So just like the issue with the latrines and the labor, this one apparently is caused by gov't neglect too (abandoned mines, no extermination quests).
I thought this was just isekai janitor with the unusual connection with a duke, but we're staring to see the bigger picture.