In ch 101 all the waifu squad basically agreed it's his fault.
And it's the dwarf who's explaining why those newcomers are reacting so extremely like that.
First he pretend there's a scenario where the MC want to kick the dwarves out.
If so, he has to consider that if he kick them out, there'll be no more drinks made by them and the others would be dissatisfy causing instability.
So rather than kicking them out, it's better to leave them here.
Thus, in a sense the dwarves are paying for their stay here by making alcohols.
And there's a panel giving a rundown of how the current residents have some such work. (Hakuren is lumped with the spiders and wolves on the 'no need to say' group lol)
like the elves being in charge of hunting/construction
mountain elves blacksmithing
But basically everyone in the taiju village all had, or made, a good reason for them to be able to stay.
Those newcomers don't feel they have such thing so they're worried whether one day he'll change his mind and kick them out.