@Kaarme
Okay, let's take the women in order.
First off were the vampire and angel, correct?
Both of them are extremely famous & infamous for their combat capabitilies.
Vampire then told the first group of high elves about the village, and they later told all the other high elves as well.
Angel got her trio of angels as well and the lizardmen.
Later, vampire's sister arrived as well. She later brought along the onis.
The few beastmen were escorted by Lasti when they traded with their village.
The mountain elves were brought along by the gatekeeper dragon.
The civil servant / demon girls brought an army (easily defeated) and then just stayed, but they lived not far away to begin with.
Wolves and spiders were the first, but they weren't part of the women group, so i'll mention them here.
Don't think i'm missing any of the groups
Excluding the civil servant girls & beastmen , the weakest ones should be the lizardmen. Who are equivalent to several elite soldiers each.
All of the other women belong to races known for their military prowess.
Most of the high elves came there only for MC's seed and a place to stay and strenghten their race, right ?
The angels obey the first angel and the 2nd vampire obeys the 1st vampire, and they're both married and given birth.
The onis obey the vampires.
The mountain elves strictly were introduced there by dragons, but...
Beastgirls are there to strenghten ties and reduce number of mouths to feed in their village.
Civil servant girls are there to 'serve' for their crimes and then decided to stay.
In fact the elder dwarves did find their way there after just hearing about the alcohol there or about other dwarves there. But the first one said he's not a lolicon and not interested in any of the women there, and frankly they're all there just for the alcohol. And Dwarves tend to be pretty good fighters if need be.
Basically, the people there made their way there through sheer strength which their famous for or were escorted by those who are.
Now with all those reasons. Explain why women from any of those groups would want to be with someone else.
The only ones I can think of are the civil servant / demon girls, because they're all nobles, and would likely have to get married sooner or later and maybe the beastgirls, because they're just too young for the MC.
And I was focusing on their strength, because as you said 'All in all, in a world plagued by wars and ungodly monsters, it would be realistic if the MC needed to build a Wall to stop eager men from immigrating into the village full of food, wine, beautiful women and where monsters don't kill the denizens but actually protect them'
That village is either home to , allied with, friends with or married into the most powerful groups of races that world knows of.