Even if he's an ugly man (by manga standards, not RL standards) of 40 years, it's quite exaggerated how much meeting him displeases all the women. In the end, why would they care so much about it? He's just some stranger they probably have no heavier business with.