See, despite how his other eventual-wives all love him, they all also are under the idea of one man x one woman marriage only.
Even Rudeus himself, even if he cares for them and want to answer their feelings, also doesn't want to cheat.
So when Sylphy gave permission, reasoning along the line of "he's an ecchi but he's also so cool that I know he can't keep it in his pant + many women would fall for him"
Rudeus married the others eventually.
Which turn out to have significant effect