Slavery in the LN and Manga worlds are more fetish than function. People who took slaves for something leisurely like sex/domestic labour would be very well off as the leisure class were a huge minority through most of history prior to the industrial revolution, even craftsmen would have difficulties gathering enough income to buy slaves to free their time from domestic labours. While the history and use of slaves is quite diverse and there were a wide range of uses and treatment of them the setting we get in this manga is a Frankenstein mix of tropes and fetishes.
Somehow, there is enough abundance of production and resources to supply a large social class we would expect from a more developed nation that is also deeply entrenched in a more primitive situation. Like a transplantation of the modern into the past.
To be honest, I do not dislike that awkward setting, though jarring at times the author often uses this within the story to deliver amusing moments like the MC's abuse of his skills. But the slavery stuff is done more as fetish and keeps hitting my pet peeve. =\