I think they'll swapped their body often. Either by accident or intentional. And then ML will see how FL is often treated. He need to see what happened to her before he can redeem himself as a husband.
Finally, some sense in my manga. Always wonder why people think it's okay for the prince to get away with this shit. It's adultery on plain sight and there's no way your royal family will approve your no name lover instead of carefully selected fiancee.