Funnily enough, that's also what came to my mind as I imagined him talking to others: correcting people after that would be uncomfortable as he would need to answer "What about Mielle then?" and probably has nothing to say about it. Not to mention she is still his biological daughter and Aria didn't have so much time to coddle him up, so it's natural that he would protect her rep. Still, I am glad that he feels sorry, and I do hope that he is more careful next time. Unlike his daughter, he doesn't strike me as a malicious man. A gutless fool maybe, but not evil.