The guy is straight up a coward right there. 5 years? Anything could have possible happened there and I doubt he would actually overthrow the duke and get rid of his wife. He might even take in more wives under the pretense of being "pressured".
And I want to know, why on earth did she even...