Theodore's father is not bad or a perfect jerk, if I'm not mistaken the former head of the Gartner family (Theodore's grandfather) had promised to unite the family with the Broderick family (Catherine's family), a political marriage then... The current earl met Theodore's mother when they were young and fell in love. Lisa knew about the Gartner County situation and accepted it, which is why they originally lived in a separate house from the main family, until Lisa died defeating a demon.
Count Gartner had to marry Byron's mother for his obligation as a noble and for not staining the honor of his own father who made the arrangements for the marriage, on the other hand the house of Marquis Broderic has a lot of influence and at some point the Gartner family were "in their debt" in the past or something like that. Regardless of all that, I agree that Theodore's father is stupid for not realizing how his son was treated after he brought him into the main house.