1. Father completely believed he slept with the mom (frankly, I'm a tad doubtful with this or I'm just forgetful.
2. It was because of worst grandmother/mother-in-law why both mother and son were mistreated (basically she was the main cause on why the father was influenced that way. There are other factors like the 2nd wife but he listens to his mother more). In mom's case, they already assumed she slept with him so she should be loyal to him according to the "root" process. Having low opinions on Ain in grandma's case was simply because she underestimated him (she wrongfully assumed his ability was weak. As an example, like the fate/Nasuverse, another person has the same ability but Ain literally has Rank EX (full immunity), grandma wrongfully assumed that Ain is like someone with the same ability (not sure if the ability has a similar name but definitely a different rank, even the simple Rank A only goes so far as resistance).
3. The father is just downright stupid. But then again, he chose the 2nd child to keep face (as he is a military man so he values combat based abilities more plus most human noblilities are unaware of said contract (remember the party where everyone was gossiping on Ain's rumored behavior). Father's just dumb/stupid. Grandma is overconfident. The kingdom... sadly lacks the necessary research or values to respect Ain's true potential and is more of a culture that values combat abilities. The father's only fault according to spoilers is disregarding MOST if not ALL that is listed to the contract/treaty.
4. 2nd wife was unaware of 1st wife's lineage and title (when she finds out later, naturally she becomes scared, anxious and stressed. Also, she has to constantly remind herself that what she did was right. Grandma's overconfidence isn't helping.
5. The brother naturally becomes spoiled and his supposed fiancee (the one he was supposed to meet at the party dumps him accdg. to spoilers). The siblings meet again after a timeskip. He becomes so arrogant and mocks Ain's group upon meeting each other again. Both nation's are at war at this point. 2nd brother challenges Ain into a duel abd laughingly loses against Ain's weakest retainer. He gets to fight Ain later and Ain has to lower his ability to not ruin what little pride he has left.