Look like trauma from his royalty pig ex wife and daughter pair still there and his wife has feeling with his brother he's in dept with, now he don't know how to treat his "normal" daughter. This guy doesn't prepare to be a good father so it seem.
I hope grandma will pull Lafilia to her side...