in the novel we get to see more. There are some loopholes, but you can just let it go if you think like the MC. The father is not afraid, per se, of his son. He is more in shock. His first born has always being a dumbass. He is going to give the land and title to his second son because the original's body of the MC is addicted to alcohol. He is going to be pretty much disfigured by the mc in the original story of our MC lol and then killed. The real son was supposed to die in the first chapter.
Well going back to the father. He is just shock to see that his son is behaving so out of character. He is always arrogant and dislike everything and everyone. He is just content to see that his first born is just showing a calm attitude.