From what I remember it doesn't really get to be a harem. Its more like the author wanted to write about how much he could abuse the MC. At least thats how I thought of it when I read the WN.
His 'fiancee' was thrust on him randomly. At first she didn't do anything she was supposed to do except buying stuff. Not telling the son anything the father let her do anything she wanted. He had no knowledge of the situation but was scolded at every turn for it whenever he expected her to do...