Though harem hijinks are much lower priority than "just because my numbers look low people think I'm inferior even though I keep outperforming the strongest/smartest people around me", there are at least half a dozen girls in love with the protagonist by the end of the first school year. And he's the dense kind of harem protagonist. And while everyone will have noticed the protagonist is actually not weak at all by then, lots of people still look down on him or hate him for various other reasons, like him being a commoner or being close to the top beauties in school.