I was hoping for a hero who actually has to try hard to over come their challenges. Sure, this guy has to train super hard, but that part is pretty much skipped over, and all we see is the end result. We don't get to see the steps he takes to get op, the incrementally harder battles he has to face, the challenges that actually challenge him.
Instead he goes from facing a few goblins to taking down the most powerful swordsman in the world (except his mother, though he beats her as well). And where does he go from here? And while he is "starting over" with magic, he's already stupid powerful there as well. Plus if all else fails he's still got his sword skills. I just don't connect with his struggles at all.
If I were to suggest an improvement, it would be to expand on his rivalry with the one girl. Contrast her skills with his hard work and natural talent. And then of course let it take longer for him to get op.