if you remove the isekai part of the story, that makes 0 difference, like, absolutely no difference, there was 0 impact on the story.
The only way to it be important was if he could see his abilities and they were all 2 or lower with all combat at F and the only reason he is doing as well as he is would be the fact that he was isekaied, but that did not happened.
I think the story would be better off if it was not an isekai, what do you people think?
btw imo the girl is totally playing him, if he can surround himself with tons of people with high level of skills, later on, she just have to kill him and take his place or marry with one of his younger brothers, those 2 have super high skills anyway