It could be an interesting premise, but I'll bet it'll just turn out like all those other isekai harems where the mc becomes a wizard hermit like that one where the guy is summoned with his childhood home and it manifests as a girl that loves him, and then he meets a princess that loves him, and then he meets a dragon transformed as a little girl...that also loves him.
Female attention seems to be the only thing these isekai authors want, so why even do an isekai story? What does it say that such a goal can only be in a world of might and magic? Pigs would need to fly before you can even imagine a girl acknowledging you?