Not allowing him to use magic isn't kind of unfair? That also has no experience value whatsoever, right?
I mean, realistically, when in combat, as a magic swordman he would always use magic. Getting stomped free by literally everyone when he can't use magic is quite unfair in my opinion, and worst, people still feel good and superior about fighting him like that.