Then yes, it does all go to complete shit and it turns into a pervert hanky-panky 'great demon lord' manga where he masters all magic known in the world, has millions (not even kidding) of hit points (big xp gains from destroying a country, apparently) and is no longer a 'barrier master' manga.