Everyone seems to have forgotten that Ouji-kun still doesn't understand what is going on, just the Himeno-chan can get angry about weird things and doesn't want to trigger that. Plus he doesn't understand his own confusing feelings yet.
If this was the only thing I had to read I'd be disappointed, but since there are 10 billion other isekai out there keeping up with this ridiculous story isn't so bad.