What bother me the most in this story is not the misunderstandings (I have seen novel and manga way worse, but also a few better ones) but the fact that nobody seem to react in any way to his incredibly awfull potty mouth.
He is incredibly rude to anyone (even to people of similar or superior social stature), but nobody seems to care at all that this kid insult them all the time... It seriously break my suspension of disbelief to see all introduced characters act the same and is the reason I stopped reading the novel a dozen of chapters after the Sumeragi arc...
Thankfully the manga is a little tamer with this so it's annoying but still readable.