Correct me if I'm wrong but I think the word on page 19 written as "wandering samurai" is supposed to be the other "Ronin" which means a student who fails an exam which I'm not sure has a word in English.
Surprised pikachu face.jpg
In all seriousness, it seems like the male lead has experienced some event with similar people in the past to invoke that kind of reaction.