madra_b's explanation was on point. People only see him as annoying but the manga already offer us 6 chapters—including the flashbacks—so far to understand his character. It's quite painful to see how he reacts sternly about empathy and sympathy things that he couldn't understand. Really great characterization from the author! His vulnerability gaves off a distance between him and other people. It's alright I think. Even I have to disagree with what Hidaka said before, "know him properly". Whether she speak for her past self or whatever, it's good enough to take things slowly. And our Heroine capable of doing that somehow.
Guh, this gonna be a heart-wrecking romance story and this is definitely just the start of their journey *shudder*.