Yeah it seems like he fell for her/he realised he liked her that way when she was about 12 and he was 18 (yikes) which is what caused him to keep away from her and he still feels conflicted about it. It's a good conflict because even though now there's nothing wrong with them getting together...