The big difference here is that Toda's just an hormones-filled teenager who fantasizes about his crush without ever trying to act upon his (pretty tame) delusions. Nakano was reasonably creeped out at first (it's kinda unrealistic that she'd never perceived similar thoughts from other people before, but this was probably the first time they were directed at her) but over time she realized Toda was more than just a pervert, and that he really appreciated her for being able to understand him; with this slow realization, she fell for him too.
On the other hand, we have a random guy on the bus which we know nothing about, except the fact that he's clearly an adult, and maybe someone with an unstable mind, who starts imagining how nice it'd be to violate her right there on the spot, and he may turn his thoughts into action at any moment.
Really, thinking this is all about looks is just plain silly.