Kazuya acting like King Simp, Ruka's petulance and semi-coercion, Chizuru's hard heart, the agonizing awkwardness of the now-forgotten Friend Who Makes MC Look Better By Contrast (what was his name, anyway? the one who rented Ruka), or anything else. The manga's biggest flaw is that it's an eroge with Kazuya as the protagonist.
The reason it's hard to empathize with the other characters is that they're never presented as people - they're elements of an eroge. Anything aside from their desirability as goals, or their utility toward (or obstruction of) the protagonist's desired goal, is inconsequential. Actual character development is rare, except as a side effect of developing the above.
Look at how quickly people seize on adorable moments like Gamer Grandma... that which is rare becomes precious. And look at how quickly the hardcore stans seize on their favorite "end" girl but can rarely make a solid case for the adjectives they attribute to her. We don't really get to know any of them as people, just snapshots.
One last thought: To me, a great storyteller goes very light on telling you who characters are
†. A great storyteller lays off the adjectives and lets us construct our own picture from the nuances of what they
show us‡.
† - Having background characters constantly remind us how beautiful the girls are is one example, but an extreme one - and to me, that goes just as much toward the whole "they primarily exist as objects of male desire and/or a game to be 'won', not as people".
‡ - Your mileage may vary on whether Freddie Einsford-Hill in this clip is actually Kazuya's great-grandfather. Certainly they're cast from the same template. (^_~)
