I don't get what's so hard to understand here.
If he hit her while everyone was watching, things would get worse for him, because she's just a normal high school girl with rich parents.
But he can beat the shit out of her brother because he's a gangster and he can't report him lol.