I have read the story, its pretty good for the most part. As for vengeance, he simply does not care about most of the others until very late in the story. Some, he starts caring for about 1/3 of the way through, but that is for other reasons than vengeance.
However, there are 3 points that you should know:
1) He never really swears vengeance against any female iirc, and almost any male for that matter (unless they personally make an effort to hurt him or his select few people).
2) There is only 1 person who he is abundantly lenient with and it is only because he cares about how killing him would affect his select few people he cares about. Believe me, he would have destroyed him on the spot the first time he cause an issue.
3) He does not punish women by grabbing their tits (well... only 1 women, and its more her ass/entire person, and she's an uber masochist who loves / begs for it). At worst he either messes with them jokingly if they are friendly, or flat out kills them if they get in his way.