Let's see then: Harem guy projected his killing intent, but because it isn't the trigger, he lives. He just told others to direct harm, therefore he dogged that bullet. The girls on the other hand, would have unleashed attacks the very next moment and became valid targets for MC's ability. The fact that killing intent most of the time goes hand in hand with willingly directing harm at others, might not be necessary to use his ability, but it is convenient for MC. He can use this fact to tell everyone, killing intent would be the trigger, to gain a tactical advantage.
Now that begs the question, what harm did all those poor doors intent to do to him and his friends? Could it be, that if the target is a non-living thing or a concept, he can use his ability freely on it?