In the original books, he didn't have an immunity from being around Kuro (remember, he barely gets to see Bertia before she comes to the school), it's just that the mind control/altering effect isn't something it puts out constantly.
They do eventually make earrings (or rings, or pendants, can't remember which) that contain some of Kuro's magic that grant them immunity so long as they are worn, and the Prince has all of their circle of friends, male and female, wear them (besides Bertia, since he can't reveal his plan to her and she doesn't really need one of the earrings due to simply having Kuro herself around)