I know right. The only reason I can think of is that he didn't want to let them know the little elf witch proclaimed to his wife. The girls would considered that cheating regardless if he declares it isn't so. Or maybe he didn't want to get the girls jealous because he was spending time with another girl?