Atsushi uses very harsh reasoning but I wish someone had pointed out that no one telling him also means no on trusted him. And that lies on him, his behaviour and his prejudice.
'Irresponsible affection' is a weird accusation anyway and it seems he was just worried about losing his heir...