Despite a lot of people here being MAJORLY suspicious, I think the Crown Prince really does love his older brother and sister. The comment about letting his guard down is probably about being with his older brother; he's too comfortable, that's why he let his guard down.
I do feel like they have a close relationship before politics got to it. I think they're like Elise and Canaria's princes-- Personally they do like each other, but politically they are from different factions so they have to fight