A LOT is left out from the LN that isnt mentioned in the Manga.
First off, Clune's grandfather knew the entire situation of Olivia, and also told Clune straight up, her peerage is not up to par compared to Ain, as they fully knew that he was a direct line of the king, and if he went back could possibly be crowned a prince of a country that made your country look medieval.
So Clune thinking she originally had too much peerage, was told later on she has too little by her grandfather even. lol.
So they knew Ain was even a Prince, before he did.