I’m guessing this was before she knew who the 9th Prince was. At the time, she only knew him as a random injured man, and he didn’t return until years later.
This event is probably the reason why he would read at the library from time to time, since he knew that she would probably be hiding there.
She does remember it, read the previous chapter again, this flashback was her flashback not the nanny's. She even thought to herself: "Ah! He was the person from 5 years ago..." in the panel right before the flashback.