Is that if the father is the one who used the time magic (or technically the Goddess at the pleading of the father), this guy is someone who realized time got turned back.
Given his lines at the start of this chapter, he probably can't remember the future events either but the knowledge that something happened that require time turning back to 'fix' is bothering him.
He probably noticed the trace of magic or something to that effect on her and assumed she is the one who turned back time.