His dad, Shinji, has knowledge of the other world, with both father and son playing coy about the happenings in the other world, and Shinji refusing to outright admit he knows to Kazura. Who's to say his dad or his predecessors before him didn't have their own adventures before Kazura, and influencing their culture as a result? It's been mentioned that the house has been passed down in his family, so it's safe to say other people have gone through the gate as well, leaving behind some fingerprints