People. He the worst trouble "was," past tense. The first time he had to go, he had to throw open all the doors to figure out where the bathroom was, implied to be shortly after he first arrived unless he has an iron bladder.
Now, in the present, he's wondering if Mary uses the bathroom because he's never seen her go for the entire duration of the manga up until now.