Wait yea, guild lady was never in the party. The set-up is amusing but it doesn't make THAT much sense, I assume they are "aware" of his presence when he is behind them and late night inhibitions/trying to sleep changes the... perception? I guess in the words of Ted's mom "Nothing good happens after 2AM".