I read the spoilers. The only way I see this as a problem is if he detaches himself so much that the servant contracts break. Leading to Tomo's hidden dimension breaking and spitting everyone out, or flat out making them disappear. Then, even if the contract is resumed after a tiny pause or started over, everyone is gone.
Is that it, or are they just really worried about him? (Maybe third unexpected option?)