@RC1138Boss Speedreader-san, you must have missed the point where the harem hero slime said that he was too powerful for the barrier to stop him. Please show respect to the pimp and his power.
Any possible explanation falls apart when you consider that the book spirit was also able to pass the barrier. My theory is that the plot opened a hole in the barrier wide just enough to allow through only the ones that needed to be separated from the group.