Shiori is often a temporary member of parties. Her reputation for making things comfortable during quests is well known at this point.
The previous guild master at the time suggested that she permanently join a party, that's Akatsuki. At first, the party worked well, they gained success, and Shiori seemed to be getting closer to the magic swordsman of the party. But when the party starts challenging more difficult jobs, and their success rate started to fall. There's the difference in equipment between Shiori and Akatsuki, her still using old equipment while the others using new ones. A few months after that, the party is outright not even hiding their mistreatment of her. It was clear that she was being exploited and that her appearance looked emaciated. The previous guild master didn't care about Shiori and didn't bother with the complaints about her mistreatment. But Akatsuki couldn't let her go because they've purposely pushed most of the food and supply expenses on her, left her with less rewards while they took more, Shiori is forced to use her savings to pay for the expenses to the point where she's practically broke(since she's trying to pay for a group of adventurers in their prime that eat and drink so much with her meager amounts of money. This leads her to not eating because she can't afford it. The members of Akatsuki basically got away with this because they kept evaluating her contributions as being poor because she isn't contributing during fighting, and the previous guild master was in on it.
Then she got left behind because the Imperial noble who requested their party told them to leave her behind when she collapsed. The party went along with that.
The reason why this is such a big thing for Shiori are these points:
One: She deeply misses Earth, being sent to isekai without a way back means she lost her place.
Two: Her desperately learning the language and writing was a way to try and find a place in her new world. But due to the new world's rules and common sense. She would have a difficult time.
Three: Her being an adventurer was trying to find a way to survive.
Four: Her being in a party would mean perhaps she maybe found her new place in the new world but ends up turning out very badly in the end.
The whole men are fearful for trying to go after Shiori is an assumption based on the details, they think that she and the magic swordsman were lovers. And the men think that it's a bad idea to try and court her only a few months after the supposed after all the abuse and supposedly broken heart. And think that her current state is very fragile and why they treat her like she's glass.