On one hand, the Guild Masters are right and clearly have been pissed with the Sword Saint for a long time. They seem to have a strong sense of responsibility with their power and authority. However, I can also see where the Sword Saint is coming from. Yes, he could have saved many lives and eased the food shortage and so on. However, he is not responsible for any of them or their city or anything else. Just because someone can help others does not mean they are obligated to do so. To say otherwise is to say that anyone who is more skilled than you is your slave. That they have to do things for you just because they're stronger or wealthier. They don't. You are not their responsibility and it is none of their business that you are weaker than them.
Imagine it if you were the stronger. Imagine that you are a good shot with a gun so the army says "Okay, we're sending you into battle. Your obligated to do so because you're a good shot and if you refuse then anyone you could have saved is on your head." Or you won the lottery and are held responsible for people being hungry because you didn't use the winnings for their benefit.
That said, if everyone did contribute to helping as many people as they can with the excess of what they have (still putting yourself first, though, as is only fair), then all of that would layer on top of each other as everyone did so and build up quickly. But, that IS what people do. That's what societies and civilizations have always done and why they exist. It is the function of society and civilization. It's what government and charities are for and why people help each other when they see someone in need. This is normal.
Basically, the problem is the Guild Masters are acting like Sword Saint owes the populace when in fact he owes them nothing as they have not contributed to him, he is entirely a self-made success in the city and tower. One could say he had help from the spirit but since that's is skill card thing it doesn't count as it's part of him. Besides, the Guild Masters talk big but I don't see them or their guilds making sacrifices for the common good, so they have no place chastising him for the same.