There must be a part of the story missing in the manga from the WN/LN...
Did the Class President girl use [Plunder] and steal MC's [Subjugation], thus making him a slave to her? Is that why the MC can't go anywhere without her permission?
That's just a huge plothole.
If not, and that is just how the MC is, then there is a limit to how subservient one can be... You are not supposed to hang on every word coming out of someone else. You have no obligation to help anyone either, you only do so because of the goodness of your heart which also has a limit to it.