This manga is so fucking cliche I predicted the outcome of the last 20 pages within the first 3. The party wasn't retarded. They were using flawed common sense and also, the main character didn't even try to talk back and simply followed those "retarded" orders. He was the one casting the spells he could've easily had just started to try casting other spells. He noticed his spells weren't doing anything but he continued to follow their orders. That makes him as retarded as the leader of the team.
This main character doesn't know how to communicate with others. And I agree with that warrior girl he was a coward in not fighting back. He just thought about what he would say to them.