I know the anime likes poking fun at Kazuma for being a pervy coward, but I really feel the show as a whole did a disservice not adding this part in. It shows why Kazuma is the leader of his group despite being the weakest, and it at least shows us that he can be cunning if given the chance.
But hey the anime is still amazing regardless.