I think I've got why the MC annoys me now.
He makes no effort to incorporate the world's common sense into his knowledge. No matter what people tell him, he's basically ignoring all of it.
Granted, he can ignore the dwarf's advice and just go into the cave because slimes can't touch him, but that's not really the point.
Even if he doesn't bother obeying their common sense, he's just being a contrarian shit by not taking that viewpoint into account when he interacts with others.