he's not closed with them; literally they're just there to keep the ministers in check and other political purposes. Even Eun So-Hyun is just there to keep her father in check (and he never wants her to be the Empress). He had to have concubines, too, cuz it's the custom and to make those ministers shut up about making heirs and shit.