They are constantly described as being completely loyal, but then they either mess up his plans or ignore direct orders at most every turn, and it's brushed off as being"personality quirks." Like when the village was attacked, but the people he sent to guard it just watched because they like seeing death, and only protected the 2 specifically pointed out when they were literally seconds before death.
He really needs to reprimand them, or at least give them a stern talking-to. They practically worship him, and he's proven himself to be stronger than any of them individually, so the chances of turning against him are slim.