That said there are only 2 legal types of slavery, debt and criminal. The children do not fall into either. The merchant also sells allot of illegal black market slaves as well as legal ones. The children were illegally sold by their land lord when their mother died, and abandoned in the forest to distract some monsters while being shipped for final sale (the merchant tried to trow only one, I think the boy, but the other jumped as well not wanting to be separated), and is not in trouble because there is a noble (who is basically a huge criminal and bully) who traveled there to buy the twins, and there are no twins in stock.
The merchant is desperate, and knows there is no proof they are slaves, but still trying to con his way into getting them back.
Lets just say things do not go well for that noble, and the local lord and knights seem to enjoy taking him down. It probably did not help that he tried to destroy the bakery Takumi is friends with by cutting off their suppliers, because the baker refused to sell Takumi's cream bread recipe and think claimed it was a stolen family recipe.