Its ok. His relationship with his twin brother is fine. He did it on purpose becoming a dull prince to avoid any faction or any mess that might happen between him and his twin brother. There is also reason that he is uninterested with the emperor stuff too, so he is helping his twin brother behind his back using the adventure name "Silver". Its a harem series, it will grow.