Nagi's real mother looks similar to Segawa. The best theory I can think of is that she was raised as a boy. And seeing how Nagi is in love with Segawa, its possible his father is trying to get his affection away from his real sister.
https://i.imgur.com/CtkujNV.jpg
@Elmousse007 Yeah, and it's selfish for a really dumb reason ("he's not my blood brother so we can date if I ruin everything, right?"). Doesn't help that incest is disgusting, on top of that plot being forced as an option.