I don't think the brother was abusive - but I think she had a severe inferiority complex of being in her brother's shadow, coupled with heavy resentment at fate reuniting her with her golden sibling no matter how hard she tried to escape his presence.
Fi sees the visiting bachelors of her past as fate again trying to put her back on a predestined path (just like the last predestined path of always meeting her brother), and intends to fight against said fate so she can carve out her own destiny in life.