In certain fields/skills/hobbies, there does seem to be a stigma against people who get into them late, and it also comes from the people who want to get into them themselves.
What Inori's mother said at the table was pretty fucked up. Sure, I can understand how she feels, and she explained her reasons later on. But I can't fathom what could've driven her to mock her own child in front of other people no less.