I think he mean when first transported to the Immortal World where he may be considered a trash compared to mortal world?
If you look at the flashback, you can the he was bullied after he was waving to another person (when he embarked on his journey) so I am assuming the bullying was after he disappeared
This story is lacking something; I can't really tell exactly what is. I'll read a couple of chapters before decide either drop or continue reading this.