This might be a "guess I didn't trust her as much as I thought" type of plot thread. People have trouble understanding their own hearts. None of his inner monologues ever talked about doubting her. He always accepted her past but, he might not have realized he was burying his trust issues. That...
Looking back the time Ichikawa spent away from Yamada was clearly a wake up call for her. Suddenly being ghosted by him opened an Ichikawa shaped hole in her Ichikawa shaped heart.