The departure came just a month after the president of the Tokyo organizing committee stepped down over a sexist remark.