The departures were the latest sign of opposition among the Japanese public to hosting the Games during the pandemic.