Top concerns include the displacement of longtime residents as rents and food prices surge. Ire over the influx of well-heeled foreigners has been building for years.