Migrant workers are living in fear following the Brexit vote, a Northern Ireland community organisation said.