Northern Ireland would be the worst hit of all economies by a Brexit, a leading thinktank has warned.