Protestors should not be allowed dictate where asylum seeker accommodation is built, Fianna Fáil have insisted.