Environment Minister Phil Hogan has been accused of “pandering to prejudice” after telling constituents a Traveller family would not be housed in their area.