An outspoken Catholic priest has defended himself after being condemned for labelling poor people “lying, drug-abusing illegitimates”.