A homeless man who said he was kept as a slave for 26 years in Wales has admitted telling police that working for a group of travellers "improved his life".