More than one-in-five farmers have said the impact of climate change could lead them to quitting the farm.