Chris Patten said he had reluctantly come to the conclusion that a new referendum was the only way to end the Brexit "shambles".