Richard Branson has had to pull out of a gruelling charity swim across the Irish Sea after a fire destroyed his luxury Caribbean home.