A man who drifted a mile out to sea in a toy dinghy had to be rescued by a lifeboat crew as he battled against the wind and tide.