Environment Minister Phil Hogan has apologised for remarks he made to a former Fine Gael secretary at a golf outing.