Health Minister Leo Varadkar has insisted ex-Fine Gael adviser Frank Flannery is not working with him “on the side” about his future prospects, saying he has grown “very close” to Taoiseach Enda Kenny in recent months.