Leo Varadkar has said his personal and family life should not be an “issue” in the Fine Gael leadership campaign.