'X Factor' judge Louis Walsh claims Gary Barlow has swiped his rightful place as head judge on the TV talent show.