The line-up has been rumoured for months, but Cowell confirmed the news yesterday.
“The combination of Nick, Cheryl, and Rita is going to be a lot of fun! They all bring a lot of enthusiasm and passion for music to the show.
“I have a feeling I am in for a bit of a rough time but sometimes you have to embrace the craziness and now, with luck, we hope to find a star.”
Ora, who had a stint as a guest judge on the show three years ago, has been seen more recently as a judge on rival BBC show The Voice.
She said she was “excited” to be returning: “This is a journey I can’t wait to embark on to find some talented singers. It’s the new generation taking over.”
It is not the first time Cowell has poached talent from his BBC rivals, notably taking Alesha Dixon from Strictly Come Dancing to Britain’s Got Talent.
He has also gone to the BBC for Grimshaw, but it is expected the DJ will stay at the helm of Radio 1’s flagship show while working on X Factor.
Grimshaw said: “I love X Factor and have watched the series religiously since it started, so obviously I’m hugely excited.”