The 'X Factor' host has revealed the 55-year-old music mogul and show boss has been misbehaving more since he and his former 'Xtra Factor' colleague Caroline Flack joined the main show earlier this year

"If Grimshaw is in, I'm out"