Speaking to BANG Showbiz at the launch of 'The X Factor' at London's Picturehouse Central cinema, Olly admitted: "I'm a bad influence on Simon, what can I say."
Asked if they had been on any lads' nights out, Caroline said: "They do become more laddy."
Olly, who came in second place on the ITV talent show in 2010, shared: "I've brought more of a lad out of Simon. He doesn't say darling as much."
Caroline added: "You're just really comfortable around each other. I think you get funnier around him."
Olly confessed he is so comfortable around Simon, who has 18-month-old son Eric with girlfriend Lauren Silverman, that he's even started using a nickname instead of his full name.
He explained: "You know you know Simon well when you call him Si."
Meanwhile, Caroline also revealed that Rita Ora, who joins fellow new judge Nick Grimshaw and returning mentors Simon and Cheryl Fernandez-Versini on the upcoming series, is the most demanding backstage.
She said: "Rita's got a massive entourage. Yours [points to Olly] is second though. I've got one person. You bring a personal trainer, in his entourage to auditions."
Olly replied saying: "I just want to look good on TV. Rita's definitely got the biggest entourage."
However, Caroline was quick to point out that the 'Poison' hitmaker is in no way a diva, adding: "Rita's really fun. She's really nice. I didn't know her before this, she's the person you want to have a drink with after, she's that kind of girl. She's fun isn't she? She's like my new Louis [Walsh]."
'The X Factor' kicks off this Saturday at 8pm on ITV.