The Edinburgh-born musician, 37, got the lowest number of viewer votes after his performance of Bruce Springsteen’s Born To Run and was eliminated from the contest alongside camp act Diva Fever.
Lee said his mentor Louis Walsh’s efforts to change his image had “backfired”.
Asked why he was ousted, Lee said: “I don’t think it was really about my ability as much as really my place in that show; I think my experience and my professionalism.
“The truth of it is I wasn’t intimidated by Simon Cowell or by the show. It worked against me in that maybe I was just a bit too polished or a bit too ‘I’ve been there, I’ve done it’, whereas some of the other contestants clearly are a lot more amateur.”
There are 12 competitors left in the show.