The 'Keeping Up With the Kardashians' stars have distanced themselves from their stepparent, who was formerly known as Bruce Jenner, and could be "done with her for good" because of the "hurtful" things she said about their mother Kris Jenner.
A source said: "There's not a lot of love for Caitlyn.
"They are so incensed that they may be done with Caitlyn for good. Everyone has turned their backs on her.
"Vanity Fair was the start of the downfall. It came out and they looked like idiots, they first saw the article on Twitter!"
Things became intense between the sisters and Caitlyn when she started bad-mouthing Kris in her infamous Vanity Fair cover feature and the sisters were "furious" about it.
The source told US Weekly magazine: "The Kardashians were furious once they realised she took jabs at Kris, a lot of times she wasn't very nice."
Kim previously confronted Caitlyn about the "unfair" comments the former Olympian said about Kris on her reality show 'I Am Cait'.
Kim said: "I read the Vanity Fair as well, and I think there are some things that you said that you might not realise are hurtful.
"Then it said, 'Had Kris been accepting to who I am, we still would be together.' And that's the most unfair thing in the world to say."
The last time the whole family came together was for Kylie Jenner's 18th birthday earlier this month and they apparently haven't communicated since.