updated 05/29/2014 AT 4:00 PM ET
•originally published 05/29/2014 AT 5:15 PM ET
Joyce Giraud and Carlton Gebbia certainly stirred things up during the time they were on The Real Housewives of Beverly Hills, but maybe they caused a little too much trouble. PEOPLE has learned that the pair will not be returning to the show in season 5.
Giraud, a former Miss Puerto Rico, accused Brandi Glanville of being racist during a long-running argument during the most recent season of the hit Bravo reality show, while Gebbia clashed with Kyle Richards, who was convinced that the Wiccan had placed a curse on her.
But a source says that while they created drama this year, “Joyce and Carlton just didn t resonate as much as the other ladies did. The fans didn’t respond to them so it didn’t make sense to keep them around.”
The source says the firing of both women, who only appeared on one season of the show, “was a long time coming.”
Gebbia didn’t seem to be upset over the news, attending the premiere of Maleficent in L.A. on Wednesday and telling E! that her exit was “a blessing in disguise.”
“Joyce and Carlton are disappointed to not be returning but they understand that it’s best for the show,” the source tells PEOPLE. “They had a great run and both of them are happy with the opportunity.”
Reporting by AILI NAHAS