updated 05/03/2014 AT 8:00 AM ET
•originally published 05/03/2014 AT 9:30 AM ET
The cards were in Ben Affleck’s favor one too many times.
The actor was banned from playing blackjack at the Hard Rock Hotel & Casino in Las Vegas on Tuesday night, a source confirms to PEOPLE.
After multiple wins, security caught Affleck, 41, counting cards and reportedly swarmed the table, according to TMZ.
“The security said, ‘We have to stop you because you’re too good,’ ” the source tells PEOPLE.
Despite reports, Affleck was only banned from the blackjack table and not the entire casino. “The hotel was really nice and told him he could play other games,” the source adds.
In a statement issued on Friday night, the establishment said, “Mr. Affleck, a valued guest of Hard Rock Hotel & Casino, is not banned from our property and is welcome back any time.”
Reporting by LIZ McNEIL and MARK GRAY