At least 1.8 million trips were wrongly charged at the suburban rate, which is double the rate within city limits, the commission said.
The city has about 48,300 licensed cabbies, and data shows that 35,558 have illegally charged a rider at least once, the city said. A smaller group of drivers is responsible for the majority of overcharged trips - 3,000 cabbies were found to have doubled the meter rate more than 100 times.
The commission has referred its findings to the Department of Investigation.
"Some of these people could face serious charges," Mayor Michael Bloomberg said. "Now, how we would ever recoup the money and get it back to the individual payers, no, but we can stop the practice and we can make sure there's penalties."
The scammed passengers overpaid by an average of $4.45 per trip, the commission found.
Officials discovered the discrepancy by scouring data from global-positioning devices that are required in the city's yellow cabs. The data goes back 26 months because GPS was first required in 2007.
A passenger complaint last year led the commission to find one driver engaging in the scam hundreds of times in one month, they said.
The city said that while 1.8 million overcharged trips is a significant number, there were 361 million taxi trips in the past two years, so the illegal fare was charged in half of one percent of all rides.
A taxi driver advocacy group cautioned that the scam appears so widespread that it might actually be the fault of problems with the technology, not deliberately dishonest drivers.
"There should be a thorough investigation before judgment is cast on an entire work force," said Bhairavi Desai of the New York Taxi Workers Alliance.
In a few weeks, taxi riders will see an alert on the television screen in the back seat when the higher rate code has been activated.