Postal employment is now below 635,000, Potter said, down from about 800,000 in 1999.
Thousands of carrier routes have also been eliminated as mail volume declines, he said.
"We have an infrastructure that, quite frankly, we cannot afford based on the income we're receiving," Potter said.
The agency still faces a potential $6.5 billion loss this year, Potter said, and even with increased borrowing and other changes it could finish the year with a $1.5 billion shortfall.
Postal rates went up last week by 2 cents to 44 cents for a first class stamp. That isn't expected to be enough to offset the likely losses. Regular increases are limited to the rate of inflation the year before and officials feared a larger, emergency increase would result in even more declines in mail volume.
The post office is seeking permission from Congress to reschedule some of its contributions to a retiree health care fund, which would reduce spending this year by $2 billion.
However, Potter stressed that the main problem is the weak economy and the post office is "anxiously awaiting a turnaround."
The post office was already working to cope with a decline in first-class mail as people turned to the Internet for personal communications as well as many financial transactions as the economic downtown caused advertisers to scale back, further cutting the mail stream.
The agency is expected to handle about 170 billion items this year, well below the peak of more than 210 billion, Potter said at a briefing at the Postal Forum, a convention for the mailing industry.
Potter has also suggested to Congress the possibility that it may be necessary to reduce mail delivery from six days a week to five. However, on Monday he sought to deflect attention from that possibility.
"People should take it for granted that they're going to get mail six days a week" until they hear otherwise, he said.
"They'll hear from us if we should ever change the frequency of delivery," Potter said.
More than 10,000 city carrier routes have been eliminated over the last several years as postal officials determined they could combine routes because of the reduction in mail volume and the increasing share that is sorted in advance by mailers.
Each route that is eliminated saves the agency about $100,000, and also may mean no longer needing a vehicle, Potter said. He said the post office is working with the workers unions on the changes, which also include a hiring freeze and an early retirement offer.