Some brains were removed to help diagnose cause of death, while others were collected for neuropsychiatric research, Dr. Jeremy Metters, Her Majesty's Inspector of Anatomy, said in a report that followed an 18-month investigation.
In one case, a hospital mortician was paid $16 for each brain he provided for research. In another, more than 200 brains were kept for a Manchester University research project that studied the brains of people who had suffered from mental illness and those of healthy "controls" for comparison.
Removing organs without families' permission was explicitly outlawed in 1999. A law passed in 1961 said organs should only be taken from corpses if relatives did not object, but Metters said permission often was not sought.
Metters said many of those involved in retaining organs didn't know they were breaking the law.
"There was a widespread belief that organ and tissue retention was in the 'public interest' and that retention was lawful because the post mortem was carried out for the coroner," he wrote.
The 20,000 figure may be far lower than the actual number of brains taken. It includes only brains still held by hospitals and universities in England, and Metters said many more could have been examined and destroyed.
The government ordered the investigation after a woman learned her husband's brain had been removed without her permission when he committed suicide in 1987. The couple's Jewish faith decrees a person must be buried intact.
Elaine Isaacs said the secrecy surrounding the case of her husband, Cyril, was "nothing short of collusion," and she expected authorities to take action against those responsible.
"I'm very, very angry about being here today," said Isaacs, who attended the launch of the report. "I've had to fight so hard to get this whole situation recognized."
"Our rights were ignored in every sense," she said.
The government's chief medical officer, Sir Liam Donaldson, said removing organs or tissues at post mortem examination without lawful authority "is an affront to families who have lost a loved one."
Metters made 34 recommendations, including greater openness by those involved in post mortems. He said public confidence for the lawful removal of organs and study of tissue must be restored.
"There is some research, important research, which cannot be carried out in any other way than by research on the brain," he said. "It would be a tragedy if my report was to undermine lawful post mortem research that has the full consent of relatives."
But representatives of mental health organizations said they feared the report had done exactly that.
Marjorie Wallace, chief executive of the mental health group SANE, said "the emotion surrounding this and previous inquiries has already almost killed post-mortem research into illnesses such as schizophrenia and depression.
"There is an acute shortage of brain tissue and many of those carrying out the research feel so stigmatized that they have ceased investigations."
The removal of organs without consent was outlawed after it was discovered that 3,500 organs from babies and children had been retained by the Royal Liverpool Children's Hospital in northwest England without the parents' knowledge.