The police fired dozens of tear gas grenades to try to disperse the crowd.
Officials said the authorities had initially allowed the demonstration by thousands of young people protesting against the Scottish judges' verdict last week that one of two Libyans tried for the bombing of an airliner over Lockerbie was guilty.
A police officer told Reuters: "People are very angry...They want to storm into the British embassy to express their anger and that's why we were obliged to intervene."
Police arrested at least 30 young people after beating them with batons and took them away for questioning.
A Libyan official said another demonstration took place outside the U.N. mission in Tripoli and said police had also intervened there to disperse the crowd. He did not elaborate.
Scottish judges last week convicted Libyan intelligence agent Abdel Basset al-Megrahi of the 1988 Pan Am airliner bombing over Scotland in which 270 people were killed.
The judges, sitting in a special court in the Netherlands, handed down a life sentence on Megrahi and recommended he serve a minimum 20 years in prison. They found his co-defendant, Al-Amin Khalifa Fahima, not guilty.
The protesters massed in the Zaouyat Dahmani district where the embassy is located, carrying four coffins symbolizing Libyans who were killed when U.S. aircraft bombed Tripoli in 1986, and chanted "Down with America! Down with Great Britain!"
Describing Megrahi as "a hostage," they burned U.S. and British flags while chanting: "La Illah Illa Allah (No God but Allah)...America is the enemy of God."
"Al-Megrahi is innocent...The verdict is political and racist," they yelled.
The demonstrators urged the United Nations, the Islamic Conference Organisation (OIC), the Arab League and human rights organisations to help get al-Megrahi released, and demanded that U.N. sanctions on Libya be lifted immediately.
U.N. sanctions have been suspended since Tripoli handed over the two bombing suspects in April 1999. To impose them again would require another vote in the U.N. Security Council but Libya says it wants its name cleared.
On Monday, Gadhafi ridiculed the "triviality" of the evidence in the Lockerbie case, reading from published reports by analysts and experts who had expressed skepticism about the verdict.
However, he failed to produce any of the new evidence he had claimed would exonerate Megrahi during the 2-hour 15-minute tirade.
Holding paperwork containing the judges' decision and evidence, Gadhafi said the court acted under British and U.S. pressure.
"They said 'Let's convict one of them ... and acquit the other,"' Gadhafi said. "This will please Libya and it supporters .... and give credibility to the Western, Christian judiciary and please the families of the victims."