David Miliband told Parliament he had taken the steps because the Kremlin had failed to properly respond to the "horrifying and lingering" death of Alexander Litvinenko.
It was the first time since 1996 that Britain had used the sanction, which was likely to be met with retaliation from Moscow.
"The Russian government has failed to register either how seriously we treat this case or the seriousness of the issues involved, despite lobbying at the highest level and clear explanations of our need for a satisfactory response," Miliband told lawmakers at the House of Commons.
Moscow has refused to extradite Andrei Lugovoi, a Russian businessman and former KGB agent, to stand trial in London over the killing. Lugovoi has been named by British prosecutors as the chief suspect in the case.
Russia's formal rejection was received a week ago by Britain's Crown Prosecution Service, which in turn spurned a Russian offer to try Lugovoi in Russia.
"The heinous crime of murder does require justice," Miliband said. "This response is proportional and it is clear at whom it is aimed."
Britain's Foreign Office declined to specify the rank or position of the four Russian diplomats to be expelled, who had yet to leave the country.
"We have chosen to expel four particular diplomats in order to send a clear and proportionate signal about the seriousness of this case," Miliband said.
Responding to the news at a special briefing, Russian Foreign Ministry spokesman Mikhail Kamynin told journalists, "The position of the British authorities is immoral and will entail serious consequences for Russian-U.K. relations."
"I would like to remind you that recently the British authorities have consistently refused to extradite citizens of other countries who are staying on British territory and who are accused of involvement in crimes. It seems to us that London's position is immoral, given this background," he said.
Kamynin said British authorities are trying to justify themselves in the eyes of the international community for their refusal to work with Russian law enforcement services regarding the extradition of Boris Berezovsky and Akhmed Zakayev to Russia.
Berezovsky, a Russian businessman and critic of President Vladimir Putin who received political asylum in London, is currently being tried in absentia for embezzling 214 million rubles from Aeroflot. He is also wanted in Brazil on money-laundering charges.that Berezovsky supplied sensitive information about Russia to British intelligence agents.
Akhmed Zakayev was commander of a Chechen resistance group and, later, representative of Chechen President Aslan Maskhadov. He is in exile in the U.K.
CBS News Moscow bureau chief Svetlana Berdnikova reports some Russian officials are suggesting tit-for-tat, calling for the expulsion of British diplomats from their country.
"Our country, too, should deport British diplomats," First Deputy Head of the Duma Committee on Security Mikhail Grishankov said. "As for changes in the visa issuing procedures, regrettably, they will hit ordinary Russians," he told Interfax.