Allen conceded at an afternoon news conference, saying the "owners of government have spoken, and I respect their decision."
"The Bible teaches us there is a time and place for everything, and today I called and congratulated Jim Webb," he said.
Webb, a former Republican and Navy secretary under President Reagan, claimed victory early Wednesday after election returns showed him with a narrow lead of about 7,200 votes out of 2.37 million ballots cast.
Allen was told by his advisers that the numbers just weren't there and a recount — which could have dragged things out for weeks — wouldn't help him, CBS News Radio correspondent Bob Fuss reports.
Virginia has had two statewide recounts in modern history, resulting in changes of only 37 votes last year and 113 votes in 1989.
After Republican Sen. Conrad Burns was defeated in Montana, the Virginia contest was the last undecided Senate race in the country, and Webb's victory tipped the scales, giving the Democrats control of 51 Senate seats and majorities in both the House and Senate for the first time since 1994.
Allen, 54, son of a Hall of Fame football coach, served as governor in the 1990s and was popular for abolishing parole and instituting other conservative reforms. In 2000, he knocked off two-term Democratic Sen. Charles Robb and won plaudits in the GOP for what some considered a Reagan-like ability to tout a conservative message in an upbeat manner.
Just a few months ago, Allen had a big lead in his reelection race and was seen as a rising star in the Republican Party. He was being touted as a possible presidential candidate, Fuss reports.
But in Webb, he faced an unconventional challenger. Supporters drafted Webb, a political neophyte, to run because of his early opposition to the Iraq war.
Allen was comfortably ahead in polls until August, when he mockingly referred to a Webb campaign volunteer of Indian descent as "macaca," a term regarded by some as a racial slur. The incident, caught on videotape, became international news. Some former football teammates from the University of Virginia also charged that Allen had commonly used a slur for blacks — something he denied.
Webb, a 60-year-old Naval Academy graduate and decorated Vietnam veteran, tried to tie Allen to President Bush and the war during the campaign. He also seized the Reagan edge, having served in the former president's administration, and used a video in ads that showed Reagan praising him.
Moving swiftly to establish himself as the winner, Webb early Wednesday began assembling a transition team shortly after he proclaimed victory.
"The vote's been counted, and Jim won," said Webb campaign spokeswoman Kristian Denny Todd. Some absentee ballots remained to be counted, she said, but Webb considered it "a formality more than anything else."
The sweep of Congress clears the way for Sen. Harry Reid, D-Nev., to become Senate majority leader and Rep. Nancy Pelosi, D-Calif., to become the first female Speaker of the House.
Pelosi and President Bush have had a contentious relationship. However both seemed onafter a luncheon meeting at the White House Thursday.
"When you win, you have a responsibility to do the best you can for the country," Mr. Bush said, with Vice President Dick Cheney sitting glumly on a couch to his left. "We won't agree on every issue, but we do agree that we love America."
"We both extended the hand of friendship and partnership to solve the problems facing our country," added Pelosi, like the president, eagerly leaning forward in her chair. "We have our differences and we will debate them ... but we will do so in a way that gets results."
Speaking in the White House Rose Garden after meeting with his Cabinet on Thursday, Mr. Bush called on all Republicans and Democrats to put the elections behind them and work on the issues facing the country.
Among them is Iraq, and Mr. Bush said he's "open to any idea or suggestion."