In other unofficial results, Sussex County developer Glen Urquhart, who also courted the tea party vote, had a 552-vote lead over Greenville businesswoman Michele Rollins, who, like Castle, was the party's endorsed candidate.
O'Donnell's shocking victory gave new energy to the tea party movement, which targeted Castle after victories by Republican tea party candidates in the Alaska and Nevada Senate primaries.
With all precincts reporting, O'Donnell had 53 percent to 47 percent for Castle, a former two-term governor and the longest serving congressman in Delaware history.
O'Donnell's win is "terrible news overall for Republicans, whose prospects for winning back the Senate in November just took a serious nosedive," according to CBSNews.com policial reporter Brian Montopoli.
O'Donnell supporters who gathered at an Elks Lodge in Dover erupted in cheers and dancing upon learning of her victory. O'Donnell took the lead early as voting results came in and never relinquished it, prompting some of her supporters to make floor-sweeping motions while cheering, "Sweep 'em out!"
"We worked hard to be here," O'Donnell said. "I cannot thank you enough for that. The people of Delaware have spoken: No more politics as usual! I can't thank you enough, because you guys have partnered with me, believed in me from the beginning, stood with me in the face of adversity."
She thanked an army of volunteers and state tea party groups who she said were committed to a cause greater than themselves.
"We're listening to the pulse of America," said Tea Party Express chairwoman Amy Kremer, whose California-based group committed $250,000 for radio and television ads on O'Donnell's behalf. "We know Americans are fed up with party politics."
Castle supporters at his election party in Wilmington stood in stunned silence as returns rolled in, but erupted into cheers when he took the stage for a brief speech.
"The voters in the Republican primary have spoken, and I respect that decision," he said.
"I had a very nice speech prepared, hoping I would win this race," Castle said jokingly, as tearful staffers and supporters looked on.
Castle did not mention O'Donnell in his speech, and he left the room without taking questions from reporters.
Interactive Map: CBS News Election 2010 Race Ratings
While attracting enough GOP conservatives to defeat Castle, a leader of Republican moderates in Congress, O'Donnell will have a hard time defeating Democrat Chris Coons in November for the Senate seat vacated by Joe Biden after he was elected vice president.
But voters nevertheless took their chances on O'Donnell, who characterized Castle as a liberal who sides with big-spending Democrats more than he does with fellow Republicans.
"I think Castle is too liberal," said Robert Manning, 56, a design engineer from Georgetown who voted for O'Donnell.
"I think Washington has done enough damage with all this stimulus spending over the past 18 or 19 months," Manning added. "It's time to get back within our budget."
CBSNews.com Special Report: Election 2010
O'Donnell, who hasn't had a steady job in years but has instead made an avocation of running for Senate, finally won after two failed Senate bids. She came in last in a three-way GOP primary in 2006 and lost badly to Biden in 2008, when she won the endorsement of state GOP convention delegates but received virtually no help from the party.
But the Tea Party Express bolstered O'Donnell's long-shot bid this year with its pledge of $250,000 for ads.
O'Donnell and her staunchly conservative supporters characterized Castle as a liberal who often votes with Democrats in Congress while masquerading as a GOP conservative. In their words, Castle is a "RINO," a "Republican in Name Only."
They also suggested that Castle, 71, was so frail that he might die before finishing his Senate term, that he might switch parties, and that he was cheating on his wife with a man.
While ignoring O'Donnell for much of the campaign, Castle and state Republican Party eventually fired back with attack ads of their own, criticizing O'Donnell, 41, for lying about her education and record, leaving a trail of unpaid bills that included unsettled campaign debts, tax liens and a default on her mortgage, and using campaign finances for personal expenses. The GOP also filed a complaint with the Federal Election Commission accusing O'Donnell of illegally colluding with tea party supporters.