Brown's startling news conference came as the Conservatives, who won the most seats in Thursday's election but not a majority in Parliament, were already holding talks with the Liberal Democrats.
Lawmakers said those negotiations stalled earlier Monday over differences on key issues including reform of the voting system, education, and changes to the tax system.
The Liberal Democrats want Britain's voting system to be changed so that seats in Parliament more accurately reflect a party's percentage of the vote demand that most Conservatives adamantely reject.
Brown said the Labour Party, which came a distant second to the Conservatives, would begin a leadership contest to replace him while he focused on talks aimed at breaking Britain's election deadlock.
"As leader of my party I must accept that as a judgment on me," Brown said, referring to Labour's poor showing in the election.
In a statement outside his office at 10 Downing Street, Brown said Liberal Democrat leader Nick Clegg had asked to begin formal coalition talks with the Labour Party and the two could form a center-left alliance. Clegg had previously said Brown's departure would likely be a condition of any deal.
"There is a progressive majority in Britain, and I believe it could be in the interests of the whole country to form a progressive coalition government," Brown said.
Conservative leader David Cameron made no immediate comment on Brown's possibly game-changing move.
Clegg said the Liberal Democrats and Conservatives had "some very constructive talks ... and made a great deal of progress. But we haven't yet reached a comprehensive partnership agreement" after four days.
He said it was "the right thing and the responsible thing to open talks" with Labour.
Cameron's center-right Conservatives won 306 of the 650 seats in the House of Commons, 20 short of a majority. Brown's center-left Labour won 258 and the center-left Liberal Democrats took 57 seats. Other smaller parties took the rest.
Brown said he hoped a new Labour leader would be appointed at the party's annual convention in September. Foreign Secretary David Miliband and Education Secretary Ed Balls will likely be leading contenders to succeed Brown as party leader.
The pound fell nearly 1.5 cents against the dollar after Brown's statement, trading at $1.4866 late Monday, reflecting some fear of Labour's continued presence in the government.
Britain has a record 153 billion-pound ($236 billion) deficit that the Conservatives have pledged to tackle faster than Labour. But Brown said his focus during his remaining time in office would be ensuring economic recovery.
Cameron's party was to meet later Monday and the Liberal Democrats indicated they too could gather again.
Clegg clearly was facing a tough choice: Trying to overcome ideological incompatibility to broker a deal with Cameron and the Conservatives or propping up Brown's defeated Labour Party.
The Conservative Party strongly opposes voting reform, as it would likely mean fewer seats for Britain's two main parties the Conservatives and Labour and would banish the Conservatives to the political wilderness for years to come.
Yet in the last election, Clegg's party earned 23 percent of the vote but got only 9 percent of the seats in Parliament.
Brown's announcement signals an end to a political career marked by great promise, considerable achievement and ultimate disappointment.
He spent a decade as Prime Minister Tony Blair's right-hand money man, but craved the top job himself. When he finally got it in June 2007, Brown faced economic crisis, a divided party, public disgust with politicians and finally defeat in last week's election.
It was Brown's fatal political misfortune to follow the charismatic Blair. Brown was brooding and awkward by comparison, and a recent run-in with a voter whom he called a "bigoted woman" showed how much he lacked a common touch. But behind closed doors, Brown, 59, was often described as warm and agreeable.
Friends also say the son of a Church of Scotland minister is dogged, determined and fiercely loyal to Labour a trait that prompted him to offer his resignation Monday so that his party had a chance at staying in office.