The decision to abandon the talks came after Yahoo chief executive Terry Semel and chief finance officer Susan Decker met in late October with Time Warner executives in New York, said Yahoo spokeswoman Joanna Stevens.
Stevens said Yahoo had "politely passed" on proposed terms and "walked away from any interest in a deal."
Two people close to the discussions said a key stumbling block was Time Warner Inc.'s insistence that it retain majority ownership in the AOL unit. They spoke on condition of anonymity because public discussions of any private negotiations were contrary to their companies' policies.
One of the people, familiar with Time Warner's position, said one arrangement under discussion had called for Yahoo to pay Time Warner in stock worth $13 billion for an 80 percent stake in AOL's growing content business, which includes its Web sites and the news, music and other services featured on them.
Under that proposal, the person said, Time Warner would keep all of AOL's Internet access business, which is in decline as users abandon dial-up connections for higher-speed cable and DSL lines.
The Yahoo withdrawal, reported earlier on the Web site of The Wall Street Journal, leaves Microsoft and Google as the leading contenders, with Google possibly combining with Comcast Corp. in a bid.
The interest in AOL comes as the company transforms itself from a declining "walled garden" focused on providing dial-up access to a provider of free content that is tapping the recent boom in online advertising.
Late last year, the Dulles, Va.-based company abandoned its longtime strategy of exclusivity and began making its rich offerings, including concerts, news, sports and e-mail, available through AOL.com for free, a model Yahoo drove to become the Web's top brand.
The Web portal side of AOL's business is worth about $11.3 billion, based largely on AOL's advertising potential, according to media analyst Michael Nathanson at Sanford C. Bernstein.
A Google deal with AOL would give the Internet search leader a way to build a portal, and grow its advertising potential, while preserving an existing relationship with AOL. More than 10 percent of Google's revenues come from a partnership in which AOL uses Google's search results and the two companies share ad revenues.
A deal with AOL would be in Microsoft's interest as it could dampen competition from Google and create synergies. Microsoft's MSN online division and AOL share many businesses, including an online portal, instant messaging services and dial-up access.
There would also be considerable overlap between AOL's and Yahoo's businesses.
Yahoo, based in Sunnyvale, Calif., is the leading Web destination, according to Nielsen/NetRatings and comScore Media Metrix, as more people head to the Internet for news, entertainment, communications and other services.