The decision leaves Sony Corp. as the sole remaining bidder for MGM's lucrative film library and its movie studio, although no sale is certain.
"As we pledged to our shareholders, we approach every potential acquisition with strict financial discipline," Time Warner chairman and chief executive Dick Parson said. "Unfortunately, Time Warner could not reach agreement with MGM at a price that would have represented a prudent use of our growing financial capacity."
MGM declined to comment on Time Warner's statement.
Time Warner has never revealed how much its offer was worth. MGM said earlier this month that media reports valuing MGM at a price as high as $5 billion were inaccurate.
A source familiar with the talks between Time Warner and MGM on Monday confirmed earlier reports that put Time Warner's bid at $4.6 billion, an all-cash deal that included the assumption of about $2 billion in MGM debt.
A source familiar with the talks between Sony and MGM said Sony had increased its bid over the weekend to $3 billion in cash and the assumption of about $2 billion in debt.
Earlier, Sony had been offering a total package that valued MGM at about $4.7 billion, including debt.
Sony's bid is more complicated, however, because it has partnered with Texas Pacific Group and Providence Equity Partners.
Sony has been negotiating with MGM since May, a process made more difficult by a snag with its partners on financing terms. At issue is how soon the partners would be bought out of their investment by Sony, sources familiar with the talks said.
Time Warner had been comfortable with its all-cash bid and was not interested in engaging in a bidding war, a source familiar with the company's thinking said Monday.
Parsons, who has often said Time Warner is interested in expanding its presence in cable television, hinted Monday that the cash it saved from the MGM bid could be used to bolster its cable presence.
Time Warner is known to be interested in bidding on the cable assets of bankrupt Adelphia Communications Corp.
"We are confident that there are other capital allocation choices that will enable us to continue to build shareholder value," Parson said.
Shares in MGM were up 31 cents, or 2.8 percent, to $11.42 in midday trading Monday on the New York Stock Exchange.