The leftist leader, who has long railed against the Washington-based lenders, said "we will no longer have to go to Washington, neither to the IMF nor the World Bank, not to anyone."
Chavez said he wanted to formalize Venezuela's exit from the IMF and World Bank "tonight and ask them to return what they owe us."
"We have a few bucks there," he said, apparently referring to contributions that Venezuela and other members regularly pay to the World Bank and IMF.
Venezuela recently repaid its debts to the World Bank five years ahead of schedule. It paid off all its debts to the IMF shortly after Chavez first took office in 1999. The IMF closed its offices in Venezuela late last year.
Chavez made the announcement a day after telling a meeting of allied leaders that Latin America overall would be better off without the U.S.-backed World Bank or IMF.
Chavez intends to set up a new lender run by Latin American nations which he has called the "Bank of the South," and has pledged to support it with Venezuela's booming oil revenues. The regional lender would then dole out financing for state projects across Latin America.
Separately, Chavez also offered on Sunday to contribute $250 million to a new regional cooperation fund.
He has often blamed the lending policies of the World Bank and IMF for perpetuating poverty, and has criticized past Venezuelan governments for signing agreements with the IMF to restructure the economy, plans that were blamed for contributing to racing inflation.
Under former Carlos Andres Perez in 1989, violent protests broke out in Caracas in response to IMF austerity measures that brought a hike in subsidized gasoline prices and public transport fares.
Enraged people took over the streets in violence that killed at least 300 people, and possibly many more. The riots came to be known as the "Caracazo," and Chavez often refers to it as a rebellion against the status quo.
During Sunday's talks with leaders from Bolivia, Nicaragua, Cuba and Haiti, Chavez predicted that "sooner or later, those institutions will fall due to their own weight."
"They will wear away — the International Monetary Fund, the World Bank and all those institutions," Chavez said.
Bolivian President Evo Morales raised complaints about the International Center for Settlement of Investment Disputes, a World Bank body that mediates disputes between governments and foreign investors. He said governments never seem to win their disputes against transnational companies there.
Chavez suggested that Latin countries could instead create their own arbitration body for disputes with big companies.
Venezuela is not the only country in the region distancing itself from international lenders.
Nicaraguan President Daniel Ortega said Sunday that he hopes to "get out of that prison" of IMF debt and that "we are negotiating with the Fund to leave the Fund."
Ecuador's leftist president, Rafael Correa, recently asked the World Bank's representative there to leave and said Ecuador paid off its debt to the IMF. Argentina also has paid back billions of dollars to the IMF.