Rumsfeld also called on Europe to provide a "master plan" to Afghanistan to help curb its massive drug trade, which has seen heroin flood Europe and Russia.
At a joint news conference with Afghan President Hamid Karzai, Rumsfeld said militants "don't want to see a country like Afghanistan have a successful democracy. They won't succeed."
Rumsfeld, who arrived unannounced in the Afghan capital, also said regional cooperation with Afghanistan's neighbors to stop Taliban and al Qaeda movements from entering the country "has been helpful, but it has not reduced cross-border" activity.
Rumsfeld did not single out any particular country for criticism, but said dealings between the U.S. military and Afghanistan with neighboring Pakistan has improved and is ongoing.
Karzai said the weakness of Afghan police forces, particularly along the Pakistan border, has allowed terrorists to increase their activities.
"We have no strong police force in the villages, so the reform of the police and the strengthening of the police force will go a long way in improving the situation," Karzai said, adding that a continued U.S. role here was "critical."
Many Afghans accuse Pakistan of not doing enough to stop militants using its territory as a base to stage attacks inside Afghanistan. Pakistan denies such claims and says it has deployed 90,000 troops to its side of the border to stamp out pro-Taliban militants operating there.
On Afghanistan's massive opium trade, Rumsfeld said the drug's spread was a "danger" not just for Afghanistan but for Europe where opium smuggled out of this country is in wide use.
"It seems to me that it is important for them (Europe) to recognize that it is a lot cheaper ... for them to assist the Afghan government in providing master over-all plan for dealing with the counter-narcotic effort," Rumsfeld said.
Afghanistan produces more than 90 percent of the world's opium and heroin supply. Many of the drugs are thought to be smuggled through Iran and Pakistan to Europe and elsewhere.
Despite the international community pumping hundreds of millions of dollars into anti-drug campaigns, opium production is increasing across the country, particularly in southern Taliban hotbeds, where the militant group is believed to benefit from income generated from the illicit trade.
Rumsfeld's remarks at a news conference came hours after the top U.S. commander in Afghanistan said the rise of Taliban resistance in southern Afghanistan reflects the weakness of the government more than a strengthening of the fundamentalist movement that once sheltered Osama bin Laden.
"The areas that the Taliban is operating in are areas that the government of Afghanistan has not heretofore had the strength and had the presence," Lt. Gen. Karl Eikenberry told reporters before boarding a plane to fly here with Rumsfeld from Dushanbe, Tajikistan.
Rumsfeld had said previously that he is concerned that the large-scale drug trafficking in Afghanistan is helping fuel the Taliban's comeback, but has also has expressed confidence that NATO, with U.S. participation, is capable of handling the violence while mainly U.S. troops hunt down al Qaeda fighters and look for bin Laden.
Eikenberry would not discuss any plans to increase or decrease U.S. troop levels, which now stand at about 23,000. Last December, on his most recent previous visit to Kabul, Rumsfeld announced that U.S. troop levels would drop from 19,000 at that time to 16,000 by this summer.
In coming weeks, NATO is set to take over command of the international security forces in the south from the U.S.-led coalition, although U.S. troops will remain. Canadian, British and Dutch troops are deploying in the region.
Rumsfeld told reporters en route here that the Pentagon has no interest in setting up more permanent bases in Central Asia. But he noted that other basing arrangements are needed to support military activities in Afghanistan.