The 83-year-old Carter walked into this highly volatile pro-Sudanese government town to meet refugees too frightened to attend a scheduled meeting at a nearby compound.
Carter was able to make it to a school where he met with one tribal representative and was preparing to go further into the town when Sudanese security services interrupted.
"You can't go. It's not on the program!" the local national security chief, who only gave his first name as Omar, yelled at Carter, who is in Darfur as part of a delegation of respected international figures known as "The Elders."
"We're going to anyway!" an angry Carter retorted. "You don't have the power to stop me."
U.N. officials told Carter's entourage that the Sudanese state police could bar his way. "Let's go, or somebody is going to get shot," said one U.N. official, as an increasingly tense crowd gathered. Billionaire businessman Richard Branson and Graca Machel, the wife of former South African President Nelson Mandela, tried to ease Carter's frustration as his U.S. secret service security urged him to climb into a car and leave.
"I'll tell President Bashir about this," Carter said, referring to Sudanese President Omar al-Bashir.
Carter later agreed to a compromise by which tribal representatives would be brought to him at another location later Wednesday. But the refugee delegates never showed up.
Most ethnic Africans appeared too frightened to speak in Kabkabiya, a North Darfur town that has long been a stronghold of the pro-government janjaweed militia.
Branson, who along with Machel was traveling with Carter, said some refugees had slipped notes in his pockets. "We (are) still suffering from the war as our girls are being raped on a daily basis," read one of the notes, translated from Arabic, that Branson handed to The Associated Press.
The note said that on Sept. 26, a group of girls had been raped, and a refugee had also been shot two days ago. Branson said it had been handed over by an ethnic African man.
The Darfur conflict began when ethnic African rebels took up arms against the Arab-dominated Sudanese government, accusing it of decades of neglect. Sudan's government is accused of retaliating by unleashing the janjaweed militia of Arab nomads - a charge it denies. More than 200,000 people have been killed and 2.5 million driven out of their homes in four years of violence.
The visit by "The Elders," which is headed by Nobel Peace laureates Carter and Desmond Tutu, is largely a symbolic move by a host of respected figures to push all sides to make peace in Darfur.
"We are here in Sudan because we want to listen to the voices of those who have not been heard and want to explore ways that we can lend our own voices to peace," said Tutu, in the Elders' arrival statement for the mission to Sudan.
While Tutu led a group to the Otash refugee camp in South Darfur on Wednesday, the U.N. mission in Sudan deemed it too dangerous for Carter to visit a refugee camp.
The former U.S. president instead flew to a World Food Program compound in Kabkabiya, where he was supposed to meet with local community members including some ethnic African refugees.
But as the meeting was set to get under way, none of the nongovernment refugee representatives arrived, and Carter decided to walk out into the town to try to talk with them. Carter went to a school where he met with one representative before Sudanese security stopped him.
"We are in the security field. We're not that flexible," said the local national security chief, Omar, after the confrontation ended.
"This (the confrontation) illustrates the challenges that communities and humanitarian workers face in Darfur," said U.N. Mission in Sudan spokeswoman Orla Clinton, who witnessed the incident.
Carter later returned to the North Darfur capital of El Fasher and where he was planning to meet with community representatives later Wednesday.
"The Elders" delegation is trying to use their influence at a crucial time -with peace talks due to start in Libya and deployment of a 26,000-strong hybrid African Union-U.N. peacekeeping force to begin later this month.
Tensions are running high after rebels overran an AU peacekeeping base in northern Darfur, killing 10 in the deadliest attack on the beleaguered force since it arrived in the region three years ago.
Carter said Wednesday that he felt "The Elders" trip was proving effective. He said al-Bashir told him this week that Sudan has committed $100 million to a fund for Darfur's reconstruction and another $200 million has been pledged by Chinese diplomatic allies.
Carter said the main goal of three-day visit to Sudan was to seek guarantees for free and fair elections throughout the country in 2009.
Observes fear the elections could be postponed and warn that this could imperil the fragile peace in southern Sudan and worsen the conflict in Darfur.
If on time and open, the slated 2009 general elections would be the first democratic election since al-Bashir came to power in a military and Islamist coup in 1989.
Carter said during a private meeting with al-Bashir in Khartoum, the Sudanese president had vowed the elections would take place.
"If the CPA fails to fulfill its commitment to free and fair elections and democracy in this country, all other efforts will be futile," Carter said, referring to the 2005 Comprehensive Peace Agreement that ended 21 years of civil war between Sudan's Muslim government in the north and the Christian and animist rebels in the south has improved life.