The 83-year-old Carter walked into this highly volatile pro-Sudanese government town to meet refugees too frightened to attend a scheduled meeting at a nearby compound.
Carter was able to make it to a school where he met with one tribal representative and was preparing to go further into the town when Sudanese security services interrupted.
"You can't go. It's not on the program!" the local national security chief, who only gave his first name as Omar, yelled at Carter, who is in Darfur as part of a delegation of respected international figures known as "The Elders."
"We're going to anyway!" an angry Carter retorted. "You don't have the power to stop me."
U.N. officials told Carter's entourage that the Sudanese state police could bar his way. "Let's go, or somebody is going to get shot," said one U.N. official, as an increasingly tense crowd gathered. Billionaire businessman Richard Branson and Graca Machel, the wife of former South African President Nelson Mandela, tried to ease Carter's frustration as his U.S. secret service security urged him to climb into a car and leave.
"I'll tell President Bashir about this," Carter said, referring to Sudanese President Omar al-Bashir.
Carter later agreed to a compromise by which tribal representatives would be brought to him at another location later Wednesday. But the refugee delegates never showed up.
Most ethnic Africans appeared too frightened to speak in Kabkabiya, a North Darfur town that has long been a stronghold of the pro-government janjaweed militia.
Branson, who along with Machel was traveling with Carter, said some refugees had slipped notes in his pockets. "We (are) still suffering from the war as our girls are being raped on a daily basis," read one of the notes, translated from Arabic, that Branson handed to The Associated Press.
The note said that on Sept. 26, a group of girls had been raped, and a refugee had also been shot two days ago. Branson said it had been handed over by an ethnic African man.
The Darfur conflict began when ethnic African rebels took up arms against the Arab-dominated Sudanese government, accusing it of decades of neglect. Sudan's government is accused of retaliating by unleashing the janjaweed militia of Arab nomads - a charge it denies. More than 200,000 people have been killed and 2.5 million driven out of their homes in four years of violence.
The visit by "The Elders," which is headed by Nobel Peace laureates Carter and Desmond Tutu, is largely a symbolic move by a host of respected figures to push all sides to make peace in Darfur.
"We are here in Sudan because we want to listen to the voices of those who have not been heard and want to explore ways that we can lend our own voices to peace," said Tutu, in the Elders' arrival statement for the mission to Sudan.