Mr. Bush's spokesman David Sherzer said the two-term president was informed Friday by the United Israel Appeal that his Feb. 12 dinner speech in Geneva had been called off.
"We regret that the speech has been canceled," Sherzer told The Associated Press in an e-mail Saturday. "President Bush was looking forward to speaking about freedom and offering reflections from his time in office."
Saturday's edition of Swiss daily Tribune de Geneve cited the Jewish charity's lawyer, Robert Equey, saying the visit was canceled because of the risk that protests by left-wing groups could result in violence.
"The calls to demonstrate were sliding into dangerous terrain," Equey told the newspaper. "The organizers claimed to be able to maintain order, but warned they could not be held responsible for any outbursts."
Protest organizers had called for participants to each bring a shoe to the rally outside the lakeside Hotel Wilson - named after President Bush's predecessor Woodrow Wilson - where the dinner was to be held. The shoe was meant to recall the moment an Iraqi journalist threw his footwear at Mr. Bush during a news conference in 2008.
Equey told Tribune de Geneve that attempts by human rights groups to submit legal complaints against Mr. Bush to Swiss prosecutors hadn't played a part in the decision to cancel the visit.
Sherzer, Bush's spokesman, declined to comment on the reasons for the cancellation.
Several human rights groups including Amnesty International and the New York-based Center for Constitutional Rights had planned to ask Swiss prosecutors to open a criminal investigation against the former president over the admission that he personally authorized the waterboarding of terrorism suspects.
"Whatever Bush or his hosts say, we have no doubt he canceled his trip to avoid our case," the Center for Constitutional Rights and others said in a statement.
Legal experts say it is unlikely Swiss prosecutors would have had the time to examine any criminal complaint against Mr. Bush and take action, such as requesting him to respond to the allegations, before he left Switzerland again.
Furthermore, an initial assessment by the Swiss Justice Ministry concluded that President Bush would have enjoyed immunity from prosecution for any actions taken while in office, ministry spokesman Folco Galli told the AP.
Widney Brown, Amnesty's senior director of international law and policy, said the group would continue to press for President Bush's prosecution next time the former president travels to a country that has committed to prosecuting war crimes and where he could expect a fair trial.
Sherzer said Mr. Bush has made several trips outside of the United States since leaving office two years ago, including to South Korea, China, Japan, Brazil, Canada, and the Middle East.