Benedict made the comments during an audience with German pilgrims, who gave a hero's welcome to the first German pope in centuries.
The man who had been known as a stern enforcer of the faith seemed almost awe struck with his first encounter with pilgrims after his inauguration, reports CBS News Correspondent Allen Pizzey.
A smiling Benedict moved down lines of well-wishers, shaking the pilgrims' hands and blessing a child given to him before taking his seat in the audience hall and starting his speech.
Speaking in his native German, Benedict told the audience that at one point during the conclave, when it became clear he was garnering many votes, a cardinal slipped him a note of paper reminding him what he had preached about Christ calling Peter to follow him even if he didn't want to go.
Benedict, 78, said he had hoped to spend his last years living quietly and peacefully.
"At a certain point, I prayed to God, 'Please don't do this to me,'" he recalled. "Evidently, this time he didn't listen to me."
He apologized for being late, saying the meeting with the religious leaders had been "heartfelt" and had run late.
"The Germans are used to punctuality," he joked. "I'm already very Italian."
Benedict was elected pope April 19 after four rounds of balloting in 24 hours, one of the fastest elections in a century. He had gone in as a leading candidate, but at 78 he was considered old to be named pope.
On Monday he also met with members of the Muslim community, assuring them that the church wanted to continue building "bridges of friendship" that he said could foster peace in the world.
Benedict made the comments in a meeting with religious leaders who had attended his installation ceremony, saying he was particularly grateful that members of the Muslim community were present. "I express my appreciation for the growth of dialogue between Muslims and Christians, both at the local and international level," he said.
He noted that the world is currently marked by conflicts but said it longs for peace.
"Yet peace is also a duty to which all peoples must be committed, especially those who profess to belong to religious traditions," he said. "Our efforts to come together and foster dialogue are a valuable contribution to building peace on solid foundations."
"It is therefore imperative to engage in authentic and sincere dialogue, built on respect for the dignity of every human person, created as we Christians firmly believe, in the image and likeness of God," he said.
The Vatican didn't say which Muslim leaders attended the meeting, which was closed to the press.
But it did release a list of those who attended Sunday's Mass, including Saeed Taghavi, head of the office of religious minorities in the Iranian ministry of culture, the head of Rome's central mosque and other Italian Muslim leaders. Two dozen Buddhist representatives were also on the list.
Also on the list was the Archbishop of Canterbury Rowan Williams; Metropolitan Chrisostomos, a top envoy for Ecumenical Patriarch Bartholomew I, the spiritual leader of the world's Christian Orthodox Church; and a senior representative of the Russian Orthodox Church, Metropolitan Kirill.
Most Jewish leaders could not attend the Mass because it coincided with the weeklong Passover holiday.
"I assure you that the church wants to continue building bridges of friendship with the followers of all religions, in order to seek the true good of every person and of society as a whole," Benedict said.
The message was significant because Benedict had left out any specific mention of Muslims in his inaugural homily, although he had directed a greeting to "believers and nonbelievers alike."
In the homily, he specifically mentioned Jews, calling them his "brothers and sisters" who were joined with Catholics in a "shared spiritual heritage." He also reached out to other Christians, calling several times for full communion of Christians.
Benedict repeated that message on Monday, telling ecumenical leaders that he fully supported the need to work toward uniting Christians divided by schism and said the ecumenical presence at his installation was a good sign.
"Your presence, dear brothers in Christ, beyond that which divides us and casts a shadow over our full and visible communion is a sign of sharing and support for the bishop of Rome, which can count on you for following the path in the hope and for the belief toward he who is the head, the Christ," he said.
In his homily Sunday, Benedict — who has a reputation as a hard-liner for leading Vatican crackdowns on dissidents — said he wanted to shape his papacy by being a "listener" and not set off by imposing his own ideas.
"My real program of governance is not to do my own will, not to pursue my own ideas, but to listen, together with the whole church, to the word and the will of the Lord," Benedict said in his inauguration homily.
The pope did not elaborate, but the speech suggested his papacy could study some pressing issues, such as greater social activism and ways to reverse the decline of church attendance and dwindling number of priests in the West. In his previous role as guardian of church teaching, he had staunchly opposed calls for fundamental changes such as ending bans on contraception or for allowing women to become priests.
"I was quite prejudiced against him at first," said Maria Theising-Otte, a teacher from a Catholic grammar school in Handrup, Lower Saxony, among the German pilgrims lining up for their audience with the pope. "But now that I've seen him, read about him, I've changed my mind. I think he came across quite human, very modest and decent," during his installation Mass on Sunday, she said.
She was with some 1,800 students from the Gymnasium Leonium Handrup who had traveled to Rome — a trip that was originally planned for the canonization Sunday for the founder of their school, Leo Dehon, but was postponed after Pope John Paul II died.
"You never know when you have a chance to do it again, to see him face to face," she said.
Later Monday, Benedict was to celebrate a Mass at a Rome basilica, St. Paul outside the Walls.
Since being elected pope on Tuesday, Benedict has sought a more inclusive image.
Benedict's effort to reach out to Jews carries an added dimension because of his membership in the Hitler Youth and later as a German army conscript during World War II. He has said he was forced into participating.
"With his German background, I certainly believe that he will be sympathetic toward Jews and I think he will continue the path of John Paul II, who made some very significant symbolic gestures," said Menachem Friedman, a sociology professor at Bar Ilan University in Israel. "But I think it is much too early to comment."