Cameron told the annual Munich Security Conference that European governments have been too tolerant of some sectors of society that publicly oppose democracy or reject equal rights for all.
He said Britain had found that many convicted terrorists had initially been influenced by so-called "nonviolent extremists" - people who aren't involved in encouraging plots, but denounce Western politics and culture - before going on to carry out violence.
"We won't defeat terrorism simply by the actions we take outside our borders. Europe needs to wake up to what is happening in our own countries," Cameron told the conference.
Both Britain and Germany have had noisy domestic debates about the impact of immigration, and the difficulties of integrating some religious communities, or those who struggle with the language of their new home.
In an attack on Britain's previous government, Cameron said authorities there had been too hesitant to intervene when some sectors of society espoused abhorrent views.
"We have even tolerated these segregated communities behaving in ways that run counter to our values," Cameron said. "We have encouraged different cultures to live separate lives, apart from each other and the mainstream."
Cameron said a culture of tolerance had allowed both Islamic extremists, and far-right extremists, to build support for their causes. "We've been too cautious, frankly even fearful, to stand up to them," he said.
Some European allies have criticized Britain for harboring hardline Islamic clerics and failing to clamp down on mosques that promote a perverted view of Islam.
Several terrorists involved in attacks or attempted plots in the U.S., Sweden, Denmark and Norway over the last two years have had links to Britain, or British-based clerics.
"If we are to defeat this threat, I believe it's time to turn the page on the failed policies of the past," Cameron said. "Instead of ignoring this extremist ideology, we - as governments and societies - have got to confront it, in all its forms."
He told the conference that developments in the Middle East should be harnessed to disprove Muslims who claim their religion cannot be observed properly within the democratic system.
"If they want an example of how Western values and Islam can be entirely compatible, they should look at what's happened in the past few weeks on the streets of Tunis and Cairo," Cameron said.
Mohammed Shafiq, of the Ramadhan Foundation - a British Muslim youth group - said in a statement following the speech that Cameron has risked angering Muslims by suggesting there was widespread intolerance within the religion.
"Singling out Muslims as he has done feeds the hysteria and paranoia about Islam and Muslims," Shafiq said. "British Muslims abhor terrorism and extremism and we have worked hard to eradicate this evil from our country."
The British leader's comments follow tensions across Europe since November of possible new terrorist attacks. Officials said last year that a sleeper cell of some 20 to 25 people may have been planning an attack inside Germany or another European nation.
Nine men were charged last month in Britain over an alleged plan to attack Parliament and the U.S. Embassy in London.
Last week, the U.S. State Department warned of an ongoing high threat-level in Britain, and told tourists of a specific risk to transit networks and airports.