From under Westminster's arches you can look down on this 700-year-old Gothic wonder and understand why it might prompt someone to write an epic novel about the construction of an abbey in 12th century England.
"It came from looking around a place like this and thinking, 'Why is this here?'" Follett told CBS News correspondent Anthony Mason. "It's huge. It's beautiful. It's costly. Why is it here?"
His latest book was a radical departure for an author who made his name with bestselling spy thrillers like "Eye of the Needle" and "The Key to Rebecca."
"My publisher certainly felt it was a risk," he said. "They would say, 'You know, you've had all this success with Nazis and secret agents. And this is about building a church in the Middle Ages? Are you sure?'"
But since it was published in 1989, "The Pillars of the Earth" has become an international sensation, selling 15 million copies worldwide. In Germany, it was voted the third most popular book, behind "The Lord of the Rings" and The Bible.
Now, 18 years later, Follett has returned to medieval England in the sequel, "World Without End." The setting is the same English abbey 200 years later.
"It's about the Black Death, the most devastating plague that the world has ever seen - killed between a third and a half of the population," Follett said. "The Black Death killed so many people, and the church was so powerless to help the victims of the plague, that it undermined the people's confidence in the church."
The sweeping story, with more than 200 characters, took Follett more than 3 years to write.
Behind his desk, he's hung a framed letter from Charles Dickens to a magazine writer. It's a rejection letter, something Follett has had the good fortune to have more or less avoided throughout his career.
In all, he's sold 90 million books. The son of a Welsh tax inspector, Follett was obsessed with Ian Fleming's James Bond adventures as a kid.
"Boy, I loved those books," Follett said. "They were the most exciting think that had ever happened to me. You know? He was so cool and tough. And he knew about things that were the most fascinating things to me as a teenage boy. He knew about cocktails! I'd never met anybody who'd had a cocktail."
By his early twenties, Follett was writing his own spy novels. In 1978, "Eye of the Needle" broke through to the bestseller list and became a popular film. It made Follett, who's never been shy about enjoying success, a rich man. He has been called a "champagne socialist," and has come to embrace the term.
From his college days, Follett has been active in the Labour Party. That's how he met his second wife Barbara in 1982.
"We met at a political meeting. How romantic can you get?" Follett said.
"Ken came in and I thought he was very arrogant and very irritating," Barbara said. "So, I went and bought one of his books, so that I could get the measure of the man, and sat up all night reading 'The Eye of the Needle.' And then to my embarrassment now, not then, I phoned him and said, 'You know, you can write.'"
Follett helped write his wife's political speeches when Barbara won a seat in Parliament in 1997, swept in on the Labour Party landslide that brought Tony Blair to power. The British press called her one of "Blair's babes."
But Follett famously turned on Blair in 2000. In an incendiary article he accused the British prime minister of "making malicious gossip a tool of modern British government."
Britain's political cartoonists feasted on the Labour feud. Their work decorates the walls in Follett's London home.
"None of them are very flattering. But cartoonists are not supposed to flatter us, are they?" Follett said. One shows Tony Blair opening a bottle of champagne. "And the cork is hitting him in the eye. And the cork is me!"
Another imagines he's written a book called "The Secret Backstabber."
"I accused Tony Blair of stabbing people in the back," Follett said. "And one of the interviewers said to me, 'Well, isn't that what you did to him?' And I said, 'No, I stabbed him in the front!'"
Barbara Follett was anxious about the article and the prime minister called her to ask what her husband was doing. She said that she and Blair had a "frank 45 minute discussion" when he visited the Folletts this past summer. Tony Blair was ceding the prime minister's job to party rival Gordon Brown. Barbara Follett was named a junior minister in the new cabinet.
And this week, Ken Follett will wait to see how the sequel to his most popular novel is received.
With the 1,000 page "World Without End," he has tried to construct a novel as monumental as a cathedral itself.
Admittedly, Follett has set the bar fairly high.
"It's a challenge," he said. "And it's a risk."