But, says Phillips, his romantic history aside, the prince is now being credited with being well ahead of the curve on some of the more popular causes of our time -- environmental protection, and organic farming and gardening.
In October 2005, Charles told 60 Minutes, "I'm just trying to say that we ought to redefine the way in which progress is, is seen. Is it progress to rush headlong into upsetting the whole balance of nature, which is what I think we're beginning to do?"
At Highgrove, his 1,000-acre private estate in western England, he has set the standard not just for organic farming -- he has a big business selling organic products -- but for organic gardening, Phillips observes. No chemical fertilizers or poisonous pest control is used there, yet he's created what's widely accepted as a horticultural masterpiece.
It's not a passion he's come to lately. He didn't follow this trend, Phillips points out, he set it.
"I think he's a man of huge vision," says Patrick Holden, who runs the British Soil Association, Britain's Organic food certification body. "In the '70s, he was talking about the environmental crisis, in the '80s he cottoned on to organic farming as a solution to build a more sustainable agriculture, and he was so far ahead of the curve that he was dismissed, and in some cases is still dismissed today. And yet, we should listen to what he says."
The prince once made the mistake of saying out loud, Phillips recalls, that he talked to his plants, reinforcing the impression that he was -- as the expression goes -- different from you and me. But, as the organic movement has taken off, Charles is now seen as a naturalist guru, and has just published a new how-to book, "The Elements of Organic Gardening." Put away those pesticides, he says. Garden by natural laws. It may be labor-intensive rather than chemical-intensive, but he's proven, he says, that it can work.
For a man whose job it is to wait for a job (as king), Phillips notes, he's proving to be a pretty good businessman.
"He's the patron saint of organics," says royals-watcher Victoria Mather. "He espoused organic agriculture and organic gardening long before it was fashionable. Now, it's super-fashionable. What a great time to release a book about it!"
The book is a lavish portrait of the rarely-seen gardens at Highgrove. The prince is very protective about the place. But it's full of helpful hints, and his philosophy that too much of nature has been destroyed in the name of progress.
Still, not everyone thinks the issue is that clear-cut.
Nick Cohen, a columnist for "The Observer" comments, "The world has 6 billion people, and you're going to get a lot more, (and) you're not going to be able to feed them with organic agriculture, or if you're going to do it, you're going to have to cut down every rainforest and turn them into fields. Only with industrial agriculture can you avoid mass famine and starvation, and Prince Charles ... just doesn't like to think like that; he's a deep conservative in some ways, as you'd expect. He's a member of a royal family!"
The prince has, Phillips remarks, developed two specialties: gardening and controversy. With his new book, he's combined both.