First-ever look into the ex-presidents' residence

(CBS News) The most exclusive group in the world is the men who have been President of the United States.

One of the best membership perks of that club is an exclusive hideaway, the ex-presidents' clubhouse, a residence across the street from the White House. It's a place for former presidents to stay when they are in town.

Pictures: The presidents' clubhouse

In a Time magazine cover story, Michael Duffy and Nancy Gibbs write about "The Presidents Club," the subject and title of their new book. Duffy, Time magazine Washington Bureau chief, showed CBS News around the clubhouse on a recent visit. CBS News is the first to tour the home.

Duffy said, "I think if you have been a president, this is a place you can come, have some privacy, have some security, be close to the action, and still feel like you're more than the person checking into the Hilton or the Hyatt."

This is perhaps the most exclusive hotel in the world - only four people are allowed to book a reservation.

"The reason this sort of overnight guest house exists goes back to Dick Nixon," Duffy said. "He keeps getting phone calls from (Lyndon B. Johnson) in Texas. He was restless at the ranch and he keeps wanting to come to Washington, and that's why this is here. It was Nixon basically trying to keep Lyndon Johnson happy."

But the first time Nixon came back to Washington after his resignation, he was forced to go elsewhere.

Nixon asked to stay in the building, but Jimmy Carter wouldn't let him.

And it's more than just the clubhouse. Duffy and Gibbs write that former presidents often form surprising alliances.

"Nixon really wanted to get to know Clinton," Duffy said. "He wanted to be sort of a secret behind the scenes adviser. He became that. Clinton turned out to rely on him for a great deal of advice."

The four-story story brownstone has two bedrooms and a dramatic presidential bathroom complete with fireplace. There's a breakfast room in addition to the formal dining room. The seal of the president is prominently displayed.

The ex-president who has used it the most?

Duffy said, "George H. W. Bush stayed here during Clinton's presidency and then didn't have to when his son became the president himself and he could stay across the street."

The authors say that former first lady Barbara Bush described the house as "a dump." It was modernized at the end of her son's presidency.

There's a reason, Duffy said, that sitting presidents - including President Barack Obama - like to talk to their predecessors.

"You come out of this job with so many scars that only you know about," Duffy said. "And really there's only few people who understand what those scars are like. And maybe a relationship like that can take you to a place no one else can."

Duffy said the clubhouse is likely to get a lot more use in coming years because the U.S. has some of the youngest ex-presidents in history.

Of course the house is large, but it's no mansion, so if the ex-president's club gets much bigger, and the man sitting in the Oval Office decides he ever needs all of them in Washington, D.C. at the same time, the presidents may have to draw straws to see who gets the prime real estate across the street.

Editor's note: Simon and Schuster, the publisher of "The Presidents Club," is owned by CBS.

For Bill Plante's full report, watch the video in the player above.