The White House, however, stresses that this is a working vacation with numerous daily briefings, reports CBS News White House correspondent Chip Reid.
"There are a number of issues that the President is following very, very closely, and expects to be kept informed about developments on those issues," Homeland Security adviser John Brennan said.
Mr. Obama has come under fire from some conservatives for his vacations this summer: first to Bar Harbor, Maine; last weekend to Florida's Gulf coast; and for First Lady Michelle Obama's recent trip to Spain.
Critics say all his should be on the dire economy and the plight of average Americans.
"Yes, he's been working so hard he's tamed the economy. And now he gets to go to Martha's Vineyard," a sarcastic Rush Limbaugh said Friday.
Not all conservatives begrudge Mr. Obama the time away from the White House, however.
"In fact, I wish politicians spent more time outside of the Beltway and less in Washington and being really in touch with the American people," independent conservative Amy Holmes told CBS' "The Early Show" Saturday Edition. "Martha's Vineyard, maybe not exactly in touch - maybe he should be in a camper. I'd like to see that. But, I'm not one of the people that's actually attacking the president on taking some downtime."
By the end of his trip, President Obama will have taken nine vacations and visited Camp David 14 times for a total of 80 vacation days since he took office. But at the same point in his first term, President George W. Bush had taken far more time away - 14 trips to his ranch in Midland, Texas, and 40 to Camp David, totaling 225 days.
Presidents, thought, are never truly on vacation. Crises often arise - for example, the Christmas Day bomber tried to strike while President Obama was vacationing in Hawaii. For President Bush, Hurricane Katrina struck the Gulf during a long stay at his ranch.
The White House says they're confident that most Americans understand that with such a high-pressure job, a president deserves and needs some time to unwind and recharge.
Still, it's been a rough summer for the administration, with Mr. Obama's approval ratings taking a hit amid the slow economic recovery and the Gulf oil spill disaster.
But compared with other recent presidents, Mr. Obama may not be in such bad shape.
"President Obama's approval ratings are certainly lower than they have been in the past, but it is worth noting they're higher than President Clinton's approval ratings were in 1994 at the same time and even higher than President Reagan's approval ratings were in 1982 at this same time. I think the Reagan and Obama situation are sort of good comparisons because Reagan also had inherited a very difficult economy," Jennifer Palmieri, of the liberal thinktank Center for American Progress, told the "Early Show."
"The president's had a lot of legislative victories but the White House understands very clearly that you don't get points with the American people for legislative victories. They want to see results. The uncomfortable truth the white house is wrestling with [is] a lot of these policies they've enacted take time for people to see results in their everyday lives … that's just going to take some time."