The island's problems even include an infestation of 4,000 rabbits, who have overrun the world heritage site treasured for its colonies of penguins and endangered birds.
Paul Langa, a former political prisoner turned museum chief executive officer, says the museum is "crippled" by lack of funds.
"We need to look into the coming 10 years with a different eye," Langa recently told a parliamentary committee. "The pain and history in that space is our pain, our history. We need to put more heads together so we can look into the problems inside."
The museum received $5 million in government funding this year, an amount Langa would like to see tripled. But two executives were suspended last year after the museum reported an unexplained deficit of $3.2 million. Government arts and culture minister Pallo Jordan says an audit uncovered "shocking mismanagement."
Meanwhile the island's most famous inmate turns 90 this summer. Mandela planned to spend his July 18 birthday quietly with his family in rural southeastern South Africa. Some of his former fellow inmates were invited to a bash on the island.
Mandela's No. 7 cell was repainted ahead of the celebration, but the blankets and bowls he used are still there, as is the bucket he used as a toilet and had to empty every morning.
Haunting photographs and stories from guides, who used to be prisoners, recall the daily hardship and degradation: lights burning all night in torment; rules forcing black prisoners to wear shorts rather than long trousers worn by mixed race inmates; censorship of letters; guard dog kennels bigger than the human cells.
Robben Island, which was once a leper colony because of its isolation, was from 1961 to 1991 a maximum security prison for anti-apartheid leaders, including Mandela. It became a museum and national monument in 1997. The United Nations declared it a world heritage site in 1999.
The museum is working on a new "visitor experience" to improve facilities and boost tourism from the current maximum 1,200 people per day to 1,800. But weather is a big obstacle. High winds and rough seas prevented ferries from sailing 65 days last year, causing disappointment to tens of thousands of tourists and depriving the museum of their $19 fare.
A new 300-seat boat, Sikhululekile or "We are Free," was launched earlier this year, halving the time of the 7-mile crossing from Cape Town to 20 minutes and doubling passenger capacity. Delivery of the ferry was more than a year late and there have been teething problems, so the museum continues to use relics like the Susan Kruger, named after the wife of a former justice minister and used to transport staff to the island in the apartheid era.
Jan Moolman has been skipper of the Susan Kruger and has lived on Robben Island since 1963. His son was born there, his daughter married there and grandchildren were baptized there. But his wife is counting the days till he retires and they move to the mainland in 2010.
"For 45 years I've lived on the sea and surrounded by sea, but now I want to get away," Moolman says, a sentiment echoed by most islanders.
There used to be more than 2,000 inhabitants - excluding prisoners. But since the island was declared a heritage site, the number has dwindled to less than 200.
A feeling of forlornness permeates the village, which is off limits to tourists. Fewer than 20 children attend the school, which has one teacher. Shelves at the store are half bare and regular power outages mean it can't stock perishables.
Frikkie Nel, a former prison nurse, treats colds and tour guides' overstrained vocal cords at the island clinic. His wife runs the post office, which wins regular awards for its efficiency - small wonder given there are so few customers. His 20 year-old twin daughters work in the curio shop, although one of them, Merle Nel, is about to move to the Netherlands as an au pair.
"I have to get away," she exclaims. "There's nothing, nothing to do." She adds that it's difficult to make plans to party on the mainland because of the unreliable ferries. The family even had to give away its poodle after a neighbor's dog was found killing penguins and authorities ordered all dogs off the island.
Rabbits have nibbled away what little vegetation grows in the barren soil - including Nel's garden. The bunny population exploded after authorities removed 100 wild cats that had been stalking rare birds, including the endangered black oyster catcher.
The flash of rabbit tails punctuates the landscape they are decimating. Wildlife officials and local game reserve owners must bring in food for other island animals, and plan to move the estimated 200 European fallow deer - regarded as an alien species and so not allowed under South Africa's strict conservation rules - to mainland parks.
Animal welfare groups initially protested plans to kill the rabbits. But now, even the head of the Society for Prevention of Cruelty to Animals, Allan Perrins, concedes that "the rabbits must go."
Despite the island's problems, the reasons for preserving its famous prison remain unchanged.
"While we will not forget the brutality of apartheid, we will not want Robben Island to be a monument to our suffering and hardship," says former political prisoner Ahmed Kathrada in a message at the ferry terminal entrance. "We would want it to be a triumph to the human spirit against the forces of evil."