Alaska: Land Of The Lost (Data)

Computer Technician Accidentally Wipes Out Info On State's $38 Billion Fund

(AP)  Perhaps you know that sinking feeling when a single keystroke accidentally destroys hours of work. Now imagine wiping out a disk drive containing information for an account worth $38 billion.

That's what happened to a computer technician reformatting a disk drive at the Alaska Department of Revenue. While doing routine maintenance work, the technician accidentally deleted applicant information for an oil-funded account — one of Alaska residents' biggest perks — and mistakenly reformatted the backup drive, as well.

There was still hope, until the department discovered its third line of defense, backup tapes, were unreadable.

"Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow. The July computer foul-up, which wiped out dividend distribution information for the fund, would end up costing the department more than $200,000.

Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.

Nine months worth of applicant information for the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.

And the only backup was the paperwork itself, stored in more than 300 cardboard boxes.

"We had to bring that paper back to the scanning room, and send it through again, and quality control it, and then you have to have a way to link that paper to that person's file," Skow said.

Half a dozen seasonal workers came back to assist the regular division staff, and about 70 people working overtime and weekends re-entered all the lost data by the end of August.

Last October and November, the department met its obligation to the public. A majority of the estimated 600,000 payments for last year's $1,106.96 individual dividends went out on schedule, including those for 28,000 applicants who were still under review when the computer disaster struck.

Former Revenue Commissioner Bill Corbus said no one was ever blamed for the incident.

"Everybody felt very bad about it, and we all learned a lesson. There was no witch hunt," Corbus said.

According to department staff, they now have a proven and regularly tested backup-and-restore procedure.

The department is asking lawmakers to approve a supplemental budget request for $220,700 to cover the excess costs incurred during the six-week recovery effort, including about $128,400 in overtime and $71,800 for computer consultants.

The money would come from the permanent fund earnings, the money earmarked for the dividends. That means recipients could find their next check docked by about 37 cents.

[rumifan]

Amazing how stupid.

Think of all the bean counters and execs with golden parachutes who want to port mission critical applications to a server, or outsource IT staff and facilities overseas. This story is really reassuring.

This would have never happened in a well run mainframe environment. The official response is so lame as to be laughable.

Also, Senator Ted (who builds bridges to nowhere) is probably owned by the auditor firm(s)who supposedly check systems controls and recovery plans? Don't they have earthquakes and tidal waves up there?

[rumifan]

Amazing how stupid.

Think of all the bean counters and execs with golden parachutes who want to port mission critical applications to a server, or outsource IT staff and facilities overseas. This story is really reassuring.

This would have never happened in a well run mainframe environment. The official response is so lame as to be laughable.

Also, Senator Ted (who builds bridges to nowhere) is probably owned by the auditor firm(s)who supposedly check systems controls and recovery plans? Don't they have earthquakes and tidal waves up there?

[michellem99-2009]

I would be upset if they did that to my computer and I am just a home user. I read the story and I wondered who wasn't paying attn and lost everything on the hard drive. I would be not too happy about it. I think they should write a Dummies book for computer techs if they haven't done so. I love to read. I am am older lady and I do not know that them geeks should know but I feel if he/she didn't know how to do it then why wasn't there a person who did know watching the rookie to be sure it was done right as this was what I was thinking as I read it. A pretty costly error for the company.

[caldwellptr]

Juneau - My case worker in the afterlife.

They never heard of RAID arrays? What amateurs...

national association of technicians? Jnunna?
I don't think medics have anything to do with it...

[jbeckett5]

No single technician should be physically able to do anything so devastating. The drive should not have been given to a tech to format until somebody _else_ had confirmed that tapes were readable by copying them to another hard drive, and the other drive had been isolated and confirmed good as well. This was clearly a management error, and the manager should lose their job.

Tech management 101: Always have at least three good reasons the unthinkable won't happen. One of them will fail you occasionally, and you'll mess up another yourself. When this happens, get somebody else involved as you think through what is going to be done next.

[jnunna]

I don't know about you guys, but I would love to hear what the national association of technicians have to say about this new phenomena.

[mitch0927]

LOL...looks like my mind wandered off there.....LOL

[mitch0927]

It's Alaska, and you know its okay to grow the whakie tobakie there. He was probably stoned or thinking about it and his mind wnadered off....

[terrapin78]

It is amazing that such a rookie would be unsupervised while working on such an important piece of hardware. Kinda makes ya think it was no accident.

If I were receiving a dividend I would be all over this.

[mitch0927]

Kind of wondering why the people receiving the dividends were having to pay for it. I know, it was only 37 cents a piece, but it's the fact the technician did such a rookie mistake, the company he works for should foot the bill. I am in the same position he is in, and let me tell you what, when our back-up tapes can't be read, we know immediately. YOU NEVER RE-FORMAT a hard drive, until you have backed it's contents up to another source. That is computer technician 101 stuff.