VA State computer systems apparently seriously degraded and/or offline

Thanks to an e-mail I received from VA State Delegate Tom Rust (VA-86), it’s come to my attention that a major information systems failure is currently affecting Virginia’s government systems. The DMV is affected:

Statewide Computer Issue Halts In-Person Driver’s License Transactions
Vehicle Titles, Decals, Etc. Still Being Processed in DMV Offices

RICHMOND – Due to a statewide computer system problem, the Virginia Department of Motor Vehicles is currently unable to process driver’s license transactions inside its 74 customer service centers in Virginia. Customers who can renew licenses through another service option are encouraged to go online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782. The Virginia Information Technologies Agency (VITA) is working to resolve the server problem that is affecting 24 different state agencies. VITA has not reported a recovery time.

At DMV, all other types of transactions, including vehicle decals and titles, transcripts, etc. are being processed normally.

That’s an interesting comment – the web site appears to work and it’s handling transactions that the in-person, on-site DMV employees can’t? As a network professional, the situation interests me and suggests quite a number of scenarios. Clearly, if the transaction resources aren’t completely dead (and they’re not, or you wouldn’t be able to do them on the web site) and the interface web site isn’t dead (ditto) then the issue must be one that affects the local sites directly. That’s either a connectivity problem (which wouldn’t affect multiple locations simultaneously) or there’s some issue with the internal systems at the DMV that’s not affecting external systems such as the DMV’s site.

Researching this led me to this report at WSLS Roanoke:

The outage, flaring Wednesday afternoon and expected to disrupt some services through the weekend, is attributed to 228 malfunctioning servers, which supply shared software and applications to clusters of state agency computers.

Twenty-six of more than 80 state agencies were hit by the shutdown, including the office of Gov. Bob McDonnell.

“We’re disappointed to have a failure, an outage of this magnitude,” Samuel A. Nixon Jr., head of the Virginia Information Technologies Agency, said yesterday. “No matter what you do, it’s going to happen on occasion.”

Ah, so what’s happened appears to be that several servers handling internal access to resources have died. The external ones that we all access via the Internet (likely located in what’s called a DMZ) were apparently unaffected. The fact that those servers can still apparently access the data means that the data stores appear to be available and intact, it’s just that the internal servers that handle the government employee’s interfaces are down.

I’m going to have to take some issue with that last comment by Mr. Nixon at VITA. I can assure you that there are companies out there (banks, Google, Amazon) who architect their environments to make it as sure as is humanly possible that such “occasions” do not occur. The story at WSLS indicates that the infamous VITA-Northrop Grumman IT contact is at the heart of this. NG’s subcontractor EMC (a network storage company) was reportedly “checking for faulty equipment” and, in the process, killed the 228 servers. That’s a helluva faulty equipment check that manages to take out a couple of hundred mission-critical systems. And do it badly enough, I might add, that it takes 5 days to recover. Where are the backup systems?

Nixon also said that the interruption was of insufficient magnitude to activate a backup system at a duplicate computer center in Russell County, in Southwest Virginia.

A full-on, deader-than-a-doornail failure is of “insufficient magnitude” to “activate” a backup system? Sorry, again, but I call bullshit on that one. Any backup system that can’t step in and handle the task for which it was designed to back up when the primary drops dead of brain failure isn’t a backup system at all – it’s a completely separate system that’s now sucking up power and providing nothing but space heating. I may not be privy to the internal discussions and the full picture of the situation but that line of crap is a man just throwing chaff into the air hoping to confuse people. If he’s serious, then he either doesn’t know what the hell he’s talking about or he’s being lied to.

Either way, an investigation needs to be performed and the citizens of Virginia need to be apprised of the results, in clear language. Either the government wasn’t clear or knowledgeable enough to articulate proper requirements or the contractor simply isn’t performing according to the contract. Either way, that needs to be fixed and fixed now.

Advertisements

One comment

Comments are closed.