City Computer Snafu Forces Document Dump, Several City Departments Forced To Re-enter Data

Although the city’s computer chief says neither she nor her computer consultants have ID’ed the exact cause of the problem, the corrupted files are deleted and massive amounts of data have been transferred to a secure server. Several departments, meanwhile, are still trying to figure out what data they lost over three days last week.

 

By DAVID REID

NORTHAMPTON – Network computer operations in a number of city departments were shut down last week, and then again this week, after a virus or some other irregularity corrupted a section of one computer hard drive that handles documents storage, the city’s top computer official said today.

Vanessa Oquendo, director of the city’s Management Information Systems, said a major corruption was discovered last Thursday in one of the city’s approximately dozen computer servers that caused the system to be shut down. Documents that were created or updated and then stored on a section of that server from last Tuesday through Thursday afternoon were deleted and had to be re-entered, she told Northampton Media.

And although Oquendo thought the problem was fixed by the time she and some of her staff left work about 8:30 p.m. last Friday night, upon her return to work Tuesday morning this week (city offices were closed on Monday, the celebration of Martin Luther King’s birthday) Oquendo found the problem had reappeared.

Luckily, the corruption of document files was isolated, affecting only departments that are housed in the three downtown municipal buildings – City Hall, the Puchalski Municipal Building and Memorial Hall – and did not impact police, fire or public works departments.

Oquendo said the computer snag also did not affect the city’s MUNIS computer system, which handles finacial and personnel data; the city’s financial information, she said, is stored on a computer hard drive located in another undisclosed location in the city. The city’s permitting server was also unaffected, she said.

But the failure of the one server caused problems that lasted until this afternoon.

For a period of time today, the network that allows many city departments to save, update or retrieve documents – letters, PDF files and some data-bases – was off-line while the time-consuming transfer of data from one “volume” of the server to a new one was completed.

“We had a massive amount of data,” Oquendo told Northampton Media.

By about 2:30 p.m. today, she said, everything appeared to be operating smoothly again.

Cause of Computer Problem Still Unknown

As to what started the problem, Oquendo said she was unsure, adding that even Dell Computer technical support troubleshooters were unable to diagnose the cause of the computer file corruption.  “I really don’t know what the cause was,” she said.

Her best guess, Oquendo said, was that some filed entered into the system from a mobile device or private laptop computer was the culprit. Despite sophisticated ant-virus software and a firewall system designed to keep outsiders out and away from city computer systems, she said, “None of these are infallible.”

Even after the problem appeared to be fixed last Friday night, Oquendo said, the computer system thought the corrupted files still existed.  It was at that point this week, Oquendo said, that all the data in that part of the server had to be transferred to another storage device and recopied to a different volume in the same hard drive.

Technically, she said, the server never crashed.

At City Hall, City Clerk Wendy Mazza said one of her staff was completing an end-of-the-year report for her department when the system went down late last week, and that the work had to be redone and saved to a hard drive on the individual computer.

“We just had to go back and re-enter everything,” said Mazza, whose department deals with documents all the time. She spoke to Northampton Media about 1:30 p.m., when the data transfer was continuing and that network was still inaccessible.

“They’re still transferring files,” said Mazza. “It’s really not a good system.”

Building Commissioner Louis Hasbrouck said today that word files, letters, PDF documents and updated spreadsheet work he performed last week was lost, and that he’s still trying to figure out which documents need to be redone

Other departments are going through similar problems.

“I’m really disappointed this could happen,” said one department head who asked not to be identified.

Mayor David Narkewicz, who thoroughly discussed the computer snafu with Oquendo, said his office was also affected by the problem.

“We had stuff in my office that was lost and had to be re-entered,” the mayor said. 

City Assessor Joan Sarafin said her tax and property records operate on a completely separate “Universe” system and were not affected. And Oquendo said the Planning Department’s document storage likewise operates on a separate and unaffected system.

The city’s website appeared to be untouched by the corrupted file problems, said Oquendo, although Mayoral Aide Lyn Simmons, who runs the website, said today she was still checking on documents saved last week.

Taking Steps To Protect City Computer Data

According to Oquendo, when the problems surfaced last week, she found the corrupted files began showing up during the day Tuesday. As a result, she had to employ a “restore” maneuver – where the hard drive returns to an earlier point in time – basically resetting the server to where it was on Monday night, when the server was still uncorrupted.

City computer hard drives, most of them located in the MIS office in the basement of the Puchalski Municipal Building, are backed up every night, with “snapshots” taken of existing computer activity several times daily, Oquendo said. And anti-virus programs are updated frequently, with “patches” applied to address the latest threats to computer security, she said.

Oquendo’s main focus last Friday and this week, she said, was to restore untainted documents already saved on the hard drive, and that her job now – if indeed the problem is truly resolved – will be figuring out what happened in the first place.

Meanwhile, said Oquendo, she will likely issue a new round of warnings and directives to city employees who use remote devices or laptops to access city servers, telling them to install passwords and take other steps to avoid duplicating the past week’s computer problems.

Asked for comment, Narkewicz said he is committed to upgrading the city’s computer networks and their security systems, while trying to make information on those computers more accessible for the public. Some of the computer upgrades, the mayor said, will continue to be funded through the capital improvement process, where a committee of officials and citizens annually evaluates the building, equipment and technology needs of all city departments.

The mayor said he and Oquendo talked in some detail “about potential holes or breaches” in computer security, “and how to build up redundancies,” something that could include new technologies to be deployed in-house or off-site. It is a top priority, he said.

“We have to protect the city’s assets,” said Oquendo.

[Editor's Note: Stay tuned for a follow-up story by Northampton Media publisher Mary Serreze.]

© 2011 Northampton Media             

David Reid can be reached at dreid@northamptonmedia.com