School System Aces Testing

The largest public school system in the United States often makes headlines—whether it's about the challenges of modernizing the curriculum for children with disabilities or it involves such debates as introducing vouchers that let students attend schools of their choice. But look past the media spotlight into the New York City Board of Education and you'll see an operation that is intent on accomplishing its goals, particularly in the area of IT.

As part of its Year 2000 solution, the board adopted a formal software testing methodology and automated testing tools over the last two years. Now the board sees the value of retaining that quality assurance (QA) solution beyond the year 2000. A permanent QA program is an investment many IT organizations may be reluctant to make, even though it is universally seen as a smart way to produce the applications that run the business.

The New York City Board of Education relies on hundreds of business and student applications. Business applications—used to maintain 1,100 school buildings and administer a district with 63,500 teachers—include everything from inventory and facilities management to accounting and payroll.

Capturing Organizational Knowledge

In 1995, the board appointed a new chancellor, Rudy Crew, who had been a popular superintendent in Tacoma, Wash., and who set about overhauling board operations to improve administrative performance and the quality of education. With an eye toward upgrading IT operations, Crew staffed the IT organization with seasoned managers.

At the same time, budget cuts forced the board to reduce its work force by 40 percent. "I saw a lot of good IT people leaving, and our knowledge base was drastically eroded. That concerned me," says Howard Schatz, a 13-year board employee who is now manager of QA. "I believed a formal quality assurance program, where system usage must be thoroughly documented, could help us recapture that knowledge."

Schatz seized upon the board's Y2K project to build his case. "Year 2000 gave me the excuse and budget to establish a QA methodology and buy QA tools, but my motivation was strictly from the QA perspective," he says.

Schatz convinced his boss, Dave Wolovick, director of business systems, to give him the job of establishing a QA program. Schatz first set up a formal, manual testing process in line with ISO 9000 standards. "This step did not make me the most popular guy in Brooklyn—it demanded a lot from our programmers, project leaders and, ultimately, our users," Schatz says.

Overcoming Resistance

The board's IT staff eyed the move toward a formal QA program with suspicion, fearing that it would slow down a department already overburdened because of the staff cuts. According to Schatz, the strict guidelines of ISO 9000 can be time-consuming to follow and frustrating for developers who are used to doing things their own way. Shifting their opinion required a re-education effort and a little internal public relations.

Once the programmers saw their workflow improve and practices become more organized, they warmed to the idea of QA. Once users saw the testing process become efficient, they, too, grew more accepting. Schatz also hired a competent, pleasant person to act as a liaison between himself and the developers. "You have to put a friendly face on this; no one likes someone looking over their shoulder," he says.

Perfecting the Testing Process

ISO 9000 standards call for a verification matrix to assure that every way a person uses a product, or every path of logic in a software application, is fully tested. According to Schatz, implementing a verification matrix for software is virtually impossible without automated tools. "Covering all of your bases was never possible before because there was just too much ground to cover," he says.

Using QADirector, Schatz automated a verification matrix. QAHiperstation gave him a way to record user actions into scripts—nearly 5,000 each day—without interrupting employees at work. "QAHiperstation is the only product on the market that offers a global record feature that works without user intervention. The product has allowed us to glean knowledge from the users of the system who know the most about our applications," says Schatz. The test bed of 50,000 QAHiperstation scripts serves as input for the verification matrix. It is being organized in a way to cover every path of logic through every program of every system.

	"I saw a lot of good IT people leaving, and our knowledge base was drastically eroded." —Howard Schatz, manager of QA

"Everything we're putting in place for large-scale system and regression testing of Y2K changes will be kept and re-used after the year 2000," Schatz says. As modified applications come through future QA programs, testers will use the tools and existing test bed to verify the accuracy of the application before it is released.

Schatz strongly believes the QA effort has improved IT operations. "The process has brought order and consistency to every project and improved the workflow tremendously," he says. "With the automated testing tools from Compuware, we are perfecting our testing technique." When production errors do occur, Schatz and Wolovick usually find that the application in question did not pass through QA.

Establishing a formal QA program is like buying insurance. "The whole point is that QA goes on behind the scenes to assure that everything runs smoothly—it shouldn't be making the news," Wolovick says.