Chapter 10: Modern Testing I

Chapter 10: Modern Testing I

Send Us Feedback

Listen on Itunes

Listen to Stitcher

Matt: Hey, everyone, welcome to The Modern Mainframe. I’m Matt DeLaere. Thanks for joining us for Chapter 10 of our series Building a Better Software Delivery Platform, in which DevOps Solution Architect Rick Slade explains how you can leverage DevOps and Agile methodologies to create a world-class software delivery system. So, let’s say hi to Rick. Rick can you believe we’re on our 10th episode, already?

Rick: This is amazing, Matt. I never thought we would last this long. It’s been great, and I appreciate all you guys do and I appreciate you guys listening to us. Thank you. And it’s good to talk to everybody.

Matt: Indeed. So, today’s episode is the first of a two-part series regarding modern mainframe testing, and we have a special guest who knows a lot about the subject. So, Rick, all throw it over to you to talk a little bit about testing and introduce our special guest.

Rick: Thank you, Matt, and I’m very happy to have Steen Brahe today with us. I’ll talk more about him in a second. I wanted to give you just a couple of thoughts about testing, because I think it is one of those things… I’ve been doing, or helpful organizations modernized their software development or software delivery ecosystems for many, many years, now and testing is such an integral part of that process. And it’s evolving. We’re getting better at it. We’ve got to get better at it, especially for those of us who are transitioning to Agile as a way to manage that software delivery effort.

So, incorporating testing into an Agile mindset and to an Agile way of doing things, particularly, especially for the mainframe, it’s something that we’ve got to continue to improve, something that we’ve got to get better at. And the technologies that Steen is responsible for here at Compuware are a big, big part of that. Learning how to leverage virtualized testing and non-virtualized testing. And I won’t go into it too much because I want them to talk about those things in the podcast, but being able to effectively utilize unit testing as a mainframe software developer, is becoming more and more important. And again, primary reasons, at least in my mind, and I’ll ask in about this later, is the result of leveraging Agile in starting to segregate work into tasks that can be completed in two-week sprints. We’ve got to get smarter about test data setup and provisioning; we’ve got to get smarter about the re-use and the ability to leverage test case scenarios over and over again. Those things become critical in a world where work is being broken down into smaller increments, and so testing has just been such a big part of our messaging, my messaging at IBM and now at Compuware, and it’s just, I think it’s critical that we get this right.

We’re actually going to dedicate two podcasts to this topic this week and next week, and so I hope that this will provide information that’s useful and interesting. With that, I’d like to introduce Steen. Steen came out from XaTester and is now responsible for our Topaz for Total Test products and technologies and capabilities. Steen, I’d love for you to introduce yourself. Tell us a little bit about your background, your role at Compuware.

Steen: Thank you, Rick. Sure, I’ll do that. So, I’ve been at Compuware about two years. Actually, today it’s exactly two years ago that I joined Compuware. I think we should actually go back in time when I started to do mainframe testing tools, and that actually goes back all the way to 2003, and at that time I was employed as a developer in a large bank. And as part of my job, I had to develop services on the mainframe in Java. And that time, if you remember, back at that time it was SOA – Service Oriented Architecture was really hot topic, and many of my co-workers, they did the same service development, but in COBOL. So, I needed to test my services, and at that time we didn’t have any tools for doing that, so I invented my own tool where I could submit some XML from a browser that would automatically test my service.

So, I’m a lazy developer, and I believe at lazy developer is a smart developer because you don’t want to do things twice, so manual work should be automated. That was what I did with this, where a simple browser solution by having some XML, had a web server that could then call the service and verify the results. Soon my co-workers on the project found out of this and they started to use it. So, suddenly 15 people were using this solution that I developed for myself. So, I thought, well, this is really a good idea, I must have hit something really interesting here. So I sat down, we designed the solution to be an Eclipse tool, and the project continued to use that one. And then it was used in the organization as an undergound tool. This was never known tool, so it was an underground tool that more and more projects found out. Sometimes we got funding to do some additional development based on feedback from developers. So, all the features that came into the product were really required by mainframe developers.

Rick: You’ve used a couple of terms I really love, Steen. First of all, the lazy programmer. I had the same conversation with my son the other day, and he used the same term, and I think you’re absolutely right. It’s not that you’re lazy in your work, you’re just smart about how work gets done so that you’ve got other time to do things. I love that. And then then secondly, this underground tool. How many great technologies have been developed under what I call skunkwork technology? So really, really cool.

Well, briefly tell us about Total Test and how it benefits our clients.

Steen: Yeah, so eventually that idea… actually just to finish that one, I stopped at the bank, got another job, got to thinking about these good ideas, so I had learned so much about what mainframe developers actually needed for testing. I knew exactly what they needed, and there were no products out there. So, at the end I contacted a partner company that I knew that had all the sales channels, all the competencies on the mainframe and so together, we developed XaTester from ground up based on these many years of experiences on knowing what developers need.

That became XaTester, we got a couple of customers, Compuware contacted us, and we got acquired by Compuware two years ago, which is a fantastic thing, it’s really a great thing to come into Compuware, this company with so many passionate explorers, think about customers. So, that’s great.

XaTester today is integrated completely into Topaz for Total Test. That was Compuware’s product back at that time. So, now Topaz for Total Test contains what originally was there, virtualized testing or unit testing, and it now contains also the old XaTester, which is what we call non-virtualized testing, where you’re testing programs in a real environment with real data, where virtualized testing is where you are stubbing out all external calls.

So, back to why Total Test, what it can do for customers. So, Total Test is a tool that developers and testers can use to create, execute, evaluate, and automate mainframe program and application testing. So, the benefit is really, you can automate manual work, and actually what developers today in the mainframe world are using, they’re using about 30-80 percent of their time on doing mainframe testing because there have not been any testing tools available. That has completely changed with Topaz for Total Test, where you can now start to automate this manual work, and that is where you can have the smart developers be much more efficient or effective in what they use their time on. Instead of using time on manual testing, they can focus on what gives value to the company, and that is to develop business functionality faster, and that is also what the developers find fun—that is to provide business functionality.

Nobody likes to do the same work over and over again. And the interesting thing is, with automated testing, with Topaz for Total Test, at the same time as you move faster, you also is able to deliver with higher quality. Because the tests are automated, you remember to test every time you make a change, and this way you can really see that you go faster with higher quality, even with less resources in the project.

And we’ve seen this as customers, that this is really true. You can go faster with higher quality and not having more resources on the project. So, it’s really, really amazing to see now that customers start to adopt Total Test, how much value they gain from it.

Rick: Yeah, it’s a game-changer with regards to technology. I know that most organizations for mainframe software development have relied so much upon what I call black box testing, where we make our modifications to our code or create our code, and we’ve got test data that we typically use or pool from production; we still have to get through a masking exercise, and actually that’ll be a topic for another podcast, someday. But then we run that code against that data and then we essentially manually evaluate the results of that run. And it has served us okay. It’s fraught with potential error, and the biggest concern is the time that it consumes because it’s such a manual effort, it requires a lot of time and again, you’re essentially at the mercy of the quality of the person doing the analysis of the test results. So, what you guys are doing with Total Test is amazing, and it’s going to have a significant impact on the industry and the ability for organizations to do a better job of testing in a shorter period of time.

You mentioned the terms, and I want to dig into this a little bit, virtualized and non-virtualized testing. Can you talk to the audience about the differences between the two?

Steen: Yes, absolutely. I can start to say that we started out by talking about unit testing, functional testing, integration testing, and found it confuses customers when talking about these terms because they are no clear definitions. Maybe unit testing, but for the other terms, different customers are using different terminologies. But when we go down to the core, what everybody are doing, they’re testing a program, and when you’re testing a program as a developer or a tester, you need to do it in two different ways. Or what you really need as a developer, you want to test the program in isolation, you want to be able to virtualize or stub out all the external calls from your program. So you as a developer in a specific test scenario is able to tell the program what data it gets. And this way, you know the source code, so you know if a certain API call to CICS or a sub-module call returns specific data, that will go into a certain if statement. So you can control that with what we call virtualized testing, and that is what you can do with Topaz for Total Test, you can actually automatically record a virtualized test case from executing your program. So that’s really a cool future. Automatically generate a test case with all the stubs, then you can play back the test case and execute the program and virtualize all these surroundings, no matter if it is sub-calls, DB2 calls, CICS APIs, access to datasets—all that is virtualized, so that is really the most shift left you can do in your development process by having developers testing the programs as soon as they make a change in the development environment without having any issues with data on the mainframe.

Rick: And if I could make a quick point, Steen, before you start talking about non-virtualized is that that’s so important in an Agile development world, because as we separate or segregate work into smaller units, the ability to virtualize pieces of the system or components that may not be there is essential to that developer getting their job done in alignment with other developers in the work that they’re doing. So, virtualized testing is such a critical capability for Agile developers. Talk to us a little bit about non-virtualized.

Steen: Sure, non-virtualized testing is then when you want to test your program in a real environment with real data. So, you are satisfied with the virtualized testing that the code works as you expect it to do, but you really want to see running in your test environment or your system test or your QA environment; you really want to see it running in the CICS region, accessing the correct DB2 subsystem, and so on.

So, that is something that you really want to do later in your development process. Well, it’s again, different from customers to customers. Some customers want to use it also in the development process, so we don’t dictate when you do the different kind of testing, but non-virtualized testing is something you really want to do to make sure that it really works in the real environment with real data. And that is something you can do, and what is unique with Topaz for Total Test is it is so simple to do the non-virtualized testing. If it’s batch testing or CICS testing, it is very simple what you can do, but extremely flexible, so we believe you can support basically all test scenarios that you would like. If it is a called module in CICS or CICS transaction you want to test, if it’s complete batch jobs with multiple steps and multiple programs that you want to test by verifying content at data set, by verifying data in DB2, all that is possible.

And the great thing here is it is so flexible and so integrated with other Compuware products that you for instance can use File-AID Compare to compare data sets after submitting a job, you can use Topaz for Enterprise Data to do test data management; that is actually some of the really cool stuff that will come within the July release that came out yesterday. So you can actually set up data in DB2 as part of a test scenario, so your test scenario becomes self-contained. And this way, the non-virtualized testing for batch and CICS, where you’re running it for real, can also establish the correct data that you need before running the test itself.

Rick: That’s awesome stuff, Steen. Tell me a little bit about… I know that developers and testers are going to leverage features and capabilities within these products at different times throughout the software delivery lifecycle. Can you talk a little bit about common use cases and how developers and testers are using these virtualized testing capabilities and non-virtualized testing capabilities?

Steen: So, the virtualized testing capabilities, that is something you’re sitting as a developer and you want to test your program, then typically the way that you used to do this, not virtualized, but the unit testing part that the most shift testing would be to use the debugger and step through the code. You can still do that, but now from that debug session that you have, you can then choose to generate a test case based on the actual data that the debug session has captured. You can even change data in the debug session and generate the test case.

So, that would be a typical thing you would do as a developer for the virtualized testing. And then you get all the stops, for all the sub-module calls, the data calls, you can immediately execute a test case and see that it works; it should do because it was the same as you just recorded, and you can then create new test cases, where you re-use the stops that you already generated. Manual change values so you can simulate external calls that would produce other data, and that can be really hard if you have a sub-module to get that sub-module to return some invalid data, if you want to do negative testing, you can do that with the virtualized testing part.

So, that’s truly how you would work as a developer, starting by generating some case cases, copying them, modifying stop values and so on to do that kind of testing.

Rick: And I love that because the way I see that being used and the way I have asked people to think about using that is that you do, as you say, the developer’s going to use the debugger as they are making their changes to the code to make sure it’s working fine, but organizations that can build an inventory of test case scenarios and actually include that into that build process, particularly in an automated fashion using some type of orchestration server, CI/CD server, Jenkins, Bamboo, whatever, the ability to execute those unit tests or those virtualized test cases as part of the build process is going to have a tremendous impact on the quality of what you’re trying to produce, and I believe a huge impact on time. Because we spend so much time in rework downstream within that software development process when problems are identified, sometimes way too late in the process, so the earlier and we can identify these problems and resolve them, I think the faster the overall throughput is and the higher quality.

You mentioned something, Steen, unit testing. And for the Java developer, as you said, unit testing using capabilities like J-unit is not foreign. For the mainframe developer, and again, a lot of us grew up leveraging, again, what I call black box type testing techniques to test. Talk to me a little bit or help our audience better understand what unit testing is from a mainframe perspective. Again, particularly for that mainframe software developer.

Steen: Yeah, so unit testing is probably the best-defined concept of the testing types. So, unit testing is testing the smallest executable code piece that you have, and for Java code that would be a method in a class, but for COBOL programming that is a COBOL program. So, the unit testing of the COBOL program is testing the execution of the COBOL program, and that could be a core module with a linger section, or it could be a main batch program that simply reads data sets, reads DB2 and produces data sets and produces DB2. And the reason why we’re not talking so much about unit testing anymore, we’re talking about the virtualized and non-virtualized testing, is it’s still the program that is the unit that you want to test. You can do that in a virtualized manner, where we can stub out everything that is where… I would call that the White Box testing because you have to be a developer that are understanding the code, and the non-virtualized testing where you’re running it live by simply executing it by providing the input data and verifying the output data, that is black box testing. So, white box/black box testing is basically the same as virtualized and non-virtualized testing.

Rick: And I love the concept of true/false testing using test automation, because you can establish your test criteria and then it’s just a matter of executing that to determine if it’s executing as intended, and I think that’s so important from an automation standpoint, and really adds to the level of quality that we’re trying to achieve.

Steen: Just one comment to the automation part, because you talked about continuous integration/continuous delivery pipelines and part of that running the virtualized tests. So, I really agree on that. But actually for the non-virtualized testing can also be part of such a pipeline and actually, we actually have a video for the launch that we did yesterday, that actually shows a pipeline where a developer promotes code with ISPW, that promotes code change from a test environment to a Q/A environment, that triggers a pipeline that will check out test cases from Git that we use to sharing test cases among developers and the pipeline. And that Git repository contains both the virtualized and the non-virtualized test cases.

So, after doing the promote, you can do automatically the virtualized test cases. If they go correct, you can then do with the deployment, so the production data sets at the Q/A environment and after that has been done, you can run the non-virtualized test cases in the real environment. And this way you have complete pipeline that will test your changes automatically by just, as a developer, promote your changes. It will automatically go through virtualized and non-virtualized testing, and you would be able to get feedback from that pipeline if you broke anything. So, it’s really powerful the way that you can do a lot of test automation with both of these approaches.

Rick: That’s amazing, dude. One final thing, because I know we’re running short on time. Tell me about the creation of test cases, because I know that that was, 10 years ago, that was an issue and it took—not that it wasn’t issue—it took time, and again, I’m looking to extract or reduce the amount of time in every piece of effort in that software delivery cycle. And I love what you guys have done and what you’re doing with Total Test in the creation of test cases. Talk to the audience a little bit about how that’s accomplished with Total Test, and then a little bit about how organizations are managing those test cases and making them available to multiple developers.

Steen: Sure. So, first of all, the creation of test cases, let’s say you have a COBOL program with a linkage section of 1,000 fields, that’s actually not a… that’s quite normal. Then that program might also read some datasets with records, also with a lot of fields, and if you want to set up that data manually, data for the linkage section, creating the records for the input datasets; producing all that data can really take a long time. So, how do you get that data? Can we do that automatically—and that is what we can do.

And I talked a little bit about it before, so when you are a developer, you execute a launch configuration with a debugger, and you debug through the code, you can, at that time, choose to generate the test case. So, in this way, the debugger, Xpediter, Compuware’s Xpediter, would then be able to monitor the execution of the test case as it does for debugization as well, but now it would collect all the data returned to the program and generate both the input data, but also the expected output data for the test case as part of that recording. And this could tremendously speed up the time for you to generate a test case, and this can be done both for the virtualized testing and for the non-virtualized testing. So, creating these new test cases is just really, really fast. And that is what we call true automation, true test automation. It’s not just about automating the execution of test cases, it’s also the automation of creation of test cases.

Rick:Yeah, I think that’s such an important capability and feature because if you’re asking developers to manually define their inputs and what their expected appear just again, it consumes time. With this approach, we’re able to capture that information automatically at the debugging time and then store that into a test case scenario.

And then what I love about it, Steen, is it’s reusable. I can reuse that test case manually if wanted, but I can also re-use the test case and execute or initiate it from an orchestration engine like Jenkins. Do you have any recommendations on the best way to manage those test cases, because I can see where they can become significant in number, is there a best practice with regard to the management of those test cases and how are our customers doing that?

Steen: Yeah, so first of all, the test cases, they are plain files in your local file system, so what we recommend is to share the files through a version control system, and there we recommend Git because that is highly used by basically everybody.

Then how do you organize the test cases inside Git? And then it’s actually a little bit longer discussion because it really depends on how your whole application portfolio is organized and how many applications you have, how many programs you have in your application. Typically, we say one application is one Git repository, and then you can divide your testing up into features in that application and use a folder structure to divide test cases up for different programs. So, it’s very flexible, the way that you can orchestrate this.

So, as I said, this is a long discussion. I can also mention that we have a best practice document at our DevOps documentation site at the web, where we actually have a complete document that talks about best practices of using version control systems for test cases, but also all other kind of practices of adopting test automation and particularly how to adopt Topaz for Total Test.

So, there you can also learn about one common pitfall that people do when they’re doing these cases: they are only testing the good path through the program where everything goes well. They forget about all of the negative testing, and that is extremely important to test negative testing as well. And also what is called boundary testing. So, you’re testing the boundary of values, like if you have an if statement that’s using a variable, you need to check or test, set up test cases, that check or test so the variable contains the boundary values of what it can be. So, that is also a true test thing that people…

Rick: You’re exactly right, it’s so critical. We’ll make sure that that document as well as the video you mentioned earlier, we add links to those pages on our Good Coding website. We’ll tell people how to get to that, but we’ll make sure that’s included.

I’m looking at the clock, and it looks like our time has about run out. I want to thank you, Steen. I think this is incredibly valuable information, and I think organizations that are serious about modernizing software delivery really have to take a hard look, a good look, at their testing process and start to think about how they can make those more efficient. Again, especially in support of an Agile software delivery process. So, thank you for joining us today, I appreciate it. I think this has been great information. And so, again, thank you and continue the good work. Matt, you want to wrap us up?

Matt: Sure, thanks Rick. Very interesting talk. We have a little housekeeping to do, first I’ll point out that we have some excellent content on with demos of Topaz for Total Test, webinars, there’s a really eye-opening white paper that shows some tangible benefits of automated testing. So, you can find all those in the resources section on

And for everyone out there listening, if you have any questions about automated testing or any other aspect of the software delivery system, join us this Friday for our live Q&A session, Office Hour with Rick Slade. That’s at 11 AM Eastern Daylight Time. If you’d like to submit your questions ahead of time, you can go on Twitter and use the hashtag #goodcoding.

We also encourage you to visit our Building a Better Software Delivery Platform page at You can set a calendar reminder for the Q&A session, view recordings of past Q&As and listen to all the podcasts in the series. So, with all of that taken care of, I’ll thank Steen one more time for taking the time to join us today and then get out of the way. So, back to you, Rick.

Rick: Steen, any closing remarks that you’d like to say?

Steen: I’ll just say that test automation is probably the most important part that you can get started on, if you have started a DevOps process. That is where you get immediate benefit and where you can really be effective in what the developers are using the time on. And we’ll be happy to help. Topaz for Total Test is the product that really will help you a lot.

Rick: I agree, concur that testing has the potential to provide the biggest business benefit and business impact with regard to helping you deliver solutions to your customers faster, more efficiently and with higher quality. So, again, thanks everybody. We appreciate your time today. I’ll mention it one more time, if you’ve got questions or need information about these podcasts and the live Q&A on Fridays, please go to to get, hopefully, what you need. Thanks again to Steen. Thank you for listening. And we’ll let you go. Everybody have a great week. A safe week. A productive week. And take care. Good coding, everybody.