Chapter 14: Monitoring and Measurement

Chapter 14: Monitoring and Measurement

Send Us Feedback

Listen on Itunes

Listen to Stitcher

Matt: Hey everyone, welcome to Chapter 14 of our series Building a Better Software Delivery Platform. In each episode, Compuware Executive DevOps Solution Architect Rick Slade examines a different aspect of the modern software delivery system. So, here’s Rick. Hey, Rick, how are you?

Rick: I’m doing great, Matt. I hope you are.

Matt: I am, thanks. Here today to discuss the role of measurement in the SDS is Compuware zAdviser Product Owner, Bill Mackey. Hi Bill, thanks for joining us.

Bill: Hey, Matt, thanks. Thanks for having me.

Matt: Thanks for being here. So, Rick, as I said, today’s topic is measurement. Can you give us a little bit of an overview on why accurate measurement and analysis are so critical to modern software delivery?

Rick: Yeah, glad to, Matt. Measurement to me is, it’s one of those critical components. It’s a discipline that is so important in the management of essentially anything. There’s an old saying, if you can’t measure it, you can’t manage it. And I truly believe that’s true, I think it’s particularly true with regard to software delivery. When I first came to Compuware, it couldn’t have been a couple of months after my first day, I was looking around and talking to folks about different projects, research work that was going on, and I ran across Bill and Jim Seronka and what they were doing with regard to providing customers information, metrics on the use of Compuware products within their software delivery ecosystem. And I looked at that and I started thinking about it, that a wonderful technology or capability to support a continuous improvement program.

We talk about, from a DevOps standpoint, we talk about continuous development and continuous integration, continuous testing, continuous delivery, continuous everything. I think one of the most important things with regard to measurement is the support it provides to continuous improvement. And I think that is as we solidify our software delivery systems, they will continue to evolve, they’ll continue to get better based on new technologies or new ways of doing things, new learnings around process. We will constantly strive to improve how we deliver software, the market’s going to dictate that. It’s one of the reasons that I think that you look at the software delivery ecosystem as a system, a critical business system that we’ve talked about in prior podcasts. But the way to ensure that it continues to operate at maximum efficiency is to measure its effectiveness.

And so, seeing the work that Bill and his teams are doing around a product that we call zAdviser, and we’ll talk more about that today, it’s great information, great dashboarding for people, stakeholders responsible for software delivery and its effectiveness within their organization, So, real glad to have Bill with us today. Bill, Matt said, is Product Manager for zAdviser products, but I won’t say any more. Bill, I’d like to turn it over to you to have you introduce yourself and your role at Compuware and maybe a little bit about what you guys are doing.

Bill: Thanks, Rick. So, first of all, I’ve been the Product Owner for zAdviser for the past three years. I’ve been with Compuware for 30 years, primarily started as a File-AID SE, then became a File-AID manager. Then, about five years ago, I entered the product management field. And that led me to zAdviser, which I’m sure we’ll get into a little bit today, which basically is our answer to measurement, and we’ll talk about that. On a personal note, I live in Atlanta, Georgia, have a wife, three kids, with my final child going off to college this fall, God willing.
Rick: Congratulations.

Bill: Soon to be an empty nester.

Rick: It’s not all bad.

Bill: Yeah, so yeah, that’s me.

zAdviser

Rick: Well, great, good deal. I know that you and the guys have done some tremendous work. Let’s get right into it. Tell us a little bit about zAdviser, how does it work? And tell a little bit about what motivated Compuware to create it.

Bill: Okay, well, great. The simple answer is zAdviser is a modern mainframe measurement tool that uses machine learning to measure and improve a development organization’s quality, velocity and efficiency.

So, just a few words about metrics, because you mentioned it already, but metrics have become an important component of Agile development and working in a DevOps environment. And one critical piece of this is continuous improvement. So, continuous improvement requires us to continuously challenge ourselves to get better. And to do that we have to be continuously measuring ourselves, and that’s where zAdviser comes in. It captures and creates KPIs that we can use to measure ourselves.

Rick: KPIs being key performance indicators.

Bill: Exactly, exactly. As far as kind of how it works, I mentioned that it captures and creates KPIs. But the way it works is that it’s a SaaS solution, it sits on an AWS platform. So, there’s nothing to install for our customers. Our customers just have to send us the data that the development tools that they use are creating in SMF records. So, all the Compuware tools, the development tools that we have create SMF records. Customers send us that, we load that into the AWS Cloud, and then we can use machine learning techniques and modern dashboarding techniques to give that information back to them to help them improve. Besides Compuware tools, we can also ingest things like whatever SCM they might be using. Compuware’s got ISPW, but if customers have other tools like Endevor and ChangeMan, we can bring that data in, and then ITSM tools like Jira, Service Now Rally, tools like that, where we have information about our ticketing and how our bugs, how much work we’re doing on different activities. That kind of information is also ingested and it gives us the ability to create these KPIs.

Rick: So, you’re not only pulling information from our tools, but you have the ability to ingest other metrics from other providers, other tools that can be leveraged within our dashboards with regard to the software delivering process.

Bill: Yeah, exactly, and the sky is the limit on that, and we’ve got some customers that have come back to us to say, “Hey, can we give you other data that you’re not asking for because it’s important to us?” And they’ve been able to supply with CSV files and we’ve been able to adjust that data as well. So, one of the nice things is there’s an unlimited amount of data we can put in and there’s tons that we can do.

You asked me also, what kind of motivated us to create this, and it was really the brainchild of Chris O’Malley, the Compuware CEO. When he got to Compuware almost six years ago, what he saw was a struggling software company that was developing 100% in a waterfall method. And we were putting out new releases about once every 18 months, like every other major mainframe vendor was at the time, and our attitude was, “This is how we’ve always done things.” It served us well for the first 25 years or so.

Rick: It served a lot of people well for 25, 30 years.

Bill: Exactly. And basically, to be honest with you, I think the industry, as a whole, apathy set in, and we got complacent. So, Chris had this vision of all the mainframe vendors being just a bunch of tired old dogs lying on the front porch, and he really wanted Compuware to become the Greyhound and sprint away from that group. So, he enabled—I shouldn’t say he enabled—he insisted that we burn the boats, move to Agile, and one of the things that sprung from that was creating a way to measure ourselves. And we had this old, what we called a VIP program, where we were being able to go back to customers and saying, “Hey, here’s how you guys are using, or here’s how much you guys are Compuware tools”. It was just a bunch of counts on, “Hey, you guys used File-AID 5000 times this month. You used Xpediter or Abend-AID, or whatever it might be.

It was helpful when it came time for customers to evaluate whether they were getting value out of the Compuware tools. So, we kind of took that as the starting point and developed zAdviser. And then on top of that, Chris insisted that this be a free service that Compuware provides to its customers, and always remain that way. So, what we’re at today.

Rick: I think that’s what it’s amazing about it, this capability is provided to any of our customers who have any of our core products, as I understand it, which is amazing. I think one of the things that I’ve preached for long before I came to Compuware and talked about it was the importance of capturing the right metrics—and we’ll talk about what those right metrics are in a few minutes—but being able to capture those real-time with the machine, I think that’s what’s important. The last thing I think you need to do or want to do is to ask your developers to execute another task with regard to the capture of this information. So, being able to collect that information from the work as it’s being done, by the software, is a tremendous benefit from a time-saving standpoint, which is a lot about what a modern software delivery system should be, so I think that’s cool, being able to capture that,

I love the idea that you’re able to extract or to leverage collected information from Jira because then we can start to look at things that really make a difference from a business standpoint, the mean time to resolution, time to market, cycle times—there’s all kinds of different things that we can leverage from a business perspective that will help us identify those key capabilities within that software delivery system that will allow us to maximize our effectiveness and our efficiency with regard to getting software out the doors. And I think that’s incredibly wonderful. Tell me customers, how many customers are currently participating in this with zAdviser.

Bill: Well, as of this week, we currently have 161 unique mainframe customers participating, so the number is rapidly rising. We hope to have every mainframe customer participating as the word gets out and I’m hoping this podcast helps in that regard.

Fast Feedback

Rick: Can you talk a little bit about the advantages, the information—why should a customer become involved in this? Why should they be collecting, sending us data?

Bill: Well, I think what the customers are benefiting from is number one, most of them are kind of where Compuware was five years ago. They’re trying to wrap their heads around, how do we become Agile? And how do we implement DevOps? And we’ve been there, we’ve been through the whole thing, so we can help them out. So, what we did probably five years ago was we did a study with Forrester to establish what are those KPIs? What should a development organization be concerned with?

So, we came up with… We came back with about a dozen KPIs and then we went about figuring out how to create them. And, because of the KPIs that we’ve created, customers now have a way of starting measuring and measuring is what they need to be doing. And that’s the one thing that most of our customers and Compuware itself was doing, is we were just kind of guessing on how to improve things. And that works in some cases, but not in all, and not particularly efficient.

Rick: I think true transparency, I think a customer sees a lot of information, they can see where things are going well, where things may not be going, where to focus their attention and their efforts with regard to the software delivery system.

But I know it provides us a lot of information. I think it’s a wonderful way to close that feedback loop. If you think about continuous delivery or continuous information in a DevOps model, one of the things that’s important is getting feedback from users of your technology and tools. zAdviser provides a way to do that without asking a lot from our customers with regard to the capture of that information—it’s pretty much done automatically. But it helps us understand where we need to focus our efforts and resources with regards to what’s working, what’s not working, in the delivery of our software delivery capability. So, good stuff.

Velocity KPIs

You talked about KPIs, Bill. What are some of the KPIs that zAdviser measures for our customers?

Bill: Well, currently, we have about 10 KPIs that we can reliably share. Now, we don’t take the term KPI lightly. We probably have 800-1,000 different data points or metrics inside of zAdviser that we can give back to customers in different dashboard forms. But when we talk about KPIs, we’re talking about the things that, at all levels of the organization, customers should be taking a look at. So, some of those include things like you mentioned mean time to resolution—Mean Time to Repair (MTTR) is what we like to call it. This is a velocity KPI. It measures how quickly a development organization is doing things like fixing bugs. This is from the time a bug is found until a time when we can deploy a fix back to production. So, we want to make sure that this is done in a timely manner for our customers.

Rick: And that information is typically coming from your Agile management system, Jira?

Bill: MTTR is one of the KPIs that comes primarily from, say, Jira or ServiceNow or wherever you’re tracking your bugs. So, we know when it was created, and we don’t ask for a whole lot of information, actually, out of these systems, just mostly dates. When was it created? When did work begin on it? How long did that work happen? What was the QA process and how quickly did we get it deployed back out into production? So, these are just dates, and that’s what these systems supply us. So, it’s not a whole lot of information, and it’s certainly not information that’s sensitive, it’s just dates.

So, MTTR is a good one. A couple of other velocity KPIs are the number of packages and components a team or organization can deploy in a given period of time. So, if we look at a two-week sprint, for example, as a timeframe, how many components are we getting done as a team or an organization or at any level, you can slice and dice the data, and that’s the nice thing about these modern mainframe tools, is you can actually break it down into the individual, the team, the organization, any date range you want to do, and you can see these numbers. So, you can kind of compare yourself to yourself week to week, month to month, quarter to quarter, and work on improving, and that’s what it’s all about, right? It’s measurement and improvement, tweaks and improvements.

Rick: Do you have the ability to compare yourself against others, maybe in your industry or other like-type companies?

Bill: Great point, and this is one of the things that we’re doing in the current quarter is what we call benchmarking. And what we’re going to be able to do for our customers is we’re going to be able to provide… We’re going to start with industry-wide benchmarks, so for example, and these are going to be… we’re looking for, initially kind of the lower hanging stuff, apples to apples comparisons, like an MTTR number. What we want to be able to tell customers, “Here’s your MTTR and here’s the industry standard MTTR.” But we’ve also built into zAdviser the ability to break things down by industry verticals, and that may not be this quarter, but it’s coming, so that a company could compare itself to its vertical, and then also a company can easily compare itself to itself just by going back sprint to sprint, quarter to quarter and watching the number move.

An interesting story there about mean time to repair is we’ve got Compuware data going back 15 years because of our VIP program, so we were actually able to look at our MTTR back when we were waterfall and then watch it over these past five years. And we used to have an MTTR of something to the effect, on average, of like 50 days to fix a bug, and that’s because some bugs were taking us years. And that MTTR is now down to seven or six days. So, we’ve watched that come down slowly, and a lot of that’s just due to the tweaks that our development, lead development folks have made. Things like creating automated tests so that we don’t create as many bugs to begin with, and then figuring out the right ratio of people fixing problems to people creating new innovation for our customers, which is obviously critical to us.

Rick: And that’s the beauty of being able to capture those type of metrics; you can see where the latency in your software delivery system is. Where the gaps are, where you’re seeing slow down or wait times for work to be done. I think it’s incredibly important.

Quality and Efficiency KPIs

Okay, I know that I’ve seen in some of the dashboards, metrics around quality and efficiency. Can you talk a little bit about those as well?

Bill: Sure. A couple of the quality KPIs that we include are things like percentage of abends escaped, and this is basically saying how much of our total abends, how much of it is actually getting out to production versus the percentage of abends that we’re capturing in test with things like automated testing and whatever our QA is. So, again, if we’re seeing too many abends getting into production, that’s generally a quality sign, and we can start focusing on creating more and more automated tests, which has been proven to reduce the number of abends getting into production. So, over time, because we’ve gotten… You get so much better over time at creating automated tests, especially for the really easy bugs, the ones that they’re easy to find and fix that all that’s left are the really complicated, difficult bugs. And there’s a lot less of those things, thank goodness. So, abends escaped is one of the things that we track.

And then we also track things like fall back ratios, how often in our SCM are we having to roll back out of production things that we’ve implemented. This is a major issue. If we roll out changes and then have to roll them back to a previous release, that’s a big deal. We don’t want to do that very often, but it happens, so that’s a KPI that we track. And then we even track things like demotions versus promotions within the SCM. This is less of a KPI and just more of an important SCM metric, but we like to see a certain ratio of demotions to promotions. We’ve got a QA group and a testing phase in our software lifecycle, so we expect to find problems and then demote them back down to development, but we don’t want that to be a huge number or that means that we’re not doing a good job in development. So, I keep an eye on that metric as well, so that’s some of the quality KPIs.

And you mentioned efficiency. The biggest thing for Compuware, an efficiency KPI that we measure is what we call a percentage of innovation. And what this is, is the ratio of how much time our development organization is spending on new innovation, new features and functions that we’re giving to our customers, versus the amount of time that we’re spending on things like fixing bugs and general maintenance. So, we need to know that number because we need to have our personnel in the right ratio as well. So, if we know that we like to do 80% innovation, 20% non-innovation or maintenance, we can develop our staff that way.

The really cool thing is, is if product management comes to development and says, “Hey, listen, we know this quarter, these are our objectives, but we’ve got to get these two new features into the product. Critical, immediate, has to happen. Our development side of the house, our management there can say, Okay, we’re going to take, of our 20%, we had our maintenance, we’re going to add… Half of those people are going to go into new development to help finish these things. But there’s a cost, and we know what that cost is because we’ve been measuring these numbers over time. So, we know if we go 90% innovation, 10% maintenance, that we’re going to get a backlog of maintenance issues and we’re going to have, in the next quarter, we’re going to have to make the decision to potentially put more people in the maintenance arena. And those are the decisions that our management can make with hard numbers.

Rick: That’s great data for planning purposes, with regard to that work. And from a resource management standpoint, that’s just incredible. Awesome, good stuff.

Customer Engagement

So, tell me a little about process for customers to get engaged with this, and then what it takes from their perspective to be a part of this.

Bill: Well, it’s actually very simple. So, one of the nice things about the Compuware… all of the Compuware tools create SMF records and they basically are putting out records as programmers do work like… Let’s say to debug a program, a record that says, we debugged a program, here’s the program, here’s what they did, how they did it, which is kind of what we call the developer DNA. And that just goes into SMF. And then, so, we have a lot of customers that are joining zAdviser and then saying, “Hey, we’ve got all this SMF data, here’s a year’s worth of SMF,” and boom, we can automatically and immediately have what they’ve done and how they’ve done it. We can come back to them and say, “Here’s what we’re seeing. Here’s your KPIs.” So, all they really have to do is turn on to zAdviser, which is fairly simple, it’s just basically sending us the data that they’ve already got. If they don’t have SMF data turned on, they just have to turn it on and start sending it to us. That’s the beauty of it. So, it’s just a matter of either streaming us the data as it gets created or sending us an extract of their SMF data every so often once a week or once a month, and the data gets noted directly into an AWS cloud SaaS solution, and they can log on and look at their data and begin. They’ve got measurement automatically. It’s really quite easy.

Rick: So, the turn around on being able to see the benefits of that data is pretty quick, then?

Bill: Yeah, it’s almost immediate. We had a customer just yesterday send us their SMFs, their Endevor data for their… Their Endevor data, and this morning, they’re looking at that data in a dashboard.

Rick: Wow, that’s amazing.

Bill, Yeah, we’ve got it all automated on our end. So, as the data comes in, it gets loaded, emails go out, saying take a look, and simple as that.

Rick: So, free service, that’s what’s amazing to me.

Bill: Absolutely.

Rick: Free service, if you’re a customer, then all of this is available to you. You’ve just got to send us your data.

Bill: Yeah, as far as I know, we’re the only mainframe company doing that, so….

Rick: Yeah, and that’s incredibly valuable. We get a lot of… It’s really valuable for us, too, so I certainly understand why we’re doing it. Good for customers, good for us. It’s a win/win for everybody. We can be much more poignant with where we’re focusing our resources and time and effort based on that feedback. So, helps us as well.

What the Future Holds

Tell me a little bit about what you’re doing next. Anything exciting in the pipeline?

Bill: Oh man, there’s just so much that we can do. And as we talk about this and you’re bringing up things we could… The sky is the limit. But one of the things we’ve been working on last quarter, or continue to work on this quarter is what we call development productivity dashboard, which is going to allow our customers to see kind of a breakdown of their SCM lifecycle. So, that’s beginning at checking out of a component and ending at the promotion to production. So this, because ISPW is so a data-rich and can give us this information, it allows our ISPW customers to truly see where the components are spending their time in the life cycle, whether that be prod, test, QA, Dev or whatever different level or phase that it’s in.
So, one of the great insights here is that they can start to see bottlenecks, and outliers show up easily, so that dev managers and team leads can make adjustments and improve. And besides all this, zAdviser also tracks work in progress, which is work done to components in a sprint that weren’t promoted to production. So, why are we working on things that we’re not moving to production and make sure that we’re not getting side-tracked by certain issues, so development or understanding the whole life cycle of development and where things are, is one of the things that we’re spending a considerable amount of time in right now.

Besides that, I kind of mentioned benchmarking. So, that’s pretty critical to us. We’re going to start delivering industry-wide benchmarks on KPIs and other metrics to all of our customers, so they can start to understand what good looks like. Customers want to be able to compare themselves to both themselves, their verticals, and the industry as a whole. From there, we’re planning on revving up analyzer and start to give customers prescriptive actions. Much like Netflix can recommend movies to you based on your past movie viewing behaviors, zAdviser will soon be able to recommend actions to take to improve development techniques based upon development habits. Isn’t that pretty awesome?

Accessing the Dashboards

Rick: Machine learning and artificial intelligence, AI, for the software developer, mainframe software developer. That is too cool. That is really cool. And in such a pragmatic, practical way of doing that, Bill. I love what you guys are doing. How is this information, these dashboards, how are they… How does the customer leverage them? Through a browser or how does that happen?

Bill: Yeah, it’s an AWS cloud solution, so they just… Obviously, security is our number one priority, so we’re giving our customers… Some of it is being done through our account group right now. Others, we’re starting to roll out logins to customers. The customers can log on their own and view it in their own browser.

The other nice thing that were starting to roll out as well, we’ve only done a couple of customers so far as kind of a trial basis, is that were giving customers their own private tenants, AWS tenants, so that not only can they view their data, but they can start to create their own dashboards and wait their data anyway they want. And we’ve got some customers… We use ELK Stack Kibana as the language and some of our customers have got some proficiencies in that area, and they’re starting to create what’s most important to them.

Oh, one other thing to mention, you asked me about where we’re going. We’re now part of BMC, so we’ve got another potential 160 products worth of data sources that can be entered to zAdviser. So, we’re looking forward to working with our BMC brethren on that.

Rick: It really is amazing. It sounds like it’s an easy… Not a lot of work for our customers other than getting the data to us, and once that’s done, or determined, then this becomes a repeatable process. So, easy for the customers, easy for us, but incredible value with that information.

Bill: Rick, it’s a no-brainer. Absolutely a no brainer.

Rick: So, if I’m a customer listening to this, how would I get engaged with that?

Bill: Well, you can start by just talking to your AC, your account consultant, your TAM, if you’re in Europe, and just say, “Hey, this is something I want to be a part of,” and we’ll get you rolling from there. We’ve got online documentation that steps you through, basically all you’re doing is setting a couple of parms to turn it on, so it’s as simple as that. You can be up and running within a day.

Rick: And I’ve seen demos of these guys showing the dashboard to customers who were looking to leverage it, and it’s very impressive. So, easy to get a demo if you want to see it?

Bill: Yeah, and lots of good webcasts out there too, on Compuware.com.

Rick: Tons of information on this thing, folks, if you want to take a look at it. Again, continuous improvement is just vital to the management of any solution, and again, you know my feelings on the software delivery system. I think it’s the most critical of organizational systems today in the world that we live in, and having that information that will allow you to understand where you’ve got potential problem areas within delivery eco-system and where you have the opportunity to make improvements. It’s just going to make for a better software delivery environment for your organization, so I hope you’ll take advantage of that. Measurement is absolutely critical.

Bill, we’re at the close of our time. I want to thank you. Man, this was awesome. I know you love talking about this as much as I love talking about software development. I appreciate it, I appreciate your passion and what you guys are doing, it’s great. Any closing, remarks for us?

Bill: Like you, I’m passionate about this, love talking to people, love talking to customers about it. Feel free to reach out to me, [email protected] And again, Rick, thanks for having me on the podcast.

Rick: Thank you, brother, we appreciate it. Matt, you want to close us out?

Matt: Sure, thanks Rick and thanks again, Bill. Really interesting stuff. And to our listeners, thank you for joining us. Be sure to join us this Friday at 11 AM Eastern Daylight Time for our live online Q&A session, Office Hour with Rick Slade. Bring any questions you have on measurement and or the SDS in general, or submit them ahead of time on Twitter using the hashtag #goodcoding. You can set a calendar reminder for the Q&A, or check out the rest of the series at https://www.compuware.com/goodcoding. And now I’ll throw it back to Rick for some closing thoughts.

Rick: Thank you, Matt. And to let everyone know, Bill has graciously accepted an invitation to join for Friday’s live Q&A on this topic, so I hope that if you’ve got questions about this or just questions in general about metrics and measurement with regard to the software delivery system, I hope that you’ll join us and ask those questions. I appreciate Bill joining us, joining me for that.

Bill, again, thank you. I appreciate everything you guys are doing. I appreciate you guys for listening, for downloading this podcast, I hope it’s a productive week for you, a safe week. Take care, everyone. We’ll talk to you soon. And good coding.