Craig: Good day and welcome, thanks for coming back to our CSI podcast from Compuware. This week, we have a very special pair of guests from the Southern Hemisphere. We’re going to be talking to Lynette and Jakes in just a while, but I’d also like to welcome back our trusty investigators from last week, actually, Atul Bhovan and Stuart Ashby. Say hello, guys.
Atul: Hello, everyone.
Stuart: Hello, Craig.
Craig: So, Nedbank and Mediro are the two companies that we’re going to be chatting to today, and this is a couple of contacts that Atul has been working with over the last few years. So, Atul, before we get in to speak to the guys, just give a bit of a background on Nedbank, and how they’re working with Mediro and explain who Mediro is.
Atul: So, Nedbank is one of the four major banks in South Africa. They’re pretty advanced, in terms of their thinking, and they really want to move forward and challenge the other banks, the other players in that marketplace. So, what you hear today, hopefully, will give you more confidence that as a mainframe organization, a mainframe-centric organization, there’s a great possibility for you, yourselves, to advance the way you work, the way you think, and bring your people as well, through a culture change. So, highly significant modernization effort taking place there at the moment.
Now, Mediro, they are our partner organization down in South Africa. They’ve been working with us for many years now, to help our customers not only implement and use our tools, but to think ahead, to think outside the box and really advance the way they utilize solutions. And their engagement with Nedbank has really been around transforming not only their SCM, but their automation processes on the mainframe as well.
Craig: Great stuff. Okay, well let’s get over and speak to Lynette and Jakes. So, good morning, guys, Jakes Olivier and Lynette Kleynhans from Mediro. Guys, thank you very much for joining me this early morning. I’d love for you guys to do a bit of an introduction to your roles. Jakes would you like to kick us off?
Jakes: Of course, yes. I’m the DevOps lead for our DevOps team, focusing on the mainframe side of things. Got about 25 years with Nedbank, now. Just short of 25 years. Started off straight after school in the print room, in the mailing room, and pretty much worked my way up through the ranks. I got an opportunity to do COBOL programming training through they Nedbank and by the end of 2001, I started my life as a COBOL developer. From there onwards, moved into the test environment space, where we manage the mainframe test environments and coordinate the testing efforts in that environment. After that, moved into software development tools and that also gave me a good opportunity to get involved with the distributed side of things. That’s pretty much led into DevOps.
Craig: Why we’re talking to you, today.
Jakes: Yeah, and it’s just been growing for the last two, three years.
Craig: Awesome. Good stuff. Well, let’s dig into that in a moment, but Lynette, please say hello to our listeners and just explain Mediro and how you’ve been working with the guys at Nedbank.
Lynette: Yes, I started at Mediro three years ago. We did an ISPW migration, that was my first big responsibility with another big bank in South Africa. That migration we did in six months, but in the meantime, Nedbank had started looking at ISPW from a POC perspective. They liked what they saw, and then we moved across. There were some really good tried and tested opportunities or things that we did with the developers, as well as from a management perspective, that really worked, So, we tried to bring that into Nedbank where it made sense to do that. Things like rolling out the applications, not in a big bang approach, but by business area and with it then train the developers in the same way. So, that really worked very well. It allowed us to do the migration at Nedbank rather quickly, as well. But Jakes can cover that.
Craig: Excellent. So, yeah, we’ll get into that. And I think guys, I’d love to kind of cover the Nedbank story from a DevOps transformation, but obviously get into some of those key components. And Jakes, I’ve been exposed to some of your previous outbound sessions, where you talked about that journey and some of the things I’d like to pick up on it was the fact that you talked about integrations with other components within a DevOps toolchain and why that was important to helping you guys start your journey. So, could you share why and how that’s accelerated your early stages of developing a CI/CD pipeline?
Jakes: Yeah, So, originally, the focus was more on, let’s call it our digital transformation journeys. And it was focused at your distributed front-end and middleware systems to start off with, and we quickly realized that we don’t want the mainframe to fall behind and any chain is only strong as its weakest link, So, we couldn’t afford to not look at the mainframe. And a lot of the disciplines were already sort of ingrained, and we just leveraged off that. So, for instance, stuff like SonarQube was pretty much already in the organization; Jenkins was being used widely. We’ve also started looking at release orchestration by XL Release, and we started looking at ISPW and it sort of just all fitted in very nicely, So, we were very lucky in that sense, that the ISPW integration into XL Release is straightforward. We had experience with scanning our code through SonarQube, So, all of those pieces would have just fell in to place fairly easily, So, we were leveraging off what we had already.
Craig: Great stuff. And I heard you reference it as an open source hybrid, which kind of seen some bold moves towards things like source code management in Bitbucket and having integrated that into ISPW gave you another speed bump along the way. Can you walk us through how you managed to get the mainframe community on board with taking on some of those tools that they’re not necessarily familiar with?
Jakes: Yeah, So, it’s always… when you start that change journey, you always get that resistance. Inherently, people do not want to change, but I think the toolchain sort of spoke for itself with a simple demo of maybe 30 minutes to an hour. We managed to get the developers fairly excited about this journey, and that sort of was half the battle, won. I know in the past, some migrations were like pulling teeth, and users were kicking and screaming all the way through. In this case, I think the benefits were So, overwhelmingly obvious that we did not have any resistance once the people grasped the concept of the possible, they actually started pushing us to get on board, which was a very pleasant change from what we experienced in the past.
Craig: So, that’s great. And lovely to see that that hybrid approach is still being adopted by non-mainframe people. We see a very similar story, elsewhere in the world. But I think key to your story Jakes, is how you managed to do such a slick migration and more importantly how your senior management was supportive. Obviously, they had their questions. How was that engagement? Talk us through the migration process and how management supported you to make the change.
Jakes: Initially, we sort of reckoned a year plus, based on past experiences around migration, and we know that the training aspect of pretty much 200 plus people, is also a major initiative. And sort of trying to balance that with all the work in progress, not sort of impacting our current projects, was always a concern as well, but we, we decided we were going to try and take a very aggressive stance and sort of approach business units. Do sort of an initial share-and-tell, almost, just to give them that idea of the art of the possible and sort of start creating the excitement within the teams. And, like I said earlier, the excitement was there, So, the people were willing to take it on, and saw the benefit, So, we had very little resistance from the development, developer community, but in saying that as well, we had tremendous support from our leadership.
Troy was great, Devi was involved with the entire process, and we actually had weekly sessions with her and any sort of issues, blockers that were raised, we sort of knew that we could pass it on to her and she would hear it for us very quickly. So, the focus was there all the way through, we had the support, and we had the buy-in from the users. So, all in all, I think those actually contributed tremendously to us actually completing the full migration in six months, as opposed to what we thought would probably run for more than a year.
Craig: Great to have both of those elements, both senior management and the people on the shop floor to desire that full. And Lynette, talk us through Mediro’s approach. I guess your approach, your experience in helping Nedbank with that transformation.
Lynette: Yeah, what I feel, and what has worked on more than one occasion, is definitely to have the collaboration of the team that you need to make the installation successful. So, we had buy-in from the NBS sysprog, from security to the dbas, from a storage management perspective, they helped us when it came to creating datasets and ensuring that the SMS rules were in place and all of these sort of things. The record guys were great and then, once we brought in the tools team, they were very keen and eager to get going on learning the tool, and supporting it.
What I find is very important, is to get the admin guys, the people that are going to be involved in the setup and the configuration of the entire ISPW, the lifecycle, your deployment process, how do your access rules work, how do the skeletons work, what is the requirements that will meet all the different scenarios within your environment, getting them involved from the very beginning So, that they have an understanding of how this beast is put together, how ISPP hangs together and how it works within their environment. So, it was very easy. We had Darryn do a lot of the stuff as well. He being a non-mainframe person, he adopted, or adapted, to working on green screen, to do the admin tasks really quickly and really easily. So, it’s a very easy tool to learn and a very easy one to get a hold of, or a hang of, and it is very important to actually bring those guys in right from the beginning and I think that really worked for us.
Craig: So, as we can hear that from the guys, the experience that Mediro brought to the table was invaluable in really kind of starting to accelerate the adoption and usage of the toolset. So, Stuart, from your experience in dealing with companies around the world, how valuable is that?
Stuart: I think it’s terrific, Craig. Everyone is special. But every customer can leverage something that Compuware has learned with an implementation and make it go faster, their implementation go faster with the lessons learned, just applying appropriate adaptations. No need to reinvent the wheel.
Craig: Excellent, let’s hear some more from Jakes on how that transformation went. Jakes, Lynette speaks of Darryn George and other members of the team. Could you let us know how have your distributed community got involved with this project, and the kind of skills that they’ve brought to this and how they have gone on the journey with you, embracing the mainframe?
Jakes: Yeah, Darryn is worth a mention. He’s a special talent. He’s really what I’d like to call a multiple-discipline specialist. He operates across multiple technologies, multiple platforms. So, he’s really a CI/CD specialist across the board. So, anything from… to mainframe, basically. He’s a real champion with, you can imagine, a team of DevOps engineers, there’s no shortage of talent within our team, but he’s been instrumental in a lot of initiatives, including this one.
But in saying that, there’s a lot of other teams, like Lynette mentioned, we’ve got our COBOL team of excellence, they assisted a lot around, SonarQube, setting up standards and quality gates. So, it was really a team effort, and everyone sort of played their part and yet the mainframe guys, they have very little experience from the distributed pipeline sort of experience, but I think building it into a nice process that’s coordinated, and that makes sense, just helps them to not fear it and fear the unknown. So, yeah, that was really a team effort. There’s a lot of people that were involved and everyone did their bit, and that just all contributed to a hugely successful effort.
Craig: Great, and Lynette, from your perspective, having somebody like Darryn George, that multi-discipline CI/CD specialist, how did that help you engagement with you guys?
Lynette: It definitely made the installation on the roll out of the ISPW, as well as all the Topaz tools, a lot easier for us.
Craig: Taking you on that next step in the journey, then Jakes, I understand that end-to-end orchestration was identified as the next step in that journey. And you worked with our friends, XebiaLabs, to implement that end-to-end piece. Could you share this what the benefits of that was, and how an end-to-end orchestration accelerated your delivery capability?
Jakes: The nice thing about XL Release is the fact that the default ISPW plug-in integration works great. Since we’ve started using XL Release with ISPW, we’ve identified some minor improvements to the process. But off the bat, we pretty much built something that’s straightforward, easy to use, easy to understand, and the benefits of using XL Release became evident very quickly. And to sort of give you an idea, in the past the admin around the change management process, so, that’s pretty much creating your documentation, listing the artifacts you’re going to work with, and which versions, what type of programs they are and sort of your implementation plan. So, you had these spreadsheets and Word documents that sort of controlled the implementation So, you had to sit there with the Word document and track the steps and make sure everyone does what they need to do. Which includes the dbas, CICs guys, I/O service providers. So, all of that used to be a big admin overlay. So, with the introduction of XL Release, we have the ability to extract the list of components that’s being promoted. We have the orchestration process defined within XL Release. So, the benefit there is, immediately, our developer community doesn’t have to do those admin pasts that could have taken up to five hours with the change. Obviously, depending on the size and complexity that might go up or down a little bit, but effectively we’ve now removed that admin work almost completely. So, where we used to sort of go from dev to prod, your admin would take you about five hours. Now, your admin takes about 18 minutes and most of that is just filling in your details about your change on XL Release and sort of starting that release and completing some of the manual steps in the process.
The rest is pretty much taken care of. And the nice thing is we built in the SonarQube scans in there, so we know we’re getting good quality source code. We’ve got our unit tests in there. We’re looking at getting our functional testing in there soon and those are going to just increase the benefits. So, we still have quite a bit of manual effort around our testing and that’s probably taking up the majority of the time in releases. So, we sort of already know where’s the next big win for us, focusing on automated testing, and that should increase our time to production quite a bit, as well.
Craig: That comment there, 5 hours to 18 minutes through driving out monotonous documentation and manual processes. It sounds like a Nirvana. You must be very popular with your team.
Jakes: That’s typically a response we get quite often, and it’s almost their disbelief. “Really? Five hours to 18 minutes?” It is, and I like to sort of do that when I do a demo is sort of start off and say, “Well, the demo should, hopefully, not take more than 18 minutes.” So, yeah, I think the benefits are obvious. And that’s not Mickey Mouse. It’s not like we’re saving minutes and it’s just going to grow. We’ll keep on pushing it to that ultimate vision of having a pipeline that just does everything for us.
Craig: So, there we hear, guys, end-to-end orchestration with our friends from XebiaLabs. I think it’s something that we aspire to. And Atul, what was the experience, and I guess the exposure that you saw, while working with the guys?
Atul: Well, I have to say, I was really impressed. So, bringing in new technology is never easy, but the team I was involved with at Nedbank, they took on ISPW, took on automation solution, took on XebiaLab’s XL Release. And they ran with it without really fearing the learning curve and they very much took it on. They started building their pipeline, experimenting, and the integration for them was very smooth.
And I think part of that is the fact that they broke down silos, they had the mainframe teams working very closely with the distributed teams, and together, they came to bring success. They’ve proven it in terms of, I’m sure you’ll hear later on, about the savings they made in the end-to-end from development to production lifecycle.
Craig: Stuart, in your global role and taking that model out to some of those larger customers around the world, when do you see end-to-end orchestration being a key thing for our customers?
Stuart: That’s an interesting one, Craig, because it makes so much sense to be able to deploy and release a package of code, especially when mainframe applications tend to be closely coupled.
But maybe right now, there’s so much other work to be doing, that the focus hasn’t zoomed out to having the release coordination orchestration, but it has to be a vital part of a DevOps solution.
Atul: If I can add a bit more to that, so, where we started at Nedbank was through a value stream mapping process and there were some very clear low-hanging fruit they wanted to address first of all, and part of that was the SCM, the source code management aspect of their infrastructure. And they’ve been working through all that was discovered in the value stream mapping exercise and then they progressed on to the actual orchestration becoming the constraint. So, that’s what they would address. So, it’s been a series of activities they have been going through and obviously they’ve done it very well.
Craig: Great stuff. So, we’re going to hear now from Jakes in the final section, just how we’re going to advance that more. We’re moving to test automation to complete their journey. And so, you touched on how testing is that next focus point for you, unit testing, functional testing. What are the next steps you’re going to be doing to accelerate that part of the process?
Jakes: Yeah, so, it’s definitely a round of engagements with the development teams and we’ve got our enterprise quality assurance team as well, they’ll sort of drive the functional testing. And I think there’s also a lot of other opportunities where some front-end testing has been automated, which we can also start leveraging off for sort of regression testing for back-end changes. So, yeah, that’s sort of the plan. Start engaging with these teams and pass on their knowledge so that they understand the benefits and understand how easy it is.
I mean, we know to create a debug session and to record that unit test is fairly simple. And yeah, so, we need to just understand that there is some effort involved in creating these tests, but as soon as we sort of paint the picture of the benefits you start reaping when you’ve got these assets and you can just re-use them in future, and that’s where the real benefit starts coming in, So, the longer you’re running with this, the more the benefit is going to grow.
Craig: Sure, and I’m hoping that once we come out of lockdown and we are able to see your journey and if people are still listening to us, Jakes, then we’d love to understand how your automated testing project goes along, as well. And Lynette, from your perspective, how can you take the next steps in this journey into other projects that you may be doing with other banks down in South Africa?
Lynette: Well, I think we’ve pretty much proven the way that we did it, having it as not a big bang approach but as a phased-in rollout. Making sure that the developers have the tools installed on their machine, that their security is set up, that their training is done very close to when we do their migration, so that they can move on to that immediately. It is a model that works.
There are definitely things that we have learned out of this project that we will then take forward. But yeah, all in all, I think it’s been a really, really good experience having the guys support us the way they do. Darryn or Jakes was always there. We had worked long, long hours. Darryn lives in Cape Town, so, there was a lot of times, and a lot of things that we did remotely with him, sitting on meetings. We worked until six-seven o’clock at night a lot of the times, but the guys were always there, and the management team was always complete supportive of what we were doing. Yeah, on a number of occasions, I hope you don’t open mind me saying this, Jakes, but Devi said to us, it’s like she cannot believe that this has just gone so smoothly. She’s never been on a project that’s actually really run so smoothly before, and she was really impressed, she really enjoyed that.
Craig: Always nice to hear. Yeah, so, I guess finally coming towards the last couple of questions guys. But a lot of the things that you spoke about here, Jakes, we look at DevOps as an industry phenomenon and you’ve touched a lot there on culture and a lot of tools have changed, a lot of processes have changed, but tell us about the change in culture within Netbank, and where it was say a year ago compared to now, having these processes in place. Tell us a bit about that.
Jakes: So, with the whole drive of the new ways of working, adopting Agile practices, just changing the organization from that happy to just support and maintain what we’ve got and just carry on. I mean, it’s lovely to have a stable system, and not have to worry about support and those sort of things, but that drive to digitize… we’ve had some great drives from our execs around the digital journey and the whole new ways of working around Agile and DevOps. You pick up that the teams are slowly but surely getting into that mindset of continuously reviewing, continuously looking for ways to improve, and ways to do things better. So, that’s exactly what we are trying to achieve with this journey. Let’s challenge each other. Let’s challenge ourselves. When things seem great, that doesn’t mean we just need to accept it and not worry about improving things even more. So, just getting into that mindset of that continuous push to just make things better and easier for ourselves is exactly what we wanted and we are seeing it.
Craig: Great stuff. So, what we’ve been doing for our listeners is talking about suggested reading; some of those industry materials that you’ve used along the way, but we’ve had quite a few of those and I think Jakes, knowing your background is more around gaming that it is around reading, I understand, tell us how you’ve managed to use gaming is a way of keeping sane in this lockdown period.
Jakes: Yes, so, I’m a big PlayStation fan, I’ve owned all the PlayStation consoles from one all the way through to PlayStation 4, currently. Very excited about PlayStation 5 coming soon. I love my racing games, I love my sport games and being able to jump on the Playstation, race against one of my best friends on a regular basis, making use of a chat room, just connecting with friends and having a chat. So, yeah, that’s really helped. You can only watch so much Netflix.
Craig: Yeah, good. And Lynette, rather than your reading material is there anything else that you guys do down there in South Africa as a hobby?
Lynette: Yeah, definitely drink a lot of wine. It keeps us sane.
Craig: It keeps you sane and keeps you happy, right?
Lynette: Yeah, absolutely. So, Jakes and they like to drink the beers. But yeah, what we did is we did definitely go out as a team a number of times during the migration, just to take a little bit of a load off and a little bit of a stress off, laugh about a couple of things that happened. We did strategize. So, definitely, drinking wine can help you to strategize and take the next steps. We work hard and we play hard, and definitely, drinking wine is one of those times.
Craig: Well, you’re in the best place in the world, arguably, to enjoy that facility. So, that’s fantastic. So, it sounds like a great story, guys, and I’m very happy to be able to share this success story with our listeners and we’d love to kind of come back to you once we can get out on those roads again and come down and see you guys, but… Jakes, Lynette, thank you very much for sharing some time with us. It’s been lovely hearing from you.
Lynette: Thank you.
Craig: Thanks, Craig.
So, there we have it. Jakes and Lynette, thank you so much for telling that story, and explaining that we should all be working in South Africa, what with red wine and games. It sounds like the perfect place to be, don’t you agree, guys?
Atul: Totally agree, and I’ve been there, I’ve done it, and I’ve got the T-shirt.
Craig: Excellent. So, Atul, as you’ve spent some time with them going through that process, just what are the key takeaways from this journey that they’ve been on, and how are they going to build on it for the future?
Atul: Okay, I think there’s a number of different points. The first one, for me, is that they’ve got a number of true passionate explorers within the organization. People, I think you mentioned it in the discussion, they know no fear, really. They make things look very easy and, I think, no path to success is linear, there’s always ups and downs, but having passionate explorers who are willing to take on a challenge without fearing failure is very, very important.
And I think the fact that they looked to make massive savings, and they actually exceeded those. So, something that was mentioned, for example, by removing the manual work of writing documentation, etc. for the change management, they managed to bring down that process from five hours to 18 minutes, and Jakes said, that is quite unbelievable when he mentioned that people. And as part of that, they also included things like the automated quality gats with SonarQube and they’re planning to take this even further by automating their functional testing in the coming months.
So, they keep on pushing. And I’ve heard them speak about Nirvana, but that Nirvana, that word has now been changed within the organization. It’s now called “Nedvana.” They are now setting the standard in South Africa of where other organizations need to be trying to reach.
Another point to add is Lynette, from her experience, she’s actually said, “We didn’t take a big bang approach,” they took it application after application, and they took each of the teams with them so that they could be trained just before the migration was done for the application. And I think that’s the right way to do it because that’s also part of the culture change. We’re not actually using the stick to force people to change. We’re bringing them along on a journey with us, as well.
And the question of a bottle of wine, it’s fantastic. But there’s a point, there, really, a serious point, and I think Lynette said it really well, they work hard, and they play hard. There’s got to be a fun element. If people aren’t motivated, then you won’t get the best out of them. So, again, amazing interview.
Craig: It’s great to get that feedback. And Stuart, I know you’re a bit of a gamer as well. I take it you’re looking forward to the PS5, as well.
Stuart: Absolutely, yeah. The COVID-19 lockdown of the world, I think people are turning to their gaming consoles. I certainly know broadband gets impacted when Call of Duty’s being used elsewhere in the house, but with the glorious weather we’ve been having in the UK, I do love reading a book in the conservatory. And being on the previous episode of the CSI podcast inspired me to actually open the DevOps Handbook again. And just picked a random chapter, as it happened, chapter five, and just reading through stuff there, it’s incredible that when you’ve been on a bit of a journey and then you touch these text areas that meant something a year or four years ago, and you read them again with the extra knowledge, the whole thing is different. So, whenever we recommend books sometimes it’s not just the new ones. The oldies are the goodies, as well.
Craig: Cool. Well, thank you guys. Pleasure to be with you. Let’s look forward to seeing you on another podcast, soon.