Security as a Journey | Let's Make Better Mistakes Tomorrow

In this episode, Kim Weins and Josh Bressers engage Stephen O’Grady, co-founder and principal analyst at RedMonk, on how improving the developer experience can pay dividends for security up and down the software supply chain. 

Transcript

 Announcer:
You're listening to Upstream: The Software Supply Chain Security Podcast. Brought to you by Anchore.

Kim Weins:
Hi there and welcome, I'm Kim Weins, host of Ustream. A podcast for those curious about the security of the software supply chain. In each episode, we talk with experts, practitioners, and thought leaders about concrete ideas and approaches to improve software supply chain security. This podcast is for everyone, both inside and outside the world of security. On this show, we highlight ways we can work together to protect the software that we all depend on. Josh Bressers, meeting that protection mission absolutely requires great communication.

Josh Bressers:
It absolutely does, Kim. I think communication between everyone in the security and the software industry, in general, is one of those things that just, you can't stress how important it is. There are an enormous number of failures that we could talk about, all because of communication. And so, I think it's enormously important and obviously knowing where we've been is so very important to innovating for where we're going in the future.

Kim Weins:
Well, Josh, we're lucky today to have an industry luminary and a great communicator with us on the podcast. Stephen O'Grady is fluent in the language and culture of developers. He's the Principal Analyst and co-founder of RedMonk, the developer-focused industry analyst firm. And the author of the book, The New Kingmakers: How Developers Conquered the World. And he joins us from the great state of Maine. Welcome to Upstream, Steve.

Stephen O'Grady:
My pleasure, thanks for having me on.

Kim Weins:
So Steve, I've heard that in Maine, sometimes people say you can't get there from here. Is that the right accent?

Stephen O'Grady:
Pretty close. Not being a native of the state I can't duplicate it, but that's as close as I can get, let's put it that way.

Kim Weins:
Okay, well then I don't feel completely embarrassed. So I recently heard you refer to the software supply chain as the soft underbelly of organizations. And I know that we agree that securing the supply chain is the destination, that's where we're going. So how ready are organizations to make the trip? And how do they get there from here?

Stephen O'Grady:
How ready are they? I'll give the standard analyst or consultant's answer, which is, it depends. It depends on the organization. What we have found, and the justification or the rationalization for the soft underbelly comment is that for many years the primary attack surface was the production systems once they get out to production. So I'm going to go attack anything front-facing, forward-facing, et cetera. And that obviously is still a focus of attack, but what we've seen in recent years are attackers get more creative and understand, all right, if I can infiltrate some of the build processes themselves, I can potentially do even more damage by spreading myself ... leveraging distribution mechanisms and so on to my benefit, as an attacker.

Stephen O'Grady:
So when we talk to organizations, I guess the short answer to the questions is that they're kind of all over the map. You have customers who are still stuck in that, I must protect the production system's mindset and everything else is secondary. And then you have customers that look at things like the SolarWinds, the high profile SolarWinds vulnerability, and just are ... they're up at night, every night, worrying about attacks of that nature. So I wish I could give you a better answer, but I think the answer is, it really depends because the first step is understanding that this is a problem before you even try and get around to try to attack it.

Josh Bressers:
I mean, I like that though. I think that makes a lot of sense because if you look at the history of the security universe, it's historically always been cat and mouse, where the defenders are chasing the attackers. And I think we've reached a point where the attackers have new attacks. And so now it's time for the defenders to adjust accordingly, right?

Stephen O'Grady:
That's exactly right.

Kim Weins:
And didn't cloud change this idea of just worry about the production systems, because we realize there's no perimeter anymore? The perimeter is gone.

Stephen O'Grady:
Yeah, the perimeter's gone. And I think one of the other pieces here is just that the nature of how we build applications today is just a lot different. In other words, I'm dating myself here, but when I was coming up, we had to build most of the stuff ourselves. Obviously, you pick up commercial frameworks wherever you can and rely on folks that have gone before you. But a lot of it was just, all right, we need to build X and so we'll go do that. And today, the good news is that there's this huge array of resources, both cloud, as Kim mentions, from an infrastructure standpoint.

Stephen O'Grady:
But on the software side as well, there's tons of project libraries, frameworks, et cetera, that can be leveraged. But this growing list of dependencies is, at once, something that will help you move more quickly and improve your development velocity. But also introduces this cascading list of new potential entry points for attackers. So, yeah, I think part of is certainly in the nature of how the infrastructure itself has changed, but a lot of it, as I said, is just the nature of the software that we're leveraging to build this stuff.

Josh Bressers:
All right, Steve. So I guess the next question, the thing I always hear when I'm discussing the amount of open source everyone is using is, I think one of the natural conclusions people will jump to is, well, I just need to use less. I'm curious on your thoughts for that particular question.

Stephen O'Grady:
Yeah. You do hear that from time to time. Less, I think, than we would've, say, five or 10 years ago. Because if you go back a little ways, there was a widespread perception in the industry that open source was inherently less secure than commercial alternatives. That was the security by obscurity axiom. And I think what we found is, is that there have been plenty of very high profile commercial vulnerabilities over time. And people are basically realizing at this point that, whether it's open source or not is not the issue. In some cases, you can make the argument that the rigor that is applied to some of the open source projects, coming as they do from the engineers, of which come from many different organizations and have to find a way to systematically build things together.

Stephen O'Grady:
That's often a more rigorous process than you'll see from some commercial organizations. So I think in some, if not many cases, the open source is probably more secure by design. But basically the net of this is that all software has vulnerabilities, whether it's open source or not. So when we hear that, and like I said, we hear that, I think, less today than we would've maybe 10 years ago. But when we hear that, part of what we're trying to point out is, by limiting your use of open source, you're basically cutting yourself off from a tremendous boom from a developmental standpoint for basically no gain from a security standpoint.

Kim Weins:
You're going to do the work yourself, but it's not going to be any more secure than the open source that you would've used.

Stephen O'Grady:
Yeah. And the analogy I used, this is actually a recent conversation, only a couple weeks ago I was talking to somebody about this and, Kim, you mentioned the cloud earlier. So there's analogy, I think, that's at work here which is, in your early days of cloud, there was this impression that cloud was somehow inherently less secure than enterprise infrastructure. We had to gently ask some enterprises, "Do you think you employ the kinds of people that work for the large clouds and work on this full time?" In some cases the answer might be yes, but in most cases the answer is no.

Stephen O'Grady:
So without getting into binary, unequivocal one is more secure than the other, I think it's very fair to say that you can have secure cloud, you can have secure private infrastructure, but you're never going to determine which is which by knowing if it's on or off-prem. And the same is true for open source. Is there insecure open source software? Absolutely. But is there insecure commercial software? You bet. So, yeah, you're never going to save yourself by lopping off one huge area of software.

Kim Weins:
So you mentioned the fact that we're developing software differently now than we used to back in the old days. And one of the things that you've talked about a lot is the frictionless developer experience. It's something you're pretty passionate about. You've written blogs about it, articles about it. Can you explain the concept and give some background on what improvements you think need to be made in this area?

Stephen O'Grady:
Sure. Yeah, so as you say, I've written quite extensively about this. If anybody wants to Google developer experience gap, you'll find a lot of it. But the short version of the argument is this, which is, as we've talked about already, we live in a world where developers have access to, essentially, infinite resources. An infinite number of resources. From cloud infrastructure to managed database services, to libraries, open source library. Every shape and size, frameworks, and on and on and on. So on the one hand, there's never been a better time to be a developer because you have this huge array of tools at your disposal. But the flip side of that is that trying to sift through and pick out which ones you want, then try to pick out how to integrate them, trying to operate them, secure them, manage them over time, is an enormous task.

Stephen O'Grady:
And so that gets to what we refer to as the gap. And what we're pushing for, and essentially to our delight, I guess is the simplest way to put it, the industry really seems to be taking this seriously and moving in this direction. Which is, just trying to think of, all right, if I'm building software, if I'm offering services, whatever my given role as a vendor might be, what is the experience of using this like? Not just, hey, I bring X capabilities to the table, what else gets used? What am I used on top of? What is used alongside of my software? What's used on top of my software? And what's the experience of using those together?

Stephen O'Grady:
Because in the original developer experience gap piece, the analogy that I used there was the iPhone. So in other words, on a given iPhone you have a phone app and you have a music app, among others. And when you're listening to music and you get a phone call, music fades out, you pick up your call. When you're done, you hang up, music fades back in. So basically, that's the ideal. Which is that we have these different pieces of software that have been ... We've thought about the experience of using together. But unfortunately, the industry right now is still very much in a pre-iPhone state. Where everyone has their given product service, API, whatever it might be, and how all that stuff gets pieced together, what the experience is like of using it together is left to the developer to figure out. Which is not ideal.

Josh Bressers:
I like this to analogy because I think, especially in the security world, there's this attitude of perfect or nothing. And I think we forget sometimes how truly new this industry is. I mean, software in general is very new. We're talking 50, 60 years max. But I think what we think of as modern security, I mean we're talking single digit years, which is ridiculously new.

Stephen O'Grady:
Right. For sure. And this is something I lament often, and this is probably not a surprise coming as a history major in college, is that the industry has very little sense of its own history. So, there's so many things where we're like, hey, we've seen this pattern before. We've seen this play out. I can tell you how this is going to end. And there's not a lot of history, as you point out, it's a very young industry. But in spite of that, there are many, many lessons which we seem doomed to keep learning, as [inaudible 00:12:16] said.

Josh Bressers:
Yes.

Kim Weins:
Well, I like to call these the daily annoyances. The things that, when you're using a piece of technology or software, that just drive you crazy every time. And it's not just about personal annoyance, it's also about development velocity. Because if a developer is annoyed because they're having to stop or slow down or switch context, they're not as productive in doing what they do best and doing what their companies want them to do well, which is to deliver new capabilities at a reasonable velocity. So how do APIs play into this? Because you mentioned this idea of every tool has its own API for its service. So talk about that a little bit.

Stephen O'Grady:
Flip side of the coin, in the sense that, it is wonderful that we live in a world where there are so many abilities, and functions available, that are just an API away. So from an application developer standpoint, it's very, very easy, typically, and lightweight, for you to go out and leverage these to do things that would just take you longer, that you would have to go figure out and build yourself. So that's the one side of the coin. But on the flip side, as I said before, I don't know if anybody here listening was a philosophy major, but there's [inaudible 00:13:44] as the categorical comparative. And I won't go into everything that's associated with that. But basically, the easiest way to explain that is that if one person litters, it's not a big deal. If everybody litters, that is a big deal.

Stephen O'Grady:
And that's a simple way to distill that down. And that's ultimately what we have from an API standpoint, which is, hey, you have one or two of these, great. I can do X, Y, or Z. But then, what happens when I have a dozen? Or two dozen? Or 100? Then it becomes much more difficult to manage. And I remember, many years ago when I was talking to a large European bank, and they were one of the first to go down the VMware path for infrastructure, and I had talked to them and said, "Hey, how's this process going?" And they said, "Well, it's going okay." And I said, "That doesn't sound that great." And they said, "No, no. Look, we're seeing the benefits that we wanted, but there's a big difference for us in terms of managing 500 physical machines versus 5,000 virtual ones."

Stephen O'Grady:
And that's what we're seeing in the API world right now. In the sense of, like I said, it's a wonderful thing that all these different services and functions are just an API away. But that process begins to ... you begin having difficulties in scaling that process, even in prosaic things like, okay, how do I select them? How do I pick one from another? How do I determine which of these is going to be the best options for me going forward? And then, you get into the logistical concerns in terms of, all right, how are they integrated? Who's managing what? Where do the responsibilities lie, support and otherwise. So yeah, it's very much a flip sides of the same coin type of situation.

Josh Bressers:
I want to turn this around on you, Steve, a little bit. And I love the message about scale, because I think that's something we often forget. Where it's easy to do something on my laptop, it's hard to do something a million times. Now, if we think about the supply chain, I think, we historically had this view of, I'm deploying one application. But once we start bringing all the software supply chain process into it all, now I realize I don't have one application, I have 7,000 open source dependencies I'm actually trying to manage. And that changes the conversation immensely because we go from one to thousands to potentially millions in some cases.

Stephen O'Grady:
Mm-hmm (affirmative), yeah, and that's part of it is that, what we're seeing are emerging questions in terms of, okay, open source is tangibly, provably delivering the results in terms of how quickly I'm able to move. Which is, at least from all the conversations we have, almost the only thing that enterprises care about at this point is just raw velocity. So it's great, but they are increasingly facing questions in terms of, okay, well, who's responsible for what? And if there's a problem with this, what's the mitigation here? What's the process and so on. So, there's a lot of different ways that organizations are looking at that. They're looking at, all right, who are our new commercial suppliers?

Stephen O'Grady:
In many cases, they're punting entirely and saying, "All right, I'm going to go with a managed service and make that some other vendor's responsibility." So there's no single way to address this, but it is absolutely a growing concern for enterprises. And frankly, appropriately so. Because, yeah, it's a wonderful thing to have all this capability at your fingertips, but then, determining ... It's the old line from Ghostbusters, who you going to call when something goes south? You have a log for shell and, oh my God, okay, what am I doing now? And who's responsible for this?

Kim Weins:
Are you offering up your phone number, Steve?

Stephen O'Grady:
No, no, certainly not. No. My basement flooded last weekend and I had that exact same process where it's like, who do I call for this? Who's responsible? I have full sympathy for the enterprises going through this, but it is very much a ... it's a process of trying to determine cost benefit. What can I use and how do I manage, responsibly, the software that we're using and make sure that we have a process for mitigation and remediation if necessary.

Kim Weins:
You're listening to Upstream: The Software Supply Chain Security Podcast. We're speaking with Steve O'Grady, Principal Analyst and co-founder of RedMonk. So Steve, we've heard your colleague, Kelly Ann Fitzpatrick, refer to software security as a journey. So do you subscribe to that line of thinking? And what do you think is around the corner on that journey?

Stephen O'Grady:
I absolutely think it's a journey. I think the only qualification I'll throw in there is that I don't think it's a journey with a destination. I don't think you ever get there. I don't think you ever get to a point where it's like, all right, we're done. We've arrived. Everything is secure now. So that's the only part of the analogy that I would quibble with in a minor way. But basically this is one of the phrases we use at RedMonk a lot is, let's make better mistakes tomorrow.

Kim Weins:
I like that. I like that.

Josh Bressers:
[crosstalk 00:19:01].

Stephen O'Grady:
That is, I think, one of the best ways ... It's from an old friend of mine, who unfortunately passed away, [Alex King 00:19:06] used to say that all the time. That, to me, I think encapsulates what organizations should be trying to think about as they're on this journey. Because, as we talked about, you're never going to get there. And what you want is the ability to make, okay, I'm a little bit better than I was yesterday. I'm a little bit better than I was the day before that. And it is very much incremental progress, but it is ... Josh used the cat and mouse analogy before, that's what it is. It's a game, the game is never going to be stopped playing, so you need to essentially be on your game, as it were, and make sure that you are trying to keep up with folks trying new and creative attacks.

Stephen O'Grady:
As far as what's coming around the corner, the biggest thing I think, for me, that organizations are trying to figure out is actually what we just talked about. Is trying to figure out, all right, for the entirety of my software portfolio, who's responsible for what? If I have an issue with some commercially backed software, great, I call them. If it's some other dependency that I'm managing, what's my process? What does that look like? And how does this change, or not, my ability to leverage certain pieces. So really what I see as the, I don't know, the critical part of the journey that a lot of organizations are going through right now is just that. Is realizing, oh wow, we're using a lot of this. It's offering us these, again, tangible, provable benefits. Which is great. But on the cost side of the ledger, how do I balance that? How do I figure out who's responsible for what, ultimately.

Kim Weins:
One good piece of news, I think, is in talking to a lot of customers and prospects in the last few months, in the post Log4j world, how many people have showed up and said, "Hey, we didn't have a huge impact from Log4j or we were able to reasonably mitigate it, but it was a wake up call for us. That we need to do something different or do something better. We're wanting to be more proactive. We're wanting to get ahead of it." So I think that's a positive development that people are taking these ... They don't have to wait till everything blows up around them, they're taking this as an opportunity to, what did you say? Make better mistakes tomorrow.

Stephen O'Grady:
Make better mistakes tomorrow, that's right. Yeah I think we're having some of those same conversations. We're basically, I'm trying to think of who it was. Anyhow, it was a large enterprise basically said, "This is our second chance. We didn't get hit this time but this is a wake up call and we need to change our behaviors. Otherwise it's basically inevitable that one of these things is going to catch us." And I mean, frankly, it may be inevitable for a lot of organizations, even if they are on their game and doing what they need to do. Because software being software, it's going to have vulnerabilities in it. But the trick is basically to try to mitigate as best you can.

Stephen O'Grady:
So, yeah, let's just say it's interesting to talk to organizations about this now in terms of how they look at it. And every single one of these is a life changing moment for some organization. Or a timeline changing moment. Events or corporations aren't people so I shouldn't say life changing, but anyway, it changes the way they do things in big ways, because we've already talked about a couple of them, and each one of these that comes along turns some other subset of enterprises into believers. That, okay, whatever this new, big vulnerability is, we got lucky. We might not get lucky another time and we need to change our behavior accordingly.

Kim Weins:
So as software security is becoming front and center, and software supply chain security is raising in focus and awareness for companies, the lines between security teams and technical teams have become more blurry. And maybe that's good. Maybe we're talking collaboration here. But what is the role of the software security practitioner and how has that shifted, based on where we're at in these new development approaches?

Stephen O'Grady:
I think what we've found in the macro sense over the last, I don't know, I'd have to go back and look to try to date it precisely, but certainly over the last number of years is that the ... We saw this with DevOps and we're seeing this in many other categories as well. Basically the more that we have these artificial bright lines between different areas of an org. Whether that's dev and ops, dev and security, on and on dev and the data side of the house. The more difficult it is to communicate, the more difficult it is to operate at velocity and operate in a way that's sufficient. So certainly what we've seen in recent years in the security space, specifically, is softening of that bright line.

Stephen O'Grady:
So in other words, the common nomenclature is shift left. So in other words, rather than ... This was certainly the case when I was a developer, you went through, you built something, and then after the fact, security team came in and told you how you did everything wrong. And so it was like, this would have been great to know weeks ago, before we're coming up on a deadline. So we're beginning to see organizations realize that the earlier they can introduce that in the process, the better. The challenge is, is that ... My colleague, Rachel, had a great piece on this a little while ago. The challenges is, is that what you see in some organizations is enterprises try to then put all of the burdens on developers. And it's like, yeah, they have a hard enough job already.

Stephen O'Grady:
So there's always a balance there in terms of, we need to shift left, but we need to shift left in a way that is not essentially going to crush developers under some huge weight of new responsibilities. Which means using tools, using every resource at your disposal to simplify things, keep them manageable, and make a developer's life easier. And that's essentially the whole point. And to the extent that organizations are practicing that, it makes sense. And that's what they should be doing. Much as they have erased some of the bright lines between dev and ops, we'll continue to see that. Like I said, the difficulties is that in some orgs you definitely see a ... okay, there's this huge weight here. I'm going to take it off security's plate and put that on developers. And that does not tend to be a terribly productive approach, in my experience.

Josh Bressers:
Indeed. Indeed. Yes, that is one of the, I think, interesting pieces that when we talk about shift left, there's, shift left doesn't just mean keep dumping more in developers. And I think sometimes we forget that in all the hype and excitement.

Stephen O'Grady:
I think a lot of people forget that, let's put it that way.

Kim Weins:
Well, and I think we can help with tooling, we can help with automation, making that kind of frictionless, making the usability of the process itself. What are all the steps across all the tools that you need? Those can all help to try to reduce the burden. I will say, in talking to a number of customers, they will always say, as one of their first requirements when looking at security tools on the shift left side, so to speak is, I don't want to get in the way of developers. I don't want to slow developers down too much. I want to make it as easy as possible. And so we hear that from security teams and we hear that from the DevOps teams that are often integrating these platforms as well. So I think people are on the same page about that. Whether we're all the way there in terms of the right tooling, the right automation, the right integration, the right processes, maybe there's always more work to be done.

Stephen O'Grady:
Make better mistakes tomorrow.

Kim Weins:
Better mistakes tomorrow. So Steve O'Grady, thank you for joining us on Upstream. I'm going to try to use better mistakes tomorrow with my kids as well.

Stephen O'Grady:
We definitely use that here as well, yeah.

Kim Weins:
And thanks to everyone listening. You've come to the right place for great discussions of the big issues impacting the security of the software supply chain. Subscribe and never miss an episode. Upstream is available everywhere you get your podcasts. If you've learned something new, be sure to share this episode with friends and colleagues and thanks for joining us on Upstream, brought to you by Anchore.