Log4j “Day Two”

On this inaugural episode of the show, veteran security leader and world-famous podcaster: Josh Bressers joins host Kim Weins to discuss the log4j security vulnerability and the way forward in preparation for the next zero-day attack.

Transcript

Announcer:

You're listening to Upstream, the software supply chain security podcast, brought to you by Anchore.

Kim Weins:

Hi there and welcome. I'm Kim Weins, host of Upstream, a podcast for those curious about the security of the software supply chain. In each episode, we talk with experts, practitioners, and thought leaders about concrete ideas and approaches to improve software supply chain security. This podcast is for everyone, both inside and outside the world of security. On this show, we highlight ways we can work together to protect the software that we all depend on. It's January 18th, 2022 and we're just over a month out from the announcement of the Log4j security vulnerability, the big scary zero-day. Let's start by getting answers to a few fundamental questions like what did we learn and what do we all need to do now going forward to better prepare for the next zero day? To help tackle those questions and many more to come, I'm joined by veteran security leader and world famous podcaster, Josh Bressers. So great to be podcasting Upstream with you.

Josh Bressers:

Awesome. Thank you so much, Kim. It is a treat to be here.

Kim Weins:

So Josh, Log4j was in a category of security holes called a zero day vulnerability. What does that even mean?

Josh Bressers:

Zero day vulnerability is something that no one knew about and was being exploited in the wild by attackers. And these are kind of the scariest, because what usually happens with vulnerabilities is a bug is found, it gets reported to the project. In this case, it would be Log4j, the Apache Software Foundation, and then they do their fix and then they release a fix and then we all have some time to update and then everything is relatively smooth. But in this particular instance, we found out the attacks were actually happening prior to the patch coming out.

Kim Weins:

This is really the worst situation because you're scrambling to try to fix the problem at the same time people are potentially attacking you.

Josh Bressers:

Exactly. Yep. That's exactly right. And that makes it, it's already hard enough to deal with an issue of this magnitude. And then when you add in the outside attackers happening at the same time, it becomes extremely difficult for anyone involved.

Kim Weins:

So this isn't the first zero day vulnerability, nor will it be the last. But what did we learn from Log4j that we didn't already know from previous zero days?

Josh Bressers:

So, I think this one was really interesting because of how pervasive Log4j is in the Java ecosystem. It's pretty safe to say if you're running a Java application, it probably contains Log4j somewhere inside of it. And so this created an interesting challenge from the context of you have to find out which of your Java applications have vulnerable versions of Log4j, because there are many, many versions and not all were vulnerable, some were vulnerable in different ways. And then obviously once you know you have a vulnerable version, you also have to fix it. But we exist in this time now where there's these tools, for example, that create SBOMs and you can use these tools to inspect your software. In the past, this would've been a completely manual process. But this particular instance, we had the ability to utilize some tooling to assist with it.

Josh Bressers:

Now that doesn't mean everyone did nor that the tooling worked correctly in every particular case. But I think this one was unique in that the tools exist. I think it's probably safe to say everyone knows they exist now. And I feel like the difference in this particular zero day versus many of the others is we've kind of started opening up that Pandora's box of what do we do with these tools? How do we make sure we can take the process and the tooling we've put together and then make sure we can use it from this point forward, which is fantastic.

Kim Weins:

So SBOM, that stands for software bill of materials. So essentially it's all the ingredients in a software application, all the bits and parts that went into that. This is interesting because you and I were on a webinar recently and we did a poll of all of the participants in the webinar and we asked them what were the challenges their organization faced with Log4j? And the number one challenge, which was selected by 86% of the people on the webinar, was that it was difficult and time consuming to identify all of the software that has Log4j. So can you kind of describe what that process is like and why it's so hard?

Josh Bressers:

Definitely. So Java is kind of a unique beast in this space. When we think of a lot of applications using dependencies, which is quite often open source libraries, what happens is you kind of have some sort of package manager we'll say, and you install the dependency and it's just sitting on the desk. Now in the case of Java, Java has been around for a long time. So there's many different ways to package Java applications. And so what we see is you could have the Log4j, they're called JARs, Java archives. The Log4j JAR could just be sitting in a directory in the application. The Log4j JAR could be in another JAR that's part of some other Java library you're depending on. The Log4j JAR could be in part of one of the other types of Java archives that exists.

Josh Bressers:

There's like WAR files and PAR files and a multitude of Java archives. And so the challenge became, where is this thing in my infrastructure? And some scanners could kind of dive into JAR files and zip files and PAR files and all these other archives, some of the scanners couldn't. And so one of the challenges we had as practitioners is we had to decide, okay, I'm going to scan something and how much manual effort am I going to put behind it? Because it wasn't always understood if the scanner you were using was getting full coverage on what your application had. So that's kind of what the biggest challenge was, is we had to make some decisions about what's a critical application? What are we going to really look hard inside of? What are we going to say, the scanner findings are good enough? And I mean, this is where every organization had a different problem. And so everyone was solving a unique issue, which is always very hard.

Kim Weins:

So that's kind of like the Russian nesting dolls where you got one inside of another, inside of another and you don't really know how deep that Log4j piece of software is. And so you don't really know if you're finding everything, if you don't go all the way down to the lowest level, the inside doll in that nest, so to speak.

Josh Bressers:

Absolutely. And then you have to imagine, take that problem and multiply it by tens or hundreds or thousands of times across your infrastructure. And that's kind of the issue we were dealing with is none of us have infinite resources, but I suspect when looking for Log4j, it sure felt like there were infinite possibilities.

Kim Weins:

It was a bit of a scramble. I know I was at a CISO Summit in December, the week after Log4j. And I asked the people that were attending the panel that I was hosting, how many of you were working on Log4j this weekend? And everybody raised their hands. So did some people have a really bad holiday?

Josh Bressers:

I think many people had a terrible holiday. This is kind of what happens in the world of security. This is something I've talked about in the past at length is I wake up in the morning and I don't necessarily know what I'm going to be doing for the rest of the day. Because something like Log4j can appear and everything I thought I was going to work on has now been sidelined and this is what we're going to focus on. And so, I mean, I wouldn't say everyone working over the holiday and on the weekend and they know why they're doing it. They know what's at stake.

Kim Weins:

Why does this matter so much? Why do we have to get better at this?

Josh Bressers:

That's such a simple question and I think the answer is incredibly difficult. The entire world we have now relies on an enormous amount of digital infrastructure like we've never seen before. And possibly that's never just happened in the history of humanity, ever. And all of this digital infrastructure, it has our money, it has our family photos, it has our lives, it has the entertainment we consume. It's all a part of this ecosystem now. And if we don't keep it secure, you end up in situations where bad things can happen. And bad things aren't necessarily bad people stealing or taking your money or deleting your pictures or whatever. It can also just be when Netflix is down or inconvenienced. If the power goes out, we don't like it. So just kind of every aspect of our life is dictated by this. 

Josh Bressers:

And so when there's something like Log4j or any vulnerability for that matter, that can be exploited or used in negative way, we have these situations where the attackers can disrupt our lives. And that means many, many things. But I think that's kind of fundamentally what it is, is so much of modern society is reliant on this technology that keeping this technology running is extremely important.

Kim Weins:

Well, most people heard about that example last year, which wasn't a zero day vulnerability, but the Colonial Pipeline ransomware.

Josh Bressers:

Yeah, that's right.

Kim Weins:

Where they shut down that one pipeline distributor in the southeast of the United States. And then based on that, the gas wasn't getting to all the gas stations and people were waiting in long lines to get gas and couldn't go where they needed to go. It was a little bit crazy. But if you take that further, you're like this could get really, really bad if this happened in a different or broader way.

Josh Bressers:

Exactly that. And so this is why focusing on the security of our infrastructure and just fundamentally everything, I think is really important. And I also think though that everyone is starting to notice. As security people, I feel like we've been proclaiming bad things are going to happen if you don't listen to us for a very long time. And they kind of didn't for a long time, but now they are. And so we've reached the point where I think it's time for security to start solving these problems and making things better. Because the world literally depends on it.

Kim Weins:

So you just wrote an article for The New Stack, which is an online publication. And it was about day two for Log4j. So what people need to do after they remediate. So you do the immediate fix, stop the blood from flowing, but what do you need to do for day two? And what do you mean by day two?

Josh Bressers:

In the world of DevOps, there's this concept we call day zero, day one and day two. Day zero is when you architect and build a solution, right? This is the development of what you're doing. Day one is when you flip the switch and you go live. That's always the most exciting day for anyone who's working on a product. But then after we turn it on, there's this thing we call day two. And day two is kind of the rest of time where we have this new product, we have this new service, but now we have to maintain it kind of forever, in theory. Like I know there's end of lifes and things like that, but let's just assume it has to be maintained from this point forward for the foreseeable future. And when we think about a problem like Log4j, okay, so we spent all this time looking for it.

Josh Bressers:

We spent all this time fixing it. Now what? As much as I'd love to say, everything goes back to normal and we're fine, that's obviously not the case. Because there are going to be more issues. We're going to have to understand our infrastructure. There's all this work we need to do. And there's all this work we did that if we don't kind of continue it, I think it's a waste. And so the idea behind day two in the context of Log4j is we've got all these SBOMs. We scanned all our infrastructure. Let's keep doing it. Let's make sure we're actually doing supply chain management. Let's make sure we're doing SBOM management and we're understanding what's in our infrastructure.

Josh Bressers:

Not only when there's an emergency, but all the time. Because if you have proper SBOM management, and a problem like Log4j appears, we go from having to scramble and look for it to just saying, okay, I'll go in my management tool and type Log4j in the search box and it'll tell me where it is. And so we're at this point where we had an emergency and now we have the ability to acquire some resources to actually solve this problem moving forward. And I think if any competent leader at this point, it would be a disservice to yourself, your employees and your company to not do something to make sure that this doesn't happen again.

Kim Weins:

Well, I think that's a great place to wrap it up, Josh. So thanks for talking about day two for Log4j. To everyone listening, this is our first episode of Upstream. You've come to the right place for discussion of the big issues impacting the security of the software supply chain. Never miss an episode by subscribing. Upstream is available everywhere you get your podcasts. And if you learned something today, be sure to share this episode with your friends and colleagues. Thanks for joining us on Upstream, the podcast focused on software supply chain security. Brought to you by Anchore.