Steve Lasker of Microsoft joins the show and talks with host Kim Weins and Josh Bressers about how the software ecosystem will generate and use SBOMs. He reveals the challenge of giant SBOMs and how Microsoft is providing transparency to customers about the components in their software.
Steve Lasker of Microsoft joins the show and talks with host Kim Weins and Josh Bressers about how the software ecosystem will generate and use SBOMs. He reveals the challenge of giant SBOMs and how Microsoft is providing transparency to customers about the components in their software.
Announcer:
You're listening to Upstream, the software supply chain security podcast. Brought to you by Anchore.
Kim Weins:
Hi there and welcome. I'm Kim Veins, host of Upstream, a podcast for those curious about the security of the software supply chain. In each episode we talk with experts, practitioners and thought leaders about concrete ideas and approaches to improve software supply chain security. This podcast is for everyone, both inside and outside the world of security. On this show, we highlight ways we can work together to protect the software that we all depend on. Well Valentine's Day 2022 is in the rear view mirror now and so is the Super Bowl, I guess we're safe to get back to the pressing work of securing the software supply chain, Josh.
Josh Bressers:
That assumes we ever stopped, right?
Kim Weins:
That is true. We never sleep at all.
Josh Bressers:
That's right, it never ends. No, it's good. I think, it amuses me that the Super Bowl had a QR code ad and everyone lost their minds over that for a little while. But no, I think fundamentally, it's been an amazing couple of weeks for supply chain. I think the single biggest thing I saw was, the US Senate had a hearing about log for Jay and I found the most fascinating part of this is, back when Harpley happened, there was very much this attitude of, "oh my goodness, open source, what's this stuff, where'd it come from?" Versus now the message was very much, "Open source is here, how do we work with it?" And it was absolutely amazing, I loved it.
Kim Weins:
We've moved past blaming open source software, that's amazing.
Josh Bressers:
That's right.
Kim Weins:
All right. So, our guest on the show today is a supply chain security leader who's committed to quality, Steve Lasker, principle program manager architect at Microsoft. Thanks for joining us on Upstream.
Steve Lasker:
Well, thanks for having me, Kim. It's great to talk with you and Josh about this.
Kim Weins:
Steve, you caught our eye when you wrote an article last month called roles and responsibilities of signing SBOMs and security scanners. And you spent a lot of the article talking about tacos, which I'll allow people to read about for themselves but the end of the article, you had a really great analogy and you described SBOMs and security scanners as PB and J. So, help us understand why SBOMs and security scanners go together.
Steve Lasker:
Great question and it's a great topic. And I think there was some [asbestos 00:02:19] wrapped in there as well, with the tacos. I tend to look at how other ecosystems that are tried and proven have worked out things, occasionally connected stuff. Do you run into the store every time you want a jar of milk? We have refrigerators. Secure supply chains. There's a lot of these patterns that I think we just need to look at other systems and how they've managed them. I think the article is there's been a lot of talk around SBOMs and security scan results. should those be converged? Are those something that are converged or is it a product might generate a security scan result at an SBOM? The analogy I was trying to tease out was, SBOMs as being facts about information that you've done, at the build, it's the build of materials.
Steve Lasker:
When you build the building, the analogy there back to the asbestos is in the 1980s, asbestos was required in theaters. It was a good thing, right? It was actually part of your build of materials. Now we realize it's great for fires, not so great for keeping us from not getting cancer. That fact of a theater having asbestos in it turns out to be good information to have in the bill of materials to then later make a decision over time as we learn new things that actually turns out to be a bad thing.
Steve Lasker:
I look asbetos of being facts about the information. Hopefully we have more facts that might... You might update your SBOM. If you realize you didn't include an important fact about the compiler flags, let's say. Then with that fact, now we can start doing better security scans because we tend to do security scans by looking at something after the fact and doing like DNA analysis of what actually is in that piece of software. There's a little bit of both. It's trying to separate the facts versus time-based evolving knowledge and then the other part was teasing out the identity of who are these people.
Josh Bressers:
I love that. I will say Steve, that I have a certain amount of personal chuckling at your story. I bought a very old house that was actually full of asbestos, and we didn't know it for sure when I bought it, I suspected it. But then we sent in some experts and they were like, "Yep, it's everywhere." And so all the pipes in the basement had to be remediated and it was one of those issues of I just thought, "Man, if I had like an SBOM of my house, I would've known that right away."
Kim Weins:
The interesting thing I think you're saying is the SBOM doesn't change, right? Unless you change the contents of the software, right? If the SBO, your bill of materials, you use the analogy of ingredients in food. And once you've produced that doesn't change. Now, if you make a new version of that, so if you updated or you add new code or you make changes, right? Then you need a new SBOM. But for any given piece of software or release of software, the SBOM should be fixed over time.
Steve Lasker:
Yeah. That's the premise, right? Of course, if you take a new build, you're obviously rebuilding because something changed.
Kim Weins:
Right.
Steve Lasker:
It plays the later version of the same package you took before or, that's important because, well, what is interesting about that new package is newer because it had something changed. There's, that chain of dependencies, if you will, and it's important to know what those are and back to the taco analogy, what the example was there was, I think I was talking about like having a party and vegans and non vegans, coming and you're making.... You're all focused on having the right ingredients, which you did. What you didn't know is one of the ingredients turned out to be contaminated. I think it was the greens or something like that. There was, it's still fact that those greens were included from a certain manufacturer, but later, as people started getting sick, it was determined that there was something wrong with that.
Steve Lasker:
Later it was determined that log for Jay actually had some problems. But because I knew of all of the things on my shelf that had those greens or all the products that had log for Jay in it, I was able to go back and cherry pick the ones that are problem and not have to re-scan everything. That's a big piece of it. It basically, it just, I think it makes our supply chain content more secure and more efficient because without SBOMs where DNA analysis takes time, right? Like scanning something, the ecosystem has evolved that we're automating builds a software every seconds, right? It's just as fast as somebody could commit something machines are spinning. If we don't have efficient systems to be able to understand what went in those and be able to diagnose it, we'll never catch these things. Most of them are human errors, right? I think of these in three categories, stupid humans, we make mistakes. Then there's bad actors, government backed bad actors that are a new weapon of war that we need to be able to mitigate for. Anything we can do to be more efficient is pretty important.
Kim Weins:
You work on the Microsoft container registry. Can you, first of all, explain what that is and what are the security considerations you have?
Steve Lasker:
Microsoft container registry, and we try not to call it container registry anymore, but that is term. It's the way we distribute the Microsoft software. It's our newest, it's been several years now. But if you think about the Microsoft download center or other ways we've distributed software, the cloud native to air quote way is to distribute things through registries. Years ago, I was driving an effort around making registries, which is a great design for having this layered way of being able to persist lots of blobs and a manifest and lots of details. Turns out it's good for things, not just container images.
Steve Lasker:
We store helm charts in there. We store arm bicep modules in there where we continue to expand the types that we can distribute in that. The MCR is our Microsoft branded products that we distribute to the world. Customers might go to Docker hub and get things. We realized several years ago that Microsoft is not just a cloud, but a software vendor. We distribute our software to our customers regardless of where they're running, including our competitor's clouds. We work with Docker to make sure that content is actually served by Microsoft. And then you can browse it on Docker hub and so forth. MCR is our Microsoft branded version of the Azure container registry, which is how we enable our customers running on Azure.
Kim Weins:
How are you thinking about security considerations? And SBOMs as part of that?
Steve Lasker:
MCR is distributed globally. It's across all regions of the world in each continent so we can make sure we have reliable coverage. It's a cure. We have Ingres into the China clouds as well as our air gap clouds for our customers. That infrastructure itself is highly secure, highly reliable. We make sure that we can protect the content that goes in. But then as we talked about, customers want to have more insight into that. We've been working on an effort where we are going to sign everything. Well, let me back up. Part of that article and a different article I wrote was separating identity from location. If you're pulling stuff from MCR directly, there's a secure, endpoint there, you can trust that's that endpoint and that's great. That's not the best practice. You don't want to be pulling from a place across the internet that you don't control in your production environment.
Steve Lasker:
Whether you're taking a Microsoft container image, a Debbie in their Alpine image, that should be in your registry. A big piece of this that started years ago was the whole concept of signing was we wanted to make sure that we could sign the content, not just as a double check that you pulled this from MCR and it's signed by Microsoft. But as you promote that content into your registry, and then you promote it into another registry and another one and another one, and you don't really need to know the path it took. I want to be able to pick up a thing and know who created this thing. So that signature needed to travel with the artifact container image, helm chart, bicep module, whatever. That's a fundamental piece that where we started the notary effort was we needed to fix the ability to signatures can travel with the artifacts wherever they go, because we want to follow those best practices.
Steve Lasker:
It turns out that SBOMs can also follow the same pattern we did for detached signatures. Then we could sign the SBOMs because I could have an SBOM that says everything in here is great, but who generated that SBOM did Josh just injected himself because he messed with the package and nobody knows it. It was critically important that not only the containers, but the SBOMs that we ship are signed as well. We've been working on that effort. Microsoft's a big company. We have a lot of software. We have a couple of hundred different teams that are shipping content in, through MCR, through this promotion workflow. We've been working with all of them to sign their content, generate SBOMs for their content, sign, the SBOMs. We will be distributing that on MCR for our Microsoft software. It is this little EO thing that came out has a little impact on all of us as well.
Kim Weins:
Basically you're giving them an ingredients label. Here's what's in it. Then you're putting like the stamp or the seal from Microsoft that said-
Steve Lasker:
The hologram.
Kim Weins:
I don't know what it is. Like I was thinking like the signet ring of the king or something in wax that basically says that this software came from Microsoft and this SBOM that we created for that software also came from Microsoft.
Steve Lasker:
Exactly and that's just the start, right? I mean, there's one of the things that has come up is like these SBOMs, especially if you start looking at some of the SBOMs or things like windows, windows SBOMs are a couple hundred Meg, like we're wrestling with this concept because if you can imagine all the components that are in that or office or other products that we, distribute as a company. We need to be able to understand not just the size and how do you store it because that itself becomes a problem with the frequency of how much we generate windows builds and SBOMs, we've been discussing like petabits storage and how we're going to handle this. We don't need all of it, right? There's a lot of builds that happen of windows during a day that just disappear.
Steve Lasker:
But the problem is that same performance thing, just I want SBOMs to be smart so I can do good scans. I want to be able to do querying over those SBOMs to understand what's in those, do they meet a certain policy? We've been doing a bunch of effort around and it's great to see some of the open source efforts going on this as well. This is always the thing of good ideas are being implemented in multiple places and how do you reconcile them? But we need to be able to say, here's a policy for what must be in or out of software. In many cases, this is coming from the EO orders, there's a policy. Then how does different software apply to that policy? Does it meet it or does it not meet it? We've not only been storing the SBOMs and generating the SBOMs, but working on tooling and languages, to be able to evaluate those SBOMs to declare if they meet given policies, those will be government policies, but it could be the Kim and Josh policy for what you want to run in your environment, too.
Josh Bressers:
I guess I'm looking for a clarifying statement from you, Steve. You're talking about these windows, SBOMs being just massive and this intrigues me. Why are they so big?
Steve Lasker:
I've been trying to understand some of that myself to be fair. Because part of me is I just want to be a plumber and don't really care about some of this stuff. It's like go for it. But we have been having those conversations and it's basically windows has drivers. There's components of windows that go back decades. They're trying to make sure that for everything that is included, what does that include and what is the SBOM for that? It's literally the stack of dependencies that are in there. It's the list of files. If a file gets replaced, which of course every time windows update runs or every time you run an installer on your machine, there is a change. Did any of those changes include something that could actually be a vulnerability? Windows has a lot of files. You could imagine all of the change tracking that's being done to make sure of that. It's aware of what was added, what was changed and what was the state of expectation.
Kim Weins:
You're listening to upstream the software supply chain security podcast brought to you by Anchore. We're speaking with Steve Lasker, principal program manager architect at Microsoft. Steve, now that you're starting to produce these SBOMs, which can be very large, you're signing them. You're going to be delivering these to customers. How do you envision your customers using them?
Steve Lasker:
There's two parts, right? There's how do customers use the SBOMs that we produce for our software? Then how do customers follow that same practice so that they can do it for their software? One of the things that we want to make sure is that this is not a Windows world. In fact, I try not to necessarily start with Windows at times because it tends to paint a tainted picture. Microsoft was a company that Windows was our main product. We would build all kinds of apps that run on Windows. Now Windows is one of the platforms we build, right? We have Linux distros that we use. We have apps that run on Android and iPhone and all kinds of different Macs and so forth. It's really important that customers can bring those SBOMs to where they need them. That's the promotion scenario.
Steve Lasker:
They need to have tools that can validate they came from Microsoft. And then they need to have tools that can do a quick check. Does this meet a standard policy that's out there? A government customer will have a set of government policies for different parts of the government in different environments. Does this particular software, not just this particular product, but this very specific build match a policy, yes or no? Those will be tools that we have to enable, not just by querying a set of services on MCR that we will serve to do that querying capability, but being able to run over those SBOMs locally as well. So they can validate those. That's the scope and problem of things that we have to think about all the cross platforms, all of our software and these massive SBOMs.
Kim Weins:
The giant SBOM problem. Now I'm scared, now I won't be sleeping at night.
Josh Bressers:
I love it.
Kim Weins:
Okay, so they're going to run a policy against it. You've mentioned EO, that stands for Executive Order. The US executive order on the nation's cyber security, I think is the full name. This was the order that came out in May of 2021 in which the government is laying out in order for the government to use software, the vendor or provider is going to have to give them certain things. One of which is an SBOM, a software bill of materials. You mentioned policies. What you are saying is the SBOM goes to that customer could be government or could be somebody else that customer could then, run some policies against it to see if it meets that. I'm assuming like a part of that is vulnerabilities.
Kim Weins:
What vulnerabilities exist within this software. They're not just dependent on you telling them vulnerabilities, but they can check for themselves and more importantly, even if you told them here's the vulnerabilities and the software at the time we shipped it, new vulnerabilities arise, post shipment or post deployment. And they can double check and do their own check at any point in time. Is that what you're envisioning here?
Steve Lasker:
Yeah, exactly. We got focused on the SBOM conversation, but we also, and we started the talk. We were talking about the time based information and those scan results should come from others than just us, right? You always want that two-factor check of sure we are scanning it. We will post updated scan results on our software and we'll make those available through MCR as well so that you'll be able to get that, but you absolutely should be running your own security scanning to make sure yourself look at solo winds, right? Like some of these, it wasn't the binary per se, that was in it. It was a change in behavior. I mean, this is where you get, where you have governmental actors that are smart enough that are doing evil, right? Cause we're very creative. You tap that creativity for the wrong place it gets scary.
Steve Lasker:
Where they knew like, "Hey, test case that run only run for a couple of seconds. As long as they don't see anything, it's fine." Sit dormant for 10 minutes, 10 hours, whatever it was. Start doing bad things. You're not going to catch that in a quick scan. You're going to catch that by doing run time scanning and see behavioral changes. Those are the things that you want to be able to go back and compare and then when that pattern exists, now I can start applying that to the software I already have. That window that's discovered and closed obviously is an important window for exploiters to try to tap into it and those try to close it down, react to it. It's a very automated needing to be automated problem that we want to enable that information to be efficient, to be able to capture and analyze.
Josh Bressers:
I think one other piece of this, Steve is as a consumer of software, I might get a piece from you, let's say Windows in this case, because that's the topic. And then I'm probably going to add something on top of that. Now I have your SBOM, I have my stuff. And then there's a vulnerability scan of, I guess the system as a whole right
Steve Lasker:
There is. And if you think about when bad things happen, it's a combination, most errors, nuclear power plants that have failed. Those are cascade of a combination of things. Absolutely being able to not just see the information in that one product, but being able to see the correlation because by itself, this port was intended to be open. It was by design this software, "Ooh, that port's convenient. Let me use that port to do some bad things." That ability to look across them is critical. You're absolutely right. And that's why, there's products... Anchore has some stuff here as well is that's where we want to enable the ecosystem to be able to innovate and build great secure solutions to keep a top these.
Josh Bressers:
Absolutely, it is exciting to me that we can talk about putting all these pieces together now. I mean, I feel and I would love your view on this as well is if you would've asked me a couple years ago about the software supply chain, I don't think it was really a topic. I feel like we've made an amazing amount of progress in really a year or two only, it hasn't been that long.
Steve Lasker:
Where we are literally at that phase where it is the next form of warfare government's use. It's extremely important to check. And having that focus, it helps, right? It's one of these things we've talked early on when containers were first being put on Docker hub and they weren't even signed. There was no concept of scanning and by the way, you can just grab anything from Docker hub and run it as part of the services in Azure whoa, whoa, whoa, wait, what?
Steve Lasker:
We needed a little bit of a check sum to go back and go this technology's cool. It's great. Definitely making us more efficient. It is the new virtualization of the two thousands of VMs, but there's some good practices that we need to apply to this. It's the cross ecosystem of consuming that content is another major place. The build versus buy is not really the questions anymore. It's not like I'm going to write all the code myself and justify. It's like you can't keep up with user demand. We have to be able to address content from all kinds of places that is constantly being updated. How do I bring it into my environment in a place where that has been used as a new web and tool.
Kim Weins:
Steve Lasker, thank you for joining us on upstream and thanks to everyone listening. You've come to the right place for great discussions of the big issues impacting the security of the software supply chain subscribe and never miss an episode. Upstream is available everywhere you get your podcast. If you learn something new, be sure to share this episode with friends and colleagues. Thanks for joining us on upstream, brought to you by Anchore.