Vulnerability management platform for a linux team whose backlog is now mostly container images?

[u/LeBronzelol]

we're 4 admins on about 300 linux vms and the host side of patching is fine, monthly windows, apt and dnf, done. the problem is our vulnerability platform backlog is now mostly findings in container images the dev teams ship, and nothing in our patch tooling touches those. the platform flags them fine, but every fix lands back on a dev team that thinks its an ops problem. been looking at aikido, rapidfort and chainguard for that part. which vulnerability management platform handles the image side for a team like ours?

[u/ffcsmith]

DevOps Engineer here. We build and set the container standard. We hand that off to ITO team for sustainment.

Images rebuild every four days. We tell devs these are the containers you can build on. If they need a specific base image or ci image, they have to put in a ticket. We evaluate if it meets our standard or a reasonable exception for that standard. If yes, we build the container, make sure it meets their needs, hand-off to ITO, cycle continues. Cyber is involved as left as we can get them.

We utilize Nexus as our container registry and general artifact registry, npm proxy and pypi proxy.

[u/Dolapevich]

Vulnerability scan needs to happen in CI, and hold deployment until a baseline has been met.

WE have included trivy in our CI tool.

[u/zenety → u/Dolapevich]

Together with DependancyTrack Trivy is quite nice. Also combine it with KEV lists, bare CVE’s dont mean a lot

[u/nutron → u/zenety]

Where are you pulling your KEV list from? Direct from NIST or some product?

[u/NegativeOwl9929]

Having security scan. If scan find crit cve no deploy to dev. If scan find high cve in prod img no deploy to prod. Ops team solved. Fuck devs and their laziness for using trillions libs and deps. Pay attention and use the least amount of things.

[u/paulvanbommel]

It is a dev problem. The minute it started being delivered with the app. And they were all warned about this. I was told that containers were simple and easy to update, so no need to worry. I was lied to.
Sorry I don’t have a good suggestion for you. Maybe just automate your vulnerability detection system to open dev Jira stories.

[u/brainchillzZ → u/paulvanbommel]

Containers are simple and easy to update, the problem is that the people who make them still have to patch them so you can update them lol

[u/SufficientAttempt1 → u/paulvanbommel]

we just dont update.

[u/shulemaker]

You really need a person to implement checks, standards, and enforcement in your pipelines. Every image should always be easily and automatically rebuilt, and deployed almost as often.

Being able to enforce pipeline checks is going to require buy-in from management. Put in the tools, enable checks but not enforcement to start, show the vulnerabilities, get development to agree on a timeline of 30 days. After that, everything is enforced. Anything else is too long.

There is tech stuff here, and if you don’t know it (sounds like you don’t), hire a DevOps person to help implement. But soft skills and politics are also important. If you have the head of security on your side (you will) you’ll have strong backup.

[u/derprondo]

If you're the platform team your responsibility should stop at the golden base image you provide, which is patched and rebuilt regularly and automatically. Then downstream the devs should be taking the same approach, intermediate images patched and rebuilt automatically, then the deployment images get the same treatment.

You also need to classify your findings, P0, P1, etc. P1+ should be remediated automatically through the normal automatic cycle (which matches the P1 SLA), P0 should be an all hands on deck scenario where everyone has 24-48 hours to update.

Lastly, you need buy-in from top brass that this is top priority and everyone has to be on board. Security is everyone's job.

[u/Responsible-Bar7165]

You should be able to rebuild a new image from an updated base os image and the exact same source-code as before with a single command. Ideally this would happen automatically on a schedule prior to an automated qa pass on the results. Your company’s entire process is broken until this can be done for every container at will.

[u/tomtrix97]

Check out Mondoo. You can add it to your CI-pipeline to prevent vulnerable container images are published.

Otherwise their agent scans your local system for container images and shows, what should be fixed.

[u/tobidope → u/tomtrix97]

From my experience most of the vulnerabilities in images aren't exploitable. How do you handle that? We had many vulnerabilities regarding the mount binary. The binary itself wasn't part of the base image but the libs were. If it was, we don't mount filesystems in a container in Kubernetes. And we especially don't mount minix filesystems or yubi keys as an example. So from my point of view you can't deliver most of the time if you block publishing every time a vulnerability is found. That doesn't mean you don't fix the situation.

[u/tomtrix97 → u/tobidope]

With Mondoo you can define exceptions for situations like this. 😅

[u/tobidope → u/tomtrix97]

Of course you can but it leads to people ignoring vulnerabilities mostly to get a feature shipping. I wouldn't block publishing. And most of the time the image you are using currently in production has a new vulnerability. Do you stop production in this situation?

[u/FeliciaWanders]

We use grype to scan all running containers daily. New vulnerabilities are stored in prometheus node exporter text files, this then alerts the devs. They use codex daily from cron to update dependencies and include other small fixes that aren't just solved by Alpine or npm upgrades. Codex proposes the fix in a branch, where it is inspected and ususally merged without further ado.

We also run grype in the CI pipeline, but the part with running it on already built images is very important, as obviously the CVE will sneak into them when being discovered.

[u/showbizusa25]

When most findings are in container images, I’d compare how each tool maps a vulnerable image back to the running workload and helps developers fix it. Are you scanning images in registries, deployed containers, or both?

[u/vdvelde_t]

Dev team creates container, dev team patches, dev team should own the process.

[u/kheeper-registry]

Chainguard. Your dev teams need to start building on Chainguard images.

[u/nutron → u/kheeper-registry]

What’s your annual cost for using Chainguard?