As some of you might know, I’m working at Sonatype. That gives me great access to the behind-the-scenes data from Maven Central. You didn’t know Sonatype were the stewards for Maven Central? Well now you know. The folk at Sonatype do quite a lot for the Java community but tend not to shout about it much. That’s a longer story for a different time.
This article is about what we can tell, or perhaps guess, what the Java community is doing to deal with the series of log4j vulnerabilities that have come our way since 9th December. As you’ll see its clear many have got the message but quite a few either missed it or haven’t paid attention. Guess which list they might be on!
Sonatype has put up a series of real time graphs that provide some interesting insight into the characteristics of Maven Central download patterns for Log4j packages. You can find the live page here
Let’s take a look at what’s on the dashboard.
Firstly are the top level numbers. As of writing on 23 December there have been 5 749 204 downloads of all Log4J2 versions. That’s about 425K downloads per day. Quite a large number but a fraction of the many many millions of daily downloads from Maven Central.
The good news is that about 65% of those log4j downloads are for versions that have one or more fixes for the Log4Shell related CVEs. At least we can claim more patched versions than not are downloaded but 65% is still pretty poor for such an important set of fixes.
Are we paying attention?
On to the first row of charts. Both show downloads per hour. The lefthand graph gives you raw numbers of downloads of Log4j artefacts by version while the right one provides a normalisation of the data as a percentage of total downloads. The normalised chart makes it easy to see that the transition from the vulnerable versions happened quite quickly. The number of downloads of the safer versions went from zero to 60% over the course of a few days and, as the situation unfolded, movement to the absolute latest version increased as well. The chart shows that even though 60% of the traffic is now for safer versions only about half of those downloads are for version 2.17.
The stacked version of the data as shown in the graph on the top left also has some curious characteristics. The strange double peaks are not new to Maven central download patterns. It suggests that CI/CD systems are being triggered and have an update to do. This is a common occurrence and the fact that the pattern is so obvious here suggests that those involved are not particularly concerned about the Log4Shell vulnerability. They seem willing to let the weekly processes do the job rather than expedite an upgrade.
The fact that this pattern continues even as safer versions are selected suggests that for most of the world patching the vulnerability has been important but not as urgent as you might think.
The wider picture
The final two charts provide geographic insight into where the updates are happening. These charts are based on best guess for physical location. The chart on the left, entitled “Log4j Central Non Vulnerable Download Percentage”. Shows the downloads by country. Countries with less that 50K daily downloads are excluded so we see only the top locations. The numbers show which countries are downloading what percentage of the safer versions. Countries like Taiwan seem to have missed the Log4Shell issues almost completely, while many others like Korea and the Republic of Ireland are mostly (80%) downloading safer versions.
The shape of the curve also tells you about reaction time and sustainable effort. Although its quite a muddy chart you can see that most countries applied an effort immediately and then , more or less, have run at that level for the period. Others like the Republic of Ireland have been increasing their efforts over time. It’s all good news but there are tiny signs that these numbers are dropping. Probably due to the approaching holidays. That does suggest that there will be many vulnerable servers left unpatched into the new year.
The final chart – the one that is a world map – is a different view of the same data but expanded to include more countries. The cut off is at 10K downloads per day so countries below that number show up as white. The other colours indicate the percentage of safe vs unsafe downloads. Green is better, red – not so much.
Can we make some predictions?
Reading the tea leaves that are download statistics is always tricky. Where the download originates, whether it’s a developer, a build machine or a cache-filling mechanism. There are many variables here so need to look at the big numbers. The good news is that updates are happening fast and the move to the safest version is occurring. The bad news is that despite many people having spent long nights upgrading their systems they still often picked a vulnerable alternative. From other data available it’s clear that the process of upgrading is still seen as a chore by many developers and the risks and consequences of not upgrading – even for something as severe as Log4Shell – are not understood or believed.
The last big news vulnerability in the Java world: ( the “Struts2” issue of 2017 which caused so much misery and damage) is still with us. The versions of Struts2 which have the issue are still regularly downloaded from Maven Central.
It’s not really a predication – more a certainty – that unless we recognise how the world has changed over the last couple of years little will be learnt by the Java community from this current situation. Take a good look at the Log4Shell situation from a technical, business and political standpoint. Think about the effort spent in emergency remediation and understand that this could become the norm. Time to fix is rapidly evaporating and applying herculean efforts on a regular basis is unsustainable.
It’s time to put process improvements, automation and some good old fashioned security education on your list for 2022.
Naughty or Nice?
There are many people who have moved heaven and earth to address Log4Shell and the follow up vulnerabilities. From the Log4J volunteer community to the developers working overnight throughout the world to apply fixes. They all deserve our thanks for literally making the world a safer place.
Let’s hope that 2022 will see us making the world even safer.
steinhauer.software