Critical vulnerability under “massive” attack imperils high-impact sites [Updated]

It is important to understand the nature of the vulnerability. It is actually very simple. Struts supported OGNL, a kind of scripting language for the JVM. Struts would mistakenly execute OGNL that appeared in the Content-Type header (which is quite shocking). OGNL can be very dangerous, because it is the surest way of introducing stupid code-injection vulnerabilities in Java code like the ones that plague dynamic languages. The best solution is to ensure your projects exclude the OGNL library.

The short answer is: 48 hours is utterly inadequate for this kind of update.

The long answer is:

Unlike, for example, a Windows flaw, or, say, an OpenSSL flaw, Struts is not a system library provided by an operating system or Linux distribution. With operating system libraries, maintenance is relatively straightforward: you hit Windows Update or run apt-get update, or whatever the case may be for your platform, and the fixed library is installed. A quick reboot to restart applications and you're good to go; all your software is now using the fixed version of the library and you can move on.

Rather, Struts is a library that is bundled with applications. Typically, every Struts-using Web app on a system will embed its own copy of the Struts JAR. Technically, this isn't absolutely necessary; you could in principle put your Struts JAR in a shared JBoss/Tomcat/Websphere/etc. location and have every deployed app use the shared JAR from the classpath, but this is rarely done because of the versioning headaches this causes. This means that updating Struts is outside the control of typical IT departments; sysadmins can't fix this. Instead, they have to get application developers involved.

An organization may have tens or hundreds of little Struts-using Web apps, all with their own Struts JAR embedded within them. Many of those apps may be essentially abandoned; the earliest affected version of Struts was released in October 2012, and I bet that there's plenty of apps developed since then that are "finished". They're still used and deployed, but they're not receiving ongoing maintenance; their developers have moved on to other projects, or even other companies.

Fixing those applications means getting the source code, updating the build scripts to change the Struts dependency to the latest version (2.3.32 or 2.5.10.1), and then rebuilding the application. For currently-developed code, that may be easy, but for a three year old app that hasn't been touched in a while? That's a little hairier. You might have to dig out older JDK versions to get it to build, find an old copy of an old internal JAR that's somehow gone missing, all the usual problems that happen when you try to rebuild an old application. That's assuming, of course, that you have the source code and build scripts, and that alone is far from guaranteed. I bet that there will be developers who find that the version in source control for some reason doesn't quite match the version that's deployed, or that they have no source at all, or that it doesn't build for whatever reason.

So, your developers have to update their Maven or gradle or (god forbid) Ant build scripts and bump the version number for the Struts dependency to grab the new version.

You then have to hope that nothing is broken. If you're using Struts 2.3.5 then in theory Struts 2.3.32 won't break anything. In theory it's just bug fixes and security updates, because the major.minor version is unchanged. In theory.

In practice, I think any developer going from 2.3.5 to 2.3.32 without a QA cycle is very brave, or very foolhardy, or some combination of the two. Sure, you'll have your unit tests (maybe), but you'll probably need to deploy into your QA environment and do some kind of integration testing too. That's assuming, of course, that you have a compatible QA environment within which you can deploy your old, possibly abandoned application.

Then you'll have to schedule an actual deployment of the updated application. If the app is world-facing, that may mean delaying until the weekend or night time or similar.

And all this is presuming that your developers even know about the problem. It's not (necessarily) super straightforward for IT to identify which apps are using which versions of Struts, so IT might well not know. And developers may very well not be tracking this stuff anyway. Struts was regarded as old-fashioned and backwards when I was writing Java just over a decade ago; I daresay it's even less sexy now. So your developers probably unsubscribed from the Struts mailing list, and probably aren't reading the release notes for each new Struts version. They've moved on to better, newer frameworks.

This kind of bug is a problem that IT will struggle to identify, and that IT can't fix themselves. Developers may well be unaware of the flaw, but developers and QA are going to be on the hook to fix it. There's no way a problem like this is getting any kind of widespread reaction within 48 hours. The wheels just don't turn that fast. This is a big hairy mess.

Agreed.



I have to respectfully disagree. This attack is being carried out through Content-Type headers. It is well within the domain of an IT department to stick a proxy in front of the Struts application and send a 418 response to anyone sending a Content-Type header containing OGNL.

It is important to understand the nature of the vulnerability. It is actually very simple. Struts supported OGNL, a kind of scripting language for the JVM. Struts would mistakenly execute OGNL that appeared in the Content-Type header (which is quite shocking). OGNL can be very dangerous, because it is the surest way of introducing stupid code-injection vulnerabilities in Java code like the ones that plague dynamic languages. The best solution is to ensure your projects exclude the OGNL library.

The short answer is: 48 hours is utterly inadequate for this kind of update.

The long answer is:

Unlike, for example, a Windows flaw, or, say, an OpenSSL flaw, Struts is not a system library provided by an operating system or Linux distribution. With operating system libraries, maintenance is relatively straightforward: you hit Windows Update or run apt-get update, or whatever the case may be for your platform, and the fixed library is installed. A quick reboot to restart applications and you're good to go; all your software is now using the fixed version of the library and you can move on.

Rather, Struts is a library that is bundled with applications. Typically, every Struts-using Web app on a system will embed its own copy of the Struts JAR. Technically, this isn't absolutely necessary; you could in principle put your Struts JAR in a shared JBoss/Tomcat/Websphere/etc. location and have every deployed app use the shared JAR from the classpath, but this is rarely done because of the versioning headaches this causes. This means that updating Struts is outside the control of typical IT departments; sysadmins can't fix this. Instead, they have to get application developers involved.

An organization may have tens or hundreds of little Struts-using Web apps, all with their own Struts JAR embedded within them. Many of those apps may be essentially abandoned; the earliest affected version of Struts was released in October 2012, and I bet that there's plenty of apps developed since then that are "finished". They're still used and deployed, but they're not receiving ongoing maintenance; their developers have moved on to other projects, or even other companies.

Fixing those applications means getting the source code, updating the build scripts to change the Struts dependency to the latest version (2.3.32 or 2.5.10.1), and then rebuilding the application. For currently-developed code, that may be easy, but for a three year old app that hasn't been touched in a while? That's a little hairier. You might have to dig out older JDK versions to get it to build, find an old copy of an old internal JAR that's somehow gone missing, all the usual problems that happen when you try to rebuild an old application. That's assuming, of course, that you have the source code and build scripts, and that alone is far from guaranteed. I bet that there will be developers who find that the version in source control for some reason doesn't quite match the version that's deployed, or that they have no source at all, or that it doesn't build for whatever reason.

So, your developers have to update their Maven or gradle or (god forbid) Ant build scripts and bump the version number for the Struts dependency to grab the new version.

You then have to hope that nothing is broken. If you're using Struts 2.3.5 then in theory Struts 2.3.32 won't break anything. In theory it's just bug fixes and security updates, because the major.minor version is unchanged. In theory.

In practice, I think any developer going from 2.3.5 to 2.3.32 without a QA cycle is very brave, or very foolhardy, or some combination of the two. Sure, you'll have your unit tests (maybe), but you'll probably need to deploy into your QA environment and do some kind of integration testing too. That's assuming, of course, that you have a compatible QA environment within which you can deploy your old, possibly abandoned application.

Then you'll have to schedule an actual deployment of the updated application. If the app is world-facing, that may mean delaying until the weekend or night time or similar.

And all this is presuming that your developers even know about the problem. It's not (necessarily) super straightforward for IT to identify which apps are using which versions of Struts, so IT might well not know. And developers may very well not be tracking this stuff anyway. Struts was regarded as old-fashioned and backwards when I was writing Java just over a decade ago; I daresay it's even less sexy now. So your developers probably unsubscribed from the Struts mailing list, and probably aren't reading the release notes for each new Struts version. They've moved on to better, newer frameworks.

This kind of bug is a problem that IT will struggle to identify, and that IT can't fix themselves. Developers may well be unaware of the flaw, but developers and QA are going to be on the hook to fix it. There's no way a problem like this is getting any kind of widespread reaction within 48 hours. The wheels just don't turn that fast. This is a big hairy mess.

Agreed.



I have to respectfully disagree. This attack is being carried out through Content-Type headers. It is well within the domain of an IT department to stick a proxy in front of the Struts application and send a 418 response to anyone sending a Content-Type header containing OGNL.