Tavis Ormandy (Tavis Ormandy) of Google’s Project Zero uncovered a major vulnerability in the Cloudflare Internet infrastructure service. Essentially, web requests to Cloudflare-backed sites received answers which included random information from other Cloudflare-backed sites! This information could potentially include confidential information (private messages on dating sites, emails), user identity information (Personally Identifying Information (PII), and potentially in a healthcare context, Protected Health Information (PHI), or user, application, or device credentials (passwords, API keys, authentication tokens, etc.)
Both Project Zero and Cloudflare acted promptly. The bug was reported on 2017–02–17 and a mitigation was in place within an hour. Public notification was given on 2017–02–23.
The duration (2016–09–22 to 2017–02–20) and potential breadth of information exposed is huge — Cloudflare has over 2 million websites on its network, and data from any of these is potentially exposed. Cloudflare has said the actual impact is relatively minor, so I believe only limited amounts of information were actually disseminated. Essentially, broad range of data was potentially at risk, but the risk to any individual piece of data was very low. Regardless, unless it can be shown conclusively that your data was NOT compromised, it would be prudent to consider the possibility it has been compromised.
While Cloudflare’s service was rapidly patched to eliminate this bug, data was leaking constantly before this point — for months. Some of this data was cached publicly in search engines such as Google, and is being removed. Other data might exist in other caches and services throughout the Internet, and obviously it is impossible to coordinate deletion across all of these locations. There is always the potential someone malicious discovered this vulnerability independently and before Tavis, and may have been actively exploiting it, but there is no evidence to support this theory. Unfortunately, it is also difficult to conclusively disprove.
The most sensitive information leaked is authentication information and credentials. A compromise of this data can have lasting and ongoing consequences until credentials are revoked and replaced.
From an individual perspective, this is straightforward —the most effective mitigation is to change your passwords. While this is on all probability not necessary (it is unlikely your passwords were exposed in this incident), it will absolutely improve your security from both this potential compromise and many other, far more likely security issues. Cloudflare is behind many of the largest consumer web services (Uber, Fitbit, OKCupid, …), so rather than trying to identify which services are on Cloudflare, the most cautious is use this as an opportunity to rotate ALL passwords on all of your sites. This will improve your security, although the primary benefit is from threats unrelated to this incident.
(Best practice is to use a long random string for each password, unique for each site, and to manage that collection using a “password manager”, such as 1Password, LastPass, or the built-in password managers in modern web browsers. Users should also log out and log in to their mobile applications after this update. While you’re at it, if it’s possible to use 2FA or 2SV with sites you consider important (using something like TOTP/Google Authenticator or U2F), that’s a meaningful security upgrade, too.)
For site operators using Cloudflare: while it may be disruptive, you should seriously consider the impact on your users. Use this incident as an opportunity to exercise your incident-handling process — discuss the specific impact to your application and what response makes the most sense (it is likely different for every site or application.) You have to balance an unknown but small risk vs. other costs. Be careful not to “scare” them, but it may be prudent to take action. At minimum, I would prepare a “stock answer” for support about this incident, and in an enterprise or b2b context, proactively communicate with customers about the extent of the incident and its effect on your application and their data. For the vast majority of sites on Cloudflare, that alone is likely sufficient.
There are very real costs to forcing a user password change — users might lose trust in your service, and it’s an inconvenience. It doesn’t appear large numbers of credentials have been compromised, so for a consumer service with limited risk to compromised accounts, it may not be worth the effort to mass-invalidate passwords and force a change. For administrator credentials, or for any sites processing highly sensitive information through Cloudflare, the lack of a quantifiable maximum exposure probably means it is worth forcing a password update. If you had any other reasons for wanting to push a password update to all of your users, this would of course be an additional factor in the decision.
Sites should invalidate authentication credentials for mobile applications and other machine to machine communications (IoT devices, etc.), forcing users to re-enroll apps and devices, if they used Cloudflare as an infrastructure provider. If you don’t want to force a change of passwords, a viable intermediate step could be to invalidate authentication tokens, forcing users to log in again with their existing passwords; since authentication tokens are more frequently passed between browser and server, they are far more likely to exist in a leak of random memory from the server than raw passwords, and forcing a fresh login is a minimal impact to users and may not require any specific messaging or notification.
Additionally, any sites not on Cloudflare but with a lot of users who use Cloudflare-hosted sites (essentially any large or consumer sites on the Internet) should consider forcing user password changes in case their users had re-used the same passwords on each site. This is probably not warranted for the vast majority of sites (anyone with security needs so high that this would make sense should be using authentication infrastructure far stronger than user passwords; otherwise, it is nearly certain users will re-use passwords with a compromised site), but it may be a possibility. Personally, I would use it as an opportunity to upgrade my overall authentication infrastructure in such a case — deploying 2FA (ideally U2F or better, or user-selectable TOTP/HOTP, not SMS), account activity monitoring, etc. — rather than rushing a change like invalidating passwords. Standalone passwords on the Internet are no longer best practice for user authentication.
If your application or website is on Cloudflare and is subject to industry or national regulation, this may be a reportable incident. (examples would be health care/medical privacy concerns with HIPAA). Your security and compliance teams should evaluate. Obviously, full compliance with applicable regulations is an essential part of security.
Subjectively, I believe this is a serious security issue and should be evaluated by any major service using Cloudflare. It is likely not an “end of the world” in severity, as unless there was concerted malicious exploitation, vulnerable data is probably fairly randomly distributed across the Internet, but presence of data in searchable caches might make small-scale exploitation possible. Since the mitigation for end users is good advice in general (use a password manager, use unique random passwords per sites, rotate on any compromise), it’s a “no brainer” for most security-conscious end users. For site operators, the decision really comes down to your specific userbase and their level of security sophistication and risk tolerance. I’m deeply appreciative of both the ongoing work by Google’s Project Zero (especially Tavis) and by Cloudflare’s rapid response to this issue.
