A map showing how internet across the east coast was affected by a massive DDoS on DNS provider Dyn... [+]
Think Friday's massive outage was bad? Worse is expected, as hackers are selling access to a huge army of hacked Internet of Things (IoT) devices designed to launch attacks capable of severely disrupting web connections, FORBES has learned. The finding was revealed just days after compromised cameras and other IoT machines were used in an attack that took down Twitter, Amazon Web Services, Netflix, Spotify and other major web companies.
In what is a first for the security company, RSA discovered in early October hackers advertising access to a huge IoT botnet on an underground criminal forum, though the company declined to say which one. (F-Secure chief research officer Mikko Hypponen said on Twitter after publication that it was the Tor-based Alpha Bay market). "This is the first time we've seen an IoT botnet up for rent or sale, especially one boasting that amount of firepower. It's definitely a worrying trend seeing the DDoS capabilities grow," said Daniel Cohen, head of RSA's FraudAction business unit.
The seller claimed they could generate 1 terabit per second of traffic. That would almost equal the world record DDoS attack, which hit French hosting provider OVH earlier this month at just over 1 terabit. For $4,600, anyone could buy 50,000 bots (hacked computers under the control of hackers), whilst 100,000 cost $7,500. Together, those bots can combine resources to overwhelm targets with data, in what's known as a distributed denial of service (DDoS) attack.
Cohen said he didn't know if the botnet for hire was related to Mirai, the epic network of weaponized IoT computers used to swamp DYN - a domain name system (DNS) provider and the chief target of Friday's attack - with traffic. But FORBES was able to find a forum post on Alpha Bay from the seller, who went by the name loldongs, which noted they had created a Mirai-based botnet. The original post was on 4 October, just a few days after the Mirai source code was made available to everyone. In a later post, in response to another user's request, loldongs claimed: "I can take down OVH easily."
RSA uncovered a botnet for hire, made up of IoT devices like connected cameras and fridges. It could... [+]
FORBES found the seller's post on the Tor-based Alpha Bay market, in which they claim to have used... [+]
Hackers have long sold access to botnets, though haven't explicitly advertised their use of IoT devices like connected cameras, fridges and kettles. The infamous LizardSquad amassed sizeable botnets for its LizardStresser "booter" - a DDoS weapon for hire - but it largely compromised vulnerable routers.
The avoidable carnage
For all the fuss on Friday, when major sites, from Twitter to Amazon's cloud, went into a mini-meltdown, the carnage was entirely avoidable. For two reasons, the web should have been better prepared for the onslaught, experts told FORBES.
First, the Chinese manufacturer of compromised surveillance and home video devices used by the Mirai botnet - Xiongmai Technology (XM) - claimed to have pushed out fixes that would've prevented its systems from being hacked in the first place. (Zach Wikholm, a researcher at security intelligence firm Flashpoint, told FORBES he had identified at least one XM device - a DVR - used in Friday's DDoS).
Cooper Wang, a spokesperson for Hangzou-based XM, told me the company had turned off Telnet (a feature allowing remote connection to devices) back in September 2015. Previously, Telnet was open on XM cameras and had a widely-known default username and password, allowing quick and easy access for hackers. Wang said customers were also now asked, upon setting up a device, to change default usernames and passwords for their web portal connection to XM cameras.
However, any device with firmware released before September 2015 remains vulnerable to those simple password hacks, Wang said. He did not confirm if protections were also being made available to XM partner products, which Flashpoint said contained the same vulnerabilities.
Wang recommended all customers update their firmware (though didn't say how to do this) and close any open ports to prevent any outside access. "To be responsible is one of [the] corporate values in XM. We have enough courage to admit the imperfection in our product, while we have more confidence to overcome all of [our] imperfections," Wang added, claiming that huge enterprises in the IoT market had similar issues. (After publication, XM said it was ordering a recall on devices in the U.S.)
If XM had shipped devices with decent protections prior to last September, and if partners had worked to improve security with the Chinese firm after the 2015 updates, those hundreds of thousands, if not millions, of devices may never have been hacked. And the Mirai botnet could never have grown to the beast that shook the web last week.
Call in the back up
Twitter, Amazon Web Services, PayPal and others could've been better prepared too, two security experts told me: anyone running a site should consider a secondary, back-up DNS provider.
The DNS is like a phonebook -- the user types in a web address, the DNS finds the right server providing the matching website, which is then connected to the browser. It's a crucial part of the web. Websites can provide more than one route through the DNS to its site, so that if one way is blocked with traffic, as happened Friday with Dyn's pipes, another can jump in and provide a new path. But that investment, it appeared, hadn't been made when Mirai came to call two days ago. Dyn is one of the more significant players in the DNS market, but there are many others to choose from. Google, for instance, charges a minimum of $0.20 per million requests per month to access a site via its DNS servers.
"Companies using third party DNS providers ultimately may not want to put all their eggs in one basket. We've already seen PayPal, a Dyn customer, add DNS services for another provider in addition to Dyn," noted security architect Kevin Beaumont for a global manufacturing company. "This will help mitigate problems for them in the future. It also works both ways and isn't a slam of Dyn - for example, companies could use Dyn as an addition DNS provider."
A permanent defense
Another tweak to the way DNS works for companies might have eased the pain for general web users. When someone enters a web address, the DNS doesn't always go through the same lookup process, routing right up to what's known as the "authoritative" DNS server. Instead, the system can quickly retrieve a previously-stored (or cached) response from a nearby server, making the whole process that much quicker. The period during which those responses are cached is known as the "Time to Live" or TTL. The shorter the TTL the quicker everything goes up in smoke if the authoritative DNS server is wiped offline, noted a security researcher who goes by the name MalwareTech. So Twitter et al should look to make their respective TTLs that much longer, they said.
"A combination of short TTL and no redundancy is what led to the issues on Friday," they added. "If [an affected site] had a TTL of, say, a day, as long as the DDoS attack is shorter than a day, most users would never notice anything."
According to CloudFlare security pro Filippo Valsorda, there's an even better solution: rather than lengthening the TTL, just ensure there's a permanent backup resource of records should anything go wrong. "You don't need to get DNS results directly from the source. Results are the same for everyone, and can be valid for a while," he told FORBES.
"Here's the point: if the global DNS system just kept replying with old results when the authoritative source - like Dyn - is offline, attacks on DNS providers would cause much much less disruption... There is no good reason resolvers should remove the results from the cache when the TTL expires, if they can't reach the source to update it."
Could Dyn have done better too? Perhaps, given Mirai's growing threat has been known about since it silenced independent journalist Brian Krebs and took down games maker Blizzard. But, according to Valsorda and Beaumont, Dyn was an innocent victim.
It was dealing with what it claimed were tens of millions of IP addresses firing requests at its servers. "Mirai was one source of the attack and there were other variants as well. Our team is still investigating and doing analysis of the series of attacks we received, but it's fair to say this was large, complex and sophisticated," said Dyn's chief strategy officer Kyle York.
With the difficulties associated in updating already-deployed, unsecure IoT products, the use of a diversified, backed-up DNS system looks like a faster-acting prophylactic for a digital disease that threatens to get out of control.
