Back in July, hundreds (thousands?) of sites noticed a surge in traffic in their analytics. Hundreds to thousands of additional visits per day above their normal traffic rates. Curiously, all this traffic identified as Internet Explorer 7 on various version of Windows, but each visit came from a different residential IP, and viewed only one page per session.
For months now, huge botnets have been producing huge amounts of fake web traffic, picking up retargeting cookies in order to later be served ads and have whoever's behind the network paid for those views. That surge has never ended. It's only grown (including becoming less easy to spot), yet nobody seems to be talking about it anymore, including the retargeting companies.
I'd love to know who's behind it and how they control so many computers.
Recently (during last two weeks or so) a lot of such suspicious traffic started appearing in Google Analytics as various versions of Internet Explorer, not only 7. We were forced to filter out IE completely to just calculate stuff like conversion rate. Great that legitimate traffic from IE was negligible anyway, but who knows how much of this traffic is still hiding there as some other browsers?
Thankfully, as far as I know we don't do any retargeting campaigns right now (but I think we did in the past, and we were targeted by those IE7 bots in July as well), but it's still awful PITA to deal with it in analytics data.
Sometimes this stuff isn't even bot traffic, it's traffic purchased from someone who fulfills by incentivizing people who play online games with virtual currency for viewing videos or clicking through to sites that serve the buyers ads. I presume they've also got into the retargeting capture game, too.
Yup. Many facebook games have a "watch this video" or "visit these sites" in an iframe in exchange for extra points, more seeds, a better pool stick, etc. Just as useless as bot traffic.
Retargeting was definitely ripe for abuse. This underscores the importance of getting MUCH smarter and pickier with your audience lists and how you actually define them and value their contribution to a given conversion.
There are a thousand more or less easy ways to pull this, but it is hard to stay unnoticed at this scale. You'd guess that every plugin/toolbox/freeware used by hundreds of thousands of people is running on some test-boxes/honeypots set up by security companies/browser-vendors.
SV will not care about this until another 2000-level crash. Then, they will be forced to care, because the revenue will stop coming in. Until then the community will do the see no evil / hear no evil / speak no evil routine.
I envision a future where a significant share of the advertising market is comprised of automated ad-bots pitching services and wares to automated web-crawlers. There's something obtusely philosophical about that and what it says about us, though I am not sure what that is.
That is already happening to an extent, except the wares being promoted are the ads themselves and they're being sold in sub-second auctions where the bidders are different algorithms operating on behalf of various companies.
I've been in the online advertising world for 10 years and it still befuddles me to see that most ad dollars still haven't switched to cost-per-action (CPA), i.e. only paying for actual conversions. One big benefit of advertising online after all is that tracking conversions and their referrers is much easier than with offline.
CPA simply doesn't make sense for all ad campaigns. When Lexus buys ads on a financial news site it's for branding -- they want to get their logo in front of their target demographic. The ads aren't optimized for clicks or leads -- sometimes there isn't even a clear call to action.
Yup, but then why bother with CPM? Just buy display ads for a fixed price in publications they believe their customers might read. Sure, it's not an improvement over what they did in the 50s but they don't have to worry about fraud.
Also with CPA, if I'm a seller, I'm not going to sell to you unless I know your landing page converts. That's asking me to do a lot of due diligence or assume a lot of risk.
There's also the problem of being sure your conversions are attributed to you.
But these aren't that much of a problem in practice, really. See how affiliate marketing works: it's CPA for all intents and purpose masquerading under a different name. The biggest difference might be that the vendor will contact you, rather than make some kind of bid, if he thinks you've the right kind of traffic.
If your audience is interested in a particular product, you quickly discover it through common sense and trial and error, as you can normally see the click-through rates.
Also, a big ticket item with a low conversion rate sometimes earns better than a smaller one with a high conversion rate. As such, maxing out your ROI is really a matter of seeing what contributes to your bottomline.
I am a pure CPA buyer, so, I agree, that I don't worry much about fraud.
The big money is in brands. P&G goes on a 6-month blitz making sure every man knows about the fusion razor, and the person that cleans the house knows what a swiffer is. They use an ad blitz to secure shelf space and consumer interest, then print money hand over fist for the next decade.
As a CPA buyer, I personally welcome rampant ad fraud because it keeps the people with real money like P&G from bidding up ad prices. Ad fraud screws brands and publishers. It helps CPA advertisers, because it keeps the brand money out of the system.
I've been in the space for the same amount of time and I'm not at all surprised by this.
It all comes down to risk tolerance for both parties involved. On one end of the spectrum, CPM models put ALL of the risk onto the advertiser, and CPA is at the other, putting ALL of the risk onto the publisher. CPC falls in the middle.
Depending on who has more leverage in the negotiation, what is possible for one advertiser may not be possible for another. But given the low-quality traffic I've seen on some publishers, and specific instances where I know publishers were arbitraging $.01 CPC clicks into their inventory counts, I'm not at ALL surprised they are clinging to CPM as long as they can. Their very existence depends on it.
High-quality publishers have little problem going with CPA in cases where they can control more of the funnel (which is often the weak link), and they can command a much higher spread as a result.
tracking conversions isn't easy given cookie churn. lots of advertisers end up paying in a last-view attribution model, so that drives a lot of dirt cheap ad purchases. and regardless of whether most advertising is sold on a cpa model, much of it is evaluated on a cpa model. the structure of ad agencies seems to be that they run $x worth of advertising and take a fraction, so they're heavily incented (and hence incent the dsps) to spend all the money. nonetheless, the campaign is evaluated to determine next quarter's spending on a cpa basis.
further, it's not clear what an action would be for a brand campaign
finally, you can demonstrate, eg by geo-fencing, that display ads work, even if many or most of them don't lead to attributable conversions
There are definitely ways to test display performance, such as the common PSA (public service ad) test to determine view-through contribution.
That said, I'd be careful painting all agencies under the same brush. There are plenty of CPA/performance-based agencies out there now, plenty that are flat fee-based, and every shade of gray inbetween.
Ultimately, ad agencies are incentivized to retain their clients, and if they have performance-based clients, just spending their money with nothing to show will not keep them in business very long. Likewise, people assume most brand advertisers at big companies are stupid. First off, you can't be a "big brand advertiser" without a "big brand" which means "a ton of cash." This also means they have fairly savvy and advanced statistical methodologies and research groups in place to help measure overall lift. Marketing mix modeling (MMM) is their lifeblood. Sure some of them have idiots working there, but at a certain point you have to look at the data at a higher level because you simply can't track everything you do in an accurate manner. Doesn't mean there isn't a way to observe lift and attribute it.
it still befuddles me to see that most ad dollars still haven't switched to cost-per-action (CPA), i.e. only paying for actual conversions
This assumes that ad networks with useful returns offer such an option.
I have no idea what Facebook have been playing at over the last month or so, but I do know that the new system seemed to charge us about 4-5x as much per real visitor to our site as the old one, despite having what appears to be the closest equivalent set up for click-based charging in the two systems. Of course, even that is only Facebook's idea of clicks, which includes non-conversion clicks such as people liking your ad itself or your page, not just the conversions you actually care about.
We pulled all Facebook advertising within a couple of days as a direct result of this and asked them what was going on. They claimed nothing had changed, and I'm told they suggested the dramatic difference was just down to "increased competition". Our theory is that a lot of other small/niche advertisers were also confused by the new model, but many didn't realise what was going on and continued running their own campaigns at the vastly higher cost per click, at least initially.
I was recently in a class where we performed a case study on Google. The professor asked if anyone had ever clicked on an ad. One kid out of 60 had clicked on ad for cheap protein supplements, but hadn't bought any. Everyone else scoffed, pointed out their use of AdBlock, and cited their gullible parents as the only people they knew who might ever possibly accidentally click on an ad. Furthermore, the strong majority of our class viewed targeted ads on Facebook and Gmail as extremely creepy and unconvincing.
There are two ad models that work in my opinion:
1. Native advertising: Ads disguised as useful content. This works for changing people's perceptions, especially if its done through official or generally impartial channels, such as news outlets. People think they're getting honest information about BP's charitable acts and Coke's health benefits, but they're really being lied to. This doesn't generate any direct sales, but it can really influence consumer sentiment on different brands which can have a massive impact on sales indirectly. This method is extremely sneaky and unethical, and that's probably why it works.
2. Snapchat's new model: Opt-in, high quality ads. I liken these ads to movie trailers. They are well-crafted and curated, and you only have to watch them if you want to. This way, people who click on them are already intrigued and have committed to watching the ad, making for a more engaged and curious user base. This method is still being developed but I can see it become big in the next couple of years.
Everyone else scoffed, pointed out their use of AdBlock
If your entire class was running AdBlock, then your sample is far from representative of the general population.
There are two ad models that work in my opinion
The thing is, opinions don't matter in this game, facts do. Do you really think major brands like Coca Cola or McDonalds or Ford spend such astronomical sums of money on advertising without being pretty good at knowing what returns they get?
You might think that no-one really buys a car because of the product placement where the successful business owner drives that model in their favourite TV show, but you'd be wrong.
> You might think that no-one really buys a car because of the product placement where the successful business owner drives that model in their favourite TV show
That would be an unusual viewpoint. People love identifying with their heroes -- I don't think anyone considers that a secret.
It most certainly isn't. I think there's a bias on HackerNews to default to hating on ads, but they can be very powerful. I'm certainly biased because I do it for a living, and the evidence I can offer is certainly anecdotal but I've run ROI positive campaigns based entirely off facebook and twitter ads. In fact I've done advertising for a $1,000,000+ grossing product where 98% of downloads could be directly contributed to an ad we ran while remaining ROI positive.
I don't entirely dismiss your point though. I think the large scale buying of billions of impressions is certainly broken. Big advertising agencies have taken to just throwing tons of ops people at the problem.
I think the real shift, and the pioneering ways forward are going to be a combination of social ads that are highly segmented, and then what I call "digital experiential" This: http://sortieenmer.com/ is a good example of what I mean.
Basically, buying banner ads really efficiently aren't enough anymore. We need to be creative again and be better about segmenting our audiences to make ads relevant again.
Besides the non-human traffic, most online ads are created to not be seen. I was even thinking about creating a blog post to call out the marketers out there to change this, but I am too lazy... specially to make them make more money.
I am so used to how ads that cover the entire page work that I manage to pass by them without even knowing which brand was advertising there.
Ads that force you to watch a video for X seconds, but in those X seconds they don't even show a logo or something, are money wasted indeed, again, IMHO.
Another example, ads that follow me. Oh god how I hate these. I visit a website, usually for a product that I already own, and the freaking ad from that company follows me wherever I go. This is money wasted.
Another example, I use Waze almost everyday to figure out the best route for my comute. My 4" screen phone sits there right on top of the radio. Whenever I stop the car, an ad pops up. But they make this ad so tiny that if I need to know what brand or product it is for, I have to pick my phone and look closer. This is money wasted.
The online advertising system is pretty broken... can I fix it?
The ads that follow you are called retargeting and the reason why they are used is... because they work! They get dramatically more clicks and conversions than other ads. There are also less obvious retargeting ads where websites trade their audiences with advertisers.
But yes, the ad system is completely broken. It's a complete security nightmare - they track you, in plain text, without telling you, they sell data about you without your consent, they are a great way to deliver exploits to millions of people (and their security policies are a complete joke). There are click-farms everywhere, pricing is not transparent, there are less-than-honest practices against advertisers all over the place, inventory gets sent to networks where they don't belong, then there are the intrusive ads, the dark pattern ads...
It would be nice to have an online ads industry that is safe and secure and honest to fund Internet content, but so much about (mass) advertising is so inherently anti-customer that I don't think that will ever happen. I hope we'll figure out ways to directly pay for content instead.
If you can get a business plan from those observations, yes, you can fix it. The advertising industry seems to be much talentless, despite the image they advertise.
Yes. You can. Google has made some acquisitions lately in this anti-fraud space.
Most of the Valley does not care, and seems to be invested in maintaining and expanding the scale of the fraud. The market will eventually punish this when conditions change to be less frothy. It will be 2000 redux, for similar reasons as to why SV lost so much credibility the last time around (pervasive traffic fraud).
>Another example, ads that follow me. Oh god how I hate these. I visit a website, usually for a product that I already own, and the freaking ad from that company follows me wherever I go. This is money wasted.
Well, they do work for some people. They just have to be managed in a way that doesn't create this annoyance. You cap the frequency or make it so that the offer is actually attractive and useful to the user. Or you limit what it triggers on to only go off when the user does something that signals their intent to become a customer (like adding certain items to their cart and then abandoning without buying).
Many things that people hate about online advertising as a whole is more caused by people mismanaging the ads than is caused by platform problems.
I will never understand the YouTube ads that show 3-4 seconds of black screen and then "allow" (not sure what controls this) the "skip this ad" after 5 seconds.
You (the advertiser) have to capture my interest in the first 10-20 frames of video, and pitch me in 5 seconds. Anything else is a waste of your money.
And YouTube only has one chance as well. He fact I'm seeing this annoying video ad means I have already installed Ad block edge and haven't reloaded the page.
I kind of like some of the retargeting ads. The web is a much nicer place when all the ads are for things I care about, rather than "100,000th visitor, click here".
Plus, at least thus far the retargeted ads have all been... friendly? Not obnoxious?
> Another example, ads that follow me. Oh god how I hate these. I visit a website, usually for a product that I already own, and the freaking ad from that company follows me wherever I go. This is money wasted.
Campaigns like this, while frustrating for existing users, aren't there to target us, they're to target someone who filled out the sales form halfway and got distracted.
If I was ever going to buy from New Relic, I wouldn't now - I've seen their retarget ads at least a thousand times. If I didn't care about click fraud, I would have made a point to make sure I cost them a few dollars.
I wouldn't be surprised to find that the larger firms are crawling their own ad platforms. When ads comprise 90% of your revenue, it's hard to imagine that someone hasn't gamed the system.
there's a certain irony when advertisers--whose intention is to manipulate the preferences/behaviors/purchases of consumers--cry foul about their manipulation systems being manipulated
I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a "ticket ID" so the (very few) real users caught can still contact the website admins to be unblocked.
The (almost feature-complete) version running at the site above was able to automatically identify and successfully block a number of bots found scraping (or, attempting to scrape) the site since 2012. Though it wasn't an easy task, it has been quite successful at blocking most of them since their second (often first) request.
In fact since the beginning, and largely upto now, most bots and botnets can still be easily spotted. Some by their user-agent, some falling into bot traps ("honeypots") which are invisible to regular users, some failing to load images (something that a regular user wouldn't do), some crawling the site at an unreasonable pace... most usually don't provide an http referer, some even appear to have a number of cookies set for your domain.
It's not always easy to spot them though - some botnets can only be undone by blocking their full IP subnet, or permanently banning their (sub)domains, like .cn, .ua or .br (if you don't have legitimate traffic from those domains).
Unfortunately, I haven't had enough spare time to complete my project (which tests all of the above conditions to spot bots, but is not ready for a release), but I could if there is some market - or I could sell the project if somebody is interested.
It's not hard to make a 70% effective bot blocker. It's extremely hard to make a 95%+ effective bot blocker without having a large number of false positives.
It is (probably) impossible to make a 99%+ effective bot blocker without having a very large number of false positives.
I've been working on this (and abusive user blocking) for a long time now and I'm roughly where you are, I have a product, it could be launched but I'm very wary of launching a half baked product that would eventually end up being disabled. I've had the help of a friendly corporate entity that gave me millions of datapoints to play with allowing me to look into their kitchen both on the support end and on the incoming traffic end as well as verified abusers.
This identified various fraud such as ad click fraud, affiliate scams, repeated harassment by the same users under different names, spam of all kinds and so on. It's been working there for more than 4 months on a fairly high level of confidence. And yet, I'm still wary of rolling it out on a larger scale.
So now 25% to 50% of impressions go to ad bots. It's known that most ad clicks from about 10% of human users, who will click on anything but buy little.
The real winner in this is Amazon. Amazon doesn't pay anybody when someone clicks on a product listing. They have to buy. Same for eBay. So there's no point in 'bots clicking on those sites.
Online ad rates have been dropping for some time now. Most of the ads run by Google Adsense (the ones on third party sites) are there because Google pushes advertisers hard to use that feature. Most of the conversions from Google Ads come from the ones on search results, which are displayed when the user is looking for something. Those are useful. Most other forms of advertising are interruptions for users interested in something else.
The classic tactic ad-based companies try as ad prices drop is increasing the ad density. That's called "pulling a Myspace". It didn't work for Myspace, which tanked. Twitter seems to be headed that way - usage up, profits not too good.
The way this is going, online advertising is going to have the conversion (to a sale) rate of spam - one in thousands.
While I understand the downvotes, the sentiment here is valid in the sense that there's been so little innovation in advertisement for the web - even mobile ads are anachronistic.
The entirety of innovation has been sizes & styles and tracking and not on presentation and usability that's complementary with content.
It's not a 'you made your bed' situation, but not wholly removed from it, either.
There's a lot of room for innovation in this space. Some friends (well, clients) of mine have a startup (Unlockable, https://www.unlockable.com/) that embeds advertising-based games into other games. The level of interactivity is way more than a bot could handle.
> Do you hate how easy it is for your computer to become slow and riddled with malware?
No, I can't say I do. If such crime is so rampant, one would think it was ripe for law enforcement to do more about it. I wonder if reporting it to the police every time one visits a relative and removes crap from their computer would start a disruption.
I went to the local electronic crimes branch of law enforcement once, when my Gmail account was broken into.
They basically asked me if I was reporting large-scale financial theft, human traficking or child porn. If it wasn't one of those, they didn't have the manpower or will to pursue it.
There are two pretty obvious approaches: only pay per conversion, or accept that not all views are human.
Just like when you advertise in a classical news paper, you only know how many copies are typically sold, not how many humans actually look at your ad.
- Each has their own tracking tag that their terms require payment be based off of
- A conversion comes in and each network registered an impression (or heck, even a click) before the visitor in question signed up.
Who gets the credit?
Well, if you don't have an agreement that states billing is done off of numbers from YOUR ad server, where you can dedupe and set an attribution model that makes sense for your particular business, the answer is that all of them will.
So you are paying 3x when realistically odds are that not all of them should get full credit. Hell, that doesn't even factor in other channels like organic, social, etc. that all likely played a role as well.
This is a problem I have seen first hand. But this is not really as big a problem as it is made out to be. The real issue here is that it is hard to predict of the given impression is a bot impression or a genuine impression. As a consequence any CPM based dealing is difficult. Difficult only means falling CPMs. That is what I have observed.
The advertisers will not have to put in more efforts to measure conversions and accordingly bid for the traffic.
I am also working in ad yield analytics, so while I wait for a DFP report to run, I'll also say, it's also in the network's interest to address this issue. In the meantime we'll just see more CPA line items and a drop in price for CPD and CPM. Being an auction, I'm sure Appnexus already sees this. When the issue is addressed we'll see a rise in CPM that reflects the increase in value with less bots. There is money in fixing it so I'm not calling the end of the on-line advertising world like the article, which is written by a general reporter not someone in the field (Cost Per Mile? oh really?)
Just wait until cross-channel attribution (particularly dynamic attribution vs. fixed model which is more commonly available) really takes off. CPA-based buys are going to come under a LOT more scrutiny when Joe Advertiser suddenly realizes that his retargeting campaign didn't actually add a ton of value to the path for the vast majority of his conversions, especially if he's giving <gasp> 100% credit to view-throughs.
That's awesome that PerfectAudience does that, and more and more DSPs are definitely waking up to that as well which is great.
What do you do to help inform what the settings should be? Unless an advertiser has a solution like VisualIQ, Convertro, Adometry, etc. (and they've all been bought up so who knows what will happen to them), or GA Premium, they likely don't have a dynamic attribution solution in place to help inform them as to what those values should be and how they are shifting over time (because in reality those weights are anything but static).
I really wish DSPs did more in the way of thought leadership on how to properly value view-throughs and other ways of measuring contribution as view-throughs are the majority of what they drive, and that metric is coming under increasing scrutiny (as it should).
I'm in advertising too. What I'm really hoping comes out of this is greater transparency in inventory. Give me greater control over my blacklist and I'll be a lot happier.
I've wondered about this before if advertising was just designed to be a big negative sum game for advertisers.
Interestingly if these same bots were capable of making real purchases and were run by ad companies this behavior may be sustainable because some people are making money. Almost like wall street.
Assuming an average consumer will spend a constant number of dollars per year - advertising would indeed be a big negative sum for advertisers. Advertisers would make the same number of dollars, except having had to spend on advertising. The problem is those who do not advertise generally lose out to those who do.
I have seen that. That mostly has to do with boost posts though. We only use custom tailored audiences for our ads and it works quite well. Albeit there's a lot of data scrubbing.
It really goes beyond that. Facebook wants to own your audience and that is fundamentally wrong.
Let's say I spend $100K advertising on FB to get people a my FB page or group. In order to now be able to reach them I have to constantly pay FB through boosts and further ads.
Why? Because they've installed a mechanism through which they limit the reach of every single post you make on your page. And, if a percentage of your likes are fake or non-responsive for various reasons (they liked the picture on your ad, couldn't care less about your content) it is very likely your posts will be to a great degree wasted on a useless audience.
In addition to this they have NOT installed a mechanism through which I can engage with my entire audience. I understand that people don't want their timeline spammed. At the same time, they liked the page. If they liked Tesla it is likely they want to hear from Tesla. Facebook provides no mechanism through which you can do this (message your entire audience) unless you pay them for every single point of contact on every single message/post.
This, to me, is fundamentally wrong. They don't own my customers. I do. A fan page or group owner should not have to pay FB over and over again to reach his or her audience. It's a scam.
In sharp contrast to this, if I spend the same $100K advertising on Google and get my customers/likes/fans to register on my site I can then build a real relationship with them over time. If they don't like what I am saying they can opt out of the emails they receive. If they do, they keep receiving them. And I don't have to spend tens of thousands of dollars every time I need to communicate with them.
A more concrete example is a product we tested on FB. It was aimed at medical professionals. With custom audiences you'd think you can reach a good number of your prospects. That isn't necessarily true. The heuristics they use can deliver anyone who, for example, likes pages with medical information. I am being simplistic. I think you get the point. Still, a good argument could be made that one could do worst outside FB. In terms of targeting a narrow audience this might mostly be true depending on the approach taken.
Reaching medical professionals on FB is expensive. Likes can easily cost you more than a dollar when all is said and done. And, generally speaking, there isn't a 1 to 1 relationship between ad spend and actions or page likes. So, you spend $100K to reach medical professionals and you might get 1,000 to 5,000 legitimate likes. In other words, you could actually spend a very real $20 to $100 per prospect. And then you have to spend money again every single time you need to reach them. We all know that a marketing message must be received multiple times before action is produced. You could very easily spend another $10 per prospect before you get a conversion. That's why I say this is a huge scam. If I spent $20 to $100 per effective member to a page or group, from that point forward they should be my members, prospects or customers, not Facebook's. And extorting money out of me in order to reach all of them with every post is just plain wrong.
The same scenario plays out far better if you avoid FB, use Google and other channels to reach your desired audience and bring people to your own environment for engagement. We've tested this with various audiences and also had plenty of prior experience before advertising on FB. We can consistently produce better results out of direct engagement with an email list as small as 200 people than by spending almost any amount of money on FB.
This is why I don't understand when I see TV ads where companies are subverting their brands to FB by telling people to go to their FB page rather than their own landing pages. They spend millions of dollars creating and airing these ads and then they have to go back and shovel more money at FB to reach the very people they spent so much money to target. That's just insane. More people doesn't mean better results.
In the end I think it is about a basic business formula: Is there a more cost-effective and higher-converting approach to reaching customers and building relationships with them than using FB ads? At the moment my answer to that question is: Yes. Absolutely. I could be wrong but my experiences so far tell me otherwise.
Eventually the common thread between all these sites was identified -- they had retargeting tags from sites like AdRoll: http://www.thesempost.com/adroll-retargeting-bot-attack-behi...
For months now, huge botnets have been producing huge amounts of fake web traffic, picking up retargeting cookies in order to later be served ads and have whoever's behind the network paid for those views. That surge has never ended. It's only grown (including becoming less easy to spot), yet nobody seems to be talking about it anymore, including the retargeting companies.
I'd love to know who's behind it and how they control so many computers.