How to defeat the gluing in J. Direct and AdWords for 600 thousand rubles per month

Over the past six months, we have succeeded in defeating the "clinking" of our contextual advertising with a budget of 1 million rubles a month.
 
 
The key to victory over Frode was the minute monitoring of traffic with notifications of abnormal changes and disabling of bad API ads, and a number of reports that reflect the situation in real time.
 
 
How to defeat the gluing in J. Direct and AdWords for 600 thousand rubles per month
 
Figure 1. Diagram of the number of visitors for keywords on dekynutam
 
 


How do you know if you are being attacked?


 
One of the first signs of "gluing" advertising will be an increase in the percentage of return funds for fraud in Direct and AdWords.
 
"In Yandex Direct, the costs for fraud are automatically returned to the balance of the advertising campaign. The number of clicks that have been filtered out by the fraud protection system is displayed in the reports "statistics by day" "general statistics" in the line "invalid clicks for the entire selected period."
- certificate of the Directive "invalid clicks" .
 
 
In AdWords, you can enable the "invalid clicks" level display on the "columns" tab:
 
 

 
Figure 2. Customized columns with the level of "invalid clicks" in AdWords
 
 
In our case, with an average level of " invalid clicks "In the Direction ≈ 10%, Yandex suddenly began to return 40% of the advertising budget, and in a month and at all 54%.
 
discussion of the level of fraud in contextual advertising in the official Direct Club, to be sure: many advertisers lose money because of fraud.
 
 
Google officially recognizes the errors of its protection system and provides advertisers with "Refund Claims" (return of the spent budget). According to the statistics of the ClickSease service, Google Ads on average returns 12% of the advertising budget.
 
 
How to return some of the money for a "slip" through AdWords [/b]
You need to send complaint about "invalid clicks" in Адвордс, after consideration of which you will refund ≈ 12% of the funds spent. Also, the ClickSease service automatically sends similar refund requests to Google every 2 months.
 
 
In our case, Google AdWords first recognized 18% of our traffic as " invalid "And returned the money for them, and when we sent a complaint of" refund claims ", Google refunded another 13% of the budget.
 
 
Yandex does not recognize the vulnerability of its own protection filters and complaints about obvious cases of fraud, sends a template response that the problem lies in the advertiser and his site.
 
 
In our case, the percentage of "invalid clicks" in the report of J. Direct never rose above the coveted 50% for any of the advertising campaigns, even on the days of the most violent outbursts, when 80% of the budget "merged" into the usually little popular ad group without calls and applications.
 
 

 
Figure 4. The level of "invalid clicks" in one of the campaigns in Direct
 
 


With what level of attack we encountered


 
Fraudulent traffic was directed to four areas of business in two cities. When connecting new advertising campaigns or starting long stopped, within a few hours "gluing" was redistributed to them.
 
 
Click Fraud was not tied to the time, and redistribution of the budget, for example at night, did not have any effect: we still received our traffic of left traffic. "Slicing" was equally active in the RSA, Yandex search, and the Google Display Network.
 
 
On the site, the bots simulated user behavior, navigated through sections of the site, selected text, naturally scrolled and moved the cursor.
 
 


Are there any remedies available?


 
All anti-click protection services have only one direct struggle tool: blocking suspicious IP addresses or placements in Y.Direct and AdWords campaigns.
 
 
In the event that dynamic IP is used against you, any anti-click service Frodom will always be one step behind the scammers in blocking them by IP: the bot will already make a few clicks on your advertisement by the time the service brings this IP to the black list and stop the display of advertising on it. In addition, after the fraudulent software can not see your advertisement, it will simply change the IP address, Hardware-ID and continue its actions.
 
 
When attacking ads on the GDN or RSA, various placements are usually used, and automatic defense systems can not detect suspicious sites for blocking.
 
 
Let's return to blocking by IP - here we come to the most interesting - if you can block up to 500 IP addresses in the AdWords, then in Yandex Direct you can block only 25 unique IP addresses for one ad campaign! Such a small black list of IP addresses is no longer relevant, since now you can safely purchase 500 IPv4 addresses for 10 thousand rubles and bypass this restriction.
 
 
Protect from the "clinking", performed at a high level, you can only two ways:
 
 
 
learn not to show ads to fraudulent users or bots, for which you need to find certain "patterns" in their behavior and characteristics;
 
 
temporarily stop specific ad groups and keywords that are attacked.
 
 
 


Cut off some of the audience to save it most of the


 
If you learn not to show click advertising to bots or fraudulent users, then they will not be able to do harm.
 
 
It is always possible to track similar behavior patterns and patterns, for example, that fraud usually goes on Windows 7 from 5:00 to 9:00 in Moscow, and set a bid adjustment of -100% for a similar audience in all attacked advertising campaigns. Functional bid adjustments in AdWords is quite extensive, which can not be said for adjustments in Yandex Direct .
 
 

We are looking for patterns in fraud through security services


 
In order to have an idea of ​​how exactly we are being attacked, and manually track the patterns in the fraudulent traffic, we connected the Russian protection service against fraud ClickFrog . The product is long known to all, popular in the CPA environment and so on.
 
 
ClickFrog quickly proved the total incapacity:
 
 
 
per day allocated no more than 40 suspicious IP addresses, with traffic from Direct to ?000 thousand conversions per day, and even recognized by Yandex 1300 "left" clicks per day;
 
 
the main tool for protecting the service is blocking by IP address, the command about which is sent to Y.Direct by API, but once the black list of 25 IP addresses is filled in, it is necessary to manually delete the last few IPs in each advertising campaign and wait for the next filling of the list, and so in a circle.
 
 
 
Then we installed the code of the American service ClickSease , aimed at AdWords, and not yet working with the Directive. The service, by the way, has, in contrast to ClickFrog, a free test period for 2 weeks.
 
 
ClickSease proved to be more useful: it began to catch 300-400 unique fraudulent IPs per day. For each blocked IP service gives statistics:
 
 
 
Internet provider;
 
the site from which the transition was made;
 
operating system;
 
unique device ID;
 
time of the first and last transition;
 
region.
 
 
From the ClickSease report, we were able to identify patterns in fraud:
 
 
in 81% of cases the device simulates a mobile OS: Android or iOS;
 
in 59% of cases the geolocation of the IP address does not apply to Moscow, with fraud directed to Moscow.
 
 

We are looking for patterns in frod by hand


 
However, even such obvious patterns were not enough to reduce harm from fraud, and to disable advertising on the mobile did not want to. Services are usually only able to give ideas for identifying similar patterns in fraud, and then it is necessary to detect fraud in the Metric (in case of an attack on Direct) and separate it into separate segments of the Yandex Auditorium for subsequent analysis and blocking.
 
 

 
Figure 5. An example of the analysis of traffic by age group in Metrics for the search for patterns of fraud
 
 
Sections of traffic that will help determine the patterns of fraud:
 
 
audience dynamics by age groups;
 
dynamics of long-term interests of users;
 
device dynamics and OS.
 
 
In the case of AdWords, the counter mechanics are clear:
 
 
 
determine the segment of the audience "infected" by Frodo;
 
we set the bid adjustment -100% for the selected segment;
 
we monitor the change in indicators: conversion, time on the site, depth of view, bounce rate.
 
 
In Yandex Direct the mechanics of the fight is more complicated and is divided into two versions:
 
 
a) you managed to find an obvious fraud pattern related to sex, age or mobility:
 
 
 
we set the adjustment of the rate -50% or -100% for the selected segment;
 
monitor the change in key indicators.
 
 
b) there are no obvious patterns:
 
 
 
We allocate frood traffic to a separate segment Yandex.Auditorium (for example, you knew for sure that from October 1 to October 20 there could not be ?000 conversions for an ad group, which was always no more than 30 visits per day)
 
through look-alike Yandex We create a segment similar to our users;
 
We set a bid adjustment of -100% for the manually generated audience segment;
 
carefully test the lowering of advertising rates for the segments created by Yandex.
 
 


We construct the diagrams that show the


 
Frod always generates obvious foci and peaks , whether it be an abstruse software with an imitation of the behavior of the real user or a group of freelancers who carry out the technical assignment.
 
 

 
Figure 6. Diagram of the number of visitors for keywords by dekynutam
 
 
Frod occurs unevenly for several reasons:
 
 
to make the attack "smoothed" you need to own confidential information and know who, when and how many conversions commits on your advertising;
 
software acts jerky, and on the minute, 10 minutes, and sometimes on the hourly chart, its actions will be evident;
 
even if "schoolchildren" work against bulletin boards against you, then they act on a specific task with the algorithm, and the anomalies generated by them will be easily tracked.
 
 
If you learn to quickly find and eliminate outbreaks, you can significantly reduce the harm from fraud. In our case, an obvious sign was an abnormal increase in the number of conversions on contextual advertising in a specific 10 minutes or one minute for some keywords.
 
 
For visualization is best suited Google Data Studio , since it is only Analytics that is able to correctly collect data broken down by time for 1 and 10 minutes, and Metric, when generating reports on decimuts, gives incorrect results.
 
 
How to build charts by 10 minutes, and not by the hour, in Google Data Studio [/b]
By default, in Analytics or Data Studio, you can not build charts in minutes or by 10 minutes, but you can do this as follows in Studio Date:
 
 
Step 1. Fromcover editing fields
 
 

 
 
Step 2. Create copies of the following fields: Year, Month of the Year, Day of the Month, Hour, Minute, and call them, for example, Year (number), Month of the year (number) and so on. Also in the copied fields, you must change the Type from the time and date format to "number" as shown in the figure.
 
 

 
Step 2. Change the type of the copied field from "date" to "number"
 
 
Step 3. Create a new field in which we write the following formula: Year (number) * 10000000 + Month of the year (number) * 100000 + Day of the month (number) * 1000 + Hour (number) * 10 + FLOOR (Minute (number) /10)
 
 

 
Step 3. Create a calculated field "Time for 10 minutes"
 
 
Step 4. Save the created field, then go back to the list of all fields and find our new field "Time for 10 minutes (decamute)". It is necessary to change its type from "Number" to "Date and time" as shown in the figure, and then back to assign this field the type "Number".
 
 

 
Step 4. Create the calculated field "Time for 10 minutes"
 
 
Step 5. Create a "Combined chart" and set our new field "Time for 10 minutes" as a parameter, as shown in the figure. Done.
 
 

 
Step 5. Create a "combined chart"
 
 
 


We set up notifications on the fronts of fraud


 
In order not to follow all cases of fraud manually, I made a report in Google Sheets, which updates the data every minute and notifies the beginning of fraud.
 
 
Google Sheets support Core Reporting API , which can be accessed through the "Script Editor" in the Tables.
 
 
Step 1. Go to the script editor to access Analytics
 
 

 
Figure 7. Script editor for accessing the Analytics Core Reporting API via Google Tables
 
 
Step 2. Write an API request to Analytics to get data about the desired metrics (for example, the number of users who have switched over paid advertising, every minute of the day, as in our case).
 
 
Google Script code for querying any data from Analytics in Google Sheets [/b]
function runDemo () {
try {
var firstProfile = getFirstProfile ();
var results = getReportDataForProfile (firstProfile);
outputToSpreadsheet (results);
} catch (error) {
Browser.msgBox (error.message);
}
}
function getFirstProfile () {
var accounts = Analytics.Management.Accounts.list ();
if (accounts.getItems ()) {
var firstAccountId = accounts.getItems ()[0].getId ();
var webProperties = Analytics.Management.Webproperties.list (firstAccountId);
if (webProperties.getItems ()) {
var firstWebPropertyId = webProperties.getItems ()[0].getId ();
var profiles = Analytics.Management.Profiles.list (firstAccountId, firstWebPropertyId);
if (profiles.getItems ()) {
var firstProfile = profiles.getItems ()[0];
return firstProfile;
} else {
throw new Error ('No views (profiles) found.');
}
} else {
throw new Error ('No webproperties found.');
}
} else {
throw new Error ('No accounts found.');
}
}
function getReportDataForProfile (firstProfile) {
var profileId = firstProfile.getId ();
var tableId = 'ga:' + profileId;
var startDate = "today"; //for example getLastNdays (14) equals 2 weeks (a fortnight) ago.
var endDate = "today"; //getLastNdays (0) is equal to Today.
var optArgs = {
'dimensions': 'ga: date, ga: hour, ga: minute, ga: sourceMedium', //Comma separated list of dimensions.
'sort': 'ga: date, ga: hour, ga: minute', //Sort by sessions, descending, then keyword.
//'segment': 'dynamic :: ga: isMobile == Yes', //Process only mobile traffic.
'filters': 'ga: sourceMedium == yandex /cpc',
'start-index': '1',
'max-results': '10000' //Display the first 250 results.
};};
//Make a request to the API.
var results = Analytics.Data.Ga.get (
tableId, //Table id (format ga: xxxxxx) .
.startDate, //Start-date (format yyyy-MM-dd) .
.endDate, //End-date (format yyyy-MM-dd) .
'Ga: users', //Comma seperated list of metrics .
OptArgs);
if (results.getRows ()) {
return results;
} else {
throw new Error ('No views (profiles) found');
}
}
function getLastNdays (nDaysAgo) {
var today = new Date ();
var before = new Date ();
before.setDate (today.getDate () - nDaysAgo);
return Utilities.formatDate (before, 'GMT', 'yyyy-MM-dd');
}
function outputToSpreadsheet (results) {
var sheets = SpreadsheetApp.getActiveSpreadsheet ();
var sheet = sheets.getSheetByName ("coeff1");
var range = sheet.getRange ('A: E');
range.clear ();
//Print the headers.
var headerNames =[];
for (var i = ? header; header = results.getColumnHeaders ()[i]; ++ i) {
headerNames.push (header.getName ());
}
sheet.getRange (? ? ? headerNames.length)
.setValues ​​([headerNames]);
//Print the rows of data.
sheet.getRange (? ? results.getRows (). length, headerNames.length)
.setValues ​​(results.getRows ());
}

 
 
 
Step 3. Set the trigger to update the data every minute:
 
 

 
Figure 8. Requesting fresh data every minute for rapid response to fraud
 
 
Step 4. Create a pivot table from the sheet updated by the desired data once a minute, and analyze these metrics to configure triggers for e-mail notifications or disable ad groups for the Ya.Direkt API or AdWords.
 
 

Figure 9. Example of setting formulas for notifications about anomalies
 
 
An example of my Google Script code for sending notifications to e-mail [/b]
function myFunction () {
var ss = SpreadsheetApp.getActiveSpreadsheet ();
var sheet = ss.getSheetByName ("notification");
var range = sheet.getRange ("D2: E4");
//The row and column are the relative to the range
//getCell (?1) in this code returns the cell at B? B2
var cell = range.getCell (? 2);
Logger.log (cell.getValue ());
if (cell.getValue ()! == "no") {
MailApp.sendEmail ("[email protected]", "Fraud notification" + cell.getValue (), "Check me" + range.getCell (? 1) .getValue ());
}
else {
}
var cell2 = range.getCell (? 2);
Logger.log (cell2.getValue ());
if (cell2.getValue ()! == "no") {
MailApp.sendEmail ("[email protected]", "Fraud notification" + cell2.getValue (), "Check me" + range.getCell (? 1) .getValue ());
}
else {
}
var cell3 = range.getCell (? 2);
Logger.log (cell3.getValue ());
if (cell3.getValue ()! == "no") {
MailApp.sendEmail ("[email protected]", "Fraud notification" + cell3.getValue (), "Check me" + range.getCell (? 1) .getValue ());
}
else {
}
}

 
 


Results: how to defeat the glue


 
Counteraction click Frodu can be divided into three groups:
 
 
a) Preemptive actions:
 
 
disabling of "contaminated sites";
 
disabling the display of advertising for the audience with frodovymi for you signs, for example, for people on tablets from St. Petersburg (more complex parameters for blocking can be used through AdWords lists and Metrics segments);
 
adjustment of rates for audience segments similar to fraud segments ("look-alike" segments are created in Ya. Audiences and Google lists);
 
blocking fraud by masks of IP networks (available only in AdWords).
 
 
b) Preventive actions:
 
 
Sending complaints about the return of the budget to AdWords and Direct;
 
an investigation "who ordered an attack on you";
 
grouping suspicious and often attacked ad groups into a single advertising campaign;
 
"Traps" for the simplest of bots, namely the hidden buttons on the site, which are visible only to the bot and when clicked on which it falls into the list.
 
 
c. Actions "post factum":
 
 
blocking by IP addresses;
 
Online disconnection of flickers: keywords, ad groups, advertising campaigns, audience segments.
 
 

 
Figure 10. Methods of protection against click fraud
 
 
Useful links:
 

 
 
Library Core Reporting API;
 
Query Explorer for easy API APIs;
 
We write scripts to automate the work with Google applications ;
 
Automate Google Sheets .
 
Analytics Intelligence to ask bot Analytics: - "Was there anomalies in paid traffic for the last year?" - and get a clear answer;
 
 
How to find out who ordered the attack on your advertisement [/b]
Any adequate competitor will minimize his damage when attackingon the other:
 
 
 
Firstly, the attacker will try not to show his advertising in the directions that are being attacked at the moment, so as not to merge your CTR and increase your CPC;
 
 
secondly, an unfair competitor will pick up such keywords for an attack on which he can stop showing his advertisement without much harm to himself.
 
 
 
In our case, the competitor also started to click fraud in 4 directions in two cities, so it was not difficult to calculate it.
 
 
To make it easier to analyze competitors with whom you cross, you can watch all included competitor announcements for each keyword in the Directive:
 
 

 
Figure 9. All announcements of competitors by the keyword
 
 
 
Who also encountered the cliques of contextual advertising - write in the comments, try to help each other!
+ 0 -

Add comment