The experience of blocking Internet advertising in a local network company
Hello, Khabarovsk citizens! Having received from the management the task to block employees from advertising on Internet sites, I decided to approach the process creatively: to understand how one or another advertisement is technically broadcast, what types of locking exist, their pros and cons. I also give an example of the implementation of the lock at the gateway level, using Traffic Inspector Next Generation, as well as the use of the options for blocking advertisements in Kaspersky Internet Security (KIS) and browser applications AdBlock and Adblock Plus for these purposes. The article will be interesting to sysadmins and other specialists working with local networks.
First in a nutshell, how can block advertising .
Firstly, there are applications for browsers, which use dozens of different algorithms and schemes for each type of content and even for some single sites (Facebook, for example). The most popular cutters are AdBlock and Adblock Plus (ADP). In my opinion, ADP is more effective of them, but under its powerful skating rink sometimes useful content also comes (about it - in the end). For me, the drawback of all such solutions is that they are placed on each computer and, moreover, on each browser. When you have hundreds of computers in your computer, this becomes a problem.
Secondly, in many antiviruses (for example, we use Kaspersky) there is a built-in option of blocking advertising. The advantage of this method is that since you are using an antivirus, you do not need to add anything to the computer - just activate the function in the menu. But since this function is not the main one, it works worse than specially applied applications.
Thirdly, it is possible to cut advertising by a network gateway. The blocking occurs due to redirection of HTTP requests of advertising sites to the address ???.0. For me, the huge plus is that the lock works right away for the entire local network. The main drawback - you need to monitor the relevance of the base advertising addresses and, if necessary, add them manually. The second drawback is that only one locking mechanism is used - by HTTP requests, and not all types of advertisements use them.
Looking ahead, I will say that in my local computer I use all three methods - they complement each other well. Details - at the end of the article, and now let's figure out what are types of advertising .
SEO and all sorts of email-mailings are not interesting for us now. We are interested in advertising, which is broadcast along with useful content. On this principle, I divided it into the following types:
1. Embedded ad units such services as "Advertising network" Yandex "and Google.Adsense.
These are text, text-graphic modules and simply banners, which are placed by the owners of the sites for earnings. In my opinion, this is the most harmful type of advertising, because it: a) often mimics the useful content and mixes with it to increase the clickability; b) shows the user what he is most likely to be interested in (all we know about big data, behavioral factor, etc., etc.); c) shows that the user (in our case - the employee) at the moment does not need, that is, distracts from work.
Let's take a few resources with this type of advertising:
Google's banner was neatly cut out by means of KIS - the layout did not suffer, no voids were formed:
1.2. Also come blocks that use the user's requests, which he entered earlier in the search engine (for example, on .zmoe.ru ):
1.3. www.bestfree.ru - a perfect example of a re-optimized SEO-site, expectedly hammered advertising to the outset. Red boxes indicate the ad units (see the picture on the left). All of them were blocked (see right):
1.4. Unsuccessful example of blocking advertising - mail Mail.ru. On the page of incoming letters "Advertising system" Yandex "manifests itself twice: an information line above the list of letters and an ad unit on the left. None of the inserts was blocked by any of the methods I used:
2. Banner exchange, teaser, RTB-network such as Rotaban, advmaker.net, Marketgid, etc.
Personalized advertising, like Google or Yandex, they can not show, but take their intransigence.
2.1. Banner-streaming on adindex.ru (advertising network Adriver) was cut by the browser application AdBlock:
Note that AdBlock cuts the entire div and iframe, so that even a white rectangle does not remain.
2.2. Here is an example of another block of the AdIndex network mimicking for content, which was also coolly cut out by the browser plugin:
3. Pop-up windows
The most violent advertising format, especially when a pop-up window does not allow to enter the site or close some content. From sites with pop-ups, I did not find anything decent for the test, so I used trash megashara.com. Browser applications coped with the task perfectly. But with the network gateway it's not so easy. On the one hand, the megashara sends HTTP requests that theoretically can block pop-ups, but, on the other hand, it constantly issues mirrors with different URLs, so in practice this can not be done. And if pop-up windows work through jаvascript, then at the gateway level, blocking them is completely impossible. (However, megashara.com in my local lock is blocked entirely.)
4. Affiliate Programs
Usually they are distributed by large online stores and lodogeneration services. For example, if you are the owner of a portal about a book, then you can use the "Labyrinth" affiliate and post their code on your website - from every transition the online store will pay you money. So, for example, the site entered. marketopedia.ru . Here's what the partner module looks like:
Such a seemingly simple thing could not be overcome because of the fact that this farm works on jаvascript. However, this type of partnering is rare, and as a rule, looks non-aggressive - you can give up on it.
But the affiliate from Amazon at sharpologist.com was successfully blocked by browser applications:
5. Advertising on YouTube.
First of all, these are videos that preview the main video, videos that stop browsing, and advertisements popping up on top of the video.
I took the the first caught roller is - in the beginning the advertising video popped up:
The advertising video was going to broadcast for a whole minute; it's good that you could skip it in 10 seconds. Then I saw a full mincemeat: both the banner and the built-in commercials:
I launch ADP - and voila, there is nothing:
6. Embedded banners are
From the point of view of blocking, this is the most difficult case, since the blocker, in fact, needs to be banned from the picture. How among all the pictures on the page to find the advertising, without affecting the rest? In browser applications there is an analysis of their sizes, because banners, as a rule, have standard sizes. Obviously, such a method does not work if the banner is of non-standard proportions.
6.1. To begin with, I took cnews.ru, that's how it is with him:
As you can see, AdBlock successfully cut all banners, including advertising of cnews.ru itself :)
6.2. But the less successful example is the partner banner on the site lezvie.info:
It is not blocked by any of our filters. We are looking at the code:
Dimensions 300x416 is a non-standard format (in any case, in the list standard sizes Google.Adsense It is not), so the banner was predictably left in its place.
7. Contextual advertising in the search
In my opinion, this is the most harmless kind of advertising, which I would not block at all. If a person searches for something in the search engine and is shown relevant ads, there is nothing wrong with that. On the contrary, they help to find what is needed, especially if the requests are commercial.
For the sake of interest, I looked at whether contextual ads block browser applications:
As you can see, ads and banners have been blocked, but there are proposals from Yandex.Market.
8. Advertising in social networks
I will not develop this topic in this article, as I believe (and the management agrees with me) that social networks in general should be blocked in the workplace.
What better blocks?
The most effective way was blocking the browser application, and ADP proved to be better than AdBlock. But there are three points that greatly embarrass me.
The problem of ADP # 1 is It cuts everything, including useful content. Take, for example, the site vedomosti.ru - it has not only purely advertising, but also semi-advertising banners with announcements of events that can be useful to the reader. And there are completely harmless link banners. Everything is cut to zero.
The ADP icon in the browser's header shows how many banners you cut out on the plug-in on this page, but what's the use of it? The feeling that you do not see something important, blocked by mistake, does not leave.
Problem ADP №2 Collided with sites that ask to disable ad blocking before they show content. And it's not some sloppy resources, but sites like forbes.com. WordPress even has a Block Adblock plugin that blocks content for those who use Adblock.
Of course, there is no such problem with the lock-blocker.
The problem of ADP # 3 is Plug-ins must be installed on each computer and on each browser. And it is not enough to install - then you will have to run after each request: someone has something useful blocked, someone does not open the site. Do I need it?
Blocking with antivirus works worse than ADP, but better than forwarding HTTP requests at the gateway level. There is nothing more to say here.
Blocking the network gateway , as I said, is due to the redirection of requests directed to sites with advertising, to the address ???.0. I used the piece of hardware. Traffic Inspector Next Generation S100 , working on OPNsense.
I'll describe in a nutshell how I set up the lock (instruction here ). Using ssh, I went to the piece of hardware and installed the curl package (https://curl.haxx.se/) with the command: pkg install curl.
Then I launched the initialization script for the list of blocked web resources update-hosts.sh . After forming the list, I launched the Unbound DNS function for Traffic Inspector Next Generation, I selected "Services" -> "Unbound DNS" -> "General settings" in the web interface. Enabled DNS-converter, forwarding of DNS-queries, registration of DHCP. In the user settings, specified the list of blocked Web resources /var/unbound/ad-blacklist.conf. Retained the settings. Now the piece of hardware has become a DNS server.
Then I installed the new DNS server as a DHCP server: "Services" -> "DHCP v4" -> "LAN" -> "Enable DHCP server on the LAN interface". On users' computers DHCP-servers should be specified by Traffic Inspector Next Generation, then the piece of hardware will automatically be registered as a DNS-server:
Now in the case of redirecting to pages with ads, its blocking occursovka.
The list of blocked resources by default is pumped from file http://winhelp2002.mvps.org/hosts.txt . But I have many questions for this list. For example, why does it by default lack addresses that use "Yandex" and Google in their advertising networks? And this means that the most intrusive type of advertising (see point 1) will not be blocked. I had to prescribe them manually. In order to add a new address to block, you need to add the address in the configuration file to the format: server: local-dаta: " <адрес блокируемого ресурса> A ???.0 "and restart the Unbound DNS service.
Instead of the resume
My recipe for blocking advertising on a local network is simple:
1. Activate the lock at the gateway level.
2. We register the ad networks "Yandex" and Google (or make sure that they are present) in the blocked addresses, as well as other addresses that you want.
3. We inform employees that, on an individual request, they can be set up an advertising blocker, and install it to those who really need it.
4. We block at the gateway level social networks, job search sites and other non-targeted resources, and then the problem of blocking advertising on them will disappear by itself :)
P.S. As I wrote above, the list of blocked HTTP requests in the Traffic Inspector Next Generation is pulled from the file winhelp2002.mvps.org/hosts.txt , but this list is not as complete as it could be. If among readers there will be prosharennye experts who know where to take more actual lists (especially taking into account the specifics of the Runet), please write in the comments. If there are such lists, I will write them in the script. By the way, I was told in technical support that OPNsense is discussing adding a special plug-in to manage such lists, but this is not yet in production.
It may be interesting
Situs QQ Online
Situs QQ Online