How fast is AMP really?

The project Accelerated Mobile Pages (AMP) from Google caused a certain commotion for ideological reasons, but the technology itself was not disassembled in detail. A few weeks ago Ferdy Kristant wrote about an unfair advantage that the AMP content receives by preloading . This made me think: how well AMP works in fact ? I've seen tests like that of Ferdie, when one or two pages were compared, but I have not yet come across comprehensive objective tests.
 
 
Evaluating the performance of AMP is not really as easy as it sounds. It is necessary to consider at least four contexts:
 
 
 
How well does AMP work in the context of Google search?
 
How well does the AMP library work as an autonomous framework?
 
How well does AMP work when booting from the cache?
 
How well does AMP work compared to the canonical document?
 
WebPageTest by emulating 3G connection on Nexus 5. Each page was built through AMP, and also downloaded from the source server .
 
 
AMP consists of three main parts:
 
 
 
AMP HTML
 
AMP JS
 
AMP cache
 
 
When we talk about the AMP library, we mean AMP JS and AMP HTML together. AMP HTML is both a subset of HTML (there are limitations that you can and can not use), and an addition to it (AMP HTML includes a number of custom AMP components and properties). AMP JS is the library that gives you these custom elements, and also produces various optimizations for documents based on AMP. Because HTML, CSS, and JS are at the core, you can completely build a document using the AMP library without using the Google AMP cache.
 
 
The AMP library should provide a certain level of consistency in terms of performance. For the most part, she cope.
 
 
Most test pages are loaded within a reasonable range from each other. But at both ends of the spectrum there were some deviations: the minimum values ​​turned out to be rather low, and the maximum values ​​were frighteningly large.
 
 
 
 
Metric
 
Min.
 
Max.
 
Median
 
90th percentile
 
 
 
The beginning of the rendering is
 
1765 msec
 
8130 msec
 
4617 msec
 
5788 ms
 
 
 
Visually full view of
 
4604 ms
 
??? msec
 
7475 msec
 
??? ms
 
 
 
The index of speed is
 
3729
 
???r3r3590.  
6171
 
???r3r3590.  
 
 
The size is
 
273 KB
 
??? KB
 
905 KB
 
??? KB
 
 
 
Inquiries
 
14
 
308
 
61
 
151
 
 
 
In general, AMP performance is relatively predictable. Nevertheless, the figures also indicate that AMP support does not give a 100% guarantee that the page will be quick or easy. As with any another technology, there is the possibility to create an AMP document that will turn out to be slow and heavy.
 
 
Any statement that AMP provides a certain level of performance depends on the attitude to the extremes and what you consider "performance". If you are trying to build an entire site based on AMP, it should be understood: although it is unlikely to get too bloated, it will not be immediately loaded with lightning speed. This will require some work.
 
 
At least this applies to the library itself. Perhaps the AMP cache will show a significant increase in speed?
 
 
Pete Meenan did me a favor and shared a script for WebPagetest, which pre-warms up connections to Google CDN so that the result is more like what you see in reality.
 
 
logdata 0
 
navigate https://cdn.ampproject.org/c/www.webpagetest.org/amp.html
 
logdata 1
 
navigate% URL%

 
 
When loading from the cache, the AMP pages showed a marked improvement in performance across all metrics.
 
 
Metric Min. Max. Median 90th percentile
The beginning of the rendering is 1427 msec 4828 ms 1933 msec 2291 ms
Visually full view of 2036 msec ??? msec 4924 msec ??? ms
The index of speed is 1966 ???r3r3590.  
3277 9004
The size is 177 KB ??? KB 775 KB ?079 KB
Inquiries 13 305 53 218

 
In general, the advantages of the cache are quite significant. Although the maximum values ​​are still depressing (the slightly increased maximum is explained, basically, by the difference in banners in different tests). But the average median, which corresponds to most AMP documents, has become better for all indicators.
 
 
The improvement is not surprising given the various performance optimizations that CDN automatically performs, including:
 
 
  •  
  • Caching of images and fonts  
  • Limiting the maximum image size  
  • Compression images on the fly, as well as generating options of different sizes with the addition of srcset for them  
  • Using HTTP /2 and HTTPS  
  • Clearing of HTML-comments  
  • Automatic inclusion of prompts for resources, such as dns-prefetch and preconnect  

 
Once again it is worth noting that none of these optimizations are does not require the use of AMP. The same is done by most of the major CDN providers. You can automate all these optimizations yourself during the build process.
 
 
I'm not saying that you need to abandon the Google cache, just keep in mind that you can and should use these optimization methods regardless of whether you use AMP or not. In the AMP itself or even the AMP cache there is nothing unique.
 
 

How well does AMP work compared to the canonical document?


 
We saw that the AMP library itself only gives a small performance boost, but the cache with all the optimizations displays the download speed to a new level.
 
 
One of the arguments in favor of AMP is that it does not need to be an "expert" to make a productive website. Although I would argue that many of the sites we've seen can be called "productive", but it makes sense to compare AMP documents with their canonical equivalents.
 
 
For the next round of testing, I took the canonical version of each page and checked it under the same conditions. It turned out that most often AMP-documents outperform their equivalents without AMP, which indicates the lack of optimization of the latter.
 
 
Metric Min. Max. Median 90th percentile
The beginning of the rendering is 1763 ms 7469 msec 4227 msec 6298 msec
Visually full view of 4231 ms ??? msec ??? ms ??? ms
The index of speed is 3332 ???r3r3590.  
8152 ???r3r3590.  
The size is 251 KB ??? KB 2762 KB 5229 KB
Inquiries 24 1743 318 647

 
Forget about Google cache for a second and compare the AMP library again with the canonical pages.
 
 
In the metrics "Start of loading" and "Index of speed", there is not much benefit from the library. In fact, the beginning of the rendering in AMP documents is even slower than .
 
 
This is not too surprising. As mentioned above, AMP documents use the AMP JS library for many optimizations and resource downloads. When you rely on this jаvascript to display the page, you immediately get a hit on the renderings. This is observed until the AMP cache enters the game, which returns normal values ​​for the start of the rendering and the speed index.
 
 
However, for other indicators, AMP clearly outperforms the canonical version.
 
 

Increase productivity. but for whom?


 
The verdict on the effectiveness of AMP is a bit ambiguous. On the one hand, for all peers, the use of AMP does not necessarily lead to faster page loading. There is no guarantee that the AMP will not slow down or catch your data.
 
 
On the other hand, AMP documents are usually faster than the original versions. Better load distribution in AMP significantly reduces overhead. Suddenly, publishers who like to include third-party scripts on canonical pages, dramatically reduce (at least, they are forced to do so) the number of third-party scripts for AMP pages.
 
 
The biggest advantage of AMP is not the library, because its functions can be implemented independently. This is not a cache - many of its optimizations are implemented through a good build script, and they all have any worthy CDN provider. This does not mean that there are no really correct things in the AMP JS library or cache - they are there. It's just that they do not give the biggest difference in terms of performance.
 
 
The main advantage of AMP is the limitations on how much content can be squeezed onto one page.
 
 
For example, here is a graph showing all the queries for the same page written under AMP requirements (right), compared to the canonical version (on the left). I apologize for the need to scroll.
 
 
How fast is AMP really?
 
Compare download schedules for the canonical version of the article (left) and AMP version (right). Limitations of AMP minimize the number of requests
 
 
The size of the pages in the 90th percentile for the canonical version is 5229 KB. The size in the 90th percentile for AMP documents from the same source is 1553 KB, the difference is about 70%. The number of requests in the 90th percentile for the canonical versions of the pages is 64? for documents AMP is 151. The reduction is almost 77%.
 
 
Limitations of AMP really reduce the size of pages and remove from them extraneous stuff. Publishers are ready to make this assignment for the content distribution services from Google, but they do not dare to do it for canonical versions of the pages.
 
 
If we evaluate AMP from the point of view of network acceleration, then the evidence is not particularly convincing. All sites have the AMP version running in addition to to the version without AMP.
 
 
Absolutely. Everyone has. Sites.
 
 
And usually these versions without AMP are heavy and slow. If you read the news on these sites and do not specifically click on the AMP button, then AMP does not improve the loading of the pages. AMP does not solve the main problem; he just hides it a bit.
 
 
Time will tell if anything will change. Perhaps, as at the beginning of the transition from mobile sites http: //m. to adaptive sites, publishers are slowly harnessing. But right now it seems that calls for switching to AMP perform exactly what you think: they promote AMP, not performance.
+ 0 -

Add comment