How to prepare mysql binlog from go

How to prepare mysql binlog from go  
My name is Artem, I work for Rambler Group in the Potok project as a Go lead developer.
We spent a lot of time taming mysql binlog. In this article, a story about how quickly and with a minimum of pitfalls to implement the mechanism of working with the bin on Go.
here or here .
The article has two parts:
1. How to quickly start the processing of records that come to the log.
2. How to tune and expand what's under the hood.

Part 1. We start as soon as possible.

To work with binlog we will use the library ...[/h]
+ 0 -

How to copy data sources to the IDE on the IntelliJ

How to copy data sources to the IDE on the IntelliJplatform.Hello! In this tutorial, consider a few popular scenarios.  - Copying the data source within a single project. - A common data source for different projects within the same IDE. - Copy the data source to another IDE, to another computer, to another galaxy.  export settings . But you do not always want to transfer all the settings, and the data sources - global ones, too, are not always :)  So it will be useful to know that the data source can be copied: on the shortcut menu, selectDatabase tools → Copy data source to clipboard. This will work with several at once.    The clipboard gets the ...
+ 0 -

DevConf 2018 already on Friday, May 18

This Friday, May 1? will be a professional web developer conference DevConf 2018. We invite you to participate. Very briefly about the program:
DevConf :: BackEnd : Aggregate thinking, rewriting the project from Drupal 8 to Yii ? Yii 2.? PHP and frameworks, Service Oriented Architecture, sending notifications, sample profiling, open tracing, Business Intelligence.
DevConf :: FrontEnd : HOC in React, WebRTC: video calls from the browser, Headless Chrome, architecture of the big legacy of the project, JSON API, Framework for individual design, Test it, Node.js and ...
+ 0 -

Go: speed up the selection of large tables from MySQL

I've been using Go to write an ad network for almost a year now. The development I lead on the server Intel i7-770? 16Gb RAM, 256Gb SSD. And in the script that runs once a day, the task appeared to select all the impressions for the past day and to count on this basis statistics for the day at once on several objects (site, campaign, banner).
According to Go's idioms everything is quite trivial:
RawBytes which ensures that the bytes from the database driver will be transferred to the user without copying. What. We will try to extract Scan into the intermediate structure with the sql.RawBytes ...
+ 0 -

DevConf: promising databases for highload

DevConf 2018 already next week! Last year Yury Nasretdinov conducted an interesting survey of promising data storage systems for highload. Video with the report is available at page of the report . And for habra-readers I offer a brief retelling.
DevConf: promising databases for highload In the beginning I'll tell you how to approach the choice of technology for a highload project.
First of all, there must be an understanding of how it works. Not only strong, but also weak points.
Knowing how to monitor and backup. Without good tools for this, this technology is early to use in production.
Sooner or later the systems ...
+ 0 -

Performance of Joomla on large volumes of content

Performance of Joomla on large volumes of content
Joomla very poorly digests the database even with several thousand articles in the _content table. With several tens of thousands of requests to the base of standard modules such as mod_articles_popular can hang for a second.
It's all about the ACL (Access Control List), an access control policy. Verification of the legality of user access to materials takes over 98% of the query execution time.
In the meantime, there are sites that do not need it. For example, a news site showing the "Most Read Articles" module in the left column for everyone. What to do in this case? Disable ACL checking in ...
+ 0 -

Someone did not notice how MySQL 8.0 came out

Unexpectedly for myself, I found out that mysql suddenly released itself the other day (April 19), but there are no articles on the hub - everyone is discussing themselves-you know-what.
I'll try to translate the squeeze out of "What's new". For those who follow the development, there probably will not be anything or almost nothing new, for those interested in time-to-time - may seem interesting. Go tell that in the original post more detailed
the analysis of all the items - here only a short excerpt with references.
+ 0 -

Designing Schemaless of the Uber Engineering data warehouse using MySQL

Designing Schemaless, Uber Engineering's Scalable Datastore Using MySQL

By Jakob Holdgaard Thomsen
January 1? 2016
Designing Schemaless of the Uber Engineering data warehouse using MySQL
Designing Schemaless data warehouse Uber Engineering using MySQL. This is the first part of the three part series of articles on the Schemaless data warehouse.
In Project Mezzanine we have described , as we moved the data about Uber trips from one copy of Postgres to Schemaless - our high-performance and reliable data store. This article describes its architecture, its role in the Uber infrastructure and the ...[/h]
+ 0 -

DevConf: Uber transition from PostgreSQL to MySQL

May 1? 2018 in Digital October will take place DevConf 2018 . And we decided to retell some interesting reports from last year's conference. There was a report with a somewhat holistic title: " " What the political instructor is silent аbout: to the discussion about the transition of Uber from PostgreSQL to MySQL " In it, the developer of MySQL Alexey Kopytov considered the differences between InnoDb and PostgreSQL at the lowest level, including the organization of data, memory and replicas. We bring to your attention a brief retelling of the report.
DevConf: Uber transition from PostgreSQL to MySQL
[h3] ...[/h]
+ 0 -

Why to put the most selective columns in the prefix of the composite index is not always good

tl; dr In this article, we will consider the case where the loop is moved the most selective attribute from the prefix of the composite index to the suffix.
And also consider what is pipeline and how with it select -see the data already sorted.

Why to put the most selective columns in the prefix of the composite index is not always good

Description of the subject area


There is an event log in the X system. You need to make an application to view log data from this system.
It is assumed that the system has its own error codes and the corresponding messages. Of these, about 10k are unique.
There are three types of messages:

  • notice ...
+ 0 -