Where is his button ?! As a simple person to download data from Kibana and Elasticsearch and do not strain the developers

Elasticsearch, Kibana and Logstash (ELK) are an excellent set of tools for collecting and visualizing large amounts of data.
Logs, logs, events - all of this is quite easy to gather, map and displayed in a single tool. Logstash stores data, Elasticsearch stores it, and Kibana displays it in graphical form.
With all the power of this bundle, of course, there are tasks that can not be realized through the built-in capabilities.
For example, Kibana perfectly displays data within a single table (index), but as soon as it comes to combining different indexes into one sample, it helplessly spreads its hands.
And the only way to solve the problem in this case is to download the data from Kibana and merge them into any other tool, for example, in Excel.
A simple example.
Imagine that your Christmas tree (ELK) collects and stores Jira events - for any change to any of the task-tracker tasks.
In this case, in the Elasticsearch index one task will store several records:
Where is his button ?! As a simple person to download data from Kibana and Elasticsearch and do not strain the developers  
If you want to build a schedule of Jira events showing the conversion of tasks by state, then with Kibana you can hardly do this. Because of the features of noSQL, you will need all the events for one task to "drive" into one Elasticsearch record.
It is rather difficult to do this without attracting developers and rewriting Logstash configurations.
The first thing that comes to mind is to download the data from Kibana to manually rotate it in normal Excel.
But the search for the unload button, to your great surprise, will fail, especially if Kibana is corporate and has limitations on the Reporting module:
What turns out, the data is, and they can not be used?
In fact, there is one secret way. Through the standard table report Data Table.
On the "Visualize" tab you can create a lot of different reports, but only in the Data Table there is a button for uploading data in the csv format.
If you work with Kibana, then you probably perceive the Data Table as an aggregate, i.e. A table in which you can calculate the total amount of anything, but you can not list all the records. However, this is not quite true.
The table has a remarkable function - Unique Count (count the number of unique elements), which you can use to display the entire list of records in the table.
Click "+" in the Visualize tab, select "Data Table" and specify the index from which to upload. In the parameters of the table, specify the aggregate - "Unique Count" and just below the field with a unique ID.
If you start the recalculation, Kibana will display the number of unique records in the index.
How to convert one number into a list of records? Very simply - with the help of the button "Split Rows"
Click on it and select "Terms" as an aggregate (breakdown by field values).
This is the key point: in the "Field", specify a field with a unique ID of records.
And, oh miracle, one number when converting a table magically becomes a table!
Here it is, the moment of truth.
What did you do? If in a nutshell, then you told Kibana that you want to count the number of unique records and also indicated the grouping of records by the characteristic, namely by the unique values ​​of the field you specified.
Now Kibana counts separately the number of elements in each group, and since As values ​​you specified a unique record ID, Kibana began counting the number of unique records grouped by a unique ID, i.e. in fact, next to each row of the table, the "Unique Count" metric will equal "1".
You got the desired result - output all the records and additionally counted how many lines in your index with the same ID.
Now, if the column in the table is not enough, add additional fields with the "Add sub-buckets" button and then "Split Rows".
Everywhere, select "Terms" as an aggregate and specify the names of the fields that you want to add to the table.
Done. You got a full unloading index.
All that's left to do is save the visualization and click the "Export Raw" or "Export Formatted" button.
"Raw" unload everything in its raw form, and "Formatted" reformats the data according to the Kibana locale.
As you can see, it's possible to do unloading from Kibana to a simple person who is not familiar with programming, although the solution does not lie on the surface.
I hope that with this little trick I will make it easier for someone to live and give the opportunity to quickly analyze the data without attracting developers for unloading.
If my experience has helped you, I'll be happy to see your comment on this article.
+ 0 -

Add comment