Labeling machine for labels - unfolding cylindrical distortion programmatically
In our application there is a feature,
like the son of my mother's girlfriend vivino - the definition of wine from a photograph. Under the hood - the use of third-party services, Tineye - to determine the most appropriate label, Google Vision - to read the text on it. The latter is necessary in order to clarify the correct product, tk. image search does not take into account the importance of certain regions, as a rule - it is textual information - year and type of wine.
However, the accuracy of both services is significantly reduced due to the fact that the label is distorted by a cylindrical surface.
This is especially noticeable in Google Vision - any text outside the central part of the label is almost not readable, although a person easily recognizes it. In this article I will describe how to reverse distortion and increase the accuracy of product recognition.
Convenience of this method is that the input parameters for the inverse transformation are the visually determined characteristics of the label (angles and upper, lower points), which allows to fully automate the process.
The next part is devoted to the definition of markers. The working code is only partially accessible in the branch. github.com/Nepherhotep/unwrap_labels/tree/detect_ellipses , t. a really working solution is covered by hacks and shamanism, so it's just that conscience simply does not allow you to spread it into githab.
Step one - convert the image into a black and white version.
Then you need to get the outline of the bottle with the label. For this we use the sobel transform. en.wikipedia.org/wiki/Sobel_operator . In short, this filter first blurs the image, and then subtracts it from the original. As a result, uniform areas remain dark, and edges (changes) are light.
The next thing to do is to determine the two most visible vertical lines, which, if lucky, are the edges of the bottle. In this case, this is true, but if you take pictures of a bottle standing next to other bottles, then this is not so.
To determine these lines, use the Hough transform en.wikipedia.org/wiki/Hough_transform
The essence of the technique is that we take a lot of lines that go through the whole screen, and we consider the average value of the pixels (for example, we take the lines from the top of the picture to the bottom). These values are transferred to a new coordinate plane and we get something like a heat map. On this heat map we are looking for two extremes - they are the lateral lines.
The diagram below shows how the left line goes to the point on the new coordinate plane:
With ellipses a little more complicated, but knowing that Huff's transformation can be applied to any mathematically defined curves, we will use this method again, but this time we will search for a set of ellipse curves.
But first you need to bring the problem to a two-dimensional form. Knowing that the bottle is centrally symmetric, take the central axis for the Y coordinate, and one of the sides for X. As values on the new coordinate plane, take the set of ellipses constructed between the central axis and the lateral side. This is possible due to the fact that an arbitrary point on the side and the central axis have only one way of connecting. Perhaps this is not very obvious at first glance, but much easier to understand if we look at the parametric formula of the ellipse:
x = a * cos (t)
y = b * sin (t)
In exactly the same way, we find two unknown extremes that will be determined by two label ellipses (curves A-B, F-E). Now that we have all the necessary label parameters (side curves, as well as upper and lower ellipses), we can apply the algorithm from the first part of the article and perform the reverse transformation.
What can be improved. First, the algorithm does not take into account the distortion of the perspective of the ellipse itself, as a consequence, the lateral fragments of the label stretch slightly more than it should. To make a correction, you need to know the real angle of the camera, or, at least, use the most typical for the phone (you can choose empirically).
Secondly, Haf's transformation works rather unstably under difficult conditions - for example, when standing bottles fall next to a bottle, and the edges of the bottle of interest can be determined incorrectly.
Thirdly, if the label is not rectangular (for example, elliptical), the markers will be defined incorrectly, and the transformation will only distort the image more strongly.
In practice, it is much more interesting to use a neural network to determine markers, because it can be trained using complex examples so that, at a minimum, the algorithm does not perform the transformation if the markers can not be determined. But for the time being I have not tried to apply a neuron for this task, so maybe this will be the topic of a separate article :)
It may be interesting
This is my first time i visit here. I found such a substantial number of interesting stuff in your blog especially its examination. Really its inconceivable article. Keep it up.Gulf Coast Western Reviews
I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help and wishing you all the success in your business.GSM Solutions
Wow, What a Excellent post. I really found this to much informatics. It is what i was searching for.I would like to suggest you that please keep sharing such type of info.Thanksthc vape juice
Thankyou for sharing the data which is beneficial for me and others likewise to see.Gulf Coast Western Reviews
Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work.GSM Solutions
Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.thc vape juice