GeoPuzzle - Collect the world in pieces

 3r33333. 3r3-31. GeoPuzzle - Collect the world in pieces 3r33333.  3r33333. I want to talk about a project that has developed over the last couple of years. It is called GeoPuzzle and is a puzzle game on the political map of the world. The goal - to put the pieces of the country into place. The idea is peeped at article "Puzzle Mercator for geography connoisseurs" I also played tetris from countries (still under DOS) as a child, but I don’t remember the name of the program. I was so inspired by the idea that I wanted to make a complete product, interesting not only for schoolchildren, but also for geography connoisseurs. The development of the project can be observed at
Github
. 3r33333.  3r33333. blogging , The source is laid out on Github . And that's all - stuck. I had significantly less free time, the motivation was gone (no decision was made), and the complexity increased exponentially. It was a pet-project, and the main task was to learn something new, so I was a little distorted with technology. On the client, of course, jаvascript (with which he worked a little then), the script was responsible for preparing the data in ruby ​​(again, a new language for me), and on the server it was erlang (I wanted to try something purely functional). Full exit from the comfort zone: it was difficult to work directly with PostGIS objects, pain when converting strings to erlang, setting YAWS is a separate topic At the next stage, I realized that I simply could not cope with all this zoo to make a full-fledged product and left think for a couple of years. 3r33333.  3r33333. 3r33333.  3r33333.

GeoPuzzle

3r33333.  3r33333. 3r3333. 3r33333.  3r33333. All this time the site was working, people even went there, but I really wanted to add one more little detail: to show at least some information about the newly solved landfill. Thus, I planned to spend the New Year holidays in 2017 with benefit. Due to problems in the prototype, I decided to rewrite everything and make the product on something familiar - Django. There are things out of the box that have significantly simplified my life, for example, the admin panel and working with PostGIS through ORM. But first it was necessary to recreate the functionality that was already working. It took only a couple of evenings, and most of the time was taken away by loading data from KML files. Not so much the import process itself, as their preparation and restoration of my knowledge of how to work with them. By the way, at that time I took the polygons from 3r3336. gadm.org
. This worked fine for countries, but with accuracy for the regions there were certain problems, so I took time out for this problem. 3r33333.  3r33333. 3r33333.  3r33333. 3r33333. 3r33333. A couple of words about administrative levels in geodata (levels). 3r33333. 3r33333. All countries are divided into many regions, which are divided into even smaller parts. There are 12 layers in such a hierarchy. 3r33333.  3r33333. For example, for Russia:
 3r33333. 3r33333.  3r33333. 3r33333. Russia (2) -> Southern Federal District (3) -> Krasnodar Territory (4) -> Vyselkovsky District (6) -> Art. Vyselki (8)
 3r33333. 3r33333. Russia (2) -> South Federal District (3) -> Krasnodar Territory (4) -> Krasnodar (6) -> Prikubansky District (9) -> Kopanskaya (10)
 3r33333. 3r33333. France (2) -> Metropolis of France (3) -> Normandy region (4) -> Orne department (6) -> Donfron canton (7) -> Donfron-en-Poiret commune (8) -> Donfron (9)
.  3r33333. 3r33358. 3r33333.  3r33333. Territorial units in different countries are called differently, but for myself I derived the following division: Country (2) -> Region (4) -> District (6). Further administrative division left for later. 3r33333.  3r33333. 3r33386. 3r33386. 3r33333.  3r33333. 3r33333.  3r33333.

Development project 3r329595. 3r33333.  3r33333. At that time, the application was simply a collection of HTML pages with a minimum of CSS. I wanted to quickly check the idea, and not to bother with the design. The idea turned out to be realizable, and it was time to make a beautiful shell for it. Since I don’t have feelings of beauty in the UI, then Bootstrap will help me. The interface is no, but appeared, and even adapted for mobile devices. But it was only the first small step in bringing the front end in order. 3r33333.  3r33333. 3r33333.  3r33333. 3r380. What is it to learn jаvascript in 2016
?! When the code is compiled into another dialect, it is template-edged, glued together to be cut into pieces. As a backendant, I was scared, but the complexity of the client part was planned to be quite large, which necessitated the use of a framework or library. I stopped at React for two reasons: it was not a SPA, but a set of components for different pages, and I wanted to quickly see the result. But before you start programming, you had to set up the environment. Now I understand a familiar front-end vendor who said that he set up a Webpack for 2 days. It turns out it was not a joke. 3r33333.  3r33333. 3r33333.  3r33333. At that time, I succumbed to the entreaties and implemented the logic of the application using Redux. Perhaps it was not a mistake, because allowed to quickly enter the topic. Formal rules allowed me to write code and make sure that it was working, without looking under the hood. Redux, with the help of its middleware, abstracted me from network interaction, which allowed me to check the answers to the server. Yes, up to this point, the client worked on its own - ajax pulled out all the necessary data with a request and checked the answers on its own. The user could cheat, peeping at the data that comes from the server. In addition, when downloading, the data arrived, which was necessary only after the correct answer. After the implementation of checking through web sockets, the process became ideologically more correct - the answer checks the code that is not available to the client. It still seemed instant to the user: sending the extreme points of the polygon to the server, checking if they were squared with accuracy, data packing for the infobox and the detailed polygon in json and transferring to the client were within ~ 200ms. 3r33333.  3r33333. 3r388. 3r33333.  3r33333. 3r33333.  3r33333. Having learned all the power of jаvascript, it is difficult to stop. Immediately there were ideas where to add animation, folding blocks, blinking and new versions of games. One of them - “Quiz”, in which by name, flag, emblem or capital, you must guess the country. However, in the process of testing, it turned out that some regions do not have flags, while others do not indicate the capital, so some countries had to hide from the list of available. At the same time, the game mode appeared on the physical map of the world - without borders of countries, for real pros. 3r33333.  3r33333. 3r33333.  3r33333.
Open data sources 3r3323295. 3r33333.  3r33333. Now there are ~ 5?000 polygons in the game, and I want to say a big thank you to such great projects as Wikipedia and Open Street Map, without which filling the base would be simply impossible. The principal requirement was to receive and update data from public sources, that is, without manual editing, because I don't want to do complicated synchronization logic. As a result, I got 2 scripts that can update infoboxes and polygons. 3r33333.  3r33333. 3r33333.  3r33333. 3r33333. Wikipedia and SPARQL
3r33333.  3r33333. 3r33333.  3r33333. What is the largest database of countries and regions? Wikipedia! Initially, I wanted to show users the whole infobox, but soon I abandoned this idea. Yes, there were important things like names, flags, capitals and other things, but there was also a lot of rubbish (phone code, form of government, GDP ). I tried to parse already collected, but found that they have a different structure. This turned out to be a catastrophe: labor costs for implementation have increased many times. It was time to stop and think. The very next day I learned about the existence of a special query language - SPARQL. In appearance, it resembles SQL - also declarative, with the keywords 3r3164. SELECT , 3r3164. WHERE , 3r3164. ORDER BY but it works completely differently. A small example that returns a list of states with their capitals in English and Russian languages: 3r-3373.  3r33333. 3r3164. SELECT DISTINCT? Country? Capital? Row
 3r33333. WHERE
 3r33333. {
 3r33333. ? country wdt: P31 wd: Q3624078. 3r33333.  3r33333. FILTER NOT EXISTS {? Country wdt: P31 wd: Q3024240}
 3r33333. OPTIONAL {? Country wdt: P36 /rdfs: label? Capital}. 3r33333.  3r33333. 3r33333.  3r33333. BIND (lang (? Capital) as? Row)
 3r33333. filter (? row = 'en' ||? row = 'en')
 3r33333.}
 3r33333. ORDER BY? Capital
 3r33333. 3r3r1616. 3r33333.  3r33333. It looks crazy, is not it? You can check 3r3146. here
. I wrote even a small 3r3148. 3r3r7676 blog post. to somehow structure your experience and help enter the topic, because any detailed materials on the Internet a little. You can learn to read such queries fairly quickly, but it took me a whole weekend to write something meaningful. A lot of “magic” from 3r3164. wd: Q3624078 and other attributes. You need to know that wdt: P31 This “essence”, and wd: Q3624078 - “sovereign state”. Unknowns begin with a question mark, and the execution of a query is precisely the search for such triples of facts that satisfy the conditions. For example, 3r3164. ? country wdt: P31 wd: Q3024240 - “find all objects that are historical states”; and then the same object participates in the other three ? country wdt: P36 /rdfs: label? capital - where he takes the capital. 3r33333.  3r33333. 3r33333.  3r33333. Somewhere in a week I had the first version of the script ready, which uploaded information about the regions from Wikipedia. And then another problem emerged - this time with the data. Some svg did not start with 3r3165. 3r3r1616. and were not recognized by the browser as valid images. Fortunately, the source file can be edited. Registering on wikipedia.org is not a big deal, but you immediately find yourself in a bath for a day. Here they have such protection from robots. So the very next night, I ruled XML and was happy about how simple it turned out, and flags and emblems appeared on the map. 3r33333.  3r33333. 3r33333.  3r33333. 3r33333. Polygons
3r33333.  3r33333. 3r33175. 3r33333.  3r33333. If for the facts we go to Wikipedia, then for geo-data - in the Open Street Map. It would be cool to pick up a local copy, learn the language of requests to 3r3178. overpass
, but I can’t even imagine how long it would take. It’s also necessary to take a hierarchy from somewhere Fortunately, one kind person is already solved this problem for me . The service even provides an API for getting information. I managed to download all the polygons from there to level 6 inclusive (district) and upload everything to Postgres, a little more than 2 Gb came out. It was not without incident - some of the polygons were so large (for example, Canada weighs more than 100 Mb in a GeoJSON zipped) that the server either fell or did not respond. It was necessary to bypass such moments manually. I downloaded all the children and merged them into QGIS. By the way, this is another example of an open source project that helped me a lot. 3r33333.  3r33333. 3r33333.  3r33333.
Problems with big data

3r33333.  3r33333. So, I have a database with the data, I launch the game and and wait wait again appear! I tried dragging the polygon - He's dead, Jim! Chrome could not handle such a volume of points and fell. The strategy "in the forehead" no longer works, it's time to think. The most obvious is to reduce polygon detailing. Empirically derived a formula that depends on the area of ​​the figure - it became better. On the working computer, the algorithm worked on the fly, while the server is strictly limited in resources. Connected redis, it became better already on the server. But the trimmed polygons are good for Drag'n'Dropʻa, when installed on the right place, the borders do not coincide with those that draw Google Maps. Well, this can be circumvented by applying a less aggressive formula to reduce detail. Since there are already 2 caches, why not try to cache everything that is possible at all ?! In Redis, infoboxes (in two languages) flew, the boundaries for which the answer is calculated, the center of the polygon, as well as the static pages of the site. As a result, the game began to work much faster, and a lot of work was removed from Postgres, which theoretically could be the bottleneck. Minus - the application does not work without Redis at all. 3r33333.  3r33333. 3r33333.  3r33333.

The first deploy is

3r33333.  3r33333. So it's time to show the project to your friends for feedback. Only a little remains: generate sitemap.xml, add robots.txt, connect metrics, add social buttons. networks and cool down! I chose AWS as the hosting. I expected to fit into the free resources. And this is a very good stack for a beginner project:
 3r33333. 3r33333. Application server (t2.micro: 1xCPU, 1 Gb RAM, 20Gb SSD)
 3r33333. 3r33333. Database (db.t2.micro: 1xCPU, 1 Gb RAM, 20Gb SSD)
 3r33333. 3r33333. file storage with CDN (5 Gb S? 50 Gb traffic) 3r-3356.  3r33333. 3r33333. caching server (cache.t2.micro: 1xCPU, 0.5 Gb RAM)
 3r33333. 3r33333. Elasticsearch + Kibana (t2.small.elasticsearch: 1xCPU, 2 Gb RAM)
 3r33333. 3r33358. 3r33333.  3r33333. This is just a list of what I managed to use. Along the way, I decided to draw up my rake 3r-3218. in the form of articles 3-333376. but quickly died. Time is running out decently, but it is not clear whether someone needs this. 3r33333.  3r33333. 3r33333.  3r33333. As a result, for the year of service, I paid something in the order of $ 1? and that was by stupidity. But the trial period came to an end, and I had to move, because The cost of owning all this farming has come close to a few hundred dollars. Equalized tariffs, and stopped at DigitalOcean. So far I have enough machines with 2 Gb RAM for everything (application server, database and cache), but I left statics and CDN on AWS. Now I learned that DO also got a CDN and a vault for $ 5 /mo, so it makes sense to think about moving this part as well. 3r33333.  3r33333. 3r33333.  3r33333.

Transfer to Open Source

3r33333.  3r33333. January evening of this year I received a letter from the Danish school. The essence of it was that they had $ 10? and they want to give them to me. But there is a condition - the source of the project should be open. Up to this point, I did not even think about Open Source. A couple of evenings went to think about and choose a license. As a result, 3r33232. laid out source codes on Github
under the GPLv3 license and received the promised $ 100. This greatly increased motivation - my project really turned out to be useful! And I rushed to the next goal - the game editor. So that everyone can create their own puzzles. For example, “Countries participating in the Second World War”, “Okrugs of the Krasnodar Territory”, “Countries without access to the sea” But for this, registration and a primitive personal account was necessary. As a result, the development was delayed for a long 3 months. During this time, I wrote a tree of regions that would pull up the data through ajax, connect localization, learn how to save the Google Map as a picture for generating thumbnails, and cut redux. Yes, he helped me deal with ddata right at the beginning, but now rather hindered. It would be necessary to drag the reducers to draw polygons on the map along with the code that would handle their movement. Fortunately, removing the binding to the global state took only a couple of days, and even moving the code to a local one simplified the application. And of course this is a good experience :)
 3r33333. 3r33333.  3r33333. 3r33333. Connecting services
3r33333.  3r33333. It turns out that many paid services provide their services for free for open source projects. I will list only those that are connected to yours. 3r33333.  3r33333. 3r33333.  3r33333. - Sentry . I think this error trapping service is familiar to everyone. When I just checked out the project, the logging was to send the frame to the post office. This only worked for the backend, but I also wanted to follow the bugs on the front end. And it’s not for nothing that I literally exhausted the free limit of messages in just 2 weeks. Most of the errors were in the depths of the google map library, which at first glance is very strange. During the investigation, it turned out that I was to blame. Corrections lasted more than a month, but it was a very useful practice of working with errors in jаvascript. 3r33333.  3r33333. - crowdin.com - localization. I plan to make the project accessible to everyone. Including that infoboxes are displayed in his native language. Filling them from Wikipedia is not a problem, but for the consistency I would also like to have an interface in the same language, but it is only translated into Russian and English. 3r33333.  3r33333. - CircleCI . No modern project can do without CI /CD, tests and automatic deployment. I chose CircleCI solely because I already worked with TravisCI when I wrote 3r33257. a library for working with Yandex.Disk
. I got the impression that it is more suitable for testing libraries, since it is easy to set the matrix of environments in which the code should be tested. But with the tests themselves I have trouble - there are not as many of them as we would like, although the infrastructure is already ready. 3r33333.  3r33333. - Coveralls . Service code coverage visualization. It also knows how to give a label to be inserted into the project's README.md. 3r33333.  3r33333. - SonarQube . The combine for quality control of a code. It checks the code according to a set of rules, considers cyclomatic complexity, monitors test coverage, and even recognizes duplication of code! Very interesting service, which I did not have time to fully understand. 3r33333.  3r33333. - Github bots. So far, only is connected. Dependabot which actualizes dependencies. 3r33333.  3r33333. 3r33333.  3r33333. I suggest in the comments to share a list of services and bots on their projects. 3r33333.  3r33333. 3r33333.  3r33333.

Bugs

3r33333.  3r33333. Analysis of bugs and problems deserves a separate article. There were funny, complicated and difficult to fix (therefore, Chukotka always stands in its place). Currently, there is one that really bothers users. When a response is received, polygons are deleted and re-created (in the react-google-maps library), and if at that moment the user dragged some, the google map continues to assume that the process is not yet complete. It looks like the polygon disappears in the process of drag'n'drop, and you can no longer grab any other. You can, of course, put a lock on handling the response in the process of drag'n'drop, but this is guaranteed to kill the multiplayer game, on the implementation of which I am currently working. I tried to find a way to programmatically interrupt dragging, but in the end I started 3r-3286. question on StackOverflow
and bug on Google Maps in the hope that he will pay attention. Until then, he added a button “the game broke!”, Which reinitializes the entire card, but does not reset the result. 3r33333.  3r33333. 3r33333.  3r33333.

What's next?

3r33333.  3r33333.
 3r33333. 3r33333. Design. I admit that this is all bad. We need to hire a designer and coder, because I am not friends with the layout and layouts. 3r33356.  3r33333. 3r33333. Monetization. Initially, I did not plan anything. The project is dedicated to basic education, which in my opinion should be accessible to all. I was very inspired by a letter from the Danish school, but almost a year passed, and during this time there was only one transfer for $ 5. Well, I did not believe that it could at least pay for the server. However, it still started campaign on Patreon . At the same time, you can probably think about the introduction of paid opportunities for teachers or organizations. For example, I have experience with integrating with Learning Management Systems - a set of platforms that allow you to create courses that are very popular in Europe and the USA. As far as I understand, even though the sources are also under the GPL on Github, this does not prevent me, as an author, from developing in parallel the commercial version. 3r33356.  3r33333. 3r33333. Mobile phones. I want to release an application for iOS /Android. Judging by the Yandex-metric, a quarter of users try to play from a phone or tablet, but it turns out that they have difficulty. 3r33356.  3r33333. 3r33333. Development. All work is done on 3r3325. Github
. I want to continue to develop the project, but to do it alone is hard. Plans to add multiplayer, make tags, ratings and filters in the Workshop, add polygons for the physical map (mountains, seas, peninsulas). There is still a lot of interesting things ahead, so one of the goals of the article is to find like-minded people. Another option is to go to the foundation, for example, 3r33314. Python Software Foundation
and get a grant. 3r33356.  3r33333. 3r33333. 3r33333.  3r33333. 3r33333.  3r33333. That is what is at the moment. Thank you for reading to the end! You can play here - geopuzzle.org , and source codes to look at
Github
. 3r33333.  3r33333. 3r33333.  3r33333. 3r33333. 3r33333. 3r33333. Minute of care from UFO
3r33333.  3r33333. This material could cause contradictory feelings, so before writing a comment, refresh something important in your memory:
 3r33333. 3r33333.  3r33333. 3r33333. 3r33333. How to write a comment and survive [/b] 3r33333. 3r33333.  3r33333. 3r33333. Do not write offensive comments, do not go to the person. 3r33356.  3r33333. 3r33333. Refrain from using obscene language and toxic behavior (even in a veiled form). 3r33356.  3r33333. 3r33333. To report comments that violate the site’s rules, use the “Report” button (if available) or 3r3333354. feedback form
. 3r33356.  3r33333. 3r33358. 3r33333.  3r33333. What if: minus karma | 3r33333. blocked account
3r33333.  3r33333. 3r33333.  3r33333. → Code of authors Habr and Habraetiket 3r33333.  3r33333. → Full version of the rules of the site 3r33386. 3r33386.
3r33386. 3r33333. 3r33333. 3r33383. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e. ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () ();
3r33333. 3r33386. 3r33333. 3r33333. 3r33333. 3r33333.
+ 0 -

Add comment