What I understood and what problems I encountered, creating a clone of Hacker News
From the translator: this article is a shortened translation of the original post web developer Jesse Horn. His work and at the same time a hobby is web design. Jesse often shares his experience and experience with other programmers, both experienced and beginners.
Some time ago I wrote a pretty lengthy post on Hacker News, which received solid support from readers. I drew attention to the possibilities of this resource and decided to try create his clone , in order to gain new experience and knowledge.
We remind you: for all readers of the "Khabar" - a discount of ??? rubles for recording on any course Skillbox for the promo code "Habr".
Initially, this seemed to me an excellent goal, where the set of tools and tactics of work were predetermined. Earlier I worked on smaller problems using different languages and frameworks. The only thing that did not exist was a large project that would allow to collect all this together.
I decided to work on a new project from the ground up using the Crystal language. This is a handy tool, which is also new. It's fast, the principles of working with it are similar to Ruby. It is statistically typed plus is open source. But just a new language is also not enough, so I decided to complicate my task, and not make a clone of Hacker News, but an improved version of this resource.
By the way, I thought that creating an ordinary clone would be an easy task. And the fact that this is not so, I realized a little later, when I started work.
Do not mark too high
I do not want to say that you should not set ambitious goals. Try to reach the stars, through the thorns or without them. To dream - it is useful, to try to achieve what you could not do earlier, is necessary for successful advancement on the professional ladder.
I said that you should not try
at once. rush to the embrasure fulfillment of a complex and large task. Choose what you need at a specific moment to implement the idea. Further I recommend to break the significant goals into smaller ones. If you have too much code, which has a lot of "garbage", then try to simplify and structure everything.
I can not even express in words how important it is to divide a large task into several smaller ones.
In my case, I ran into various new problems that I had not met before. First, I worked Crystal only once - when I created the application, the complexity of which was not much higher than the complexity of the program like "Hello, world!". In the past, I used Python, Lua, PHP. The new language was the first obstacle to the implementation of the project.
The second was the framework. . Crystal, being a new language, still does not have too many frameworks and documentation for them. I decided to use Kemal for my clone. I was hoping that the framework is intuitive, since I've met exactly those in my work. But there was another case here - some things were not easy to understand. For example, some parts of the code looked quite normal, but for some reason they did not work. And to find the cause of the problem on the Internet was difficult for the same reason - the novelty of the language and the framework itself. In some cases, I had to stop for a long time and deal with the cause of the problem. Until now, I have never looked in the source, for example, the same Flask.
The third obstacle is - The specific goals that I have chosen for the project. In order to make my clone more useful, I decided to add a number of functional features. I worked with Hacker News before, so I know a lot about the advantages and disadvantages of this resource. It occurred to me that it would be good to use the analytical data provided by GitHub. Using them, I could add statistics on the number of views and user activity on the site.
So, I decided that my clone should show at least the views and clicks for each post. Such a function would probably be useful for users, and its implementation would be a good practice for me. But I did not think it would take so long! Now the project is more a proof-of-concept than a ready-made resource. In addition to views and clicks, I also decided to add a display of the appearing comments so that they appear in real-time. If not for this additional task, the project would be completed much faster.
Now the project supports authorization, placement and sorting of posts, interaction with other users by commenting and viewing profiles, etc.
The fourth problem is , with which I came across, is chaos. Migrations, models, views, API controllers, CSS and jаvascript code are accumulating very quickly. And all this turns into a mess. I tried to solve the problem, and I almost did it. But there are some problematic places in the code of my project - you can easily notice it. So, in many places the code is repeated, endpoints are not organized, naming is complicated, which leads to an increase in the amount of code and a large number of queries to the database. Gradually I solve this problem.
In many places the code is repeated
When the project started, I solved the problems as they appeared. For example, you had to check whether the user was authenticated - and I found an original solution that took three lines or so, depending on the route. I was in a hurry to implement the solutions found.
In general, I had to follow the principles of DRY from the very beginning. Do not repeat! I had to write an intermediate program that would do what was necessary, and then simply apply the solution found in those cases where it was required. In the future, I will adjust my project based on this principle.
Naming endpoints and organization
I used to develop an API in the Pioneer team. We used REST - the style of architecture that I actually swore in as a new member of the Pioneer team. In this team I worked on small elements of a huge application. But in my new project I had to write everything from scratch.
"RESTful API" should have a Uniform Interface, as it says in "Wiki". In my project, I organized everything in such a way that endpoints were defined in source files, which were called functions or purposes, respectively. Registration and exit from the application were registered in "auth.cr". The user's login was registered in "user.cr".
Undoubtedly, this part requires recycling. In the future, I organize everything in such a way that the actions concern the endpoint "/user", simply because all this has to do with "User". For example, to improve the structure of the project, I will divide the code into more convenient and manageable parts with a large number of files. Create the folder "src /user /auth /", which will contain the functions of login, registration and exit the user from the application.
Too many requests from
In this project, I try to scale my ideas. Too many requests are being made to the database. So, if the application had one hundred users, I would just go crazy, trying to find out how many requests to the database are being executed every few seconds. Now the client updates the status of the messages every 10 minutes. In addition, the post cyclically determines whether it was viewed by the user. During each check, a request is made to the database, this is performed according to the current filter. And all this greatly loads the server.
All that is described above are only examples of the problems that I always face. I can not describe them in full. Nevertheless, now I'm thinking about documenting my problems and solutions. I hope that all this will help other programmers see and understand my mistakes and avoid their own. I know that if I use a combination of Redis and MySQL or Postgres, I can speed up the execution of queries. But for now, I will focus on this.
I hope that you liked the article and you were able to learn something useful for yourself.
Online course "Profession frontend-developer" .
Educational online course "Profession of the web developer" .
Practical course "Mobile Developer PRO" .
It may be interesting
I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done!
Took me time to understand all of the comments, but I seriously enjoyed the write-up. It proved being really helpful to me and Im positive to all of the commenters right here! Its constantly nice when you can not only be informed, but also entertained! I am certain you had enjoyable writing this write-up.