How we created a technological product and fell to the bottom of
I want to share with you the story of a good life and a long, slow and painful death of one, once big, real estate portal. Who was not ready to change and make sharp movements in order to adjust to the changing market. The product, which for 10 years has gone from TOP-10 to the bottom. This is the story of the importance of the thread that connects the developers, with their understanding and mindset, and the managers /directors /managers with their approach to project management.
I came to this job in 201? a simple junior .net. Mon (further real estate portal) was a real estate aggregator, in addition to the database there was a high-quality thematic news section. News, articles, laws and many different materials on real estate. A large department of classroom journalists wrote very cool articles, interest in which was very high among users.
The site itself was free to place ads, which attracted users (private sellers, real estate agencies). All income was formed due to the usual in this area advertising placement of text and graphic blocks (TGB) from developers, promoting their new buildings and LCD.
During the heyday (2008-2012) the site had traffic to over 1 million unique visitors per month. And he gave a good conversion for advertisers. These were excellent indicators, considering that PN worked only for Moscow and St. Petersburg. In the state there were 2 programmers and many journalists and editors. In fact, all maintenance, support and refinement of the PI was carried out by only 2 employees (How many people designed and developed the original history of the PNC silent).
This concludes the positive part of the story.
The fall is
After the first wave of the crisis, the site began to appear problems with traffic and very serious. He just began to fall. The market began to change and quickly enough. Competitors began to push, because while our PN was resting on its laurels and did not develop anywhere, others were making their products more modern and meeting the expectations of users.
In addition, the search engines began to love our site less and less, which by the way was practically without any CEO. Yes, there were some templates for h? title, description, a small link, and, in general, everything (after the beginning of the fall and until the very end, the head of the PN started the CEO battle, which we were engaged for almost 5 years).
In the second wave of the crisis in 201? the indicators fell below 30?000 unique visitors per month. Approximately at this time I got a job.
That's where my technical part of the story begins. .
The first 2 years I worked as a junior developer. He did everything that was given to me. I had to learn a lot from scratch. However, I was lucky with my development manager. He was a little over 40 years old, an old school programmer, he knew a lot and knew how. Probably his main credo was - it works well (I'm very grateful to him, because I have learned a lot of knowledge and various approaches to development).
Our stack of technologies was as follows: MS SQL Server 2012 + ASP.NET MVC 3 . The base contained everything in itself. Even photos in binary form, for 3 fixed sizes (big, large, small).
Backend included several modules:
General Admins Office
Admine the SEO
Robot parser KLADER
The import robot of XML-feeds
All this time I just did tasks to make the code work and there were no errors. Especially not delving into and not thinking about further development. But the time has come and it became necessary.
In the second year of work, my supervisor left the project. And I was left alone with this ancient machine, which was on the strength of 3% of my code.
It was a wild stress . The CEO was asking everything from me, and I sometimes had no idea what and how it works. Naturally, life forced and gradually I penetrated into all processes and realized that that approach "works well" is not for me. At this time, I read a lot of materials on design, was educated in the field of popular technology and frameworks. I went home from work and thought about what and how I would do tomorrow. And when I went to work I thought about whether it was right that I decided yesterday. I wanted to understand how to do everything correctly and conveniently. To avoid having to do crutches, duplicate code, test everything live, to unmount everything manually and to give up good ideas because of some mistakes in designing in the past.
By this time, we did a lot of SEO, abandoning the tasks focused on improving the UI. However, traffic fell lower and lower. And at some point the project was frozen. I became involved only in the support and correction of bugs So it sounded official.
In fact, I began to completely rewrite the system from almost 0. I had to do it secretly from the management. After all, we never got time. Always said that you need faster, faster. That we are so far behind from everyone. And you have to make a competitive product in 4 hands and better than others. Therefore, if I said that I planned to do everything, then you understand what answer I would have heard.
So, rolling up my sleeves, I started my plan. I chose the classical three-level architecture. The .Net Core + SQL + Mongo backend, the frontend is Bootstrap + jQuery + KnockoutJS .
Organized the Data layer. Interfaces, abstractions, repositories are all as it should be. There was a layer on stored procedures (the blessing I quite well began to understand SQL). For mapping, I chose Dapper. It is simple and straightforward. Refused InMemoryCache in favor of Redis, to bring the cache to a separate server. Then there was the level of business logic. All the same interfaces, services, DI. So the basis appeared in the form of Data-Layer (Stored Procedures + Dapper + Redis) and Logic-Layer. It all took about 3 months.
And then, after almost a year of working alone, I managed to get an assistant, insisting that there were many tasks (and there were a lot of them). Together, everything went much faster and better, and most importantly even more rationally.
The first thing we developed was API for photos . It was a simple WebApi, which, on a Get-request, gave a picture of the desired quality and size from the disk. We switched to SSD and forgot about the base of pictures as a terrible dream. It's hard to describe how quickly the average page of the site began to load after allocating a separate pool for this.
We refused to use KLADRA in favor of FIASA . We have written a high-quality data parsing service from FIASA to our database, taking into account our features. Tied to it the service for geocoding houses. Everything worked almost like a clock. Only sometimes there were bugs associated with the duplicate locations or streets in the FIASA database.
Next long wrote a new personal cabinet , having allocated it from a site. Long designed and planned it to be user-friendly. And also fast and functional. We tied to it the payment, and then the fiscalization of checks (yes, we could not afford to use ready-made integrated solutions). In general, a good user service was made. And they were happy with it.
Finally we reached robot import XML-feeds . Made a convenient validator and good logging for customers. The new service was so optimized that if the old one (using EF) worked for about 6-8 hours, the new one processed the same data volumes in 2-3 hours.
After all, they raised domain with documentation for everything that only is. All the points for the users and clients of the portal were laid out on the shelves, as well as a part of the documentation that would be useful to the developers. And this is really important!
The last step was optimization of the base . We completely reworked it. We cleaned everything that was superfluous. Have achieved accelerate the search speed from 4-5 seconds to ~ 300 msec . They created indexes, wrote complex queries, used hints and even made partitions of statistical tables.
Unfortunately, we did not manage to get to the site right away. Because almost all the tasks with the site were related to SEO, which took decent time. New pages, new collections, new rules. More, more, more pages. It was necessary to constantly something to correct in a site engine that did not allow to transfer it in parallel on the created basis.
Here the technical story ends and a sad epilogue begins.
It is worth starting with the fact that the general director of the site was not connected with the IT-sphere. Therefore, many decisions were made by him incorrectly and solely. Very often the discussions came to a dead end, tk. Our new ideas were not accepted, because of
"This is not necessary, I'm talking to you as an expert in real estate"
"So no one is looking, it's a low-frequency request, I'm sure"
And after some time, when he himself came to this, our ideas were offered to them with a claim, why we did not say before, or with sincere surprise
"I could not say that, it's absurd"
We could not come to a consensus, as a result, we (the developers) just conceded. And they did as they asked us. Programmatic pain - do not need "features".
I would like to mention that there have always been problems in finance. No attachments, except as in SEO was . Even keeping 2 programmers it turned out to be expensive. To compete with new portals from TOP-10 with such level of financing and management, obviously, it was unrealistic.
As a result, we have a real estate aggregator portal platform. Scalable, expandable, working quickly and technically ready for any disasters. With great potential. A good code, a minimum of crutches and few bottlenecks.
However, this did not give any positive result. At the time of my last working day, despite the presence of 4 million unique pages in the search engines, the traffic has rattled fluctuating around ?400 uniques a day. And this, I think, states death.
P.S .: From all this story I could personally make one main conclusion. You can make a good product from the point of view of the developer, but it will be absolutely not claimed, tk. does not have proper management. If the thread between employees that keeps the business afloat is torn, then your product will certainly sink.
It may be interesting
I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done!
Took me time to understand all of the comments, but I seriously enjoyed the write-up. It proved being really helpful to me and Im positive to all of the commenters right here! Its constantly nice when you can not only be informed, but also entertained! I am certain you had enjoyable writing this write-up.