ICE Scoring - the tool for choosing the best ideas of the product
Working with priorities is a task that requires preparation, experience and consideration of a variety of technologies, scientific approaches, and authoring methods.
This article is a translation of the material from the site Hackernoon.com. Her author proposes the use of her own priority assessment tool within the ICE Scoring method. In this article, the approach is described in detail and a simple and accessible example is understood, understandable to any product manager.
Itamar Gilad - a well-known consultant in the field of product management and a successful speaker. In his many years of experience - management positions in product management in Google, Microsoft and other well-known companies. We propose the translation of his article:
Let's say you manage a product for a small business and its customers. Your goal is to improve the degree of customer involvement and retention. You have two ideas on the agenda:
The implementation of the main dashboard, which allows the business owner to track the statistics of involvement and all trends.
Chatbot (chatbot) to automate communication with customers.
The idea with the tool bar arose several times in negotiations with customers, and you feel that it has good potential, but there is a risk that it will be used only by experienced users.
The idea with the chatbot like the whole company, and the management is quite optimistic about it. Also, the feature looks good for customers.
What would you choose?
These issues of prioritization are the basis of product management. The fee for incorrect selection can be very high and include the cost of development, deployment, maintenance, as well as other unscheduled costs.
We are often tempted to make a decision based on unconvincing signals: the views of the majority, the opinions of the bosses, industry trends, etc. But time shows that these signals are accurately at the level of the random number generator.
This post is about how, in my opinion, to find the best ideas. It consists of three parts:
Indicators of ICE
Levels of trust
ICE Scoring Is a method of prioritizing that was first used by Sean Ellis, known for his active participation in the formation of companies such as DropBox and Eventbrite, and the promotion of the term Growth Hacking. ICE was originally designed to prioritize growth experiments, but soon began to be used to evaluate any ideas.
In ICE, you evaluate ideas in this way:
ICE = Influence * Ease of implementation * Confidence
Effect of demonstrates how the idea will positively affect the key indicator that you are trying to improve.
Ease of implementation or simplicity is an assessment of how much effort and resources are required to realize this idea.
Confidence Demonstrates how confident you are in assessing the impact and ease of implementation.
The values in the ICE are estimated on a scale of 1 to 1? so that all factors have a balanced effect on the final number. Under the values 1-1? you can mean anything, the main thing is that the values are consistent among themselves.
Now let's look at an example of how this works.
So, you decided to calculate ICE scores for two ideas: dashboard and chatbot. At this early stage, you use rough values solely based on your own intuition.
Influence - you assume that the dashboard will significantly increase the retention of users, but only experienced ones - you give 5 out of 10. Chatbot, on the other hand, can become an innovative solution for many customers, so you give it 8 out of 10.
Ease of implementation - you estimate that for a dashboard it takes 10 man-weeks, and for a chat-bot - 20. Later from the team you will get better grades. You use this simple table (chosen by you and your team) to convert your estimate into Ease:
Thus, the toolbar gets the value of Ease 4 of 10 and chatbot - the value of 2.
Calculation of confidence
There is only one way to calculate confidence - this is the search for supporting evidence. For this, I created a tool that can be seen below. It lists the common types of tests and evidence that you can have, and the level of confidence they provide: test results, longevity date, self-confidence, thematic support, other people's opinions, market data, etc.
When using the tool, consider what indicators you already have, how many of them and what you need to get more confidence.
If your product or industry has another proof check, do not hesitate to create your own version of this tool.
Let's go back to the example to evaluate the tool in action.
Supporting the evidence for chatbot: personal confidence (you think it's a good idea), thematic support (in the industry they also think that this is a good idea) and the opinion of others (your bosses and colleagues consider this a good idea). This gives it a common confidence value of 0.1 out of 10 or Near Zero Confidence. The tool clearly does not consider opinions a reliable indicator.
What about the dashboard? Here personal confidence (you think that this is a good idea) and occasional support (several customers asked about it). This actually increases its confidence level to 0.5 out of 1? which is a low confidence. Unfortunately, clients poorly predict their future behavior.
ICE scoring in this case:
By this point, the toolbar looks like the best idea, but our tool shows that you have not gone beyond the limits of low confidence. For the time being, just enough information is not enough.
Verification of the assessment and feasibility
Next, you meet with your colleagues responsible for the development and UX, and together you begin to evaluate both ideas. Both projects seem feasible at first glance. The main developer offers a rough estimate of labor costs: working with the toolbar will take 12 man-weeks for the release, and with chatbot - 16 man-weeks. In accordance with your Ease scale, this gives ease of implementation in 4 and ? respectively.
In parallel, you make detailed calculations. With closer examination, the dashboard looks a bit less promising and gets 3. Chatbot still looks on 8.
The use of the trust tool shows that both ideas now pass the Estimates & Plans test and get some confidence. Now the toolbar moves to 0.8 and chatbot to 0.4.
Chatbot rehabilitated a little. However, the level of trust is low for a good reason - it's mostly numbers out of nowhere, and you understand that you need to collect more evidence.
You send a questionnaire to existing customers, offering them to choose one of 5 possible new features, including the chatbot and the toolbar. You get hundreds of answers. The results are very positive for the chatbot - this is feature # 1 in the questionnaire, and 38% of respondents choose it. Dashboard takes 3rd place with 17% of the vote.
This gives both functions support to the market, but the chatbox score is higher by 1.5. For the control panel, the confidence also increased, but only to 1.
Obviously, the chatbot has moved forward a lot. It seems that your colleagues and industry data have proved their worth. Should this be taken as 100%? Probably not - the project is quite expensive, and we have all that average confidence. Unfortunately, the results of the survey do not give a very indicative signal. We continue to work.
The word to customers is
To learn more, you run a custom research for 10 existing customers, showing them interactive prototypes of both features. In parallel, you conduct telephone interviews with 20 survey participants who chose one of the two proposed features.
The study shows a more interesting picture:
8 out of 10 participants found a useful dashboard and said they would use it at least once a week. Their understanding of this function was well correlated with what you had in mind initially, and they had no problems with its use. Telephone interviews confirmed the understanding and desire to use the feature on average once a week.
9 out of 10 participants in the study said they would be willing to use the chatbot. Their level of enthusiasm was very high - everyone immediately realized why this could be useful and many asked him as soon as possible. However, there were problems with usability, and some customers expressed concern that their customers did not like the repetitive and "jammed" responses of the bot.
This qualitative research gives you more food for thought. The toolbar seems more popular than you expected. Chatbot now resembles a project with a high level of risk and a high price. When you look at our trust tool, you assign the confidence bars 3 and 2.5 respectively to the toolbars and chatbot.
You adjust the effect like this: 6 for dashboard and 9 for chatbox. Finally, based on the usability study, you understand that getting a quality UI for a chatbox will require more work - you reduce Ease to 2.
The table has changed again, and the toolbar is now in the lead.
You bring results to your team and your management. According to ICE results, the toolbar should be declared the winner, however, on the other hand, the confidence indicators for both features are far from high. Not wishing to release a potentially good feature, the team decides to continue testing both.
Final tests and winner!
You decide to start by creating a chatbot version for a minimally viable product (MVP). The development takes 6 weeks, and you launch an MVP for 200 respondents who agreed to participate in the testing. 167 people activate the feature, but its use drops sharply day by day, and by the end of the second week you have only 24 active users.
In subsequent polls, a clear picture emerges - the chatbot is harder to use, it is much less useful than the participants expected, and, worse, it creates a negative for clients who value personal contact.
You can modify the MVP chatbot and make it much more useful for your customers, but it takes about 40-50 person-weeks.
It is also obvious that much fewer customers than expected earlier will be called a feature useful. Therefore, you reduce the impact from 9 to 2. This significantly changes the feature, so you can no longer trust the results of user research, so reduce the confidence to 0.5 with the help of the trust tool.
You run the MVP toolbar on 200 other clients for 5 weeks. The results are very good: 87% of participants use this feature, many of them daily. Feedback is overwhelmingly positive. You understand that the impact is higher than you expected - 8 points instead of 6. The development team estimates that it will take 10 more man-weeks to start the toolbar in its entirety, so it's easy to implement. 4. As a result, you increase the trust rating with 3 to 6.5.
At this point, prioritization becomes quite simple. Now everyone agrees that the dashboard is the right thing to develop the product. You keep the chatbox feature in your ideas bank, but it will naturally stay "on the bottom," given the low ICE.
1. Stop investing in bad ideas
Our example shows how risky it is to bet on features that require a lot of effort and are based on feelings, opinions, given industry, market trends, etc. Most of the ideas in fact are much less useful and more expensive than we think before working out. The only real way to find the best ideas is to test them and reduce the level of uncertainty.
2. Worry about the benefits, not the results of
Adding a phase of prioritization of features reduces the speed of product development - so it seems at first glance. But in fact it does not reduce, but increases speed. Thanks to the assessment of confidence, you simply do not do some bad features. It also focuses the team on specific short-term goals and increases team productivity. This processallows us to learn about the product, consumers, the market and ultimately get a better product that has already been tested by real users. Therefore, we are waiting for fewer surprises on the day of launch.
3. Encourage the diversity of approaches
In fact, we often have to choose not between two ideas, but between dozens. We reduce the cost of developing an idea based on confidence in it. This allows us to test many different ideas in parallel and avoid the pitfalls associated with traditional long-term planning.
In this example, the team tests 4 ideas in parallel, executing several projects (yellow boxes), each of which gradually completes the idea and tests it to increase confidence.
4. Get the layout of the leadership and stakeholders
Usually, when I explain this method, people are most concerned about how to get the consent of their management and stakeholders to implement such a prioritization process.
Can we limit their power over the product? You will be surprised. I've heard a lot from managers that they have to dive into the process of making product decisions because of the lack of strong options. A weak or strong option is, of course, a subjective concept, but until you see the real state of affairs with real evidence and a clear level of confidence in the evaluation of the feature.
On the other hand, the next time that the CEO makes you do your super-idea, show him how the idea is assessed with the help of factors of influence, effort and confidence, how ICE indicators on this idea are compared with the indicators of other ideas, and how we can test her to clarify the factor of confidence.
About the shortcomings of the ICE method, as well as about the alternative method of prioritization, you can read in our last article " RICE and ICE Scoring: simple prioritization techniques for advanced product managers ".
Was this article useful to you? Would you like to read more about this author? Please, tell about it in the comments.
It may be interesting
visit this site
visit this site
visit this site