Transcript — Part B

You launched your new filters, and they offered, I don't know, 50% off on everything.

Of course a number of orders will increase, that's not very surprising. While the average order value will probably decrease, right, but it might increase the revenue overall, right? Has nothing to do with you. So how can you then indicate your impact, right? Now, that's something that I struggled with for a while, just a moment, Vinu, just a second.

Well, to measure, to kind of—

To indicate your impact, we need to do two things. We cannot just measure that big goal on the top.

Right? That's not going to happen. We need to measure at least at 2 levels. If you're working on filters, you need to measure the impact on filtering. That means maybe you know which filters people are using the most, and you need to understand if people with your new changes now use filters better or faster or, you know, better in some way, right? That's important. So if you're working on search, you need to measure search. If you're working on navigation, you need to measure navigation. But then you still need to, of course, take a look at what's happening globally, right? But these 2 levels are very important. The local level of what you're working on, and ideally if you're working, let's say, in a unit of representing a particular flow or product, right, you need to measure success on the product level as well. It's just going to be a little bit more difficult to get your way all the way to the head of the KPI tree in your unit, because it is being influenced by many other people too. Okay? And as you keep going down that tree, right, it kind of becomes even more challenging, because if you look at the orders, right, it's broken down, it's visitors and conversion. If you increase both, that's great. If you increase one of them and the other is stable, that's great. If both of them going down, that's bad, right? But then if you're all the way on level 5 and you're working on filters on level 5 to drive particular metric in there, right, it will be very difficult to make a strong case that you worked on filters and improved revenue.

Okay? So everybody still with me? Nobody left?

Just yet? Vinu, you have a question? Could you maybe write it in the chat so I don't interrupt everyone? Would it work?

Sure.

Okay, perfect, thank you. And I'll be dropping in questions in just 5 minutes when I finish.

So this also can represent itself in a very different way too. It can be quite advanced. So, for example, this is another example here. Where you have something that's called a Norse star metric. Right here. Anybody has a Norse star metric in your organization, in your company?

By any chance?

Yeah? Well, some people do. So Norse star metric is perceived as being the important metric, right, that entire organization is aware of, and everybody is trying to improve that metric. Now, behind that metric, and that's often very important, it's often not mentioned, there are significant business metrics that are the company is trying to hit, right, or improve. Like, for example, annual recurring revenue, or monthly retention, or premium subscriptions, or anything like that, right? But what is being disclosed to the organization is this metric, right? This Norse star metric. Now, this Norse star metric can be influenced, again, by a lot of different initiatives. Right? And so you could have something like this, where there is an initiative to improve, let's say, launching new marketing campaign, social notifications, and others, right, kind of closer to the product side of things. So even design side of things. But then what is really important is when you start measuring, right, that you need to understand how well you're contributing to that goal. And to understand that, we need to attach some sort of metric that represents the success of that initiative, right? So if we imagine that this is working as expected, right, what is going to change? If we imagine that this is working as expected, what is going to change? And kind of keeps going. And sometimes product managers would assign it or attach it to epics, in Jira, right, or in Asana projects, right, and stuff like that, right? And before that, there will be either research done, right, or there will be assumptions, experiments, hypotheses, right, that will be used or will be— I guess the initiators of some initiatives that are coming through. So you're basically creating this flow of value. This is what we call the value architecture, right? All the way on the right you have the business metrics. All the way on the left you have design initiatives or product initiatives. And in between you have product metrics that are supposed to influence the Norse star metric that then influences business metrics. The only thing that's missing hereRight, and just to visualize it to you, I'm going to show you a real thing, here, and I'm going to drop in a link to you, so feel free to use it as well if you like. So this is coming from a client project I was working on, and then I'll jump into a question in just a moment, uh, here.

And this is basically how I visualize that flow of value.

Right, um, so— okay, excellent, I can see a lot of pointers in here. Beautiful. And so what you see right here is basically the same thing: you have a business goal all the way on the top. So imagine you're working for a manufacturing company that produces domestic appliances. They want to sell faster. That's all they do.

The challenge is that they actually are selling premium appliances, which have quite expensive— right? So people need time to commit to purchase, right? So that's a bit of a story. So I need to understand: what can I do as a designer, right, to drive that goal?

I also do, like, whimsical, by the way, we, so that's— we have something in mutual.

So we can break it down, right? If we want to reduce time to purchase to help people buy faster, I need to understand what that means. Right? And so I sit down and I try to translate it: what does it mean? Well, to me, that means that customers should fast recognize the value of the products, and also— if they're buying something like a washing machine, right— be able to make best fit choices.

Because if they cannot make a best fit choice for them, or if they've gotten the choLet's say my job, really, when I start measuring, is to make sure that I cover all the bases to guarantee success to the highest level I can. Which can be very difficult, of course, but I cannot miss the big picture. So what we get here is another side of the story, which is often forgotten. Like, for example, frustration. When people are frustrated in some way, and they can be frustrated by the price as well, right, then they will not buy. If the web page is not responsive, that's a problem. If the search is not working well, that's a problem. If a configurator is not working well, it's a problem. If error recovery rate is low, that's a problem. So all of those things are endangering basically the risks of people abandoning prematurely, right, and that's something that's quite challenging, right, for us to get right. But we need to do our best.

But on the other hand, there are also very other different things that kind of go into this, right? Like, for example, you can say, well, I want people to buy faster, while sometimes they can bypass the website altogether by ChatGPT, talking to ChatGPT, please don't talk to ChatGPT, talk to Gemini, or Anthropic instead. Or referrals. If somebody recommends a specific model to use and to buy because they love it and, you know, and everything, that will probably increase your chance of buying it, right? Okay, so these are the free stories, right? It's not necessarily comprehensive, but we need now to break down. So let me just explain that. And we break down further, right, and we say, well, how do we reduce time to make a decision?

What does it take?

Well, for people to be able to make a decision what to buy, we must reduce the time for them to find a candidate that will be basically the washing machine or not, right? So they need to find the one from a list of 70. They need to find the one. That's important. But on the other hand, sometimes people search a bit differently. They kind of need a couple of candidates, and then they compare.

Right? Maybe they would just copy-paste all of that stuff to Gemini or Anthropic or Claude or ChatGPT or whatever, and then compare there, but they need a couple of options. People think differently, right? And on the other hand, when they make a decision to buy or not to buy, right, for this to be reduced, I need to track how successful our landing pages or category pages or listing pages are in helping people make a decision in the first place. Right? Meaning, can somebody coming to that page giving the requirements and, you know, measurements that they have, say, this is a washing machine that's for me or not? Because if they're paralyzed, that's a problem. And so we can also run a session to understand why and when users fail.

Right? Still with me? That's level two. Usually you might need two to four levels like this.

And then we keep going because, of course, if you want to reduce time to top candidate, now we can do stuff on a product level. So we're moving from the business side to the product level. And on the product level, well, we can help people compare faster. Maybe you can increase the number of video reviews. Maybe you can try to reduce the drop-off rates or improve guides or improve filters or improve the use of configurator or something like that. This is easy for us to think of. I mean, this is more closer to the product that we're working on, right? But then every now and again, as I'm jumping from top to bottom, I feel like I'm lost or I'm stuck or I'm paralyzed. So I jump all the way down. And all the way down, what you will find is this different boxes. The purple boxes representing a user need or job to be done or anything we know from user research. The yellow boxes represent design initiatives. That we are going to take to drive that goal that we have all the way on the top. The blue boxes represent what we're going to measure to assess how successful all of this has been if we imagine that this already is implemented. Because if all of this is already done, right, then we must have moved something ideally. We don't just take it randomly or pick it randomly. We believe that something will change, and we'll be working for this in something like 20, 25 minutes from now, right, to kind of see how it's built up from the beginning till the end, right? But then we also need to see about the metrics. Right? Again, how are we going to define what we're going to measure? How often we're going to measure? Basically, it must be a number and time bound. But then, of course, when you go in a big business meeting, you don't want to talk about the percentage of users who have referred at least 1% to our product over the last 6 months. That's a bit of a word of mouth. Instead, you talk about average recommendations per month or something like this. Or recommendation rate score, whatever you want to call it. And then you try to connect the dots. And now I have a story to tell. I never produce any keynote presentations or anything like that for presentations about metrics because I'm just guiding people through this. I'm telling them a story. For me, this is a storytelling tool. So I'm telling them that story. Well, our goal is to reduce time to purchase. We've been thinking that to do that, we need to make it easier for people to make a decision and make that decision faster. To make that decision faster, we need to understand what drives people to a selection of one candidate for the job or a selection of candidates that are relevant to them and at the same time, we need to make sure that they can actually make a decision relatively quickly as they land on our landing pages. And to do that, we decided that we're going to work on a bunch of design initiatives to improve filtering speed, to improve comparison speed, to improve this and this and this, and this is how we're going to measure that. Right? To understand if our filtering speed has improved, we're going to check the average time that people need based on our analytics to set up for 5 most frequently used filters successfully in the first session that they're when they come to the page. Right? Or, for example, we'll be looking at the percentage of new users who apply at least 5 filters in the first session or anything similar to this. Right? And so we have the specific, very low level, right, design initiatives to do that. Right? And the reason why we're doing it in the first place is because we know from research that we conducted whatever, right, that 7% of users abandon landing pages within the first 30 seconds. Right? Some of those things might be coming from research. Others might be coming from very strong assumptions. Right? And that's fine. We will need to just mark them differently. But that's my storytelling tool to connect that business call with design initiatives.

And, you know, why we're doing it in the first place.

Okay, does it make sense, everyone?

Right? Now, this does not take weeks at most. It would take you something like 2, 2 and a half weeks or 2, 2 and a half days once you've done the prep work. Right? So you need to understand what you want to do. And by the way, so this is not something that one, you know, one person team would do. Right? If you were the only designer in the team, of course you can't do this. But if you're working in that case, it was actually a group of designers in different units. Right? So they were working on different parts of this simultaneously. So you, on your own, might have only, you know, these two columns and this is your entire tree. And that's okay. But, you know, what you're doing, you know how you're going to measure it. You know how you're going to report it. And you know why you're doing this and what you're trying to achieve here.

Erin is wondering, what do the icons on some of the cards mean? Well, basically, to me, that means Gauche is actually like metric.

This is something just an indicator for me that this is a KPI, something that I'm going to report. Right? Because you might also have multiple metrics in here that you're measuring.

Right? And then you kind of go from there. So the icons are sorry about that maybe. Relatively randomly picked. I just tried to make it more or less meaningful.

Sarab is wondering, do we always need to have statistically significant metrics?

Do we derive a bare minimum metric in situations where journey is mapped on a short timelines? And indirect sources like 4 or 5 members of sales teams and so on? I will get into that, but I'll leave it closer to the end because when it comes to explaining the impact that you're producing and why you want to do certain things, especially when it comes to prioritization,.

It depends on the company very much. Like, if I go in and I want to be statistically significant, right, well, then obviously I need to go and go all the way in the quantitative testing and that means surveys and that usually going to mean I need to bring in a lot of people here. So, for example, this is a wonderful little tool that I use a lot. By the way, which is called Sample Size Calculator. And if you, let's say, you know, targeting 10,000 people, right, you need to understand what they mean by statistically significant. And actually, for a long time, I had no idea. I mean, I don't have a statistics background. I'm not a data scientist by any means, but I need to understand what is expected of me when I start talking about numbers. Right? And what is expected of me is that we have a margin of error of 2 to 5 percent, which is industry standard. And the confidence level of 95%. So if there are elections, for example, happening in your country, right, and you have a candidate getting 30%, right, usually it will be this data, the candidate has 30%, will be also mentioned.

The margin of error. If the margin of error is 2%, that means that in reality that candidate had between 28 has 28 and 32%. So 30 plus minus 2. And so that's an industry standard for margin of error. This is the plus or minus figure reported usually about how accurate your data is. But then you also can get a confidence level. Basically showing how confident you are that this output that you get is correct. So, for example, if you want to go statistically significant, right, this is going to be quite a journey because if you have, let's say, 10,000 users, right, and you want to be statistically significant, right, well, then you need to bring in 2,000 responses right giving that most of the time you have something between 15 to 20 percent participation rate in the survey at best. You actually need to bring in all 10,000 people in this survey to get any statistically significant results. You can, of course, relax it depending on the company because in some cases you could say 5% is also fine but then you still need to have 370 respondents which means because it's usually makes for 20% we need the 5 fold the number of the people involved and those people shouldn't be random right we need to segment them properly there must be representing the user base and it's not easy at all so I always try to find a way to articulate that.

Qualitative testing right with bringing 15 to 18 people as I can also show you if you're interested can be it is not statistically significant but usually after 18 every 18 people it becomes stable you know maybe I should show this I mean I'm navigating back and forth but I hope it will make sense well you know what I'm switching all the way but let me just show this so as you have as I mentioned I working with the European Parliament right and in there one of the things that we had is we were working on something that we call TPI and this is the spreadsheet that we worked on and the way it worked is very simple just to put all of this into perspective right kind of moving straight to metrics so this is one of those hold on let me just double check here we go this is one of the notes from participant number 16 on June 4th TPIs task performance indicator and we're basically tracking how well people are performing task so we went in in that environment and we gathered tasks from search from stakeholders not from me right but from stakeholders representing what are very specific tasks that people are doing on the European Parliament website first of all I have to ask you how many of you actually going to European Parliament websites at least every month.

It's always very disappointing.

Maybe Tamarind does.

Okay then there was silence. Fine. Fine. I'm not judging at all. Right. So we basically what we do is we look at the stars that people are performing there. Right. And we look at how successful they are. We look at whether they start with navigation and search and whether they success and or fail. So let me give you an overview about what the report looks like.

So report will look something like this here.

Right here. So let me give you just the overview of this task that we have put together here.

Hold on just a moment. Here we go. Task. Okay. This is not the one. It's a secondary draft. That's right. So let me just move this here.

Okay. Here we go. So what we get here let me just scroll a little bit. We will get tasks like this here. Like for example what date did the European Parliament adopt the Treaty of Lisbon? I mean I know this is a question that bothers you every day you wake up right but this seems to be like the simplest question of them all. I mean how difficult can it be? It's almost like a homeschool school question right well the correct answer is February 19 2008. The success rate rate for this task among 18 people sorry 20 people that we invited here which we of course representative for the audience that we're talking about was 35%.

You could say uh how can it be this is just so weird and strange well because Google was lying to people well not lying just giving a wrong answer and this has happened more often than you think not because of AI by the way but because Google doesn't always distinguish between things like retify adopt agree sign on and so on it might seem like they're similar or the same but they are not right and we're talking about environment where every word matters right so of course people would often go to Google they start on the European Parliament's website but then they would go to Google and then they would get us the wrong answer even worse than that if you look at some of them which French national law has implemented the general data protection regulation GDPR anybody knows it by heart like maybe you looked it up yesterday on the European Parliament's website or.

No so again the success rate was very very low nobody could find it right and so we look at what is happening here now there again there are 12 questions like this and so 12 tasks like this and so then out of that for each participant that we bring in and we brought in all together I think we invited 24 25 not all of them could come but here is the data right from all of them we measure or we track you know search or navigation do they use primarily search or primarily navigation we track if they fail or succeed what does success mean here by the way what does it mean

well exactly to get the right answer not to get the right page right but to get us to the right answer right so we track if they're successful not and we also track how much time they need for it basically I have my iPhone 5 I take my iPhone 5 I click on a button when they actually start and then I click on a button when they finish and that's the timer there are two people on every call this is moderator testing two people on every call in the in the call there is the person who is a moderator and there is the person who is a note taker right the moderator does not speak to that person who is participant right no because that would influence the timing right and you know often people want to know the truth so they would come they would kind of raise their hand and say but did I get it right it doesn't matter that's not the point but did I get it right but it kind of gets going like this right but eventually from this we gather all this data and we know that across these different tasks people had different tasks success rate right which brought us in the very beginning to 46% success rate total median time on task around 2 minutes 30% use navigation 70% used search right now with this then uh basically this year later this year we do another round of testing after we implemented the new navigation and so we broke it down into class and content success rates content success rate meaning people give us a wrong right answer class success rate means people landing on the right page and we also measure the time the same story right plus we also added few more tasks because we also worked in some other sections of the site so as a result of that right we look at the success rate the success rate for navigation is 81% I'll talk about it in a moment because the industry standard and this is where you want to be for success rate is beyond 80% right but then the content success rate so people giving us the right answer was only 52% right and then the total median time on task was 1 minute 9 basically within that year of our work while we improved navigation revamped it all together what happened is well we brought people from success rate of 46% to success rate of 52% which is not that much given that the content hasn't changed it's not very surprising right but when it comes to getting to the right page right 81% is a good number however what's more interesting to me right the total median time on task here was 2 minutes in the in initially and the total median time on task here was 109 so basically we just help people fail faster which is not necessarily what we're aiming for but the goal is once we improve the content that people will be getting to also correct answer faster and now we have a definition a notion of success right but before you fall asleep with this who doesn't want to see more spreadsheets in their life right I want to give you kind of a more structured approach to this that will hopefully make sense um I know that there are a lot of questions coming in so if you could maybe stay until the end I just don't want to lose too many people uh at this point so let me just explain how it all works and what we do with this now the important part that we kind of need to keep in mind and this is kind of where I I just want to kind of make sure that we don't do the same mistakes what does it mean to measure things at all well usually it means a very simple thing it means that you need to show the cause and effect of whatever it is that you're producing right however you might have heard a thousand times already somebody telling you well hold on for a moment correlation is not causation right you might have heard this before do you know what it actually means.

Especially in the world of metrics right well causation means we did a b happened right and there is a direct connection between them one influenced another right but then you might also have of course a correlation which means well we did a b happened but not because of us but because of c right so there are other factors that are influencing that as well there's a wonderful little illustration uh right here on Claude Claude I repeat Claude not ChatGPT right which basically is all in to showing that you know how it could work right so usually this would be called confounding so there is something else that influenced a or influenced b at the same time so it's nice to see this like you know you basically have a direct causation between a and b right so you have cause and effect here however very often data in analytics in data science you'll be calculating this correlation score this r score so that score the r score usually must be at least more than 0.5 to indicate that this is a good correlation right but it doesn't mean that it's actually causation right but then what you also will find of course is that in situations like this here you might see you know we are mapping shoe size against reading level I mean this is ridiculous I know but this is what a lot of designers do when they start talking about measurements right because you might be saying okay well filters over here conversion rate over there right that's a really bad story to tell because it maybe does not count for things that are happening outside of your control right and so that might be that I don't know of course the age there is age that actually causes both right and somebody will be very good at sporting that and pointing that out so that story must be much better so first of all we do need to kind of make sure that that what we're measuring here in somehow fits into what people uh what is um kind of producing that effect now why does it matter James that KPI template um I think it's already available here on the Google Doc if you'll find it here somewhere by the way um design KPI 3 KPI 3 this one so feel free to use it here here right uh now the reason why I bring this up is because I need to know what a tap into so please don't leave just yet because I think that this is the most important part of everything now most of the time when we think about what is important for the business these are the common strategic priorities that you will find right we want to increase revenue we want to decrease costs increase new business increase existing business and also increase shareholder value however we need to really understand it much more broadly than we usually do when we think about revenue typically we think about the money that comes in it's not quite necessarily the case right because there are many different sources of revenue and one of the best ways to increase revenue is to increase loyalty for example and as a designer one of the jobs that you usually have is to build trust and confidence and increase loyalty as well loyalty is what drives revenue in the end also when you think about cost we usually think about the highest cost that the company has usually people right but also could be you know software tooling organizations slowdowns and things like that right no this is not the full story either what is the highest cost that every organization has in terms of you know not just people but specific domains or specific areas that are usually the most expensive ones in every company in every organization time that's a good point you know anything else stuff.

For sure

Infrastructure remote collaboration resources logistics very good yes you will find that a lot of time the most I mean depending of course on organization and the company you know it doesn't matter by the way if it's nonprofit or if it's a public service the there is a strong desire to decrease costs increase efficiency for sure right but the point is that very often you might have very expensive sales right where the cost of acquisition is just incredibly high especially in between can be incredibly high so if you can tap into this and really make sure that people understand that you're contributing to something that affects the costs of it and I will show you in a moment how this would be very very very helpful right and the others you know they're kind of obvious so why does it matter and how does it actually apply to you very simple like my little hack if you like to make it work a little bit better is to tap into what organization is usually measuring and for that I need to understand how they what exactly they measure because there are of course a lot of different metrics that can be tracking right some of them are strange to say the least right for example if I think about some organizations I working with sometimes you will find people obsessed about things that are a matter of interpretation like for example sometimes I would be in a company or an organization working with teams that feel like a success is increased session duration so.

There will be celebrations of you know we jump from two minutes 47 to 311 so what what does that mean so what happens in this 311 minutes three minutes 11 seconds what happens in 247 what does it mean are people confused and they can't figure it out so they stuck or they are basically know exactly what they're doing and so they just engage more even when you look at let's say social media right even that of course everybody wants engagement and everybody wants to keep people on the platform right but that's basically just people wasting time in enterprise world this is just incredibly decremental you want people to complete tasks faster but that's not the only requirement right because you want people to be efficient the other requirement is that there is a high accuracy as well and that there is a very little you know feedback loop in terms of errors and recovery from errors and stuff like that session duration you can never just measure session duration and you know have it defined as a definition of success that's just very strange right on the other hand also traffic growth that's easy if you throw a lot of money in Facebook advertising and LinkedIn advertising you can bring people on your site that's not a problem but you don't want to just grow traffic you want good leads you want maybe five really good leads a day rather than 5,000 people who don't even interested in your domain just because they're coming from a LinkedIn advertising it's very strange and then even click through rate well if people are clicking through a particular flow that is not necessarily driving any benefit to them or to us you know what's the point if they click through but then they abandonment on a shopping cart page let's say is tremendous so what's the point of higher click through rate can I to understand where things are not working and where they are working something some more obvious like diagnostic metrics system uptime security incident rate and stuff like that sure this is very important right but on the other hand also we have a lot of business metrics and I think there are two that are really important on the one hand we talked about Northstar metric many organizations cover net promoter score right NPS which is a horrible horrible metric which is very easy to game by the way right but and that's something I have to mention if you're familiar with NPS this is just you know how likely are you to recommend something to a friend a colleague it's a horrible metric very easy to game hides your success all the bad stuff right but a lot of businesses measure it for historical reasons because it's a proxy to loyalty right and in fact there is a study for my belief 2016 which showed that there is a connection causation correlation again right this is a connection between the NPS score and revenue right so there is that there is an academic paper on that but also and that's something that is usually not being told there is another paper from 2017 I will need to find later which basically showed if you improved usability and accessibility and UX NPS tends to go up as well right so you can try to map your success into NPS too but I wouldn't even try it's just a really horrible metric to use but what's more important is that a lot of organizations measure customer lifetime value have you heard of this before

it's CLV or CLTV because why not we can have both abbreviations because life is better with more abbreviations in our life right so off we go we have CLTV and that's magical I really want you to encourage to encourage you to think on this and and consider this and what does it mean again well CLTV guess what it's a customer lifetime value which means it's a customer value multiplied with the lifetime right that's it I mean that's not very difficult right so customer value multiplied with the average customer lifespan great okay and how do we measure customer value then you might ask great question I asked it myself as well well for that we can go in and look into average purchase value right and average frequency rate this is the official formula if you like of how it's calculated CLV is the average purchase value value so what's the average purchase value that you know people buy like for right how often they buy and how long they stay customer does that make sense

the tricky part is this one because what does it mean average customer lifespan how many washing machines have you purchased in your life

one two let's go for the record here zero James you don't believe in washing machines

okay one two all right excellent so if you let's say bought a washing machine of a particular brand right and you used it for six years are you still its customer I mean you probably don't purchase another washing machine for a while right so what's then your lifespan I mean different teams define it differently sometimes you're talking about um you know somebody who is exposed to for example signing up for a newsletter or is connected to in some way right to the brand right and so as long as they are interacting with the brand they are still active customer and as long as they're active that's their lifespan right there are different definitions of that that turns out to be quite tricky at times right but this is quite obvious we can measure that for a given user customer that is obvious we can measure that that's quite obvious right as well and that as well and this is what is driven by loyalty as well so why am I showing this to you this this the gold mine for me because my work right in terms of design right fills in I try to always map into my work helping increase average purchase value if it's let's say e-commerce right increase average frequency rate if I can and increase average customer lifespan through loyalty and good experience and good support and everything and this is so important because once you have customer lifetime value then you can measure something else which businesses are often obsessed about by the way which of course the customer acquisition cost right because if that customer acquisition cost is too high right cost per acquisition here if it's too high and it's higher than customer lifetime value then the company really has a problem right but if you can increase customer lifetime value you uncover tons of resources right that can be they invested into acquisition if needed so a lot of companies will be absolutely I mean delighted to see you making a strong case that you actually someway contributed to customer lifetime value because you can right just to kind of make that point here but then again when I look at the side of kind of at the UX side of things right kind of need to break it down right to something that I can work with so lifetime value is something that I always going to work on but there's nothing that's better than tars I mean I don't know if you've heard of this before right anybody has seen or used tars maybe in in this is absolutely like just really fantastic it's very compact it's very simple it's very straightforward and I try to use it as much as I can so this will be slightly technical so please don't run away although I see Oksana running but not running away hopefully here we go uh so with this tars is basically a combination of four metrics it's target users active users retention rate and satisfaction score all of it used to evaluate the impact of a particular change that you're producing features change of flow and stuff like that and also measure success so let me explain to you how this works it's actually very straightforward first part is T being target users so we quantify target audience basically asking what percentage of all our products users have that specific problem that our feature aims to solve right that's T by the way in general this would look pretty much like this so we have a target audience right maybe 100% maybe 60% doesn't matter we'll look at how many people of those that we targeted actually have adopted the feature adopted meaning they used it at least once then we'll look at from those who adopted the feature how many actually retain customers or retain users of that features that they keep using it over and over again and then we'll look at from those people who have retained how many unsatisfied very simple funnel right but then you have basically three metrics the adoption rate already available on the business side of things or product side of things retention rate already available on the product side of things satisfaction score well that would be your first UX metric if you like right now how do we do that well again the target we can choose depending on you know if you're targeting a right segment or not adoption is a bit tricky with adoption we're asking what percentage of active target users actually use our feature to solve that problem now sometimes you will be very surprised that this number is very low sometimes it can be very high right now you should be expecting something between 40 to 60% if you actually solving a particular problem that users are having right if you have studied the problem space enough if you just experimenting could be as low as 5% right but that's important for us to understand because if that number is very low like for example 10% it might have nothing to do with the design of that feature why you did an excellent job you did everything right that adoption has nothing to do with your design what can it be depending on what can it be kind of what can this drop in or very low adoption be uh influenced by time.

Bad customer experience wrong persona wrong target audience very good high price really has nothing to do with you marketing usefulness not solving a problem lack of value that's a lot of stuff here awareness yes very often people need first of all to go through this journey of discovery they need to be able to discover that feature they need to be able to understand what you're proposing there they need to be able to understand how it's going to replace the existing way of working they need to spend time trying that feature they need to spend time trusting that feature or building trust with that feature this does not come overnight unless it's a very simple feature that you're producing right so this might take months until people go through all of those stages and even years later you will still find people who are using the old way of working despite that feature being around for a long time so that adoption is a journey right and that requires discovery it might be that you just have you know something like a bad label or bad naming or bad positioning or you know you heated somewhere in filters and people are not even looking there right and so this might not have nothing to do with your design right and then you kind of end up with the situation where you might not have the uh the right adoption rate now Mark is bringing good point no hockey stick it happens as well so if you do solve a real problem that a lot of people have and complained about they will try things out so if they're very frustrated with what you have they will give it a try and then you can have a hockey stick right but a lot of times you find it not getting there unless you're solving a really big problem right so you kind of need to wait and see what's going to happen at least like if you see let's say again 10 or 15% adoption of the three months or two months that's sign of something is really off here right and then you have retention and typically when it comes to retention we're just tracking how many people reuse and keep coming back to that feature to use it again right typically high retention is a significant um thing for us to keep in mind because it means that we have a relatively high strategic importance I mean this is not something that I define but the creators of tars have defined as that seems like if people really you launched a new feature and has a very high retention that means that that feature is important right and out of that finally we get satisfaction score where we basically asking a very simple question this is the simplest you know survey prompt you could have like for example from those people who actually keep reusing the feature we ask so overall how easy was that to use or how easy was it to solve your problem after you use that feature today they go through the journey they use that feature we ask them was it difficult more difficult as expected much easier than expected and so on so you have this funnel right if you have big drop-offs in between you have a trouble so you probably need to look out what happened if the drop-off is like 10 15% that's not an issue if it's a drop-off it's natural but if you have a drop-off of 40% or more that's a real problem then you know what to work on it so why am I showing this to you now because now we can do some math right now we can say what if we calculate s by t score when they are right

just a second everyone all right thank you right I'm just going to finish with this in just a second everyone and I will be staying until I answer your last question I promise right but basically we can say well maybe we're targeting 80% of users from them we found that 50% adopted in the first three months from those who adopted 70% retained from those who retained 90% are satisfied and then we multiply all of this this gives us 25.2% of what

what does this number represent

success success well jobs to be done Matt yes so from those customers that we're targeting like in this 80% of the entire user base 25.2% are very satisfied with that feature right that's success on that local level because we're targeting specific group because remember what I mentioned at the beginning we are working on filters we measure filters we're working on feature we measure that feature if you don't measure that and you only measure the overall global impact have very little chance to create a nice story to tell and out of this you can flip it and divide the kind of this um the success satisfaction rate by target base and you get this number and you can say sure numbers are great well the reason why you would do this is because then you sit down with your product manager and you map out your features against s t score and their level of retention remember what I was mentioning if you have more than 50% strategic uh retention that strategic importance right but also sometimes teams would define business strategic importance differently right uh the name of it is ananya tars t a r s and then you have four quadrants right you have some quadrants that uh one quadrant that you could call core features so you have some features that live over here very high strategic importance meaning very high retention people use it a lot right and also very high satisfaction rate right hmm so if people really use it a lot and they are very satisfied that's amazing the worst thing you can do is to break that flow or break these features they are becoming the gold mine of everything and now you know what works right on the other hand you might have something that lives over here people keep reusing it a lot but they are very unsatisfied why would they do that everyone imagine yourself why would you hurt yourself using the same flow the same feature over and over and over again with a very very low satisfaction rate why would you put yourself through trouble and horror of it.

Very good chain people sometimes cannot choose right they have to use it they have to use that flow they have to get through it they know it's horrible they need help right so this could be perceived as a liability feature it might be a very good idea right if it's an important flow and it doesn't affect just uh you know a small niche right usually this would not be the case because we're kind of having s by t score right here so if we have this being a big problem right uh we probably might want to fix it if it affects a lot of people and then of course we have the other two like for example this one where we have two features right and they would be called project features because you know they have low retention low satisfaction maybe they're important maybe we need to remove them maybe we need to replace them probably not going to be priority and then this is becoming interesting why would you have this where you have very low retention very high satisfaction what does it mean just.

Totally overperforming right why would that be like very low retention very high satisfaction what could it be a sign of.

Engagement discoverability on a need to visit once to complete the goal maybe so you don't have to use it a lot when discovered it's highly used ah very good Sara uh expert feature very useful but not frequently needed things like that it could also be that you just picked the wrong target audience and you discovered something that is specific segment of your users can heavily benefit from but they are really really happy with it maybe there is something else that you might want to study and explore to broaden up that feature or that you know product or whatever it is that you're working on to make it more available to a different audience or maybe adjust it for the needs of other audiences as well right and now just by having three metrics three metrics right you can have a meaningful conversation with your product manager looking into what should we be working next are we happy with the way things are what should we be fixing what should be our priorities and this brings you kind of to a place where you can feel slightly more comfortable and for me this metric this tars is one of a few that I would be using I just uh three more minutes everyone and then I'm done and happy to drop in your questions right because this is a toolbox that I'm using but that depends on the environment I know that many of you might be working in enterprise world where you don't have access to users who may be very limited access to users so then my job is to find a way to get as close as I can to people who are talking to users or I would say customers or end users right so in B2B this is probably going to be the case on the other case on the other hand what is always quite helpful also is to really recalibrate your metrics right so this is just a few that we have established over the years right but I also have more in the Google doc listed but what I think is important to understand is that when it comes to B2B or corporate world enterprise world we need to speak a different language right so there we are looking at efficiency we're looking at failure tolerance we're looking at error recovery and things like that right but also and that's also very important uh you can never you cannot always frame a task in a simple way like for example a lot of journeys people take in some enterprise products are not linear they have different entry points people go through loops people go back and forth they switch up and down they have different views which they have to navigate you can't just give them a task and measure time that they need to complete it right we need to see just how successful they are in general so time might be less of an essence there than actual success identifying a problem fixing things and so on and sometimes you can measure it only once your life in production if you have analytics and logs right using hot jar and you know full story and others but my toolbox looks like this for a given project I will never pick more than four metrics right so again if I'm working let's say on filters I'll be measuring filters but also I would usually define a very small scope maybe five six tasks that would be related to filtering and to bring 15 to 18 people to see how successful they are right and then measure that but also you might be want to measure things like search quality score if you're working on search that's only reasonable and sometimes if you're working let's say on uh forms right we are looking mostly not even at the success rate of form submission but rather the accuracy of the input that people are providing right and then there are of course a few others for example here on the right you find a lot of surveys right here right some of them are very specific right very often it's a good idea to just track the frequency and the volume of service desk inquiries right and if I start usually you don't need more than this this could be a wonderful beginning of your measurement journey where you basically establish a very simple spreadsheet right which has you know not that much stuff it has three different metrics defined right so you have a success rate for maybe four five six journeys that are very important right again whether you actually talk to customers and get the insights from them uh or whether you run tests with them or you look in analytics you need to get to the some point how successful they are right and so you might have this and again just for reference whenever you go beyond 80% you're in a good place whenever you are below 60% sorry about the typo you're in a bad place you also might want to send a little survey if you like right via email or preferably within an app or a product after people try to complete a particular flow right and see how successful they were by asking how easy it was to use right this is called SEQ and that's also fine right and you basically get to the point where if you above 4.5 you're good sometimes you also will find companies using UM UX slide which is the ability metric for UX slide which is basically two questions or two statements that you agree or disagree on the scale from one to seven I believe uh from zero to seven sorry the systems capabilities meet my requirements and it's easy to useRight, there are a couple of guidelines and little cheat sheets that I'm using right. There's obviously the notion of leading indicators and lagging indicators, um, this is a nice view of this. Just in case you've never heard of it before, leading indicators are predictors of success, right, and lagging indicators basically telling you how successful you have been. Typically business metrics are lagging because you can know if there is some impact on revenue or profit only once you made some changes, right. But if you, let's say, improve filtering experience again—sorry about that—thenWird. So it can become legging then as well.

And also, one thing here: all the way at the bottomYou probably want to see how you're doing, so that means that this critical task that people are supposed to be performing must be established already. That's a threshold you absolutely should not cross, because if you spend work and it gets worse, this will backfire.

Right, then as you keep designing and refining, and it's okay to kind of launch something and see that there is a regression, right, but as you keep going, hopefully it's going to get better and you just need to measure it. Usually we measure at least once every 6 to 8 months, but that depends on the velocity of the team, on the organization, and so on. And also, there must be some sort of a target state that you are aiming for, again, for key tasks that you want people to perform, you want to be maybe 80, 85 successFor me, that's weird.

No, please don't—

That's nice.

Recording in progress.

Interesting.

Sorry, we're— Okay, hold on.

Okay, that's pretty strange. So, sorry everyone, I— I've just realized that my machine has, um, hung up. So I'll be right back in just a second.

Okay, sorry everyone. For some reason my machine just decided to crash, so I have to restart it now. I'll be right back in, like, a minute.

Sorry about that, everyone.

That was fun.

And if you have to re- have to jump, that's of course not a problem at all.

But for some reason my machine is not befriended me today.

Okay, so I just relaunching. Here.

What a fun time to be here. So, sorry about that, everyone. So I'll be just coming back in to answer any questions that you might have.

As well.

My machine decided it's probably a good idea to just stop.

Uh, for some reason. That's very strange.

So just a second.

Everyone.

Feel free to drop in your questions in the chat. I'll be answering them in just a moment.

Oh.

Just 12 minutes.

All right. Well, fantastic. Well, what a pleasure. It was, like, just the perfect timing. Just perfect timing. But you're still here. That's very surprising to me.

Welcome back, everyone, to the continuation of our journey here. So let's see how this goes. So first of all, thank you so much for your patience, everyone. Uh, let me start sharing the screen. Let me just dive into questions. I mean, from my end, I think maybe one final note I wanted to share in this. It seems like in the beginning, especially for designers, right, it might feel like actually measuring and establishing some sort of metrics is extremely technical. No. It's not that difficult at all. I mean, the question is— and this is something that I would always encourage everyone to think of— what can you do in an organization that would not require too much effort on your side? And that you can sell easily. I would ask myself, do you get access to users? Can you?

Can you get to 5, 6? Like, such small. 5, 6 people that you can actually speak to. And all I always start with the same thing. I always say, you know what, just guide me through this product. What do you see there? What do you use every day or every other day? And they start having the conversation, explaining to you what they see in front of them, right? And then you can kind of learn quite a bit from what they're doing, what they're using, what they're not using, and so on. That's always a really, really good, uh, kind of thing to keep in mind.

And you do this a couple of times, and different people will tell you the same story, or different story, that depends, of course, on how they use the product. And get a kind of a bit of a sense about what you can do. This is a great beginning. If you don't measure at all anything, right, but you then will spot some relatively important things that are actually worth, maybe, working on and kind of move to the proper space to then kind of understand and so on and so forth. If you can't do that, well, what else can you do? Desk research you can do. Looking into Google reviews about what people say about your product could do something, right? Could be also quite helpful.

Also, again, talking to customer success, talking to customer support, talking to sales. And, you know, one of the things— and this is very ironic— one of the first things I usually do is setting up— and this is, like, how boring my life has become, I think. The first thing I do, I set up working groups. The very first thing I always do, because there is nothing that's more impactful than sharing your ownership with people who care. The problem is that usually you have two types of people in every organization. There are your stakeholders, for sure, but there are also key influencers that are usually invisible, and very often you don't know of them. But these are people who maybe had some success with UX in the past, and maybe they would love to help you support you and push for you, and even find sponsorship for you, or support for you, if they even know that you existed. Right? But for that, you need to create some sort of connection. The best situation you can be in is when you say, well, we decided that we're going to do this thing, and this is why we're doing this, and this is the KPI tree, and all of that. And by the way, we also consulted our stakeholders and also this person and that person and that person, right, who basically is, you know, pushing for this as well. So if you have people on your side already, that's incredibly powerful story to tell.

It's very difficult to reject on spot. Right? So there is that.

Also, by the way, what I've forgotten to mention, entirely, because of course I'm crashed. I also have this little thing coming on on Maven. If you're interested, let me know. If you— if it's too expensive or whatever, just send me an email. You'll find my email here.

And, you know, obviously I would like to make it work for you either way. Um, but there are also more coming in. I mean, this is where we actually go in and we dive into all the specifics and all the detail from scratch. Right? Really building out those KPI trees for your specific use case. Not for my lorem ipsum project of kind, right, but for your project, whatever it is that you're working on, as long as you can share it, because it might be under NDA. Right? Um, and if you'd like to, I would like to see you there as well. Um, beyond that, let me see somebody's asking about the mirror link. All of this is here.

So feel free. Please don't release it, like, on LinkedIn or something. I have to clean it up and double-check. Um, but hold on. Is it the right one? I'm not sure.

Wait, wait, wait, wait. This is not the right link.

This is a different one, but this is the same anyway.

The— oh, the mirror board. I didn't share the mirror board with you.

Did I not?

Wait, wait. Now I'm totally lost here.

So the link was maven— mave26, right?

Something.

Yes.

Very confusing. So that one.

Also, while on it, finally, I also put together this UX strategy canvas. It's a bit broader. Right? Because in there, what we're doing is, you know, usually measurement is related to strategic work as well. That's something also we do in the training too. Right? Which is basically sort of a— UX strategy canvas. It's not quite finished yet, so I haven't finished releasing it. I will release it when it's finished. Right? But if you find it useful, meaningful, you can also use that, of course, as well.

All right. So jumping to your question finally, everyone. Sorry for keeping you, uh, here. For this, let me just start from somewhere.

Oh, Carys, thank you so much for watching it from Singapore. I think that's fantastic.

Uh, if.

Can afford buying 5 coffees, then you have access to users. That's wisdom coming right here. Uh, yes, this is going to be recorded, and it will be available. Right? Let me just double-check. Erin is saying, "I like the metric structure you've provided. I measure weakness in organizations is NPS or reliance, because no one has adequate result in other point of view." Absolutely. I would say— I mean, we have to be honest here as well. I've never seen a single company that measures only NPS. There is NPS and dozens of other metrics as well. And NPS is something that is easy to compare with competition, because a lot of data of that kind is actually public or available. You can find it.

So I think I guess I— I understand why companies measure NPS. But this is never the single metric. And so I'm never kind of starting to hang out on it. Right? Instead, sure, I'm looking at what is closer to what I can deal with.

Same is same: I can't claim percentage-wise that way as it is in quantitative data, though. Hence the question, do we need to measure in a focus group of 50%, 50 users?

Well, I can give you a very— well, non-scientific, I would say, chart which basically shows you that you don't need to get more than 18 people for a particular type of, uh, testing. A specifically— let me see if I can find it here.

Uh, keynote. Open up keynote. Sesame. So this would be UX, uh, here we go. This one.

Just a second. We'll open up. Typically you are perfectly fine if you bring in 15 to 18 people. You will get a pretty stable results. Um, which you don't need to— you just don't need to bring more people in. Typically it will look something like, let's see if I can find it, like this. Is it here?

Okay.

Measurements.

Yes, I will have to probably this one. Yes. So typically this is what you would get.

Let me just come back here.

Right? So if you bring people in and you measure task success rates and also the time on task, typically— but I mean, it's very important that you pick the right audience. Right? If it's a random audience, then you will have random results as well. But if you pick the right audience and you give them, let's say, 5, 6 tasks, typically at between 15 to 18 participants, both time on task and also the success rates begin to stabilize. So you really don't need more than that. Right? And this is coming from the top task methodology, just in case you're curious. Top task methodology coming from, uh, Jerry McGovern.

Right here.

Right? And I'm— I've been having a pleasure to work with Jerry along the side, and he's a researcher as well. So, um, I very much trust his insights on this tool. Right? You don't need 50 people.

I mean, if you want— if you have access to 50 people, great. Yes. But usually you don't need this.

Right? Um, let me just double-check more. That's coming in.

What happens, uh, Clary is wondering, what happens if we design it to not have access to the data? Well, the question that I would ask is, what data do we have access to? I think very often the situation is a little bit different as how it's portrayed. So let me see if I can show you see you. Because very often the reason why you don't get access to data is not because you're not allowed to touch on data, but you— it's not clear to some people who decide whether you get access to data or not why you need it. So it could be very much that you just don't trust you. Frankly, or they don't believe that your needs are more than merely technical needs. To just validate something. And they don't want to disrupt a, you know, carefully orchestrated relationship with, uh, with the customers, especially if they're high volume of high-paying customers. So this is often why you don't get access to data. But if it's not the case and there are other reasons, it could be that there are no customers. Yet? Or you have NDAs or you have IP issues or anything of that kind, which is all fair. Right? But I would say what can I get access to?

Right? And sometimes, at least analytics is something that you can actually start with. A service that something you can start with. But there is probably— probably going to be a quite easy path to get some data by talking to customer service, because usually they're on your side. If you have them.

Right?

Oh, thank you so much, everyone. Uh, Himanshi is wondering, "Starting from scratch as someone not from data background, I'm not— I don't have a data background either, by the way. Is there something I can do to learn data analytics language directly linked to the business outcomes?" Yes. Two resources I can highly recommend. Uh, totally unrelated, but very useful. One of them is, uh.

Ryan Ramsey is great. Ryan Ramsey is running a lot of stuff, and he also has books. Uh, well, not books, but, uh, holds a courses. But also this wonderful little book, which is "Business Thinking for Designers."

Which is very good, I have to say. There is also another book, I think a lot of people are speaking about it, and it's kind of a little bit hyped, but I really like this book. Just to understand how business thinks, it's personal MBA. Some of you maybe already have it. And the third one, which I think is not something that you might be expecting, there is a fantastic little book on dashboards design.

Uh, let's see dash met David, I think. Dash data data dashboard.

Hmm. You see? So sometimes I do have to, uh, search on myself on LinkedIn because I always forget where it was. Just a second. Not this. I think it's here. So this is the post where I posted about it. But this is a great— oh, here we go. This one. For some reason it's, uh, unindexed from Google, so you cannot find it from Google. And it's also a bit broken. If you click on the homepage button, it just loops and disappears. But this entire book is available. And it actually dives quite in a lot of detail about metrics and how to track things, how to establish things, and how to basically bring them into something that's a bit more meaningful. Like, really going into all the fine technical details, which is very approachable, I would say.

So I found it very helpful. It's not necessarily, like, let's say, a book about metrics at all, but it's a very nice resource to have.

Right? Uh, let me see what else. Here we go.

Please do repeat your questions if you had them all the way on the top, because obviously they disappeared once I came back here.

Uh, Richard was wondering, "What is your approach in defining metrics for a new product where things are moving fast and there is no initial user base?" Okay, that's also a great question. So let me also share this. On the mirror board, okay, maybe I should show this, uh, mirror board. Here we go.

So I'm going to show you, uh, kind of a work in progress. Right? So what you have here— right— is something that we use that's called event storming. Some of you who are in this call will be remembering this: Artur, you remember?

Oh, yes. Who will remember? Forget that. Right? This is event storming. On an event storming, what I try to map out, uh, things that are most useful to address when you are—

This is also something you find here on the Maven board. It just empty because it's a template here.

Right? This is basically, uh, the best way to improve experience is to track two areas of improvement. First area is where people struggle, second area is where people succeed. Our job is to dial up successes and dial down frustrations. But for that, we need to map out where frustrations are and where successes are. So if it's a new project, I would take a look and say, okay, well, at some point the user comes to our product for the very first time, and there will be a journey that they're going to take to get to a successful outcome. Across that journey, there will be a lot of things happening. As you can see here, it's not linear. It's quite chaotic. Right? There is a lot of stuff that could be happening at the same time. And there are reasons for that. If you want people to get to a particular goal, very often a lot of things must work together. It's not a cust kind of linear journey. Right? A lot of things must work together. Similar to marketing. When people buy a book, they don't buy a book with a first time they hear about it, most of the time. They need to hear about it from a podcast. Then read a review. Then stumble upon it in mentioned somewhere. Get recommendation from a friend. And eventually there is absolutely no way for them not to get that book. I want to achieve the same for success of a person who is jumping in a in a, you know, product. Uh, and to understand how we can help that with that. So I map out two things. Where they usually get frustrated and what are the problems that they will be experiencing across the journey, where we need where they need our help. And where they can or will or should be successful.

Then once we do this, we need to prioritize. Typically the way of how we prioritize is by voting. I don't do voting. Because voting is very biased and it's not very helpful. No. I have every single designer in the room to walk through the pain points and walk through the success moments and pick maybe three or five of them. Three or five events that they believe are very significant and very important. So they do. And this is kind of what a group would look like. Right? They just start working on it. And then to each of the event, we consider four areas. We consider who needs to be involved as a stakeholders. We consider some design ideas and features and initiatives. We consider risks we also consider metrics. So every single event that the user is experiencing is becoming a blob like this.

Right? So we want to help people do this better. Or well. Good enough. Right? So there are some ideas that we have on how to make it work. There are some risks of things that can go wrong. We need to mitigate them or at least consider them. There will be different teams that need to be involved. And if we know that we have a better relationship with somebody, right, then we probably need to do a bit more politics in order to make sure that it actually does work. Plus, we establish metrics. Right? So we have this cluster. And every individual team or person will be working through this. And then we bring it all together. The outcome of all of these brought together looks sometimes like this. Here.

Where different people who have chosen independently. I really think, like, way too early most of the time we jump in the group thing totally eliminating the chance of people getting to the point where they can think and they have time to think. People need time to think. We often forget that. So people think independently. And then we create this mega clusters of the different things that people were thinking. Along with ideas. Along with risks. Along with stakeholders. Along with metrics sometimes as well. And so we have this. This ultimately by its volume sheer size of it indicates priorities. This is what a lot of people decided would be important to us to improve. This is the second thing that a lot of people decided would be important for us to improve. And then there are a few other things that people also wanted to improve. And this on its own is a bit of a filtering exercise. But also brings you to the point where you see what everybody is thinking without a lot of individual ideas being drowned in the ocean of, you know, compromises.

Right? And so then you can say, all right, we're going to work on this. Like, for example, one person here decided to work on transparent delivery and tracking. The other person here transparent information of what will happen when something goes wrong. The other clear product detail. Fourth clear product detail. Fifth the site speaks the same language as the user. Right? Things like that. And there are tons of ideas. That are already here. Right? Which I find remarkably valuable. And usually they would also come with a bunch of metrics that people individually thought of.

And again, this does not take that much time. It's like a workshop that you do with, uh, designers maybe one, two hours. And then I have to clean up and then I have to bring it all into clusters and do something with it. But then this is something that I can also talk with the product manager about. Right? And in terms of how we're going to prioritize it. It just there are different ways of prioritization. Right? Um, I use this, for example, you know, if I need to decide what I should work on, I kind of look first at the impact that we're expecting and the urgency. Very often you will find product managers and also designers estimating based on impact and effort.

That I find very difficult because what we're estimating here, you might have seen this before everyone, by the way, this ROI matrices, stuff like that. Right? I find it very unuseful.

Actually. Mostly because what we're estimating here is shipping. Or delivery. But not necessarily the delivery and impact over time. So sometimes you'd be using something that looks much more like this, which is called effort value curves.

I don't know if you've seen this before or not. Right? But the point is you will have different.

Trajectories, I should say, about when you choose, let's say, to work on the feature you know, you know how much time it's going to take in order to ship it. Well, at least you can estimate it. If it's a legacy project, it will take a long time. If it's a quick win, it will take a little time. And you have some expectations about the value it's going to bring over time. Along with effort, it is going to take over time. It might be that the feature that you launch will bring a lot of value. But also will create a lot of developments and design debt that you then need to clean up later. Right? So that I find helpful. But then also I still try to consider this. Like, how urgent is this? And how critical it is.

Right? What are we talking about in terms of, like, again, impact?

And so I frankly don't even know. Like, sometimes I'm wondering, am I still a designer or am I now a product designer manager or something? What is it that then my role that is? The important part that I shouldn't step on the toes of a product manager. So if anything, should be working with them rather than for them. Or instead of them.

Right? And it's one I'm glad you touched on building alliance. Oh yeah, that's so important. Very, very important. How to strongly advocate for CLV from PJ. Hello PJ. How to strongly advocate for CLV in a company that wants to start measuring NPS. Great question. I haven't answered to that.

So there is something that's called feedback score analysis. Feedback gap analysis. Sorry.

Which is an alternative to NPS. For me, this is the gateway to measuring customer lifetime value. Right? So this is a very simple way where instead of saying how likely are you to recommend something to a friend or colleague, you do the opposite. Where you basically establish five statements.

That are important to you for a given project. Right? And maybe what you're trying to achieve. And then you run a survey. And you see how people agree or disagree with those statements. Right? So basically also like all this point on the level from zero to five. Right? And you kind of map it out. Now, why does it matter? Because of course it will depend on the audience that you're targeting and stuff like that. Right? But it matters for a simple reason. There is a little Google sheet and form that you can use here. You can create obviously this sort of charts. Getting an understanding about where people are. What I tend to do is use this. But then I sneak an anti-NPS question in here. I ask have you recommended a product or a service to a friend or colleague? And have you recommended it to zero people, to one people, to three people, to two people, to three people, to four people, more than five people?

Right? Obviously the higher you go, the better it is. If it's no, then it's no. Right? So it's like NPS, but it's not really NPS.

So it's kind of I think it's kind of a way of cheating or hacking around NPS in some way. So there is that. And once I've done this, right, I try to create an understanding about lifetime value, what it actually means.

Right? Meaning how long are those customers staying. And I think about retention and kind of try to slowly build up the case. In fact, in one of the organizations I had to make exactly that case. This is why I kind of it's kind of burning memories about my, um, journey through this. I'm not sure if I succeeded or not, frankly. It was a very long journey.

But I know where you're coming from.

Thank you very much.

Yeah. My pleasure. Of course, PJ.

All right.

Uh, let's keep going. So let me just double check what else is here.

Oh, this session is being recorded. So you will receive it as well.

So I'm just lost, uh, track of where we were.

Hold on. Let me just double check. I have to see. Can I search through PJ? No, I think I'm not. Uh, just going to jump in. Uh, Georgie is wondering. Oh, hello Georgie. If you have many, many hypotheses and solutions, how do you prioritize? Okay. That kind of covered that, I think, partially. But there is a little bit more to that, of course. I mean, kind of wrote this article about complexity. Which is, again, specifically for complex environments. Right? But I think in many ways this is something I also try to keep in mind. So if you're looking into prioritization specifically, maybe this article will be helpful. So what I cover in this is one simple thing. Which is basically this. Here.

What was that? Uh, no. Here. So I kind of I mean, I don't know if it's right or not. Because this is just me doing things. Right? But to me, what's really helpful is to think about three things when I prioritize. And I'm not trying to be difficult. But I don't really I.

Don't want to underestimate the effort that is required to ship something that needs to be a part of a conversation. But before we even get there, I want to work on something that's going to have an impact. Right? And so for that, I'm looking into frequency. Of an error or task that people need to complete. Right? How often do they need to do that thing? Then I look at severity. What are the consequences if it's not working or if it's broken or something like that. And also frustration or effort. Not that we need to ship. But that users need to bring into the system to make it work. So I always try to consider those things before I start working on, you know, impact, effort, metrics, or anything similar. Right? That just seems to be much more reasonable. Because again, at this point, I'm prioritizing what hurts the most. Or where people succeed the most.

That's sort of my general journey there.

Right? Uh, Parth is saying I think a lot of metrics results come from longitudinal observations. For example, retention overtime satisfaction. How do we define how long something needs to be measured?

Yeah. So at times it feels like it's almost, um, a guess. Right? I would say that depends on the type of thing you're trying to influence. Like for example, if it's onboarding, it just doesn't matter or doesn't make sense to measure, you know, the how many people take onboarding in the first two weeks. Because they either do it in the beginning or they will never be able to kind of find it again. Maybe. Right? But if it's something like, let's say, slightly more complex flow, uh, then it might take more time. I would say.

Somehow I kind of think about either the first session. So that would be something like time to first value. Time to first success. Or it would be the first journey that they take. Maybe a bigger journey.

That would be representing some success in some way. And then the third level would be more global, which would be loyalty, maybe a couple of months later. If they keep reusing the same flow, and how often they encounter errors there. So I never think about years. Because you know, within a year, so many things can happen. But I would probably think about the scale of first session maybe first few days if they need until they go through the important flow and important first flow. And then loyalty long term.

That's sort of the scale that I would be thinking of here.

Uh, Kirilos is wondering. I'm a product designer and I love business and helping business grow. I'm targeting international companies from your experience. What can I do to show that has most value international companies look for?

I mean, typically I would always say that one thing that we definitely need is a way to explain the work of design not for the lens of design. I always try to almost forget words like consistency, or, um, I.

Even sometimes brand perception, to be honest. Because when I'm trying to make a business case, I do need to speak about how it's going to affect some of those things that I mentioned. Right? The live customer lifetime value, right, is being one of them. The value, the order, the frequency of purchases, the activity, the long term kind of loyalty, and things like that. Right? That's what I'm trying to impact. Plus, on top of that, reducing costs. By improving efficiency. Right? And also increasing revenue. For example, by improving conversion. Right? Or also clarity. Right? Or also search. And also filtering and also navigation. So I always have those things in my head. And I always try to fit into them. So I would never make a case for consistency, for example.

That's just usually not working.

Jasleen, I'm very happy to hear that. Thank you so much for joining in as well.

Right?

Uh, yes, here, too. If anything, please send me a message and I'll be happy to help.

All right.

Uh, Sarap is wondering, uh, how have you found that it's always possible to derive a math model to a manual value? No. If you think about brand perception, for example, you can't just say here's a number and the formula of how it's going to work. That's just not how this works. Right? Um, what you can do, however, I think, is to try to qualify it. What I mean by that, for example, if you want to measure brand perception, you cannot have a scale from 0 to 10 or something. But what we usually would do is we would send a survey where people can get a choice of 20, 25 different adjectives. And they can select an adjective that would be more representative for that brand or for that company that we are talking about here. And then we see what happens here. And we see how people perceive it. Like what adjectives they're choosing. But you cannot really connect it to a particular model. Right?

So there is that.

By the way, dear friends, I just noticed, by the way, that I'm a little bit behind schedule here. Because apparently we have another, uh, lightning session that I need to run in five minutes, which will be about something entirely different. I believe AI. That's fun. I totally forgotten about this. But if this is coming in five minutes, if you'd like to join in, this will be fun. Hopefully my machine will not crash then.

Right?