Transcript — Part A

First thing that's important to know here: once you start measuring—and sorry that I'm kind of keep looping that slide—once you start measuring, you become accountable for what you're measuring. Right? So if you don't have any metrics defined right now, everyone, then you know you just put pixels in the world and these pixels eventually land in somebody's system, product, feature, and so on and so forth. And so these pixels then are never measured, and in the end nobody can come to you and tell you, "These are bad pixels." Right? Or, "This was a bad idea." This is a poor execution. Right? However, once you start measuring, your impact, right, or once you're measuring, like, have you X metrics in place, if you're not performing and you're delivering something that does not produce the result that is expected from you, there will be questions. And if you do this a couple of times in a row, there will be more significant questions. Does that make sense?

So if you decide to measure, you better have a good reason why, and you better have enough, um, confidence that you can deliver.

Otherwise it's going to make you look bad, and also other people around you. As well. Right? And so if I want to measure something that would be actually making some impact, right, I kind of need to be very careful about what that is. And so we need to understand what that thing actually is, right, and how to choose it. For that, we need to understand what actually success means. You know, we often speak about that, you know, UX, the job of the UX, is to make customers happy, satisfied, and all of that. I don't think so. You can have a very, um, unhappy customers who are incredibly successful. If we look at Ryanair as an example, nobody enjoys a Ryanair experience of flying. I mean, some people maybe do, but I would be very cautious talking to these people. Right? I think, but however, what is important is that while they maybe not necessarily very happy with the treatment and the experience in case you don't know what Ryanair is, they're pretty much satisfied with the outcome of it, which is cheap flying. So successful being successful is much more important than being happy or satisfied. And then we need to talk about design and UX metrics and how they fit in the overall kind of scope of things. Right? Because they must work together with product metrics and business metrics. And then we also need to think about the stakeholders, the mapping of them, how to manage risks, and ultimately how to measure how often to measure. Right? And how to report success as well.

Right? Now, with this, I think it's just really a good idea to just keep going and start. And what—oh, hold on, this is something I want to kind of come to later—when we actually start measuring, we kind of get to a more strategic part, whether we want it or not. But let's first look into the impact that we're producing. Because I think this is often overlooked or misunderstood. And by the way, it's not going to be just me presenting. Don't worry. And I do welcome your questions. I would like to hear and see your questions, so feel free to write them in the chat, or just unmute yourself, but please raise your hand first. So this is a very open environment. I'm not going to do the formal thing. Right? So—but please do pay attention, because I think that those things are really difficult to explain in a way that would be—you know, in a podcast or so. You kind of really try to convey the message here. So some of you might be very familiar with this. This is a relatively old paradigm about what represents the layers of user experience, what represents user experience. Right? Now, if I look at you, you are incredible designers. Right? All of you. You are smart cookies. Each of you. Right? Very smart cookies. And if we look at the UX and what you're supposed to do today, the expectations about what a UX designer is supposed to be doing, right, have shifted. Now, in the past, I was—I myself was a webmaster, then I became a web designer, then I became a UI designer, and now I'm supposed to be something like a full-stack designer who knows everything from research to UX writing to MCP to AI to design to working with stakeholders to working with the business to understanding developers to debugging to looking into dev tools. I'm supposed to do everything. That's not fair. So in many ways, it feels like I need to be able to do everything. Right? Oh, I got muted?

Did I?

No?

Sorry. That was probably me.

Right? But then if you look at this model, right, we have this really relatively straightforward model where we have kind of the UX is moving between somewhere between concrete and between abstract. And all of it is considered to be UX. And on the top layer, we have the surface, the UI layer, the visual layer. I mean, very good at this. Think about the button.

Can you imagine that button?

A perfect incredible beautiful button. I think all of—oh, I think some people really hard those buttons. In reality, you are the button. You can create the most remarkable button ever. This is not going to be a problem. And probably it's not going to take days to create that perfect button. We can have our preferences around the, you know, border radius and stuff like that. By the way, that would be very curious for me to find out. What's the perfect button roundedness?

The perfect border radius?

Give it a go. I'm just curious. Oh, 25. You're being very generous, Tamerlane. Like, 25 radius button. 100? I mean, I'm not judgmental at this point, but I kind of am. I mean, 100 for button? That seems to be a little bit excessive. So 16, or somebody likes agreed. I can tell. All right. Anyway, but you know about that button. Right? And actually, when you look into the surface layer, we're very good at this. But then we need to think about the skeleton, the information architecture as well. Right? So here we're thinking about the layout, the responsiveness, the content design, and so on. Here we're thinking about information architecture. Like, how do we architect new navigation? How do we architect maybe filters? How do we group things? And so on. Along with it come interaction design and accessibility. And then as we keep going to more abstract stuff, we need to figure out functional, non-functional requirements, content requirements, strategy. Right? Business needs and user needs and map them out. Now, this is something that I learned—this is really like, what, 20 years ago? And yes, the recording will be available, so don't worry if you need to drop out. That's not a problem. This is something that maybe many of us have seen many, many years ago. Right? But now I think that if I think specifically about measurement, I kind of really like to think about these models instead. And the point is not, okay, this is a better model, this is a worse model. That's not quite that. But I always try to keep it in front of me when I think about measuring pretty much anything. So what we have here in front of us is a wonderful adaptation by Jamie Mill, which basically maps out three things. Reality, problem space, solution space. So when we're measuring, what exactly are we actually measuring?

So imagine you, you know, you worked on a feature. That feature has shipped. That feature has some impact on some users, apparently. And now you want to measure how well you worked on it, almost specifically the impact of it. Are you measuring reality? Are you measuring the problem space? Are you measuring solution space? Are you measuring something in between? What do you then measure?

If you establish some sort of metric.

What is that?

Right? So if you want to, you can write in the chat. Right? But for me personally, what I'm trying to measure is the impact of my solution—right—that I'm launching on the problem space.

Right? So what does it all mean? Well, first of all, we have reality. That reality is very complex, very contradictory, very evolving, and very difficult to understand. Right? I'm allergic to a lot of things, I have to say. At least four. I'm allergic to shellfish, unfortunately. That's a bit sad for me. Right? I'm also allergic to spirulina, which I discovered when I was on a trip in Toronto. That didn't go well. Right? I'm also allergic to customer journey maps, apparently.

That's—I will talk about a little bit later. And I'm highly allergic to ChatGPT. I don't know why I cannot touch ChatGPT. This is like really—I mean, this is it. I cannot. So by the way, we can talk about that separately if you're interested. But those two last things are out of control. ChatGPT is I cannot touch. Linear journey maps are problematic for me. Why? When we think about journey maps, this is a way that we developed, which is a good way, of course, to map out what happens in a product timeline. Right? User's timeline, or journey. So we map out the journey into steps and phases, and then we see what happens across each of these parts of the journey. Right? However, the reality is much messier than that. Because people are never following a linear path when they go anywhere. It's always a lot of going back and forward and sideways and here and there. What basically the journey map represents, be it the customer journey map or user journey map, what they represent is an ideal journey that a lot of people never take. That's an approximation of how we perceive reality, and what we think it actually represents. So in some ways, Claire is saying it's indeed way too idealistic and very often does not consider the shortcuts, the roundabouts, and everything that happens in between.

Why am I telling you this now? Well, because when we think about this, my job is to be able, in some way, even although I never have enough time for research, even although I never have enough time to make a convincing argument that we need to spend more time to users. Maybe sometimes you don't have access to users. You have limited access to users. But I need—in order to make any impact at all, unless I'm just operating on hunches, assumptions, and experiments—I need to map that reality in some more or less meaningful way into the problem space. Now, I'm just recently coming from a project with a power grid company that's managing electricity in the Netherlands. And when I look at some of the products that they're producing and designing, I have no idea what it represents. I have no idea. Designers, they're often go through training in physics for 8 to 9 months before they start designing anything. So if you need to understand the problem—kind of the reality—you kind of need to go into that level of detail. Right? If you really want to make an impact. However, we are living in reality which is often limited. Right? By—you know—our resources are limited, our timing is limited. We have to ship. We have to deliver. So what do we do then? Well, I need to understand user needs. I also need to understand the domain. And I also need to at least in some way observe behavior if I can't do that because I don't get access to users. I need to find people who are the closest to users to do that. The best way for me, of course, is customer service. But also customer success teams. Right? And also sales. Because they speak to customers all the time. So I really try to spend enough time—and that's important—not a lot, but enough time to start feeling that I can swim here. Because if I don't understand it, right, and I don't have a proper mapping, I can, of course, jump into solution space and just design. But then chances that this design is going to make an impact, or meaningful impact across the organization, are sporadic. It's random. Right? Because you basically just operating on hunches or assumptions. Now, I know—and because I've been there before as well—that sometimes you have to do just that. You don't have a choice because you're given a task and you're performing the task. I always have to smile when I hear about the double diamond because my life is reverse double diamond. Instead of figuring out what to work on by researching, I'm giving a task and then I have to validate it to make sure that it's actually fitting an agenda. Right? And so very often I'm thrown in the water of that solution space. But then I try still to get at least some understanding of what's going on here or here. Because otherwise I'm just designing pixels. Why doesn't—which might not have any impact—so why does it matter? If you work here and you start from here and you measure the impact of that, it's almost like a casino game, really. Right? You ship something, you have no idea if it's going to work or not. Absolutely. And so it can make it very difficult for you to then show that it actually improved something in some way. Does it make sense? Everyone? I mean, it's not particularly kind of fascinating here. Right? But that's sort of a very, very important thing to consider. But then we keep going. Because if I think about this, right, there is always this notion that comes up in almost every conversation I have with a business. It's all about value. You might have heard this before. Like, it's all about the value generation. It's about value innovation. It's about value delivery. It's about value recapturing. It's all about value. And it's not surprising that sometimes I will find, you know, you accent design teams, that actually working in a delivery team. And I always ask, what does it mean? It's not like we are DHL. It's not like we're delivering packages. No, this is a value delivery team. Right? And very often now I'm jumping in conversations where people are speaking about value architecture.

And it's all sounds like a very, very big, big term. So we're here, by the way. We should improve.

Improve my lighting. Sorry, everyone. Right? But value architecture is very important. Because this is a way to connect the company's actions to the value it creates for customers. In other ways, just to avoid the big terms, we have a business goals that have been given to us. And we need to break it down in some way given whatever abilities, resources that we have. Right? Into some sort of manageable design initiatives that will bring the company closer to that goal. So there is always this gap because we need to visualize in some way how we are here working on those design initiatives driving that business goal. So to do that, we often use something that's called value architecture to visualize that gap. It could look like this. Anybody seen something like this before?

So this is actually something that a lot of marketing teams will be using. And this is called a KPI tree. In a KPI tree, you take a goal, a business goal, and you break it down into sub-goals until you actually get to a low-level initiatives. Right? The way of how it works here is, well, you maybe have a business goal which is revenue. Please pay attention. Everyone, I know that you might be doing something else in the meantime. Because this, I think, is the crux of getting everything right. So we have this business goal. And that's revenue. There is a metric and there is a KPI. Well, every metric is just a bunch of numbers. Right? But every KPI is an important metric that is worth paying attention to. So every KPI is a metric, but not every metric is a KPI. KPI being a key performance indicator.

KPIs are usually much more strategically important and so on. Right? So if we want to improve revenue in organization, how do we do that? Right? Well, we need to understand what it is influenced by. Right? So what goes into a revenue, what's going to drive revenue to increase? Easy peasy. It's orders. Number of orders specifically. And the average order value. Right? If both of them go up, then revenue definitely goes up. I think that we all agree on that.

Right? So if the number of orders increases and also the average order value increases, that's definitely going to go up. However, if the number of orders increases but average order value decreases, this might not go up.

It might go up a little, or not at all. It might even drop. Does it make sense?

Right? Because they are connected. It's not like one is disconnected from the other. So if we—like, the best chance we have is that one of them is stable more or less stable and the other grows. Then we grow. Or even better, both grow. Which is very rare to achieve. Now, this is the head of that KPI tree. Right? And difference I'll be looking into your question just after this. I just want to make sure that I explain it. Right? So if we look at kind of really improving this, we need to improve both. The problem is that this is only the head of that KPI tree. And if you break it down closer to what you are doing, right, your design initiatives, you might be somewhere in branch level 5, 6, or 7. In other words, what it means is—and that's important because there are a lot of people who are working very different units trying to reach that goal from different perspective. Some of their initiatives will be successful. Some of them will be conflicting with what you are trying to do. Right? So you might be doing everything beautifully well and incredible with a lot of attention and all of that, and this is not going to increase that business goal or drive that business goal because somebody else maybe is working on something else not necessarily because they're poor or bad in what they're doing, but because it's conflicting with your goals. Right? And also, it might be just seasonal. It might be that you're working on filters and you refine those filters and you ship those filters and all of a sudden, indeed, the orders number of orders increased. That's great, but it has nothing to do with you maybe. Because marketing launched the same campaign like that same week when you launched your new filters and they offered, I don't know, 50% off on everything. Of course, the number of orders will increase. That's not very surprising. Well, the average order value will probably decrease. Right? But it might increase the revenue overall. Right? Has nothing to do with you. So how can you then indicate your impact? Right? Now, that's something that I struggled with for a while. Just a moment, Vinu. Just a second.

Well, to measure—to kind of to indicate your impact, we need to do two things. We cannot just measure that big goal on the top.

Right? That's not going to happen. We need to measure at least at two levels. If you're working on filters, you need to measure the impact on filtering. That means maybe you know which filters people are using the most. And you need to understand if people with your new changes now use filters better or faster or, you know, better in some way. Right? That's important. So if you're working on search, you need to measure search. If you're working on navigation, you need to measure navigation. But then you still need to, of course, take a look at what's happening globally. Right? But this two levels are very important. The local level of what you're working on and ideally if you're working, let's say, in a unit of representing a particular flow or product. Right? You need to measure success on the product level as well. It's just going to be a little bit more difficult to get your way all the way to the head of the KPI tree in your unit because it is being influenced by many other people too. Okay? And as you keep going down that tree, right, it kind of becomes even more challenging because if you look at the orders, well, it's broken down. It's visitors and conversion. If you increase both, that's great. If you increase one of them and the other is stable, that's great. If both of them going down, that's bad. Right? But then if you're all the way on level 5 and you're working on filters on level 5 to drive particular metric in there, right, it will be very difficult to make a strong case that you worked on filters and improved revenue.

Okay? So everybody's still with me? Nobody left?

Just yet? Vinu, you have a question? Could you maybe write it in the chat so I don't interrupt everyone? Would it work?

Sure.

Okay, perfect. Thank you. And I'll be dropping in the—in questions in just five minutes when I finish.

So this also can represent itself in a very different way too. It can be quite advanced. So for example, this is another example here. Where you have something that's called a North Star metric. Right here. Anybody has a North Star metric in your organization, in your company?

By any chance?

Yeah? Well, some people do. So North Star metric is perceived as being the important metric. Right? That entire organization is aware of and everybody is trying to improve that metric. Now, behind that metric—and that's often very important, it's often not mentioned—there are significant business metrics that are the company is trying to hit. Right? Or improve. Like, for example, annual recurring revenue, or monthly retention, or premium subscriptions, or anything like that. Right? But what is being disclosed to the organization is this metric. Right? This North Star metric. Now, this North Star metric can be influenced, again, by a lot of different initiatives. Right? And so you could have something like this where there is an initiative to improve—let's say, launch a new marketing campaign. Social notifications and others. Right? Kind of closer to the product side of things. So even design side of things. But then what is really important is when you start measuring, right, that you need to understand how well you're contributing to that goal. And to understand that, we need to attach some sort of metric that represents the success of that initiative. Right? So if we imagine that this is working as expected, right, what is going to change? If we imagine that this is working as expected, what is going to change? And kind of keeps going. And sometimes product managers would assign it or attach it to Epics in Jira. Right? Or in Asana projects. Right? And stuff like that. Right? And before that, there will be either research done—right?—or there will be assumptions, experiments, hypotheses. Right? That will be used or will be—I guess the initiators of some initiatives that are coming through. So you're basically creating this flow of value. This is what we call the value architecture.

Right? All the way on the right, you have the business metrics. All the way on the left, you have design initiatives or product initiatives. And in between, you have product metrics that are supposed to influence the North Star metric that then influences business metrics. The only thing that's missing here are design metrics. Right? Because those things are very clearly, you know, product metrics. But what about the user's experience? Right? That's what's often missing and that's what I think we should be kind of matching. And this gap is something that we need to bridge. Right? And just to visualize it to you, I'm going to show you a real thing. Here. And I'm going to drop in a link to you. So feel free to use it as well if you like. So this is coming from a client project I was working on. And then I'll jump into a question in just a moment. Here. And this is basically how I visualize that flow of value. Right? So—okay, excellent. I can see a lot of pointers in here. Beautiful. And so what you see right here is basically the same thing. You have a business goal all the way on the top. So imagine you're working for a manufacturing company that produces domestic appliances. They want to sell faster. That's all they do. The challenge is that they actually selling premium appliances, which have quite expensive. Right? So people need time to commit to purchase. Right? So that's a bit of a story. So I need to understand what can I do as a designer? Right? To drive that goal.

I also do like whimsical, by the way, we. So that's—we have something in neutral.

So we can break it down. Right? If you want to reduce time to purchase to help people buy faster, I need to understand what that means. Right? And so I sit down and I try to translate it. What does it mean? Well, to me, that means that customers should fast recognize the value of the products and also if they're buying something like a washing machine, right, be able to make best fit choices.

Because if they cannot make a best fit choice for them, or if they've gotten the choice paralysis, they will definitely not buy. Right? Also, they must understand the value because it's a premium product. And to me personally, what that means—and this is my translation of it, right—that means that if I think about time to purchase must be reduced, I think about, well, from the user's perspective, right, it means that people need to be able to make a final decision faster.

Maybe there is something that's blocking them. Right? Still with me, everyone?

Right? But that's not the only journey. Remember, this is not a customer journey map. Because we need to look at different sides of it, which kind of influence that direction all the way to the top. In fact, I would say my job, really, when I start measuring, is to make sure that I cover all the bases to guarantee success to the highest level I can, which can be very difficult, of course. But I cannot miss the big picture. So what we get here is another side of the story, which is often forgotten. Like, for example, frustration. When people are frustrated in some way, and they can be frustrated by the price as well. Right? Then they will not buy. If the web page is not responsive, that's a problem. If the search is not working well, that's a problem. If a configurator is not working well, it's a problem. If error recovery rate is low, that's a problem. So all of those things are endangering basically the risks of people.