Understanding how users experience your website | Search Off the Record
2024-10-31 ยท en automatic
[Music] hello hello and welcome to another episode of search off the record a podcast coming to you from the Google search team my name is Lizzie and today we have a very special guest with us Eva who is a ux researcher on the bill in the blank team at Google can you tell me uh what you're working on uh gladly so hello hello uh I'm Eva um I'm quantitative user experience researcher um just freshly transferred to Google shopping uh congratulations thank you I'm super happy about that and uh before shopping I worked on our internal uh search and I also worked on Google assistant at some point so that means basically my whole Google career has spent in some shape and some mode of search and uh and with uxr you've been this is your entire career or did you make a career transfer that's my entire career in Google but uh as you will learn almost nobody who does uxr in Google actually um studied to become a uxr we all come from very diverse backgrounds and especially if you do something that I do which is like a special flavor of uxr quantitative uxr yes I heard you mention this so it's not just uxr it's quantitative um yes and this is something very unique to big companies um the smaller the company the the less specialized uh people are right so if you're just starting a very small company you could have the same person doing both research and design or doing product management and user research but then the bigger the company grows the more specialized roles become so like the first typical split is like in design and research then the next one once you have a lot of researchers right some of them specialize in uh methods that Focus like to get some rich user data right so you really want to talk to a couple of users in depth and some other uxr specialize um uh in methods that get you a lot of data but maybe not uh at the same depth as um uh as qualitative uh colleagues in fact everybody can do user experience research right you don't have to have this in your title to be allowed or able to talk to your users right and in fact every of course of course so can you tell me what is uxr so it's researching ux so it does that mean you're doing something really academic you're reading a lot you're you're research reading research papers like what what is what is the thing yes so I can I hope hopefully I can explain so first ux stands for user experience and user experience is basically like a fancy name for person's thoughts feelings and action before they start interacting with any system while they interact and after it right so it doesn't mean just user interface it doesn't mean just usability it's just like this General fluffy thing right that goes around the uh the use and um there is somehow academic branch of this I would say right so you've for sure heard of um human computer interaction okay yes this like the the degree that you would get or something exactly so this is like academic discipline that uh that studies interaction between well humans and machines right okay uh but then um ux research is kind of more like applied version of that that happens in Industry because usually like when you do academic research you want to make sure that what you learn is very generalizable and that it applies to many if not all humans and so on but in industry very often you don't necessarily care to learn things that are applicable overall right you just need to learn something about your product and you need to learn it fast right so um compared to Academia I would say on average we apply a little bit less rigor to our methods but that's okay right because at the end of the day like we want our insights to be very practical and very applicable right you want to fix your product or you want to make it better which is much different from some like academic kind of goal of uh understanding humanity and how humans behave in general and what's the goal with this whole thing like it we're studying how people are interacting but then what do we do with it yeah so I mean ideally we improve our products right so we either build better products to begin with OR we improve the products we've already built or we learn something for the next time we are building a product um and of course having a good product is also not a goal in itself right so if you own a website like let's say you own a web store your goal is ultimately to sell products right to sell something to to your users but then um if your website is such that people cannot find stuff in it that they are frustrated with it that they are not satisfied with their experience at all they will not come back right because almost like there are very few products where users are really like locked into they have to use this one they don't have an alternative like you will suffer through this stat experience because you need to make the or whatever exactly so like if you have to use a particular software for for work maybe manufacturer of that software is not necessarily like super concerned with you being super happy with this product but in general like for most companies like you know getting your visitors fast to to their goal getting them to come back getting them to recommend your product further right these are all somehow related to how good their experience was with your product so if I have like let's talk practically how would I know that I need to do uxr uh a study or to start looking into this and let's say I I have a small website it's not a big company I don't have a team do I just is it sort of a chicken and an egg thing like do I know that it's there's a problem and I need to bring in uxr or apply something from there or can I bring them into then uncover the problem so how do you know that you need one well it's easy because almost always you need one right are you just biased though no there is no such thing as a perfect product but of course you it doesn't mean that like uxr is the first thing that your small company should do um if you are building something that has like very well established examples in the industry right uh you know if you are building a a a website or like online auction site or so on it's it's already quite well known right like what works and what doesn't work you mean like what's expected from a user who is making a bid like let's say it's the auction side example yes and like how to present your products you know how to filter through them and so on right so in that sense you know your your your uxr and you cannot hear my quotation marks here but like your uxr might consist of you going out looking at the examples of products that you think or somebody else thinks work well right and just kind of learning from that and that's already some form of uxr right you're you're probably already doing it naturally um as you're paying attention to how using websites and we kind of know uh what would be like what how to find the menu there's probably General convention that you should put a menu like this and if you're doing something that's not standard or unusual then that's probably a sign or it's going to be more difficult for people to find it if they don't know where the thing is exactly uh um with the like small difference you know like if you're learning from other people's experiences you as a developer might focus on you know how does this thing technically work but what uxr would probably focus a little bit more is like how users flow through products you know like you know okay so okay I now imagine myself as a user I want to like buy a certain product like so which actions might I take and then you could take an inventory of you know how are you guided as a user through different products right so this is a little bit I guess difference between uxr and other research it's really like you always trying to to like understand the experience from user perspective not like from technical perspective or even from your own right because that's I guess a big maxim of uxr is you are not your users right and of course uh you are maybe representing some of your users but um the moment you start building your your site or your product you just understand it too well right and then it's very hard to imagine yourself as a as somebody who is very different from you right you we become like as we develop our products we really become experts in our products and then it's very hard to like take this novice view of our products right it's the curse of knowledge like once you know where I I know exactly where this thing is on my website so I can navigate to it and then you forget or it's hard to unknow that piece of information so if you're doing like a DIY approach for uxr how can you do you like need to imagine that you're a different user or do you need to go talk to people do you show them your website on the street like how can you go about doing this yeah so like all of the above so it very much depends on like how important this is to you and how Keen you are and how much time you have and how many resources you have right but uh if you if you let's let's for for a moment assume you will do DIY right so let's assume you are have very low budget I'm not going no budget no agency so I I just want to do I might be it's me I'm working on yeah I have a documentation site developer documentation and I want to dabble in uxr do I need to have a project in my first or like a problem in in mind yeah I guess you should have some sort of a problem but for sure almost certainly already have one so your problem might be something like you might experience high bounce rates or low conversions you might receive some negative feedback right and you might not know oh does this feedback reflect all my users or did one person like have very bad seral this morning right you might uh not even know who your users are or what they need right you might have a great idea for an app but but you're like okay but who would really use this um it might be that you observed some sort of drop in usage right that you cannot explain from external factors um or it might be that you're just anyways deciding to redesign your service so then why not like look a little bit into like how to make it easier for your users and um there are many many resources I mean free online resources right to help yourself but as you said without knowing anything it's very hard searching in the dark for like uxr resources is maybe too broad of a net um but I relate to what you were saying like uh finding the problems do make their way known to me so we do have some things on our site uh where we can see the helpfulness rating for example like a lot of people are having a bad time on this page or they don't like it they hate it they're clicking that downvote button but I don't necessarily know why yeah so would that be where uxr best practices would come into play to sort of like investigate is it like investigative journalism a little like being a detective detective about like why is this happening and then you sort of like start to investigate the metrics that or at least the data that you have yeah potentially yeah exactly so you you of course you would first start with the data that you have because that's cheaper than generating new data um but um even if you um are able to determine exactly where the thing happens right you might still not EX exactly know why and this is where it's actually helpful to also be able to talk to few users so in general there is um I mean there are many ways you can split uxr methods but kind of the the the most useful um way if you are a small businesses to think about okay there are some methods which don't require like users and there are some methods which do require users and it's a little bit unintuitive that you can do uxr without talking to users uh but uh there are many methods like the idea is you know uh you will ultimately do user research with users but before you are even ready to put stuff in front of your users because that requires time and money yeah it's like you need to show them a demo or something like I guess like talking to users are are we actually talking about like verbally speaking with them like bringing them in for interview or like a survey something that's a little bit more scalable so I mean when I say talking to users it might just be like any shape of users giving you like their attit udes right so could be through a server way could be through an email form could be through actual conversation could be through them making like a cool video for you while they are using your product there are all these methods right but before you get there there are also certain methods uh where you kind of emulate your users and how they move through your product and these can already help you to somehow like tackle some very big and glaring problems in your interface right because the thing is you do you talk to users let's let's say an interview right you talk to them you talk to five of them you have to you know there there there is a lot of admin related to like talking actually to users and then you fix it and then you have to talk to them again right because why I guess why to to make sure that you fixed it right I mean a new problem of course this is this is the ideal flow right it's very rare but then like if you it's very expensive to to talk to a user for the purpose of them telling you hey this button is different color than that button right yeah like pointing out the obvious I already know this thing yeah so there are certain methods like I will name my favorite ones cognitive walkthr and heris evaluation which are very structured method that would help you as um as a um developer product developer or designer or whatever is your role to somehow identify these like low hanging fruits without having to involve users right so this is It's like a low cost yeah exactly so it's not necessarily enough right so of course like if you can talk to users you should but if it's better than nothing right it's always better to to apply more methods uh okay so can you tell me about the first one you you you put in some big words that I've never heard before so so the first one cognitive walkth through walk through so it has a very fancy name it's not a very fancy method it's it's um in fact um sometimes my colleagues like to tease me because I really like this me method and it's a it's a very vintage one right because uxr is such a fast moving field uh and we are always getting more and more data about users if you end up using a method from the 90s it's already like but I think it's a very cool one right so the whole idea of this method is that you somehow first you start by describing to yourself exactly who your users are and what do they know and what do they not know right so for if you are if you have a online store right you might start by thinking okay so they are they are in their 20s you know they speak English and so on right you make a lot of assumptions about your users they should be realistic and then um you make a list of like tasks that users are accomplishing with your product right so if it's a if it's an online store they might be buying something they might be returning something they might be comparing prices and so on right and for each of these tasks you you follow a very structured like way how users move through the screens from the beginning to the end how they currently move through or how they should so uh um H the ideal flow right so like how the way that person who designed the system imagine they would move right and then at every point you ask yourself okay so this is like this is how I think they should go will they do this or not right and then you kind of try to justify why they would go the right way or why they wouldn't and you just repeat this a lot it takes maybe one two days to evaluate like a decent sized website but at the end you somehow end up with a bunch of like assumptions right like they would do this because they are like this or they wouldn't do this because product is like that and then all you have to do is take this list of assumptions and address each one of them right and say okay this one was reasonable I leave it this one ahuh okay if I fix this bit then they will not have this problem anymore right so you have kind of generated a list of problems in your uh product without I mean it's not without bias of course because you're making it's difficult if you designed it this way then then you go through this walk like this is how I designed it h it's working yeah but that's that's not a very good right like because I said so is not a good argument or for like why why a certain experience should be like that and if you're really honest to yourself and your product right you will see okay hey actually like this font is quite hard to read you have to be like into like discovering the problems when I'm doing this exercise I'm going to be critical about this thing even though it's my that I made of course and I mean this is this is the thing right so if you really want to like if you genuinely want your product to be better then you will also be like honest in your whichever method you're applying but if you are trying to and this happens a lot right if you're just trying to satisfy some check mark right if you want to say oh we did user uh research for this product but you don't really care to improve user experience right then of course well then you're set up for failure from the beginning yes exactly so so like uh all the advice I'm giving today is really only meant if you genuinely want your users to have better time with your product yes okay so the second one so cognitive walkthrough uh I'm think I'm using my brain that's the cognitive part is like I'm imagining I'm putting on different hats and I'm imagining how this type of user it sounds like user Journeys almost but like in a practical way yes it is right it is really like you are analyzing user Journeys without like having to adopt a super fancy framework right um and then the second fror theistic evaluation evaluation okay yeah all these fancy words right so this one is actually even simpler you mentioned before remember you said oh there are certain standards that probably I can follow to make my website I don't have to necessarily invent everything like from scratch and that's that's actually what it is right you take a list of Standards you take all screens and this list of Standards can be anything that's relevant to your product right so if you're analyzing a website let's say we have have a like a a knitting and crochet yarn website that's selling supplies and stuff like that is there a standard for like how knitting websites should be laid out so there isn't a standard how knitting websites should be laid out but there is a standard how retail websit should be laid out and and even more generally there is somehow standard how website should be laid out uh right so um but then if your product is something else let's say you're building which by the way you can totally analyze usability of buildings right um then there will be a different set of Standards right so your first goal is to somehow identify which principles you should apply and these are actually like really really high level ones right so they are they're saying stuff like contrast should be good right and what does it mean that contrast is good you will still judge with your human eyes It also says well you should like anticipate that people will make an error and prevent it right or if they make an error you should allow them to recover from this error easily so these are it's very simple like close the entire app or something like that exactly and then all you do is like you take this list of principles you take every single screen of your product and then you just check that all these principles are satisfied for every screen and that's all right so it's very very simple where do you find this list of standard like this magical so I I don't know if I'm allowed to plug something in here that I'm in no way affiliated with but there are many uh so there are many like ux communities around the world but actually for me like the most useful website and the place where I learn the most about these things is um his website uh from neelsen Norman group so neelson and Norman are basically two fathers of this field of uxr right or ux in general so uh if you just look up Neilson Norman group they they have hundreds if not thousands of pages of different resources where you can learn about different methods you can learn how to apply them when do they apply right because I did say before well the more methods you do the better it's not strictly true right because at some point you get overwhelmed with the amount of data or like suggestions that are out there I mean it sounds like it's sort of a a benchmark for the the industry that you're in you're kind of looking for what is like sort of high level agreed upon but even there you mentioned that there might be several sources or opinions on like how to do this thing it's not like there only one answer like where the menu can be the menu button absolutely and again there is no one way to check whether the menu is in menu button is in the right place right so it can depend on like what are your uxr skills that you have or don't have right what will resonate well with your stakeholders right so like I can run thousands of surveys but if my director only wants to watch videos of users to make their mind right make up their mind then then my surveys are not so much um useful right and this is what I meant about this difference between Academia and um and Industry right because ultimately what you're doing as a uxr also has to be very practical right so but uh putting all this aside right so like what this website also offers is you know it will give you good advice okay so you know if you only have a like a low Fidelity prototype right you can't really measure time it takes users to do something right so if you have a paper prototype right so if you don't yet have a page or website then you can't really uh like measure things that have to do with also how page functions right so you might have latency you might like have to um like include for scrolling and so on which you don't really have on a paper prototype but like things you can then like check that the structure of menu makes sense right you can do some sort of information architecture you can um you can make sure that all the titles of the sub pages are understandable that all the sub menus are where they should be text doesn't take too long to read you know with your human eyes and so on right so depending like in what level of development you are with your product you might use different methods right or imagine you only have 100 users you cannot run a survey with 100 users right because what do you mean right you can't oh that's a good point well right but but what I mean let's say you email all your 100 users and let's say they really love you so you get an awesome response rate of 20% right so that means you got 20 users to tell you oh they are like satisfied with your product between three and five right I mean you don't ask them to give you a number you ask it with a better question but just for the caricature okay so now you have you have you know you have certain number but like your confidence interval is just so big that you would never really do statistics or try to model this data when it's such a small data set right so it might okay so it's not a big enough data set for it to be like meaningful or like reliable I guess so and also it might make more sense than to just email your 100 users and be like hey can I chat with you for 15 minutes right and if 20 of them tell you yes you will talk to them for what H somebody do the match right so like you will talk to them for five hours and you will get like much deeper because even let's let's say they let's say all your 20 users told you they they love your product right you scored the highest uh grades or the lowest grade you still don't know why right yes so that's why you need the the qualitative ex research this is the the talking to them and learn more about like why was that difficult exactly exactly and uh and I mean in general I think you always like it's very good to always triangulate or like use different methods to get at the same question right because every method has its own kind of uncertainty right because like if you talk to five users and in general like five users is like a good amount we can talk about this like why is this good amount because it sounds very low right but then like the problem there is okay like you have to put a lot of thought into who your five users will be like and how representative they are and so on and you can never like from these five users you know like what are the pain points but you don't know like how common they are right because you didn't check with all your users on the other side you know if you run a survey you can check the frequency of paint points but you won't know like the juicy details right they cannot show you on their screen hey look here I get stuck right or hey look this is now off my screen because my I'm doing this on a small screen and you designed it for a big one right so it's really like every method will uncover like a different type of information for you to make your decisions right is there a recommended approach where you sort of do the the qualitative five user thing to find out what the problems are and then do the bulk survey to like validate this thing is this like a common approach so this is a super common approach yeah uh so this is something that I do a lot with my colleagues right so they will uh they will run qualitative research and they will identify certain themes and so on and then I will go quantitatively right because when they give me a list of these problems let's say then I can turn them into survey questions right and then I can really assess what's going on there sometimes it can also go the other way around right I could identify in my survey that uh people have a certain issue right let's say they feel that the information is outdated on the website right but it could still mean many things so then I can give this to my qualitative uh colleagues and I can tell them hey users are reporting this can you go and check like what exactly do they mean right ah okay so you kind of find the theme the high level theme that like uh it was reported that uh things are out of date or difficult to use can you find out why yeah can you find out like or how widespread this issue is this kind of thing so so yeah I guess the widespread issue is more more like the Quant area and Y is more like a qu I mean it doesn't mean you cannot also get into y with with qual with quantitative it just needs more like it takes much longer right and it's much uh like you get much more noise in your data because um ah so you bring us to a good point with the data how do you know that you have enough data and that it's good to use clean enough that you have and how can you tell this if you are not uh like a dat Nerge is this something that the average website owner could do yeah yeah it can right so I mean I guess like um how to say you have exactly enough data I mean if you're doing something Fancy with your data um you will usually know what you're doing right if you're trying to Model A Certain phenomenon or develop a certain metric you will then also be able to calculate um exactly how much data you need right and like you can calculate how certain you are in your data but that's not something you will DIY right that's really something that that has like its own kind of like other bucket but if you just have some data right and you want to know okay what what's going on with this data like is this is this good I would say any data any amount of data is better than any smaller amount of data right oh well you always want more let's give a practical example so let's say that I'm running a survey uh on our developer documentation site I'm running it for 24 hours yeah and I've had maybe 100 responses is it enough how many users do you have to begin with um 10,000 okay so that means your 100 users are about one are they 1% oh my God I don't want to embarrass myself number this is even like this is a fake example I should give you uh easier numbers to do some kind of computation no no it is 1% right so right now what you have done is you have taken the data from your from 1% of your user right um if if this these 100 responses would be like equally distributed right so you are sure that you got all user slices in them it's probably enough um and depending what question you're trying to answer right if you're trying to understand okay why are you on my documentation page you can do this with 100 right but if you're trying to like track certain metrics over time right so if you're having them give you their I don't know like um um their satisfaction and if you want to turn this into a dashboard and track it over time it's probably not enough right because your confidence interval will be very big it depends on like how much data you have uh and then if you see a different number in a month measured from 100 responses you still cannot really judge if if the things have changed because the confidence intervals will like overlap right so there is like too much kind of possibility for an error so with 100 participants you can really like learn things about like okay so like are you having certain issues right like um what's maybe like even something a little bit about demographics but the problem is as long as they're not representative uh it's more just going to be very informative but not enough to do some sort of like metric development or something so this is where you come to the statistical significance number there's like some magic number that you need to hit well there is there is actually a magic number and I'm now worried I will give you very bad advice right but like normally when you're doing like this sample size calculations why why are you worried because uh because I don't want to give bad advice to uh to somebody and then like have their business go up in flam because I said the number on a podcast however uh you know how we said well there is five participants are enough for qualitative research there is actually a good reason for that right of course five will not uncover all issues but the added value uh of insights from the sixth one is much lower than adding your second or third right so it's kind of like a curve so from one participant you learn a lot from if you add a second one you learn almost double but as you keep adding participants like and doing interviews you're just hearing same and same stories okay so you reach sort of like a plateau where you no enough and you you don't need to keep doing this to keep hearing the same sentiment again and again yeah yeah exactly what I wanted to say earlier is actually like before you start doing any fancy like sample size calculations or whatever I would say like you can use your human eyes on your data right so and this is something I love recommending to people just like use your human eyes and look at the stuff and you will already learn so much right so like if you just plot your data right so let's say you ran a satisfaction survey on your little website for 24 hours and you got certain scores right and you can calculate an average and let's say average is 3.5 but this average doesn't still tell you okay do I have like a lot of haters and a lot of like people who are completely obsessed with my product or is everybody Mech about it right the average doesn't tell you that so if you plot your data you will start to understand you will see well what's the shape of my data is everybody agreeing with with each other or do I have like multiple groups that might have multiple attitudes right interesting uh yeah on or then like if you you know let's say you record how long people take on a certain like Sub sub page of your website right so if you just try to trace these timings right so if you open this website and then you sit there and you wait as much as users wait there and then you proceed to the next action you start understanding like oh wait why is this user now sitting on this page for five minutes when they only need to enter one number in the field like what's going on there and then you start thinking and you realize ah I'm asking them to put in their credit card number and they have to go and get their wallet right so I cannot you know I cannot assume that they will proceed like in 5 seconds when they are actually getting their wallet right so it really helps you just the kind of of like look at it think about it and and try to maybe like Reen what they have done if you have some sort of like information right or if you see that they're bouncing between the two pages um then you should also go and bounce between these two pages as many times as they do because then maybe you'll realize ah okay wait actually these two terms are too similar and they don't know which one is the correct one right so they or they're like checking your shipping uh list of countries that shi to you and whether or not I want to make this purchase because I found out it cost the same amount to ship me this item as it is I don't know something like this yeah pretty much and and another thing I wanted to call out right so like as long as your data matches your research question you don't need to have like a ton of data right like if you think about it so you mean enough data to then fix something for users like if you have five people telling you it's a problem it's enough to then maybe make fix definitely so if if if these are your real users and five of them out of five right in the interviews are telling you this is an issue this is something right but just because something is an issue it doesn't mean it will end up being prioritized right because there is other factors it might be just too hard for you to fix or something and this is I guess somehow like what uxr are trying to do when conducting our research right we try to make sure that whether are five or 100 or 384 data points wherever they are coming for this should somehow like represent the slices of your users right and it should be like carefully considered going to inv but like if you heard five times something is an issue M and you had no clue that this was an issue for even one person before I mean you learned something right yes I mean I learned a lot on this episode and this has been super useful for me and I'm like feeling re invigorated to think about my survey for my site because it's something that I've been wanting to do and I think it would be now a good time of the year to start thinking about asking people what they're having troubles with before planning for for next year but as we close out is there like one takeaway that you wish that uh people would come away from this episode uh having either learned something or like a hot take about uh uxr that you want to clear up huh so I don't know if this is a hot take but uh I wanted to say like whatever type of research you want to do somebody has done it before right and there is probably a standard for it or a framework for it right and it's always better to apply something that's been like used and reused and validated then trying to like come up with your own questions right because if we are not super trained like the be Arts we might not even know like how to avoid bias in questions or like um ju even understanding just because you know your users are satisfied like what does it mean like why do I care right so just looking it up and understanding aha okay satisfaction is like highly correlated with like retention rate then you know why you're doing that right so just like go out look it up right look up uh is there a resource that we can put in the description again I'm mentioning this Neilson Norman website and if people wanted to find you on the internet where could they find you so uh again we will put a link I guess to my LinkedIn because if you L you try to if you try to spell my last name from me pronouncing it good luck with that sounds good so we'll put the link uh in the description thank you so much for joining me today uh and that's it for this episode next up on search off the Record we're going to continue talking about ux uh this time about the relationship between ux and SEO so stay tuned for that one and bye [Music] bye we've been having fun with these podcast episodes and I hope you The Listener have found them both entertaining and insightful too feel free to drop us a note on LinkedIn or chat with us at one of the next virtual events we go to or inperson events if you have any thoughts and of course don't forget to like And subscribe so you can stay tuned for the next episode thank you and goodbye [Music]