Paul Blunden (UX247): Hi, I’m Paul Blunden. I’m founder of UX247, and today I’m going to be carrying out another interview in my series on research programs. This series is seeking to understand the similarities, differences between research programs, what are the benefits? Why we should think about using them and how they fit in with all the other research we do.
Today, I am delighted to be speaking with Tomer Sharon. He’s written widely on research design, thinking various things. He’s got 3 books out. He’s held roles in Google, Goldman Sachs, WeWork. He’s a industry Guru, to say the least. He’s going to be talking to me about continuous research. And I happen to know that’s already got a link to repositories. So I’m really looking forward to finding out more. Let’s hear what he’s got to say.
Paul Blunden (UX247): Hi Tomer, I suspect, for many of our subscribers you won’t need an introduction, and, in fact, many of my colleagues were very excited when I told them I was speaking to you this week, but for the minority, who don’t know who you are, can I ask you to introduce yourself.
Tomer Sharon: Hi, Paul, thank you for having me. Tomer Sharon. I am based in New Jersey, U.S.A. And I’ve been let’s call it a ux guy for many, many years. Mostly focusing on research main kinda highlights are 3 books I published. I’m not gonna go over them now. But 3 books I published, and work workwise I guess the most interesting ones are researcher at Google, mostly focused on Google search and within search, mostly on sports search results. And then head of Ux at WeWork and head of user research and metrics at Goldman, Sachs.
Paul Blunden (UX247): So you are what I would class as a proper industry veteran. So really pleased to be talking to you, and I could see looking at your background, you’ve been in research obviously a long time. How did you get into? What sort of inspired you to get involved?
Tomer Sharon: Oof we go a long way back. So we use none of these words research in in the ux at the time. I just had a feeling at the company I worked for at the time that we need to start having people who use the thing use it and see what happens. Started kinda self educating myself about it. Then I became a designer. Pretty bad one. Nobody hears and then, at some point, I think it was before Google, I realized that I needed to make a choice because I was happy with kind of being both. And then I needed to make a choice so deep look in the mirror. Maybe, I don’t know if it’s decide, but feel that research is, is more for me than design and that’s it. From that point on. It was all research.
Paul Blunden (UX247): Well, I think we’re very happy you made that choice because certainly the books you’ve written and the papers are a real library of intelligence for all of us.
So this interview series about getting to grips with different research programs. And I was so delighted when you agreed to speak to me, because among many other things, you are the brains behind continuous research. And it was something I’ve read about, but I’d really be interested to have you tell me, you know, what is it? What problem does it solve?
Tomer Sharon: So, first I’ll say what it is. I kind of thought about it at first. It’s fast-based research that’s open-ended in in nature. It’s not dedicated to a specific topic. Doesn’t have any you know, requests behind it from the team or research questions. It’s this research that happens all the time, and I’ll maybe I should say a few more a few more things about it, because, people might ask, Okay, what’s the difference between the research we know and what you’re talking about. So in kinda traditional research or research study or traditional research study, I would say, or, as I call it a dedicated study, because it’s dedicated to a very specific topic.
There are steps a researcher must take, you know. It needs to start with, you know, initiated by, you know, either conversation questions from from the team request. Whatever it is you need to plan it. You need to find participants. You need to schedule with them. You try and make that as kinda in a small amount of time as possible that you let’s say, interview these people and then you come up with a you know, analysis or report and share it with the team, and so on. With continuous research. You still do those things, but you skip a few steps and you change completely the way it’s done.
If a dedicated study is going to happen within 3 weeks. Continuous research happens all the time. So you might interview a person or a user once a day and then you’ll just do that every day, let’s say. And let’s say 15 min. You interview a user every day and then you can ask them the same questions, and over time you have and answer a body of answers to that same question which might be, a big question. Important question.
But the questions are very open ended. I think I need an example because I feel it’s too vague. So I’ll say how. How, I’ll you know, give an example of how we started it at WeWork. At the time we had people join, we work as members, as they were called, and they could leave anytime they wanted, but when they wanted to leave they would initiate that by filling in an online form. Hey, WeWork, we wanna leave, and so on and so forth. We’re out the time.
Sometimes it was there was some time between that form gonna submission and the actual day where they when they left, because, let’s say, they submitted that form on the 4th of the month but they paid until the 30th so they’ll stay and leave on the 30th. So we had some time.
So we kind of connected with that database of people who informed we work that they’re leaving, and we asked them to be interviewed. And it’s not a it’s it wasn’t an exit interview. It wasn’t, you know, our goal wasn’t to persuade them to stay or anything. We wanted to know a few things. And we just interviewed as many people as possible. I think we did that with hundreds of people. It was very short 15 min, and we asked, Why are you choosing to leave? We asked. What was good about. We work? What went well, and how could WeWork improve?
And if they had trouble answering that, we I specifically kinda with more sizzle on that question, and said, if you had 15 min with the CEO, what would you tell him? Things like that. So I didn’t know nobody, you know on the team knew what they’re gonna answer. They could speak about you know, bad coffee, or a noisy environment, or the beautiful design of the space, or whatever it is. But they chose what to talk about.
In any case, we collected all of that, and added that into a database and then eventually we could say with very high confidence, we could say. Here’s why people leave, WeWork. And at the time we found that there were if I remember correctly, 73 reasons why people leave WeWork. But that’s too much right? So we saw that you know, the top 5 reasons represent 50% of people who leave. So okay, let’s take care of that. So it gave us a pretty good idea of what needs to be not what needs to be done. But what needs more focus. What needed more focus from us now. The beautiful thing about it is, let’s say we could always add one more question.
So let’s say I love examples about coffee. So let’s pick coffee. Let’s say the person responsible for buying coffee, for WeWork is very interested in in feedback about that. So we could add for 3 weeks we could add a question about how satisfied are they with our coffee? Why or why not why, you know, yes or no, and then we could provide an answer to that to that person. So it’s very easy. We don’t need to tailor a study, invite participants. We had them already.
And especially when they’re when they’re leaving, they were, they felt very open to kind of say things.
Paul Blunden (UX247): That answers a lot of my questions or one of the big questions I had about Continuous research. When you sort of said it was not with a purpose compared to a project, but actually there is a bit of a purpose there around the theme. I guess so is it that the theme is bigger, longer, rather than very project specific?
Tomer Sharon: I think the questions that we asked that we work are very well representative of the type of questions that are good for this for this type of research. Because they are kind of they’ll always be relevant. You always want to know what’s good and what’s bad about your product. Nobody needs to ask for that. Nobody needs to, you know. Bump into something and then say, Okay, we need to know this. These are questions that everybody’s always interested in.
And there are a few more like that. Or maybe not a few more like that. That are not very specific and dedicated, and this is where it fits really well.
Paul Blunden (UX247): Gotcha. And you mentioned. You know the comparison with other research methods. Does continuous replace discovery research or evaluative research?
Tomer Sharon: No, I don’t think so. And that was not the intent. And even back then we didn’t stop any, you know some, if we needed to do a dedicated study, which is, which was obviously more than one question that you add to this continuous effort.
Then there was definitely room for it. So it does it replace it, but I think it. It provides another arm, another tool for getting sometimes quick feedback with those questions that we added and an overall look about a specific topic that that you’re interested, not specific, a topic that you’re interested in which might be, you know, as I gave him the example, what’s good or what’s bad about our product?
Paul Blunden (UX247): And in the one of the articles I read that you’d written, you talk about the project versus product, mindset. Could you share a bit more about that.
Tomer Sharon: When you have a project with a with a kind of start and endpoint, then then dedicated research is probably mostly suitable for that. And when you talk about a product, it’s almost like the difference between working in an agency and working in a product company.
When you work on a project or in an agency. And I’m not saying good, bad! I’m just saying there’s a difference. You come in even if you put a, if you will, a magnifying glass on something, and you leave with a product you keep doing research as a researcher. You see how the product is growing of the product is changing. You have all this history with the product and with the team. And then you can do kind of longer term studies such as such as the such as ones with using continuous research. That was the point I think, in that in that distinction.
Paul Blunden (UX247): Gotcha, and I think you 1st wrote about it in 2018. Has the approach changed or matured, adapted over time?
Tomer Sharon: Hard to answer, I mean for me not really I honestly, I don’t really know about other companies who you know adopted it and then changed it, maybe probably. But I think it’s another way. Another tool for people to use. And as always with things that I kinda put out there. I’m happy to see people kind of adjust them and kind of fine tune them to their needs, so I guess the answer is probably yes, but I don’t know. You know many details about that, because.
Paul Blunden (UX247): Got you, you know.
Tomer Sharon: I’ve even my own bubble.
Paul Blunden (UX247): Yeah, as we all do. I wanted to ask, because I think you are kind of say yourself when you explaining it. It needs more explanation, because it’s almost, you know, an intangible at some point when you’re trying to convince a budget holder who might not be a researcher how to do this kind of thing. How do you go about convincing them.
Tomer Sharon: I don’t know. Do you need to? I mean there are things there are things that are. Maybe I’m naive still. But there are things that are still you know, nobody’s going to argue with a researcher about them. So it’s to me. It’s like, you know, picking a method is one of them. Maybe they’ll have questions. But picking a method is when deciding on the length of a session, you know nobody’s. You don’t need any approvals for a session to be 30 min or 45 min, or or 60 min it. It’s completely up to the researcher.
So I think a researcher when they have a question in mind. I think the only thing is kind of working with the team on is this question. Interest is the answer to that to this question, interesting for you and if it is, then we’ll do what we can to provide an answer with high confidence. And this is the method we’re going to use. I don’t look for approvals for methods. Maybe some people do. But I don’t.
Paul Blunden (UX247): Yeah, no, okay. That makes sense. I guess. Where I was coming from is feels like an organization doing research is quite project mindset. Who’s then got to sort of invest in a continuous research program might find that quite hard to sort of square away in terms of what do I get back? There’s all this going on, but because it’s not attached to a project, if you like, or a product.
Tomer Sharon: Yeah. The biggest challenge I’ve faced was not with, you know, my stakeholders. It was with researchers. Because to researchers, that’s a it’s a meaningful change. I mean, there’s no, there’s almost It doesn’t follow. They’re gonna what they do on a kind of regular basis. So that was the main challenge for me, especially. And I’m sure we’re gonna get to that. But especially when I talk about kind of adding those insights from what we learned to a repository. So researchers were sometimes more difficult to persuade, I would say.
Paul Blunden (UX247): Right. Well, you’ve linked neatly to, I think the question I wanted to ask you which was about. And now we know kind of what it is and why people need it, what do organizations need to do to make a success of it? And I think that’s it sounds like repositories is key.
Tomer Sharon: I think it is I mean, everything we learned there we kind of added into a repository we created at the time and it was I don’t know. To me it’s felt like it. It’s worth its weight in gold. And it fits really well, kind of continuous research to a repository where you atomize your insights, not into reports, but into single insights. And we had a lot of these single insights during these continuous conversations with our exiting users.
Paul Blunden (UX247): And do you do you think organizations need to sort of change their approach to using insight if they’re going to go down this route of continuous?
Tomer Sharon: I mean, really, nobody cares. What they care is that they have a question, and they want an answer, and I think it’s in a way it should be. But it’s gonna take a long while to get there, but it should be almost like a electricity. You put the plug in and it works. Do you care what happens in the wall? No, you don’t. You have a question, you get an answer. Do you care what the researcher does.
Generally speaking, you care. You want it, you know, as electricity. You want it to be safe. You don’t wanna burn the house down. You don’t want people to die, that it’s all true, but the same is with research. You don’t want it to cost a million dollars. You don’t want it to, you know, take a million years, and so on. But generally speaking, nobody really cares about the details of what you do.
People have questions, have knowledge, gaps, and research is supposed to fill them with answers that, help them either be more confident what they do, or a lot less and but also kind of paint the way to towards you know what would make it better. So yeah, I don’t. I don’t think people are kind of very interested in the details. Most people, I would say.
Paul Blunden (UX247): Gotcha. And I, I suppose, hearing you talk about it, whether it’s continuous research or any other research method. If you’ve got a repository, and it’s set up correctly, and you’re atomizing the insight and turning it into nuggets you can act on. Then, if culturally, the organization is getting used to using the repository to go and answer its questions. Kind of continuous doesn’t make a difference. Is that would that be right?
Tomer Sharon: I think continuous research feeds that repository with a lot of insights. And that’s becoming very, very useful. Once people try to retrieve kind of knowledge from the repository so I always like to say that our repository was only useful once it had a thousand insights or nuggets. And as we did more and more of that, I think when we when I left we work we had 10,000 nuggets in the in the repository. What happens then is that it saves your research. You have the repository in a way, is hiding answers to questions that you have that otherwise you would run a dedicated study for.
And it happened to us many, many times when we kinda somebody asked for research. And we said, Let’s look at the database first.st Blah blah blah blah blah boom. We had an answer. So because the problem with the kind of the other option. And when I say repository, I mean a repository that is kind of using nuggets, not a repository of reports. When you had reports, you know, as a researcher, you make a choice sometimes what to include in the report and what not to include
The biggest example which I’m pretty sure every researcher faced or probably facing all the time is, you know we had, let’s say, 5 research questions. We have answers to these questions, but we also find 7 answers to questions nobody asked, or we were asked to do a Usability test, and we did it, and we have, you know what we need. But we also we found so many things. Let’s say 70 things that I don’t know that that we wanted to report on.
But you know that your team’s attention span is not too big, and probably they should focus on these 10 things. It’s a matter of approach. Some people would say, Report everything. When they have time they’ll fix everything some people say, if you report everything. They always choose to fix the things on the bottom of the list because it’s easy. And then we solved 40 problems. But they didn’t solve the top 5 ones and so on.
So it’s a matter of approach. But sometimes researchers are facing a decision what to include in the report, and sometimes they don’t include a lot of things, and with the repository, or with this kind of different approach of treating what you find as nuggets, single insights. It makes, you know be open. It gives you access to all of these almost lost findings that maybe when you ran the study you weren’t really interested in but in the future somebody can ask if you didn’t run a study about coffee.
Somebody might ask, you know, what do we know about coffee or people. You know, people have any feedback about coffee. So, and you never ran a study about coffee. But maybe 17 people commented about coffee and said some things that were useful that were added into the repository, and then you have it. You have your answer, most of your answer, or whatever it is that that you needed. That’s the beauty of kind of nuggetizing everything because it might become very useful in the future. And it happened to us many, many times.
Paul Blunden (UX247): Yeah, I can see that would offer huge benefits. I think that’s probably at the heart of my question about culture, because if an organization’s not used to consuming research in that way, then you, you know perhaps as well that’s the investment case for continuous if you’re trying to push it.
Tomer Sharon: Is where we had a lot of challenges, I would say, kind of. We wanted to. We wanted people in the organization to use the repository on their own, without us. You have a question. Go, type it and you will get an answer. That was the hardest part, I would say, kind of promoting it and having people gonna actually use it.
Paul Blunden (UX247): Gotcha. And would that be the biggest challenge also of sort of getting continuous research programs up and running? Or are there other challenges with that.
Tomer Sharon: you know if you have a research team and you know there are researchers there, and they’re running these studies, not these dedicated studies traditional research all the time. When you introduce this, you you’re adding more to their plate. So., it’s something that is always there. So it’s almost like shifts. Okay, who’s interviewing today? Who’s interviewing tomorrow? So it added, it adds some kinda logistics to it. Because it’s, you know, as we said before, it’s not really replacing anything else. But once people understand, and mostly when they see the benefit of it, then they’re happy to do it.
Paul Blunden (UX247): OK, understood. You mentioned as well. I mean, you’ve given a really good example with WeWork. But I wonder if there were any other examples of where you you’d sort of put in continuous research and what the outcomes were achieved that might help people sort of think, yeah, this is for them.
Tomer Sharon: This is for them. What do you mean?
Paul Blunden (UX247): In terms of a program and approach for their business.
Tomer Sharon: yeah. I also had a startup up until year ago, and we applied it there as well with our kinda end users. We had kinda 3 types of users. But with our end users so yeah, we wanted, we wanted to hear from them basically same questions as with WeWork. But completely different topic. That was also again. We just repeated it. Day in, day out and eventually you know, stuff started almost like popping up in the in the database. It was very clear that we had some insights there, were hidden, things that we missed and things that we could only get an answer because we have them in in the repository.
And after implementing continuous research it seemed like it was relevant. So it’s not just for big companies. It’s not just for for small companies. I think it’s a matter of, you know, having a product out there. So it’s not, for you know. It could be in either in a big company or a small one or startup that you know there are periods of time that you are. You don’t have a product yet or you’re working on something, but you don’t have users yet.
So I think it’s that’s less kind of fitting. You can. You could always. I mean, there’s no bad thing of talking with the user every day but for the purpose of what I’m trying kind of to learn from continuous research. I think it’s more fitting once you have something out there that people can. I always like to call it kind of poke and use, and then it works best. I think.
Paul Blunden (UX247): Okay, now, we’re getting close to time. But I think the penny starting to drop for me, and I wanted to go back to one question just to probe a bit. So it feels like if we’re running an organization, particularly in a brand. I mean, I run an agency so slightly different. But if I’m in a brand and I’m running a research team, if we choose to run continuous research, am I right in thinking you’re not describing this as we have a continuous research team. It is something that the research team does on top of the other research. They’re doing just because it’s kind of it’s regular. Small. Is that how operationally this works?
Tomer Sharon: That’s how I worked. But I’m assuming. And I’m you just reminded me of something from the Google days at the time. Nothing about continuous research. But at the time Google started they call it a rapid research team. And these were kind of more junior researchers led by a senior researcher. And all they did was usability testing. That’s it. So they decided at the time to dedicate a team to a method. I think only when you’re not when you’re Google, but when you’re big enough, then you can. You can play with these things and see how it works. I never tried it, so I don’t know if it’s and I don’t know if it’s if it has any kind of strong benefits. No idea. Maybe.
Paul Blunden (UX247): Well, look before I wrap up. I wanted to ask, is there anything else that I should have asked you that I haven’t. That would sort of reveal more about this area.
Tomer Sharon: I think the biggest thing is trying it out. Kind of not being afraid. It’s not, you know. You don’t need to learn anything new to be able to do that. If you’re a researcher, you already have the skills. So I think most of kind of the adoption is gonna held by the elephant in people’s minds. But it doesn’t really exist.
Paul Blunden (UX247): Okay, thank you. Well Tomer, it’s been a pleasure speaking to you. Thank you so much for giving me your valuable time. I’ve learned well, everything I wanted too about continuous and more. Thank you very much. I’m sure our subscribers are going to really enjoy listening to this interview.
Tomer Sharon: Sure. Thank you. Paul.
Paul Blunden (UX247): What an interesting interview with Tomer. I hope you enjoyed it as much as I did. I think I finally really understood what he’s talking about with continuous, because I was struggling a little bit with, well, there isn’t a thing. It’s different to product research where there’s kind of a goal and everything else or project research, I should say.
But actually, there is a thing. It’s a bigger thing. And it’s answering these questions over time and clearly, if you’ve got a decent repository set up, and you’re using atomized research and nuggets, and you’ve got a culture where you can go looking for answers rather than asking for research to be done. It could be hugely valuable and actually a massive cost saving. Fascinating, so much to think about. I’m sure you are feeling the same.
Anyway, if you want to learn more about this kind of thing, obviously, I’m sure Tomer would answer your questions or search him online. But if you want to get involved in any of these sort of program discussions. Please give me a shout. My name is Paul Blunden and I’m founder of UX247 you can find me on Linkedin message me there, by all means. Otherwise, email via our website. That’s hello@ux247.com. And of course subscribe to this channel, and there’ll be more interviews coming along soon. Thanks so much for watching.