Category Archives: research

Reece’s Secret Sauce to Product Success – October 2022 Wrap

Posted on by
Reply

How well do you really know your customers?

Do you have to reference a user testing report to try to understand their needs?

When you are designing and building solutions, how easy is it to test your assumptions?

Do you need to kick off a testing with a 3-4 week lead time, to recruit, book and run sessions?

Could there be another way?

The folk at reecetech thought so. 

Immersion, Immersion, Immersion – Branch Time

Starting from their grad program, and soon proliferating throughout the rest of their teams, new Reece employees are given an induction like no other. They are sent to one of the 644 Australian branches to immerse themselves in the business.

It’s back to apprentice mode, as you get first hand experience of the customer sites. The 7am morning rush, helping setup customers, picking and packing orders which come in from all areas (overnight orders, phone, app and in-person), receiving stock, organising deliveries,  reporting back when things don’t go quite right. And seeing Reece’s customer obsession service standard for yourself.

Not just for research purposes, or standing on the sidelines taking notes. But by working side by side with the branch staff, serving customers. 

Everyone has done Branch Time. 

Even the CEO.

How long can vary for different areas, depending on their needs. But the standard stint for a Product Manager is around 6 weeks.

The Pros and… More Pros

But sending every new hire to Branch Time is a serious commitment – in both time and dollars. 

So what’s the upside?

Firstly, accelerating your understanding of the business, and building a strong foundation to make decisions in the future. This is not always tangible or measurable by reporting, but what better way to fast-track your decision making capability?

Also, forming connections and relationships with the branches, and understanding the mayhem of retail. You even get direct experience with using Reece’s internal systems, such as TRS. You always have access to feedback, as you’re constantly in contact with branches. 

And, of course, establishing deep empathy with your customers, the majority who are tradies. Understanding and becoming intimate with their customer problems, so you can ensure the right solutions are developed to deliver the right outcomes. Knowing their world also enables you to contribute and create meaningful OKRs (targets). 

But it doesn’t stop there: 

  • Branch managers and staff also benefit from building relationships with somebody in head office.
  • After Branch Time, you have ongoing access to the branch network, to continue to foster relationships, to interview or to validate ideas and concepts with staff or customers. 
  • New team members also go through the same Branch Time experience, so there is a shared understanding and common ground established.

Thank you

Thank you again to Cameron Rogers and Nikki Pecora from reecetech for sharing, to our volunteers (Nosh Darbari, Yau Hui Min, Steve Bauer) and to our lovely hosts Lexicon and A Cloud Guru.

After Research: Creating Useful & Well-Executed Research Outcomes – February 2022 Wrap

Posted on by
Reply

Following a number of sessions on gathering customer feedback, we were fortunate to be joined by seasoned researcher Jess Nichols to share some insights on the next stage of research – bringing it all together and synthesising your qualitative data, creating reusable and actionable insights and advocating your research across your team. 

Setting up for Success – Do Not Do Research in a Bubble

Research is there to mitigate business risks. 

Therefore, one of the worst outcomes is for your research to be ignored, shelved, or not utilised. 

Alleviating this risk begins before you even start conducting your research. Give some thought to what successful research looks like. Are you trying to drive to specific research outcomes? What is the wider business context? Are there strategies or OKRs that can act as your guardrails? Work with your team to ensure you are solving the right knowledge gaps for them. Having this North star can help to provide clarity in what questions you need to answer with your research.

Synthesising your Data

Once you have collected all your feedback and conducted your interviews, it’s time to collate your research and find the patterns in the data. This is crucial for connecting your data to any desired outcomes. 

Participants will commonly try to contextualise questions with their understanding of the problem or situation. They may apply their own biases to their responses. So you should not take their responses at face value. Understanding, and classifying the data into behaviours (what people do) and attitudinal (what people think) can be beneficial. Try to drill into the responses to find the underlying pain points. 

"What people sat, what people do & what they say they do are entirely different things" - Margaret Mead

You can then form insights from the patterns of behaviours or attitudes.

Some tips for when you analyse your data:

  • There is no single right way to analyse your data. So just start, and pivot along the way.
  • Use your OKRs to guide you.
  • Be comfortable with conflict.

Sharing your Insights

One of the aims of research is to have no surprises at the end. Share as you go. This can help to identify what resonates, what may be controversial and need more care, or what can derail conversations and should be avoided. 

Research also won’t be useful if your teams don’t understand it. Creating stories can be a useful vehicle to deliver insights. Your team is more likely to remember stories that they can connect with, which can make it easier for them to incorporate the customer insights into their work. 

  • Tie the insights back your original research questions;
  • Advocate for your customers’ needs (especially those which may not have been considered by your team in the past);
  • Help your team understand how to action what you’ve learnt about your customers. 
Insights are the sticky stories you want to tell about your customers.

There will always be biases, from both your participants (friendliness, social desirability, habituation) and your own (confirmation bias, cultural bias, halo effect, etc). The main thing is to recognise them and to then try to minimise them. 

Ensure your Research makes an Impact

As well as being the biggest advocate for your research, find and partner with others to help champion and influence behaviours.

A handy way to approach this is by:

  • People: to amplify your learnings. You can start with designers, marketing and product marketers, other product managers and your research community.
  • Processes: to add traceability to your findings. Insert relevant insights into the product development process, through user stories, requirements documents or annotations in designs.
  • Platforms: to store your research for future use. Upload your presentation to your internal wikis. Bring up relevant insights during meetings. Share bite-sized insights over chat.

Your research will not always have a clear or direct impact on a business outcome. Sometimes the result will be more subtle, and change the direction or the way we understand our customers over a longer period of time. Either way, celebrate the impact you make, big or small. 

Successful research involves a level of humility. Not just listening to your research participants, but listening to your internal stakeholders so you can be effective with them using it.

Thanks

Thank you to Jess Nichols for sharing, to our volunteers Nosh and Steve and our event sponsors Pluralsight / A Cloud Guru.

Further Reading and Resources

You can find Jess on LinkedIn, Twitter or her website http://www.jessnichols.com.

Some resources mentioned during the session include:

After Research: Creating Useful & Well-Executed Research Outcomes

Posted on by
Reply

So you’ve completed your customer interviews – but now what?

How do you make sure that you’re creating the right insights based on all of your data? How do you advocate for your findings across product development, especially when they conflict with business objectives?

In this presentation, Jess will share how to set yourself up for success in the most important part of the user research journey – After Research. Learn how to effectively synthesise your qualitative data, create reusable and actionable insights & advocate your research across your team.

RSVP Thursday February 24, 2022

Our Speaker:

Jess Nichols

Jess Nichols – Principal, Research & Insights, Pluralsight
Jess is a research leader at Pluralsight (formerly A Cloud Guru), where she helps teach the world to cloud. She has spent the past decade focusing on discovering actionable insights through qualitative research approaches to ensure customer centricity across the product journey. Throughout her career she has worked globally, including several years based in San Francisco, working for companies such as Twitter, Uber and Deloitte.

Our Host:

TBD – we’re working out if this will be via zoom, in person or hybrid

A Practical Guide to Customer Feedback – July 2021 Wrap

Posted on by
Reply

We can always benefit from getting closer to our customers. But how should we go about it? In July, Dipa Rao shared some stories from the trenches, and some practical advice to help us navigate our way through. 

When do we need customer feedback?

Always! We should get customer feedback as often as possible. And at different stages of the product life-cycle.

  • Understanding the problem space: What are the problems our customers are trying to solve? What are their current solutions and alternatives? What are the gaps?
  • Validating solutions or ideas: What is attracting new customers, and is there information or data that they want to carry forward? Or perhaps designing a mockup to gauge interest, before completely building out new functionality.
  • Prioritisation: We often have ideas from many different sources, such as from our call centre and frontline colleagues, management or even directly from our customers. But where should we start? Surveying our customers to rank importance can be beneficial, to ensure we direct our limited and precious resources in the right places.
  • Any change, big or small: Depending on the size of the change, we can employ different techniques to gather feedback, from limited betas to feedback forms post launch.

How to get feedback?

When designing a method to gather feedback, there is no perfect solution. Depending on our skill sets and resources, this could end up looking different for each of us. Net Promoter Score (NPS) could be a good start. However, it is not specific by design, so it may not entirely meet our needs.

Whether we decide to use email, or create an in context web/app form, or even instrumenting a survey with google analytics, try to make it:

  • Easy;
  • Have minimal set up; and
  • Repeatable

How to prepare?

Expectations: Letting both our internal and external stakeholders know what to anticipate will often make our lives easier. 

External customers – Why are you asking me? When will I hear back? Will I hear back? What are alternative paths for support?

Internal customers – Awareness of our activities for support (if needed). Sharing feedback and insights, some which may be distressed feedback.

Analysis: Ensure there is time and capacity to analyse feedback, before trying to get it. If not, don’t bother getting it and wasting our customers’ time. We may also need to mash data together from different systems, so finding an easy and/or repeatable process will be important.

Bureaucrazy! Never underestimate the amount of bureaucracy that may exist in large corporations. From setting up a shared email address, standing up a new platform, covering the legal and privacy aspects of engaging with customers, or ensuring our proposition is aligned to our marketing and brand guidelines. All of these things can take time. 🙁

Types of feedback

When the feedback starts rolling in, it can come in different shapes and sizes. So it can be useful to categorise the feedback, and to learn when to take it with a grain of salt.

Shiva (the destroyer): This feedback can be brutal and destroy imperfections. But don’t take it to heart, as this may be more indicative of a lack of loyalty or trust for our overall product, brand or company. Remove the emotion, and take the feedback for what it is. Feedback from Shiva can impact our morale (or our teams), so take in small doses.

Vishnu (the preserver): Feedback from Vishnu is generally pragmatic and more balanced, and can encourage us to keep going. We’re on the right track. 

Devi/Shakti (the creator): We can consider Devi as expert or superusers, who will give detailed feedback, and potentially challenge our thinking or approach. A great way to foster new ideas and allow them to grow.

And then there are ‘other’ types of feedback. 

Got feedback, now what?

Once we have feedback, we should analyse and share the insights. Feed the other parts of the business. Construct a shared understanding. The feedback can also help motivate our teams. And where possible, we can also respond, to open a dialogue, so that we can build empathy with our customers, to allow us to build better products.

Thank you

A big thank you to Dipa Rao, our volunteers Gwen and Nosh, and to our generous host and Zoom sponsor, A Cloud Guru – they’re on a mission to teach the world to cloud.

Resources and Slides

You can find Dipa on LinkedIn and Twitter. 

Some of the resources mentioned during this session included:

Continuous Discovery IRL – June 2021 Wrap

Posted on by
Reply

More companies are realising that the path to success includes knowing their customers better. As they commence their journeys towards continuous discovery to deeply understand their customer and their problems, there can be some lofty expectations. However, the reality is often very different, and can be fragmented, chaotic and full of barriers. 

In June, we caught up with Ben Ryan who shared 3 lessons from his experiences at FatSecret.

Lesson 1: Challenging the Status Quo

How well do we really know our customers? Just a little, ie. at a surface level? Perhaps it’s as little as reading the feedback or complaints that actually make their way to us. Or possibly even less than that. 

Recognising this as a problem and the need for change is only the first step. However, making the case can be another story. 

Resistance can coming in various forms:

  • Sacred cows – firmly held beliefs that are rarely questioned and/or exempt from criticism or opposition, even in the face of contradictory evidence;
  • Brittle foundations – legacy systems which are too hard to change;
  • Sunk cost fallacy – ‘we’ve already invested so much in this direction/system/platform, so it’s too late to change now’;
  • Or just being comfortable – changes require effort and moving towards the unknown.

Teresa Torres talks about having weekly conversations with your customers.

However, FatSecret was far from that. They had convoluted pathways for customers to make contact, and a small team (of 3 people) already inundated with 200-300 emails per week. This drove a fear of scaling, and not being able to handle additional volumes if the floodgates were opened. Furthermore, a dispersed customer base, with less than 1% within Australia, introduced additional geographical, cultural and linguistic challenges.

As an app, the other option for your customers is to leave public (1 star) reviews on the app store for others to read. Not a great alternative.

Even though there was no clear line to customers, they got scrappy and improvised. They also kicked off a longer process of a qualitative and quantitative research project with an external agency. Its brief was to gather insights, and begin co-designing the future vision. Once they started to build things that customers were vocalising as their pain points, retention started to improve, and there was an increase in active usage and engagement.

Lesson 2: Revisit old ideas with fresh eyes

There will always be some ideas that don’t work out. But it is also good practice to revisit those ideas later. What was wrong with those ideas, and what were the conditions the first time around? Are those conditions still in play, or has the environment changed? Do you have new lessons or understand things differently now? With a different perspective, you may find elements worth pursuing or that can be repurposed. And other times, they may turn out to still be bad ideas.

To try and place customer experience at the forefront of decisions, FatSecret introduced a discovery phase at the start of projects. Surely doing discovery about a business objective would help them identify customer needs earlier, to ensure they were solving customer problems?

Unfortunately not. 

In reality, Project Discovery was unsuccessful as:

  • Discovery didn’t begin until the project had been incepted around a business goal;
  • Having a big deliverable at the start of the project, required time to do the customer discovery;
  • There was high overhead (and difficulty) finding target users with the problems that aligned with the business goal;
  • Discovery became a reflection of the customer experience, producing only a limited and blinkered view of the customer problems, and not going deep enough into the wider customer journey to understand the context of the problems.

Lesson 3: Continuous Improvement

Invest in the time for reflection.

Similar to re-evaluating past ideas, it can also be beneficial to circle back to past decisions and assumptions, and challenge your original thinking. Were the assumptions correct? Would the potential outcome be different from what you know now? Are you pursuing the best options, or are there better opportunities available?

Four years on, FatSecret has started moving towards Continuous Discovery to generate insights. Customers are interviewed on a regular basis, and then mapped to overarching archetypes, to understand their various drivers and how they will respond to different types of obstacles.

There’s still a long way to go, but FatSecret have progressively put distance between what used to be their status quo, and where they are now.

Resources and further reading

Some of the resources mentioned during this session (and a few bonus ones too):

Thank you

Thank you to Ben Ryan (Head of Product at FatSecret) for sharing, our volunteers Gwen and Nosh, and to our generous host and Zoom sponsor, A Cloud Guru – they’re on a mission to teach the world to cloud.

Slides and video

Find your ‘Get out of Hell’ cards here – July 2020 Wrap

Posted on by
Reply

With everything becoming remote and distributed, businesses are forced to adapt. Explore new opportunities, or find a silver lining. The alternative to wither and become a mere memory. And we’re no different.

Taking advantage of lockdown, we had Jock Busuttil make his long awaited return to Product Anonymous in July, all the way from London, to share some of his experiences of an all too familiar place – product management hell.

The Symptoms – What does Product Hell look like?

There are many common indicators that you may not be in the healthiest product environment, such as:

  • Not allowed to talk to customers. The complete opposite of continuous discovery, and not validating your ideas with customers. From concept, to build, to launch – talking to customers is always important. 
  • Unable to plan, because you’re too busy dealing with emergencies. Although it is important to put out fires, it can also wear you down. It’s equally important, if not more, to know which direction you’re heading. Having enough foresight to know which areas you need to invest your time and resources in, and which areas or features should be retired. 
  • Screw research, let’s build. The build trap. Do we really need to say anything else on this one?
  • But we have OKRs – hundreds of them! If you have too many Objectives and Key Results, which ones actually matter? And if different business units have different objectives, and lack of transparency across the rest of the organisation, how do you actually align with each other? 
  • Flip-Flopping between Very Important Goals. Do the goal posts keep moving back and forth from quarter to quarter? Oh no, that’s not important anymore, let’s move on to something else instead. Maybe keep your research handy for the next time it becomes a priority again. Probably next quarter.
  • No buy-in for my product strategy. If you’ve done all the adequate research, and validated those assumptions, and know the balance points – who better to drive the strategy? Or should we go by the opinions of everybody else instead?
  • Each board member has their own interpretation of the strategy. Whether this is to minimise the effort for their teams or maximise the benefit for their team, neither is healthy, nor going to help to align everybody’s efforts.

The Causes – Why are you in Product Hell?

So you’ve discovered you’re in Product Hell. Population: one. But how did you get here? Here are some possible and likely causes:

  • No clear corporate strategy or goals. Is your company vision to be the market leader in something generic? A good corporate strategy should be rooted in customer outcome. A true north star to align all your efforts. But what does a clear corporate strategy even look like? Here’s a fantastic example from Tesla.
The Secret Tesla Motors Master Plan – 2006
  • Lack of alignment. To ensure alignment, you may need to prioritise the things to focus on. But prioritising is also about calling out the things that you won’t be spending energy on, right now. 
  • Wrong strategy (for now). You may have a strategy that has worked for you in the past. But that doesn’t necessarily mean it’s still the right strategy now. Have the market conditions shifted? Has the competitor landscaped changed? Nobody, including Zoom, were planning for Covid to happen.
  • Wrong measures of success. NPS, Revenue and Market share are not tied to human outcomes. These metrics could change due to external factors, without you doing anything. 
  • Scared of user research. Too many companies are scared to approach their customers to see how they are doing. How do you uncover unmet and underlying needs if you never talk to your customers?

Getting out – How to escape Product Hell?

Now that you’ve identified the problem, what can you do about it?

  • Start with real user research. Deepen your customer insights. Understand their needs. The problems they need solved And what they would be willing to pay for. Like all things product, it starts with the customer. 
  • Make your product strategy before somebody does it for you. Gather the research. What does the data suggest, and what needs further validation? Ensure you use the right research for the right situation – different techniques will have different biases built in. Be aware of the biases, so that you can balance your view with other research techniques. Use the insights to form a compelling product vision and strategy. 
UX Research Methods – Nielsen Norman Group
  • Influence the corporate strategy with your product strategy. Talk to your leaders to understand each of their concerns and motivations. Create a shared and aligned vision, and get them to agree with your product strategy. It might be a long path, however, it can be done. 
  • Call out your Board’s lack of alignment… tactfully. This could also apply to your executive leadership team, or any other management layer or structure in your organisation. Warning: Proceed with caution!

Jock Busuttil, Founding Director at Product People

Jock is a freelance head of product, author and conference speaker, having spent nearly two decades working with technology companies to improve their product management practices. From startups to multinationals, his clients include the BBC, Brainmates, and the UK’s Ministry of Justice and Government Digital Service (GDS). In 2012, Jock founded Product People Limited – a product management services and training company. And his book, The Practitioner’s Guide To Product Management was published in 2015. You can find more of Jock on LinkedIn and Twitter, or on his blog – I Manage Products.

Watch for Jock’s upcoming Product Management masterclasses in October. Keep an eye here for details: https://productpeo.pl/linktree/

Thank you to our sponsor: A Cloud Guru

We’re on a mission to teach the WORLD to cloud. A Cloud Guru is the largest online cloud school on the planet. Our training feels more like logging into Netflix or Spotify – it’s entertaining and playful. The people are the #1 reason employees say they stay at ACG. We’re a quirky, tight-knit crew that cares about our customers and each other. No egos here. Our leaders encourage thoughtfulness, compassion, being humble, and we have a bit of fun along the way.

Slides & Video

Using Mental Models for Product Success – November 2019 Wrap

Posted on by
Reply

Can you believe that another year has flown by? With so much happening, it’s not surprising that it has gone by so quickly. Eight meet-ups, ranging from roadmaps to Wardley Maps, exploring continuous discovery and mental models, diving into OKRs and NPS, and putting ourselves in the shoes of some entrepreneurs. Amongst all this goodness, we also had another Leading the Product Conference and Product Camp!

For our final event of the year, Tafida Negm, an independent Human-Centred Researcher and Designer with a background in Marketing and Psychology, took us through the Mental Models framework by Indi Young.

The Problem Space and the Solution Space

Most of us will be familiar with Gartner’s Design Thinking, Lean Start Up, & Agile Delivery diagram. However, according to Indi, that’s all part of the Solution Space where past work informs future work. Why do we feel so comfortable here? Because we’re rewarded for ideas – and we’re rewarded for speed of delivery.

With mental models, we try to move earlier in the cycle and focus on the person and what they are trying to achieve.

  • What are they thinking?
  • How are they reasoning their way towards their intent?
  • What are they feeling?
  • What are their beliefs that underpin their (in)decision or actions?

If we can understand this & develop true empathy, then we can have a better opportunity to design an aligned solution and have the customer think:

‘Wow, it’s like that product was made for me’

What are Mental Models?

This is a bit of a loaded question, as it is applied in so many different contexts, from psychology, to machine learning and behavioural perspectives, there are little different nuances.

Indi Young defines them as: “Mental models give you a deep understanding of people’s motivations and thought-processes, along with the emotional and philosophical landscape in which they operate.”

You may have seen them represented as a skyline, but we’ll delve into that a little more in a moment.

Mental Models – Photos courtesy of Steve Cheah and Irene Toh

Listening Sessions, Cognitive Empathy and Patterns of Intent

Tafida led us through a few exercises because what better way to learn than to get hands-on? We started with listening sessions, where we used active listening to try to develop cognitive empathy. 

The aim of cognitive empathy is to gain that deep understanding of people. You want to understand so well that you could walk in their shoes and make decisions exactly as they would.

How many listening sessions should you do? As many as you can, until there are no new themes coming through.

https://www.instagram.com/p/B5INVyqHyRd/?utm_source=ig_web_copy_link

After conducting listening sessions, it becomes time to document and synthesise the results by grouping or looking for patterns based on intent (which creates the towers in the skyline visual). We group the towers into mental spaces.

Next comes identifying the different mental spaces that people are going through. Then you can start arranging the concepts into your own skyline.

Once you have built your mental model, which includes the thinking styles that people go through, you can use it in many ways:

  • Map your organisation’s support underneath the respective towers to show where you have gaps.
  • Overlay competitor capability for analysis.
  • Broaden your market by supporting more thinking styles.
  • Overlay other data (usability metrics)
  • Many more…

When to use Mental Models?

There is a range of scenarios where researching Mental Models can be useful, such as:

  • When you want to innovate in a new direction;
  • Strategise broader and farther than your current solution;
  • When you spend a lot of time re-architecting / re-inventing;
  • When your existing user research is fragmented; or
  • You recognise you are out of touch with your audience:
    • You think everyone is your user
    • Make-believe and assumptions drive design decisions
    • No improvements after test and iterate cycles.

Another great benefit is the research can be re-used for other problems, as we’re not focusing on solution space.

Resources and Further Reading

Tafida Negm’s key takeaways from Indi Young’s Mental Models Methodology

Thank you Envato for hosting our last event of 2019, and all our Prod Anon volunteers that have helped throughout the year.

envato

The next Product Anonymous will be February 2020. Sign up for the newsletter, follow on Twitter or Meetup to stay in touch over the break. Have a festive amazing couple months!!

Using Mental Models for Product Success

Posted on by
Reply

Lately ‘outcome, not outputs’ is the subject of lots of conversation within product & agile circles. How do you know what those outcomes should be and what will truly support your customers?

RSVP for ProdAnon’s November session on Thursday November 21st to find out.

No single methodology will help you create the perfect product, but you can increase your odds by understanding people’s deep, messy thinking and reasons for doing things. Mental Models gives you the tools to uncover and design for those reasons. 

Developed by Indi Young, this framework helps you curb assumptions & cognitive bias through a bottom-up approach to data analysis. With less bias and greater clarity of opportunities, this approach will help you more closely align design possibilities to your customer’s needs and your organisation’s capabilities.

Our speaker, Tafida Negm will walk us through some of the important concepts within Mental Models after introducing what it is and why you should incorporate it into your toolkit. There will be a few activities to aid you in having a go and gaining some confidence in trying it yourself when you get back to the office.

About the Speaker

Tafida Negm is an independent Human-Centred Researcher and Designer coming from a Marketing and Psychology background. As a consultant, she has gained a wide variety of experience across for purpose and commercial contexts helping lead teams from discovery through to launch. Witnessing the value research has delivered in shaping products and services, she has been on a mission to continually hone her research skills. Having spent the past year learning from Indi Young she is passionate about spreading her love of problem space research.  

RSVP now. Doors open at 6:00 pm. Talk starts 6:30 pm

May wrap-up: Working with User Research

Posted on by
Reply

WOW! What a fantastic evening! If you’re interested in user research, I hope you were there! Otherwise read on for the summary.

Big thanks to Fiona Knight, one of our volunteers, for taking great notes!

Using Experience Sampling for Rapid Insights into User Needs

George Cockerill started the evening off with a discussion about how his team approached gathering user research in a project building a smart assistant for students at Deakin University.

The team decided to use the Experience Sampling method. Other methods like interviews & diary studies have participants recall their past whereas experience sampling gave them access to detailed immediate needs.

With experience sampling, you ask short, easy to answer questions throughout the day. For this project, the same two questions were sent to participants eight times a day over a week. Occasionally the team would add an optional question. Experience sampling allowed the team to get to very recent needs & the student’s behaviour.

To collect the data, George used PACO, a free app that builds experiments quickly. It’s great for experience sampling and since it’s an app on participants’ phones they can take photos to include with their answers.

Tips for success

  • Get maximum value for the method – be clear about the research goal
  • Don’t rush, plan meticulously & plan to be flexible. Plan even when you need to go fast (it actually helps you move quickly!)
  • It’s a complex set up – test it; test your questions and mechanics
  • Get the most from people – recruit and onboard carefully. Establish how much data points you need to help determine how many participants you should have. When you screen them, ask questions along the lines of what you’ll be asking during the research. If they can’t answer in the screener, they probably won’t provide good data during the research. Your onboarding needs to have very clear instructions. George made a video to help explain the app and expectations of the participant (including what they needed to do /when they’d get paid).
  • Monitor it! Experience sampling is not a ‘set it & forget it’ method. You can be encouraging to the participants or ask them to tell you more about a specific thing. It’s awesome if you can instill a shared purpose of what’s going to happen with the data they provide.
  • Use the data to help refine the problem. Start finding themes, create categories, map answers to categories to find patterns, map data to graphs and examine categories to fine tune them. Look for trends. With participants using their phones, images will pick up detail the users might not think to say like how they solve their problems, their hacks, etc. You can combine the data with other research (both qual & quant). For this project, George combined experience sampling, resonance testing, journey maps from user interviews and ‘day in life’ models.

Currently the student smart assistant is in a pilot with a set of students. We’re looking forward to hearing how it goes!

George Cockerill on experience sampling #prodanon

A post shared by Product Anonymous (@product_anon) on

George recommended two excellent references that he found helpful in planning and running the research:

Why is marathon running important when introducing user research to an organisation?

Electronic Arts operates 8 research labs across 4 countries but it’s still early days for the games industry to embrace user research. In the APAC region, Kostas Kazakosis the 1st UX researcher at EA and thus needed to help communicate the value & importance of UX research.

Using ‘The Reflective Practioner‘ by Donald A Schon, Kostas reflected on his journey as a marathon runner and introducing UX research to a company – both ongoing!

Kostas finds there’s 8 stages to long distance running

  • Excitement where anything is possible! The organisation has no prior exposure to formalised UX research.
  • Denial when doubt starts to creep in. Here Kostas realised he didn’t know much about the FireMonkeys’ development process.
  • Shock where everything seemed to be really difficult. Kostas had to show the value of UX research, create a research space and creating research protocols to suit the games industry. Kostas’s plan was relying on the importance of dialogue: talking to people, understand what they do, and showing UX research helps them. This is where empathy is important but how do you do this? You empathise with the data and people and match it to research question.

  • Isolation or the fear you’re not going to make it. Having talked to as many people as possible, you need to accept the feedback and adjust your approach
  • Despair usually happens about mile 19 in a marathon. When you start to question if you really can do this, if you’re ready to do this and will it be ok?
  • The wall at mile 22 is when your brain doesn’t talk to your body anymore. You need to take one step at a time to keep going & identify mistakes and address them quickly. At EA, this meant all that earlier work to establish relationships with the team enabled collaboration. He had helped educate them & gave them ownership. The teams had begun the process of seeing the importance of data & value of user research.
  • Affirmation at mile 23 is feeling like you have a break thru. This is the stage Kostas is currently at work. He’s seeing people want to get involved in the UX research and wants to be able to sustain that team participation.
  • Elation at the finish line of mile 26 is when you’ve achieved the goal and need to shift your mindset towards the future. At work, this is what Kostas is working towards.

In summary, the DECEMA frame of reference that Kostas described – Dialogue, Empathy, Collaboration, Empowerment, Mutual respect and Advocacy – can take you a long way when introducing user research to your organization!

Thanks to both George & Kostas for a great evening and to Seek for hosting us!!!

seek