There’s a saying that good judgment comes from experience, and experience comes from bad judgment. At our last event, Dave and Amir leaned into that idea – hard. Hosted as a game show loosely based on Who Wants to Be a Millionaire, they spun a wheel, pulled audience members up as contestants, and used the chaos to share some of the most honest leadership stories we’ve heard on a Product Anonymous stage.

Here’s what came out of it.

---

Earn the Right

Product people are opinionated. We’ll swim in anyone’s lane – engineering, leadership, customer experience – and we have thoughts we’d like to share. The problem isn’t the opinions. It’s the timing.

Both Dave and Amir had stories about arriving somewhere new with ideas, and learning the hard way that credibility isn’t assumed – it’s accumulated. The concept they came back to: idiosyncrasy credits. Deliver consistently, meet expectations, make the people around you look good – and you build up the right to eventually push back, deviate, and step outside your lane. Skip that step, and nobody’s listening anyway.

---

Set Yourself Up

Dick Fosbury revolutionised the high jump by going over the bar backwards, head first, not seeing where he’d land. What made that possible wasn’t just the idea – it was that his school had just introduced foam landing mats.

Bold moves require crash mats. Amir had a vivid story about needing to challenge a major decision that had already been signed off at C-level – and the quiet, patient groundwork he had to lay before he could make that case without torching his credibility or his career.

The lesson: it’s not the boldness that works. It’s the preparation that makes the boldness survivable.

---

Shoe Maintenance

If you’re walking down the street and step in dog shit, how do you react? Because as a leader, you’re going to be stepping in a lot of it.

Many product people treat dealing with mess as a distraction from the real job. Dave’s point was simple: dealing with mess is the job. We’re drawn to an idealised version of leadership – vision, strategy, craft. Most of the time it’s closer to garbage collection. The sooner you make peace with that, the better a leader you’ll be.

---

Lead the Humans, Not the Projects

Products fail. Companies get acquired. Strategies get shelved. After 25-plus years, Amir’s conclusion is that almost everything you deliver is temporary – except the people you help grow.

Dave told a harder story about a redundancy round, and what happened in the hours after he’d delivered the news. The idea he landed on: leadership shadow. Everything you do and say casts a shadow across your team – your standards, your values, how you treat people on the way up and on the way out. It accumulates quietly, over dozens of interactions. And sometimes it comes back to you in ways you don’t expect.

---

Play the Long Game

Dave got his Miro year-in-review: top 1% of users. He assumed the most-used feature would be Post-it notes or user flows.

It was tables.

He’d become what he’d sworn to destroy – and then made peace with it. Leadership runs on data, budgets, capacity planning, and the unglamorous work that keeps things moving. Stanford professor James G. March called it poetry and plumbing: most people who want to be leaders are drawn to the poetry, but most people who are leaders spend the majority of their time in the plumbing. The best ones aren’t too proud to do either.

---

Thanks to Dave and Amir for bringing both the stories and the game show energy. And to our five brave participants – Lisa, Pranav, Jaden, Kate, and Malay – for being good sports.

Our Speakers:

Amir Ansari: With over 25 years’ of experience in the field of design and design leadership, Amir has worked across the private and public sectors, in-house and agency, managing design practitioners and leaders to build long-lasting and impactful products and services. Amir is passionate about democratising the craft of design and sharing his learnings – doing it with a smile and a hug.

David Bacon: After graduating from Monash with a Bachelor of Science, David worked as a penguin keeper in England which somehow led to a career spanning marketing, behaviour change, innovation and human-centred design. David has experience delivering digital products and services in financial services, healthcare, retail, government. He has the goal of weaving humanity into technology. He still dreams about penguins.

There’s a moment in almost every product career where you build something thoughtful, ship it carefully, and then watch people completely ignore it. Not because they don’t care – but because changing behaviour is genuinely, stubbornly hard.

That’s the problem Alex Waddell, behavioural and implementation science practitioner, researcher, and lecturer, has spent her career trying to solve. At our recent Product Anonymous session, she gave us a fast, hands-on tour of behavioural science and implementation science – and how product people can actually use both.

---

First: What even are these sciences?

Behavioural science is an interdisciplinary field drawing on cognitive science, economics, psychology, sociology, and neuroscience to study human behaviour and design strategies to change it.

Implementation science – which was born out of healthcare, where it takes an average of 17 years for research to reach the patient bedside – asks a different but related question: how do you get something that works into an organisation or system with speed, fidelity, and quality?

One is more micro. The other is meso and macro. But both ultimately boil down to the same question:

“Who needs to do what differently?“

Alex said this was the one thing she wanted us to take away. It’s deceptively simple – and it cuts through a lot of the vagueness that plagues change initiatives.

---

Step one: Get brutally specific about the behaviour

Before you can change a behaviour, you have to specify it. And this is harder than it sounds.

Alex asked the room: what behaviour am I doing right now? In about thirty seconds, the audience shouted out ten different answers – presenting, talking, staring, moving around the room. They were all correct. And that’s exactly the problem. If you can’t pin down the specific behaviour you’re targeting, you can’t measure it, and you can’t change it.

To get specific, Alex introduced the AACTT framework – a tool for mapping behaviour across five dimensions:

• Action – what can be observed and measured? If you can’t observe it, you can’t change it.
• Actor – who specifically is doing (or not doing) this behaviour?
• Context – the physical location, the social setting, and critically, the emotional context.
• Target – who is the behaviour being performed for or with?
• Time – when is it happening?

The emotional context piece generated a lot of discussion in the room. Participants reflected that we tend to design for the rational user, in a neutral state. Real behaviour happens in emotional contexts – rushed, distracted, anxious, under social pressure – and those contexts matter enormously.

---

Step two: Stop going straight to solutions

Once a behaviour is specified, the instinct is to jump to solutions. Alex made the room do exactly this – using a photo of a woman examining apples in a supermarket, she asked how we’d increase apple sales to her.

The ideas came fast: discount signs, taste testers, better labelling, smaller bags for easier carrying, large-font labels, recipe cards. All reasonable. All based on assumptions.

Then she asked: what assumptions are we making about this woman?

The list was long. That she has grandkids. That she has poor eyesight. That she can afford apples. That the apple is even for her. That she buys apples at all.

“We need to be really, really careful about assumptions. People are not like us. They don’t have your lived experience, they’re not from where you’re from, they’re dealing with other things.”

The exercise was a pointed reminder that we design for ourselves more than we realise – and that good research with and for people is non-negotiable before you reach for solutions.

---

The frameworks: COM-B and EAST

Alex introduced two frameworks worth keeping handy.

COM-B

Synthesised from 33 theories, models and frameworks by Susan Michie and colleagues at UCL, COM-B says that for any behaviour to happen, a person needs three things:

• Capability – psychological and physical ability to perform the action (knowledge, skills, stamina)
• Opportunity – external factors that make the behaviour possible (time, resources, location, cultural norms)
• Motivation – internal drivers, both reflective (conscious plans and beliefs) and automatic (impulses and habits)

One insight that landed particularly well: motivation includes beliefs about other people’s capabilities, not just your own. Alex drew on her PhD research into shared clinical decision-making – where some clinicians assumed certain patients didn’t want to be involved in decisions about their care, and so never offered it. A belief about someone else’s capability, cutting off behaviour before it started.

EAST

A more action-oriented framework from the UK’s Behavioural Insights Team, EAST gives you four levers:

• Easy – reduce friction, simplify messages, use defaults
• Attractive – make it stand out, use images, personalise, consider rewards
• Social – show what others do, use networks, make commitments visible
• Timely – prompt at the right moment, consider habits, plan for barriers

Alex walked through a famous cautionary example: a sign that read “Littering in our local area has increased” – intended to encourage people not to litter, but framed as a social norm that actually increased littering. When you tell people that something bad is happening a lot, you inadvertently signal that it’s normal.

The lesson: always test before you roll out. Behavioural interventions can backfire.

---

There are 93 behaviour change techniques

One of the most eye-opening moments of the evening was learning there are 93 identified behaviour change techniques – and that the instinct most organisations reach for first (training, education, an online module) is just one of them.

“People most often say, ‘let’s just do some training and they’ll [the people whose behaviour you are trying to change] will do it.’ They probably won’t. Or they might, and probably not everyone.”

The richness of options is the point. If your current approach isn’t working, there are dozens of other levers to try – and a growing body of evidence about when each one works.

---

The key takeaway

At the end of the session, Alex asked the room to spot how she’d used behavioural science in the workshop itself. The answers came quickly: prizes for participation (incentives), the active listening exercise at the start (priming for engagement), the Slido polls (social proof, visibility), the hands-on activities (behavioural practice) rather than passive listening.

It was a neat way to close – and a good reminder that these frameworks aren’t abstract. They’re design tools. Every product interaction, every onboarding flow, every survey you send is a behaviour change attempt. The question is whether you’re doing it intentionally or not.

---

Want to go deeper? Alex has compiled a resource guide with links to the frameworks mentioned – including the behaviour change technique taxonomy, the Theoretical Domains Framework, and the EAST playbook. Grab it, then find Alex at coformed.com.au to see how she helps teams turn good ideas into lasting change – and get in touch. Connect with her on LinkedIn too.

Thanks to our hosts

Thanks Zendesk for being our wonderful host for the evening!

Editor note: citations for works quoted or referenced here are included as direct links to the source material.

The following is a recap of a panel discussion on AI in product, engineering, and operations. Panelists were Daragh Kan (Head of Product, Cuttable and hospitality entrepreneur), Dave Slutskin (Cadence), and Melissa Cupples (Head of Intelligent Products and Automation, Humanly Agile). The session was facilitated by Liz Blink, Co-founder of Product Anonymous.

---

Over a year ago, a small Slack channel was set up with a simple purpose: help each other figure out how to use agents. It looked really hard then. It still is – but the pace of change has been remarkable, and the conversation has moved with it.

This panel brought together product and engineering leaders who are living in that change every day: building AI products, running their teams with AI, and trying to work out what the role of a product person actually looks like in 2026. There were no tidy answers – the panelists were the first to say so – but there was a lot of hard-won experience, a few genuinely useful frameworks, and one story about chicken salt that the room won’t forget.

If you’re a product person trying to stay on the right side of this shift, there’s something here for you.

---

Who’s in the Room

Daragh runs product at Cuttable, an AI-powered ad tech startup working with e-commerce and service brands – taking brand assets, augmenting them, and pushing them out as ads across Meta, TikTok, and other platforms. Outside of that, he’s been running bars and restaurants for about 20 years, which gives him a unique lens on bringing AI into non-technical, sometimes smartphone-free environments.

Dave has spent 25 years in tech – writing code, managing products, and leading developers. His current company, Cadence, looks in minute detail at how developers use AI coding agents, helping individuals and their managers understand what good looks like in a world where that’s still being figured out. “I suspect if we come up with any answers tonight, I think we’re probably doing it wrong – because there’s no right way to do any of this stuff at the moment.”

Melissa leads engineering and product at Humanly Agile, an AI consulting and product-build company. Her team builds AI products for clients – internal and customer-facing – while also using agents to run their own operations. “Pretty much everything we do is powered by agents that allows us to do 50 times the work we could do with our very small and constrained teams.”

---

The Pitfalls

Liz: You’ve all stumbled along the way. What were some of those moments?

Melissa: I come from an engineering background, so I was using AI tools early – back when everything was being introduced. That was my first experience of how things could go wrong. Using agents and putting system tools in that environment, you quickly see how things spiral out of control. Things are quite good in a greenfield environment, but as soon as you hit any complexity at all, it can get very unwieldy. A lot of what we’ve learned is about how you control your environment, your context, your structure. There’s quite a lot of thinking and scaffolding that needs to be in place for effective use of AI in any context.

Dave: There’s a mental model thing we’re all developing, and I think it’s weird. Sometimes it feels like you’re working with a co-worker, and sometimes it feels like you’re working with a tool. With a co-worker, when something’s misaligned, you need to work it out – because it’ll come up again. With a tool, you can just kill it and start a new one. What I see with people who are new to this – and what I sometimes still catch myself doing – is spending a bunch of time arguing with the agent for no reason. It doesn’t get you anywhere. There’s no value in those tokens you spent, or in the tokens it spent apologising and justifying. I think one of the pitfalls is this uncanny valley – it sort of feels like a co-worker, but it’s not.

Daragh: There’s also this single-player mode problem. Before AI, if I needed help from Dave, there was a visible record – a Slack message, a conversation. Now a lot of that’s just happening in chat windows. On the flip side, I can get immediate help without waiting a week. But I’m noticing there are fewer questions being thrown around. People are quieter. And it’s also really easy to start with very little planning, which can be good – throw some ideas out and see what emerges – but it can also mean you’re three minutes from deploying something you haven’t properly thought through.

---

The Joy

Liz: What was your “oh, f**k yes” moment?

Melissa: For me, it was truly solving a customer problem that could not have been solved without generative AI. A paradigm shift of: wow, this wasn’t even possible before. That’s why I get up in the morning – we can actually look at all of these problems that until now couldn’t be solved.

Dave: For me, it’s doing things I wouldn’t have got around to. I can just say: go look at all our competitors, understand what they did last month, find any new ones that have appeared, map the landscape. That’s foundational product work I used to revisit every six to twelve months. Now it just happens. I still find it critical to build product vision and strategy in my own head – I’m not outsourcing that – but having the data pulled together and accessible? That’s pretty fantastic.

Daragh: I fired our agency managing restaurant ads and inherited a Google Ads account I hadn’t touched in five or six years. I opened it up with Claude, said “what is going on here,” and it kept asking for more access – Analytics, search terms, Shopify data, organic rankings. I kept saying yes. And then it came back with an idea for an ad that I would never have thought of, because it had pulled threads together across all of that data. We made an ad for our chicken salt. It was great.

(Chicken salt. The audience loved this.)

---

Evangelism and Pushback

Liz: Have you ever had to be an evangelist? Or are you ever behind anyone else?

Daragh: We’re not regulated – we sell shoes and things, it’s all visible. I’m admin on everything and I can just enable things, so I’ve never really had to fight for it internally.

Dave: What I’m seeing a lot is that it’s often marketing or sales who think AI will magically make their life easier – and maybe it can – while the tech teams and product and design people are pushing back. And when you dig into why, it’s obvious: they have tight deadlines and no time to experiment. The CTO wants AI-first everything, but they haven’t set up the conditions for that to work. You can’t just hand someone a $20 licence and say “experiment, but don’t miss any deadlines.” The process changes, the workflow changes – people need space to learn that.

Melissa: I’m actually the constraint-maker in our business. I’m the one slowing things down, asking: are we solving the right problem? What are the constraints? We’re a small team, and my colleagues are even more enthusiastic about this than I am – but if we need to stop and rethink rather than keep going in the same direction, that’s encouraged. You need a really supported team and aligned incentives for that to work.

---

From the Audience

On arguing with agents

Audience member: I find it useful to say “don’t do this again” – because it stores that and it doesn’t happen next time.

Dave: That’s a really good point. The mental model is: I’m training this thing to create better conditions going forward. Not literally training, but steering. What I was talking about is more when it misses something in the middle of a task and I find myself swearing at it – which is fun, but doesn’t help.

---

On deterministic vs. stochastic decisions

Audience question: Do you have a conscious process for deciding what should be deterministic code versus what goes to the LLM?

Melissa: This is architecture 101, and it’s increasingly important given how token pricing is evolving. You have to assess how deterministic the outcome needs to be – is it customer-facing? What industry? Does it involve sensitive data? There’s a spectrum from a fully deterministic, orchestrated workflow through to autonomous, goal-based, multi-agent reasoning. I personally spend a lot of time at the deterministic end – the LLMs are used for things they’re genuinely good at: generation, judgment, working with messy data. Everything around them is code, harness, and structure. We started with more autonomous agents, swung hard the other way, and now good frameworks are emerging that let you move back toward goal-based reasoning while still having visibility and control over what’s happening. The most interesting question for me right now is: how do you build agents that are genuinely trustworthy? I don’t think that’s been answered well yet.

Dave: One thing I’d add: you don’t usually have to think about the cost of an individual request in traditional product development. It’s approximately zero. Now it’s not. From a product perspective, if you’re building AI products, you have to understand whether this is going to be a problem for your margins.

---

On shared context management

Audience question: Individually, I’m converting a lot of context into markdown files. But I’m part of an 850-person organisation with many PMs all in their own local environments. Has anyone cracked shared context management?

Dave: It’s something I think about but don’t have a good answer for yet. We use Notion, which helps, but it’s not perfect. I have delusions of restructuring everything properly, but that’s not happening soon. The interesting question is: how do you get information into a structure where people can see the benefit of contributing to it?

Dave (continued): There’s a fuzzy but real boundary between personal context and organisational context. Same with skills – the immediate instinct is to share everything across the team, but I actually think that’s often an anti-pattern. People develop their own skills, know how they work, know how to use them effectively. Sharing too broadly can dilute that. Some context should be team-wide; some probably shouldn’t.

Melissa: We’ve turned context into a repo. Backend, frontend, and the repo itself as context. For shared context, we use version control – submitting a PR to update context, which has a governance layer built in. Everything is versioned. We share all our skills, everything goes through engineering processes. It’s interesting how many layers this has – what’s local, what’s shared, what’s in a vector database, whether you’re fine-tuning a model. Different strategies depending on what the context is for and how you need to access it.

Daragh: The danger is that if you’re the person who owns this, suddenly everyone’s coming to you all day and you have other work to do. And telling people how to do things rarely ends well. What I’ve found is: make people jealous of what you’re doing. Circulate outputs. Show the value. Let people come to you asking “what are you doing?” rather than mandating a process from the top.

---

On expanding your skills beyond your role

Audience question: People in product and engineering are starting to do things that sit outside their traditional role – writing code, making design changes. Where does that work well, and where does it cause friction?

Dave: I’ve seen friction in organisations where roles are smearing. I wrote some designs but is the designer happy with it? I wrote this code but are the developers happy with it? There are real human challenges when no one has communicated whether that stretching is sanctioned. People don’t know what their role is, and leadership often doesn’t know either – because no one at any level of the organisation has figured it out yet.

Liz: Our roles have changed, but we haven’t necessarily realised it yet. “That’s my role, that’s your role” – those boxes might already be outdated language.

Melissa: I walk around with what feels like split personalities – I have both an engineering and a product mindset, and they approach problems differently. There’s a natural tension between them. I wonder: if you don’t have the metacognitive awareness to know which hat you’re wearing, is it too easy to be led by the AI or external forces in a direction that loses the productive friction that tension creates? That tension often leads to better outcomes. Smearing – [laughter from the room] – might actually reduce something valuable.

Daragh: Process creep happened before AI. With AI it’s just going to go faster. Everyone optimising locally doesn’t necessarily make the whole system better.

---

On staying in the loop vs. letting it run

Audience question: I’ve been building prompts and augmenting my skills for three years. I always imagined automating more and more, but I keep wanting to stay in the loop because any response could go the wrong direction. Am I thinking about this wrong?

Dave: The best harness for a coding agent includes an engineer. Ideally an experienced one. It’ll make trade-offs it doesn’t know are trade-offs – UX versus security, scalability versus cost. You’re responsible for those decisions. You can get it to work autonomously if you plan seriously: maybe an hour to an hour and a half of upfront planning for a sizeable change, and then it’ll go away and work for a few hours and get something close. But you’ll still iterate. The human still has a huge role. It feels like we’re close to full autonomy if you squint, but I don’t actually think we are.

Melissa: What’s interesting is: imagine if we set up our human team members for success the way we’re now starting to set up AI. A new engineer joins, looks at the codebase, reads the README, and has no idea what’s going on. But now there’s real motivation to document things well – because the AI needs context. Documentation is becoming part of the CI/CD pipeline. That’s a side effect I didn’t expect.

---

On evaluations and model stability

Audience question: What’s your evaluation strategy, especially with models changing and updating underneath you?

Melissa: It’s not that different from traditional engineering. You start with “what does success look like”. You break that down into discrete, quantifiable metrics, which become your test cases. For non-deterministic parts of the system, a simple pass/fail string match doesn’t work – you’re testing for intent, quality, tone. We use synthetic test cases, human reviewers to establish ground truth, and LLM-as-judge depending on the architecture. It’s always multi-layered, and it always starts with: what are we trying to achieve?

Dave: We’ve seen models change in ways we didn’t ask for. We watched one model get noticeably worse at coding over a period of time – apparently because resources were pulled to train the next version. We’ll have things just suddenly start failing with no changes on our side. That’s why running regression suites regularly matters. It’s not that different from any dependency that can silently change on you.

---

On model portability and architecture

Audience question: How much do you think about building products that aren’t tied to a single model or provider?

Daragh: Everything is built to jump ship at any second. There’s no loyalty to models. If a new one comes out in the morning, we’re ready to move.

Melissa: We’re model-agnostic but deliberate about choice. Different models are better for different things – some are great at JSON schema validation, some are better for generation and imagination. Token costs and prompt engineering differences between providers are real: a prompt that works beautifully with one model often needs significant reworking for another. Once cost becomes a real constraint – and it’s getting there – you’ll need to invest proper time in working out the minimum viable model for each job.

Dave: One nuance: if you want personality from a model, that’s where it gets hard to port. If it’s behind the scenes and nobody sees it, you can often swap models around fairly easily. If users interact with it directly, the personality will differ and your prompts will need fundamental rethinking. Also worth noting: with Claude models, capitalisation in prompts works well. With OpenAI models, it tends to push things too far. These are the kinds of differences you only learn by doing.

There’s also the “bitter lesson” thinking from the Bay Area – the idea that you want to benefit from future improvements to models, and the best way to do that is to describe the outcome you want rather than giving very detailed instructions about the process. That way, when the next model is smarter, it naturally does a better job. If you’ve over-specified the process, you’re locked into an approach that might not transfer.

---

On security, PII, and prompt injection

Audience question: What level of thought are you giving to things like sensitive data, PII reduction, prompt injection, tool poisoning?

Dave: Prompt injection and tool poisoning only matter if people are talking directly to the LLM. In our product, nobody does. It’s all behind the scenes, prompted in fixed ways, with data we’ve verified. Indirect prompt injection – where data in the context becomes the attack vector – is real, but the base models are getting better at resisting it. The fundamental challenge is that the control channel and the data channel in LLMs are mixed. There’s no technical way to separate them fully. A lot of money has gone into solving this. If we haven’t found the answer by now, it might not be possible to solve it completely.

Melissa: It comes down to the harness. What are you actually passing into the LLM? What’s the minimum viable payload? We won’t pass sensitive client information – it gets abstracted out. Some agents don’t get permissions to send emails; that happens at the end of a verified process. Security has to be first in how you think about architecture: what’s the worst case scenario, and what do you need to control for?

---

Closing: What Should We Do?

Liz: What’s your advice to the product people in this room?

Dave: Because it’s an art, you have to understand the grain of the material. You have to know how things feel and what works. The only way to get that is experiential. There are no courses that can really teach this yet. You have to be doing it – at work or outside of work – as frequently as you can.

Melissa: I’m actually hopeful. There are parts of the product function that we won’t have to do anymore – the product ops, the ticket-writing, the traditional product owner work – and that’s a relief. So many PMs I speak to are stuck in the weeds and only get 30 minutes a week to do actual product strategy. That excuse is disappearing. But if everyone has access to the same tools and can spin up a greenfield product easily, how do we differentiate? That question doesn’t have an answer yet, and I find that genuinely exciting.

Daragh: I was someone who went into the terminal only when I was doing something I shouldn’t be doing. That was four months ago. Now I have two Mac Minis and I’ve set up an MCP server for my rooftop bar so the team can check bookings, schedules, cancellations. Some of these words weren’t in my vocabulary. The only difference is I just decided to give it a go. The distances between “I don’t know how to do this” and “I’m doing it” are a lot smaller than they look. Find something you’re personally passionate about solving – that’ll carry you through when it gets hard. Ask for help from your colleagues. And do it together – because as an empowered product team, you will go faster together.

The Thread Running Through All of It

If there was one thing the panel kept returning to, it wasn’t a tool or a framework or a model. It was the human stuff.

Who owns the shared context? Who decides which roles can stretch into which? Who gives product, design and engineers the space and time to actually learn new ways of working instead of just mandating AI-first and expecting magic? How do you preserve the productive friction between different mindsets when those mindsets are increasingly blending into one person?

None of these questions are new. They’ve shown up at every technology transition. But AI is accelerating the timeline and compressing the window to get ahead of them.

The honest answer from all three panelists was some version of: we don’t know yet, but we’re figuring it out by doing. There’s no shortcut to the experiential knowledge of working with these tools daily – what they’re good at, where they’ll go sideways, when to push and when to restart. The gap between “this seems impossible” and “I’ve set up an MCP server for my rooftop bar” turns out to be smaller than it looks, and shorter than you’d think.

So: find a problem you actually care about. Try something. Ask for help. And if you and a colleague are both stuck arguing with the same agent about the same thing – maybe the conversation you actually need to have is with each other.

Thanks to our host

InvestorHub is a Melbourne-based scale-up with the mission to make being listed a superpower. Our software allows listed companies in Australia and the UK to manage their end-to-end investor relationships, from their investor-facing website, a detailed investor CRM, analytics, and more. We’re hiring for Product Managers, Product Designers, Engineers and more right now!

We’re all trying to figure out this constantly changing AI thing. How can we use it day to day? How do we use it to make our products better, more useful, more whatever your current KPI/OKR is.

So we were very excited to have Caitlin Blackwell from SEEK to share some of their experiences with incorporating AI into not only production applications but also a very well established product.

Caitlin shared a couple of their experiments which have been running in production.

Experiment One:

Use generative AI via a chat to refine and improve job recommendations. They decided to use an LLM to retrieve a person’s preferences in order to refine the recommendations received.

What they learned: They had multiple objectives such as gathering information, adoption, satisfaction which made the measurement hard with no single primary objective. It was hard to measure the actual change on the recommendations relevance

Experiment Two:

Currently there’s a ‘You are a strong applicant’ badge which helps you understand your fit for the role based on your profile information and the job listing. This experiment was to make this feature more valuable and grow its usage. Using the sparkly ai icon, they rolled out an experiment which helped you understand the next steps and what skills you could grow.

Once again, the adoption was lower than expected – which left them wondering if the sparkly ai icon isn’t understood or do people just not want to use ai tools? They chose a model which was cheaper but they found it was blunt, not very conversational nor friendly. This led to spending more time tweaking the tone than had they gone with other models that were more expensive but nicer out of the box. They also fell into the trap of too many objectives again.The goal was to increase quality applications. This could be a great way to drive people to update their profile in order to have better matching but that was not the primary goal.

Overall, Caitlin recommends:

• Have a single objective for each experiment. Don’t try to do too much or get confused about your goals.
• Don’t go too big. Focus on small well known problems that you know have desirability. Or experiment cheaply to determine the desirability eg fake door tests.
• There is so much uncertain & unknowing right now, that you should be going fast with ideas.
• Use your experiments to determine the costs overall (human & AI) and think about how much you want to be spending on the experiment. Make sure you have buy in from leadership.
• Don’t get into analysis paralysis with the technology choices. Pick a model & go for it.
• Realise you might not have the adoption you had hoped for. Think about how you will maximise the experiment if you don’t have high adoption.

Thank you!

Thank you Caitlin for sharing some of the challenges your teams have been working through.

AND thank you to EasyGo for hosting the evening! Thank you for your hospitality & letting the community eye up the race car! 😉

Both Seek & EasyGo are currently hiring so go check them out!