We’re all trying to figure out this constantly changing AI thing. How can we use it day to day? How do we use it to make our products better, more useful, more whatever your current KPI/OKR is.

So we were very excited to have Caitlin Blackwell from SEEK to share some of their experiences with incorporating AI into not only production applications but also a very well established product.

Caitlin shared a couple of their experiments which have been running in production.

Experiment One:

Use generative AI via a chat to refine and improve job recommendations. They decided to use an LLM to retrieve a person’s preferences in order to refine the recommendations received.

What they learned: They had multiple objectives such as gathering information, adoption, satisfaction which made the measurement hard with no single primary objective. It was hard to measure the actual change on the recommendations relevance

Experiment Two:

Currently there’s a ‘You are a strong applicant’ badge which helps you understand your fit for the role based on your profile information and the job listing. This experiment was to make this feature more valuable and grow its usage. Using the sparkly ai icon, they rolled out an experiment which helped you understand the next steps and what skills you could grow.

Once again, the adoption was lower than expected – which left them wondering if the sparkly ai icon isn’t understood or do people just not want to use ai tools? They chose a model which was cheaper but they found it was blunt, not very conversational nor friendly. This led to spending more time tweaking the tone than had they gone with other models that were more expensive but nicer out of the box. They also fell into the trap of too many objectives again.The goal was to increase quality applications. This could be a great way to drive people to update their profile in order to have better matching but that was not the primary goal.

Overall, Caitlin recommends:

• Have a single objective for each experiment. Don’t try to do too much or get confused about your goals.
• Don’t go too big. Focus on small well known problems that you know have desirability. Or experiment cheaply to determine the desirability eg fake door tests.
• There is so much uncertain & unknowing right now, that you should be going fast with ideas.
• Use your experiments to determine the costs overall (human & AI) and think about how much you want to be spending on the experiment. Make sure you have buy in from leadership.
• Don’t get into analysis paralysis with the technology choices. Pick a model & go for it.
• Realise you might not have the adoption you had hoped for. Think about how you will maximise the experiment if you don’t have high adoption.

Thank you!

Thank you Caitlin for sharing some of the challenges your teams have been working through.

AND thank you to EasyGo for hosting the evening! Thank you for your hospitality & letting the community eye up the race car! 😉

Both Seek & EasyGo are currently hiring so go check them out!