Indeed’s relevance problem
Indeed is the #1 job site in the world, reaching 300 million people every month. Shortly after I joined the company, our VP of UX asked my team to develop a strategy for making progress on an embarrassing problem: we kept showing people terrible job matches.
Actual job seeker quote
“Indeed sends me at least 5 jobs a day and not one of them is relevant to my career path, my profile, or anything in my CV. I do feedback on all the job alerts and I still keep getting the same irrelevant crap.”
Indeed had always been a search platform: people searched, we showed them jobs. But the company was investing more and more in recommendation products: emails, the homepage, and other surfaces where Indeed chose the jobs people would see.
The problem was that we were still pretty bad at that.
Actual job seeker quote
"A puppeteer? Where is that in my resume? You think I would be a good fit to put my hand in a sock and make voices?"
And as our number of recommendations grew, so did our complaints.
The problem at scale
Indeed tracked positive outcomes obsessively: hires that came from search-and-apply or from a recommendation we provided. Our recommendation products were becoming a greater part of our success story over time:
But alongside this success came some alarming signals:
People weren’t getting interviews for the jobs we were recommending, were reporting these jobs as bad matches, or worst of all, getting autorejected by employers after we just told them they’d be a great fit for a role. Leadership's initial read: we weren't respecting job seeker’s real job preferences, and that was a major contributor to these negative outcomes. They asked my team to understand preferences deeply, chart how they played into search and recommendations, and propose a concrete plan forward.
Our design goal: understand why our matching strategy was missing the mark in a recommendation-led world. Build a long-term preference strategy, and partner with product teams to ship tactical wins along the way.
Team & Stakeholders
A different kind of team
When this problem was handed to me, my team was about a month old. Indeed had recruited me to start the Job Seeker Accelerator, a startup-style strategy team designed to:
My team would later tackle many different gnarly Indeed problems, from SERP flooding and location data strategy to core journey work, but none of that had happened yet. At the time of this project, we were new and unproven, with a lot of ground to cover.
Core team
Me (UX Director)
Lead Product Designer
Sr. UX Researcher
Sr. UX Content Designer
I work most often as a player-coach. I’m allergic to hard lines and seniority-based hierarchy. On my teams, people step up to do whatever the work needs, regardless of title. On this project, I was the strategic driver and director, but I'd call the rest of them co-leaders in every way that mattered.
A complex stakeholder landscape
My small team needed to coordinate and partner with many people across R&D, plus the embedded designers and researchers on every one of those teams. The right collaboration strategy was going to be crucial. Because this was a leadership-sponsored project, the initial perception from product teams was always going to be "who are these people, and what are they doing in my space?" Our job was to listen and partner so closely that we became, temporarily, part of each team.
Project sponsors
VP Product
VP UX (my boss)
Everyone was remote, and important stakeholders sat in different timezones.
Collaboration Strategy
A leadership-sponsored project landing inside other teams' product areas meant the perception of my team mattered as much as the substance of our work. We set up four practices that ran throughout the project.
A workshop-led kickoff. My team brought everyone we needed to influence into a single 60-minute working session: our VPs of Product and UX, the individual product PMs, and the embedded designers and researchers from every team. We aligned on the problem, captured every leadership assumption out loud, and named the success metrics together. This bought us buy-in and a sense that we were building this together.
Async documentation as the default. I maintained a living brief covering scope, milestones, RACI, and open challenges, and emailed weekly updates linking to it. This let our distributed working group, and anyone else interested, see what was happening without scheduling a meeting. Indeed's heavy doc-comment culture turned this into a constant async dialog that resolved misalignments before they started.
Primary collaborators
Director of Product - Profile
VP Eng - Profile
Sr. Product Manager - Search
Sr. Product Manager - Homepage
Product Manager - Email recommendations
A council with embedded UX Researchers & Designers. We set up a biweekly working group with the embedded researchers and designers from each involved product team. They had the deepest read on their products and would help us ship the tactical work. Bringing them in early and soliciting feedback at every milestone bought us champions on each team.
A monthly product meeting. I convened our project sponsors and all embedded PMs each month to share progress. I led the meeting but tagged in the other PMs for their updates. This created a "one team" atmosphere, with everyone presenting to leadership as a united front.
Early Insights
Getting up to speed, fast
We had to map the problem before we could chart a path forward. But a long, siloed discovery phase wasn't going to work. We planned to launch a core tactic with one of the product teams almost immediately, so we could earn trust while the deeper learning continued in parallel. That meant getting up to speed fast.
We ran three research studies in parallel in the first 4 weeks, alongside our project scoping and larger planning processes. The first job was to understand the shape of the problem, which meant more than understanding the user experience. We had to also deeply understand the historical internal context that led to things being how they are:
10
30-min interviews with project stakeholders and subject matter experts
20+
30-min interviews with project stakeholders and subject matter experts
There were two core insights that shaped our early plans:
Insight 1: We had prioritized gathering more data rather than better data
Over time, Indeed product teams had gone their own ways. Each was building for a different part of the experience, in a silo, gathering data on its own terms without talking to the others.
7
30-min interviews with project stakeholders and subject matter experts
Because preferences were captured in different ways across our experience, with different language and interactions, we couldn't actually use the data to improve the jobs we served. When we talked with the teams, every one of them said the same thing about their approach: they wanted to collect more now, and untangle it later. The cost of that approach was a really awful experience for job seekers in the meantime.
Insight 2: We asked users for data we didn’t use to improve their experience
Even where we were collecting preferences, a tangled web of behavioral signals and resume inferences (combined with not respecting what job seekers told us directly) produced really bad experiences.
A real job seeker example: Erin*
*PII has been changed to protect her identity.
Erin was a real Indeed user. We encountered her feedback during our meta-analysis, then worked with data to piece together her story.
Erin worked from home running her own online fiber arts store. She had a disability that limited her ability to drive, and she didn’t have a driver's license.
Resume data from Erin
Erin’s Fiber Arts
Owner operator
June 2021 - Current
I own and operate this ecommerce shop where I make hand stitched patches and wall hangings, sell and ship all on my own.
To form recommendations, our Homepage and Search teams extracted data from users' uploaded resumes. In Erin's case, we inferred "owner operator" to mean “CDL truck driver, owner operator” and started recommending driving jobs.
Normalized job title
Owner operator driver
Indeed extracted title
At the time, we didn't show users this inferred job-type preference (don’t worry, we fixed this as part of this effort). So Erin didn't know we were using it to shape her experience. All she knew was that we kept recommending jobs she didn't want.
Erin kept telling us she didn't want these jobs. We just weren't listening. She disliked more than 25 CDL A truck driver jobs and saw no change to her experience. Every time she clicked one of these jobs (often to dislike it), we inferred that click as an "interested" signal. Our inferences outweighed what she was telling us directly.
A complaint Erin submitted
“Here is a glaring example of Indeed ignoring the information clearly stated in my profile. I can not physically drive. I do not have a license and cannot get one due to my disabilities yet when I scroll through my recommended jobs it’s full of driving jobs.”
Even though she’d disliked these jobs repeatedly, and reported some, she continued to get CDL A jobs, leaving her frustrated and unaware of how to move forward on Indeed.
Plan
Two tracks of work, running in tandem
After our initial planning phase/discovery sprint, we needed to move fast. Leadership wanted progress, and the longer we spent learning in a silo, the less credibility we'd carry into the work that mattered.
So we ran two tracks at the same time: a tactical phase to nail the experience on one product, and a strategic phase to keep the deeper learning going alongside it. The tactical phase was designed to feed learnings into the strategic one.
Phase 1: Invite to Apply (I2A) Email MVP
Goal: Solve preferences gathering and usage end-to-end on a single recommendation product, so we could ship fast, learn fast, and earn the credibility to do bigger work. We picked I2A because it was one of our most complained-about products.
Phase 2: Global preferences strategy
Goal: Apply what we learned from the MVP, plus deeper discovery, into a scalable, consistent, Indeed-wide way for job seekers to tell us their preferences and manage them later.
MVP
Phase 1: MVP Scope & Plan
We wanted to scope this as small as we could. End-to-end preferences for one product surface: the Invite to Apply (I2A) email, the associated feedback form, and a centralized place for users to manage saved preferences.
I led this phase for two reasons. The strategy was only going to be as credible as our first tangible result. A long-term framework with no shipping product behind it would die on the vine the moment our team rotated off. I also recognized that this would buy the team time to do longer-term strategic research alongside this tactic, and that the learnings from both were what we needed to create a successful long-term strategy.
Timeframe
8 weeks from kickoff to shipped MVP, including discovery, design sprint, user testing, and rollout.
Success Criteria
↑
Increased preference data point collection on I2A emails (a leading indicator that users were willing and able to engage with our new form)
↓
Reduction in I2A-related complaints
↔
No regression in email open or apply rates
🧠
Qualitative signal from user testing that the experience matched how job seekers actually thought about preferences
We agreed in advance that any one of these failures would hold the launch: flat or dropping data point collection, an open or apply rate drop greater than 2%, any increase in complaint volume, or testers who couldn't articulate what their preferences were doing to their experience.
From Discovery to Final MVP Designs
In 4 steps, we reached our final design:
1. Design Sprint
We prepared user stories and problems to solve based on the discovery results. These formed the basis for ideation in our design sprint which involved members of every team involved in the greater project.
We turned the top ideas from this session into annotated wireframes, landing on two potential directions to carry into testing.
2. User Interviews
Our protocol was designed to surface mental models first, before our prototypes colored the conversation.
Study protocol
Intro + job search overview (5 min)
Build rapport, set expectations, and understand job search journey
Contextual inquiry (15 min)
Understand how participants search for a job, define their search criteria, and tell Indeed what is a good fit and not a good fit
Prototype review (30 min)
Conduct a “think aloud” exercise as participants walk through 2 dislike prototypes, asking them to share their reactions, questions, challenges and roadblocks.
Reflection (10 min)
Ask participants to reflect on the ideas they’ve seen and what aspects they found useful or not
Two findings shaped everything that came after:
Job seekers can name what they don't want far more easily than what they do want. Saying "I don't want to drive more than 30 minutes" felt easier than "I want a hybrid role at a values-aligned company."
The natural moment to capture that signal is in response to a specific job, not on a static preferences page. We found showing someone a job they'd never take and asking why is a much richer interaction than abstract questions in a settings menu.
A design strategy pivot to negative preferences
Leadership had asked us to gather better positive preferences (what job seekers do want). The data was telling us to build for the negative ones first.
We pivoted the prototypes to focus on dislikes. Because we'd engaged leadership from the beginning and shared every discovery as it came, the pivot landed without pushback.
3. Unmoderated testing
We used our Round 2 learnings to develop two very different prototypes for how we could gather dislikes. We tested these against each other in unmoderated think aloud tests with job seekers.
There were pieces from each of the prototypes that we took into our final design.
Releasing the MVP
Our final flow let users:
See exactly what aspect of the job they were disliking (the specific company, location, pay, schedule)
Use clear, neutral language ("Not interested" button replacing the confusion of checking something you don’t want, a pain point revealed in our earlier research)
Apply granular feedback on each detail rather than a single thumbs-down on the whole job
Manage every dislike they'd ever given in a single place
4. RITE testing
After settling on a final design, we then ran RITE testing (Rapid Iterative Testing and Evaluation), a UX methodology that focuses on identifying and fixing usability issues immediately after they are observed, rather than waiting for the end of a study. We conducted 10, thirty-minute moderated sessions with active job seekers, and as we went, we worked with our developer to iterate language, interaction patterns, and placement between sessions. By the end, we had a flow that felt simple and scalable.
An A/B Testing Learning
When we released our new version of the form to the wild at 5% of our job seeker base, we saw a staggering drop in one of our core metrics: number of preference data points collected. So we jumped into FullStory, our session replay platform, to watch Job Seekers in the test group use the experience live.
What we found was that one of our trust and security teams had released a terms and conditions pop-up (unrelated to this project) that covered the feedback buttons. 🤦🏻♀️
We worked with that team to make a revision so that the form didn’t obstruct the preference page, re-tested, and saw the results we wanted.
The learning: when the numbers feel counterintuitive, always see if you can directly observe the behavior driving them.
MVP results
The MVP shipped on schedule. Across the I2A email and the connected feedback form:
18% increase in dislike collection, meeting our goal of increasing preference data point collection.Users were giving us more signal we could use.
6% increase in apply/sends for users with dislikes on file. Job seekers interacting with our new experience were seeing more jobs they wanted to apply to (original goal was to do no harm).
Neutral open rates with fewer customer service complaints about the email (meeting our goal to do no harm).
9% reduction in email opt-outs. We saw fewer users disengaging from the email experience entirely, an unexpected boost.
4% reduction in I2A related complaints, meeting our goal of reducing complaints.
Learnings from our MVP Release & Continued Discovery
While Phase 1 shipped, we ran deeper discovery in parallel to close a risk I'd flagged early: our backend assumed preferences were stable, and nobody had tested that. Our researcher ran a 10-day diary study with 24 active job seekers, logging every search, the criteria they applied, and what made them adjust. We combined those findings with our rapid Phase 1 studies and live MVP usage.
What validated the I2A direction
Job seekers valued predictability, seeing the same feedback form everywhere they interacted with a job.
Dislikes beat positive preferences as a matching input. More durable, easier to collect, more directly tied to satisfaction.
Users wanted one home to review and edit every preference they'd given us.
What challenged our strategy
Preferences shift mid-search, not just over weeks. Users ran a search, saw the results, and adjusted on the spot. Sometimes major (dropping remote-only after seeing the local market), sometimes minor (ten more minutes of commute). It happened constantly.
Job seekers search in parallel. 49% were looking for two or more job types at once, applying across 3 top-level and 6.5 specific occupations in a single search. Preferences that fit one job type were wrong for another.
Almost nobody updates their official preferences. Under 1% did in any given week, even when their search behavior showed obvious changes.
The shift in thinking
Indeed had built search and storage around explicit preferences users almost never updated. Teams imagined a clean, stable list tied to one job target. The reality was a fluid set of must-haves, nice-to-haves, and tradeoffs users recalculated constantly.
A system built around static collection was going to fight users. A system that captured what they said in the moment, and let them adjust without a trip back to a preferences page, was going to win.
Our (updated) design goal: build one preference framework that works across every product, and help our data teams see why the backend needs to flex.
Phase 2
Extending the System: Global Preferences
Our goal for phase 2 was to take our MVP learnings and develop a global, scalable, end-to-end preferences experience and a centralized place for users to manage saved preferences. Make it flexible enough to support the world of today (static preference sets) and the future we wanted (flexible preferences).
Timeframe
6 months to close out this project (initially scoped at 4 months)
Success Criteria
↑
10% Increase in preference collection across products
↓
5% reduction in relevance-related customer service complaints
Beyond leadership’s success criteria, my team had another goal: change minds across our backend teams about the flexibility of preferences. Motivate teams to build for a flexible future.
The Final Deliverables
1. Global feedback flow
We turned the dislike collection flow we'd validated on I2A into the standard pattern across Indeed's job surfaces. We worked with designers embedded on all teams to translate the framework to their product, and released the flow globally.
2. Unified preferences page & entry points
We built out a single home where users could review every preference they'd given Indeed, edit them, and see how they were being applied. We replaced the patchwork of surface-specific settings we kept in their profile, and we respected this list across all products, no longer showing jobs with aspects seekers disliked.
To build trust, we showed job seekers how their preferences were working. We called out when a job was removed because of a dislike, highlighted matched preferences on job pages, and gave in-context controls to adjust or undo. Preferences were reflected back, acted on, and easy to change.
3. Shared backend
Any preference saved anywhere on Indeed was stored on the user's profile and accessible to every surface. Surfaces stopped maintaining their own schemas and started writing to a shared one.
4. Integrated dislike framework for our design system
Our designer summarized everything we'd learned about how job seekers think about feedback form structures and what testing had taught us about the right way to collect data from job seekers. The framework was specific enough that surface teams could implement net-new job feedback experiences directly from it without rerunning research, and we integrated its guidelines back into our design system documentation.
5. Content model
Our content designer built a shared taxonomy and language for every preference type (job title, pay, location, schedule, industry, and others). Each preference got a canonical term, terms to avoid, the question we'd ask the user, and the confirmation phrase we'd use. The model removed the language inconsistency that had been one of the original root causes of bad data.
Final results
Impact for Job Seekers
18% increase in dislike collection, exceeding our goal of 10%
8% reduction in relevance-related customer service tickets, exceeding our 5% reduction goal
40% increase in users who said they'd recommend Indeed after using preferences, an unexpected boost
For the business: Matching teams gained the data coverage to include dislikes in their matching algorithms. Surface teams using the unified system cut design time and increased consistency. The shared backend gave us a single source of truth for preference data instead of separate ones to maintain.
What didn't get solved (right away)
We were honest with leadership that we hadn't solved everything. We'd consolidated preferences and given the business reliable matching data, but the bigger shifts the project surfaced were conceptual ones, and conceptual change moves slowly.
The diary study we ran surfaced real pain points around how dynamic preferences were and how poorly Indeed's underlying models handled them.
Our final deliverable included a forward-looking vision document with five future directions: refining preferences in the moment, updating job card design to reflect filters, saved searches, onboarding archetypes based on behavior, and explicit prioritization and tradeoff tools.
Over the next few years, I watched every one of these directions land on teams' roadmaps.
Biggest Lessons Learned
Plan for the company you're in, not the company you wish it could be. I had just come to Indeed from a much, much smaller company with a much more collaborative product culture. I thought our discoveries would change minds quickly. Instead, I found entrenched thinking developed over years, and a reasonable reluctance to rearchitect a data model that other systems depended on. Project wrap-up and socialization took two months longer than I'd planned. Worse, we became "the preference team" in people's heads, and it was hard to extricate ourselves. On future projects, I built a longer adoption tail and clear exit strategy into the plan from the start.
Push the strategic discovery earlier. The diary study was the single most valuable piece of research we did, and its findings reframed Indeed's data collection strategy. Running it in parallel with the MVP worked, but some MVP design decisions would have been different if we'd known what the diary study was going to show. I learned to push the highest-leverage discovery as early as it can run, even when there's pressure to start shipping.