The people teaching AI to use a computer.

Before the first click

Two people could read the same request, pursue different objectives, and each produce a convincing account of what they had done. If either demonstration relied on an unsupported assumption, the model would be learning from it too.

I knew the problem from my own work. I began in data annotation, ranking model responses and RLHF, then demonstrated computer tasks myself. My early demonstrations failed on errors the guidelines named without showing how to avoid them.

Quality reviewers could flag those errors. When someone kept making one, the reviewers sent them to me for coaching.

I taught people to work through the intended outcome and recognise what still needed clarification. Their actions and written rationales also had to follow the information available to them. Knowing a website well could make this harder: people would describe something from memory before it appeared on screen.

As Training & Process Lead, I owned the training and the process our teams worked to. Changes to the client’s guidelines came through me, and I briefed the team leads before they briefed their teams. I designed the full onboarding journey and built roughly fifty training videos, from the mechanics of the work to complete demonstrations and recurring errors. Human reviewers evaluated trainees’ work along the path to production certification.

A curriculum shaped by the work

Each week I ran a session on errors from production, followed by a workshop on the idea behind them.

Those sessions complemented the courses I developed on reasoning, language, evaluation and platform use.

The tools around the training

I designed a platform lifecycle board in Whimsical that guided onboarding across training cohorts, and an offboarding map that showed leadership task flow, escalation paths and exit decisions.

To see whether trainees were learning in the workshops, I built a tracker that matched meeting records against an end-of-session quiz. Those who stayed and answered also performed well across the project. Less engaged trainees heard from me privately: I asked what would make the sessions worth their time. Their work improved after those conversations.

Two weeks to production

The usual onboarding path relied on human evaluation. For one intake, more than a hundred people moving across from other projects needed to reach production within two weeks. I designed a shorter training path focused on the knowledge they needed for this work, and added a grader built on a large language model to help handle the volume. I wrote the rubric and specified the evaluation workflow; engineers implemented the grader.

Reviewers checked every submission, including those the grader had passed. The intake reached production readiness within the two weeks.

What the errors taught me

I wrote up the habits I taught and how I coached them. The question at their centre works just as well when an AI hands work back to a person.

Read the essay