Master Turkers

Amazon’s Mechanical Turk is an amazing service where one can create a simple task that can be micro-out-sourced to many people over the web, each of whom performs a small parcel of it. For instance, if you wanted 1000 people to highlight faces in photographs, think of synonyms for words, or provide from-the-hip feedback on your website, Mechanical Turk is ideal.

For reasons that are unclear to me, people seem to be willing to work for far below a minimum wage performing pretty dull tasks. As an experimental psychologist, I’m torn between feelings of data lust at the number of participants I could thus thriftily recruit, and concern about the quality of their data. What kind of person is willing to engage in dull tasks that must feel meaningless from a worm’s eye view? Where’s the incentive to do a good job?

It seems to me that there might be a market for tasks that require more effort, skill or thought, for which one would like to be able to cherry pick the participants. For this to work, you’d need a rich reputation scheme to Mechanical Turk, to pick out the Master Turks.

I’m picturing myself in holidays as an undergraduate. If someone was willing to pay more £10/hour (roughly what I was earning as a medical secretary), I (or my more talented peers) would have happily:

  • researched historical facts for a novel
  • proofread a doctoral thesis
  • helped with market research for a business plan
  • written a catchy jingle
  • filmed a youtube video using your product
  • written a program to generate verbal reasoning or arithmetic questions for an exam
  • provided summaries of white papers

You could imagine non-fixed-rate payment schemes, e.g.

  •   a competition where the best submissions divide the spoils
  •   an auction, so that more enjoyable tasks would be bid lower

And, deliciously, you could create a meta peer-review system where other Master Turkers’ task is to rate the submissions you’ve received.

Stack Overflow is going to transform the programming job market by making answering people’s questions satisfying, and then providing a metric of someone’s expertise that will help them land a job.

There have been many precedents of this kind of idea, but it seems strange that none of them have taken off. This feels like a way to demonstrate one’s abilities on potentially interesting tasks that would provide a portfolio of work to supplement a job application.