<story><title>Amazon’s Mechanical Turk Has Reinvented Research</title><url>https://daily.jstor.org/amazons-mechanical-turk-has-reinvented-research/</url></story><parent_chain></parent_chain><comment><author>Radim</author><text>Mechanical Turk is great for &amp;quot;open&amp;quot;, public research. We used to use them a lot for machine learning tasks (data cleanup, model comparisons, label annotations), along with similar services like CrowdFlower &amp;#x2F; Figure Eight. We saw two primarily issues when applied to &amp;quot;non-open&amp;quot; (commercial) projects:&lt;p&gt;- business-related data too sensitive to share with strangers (contractual obligations, too much risk)&lt;p&gt;- some tasks required non-trivial subject matter expertise and context to annotate properly (quality control issues)&lt;p&gt;For this reason, we gradually moved to an in-house team of long-term annotators. It&amp;#x27;s not much more expensive (moms on maternity leave, students…), but infinitely more flexible and safer for our purposes. YMMV.</text></comment>
<story><title>Amazon’s Mechanical Turk Has Reinvented Research</title><url>https://daily.jstor.org/amazons-mechanical-turk-has-reinvented-research/</url></story><parent_chain></parent_chain><comment><author>chx</author><text>Data cleanup is also a great use. For example, we needed to parse an amount of address data, we just needed country, state (or equivalent), city, we gave out each address three times, whatever results were the same at least twice was accepted. We had over 92% where all three were the same, another 7% with 2-1 (required review), less than 1% needed either manual cleanup before re-Turking or just manually entering some of the more gnarly cases. We considered it a truly massive success, price efficient and absolutely unbelievable quick.</text></comment>