Powered by Drupal
You are here
Submitted by james.wiedner on Thu, 10/11/2012 - 15:55
Even in our always-increasingly automated world, there are still tasks related to information retrieval that require the sort of human cognition that we have so far been unable to replicate with even the most sophisticated of computers. Such tasks are known as human intelligence tasks (HITs) in the parlance of the field. With the sheer volume of electronic data increasing exponentially, relying solely upon trained editorial judges is quickly becoming both cost- and time-prohibitive. The alternative of crowdsourcing the task(s) of parsing through immense amounts of data has proven an effective alternative method in the field of IR.
While the use of crowd sourcing was once a “niche phenomenon,” it has become “an accepted solution to a wide range of data acquisition challenges” over the past 5 years. (Eickhoff et. al., 1). However, even though quality control measures such as built-in redundancy help to assure accuracy in HITs, the incredibly large scale and relative anonymity of individuals in the context of modern crowdsourcing has inherent quality control shortcomings. For this reason, an analysis of the otherwise anonymous crowdsourcers has become an area of interest both in business and academia.
A recent study analyzing the demographic data and personalty traits of a group of individuals engaged in HITs arrived at conclusions that were frequently unexpected, but methodologically sound. (Kaza et. al.). The study concluded that, for reasons that might be disputed, the evidence irrefutably revealed that demographics and personality traits could be accurately predictive of the quality of HITs produced. For example, due to factors that the researchers only speculate upon, American and European workers in a crowdsourcing project consistently outperformed their Asian counterparts. Another ostensibly counterintuitive finding was that a worker’s education level was not a significant factor in predicting the accuracy of an HIT worker. A worker’s reading habits, on the other hand, were a notable factor in predicting accuracy. The takeaway from the study was that the demographics and personalities of individual members were highly predictive of quality of overall results; even the largest of crowds is still a collection of individual minds. To paraphrase the authors, the faces inthe crowd make of the face of a crowd, and what that face looks like is of significant import.
On the other hand, there have been studies suggesting that factors commonly accepted as predictive of HIT quality such as the nationality of a crowd’s workers are not nearly as accurate in their predictions than an analysis of the primary motivating factor of a worker. At least one study has shown that there are two basic types of workers, delineated by their primary motivation: monetary gain, or entertainment (in the form of diversion, competition, etc.). The latter group was found to be (1) more accurate, and (2) performing at their best in a gaming environment. Given these facts, it only makes sense that gamesourcing be explored and employed.