An introduction to crowdsourcing for language and multimedia technology research

Language and multimedia technology research often relies on large manually constructed datasets for training or evaluation of algorithms and systems. Constructing these datasets is often expensive with significant challenges in terms of recruitment of personnel to carry out the work. Crowdsourcing methods using scalable pools of workers available on-demand offers a flexible means of rapid low-cost construction of many of these datasets to support existing research requirements and potentially promote new research initiatives that would otherwise not be possible

The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also estimate that the time taken for half of the workers to leave the MTurk pool and be replaced is about 7 months. Each laboratory has its own population pool which overlaps, often extensively, with the hundreds of other laboratories using MTurk. Our estimate is based on a sample of 114,460 completed sessions from 33,408 unique participants and 689 sessions across seven laboratories in the US, Europe, and Australia from January 2012 to March 2015

Female economic dependence and the morality of promiscuity

This article is made available through the Brunel Open Access Publishing Fund. Copyright @ The Author(s) 2014.In environments in which female economic dependence on a male mate is higher, male parental investment is more essential. In such environments, therefore, both sexes should value paternity certainty more and thus object more to promiscuity (because promiscuity undermines paternity certainty). We tested this theory of anti-promiscuity morality in two studies (N = 656 and N = 4,626) using U.S. samples. In both, we examined whether opposition to promiscuity was higher among people who perceived greater female economic dependence in their social network. In Study 2, we also tested whether economic indicators of female economic dependence (e.g., female income, welfare availability) predicted anti-promiscuity morality at the state level. Results from both studies supported the proposed theory. At the individual level, perceived female economic dependence explained significant variance in anti-promiscuity morality, even after controlling for variance explained by age, sex, religiosity, political conservatism, and the anti-promiscuity views of geographical neighbors. At the state level, median female income was strongly negatively related to anti-promiscuity morality and this relationship was fully mediated by perceived female economic dependence. These results were consistent with the view that anti-promiscuity beliefs may function to promote paternity certainty in circumstances where male parental investment is particularly important

The martyrdom effect : when pain and effort increase prosocial contributions

Most theories of motivation and behavior (and lay intuitions alike) consider pain and effort to be deterrents. In contrast to this widely held view, we provide evidence that the prospect of enduring pain and exerting effort for a prosocial cause can promote contributions to the cause. Specifically, we show that willingness to contribute to a charitable or collective cause increases when the contribution process is expected to be painful and effortful rather than easy and enjoyable. Across five experiments, we document this “martyrdom effect,” show that the observed patterns defy standard economic and psychological accounts, and identify a mediator and moderator of the effect. Experiment 1 showed that people are willing to donate more to charity when they anticipate having to suffer to raise money. Experiment 2 extended these findings to a non-charity laboratory context that involved real money and actual pain. Experiment 3 demonstrated that the martyrdom effect is not the result of an attribute substitution strategy (whereby people use the amount of pain and effort involved in fundraising to determine donation worthiness). Experiment 4 showed that perceptions of meaningfulness partially mediate the martyrdom effect. Finally, Experiment 5 demonstrated that the nature of the prosocial cause moderates the martyrdom effect: the effect is strongest for causes associated with human suffering. We propose that anticipated pain and effort lead people to ascribe greater meaning to their contributions and to the experience of contributing, thereby motivating higher prosocial contributions. We conclude by considering some implications of this puzzling phenomenon. Copyright © 2011 John Wiley & Sons, Ltd

Bias Blind Spot: Structure, Measurement, and Consequences

People exhibit a bias blind spot: they are less likely to detect bias in themselves than in others. We report the development and validation of an instrument to measure individual differences in the propensity to exhibit the bias blind spot that is unidimensional, internally consistent, has high test-retest reliability, and is discriminated from measures of intelligence, decision making ability, and personality traits related to self-esteem, self-enhancement, and self-presentation. The scale is predictive of the extent to which people judge their abilities to be better-than-average for easy tasks and worse-than-average for difficult tasks, ignore the advice of others, and are responsive to an intervention designed to mitigate a different judgmental bias. These results suggest that the bias blind spot is a distinct metabias resulting from naïve realism rather than other forms of egocentric cognition, and has unique effects on judgment and behavior

Challenges in Complex Systems Science

FuturICT foundations are social science, complex systems science, and ICT. The main concerns and challenges in the science of complex systems in the context of FuturICT are laid out in this paper with special emphasis on the Complex Systems route to Social Sciences. This include complex systems having: many heterogeneous interacting parts; multiple scales; complicated transition laws; unexpected or unpredicted emergence; sensitive dependence on initial conditions; path-dependent dynamics; networked hierarchical connectivities; interaction of autonomous agents; self-organisation; non-equilibrium dynamics; combinatorial explosion; adaptivity to changing environments; co-evolving subsystems; ill-defined boundaries; and multilevel dynamics. In this context, science is seen as the process of abstracting the dynamics of systems from data. This presents many challenges including: data gathering by large-scale experiment, participatory sensing and social computation, managing huge distributed dynamic and heterogeneous databases; moving from data to dynamical models, going beyond correlations to cause-effect relationships, understanding the relationship between simple and comprehensive models with appropriate choices of variables, ensemble modeling and data assimilation, modeling systems of systems of systems with many levels between micro and macro; and formulating new approaches to prediction, forecasting, and risk, especially in systems that can reflect on and change their behaviour in response to predictions, and systems whose apparently predictable behaviour is disrupted by apparently unpredictable rare or extreme events. These challenges are part of the FuturICT agenda

Cooperation and Contagion in Web-Based, Networked Public Goods Experiments

A longstanding idea in the literature on human cooperation is that cooperation should be reinforced when conditional cooperators are more likely to interact. In the context of social networks, this idea implies that cooperation should fare better in highly clustered networks such as cliques than in networks with low clustering such as random networks. To test this hypothesis, we conducted a series of web-based experiments, in which 24 individuals played a local public goods game arranged on one of five network topologies that varied between disconnected cliques and a random regular graph. In contrast with previous theoretical work, we found that network topology had no significant effect on average contributions. This result implies either that individuals are not conditional cooperators, or else that cooperation does not benefit from positive reinforcement between connected neighbors. We then tested both of these possibilities in two subsequent series of experiments in which artificial seed players were introduced, making either full or zero contributions. First, we found that although players did generally behave like conditional cooperators, they were as likely to decrease their contributions in response to low contributing neighbors as they were to increase their contributions in response to high contributing neighbors. Second, we found that positive effects of cooperation were contagious only to direct neighbors in the network. In total we report on 113 human subjects experiments, highlighting the speed, flexibility, and cost-effectiveness of web-based experiments over those conducted in physical labs

Attitudes toward hydraulic fracturing: The opposing forces of political conservatism and basic knowledge about fracking

publisher: Elsevier articletitle: Attitudes toward hydraulic fracturing: The opposing forces of political conservatism and basic knowledge about fracking journaltitle: Global Environmental Change articlelink: http://dx.doi.org/10.1016/j.gloenvcha.2016.03.004 content_type: article copyright: © 2016 Elsevier Ltd. All rights reserved

Mortar-based systems for externally bonded strengthening of masonry

Mortar-based composite materials appear particularly promising for use as externally bonded reinforcement (EBR) systems for masonry structures. Nevertheless, their mechanical performance, which may significantly differ from that of Fibre Reinforced Polymers, is still far from being fully investigated. Furthermore, standardized and reliable testing procedures have not been defined yet. The present paper provides an insight on experimental-related issues arising from campaigns on mortar-based EBRs carried out by laboratories in Italy, Portugal and Spain. The performance of three reinforcement systems made out of steel, carbon and basalt textiles embedded in inorganic matrices has been investigated by means of uniaxial tensile coupon testing and bond tests on brick and stone substrates. The experimental results contribute to the existing knowledge regarding the structural behaviour of mortar-based EBRs against tension and shear bond stress, and to the development of reliable test procedures aiming at their homogenization/standardization

How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a number of workers to work on a set of similar tasks. After completing the tasks, each worker reports back outputs. The requester then aggregates the reported outputs to obtain aggregate outputs. A crucial question that arises during this process is: how many crowd workers should a requester hire? In this paper, we investigate from an empirical perspective the optimal number of workers a requester should hire when crowdsourcing tasks, with a particular focus on the crowdsourcing platform Amazon Mechanical Turk. Specifically, we report the results of three studies involving different tasks and payment schemes. We find that both the expected error in the aggregate outputs as well as the risk of a poor combination of workers decrease as the number of workers increases. Surprisingly, we find that the optimal number of workers a requester should hire for each task is around 10 to 11, no matter the underlying task and payment scheme. To derive such a result, we employ a principled analysis based on bootstrapping and segmented linear regression. Besides the above result, we also find that overall top-performing workers are more consistent across multiple tasks than other workers. Our results thus contribute to a better understanding of, and provide new insights into, how to design more effective crowdsourcing processes