Optimal Crowdsourcing Scheme for Open Questions on Web

Abstract

Crowdsourcing has become immensely popular with hoards of requesters on web to solve a wide range of data acquisition problems with hundreds of thousands of workers regardless of their locations. The majority of crowdsourcing markets are occupied by open questions with unstructured answer formats and no possible answers are suggested in advance. A key challenge of crowdsourcing markets is to ensure the quality of the answers from the workers with different abilities and reliabilities. The collected answers are often consolidated either using a simple majority voting strategy or more sophisticated solutions. In this work, we model the group behavior in the crowdsourcing service based on entropy and Chinese Restaurant Process. Then we design an EM scheme to dynamically recommend suitable answers to open questions with infinite answer spaces by using logistic regression model. Comparative study conducted using live experiments demonstrates the superiority of our crowdsourcing scheme for open questions.