Università degli Studi di Udine OpenUniud - Archivio istituzionale delle tesi di dottorato

OpenUniud - Archivio istituzionale delle tesi di dottorato >
Udine Thesis Repository >
01 - Tesi di dottorato >

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/10990/815

Autori: Maddalena, Eddy
Supervisore afferente all'Università: MIZZARO, STEFANO
Titolo: Crowdsourcing Relevance: Two Studies on Assessment
Abstract (in inglese): Crowdsourcing has become an alternative approach to collect relevance judgments at large scale. In this thesis, we focus on some specific aspects related to time, scale, and agreement. First, we address the issue of the time factor in gathering relevance label: we study how much time the judges need to assess documents. We conduct a series of four experiments which unexpectedly reveal us how introducing time limitations leads to benefits in terms of the quality of the results. Furthermore, we discuss strategies aimed to determine the right amount of time to make available to the workers for the relevance assessment, in order to both guarantee the high quality of the gathered results and the saving of the valuable resources of time and money. Then we explore the application of magnitude estimation, a psychophysical scaling technique for the measurement of sensation, for relevance assessment. We conduct a large-scale user study across 18 TREC topics, collecting more than 50,000 magnitude estimation judgments, which result to be overall rank-aligned with ordinal judgments made by expert relevance assessors. We discuss the benefits, the reliability of the judgements collected, and the competitiveness in terms of assessor cost. We also report some preliminary results on the agreement among judges. Often, the results of crowdsourcing experiments are affected by noise, that can be ascribed to lack of agreement among workers. This aspect should be considered as it can affect the reliability of the gathered relevance labels, as well as the overall repeatability of the experiments.
Parole chiave: Information Retrieval; Crowdsourcing
MIUR : Settore ING-INF/05 - Sistemi Di Elaborazione Delle Informazioni
Lingua: eng
Data: 3-apr-2017
Corso di dottorato: Dottorato di ricerca in Informatica e scienze matematiche e fisiche
Ciclo di dottorato: 29
Università di conseguimento titolo: Università degli Studi di Udine
Luogo di discussione: Udine
Citazione: Maddalena, E. Crowdsourcing Relevance: Two Studies on Assessment. (Doctoral Thesis, Università degli Studi di Udine, 2017).
In01 - Tesi di dottorato

Full text:

File Descrizione DimensioniFormatoConsultabilità
thesis.pdfTesi 8,16 MBAdobe PDFVisualizza/apri

Tutti i documenti archiviati in DSPACE sono protetti da copyright. Tutti i diritti riservati.

Segnala questo record su




Stumble it!



  ICT Support, development & maintenance are provided by CINECA. Powered on DSpace SoftwareFeedback CINECA