The crowd workers outperformed the graduate students who had more expertise than them. The ratio of “yes” answers (results were valid) to “no” answers (results were invalid) regarding the task questions was compared between the two experimental groups, and it was found that the graduate students were more inclined to answer “no” than the crowd workers. A suggested explanation for this was the fact that graduate students had a preconceived idea of what types of results were valid, while the crowd workers’ responses more closely matched the tutorial from their training.