1Cademy - Auditable-Artifact Reporting Standards: Data Statements, Datasheets, Model Cards, Reproducibility Checklists

Learn Before

Research

Concept

Auditable-Artifact Reporting Standards: Data Statements, Datasheets, Model Cards, Reproducibility Checklists

Auditable-artifact reporting standards are a tradition in machine learning, NLP, and adjacent fields that attaches standardized disclosure documents to the artifacts a research community produces — datasets, trained models, and paper submissions — so that downstream users can independently audit how each artifact was made, what it contains, and what claims it can support. Four founding works define the tradition: (i) data statements (Bender and Friedman, TACL 2018), structured documents attached to NLP datasets that disclose curation rationale, language variety, and speaker and annotator demographics; (ii) datasheets for datasets (Gebru et al., CACM 2021), a 57-question form covering motivation, composition, collection, preprocessing, uses, distribution, and maintenance of any dataset; (iii) model cards for model reporting (Mitchell et al., FAT* 2019), short documents accompanying trained models that report intended use, factors, evaluation and training data, subgroup-disaggregated metrics, and ethical considerations; and (iv) the NeurIPS 2019 reproducibility program and ML Reproducibility Checklist (Pineau et al., JMLR 2021), which attaches an explicit reproducibility checklist to conference submissions and pairs it with a code-submission policy. The shared discipline across all four is that each artifact ships with the documentation a third party needs to audit it, rather than relying on prose buried inside the paper. This tradition is methodologically distinct from the IR-evaluation reproducibility line (reference implementations, statistical controls, split-design hygiene): it is about per-artifact disclosure rather than benchmark-level evaluation variance.

Updated 2026-05-18

Contributors are:

Who are from:

References

Reference: aclanthology.org q18-1041

Learn Before

Related

Learn After