statcheck

Credits

This is a python implementation of the R package statcheck (ver. 1.4.0-beta.4) published by Michèle B. Nuijten [MicheleNuijten]. The original package can by found at her Github page. The code relies heavily on Nuijten's work and is currently only a python implementation of the original package, with the goal of making it more accessible to the python community. The original package was published under the GNU General Public License v3.0. The curent implementation is published under the MIT License. To ensure usability, all the original tests were recoded to the python version.

What is statcheck?

statcheck is a free, open source Python package that can be used to automatically extract statistical null-hypothesis significant testing (NHST) results from articles and recompute the p-values based on the reported test statistic and degrees of freedom to detect possible inconsistencies.

statcheck is mainly useful for:

1. Self-checks: you can use statcheck to make sure your manuscript doesn’t contain copy-paste errors or other inconsistencies before you submit it to a journal.
2. Peer review: editors and reviewers can use statcheck to check submitted manuscripts for statistical inconsistencies. They can ask authors for a correction or clarification before publishing a manuscript.
3. Research: statcheck can be used to automatically extract statistical test results from articles that can then be analyzed. You can for instance investigate whether you can predict statistical inconsistencies (see e.g., Nuijten et al., 2017), or use it to analyze p-value distributions (see e.g., Hartgerink et al., 2016).

How does statcheck work?

The algorithm behind statcheck consists of four basic steps:

1. Convert pdf and html articles to plain text files.
2. Search the text for instances of NHST results. Specifically, statcheck can recognize t-tests, F-tests, correlations, z-tests, -tests, and Q-tests (from meta-analyses) if they are reported completely (test statistic, degrees of freedom, and p-value) and in APA style.
3. Recompute the p-value using the reported test statistic and degrees of freedom.
4. Compare the reported and recomputed p-value. If the reported p-value does not match the computed one, the result is marked as an inconsistency (Error in the output). If the reported p-value is significant and the computed is not, or vice versa, the result is marked as a gross inconsistency (DecisionError in the output).

statcheck takes into account correct rounding of the test statistic, and has the option to take into account one-tailed testing. See the manual for details.

Installation and use

For detailed information about installing and using statcheck, see the Documentation file in the github repository, or refer to the R documentation.

Installation

pip install statcheck

Example Usage

from statcheck.checkdir import checkPDFdir
dir = 'path/to/pdf/directory'
Res, pRes = checkPDFdir(dir, subdir = False)

# Res is a pandas dataframe with the analysis of statistical results
Res
# pRes is a pandas dataframe with extracted p-values
pRes

Running tests

pip install pytest
pytest tests/

statcheck.io is a web-based interface for statcheck.

Author of the Python implementation

** Hubert Plisiecki **

• Github
• ResearchGate
• LinkedIn

Citation

---
@misc{MicheleNuijten,  
  author = {Michèle B. Nuijten},  
  title = {statcheck},  
  year = {2021},  
  url = {{https://github.com/MicheleNuijten/statcheck}}  
}
---