The National Institute of Standards and Technology (NIST) has developed a large-scale benchmarking platform called JARVIS-Leaderboard to improve reproducibility in materials design research. The platform, funded by the CHIPS for America program, aims to address the current issue where only 5-30% of research papers are reproducible.
JARVIS-Leaderboard covers five main materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). It allows users to set up custom benchmarks and enables contributions in the form of datasets, code, and metadata submissions. The platform currently has 1,281 contributions to 274 benchmarks using 152 methods, with over 8 million data points.
The project’s goal is to provide a comprehensive understanding of how current and next-generation materials impact semiconductor device performance, which is crucial for U.S. semiconductor manufacturing. By facilitating benchmarking and enhancing reproducibility, JARVIS-Leaderboard aims to help researchers understand the strengths and limitations of various computational and experimental methods in materials science.
Keywords: benchmarking, reproducibility, materials design, quantum computation, artificial intelligence
