lfoppiano/MatTPUSciBERT

Material SciBERT (TPU): Improving language understanding in materials science

Work in progress

Introduction

SciBERT-based model pre-trained with materials science scientific fulltext

Authors

Luca Foppiano Pedro Ortiz Suarez

TLDR

• Collected full-text from ~700000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM-PF/en/), dataset called ScienceCorpus (SciCorpus)
• We added to the SciBERT vocabulary (32k tokens), 100 domain-specific unknown words extracted from SciCorpus with a keywords modeler (KeyBERT)
• Starting conditions: original SciBERT weights
• Pre-train the model MatTpuSciBERT from on the Google Cloud with the TPU (Tensor Processing Unit) as follow:
  • 800000 steps with batch_size: 256, max_seq_length:512
  • 100000 steps with batch_size: 2048, max_seq_length:128
• Fine-tuning and testing on NER on superconductors (https://github.com/lfoppiano/grobid-superconductors) and physical quantities (https://github.com/kermitt2/grobid-quantities)

Related work

BERT Implementations

• BERT (the original) https://arxiv.org/abs/1810.04805
• RoBERTa (Re-implementation by Facebook) https://arxiv.org/abs/1907.11692

Relevant models

• SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert
• MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K paper limited to 4 MS families http://github.com/m3rg-iitd/matscibert
• MaterialBERT: Not yet published
• MatBERT (CEDER): BERT, from scratch, 2M documents on materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT
• BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/

Results

Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft)

NER Superconductors

Model	Precision	Recall	F1
SciBERT (baseline)	81.62%	84.23%	82.90%
MatSciBERT (Gupta)	81.45%	84.36%	82.88%
MatTPUSciBERT	82.13%	85.15%	83.61%
MatBERT (Ceder)	81.25%	83.99%	82.60%
BatteryScibert-cased	81.09%	84.14%	82.59%

NER Quantities

Model	Precision	Recall	F1
SciBERT (baseline)	88.73%	86.76%	87.73%
MatSciBERT (Gupta)	84.98%	90.12%	87.47%
MatTPUSciBERT	88.62%	86.33%	87.46%
MatBERT (Ceder)	85.08%	89.93%	87.44%
BatteryScibert-cased	85.02%	89.30%	87.11%
BatteryScibert-cased	81.09%	84.14%	82.59%

References

This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers

Acknowledgements

TBA