| # Joint NT-ESM2 DNA-Protein Models | |
| This repository contains jointly trained Nucleotide Transformer (NT) and ESM2 models for DNA-protein sequence analysis. | |
| ## Model Components | |
| ### DNA Model (`dna/`) | |
| - **Type**: Nucleotide Transformer for DNA sequences | |
| - **Context**: 4096 tokens | |
| - **Training**: Transcript-specific coding sequences | |
| ### Protein Model (`protein/`) | |
| - **Type**: ESM2 for protein sequences | |
| - **Variant**: Large model | |
| - **Training**: Corresponding protein sequences | |
| ## Usage | |
| **IMPORTANT**: Both models are masked language models. The DNA model uses the Nucleotide Transformer architecture which requires `trust_remote_code=True`. | |
| ```python | |
| from transformers import AutoModelForMaskedLM, AutoTokenizer | |
| # Load DNA model - requires trust_remote_code for custom NT architecture | |
| dna_model = AutoModelForMaskedLM.from_pretrained("vsubasri/joint-nt-esm2-transcript-coding", subfolder="dna", trust_remote_code=True) | |
| dna_tokenizer = AutoTokenizer.from_pretrained("vsubasri/joint-nt-esm2-transcript-coding", subfolder="dna", trust_remote_code=True) | |
| # Load protein model | |
| protein_model = AutoModelForMaskedLM.from_pretrained("vsubasri/joint-nt-esm2-transcript-coding", subfolder="protein") | |
| protein_tokenizer = AutoTokenizer.from_pretrained("vsubasri/joint-nt-esm2-transcript-coding", subfolder="protein") | |
| # Example joint usage | |
| dna_seq = "ATGAAACGCATTAGCACCACCATTACCACCACCATCACCATTACCACAGGTAACGGTGCGGGCTGA" | |
| protein_seq = "MKRISLHHHHHHHQVTVRWD" | |
| dna_inputs = dna_tokenizer(dna_seq, return_tensors="pt") | |
| protein_inputs = protein_tokenizer(protein_seq, return_tensors="pt") | |
| dna_outputs = dna_model(**dna_inputs) | |
| protein_outputs = protein_model(**protein_inputs) | |
| ``` | |
| ## Training Details | |
| - **Joint Training**: Models trained together for cross-modal understanding | |
| - **Batch Size**: 8 | |
| - **Data**: Transcript-specific coding sequences with corresponding proteins | |
| - **Architecture**: Maintained original NT and ESM2 architectures | |
| ## Repository Structure | |
| ``` | |
| ├── dna/ # NT DNA model | |
| │ ├── config.json | |
| │ ├── model.safetensors | |
| │ ├── tokenizer_config.json | |
| │ ├── vocab.txt | |
| │ └── special_tokens_map.json | |
| ├── protein/ # ESM2 protein model | |
| │ ├── config.json | |
| │ ├── model.safetensors | |
| │ ├── tokenizer_config.json | |
| │ ├── vocab.txt | |
| │ └── special_tokens_map.json | |
| └── joint_config.json # Joint model configuration | |
| ``` | |
| ## Citation | |
| If you use these models, please cite the original NT and ESM2 papers along with your joint training methodology. | |