| | --- |
| | license: apache-2.0 |
| | pipeline_tag: text-generation |
| | datasets: |
| | - codeparrot/github-code-clean |
| | - bigcode/starcoderdata |
| | - bigcode/the-stack-smol |
| | tags: |
| | - diffusion |
| | - llm |
| | - diffreaper |
| | - dllm |
| | - mercury |
| | language: |
| | - en |
| | --- |
| | # DiffReaper 3 |
| |
|
| | DiffReaper 3 is the third revision of DiffReaper; a experimental 1.5B parameter Discrete Diffusion Language Model (dLLM) designed for high-throughput parallel token prediction. |
| | Unlike traditional autoregressive models, DiffReaper is optimized for non-linear sequence refinement across mixed Python logic and natural language corpora. |
| |
|
| | ## Model Details |
| | - **Architecture:** 24-Layer Transformer Encoder |
| | - **Hidden Dimension:** 2048 |
| | - **Attention Heads:** 16 |
| | - **Objective:** Discrete Masked Diffusion (Mercury-style) |
| | - **Training Precision:** BF16 |
| | - **Context Window:** 1024 tokens |
