README.md

---
language:
- en
license: openrail
library_name: diffusers
tags:
- diffusion-llm
- parallel-generation
- custom-transformer
- cropmark
datasets:
- OpenAssistant/oasst1
metrics:
- cosine_similarity
base_model:
- darwinkernelpanic/DiffReaper-Talk
---

# DiffReaper-5

DiffReaper-5 is a **Conditioned Diffusion Large Language Model (DLLM)** designed for high-throughput, parallel conversational text generation. Unlike standard autoregressive models (GPT-style), DiffReaper-5 operates in the continuous latent embedding space, denoising an entire response sequence in parallel.

## Model Details

- **Architecture:** Custom 12-layer Mercury-inspired Transformer.
- **Task:** Conditioned Text Diffusion (Prompt-Response).
- **Latent Space:** 1024-dimensional continuous embeddings.
- **Training Objective:** Cosine Similarity Regression (Directional Loss).
- **Sampling:** 10-step iterative parallel denoising.

## Usage (Inference)

Unlike autoregressive models, DiffReaper-5 generates the entire response in parallel through iterative denoising. Use the following logic to run inference:

```python
import torch
import torch.nn.functional as F
# Assuming DiffReaperModel is defined as per train_autogrow.py

def generate(model, tokenizer, prompt, steps=10):
    model.eval()
    with torch.no_grad():
        p_tokens = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
        p_emb = model.token_embedding(p_tokens[:, :32]) # Hard conditioning
        
        # Start from pure noise
        r_noise = torch.randn(1, 32, 1024).to("cuda")
        
        for i in range(steps):
            t = torch.tensor([1000 - (i * (1000//steps)) - 1], device="cuda").long()
            pred = model(torch.cat([p_emb, r_noise], dim=1), t)
            r_0_pred = pred[:, 32:, :] # Extract response
            r_noise = 0.4 * r_noise + 0.6 * r_0_pred # Iterative refinement
            
        # Map to vocab using Cosine Similarity
        norm_weights = F.normalize(model.token_embedding.weight, dim=-1)
        norm_r = F.normalize(r_noise, dim=-1)
        logits = torch.matmul(norm_r, norm_weights.T)
        return tokenizer.decode(torch.argmax(logits, dim=-1)[0])

# --- Loading Example ---
# model = DiffReaperModel(vocab_size=50257, n_embd=1024, n_head=16, n_layer=12).to("cuda")
# model.load_state_dict(torch.load("cropmark_latest.pt"))
```

## Fine-tuning

To fine-tune DiffReaper-5 on a custom dataset:
1. **Objective:** Use `1 - F.cosine_similarity` between predicted and target embeddings.
2. **Conditioning:** Ensure your data loader provides a fixed-length prompt prefix followed by the target response.
3. **Architecture:** Maintain the 1024-dimensional latent space to stay compatible with the weights.

## 📈 Diagnostic: Cropmark

The model's progress is monitored via the **Cropmark Diagnostic**. 
- **Cropmark** tests the model's ability to manifest a response (e.g., "I am good, how are you?") from pure Gaussian noise given a fixed prompt.
- Results are logged in `checkpoint_log.txt` and uploaded periodically.