| # Test LoRA Adapter | |
| This is a test LoRA adapter (randomly initialized without tuning) with customizable target modules. | |
| ```bash | |
| python create_test_embedding_layer.py | |
| ``` | |
| ## Configuration | |
| - Base model: meta-llama/Llama-2-7b-hf | |
| - LoRA rank (r): 8 | |
| - LoRA alpha: 16 | |
| - Target modules: embed_tokens, lm_head, q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | |
| ## Weight Shapes | |
| - embed_tokens.lora_A: (8, 32000) | |
| - embed_tokens.lora_B: (4096, 8) | |
| - lm_head.lora_A: (8, 4096) | |
| - lm_head.lora_B: (32000, 8) | |
| - q_proj.lora_A: (8, 4096) | |
| - q_proj.lora_B: (4096, 8) | |
| - k_proj.lora_A: (8, 4096) | |
| - k_proj.lora_B: (4096, 8) | |
| - v_proj.lora_A: (8, 4096) | |
| - v_proj.lora_B: (4096, 8) | |
| - o_proj.lora_A: (8, 4096) | |
| - o_proj.lora_B: (4096, 8) | |
| - gate_proj.lora_A: (8, 4096) | |
| - gate_proj.lora_B: (11008, 8) | |
| - up_proj.lora_A: (8, 4096) | |
| - up_proj.lora_B: (11008, 8) | |
| - down_proj.lora_A: (8, 11008) | |
| - down_proj.lora_B: (4096, 8) | |
| ## Usage with SGLang | |
| This adapter contains randomly initialized weights for testing purposes only. | |