README.md

---
license: apache-2.0
---
# LoRA Encoder (FLUX.1-Dev)

This model encodes LoRA models for FLUX into embedding vectors, unlocking the capabilities of the LoRA models.

Taking the LoRA model [VoidOc/F.1_Animal_Forest_LoRA](https://www.modelscope.cn/models/VoidOc/flux_animal_forest1) as an example, the LoRA encoder can be used in the following ways.

## Method 1: LoRA Usage Inference

Given a LoRA model, with no additional information and using an empty prompt, the LoRA encoder can directly activate the LoRA's capabilities, allowing inference of its intended use.

Prompt: `""`

|Without LoRA Encoder|With LoRA Encoder|
|-|-|
|![](./assets/image_1_origin.jpg)|![](./assets/image_1.jpg)|

## Method 2: Trigger-Free Activation of LoRA Capabilities

Activate the LoRA's capabilities automatically without needing to specify any trigger words.

Prompt: `"a car"`

|Without LoRA Encoder|With LoRA Encoder|
|-|-|
|![](./assets/image_2_origin.jpg)|![](./assets/image_2.jpg)|

## Method 3: LoRA Strength Control

An additional parameter `scale` is provided to control the influence of the LoRA on the generated image.

In the example below, the prompt is `"a cat"`. When `scale=1`, the LoRA exerts maximum influence, resulting in an image showing both a character from Animal Crossing and a cat. When `scale=0.5`, the LoRA's influence is reduced, producing an image of a cat character from Animal Crossing. The optimal `scale` value depends on the specific LoRA model; we recommend using larger values for character-based LoRAs and smaller values for style-based LoRAs.

Prompt: `"a cat"`

|`scale=1`|`scale=0.5`|
|-|-|
|![](./assets/image_3.jpg)|![](./assets/image_3_scale.jpg)|

## Inference Code

```
git clone https://github.com/modelscope/DiffSynth-Studio.git  
cd DiffSynth-Studio
pip install -e .
```

```python
import torch
from diffsynth.pipelines.flux_image_new import FluxImagePipeline, ModelConfig


pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors"),
        ModelConfig(model_id="DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev", origin_file_pattern="model.safetensors"),
    ],
)
pipe.enable_lora_magic()

lora = ModelConfig(model_id="VoidOc/flux_animal_forest1", origin_file_pattern="20.safetensors")
pipe.load_lora(pipe.dit, lora, hotload=True) # Use `pipe.clear_lora()` to drop the loaded LoRA.

# Empty prompt can automatically activate LoRA capabilities.
image = pipe(prompt="", seed=0, lora_encoder_inputs=lora)
image.save("image_1.jpg")

image = pipe(prompt="", seed=0)
image.save("image_1_origin.jpg")

# Prompt without trigger words can also activate LoRA capabilities.
image = pipe(prompt="a car", seed=0, lora_encoder_inputs=lora)
image.save("image_2.jpg")

image = pipe(prompt="a car", seed=0)
image.save("image_2_origin.jpg")

# Adjust the activation intensity through the scale parameter.
image = pipe(prompt="a cat", seed=0, lora_encoder_inputs=lora, lora_encoder_scale=1.0)
image.save("image_3.jpg")

image = pipe(prompt="a cat", seed=0, lora_encoder_inputs=lora, lora_encoder_scale=0.5)
image.save("image_3_scale.jpg")
```