Emotion-conditioned scene generation: a caption and an emotion condition the diffusion model, and classifier guidance steers sampling toward the target emotion
The pipeline: a scene caption and a target emotion condition the diffusion model, and an emotion classifier can guide or reinforce sampling. Illustration.

The question. Text prompts control what an image shows far better than how it feels. Can Stable Diffusion be steered cheaply toward a target emotion, and which conditioning route works best?

Approaches

Three ways to condition Stable Diffusion 1.5 on eight emotions (amusement, anger, awe, contentment, disgust, excitement, fear, sadness), trained on EmoSet, with RAF-DB for portraits:

  1. Learned emotion tokens with LoRA. Eight new tokens such as <awe> are learned jointly with LoRA adapters (rank 32) on the UNet’s attention layers.
  2. Classifier guidance. A noise-aware classifier on the diffusion latents steers sampling with its gradients, with no change to the UNet.
  3. Multimodal conditioning. BLIP captions combined with EmotionCLIP emotion embeddings, with an optional emotion-classifier loss during training.

Results

  • Efficient. LoRA trains 0.4% of the model’s parameters, which makes training 15× faster and uses 80% less memory than full fine-tuning.
  • Measured. Every approach is scored by an emotion classifier on its own generations (EmotionCLIP for scenes, a ViT for portraits), with confusion matrices and per-emotion accuracy.