README.md

---
license: apache-2.0
datasets:
- laion/laion-art
language:
- en
library_name: diffusers
pipeline_tag: image-to-image
tags:
- jax-diffusers-event
base_model: runwayml/stable-diffusion-v1-5
---

# Color-Canny CantrolNet

These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning. 

You can find some example images in the following. 

## Examples

#### Color examples

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_1)](./1.png)

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_2)](./2.png)

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_3)](./3.png)


#### Brightness examples
This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7).
![images_4)](./4.jpg)


## Limitations and Bias

- No strict control by input color
- Sometimes generate image with confusion When color description in prompt

## Training

**Dataset**
We train this model on [laion-art](https://huggingface.co/datasets/laion/laion-art) dataset with 2.6m images, the processed dataset can be found in [ghoskno/laion-art-en-colorcanny](https://huggingface.co/datasets/ghoskno/laion-art-en-colorcanny).


**Training Details**

- **Hardware**: Google Cloud TPUv4-8 VM

- **Optimizer**: AdamW

- **Train Batch Size**: 4 x 4 = 16

- **Learning rate**: 0.00001 constant

- **Gradient Accumulation Steps**: 4

- **Resolution**: 512

- **Train Steps**: 36000