• 2023/05/11: We release new checkpoints for sketch control from controlnetv1.1. We utilize the SAM palette as the latent input in an image2image manner and also implemented the rectangular palette via controlnet.

• 2023/04/17: We release a simple code for color controlnet. Our team has incorporated either rectangular or SAM palette to effectively control the color.

We provide two color condition inputs:

1. Rectangular downsample color palette.

2. Segmentation mask from SAM

Download the pretrained weights.

Training code: our model is compatible with the training code of diffusers controlnet.

Version 1: Run infer_palette.py to reproduce the results.

Version 2 (recommended): Run infer_palette_img2img.py to reproduce the results.

The first cell pertains to the sketch condition, while the subsequent cell represents the rectangular palette condition. The third cell corresponds to the SAM palette. The final four cells showcase four distinct images that were generated using varying seeds.

When utilizing the SAM palette in an image-to-image capacity, it is likely that the results obtained will be low-contrast, as the color of a mask is limited to a single value. However, the implementation of a rectangular palette can serve to address this limitation by introducing color variations into the denoising process. An ablation study example is presented below to further illustrate this concept.

With this model, we can easily edit the color of an image by simply scribbling.