[New Preprocessor] The "reference_adain" and "reference_adain+attn" are added #1280
Replies: 48 comments 69 replies
-
|
In our test:
|
Beta Was this translation helpful? Give feedback.
-
|
Can style fidelity be added to the API as threshold_a. Looking forward to fully using all of these new preprocessors. |
Beta Was this translation helpful? Give feedback.
-
|
i think it is already - but not tested |
Beta Was this translation helpful? Give feedback.
-
|
sorry for repeating request, but i don't know if you see my reply in previous thread, i hope you add option to reference feature, to use multiple photos as reference instead of one photo, i guess this will solve output images is not look alike the original person 🤞🤞🤞🤞🤞 |
Beta Was this translation helpful? Give feedback.
-
|
multi-controlnet is in setting, implemented several months ago. see also the readme |
Beta Was this translation helpful? Give feedback.
-
|
yah i know about it, but i didn't get good results with it in this case, my request is like make it like lora training, by add ability to add multiple photos to the same controlnet reference with same person or style "Architecture style for example" in different angels and resolutions to make the final photo, and if possible produce a file like lora form this photos to be used with controlnet reference later, instead of adding the same multiple photos again every time, this will be revolutionary 🤞🤞🤞🤞🤞🤞🤞🤞🤞🤞🤞 |
Beta Was this translation helpful? Give feedback.
-
|
it is not working at batch tool in img2img. Could you check it? |
Beta Was this translation helpful? Give feedback.
-
|
hi all we fixed the multi-reference weight bug in 1.1.172 |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
prompt conflicts style - try "prompt is more important" or remove conflicting prompts |
Beta Was this translation helpful? Give feedback.
-
|
I figured it out - the mugshot lora was causing the color defects - probably for anime only |
Beta Was this translation helpful? Give feedback.
-
|
so remove your tests if it was mistake on your part , they are confusing |
Beta Was this translation helpful? Give feedback.
-
|
I did. My bad I'm not familiar with github discussion culture. |
Beta Was this translation helpful? Give feedback.
-
|
Did something change with lineart_standard? I'm getting a weird hidden halo effect when using lineart_standard that doesn't appear with the same input using the other lineart preprocessors. The background warps to form an outline around the input image's lines.
Also, tons of artifacts especially on faces. Again, just with lineart_standard, doesn't appear when using other lineart preprocessors. |
Beta Was this translation helpful? Give feedback.
-
|
check input image resolution - mistakenly used low-resolution or thumbnail image |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I'm working with some old legacy files that are low resolution in this case, and upscaling is causing too many changes. :/ |
Beta Was this translation helpful? Give feedback.
-
|
then just use [invert] instead of lineart_standard |
Beta Was this translation helpful? Give feedback.
-
|
Invert's also giving a weird artifact pattern with low res inputs, actually. But I figured out a workaround. Since Controlnet was already getting the lines right, I used the low-res image in txt2img then prompted for black and white lineart and used LoRAs to encourage it to remake the image as just lineart. Eventually I was able to recreate the input at 512 a little better than upscaling, which was causing gaps. |
Beta Was this translation helpful? Give feedback.
-
|
The new feature is powerful and will have a lot of use cases. Thanks for this! One thing I've noticed is that the images it produces are often slightly blurry compared to images generated without using the new feature. Is this a known issue? |
Beta Was this translation helpful? Give feedback.
-
|
Still, there is almost no face recognition, |
Beta Was this translation helpful? Give feedback.
-
the photorealistic results also depends of the checkpoint u use as they trained with limited dataset. i used to play around with face generations in artbreeder back in a days) that thing is still great |
Beta Was this translation helpful? Give feedback.
-
|
Hi so previously I made a bug report about an issue that consists of incorrect color tones that I assumed was only limited to "Controlnet is more important" But my assumption seems to be wrong. It has been happening on balanced mode with style fidelity 0.5 and 1.0 too and in all 3 "reference" pre-processors Here is the reference image: Here is all reference pre-processors with Style fidelity 1.0 in Balanced mode And here is all reference pre-processors with Style fidelity 0.5 in Balanced mode As you can see, it seems to be collapsing even at 0.5 style fidelity and the color tone seems to be more dull too. What I also did notice is this does not happen much on realistic reference images but it happens a lot more on Anime art style images that u choose for reference images. So I am guessing that is where some issue lies in |
Beta Was this translation helpful? Give feedback.
-
|
it can happen when your base model refuse to draw colors as saturated as the example - you can disable cn and generate a few images with your base model to see what color range your base model supports |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I noticed this issue as well and my base model is very colorful/vibrant without cnet enabled. I'm on 1.1.189. Here's my generation info minus prompt since it's nsfw: Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 4017221646, Size: 512x768, Model hash: f725cdf24b, Model: euclaseBlend_v12, Clip skip: 2, ENSD: 31337, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 1, 64)" |
Beta Was this translation helpful? Give feedback.
-
|
I did some testing with ref_adain+attn. Style Fidelity=1.0 Input image first gen with only quality tags:
No surprise, he looks like a female because of model's bias Now I add 1boy in prompt of the prompt:
Close enough. Facial hair, black hair is there, I don't need to use more prompt. This is the interesting part. Example that hires fix fixes some artifact at cfg9 So the conclusion is use fewer tags + low cfg? Need more testing to confirm this. |
Beta Was this translation helpful? Give feedback.
-
|
I also noticed lowering the CFG value resolves issues where certain models output very dark generations when using the new reference_only feature. |
Beta Was this translation helpful? Give feedback.
-
Could this be related to the problem I am having, scroll up a bit and you will see a post about a pink haired anime girl problem. The generations it produces make it very dark even after testing with multiple checkpoints. Checkpoints that can even do vibrant colors do this too. I noticed this because there are times where it would spit out a color correct generation once in a while so I am not 100% sure if its a model checkpoint problem or not. Although I was able to get it working with just one checkpoint model that has vibrant colors. But still its hard to believe something like Abyssmix wont be able to produce vibrant colors like the pink haired girl |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for providing this awesome preprocessor! Through reference_adain+attn, it is possible to learn illustration composition very well, and can fine-tuning character through prompts. Using Eunice Ye's Naraka character image as a reference, the generated images are compared (the left one is the original picture):
|
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to use them with diffusers yet? |
Beta Was this translation helpful? Give feedback.
-
|
Can someone explain how the method work? What is the attention mechanism of the reference-only? |
Beta Was this translation helpful? Give feedback.
-
|
I draw a simple graph for your convenience. The hidden features of a reference image within a self-attention block are initially stored in a list. These attributes are subsequently employed as image conditions to perform "cross-attention" within the same self-attention block. From my understanding, we actually perform two rounds of diffusion in this pipeline.
Regarding the check the code here |
Beta Was this translation helpful? Give feedback.
-
|
Thx for the explanation. What is reference_adain and style fidelity? |
Beta Was this translation helpful? Give feedback.
-
|
I wrote a blog post about reference-only mode. please check here |
Beta Was this translation helpful? Give feedback.
-
Update in 1.1.178Hi all, reference preprocessors are now supported in "inpaint only masked" Although you need to make some adjustments to the window, the result is good when you are done. This can achieve effects like moving objects from one image to another, like moving face, etc ExampleInput image
reference image
Mask A1111 Inpaint setting CN 1 setting CN2 setting (no input image) Meta woman face, best quality Result More results You need at least 1.1.178 to use it. |
Beta Was this translation helpful? Give feedback.
-
|
edit: nm, tired and forgot I already saw that information in the pics above and tried it myself with no luck. kinda shows how deep into the rabbit hole I've gone lol I'm having issues with the reference face not being transferred onto the output image at all. I've tried just about every permutation of settings I could think of. I didn't see what masked content mode you're using though. Is it original or latent noise? |
Beta Was this translation helpful? Give feedback.
-
|
edit: nm, answered my own question. tldr: allows adopting more stylistic aspects of the prompt via high denoise WITHOUT the original img2img reference image getting garbled up due to said denoising Should we be setting the conditional mask weight inside the xyz plot as you've done here? |
Beta Was this translation helpful? Give feedback.
-
|
I assume this is coming soon on SD and ControlNet! |
Beta Was this translation helpful? Give feedback.
-
i have this setup, it outputs kinda specular face, it is still better to train lora or dreambooth, yeah its just one pic but training in sd gives more natural output |
Beta Was this translation helpful? Give feedback.
-
That's so cool. Would it be perfectly possible to change him into clothes or hands? It'd be perfect if it could.... Then no more crumpled faces and hands, hahaha..... However, wouldn't it be better if I could go through the options in the preprocessor, for example, if I want to restore my hand, I use reference_hand,reference_clothes... This way you don't have to drive too much A controlnet to speed up build?? 这太酷了, 把他换成衣服或者手部是否也可以完美实现呢? 如果可以那真实完美了....那就不再有奔溃的脸和手了,哈哈哈哈..... |
Beta Was this translation helpful? Give feedback.
-
|
I didn't see it mentioned so figured I would ask: does it improve anything if we use a txt2img reference image that's the same size as the txt2img output? example: txt2img outputs a 512x512 image and uses a 512x512 image for the reference |
Beta Was this translation helpful? Give feedback.
-
|
you want image cropped or stretched? i personally match aspect ratio or crop, also i dont use below 512res as ref |
Beta Was this translation helpful? Give feedback.
-
|
Hmm I noticed that lllyasviel posted some reference photos that weren't the same size as the output. So you're saying you prefer matching the reference aspect ratio to the output aspect ratio, or you crop the reference so that it matches the output size exactly? If I summarized that correctly, does that have visible differences? I'm not noticing anything on my end regardless of absolute size or aspect ratio, but I'm still unsure. |
Beta Was this translation helpful? Give feedback.
-
|
I just updated my PyTorch version to 2.0 on MacOS and images are collapsing again on some models when using reference_only / reference_adain / reference_adain+attn. This wasn't the case on the previous version of PyTorch. I'm running Controlnet 1.1.189. |
Beta Was this translation helpful? Give feedback.
-
|
Are you on the latest version of the webui? |
Beta Was this translation helpful? Give feedback.
-
|
Yea, updated to the latest (89f9faa) to see if it fixed the issues, but it didn't. |
Beta Was this translation helpful? Give feedback.
-
|
Any luck? Same problem here. |
Beta Was this translation helpful? Give feedback.
-
|
Can the reference preprocessors be used in combination with controlnet models? I want to do img2img and preserve the semantic of the input image (depth, segmentation masks), and apply the style of a second image to the first image, would that be possible with the current state? |
Beta Was this translation helpful? Give feedback.
-
|
yes, reference_only works best for me |
Beta Was this translation helpful? Give feedback.
-
|
is there a way to get a bunch of MidJourney images and pass it through this without prompting but still get a great output? want to do an article comparing the progress but won't be efficient to manually prompt for 50 images :( |
Beta Was this translation helpful? Give feedback.
-
|
Would it be possible to have a negative reference? I give ControlNet an example of how it is not supposed to look like - that would be a cool feature! |
Beta Was this translation helpful? Give feedback.
-
|
when these will be available outside web ui? |
Beta Was this translation helpful? Give feedback.
-
|
When you use reference model and lineart model (two controlnets) then you can recreate reference image kinda almost perfectly Steps: 32, Sampler: DPM++ 2M Karras, CFG scale: 9.5, Seed: 2228333885, Size: 768x768, Model hash: fc52756a74, Model: dreamlikePhotoreal20_, Denoising strength: 0.66, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)", ControlNet 1: "preprocessor: lineart_realistic, model: control_v11p_sd15_lineart_fp16 [5c23b17d], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 64, 64)" |
Beta Was this translation helpful? Give feedback.
-
looks great! would be even more cool if it possible to change composition keeping the referance face/ how is that possible to do? |
Beta Was this translation helpful? Give feedback.
-
|
Is there any resolution requirement for uploaded images? I upload a picture with an indeterminate scale resolution, for example 712x1321 image as source, error will occur RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640 |
Beta Was this translation helpful? Give feedback.
-
|
One more mistake: RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 2 for tensor number 1 in the list. Strangely enough, when I reduced the number of prompt words, the errors disappeared, It appears when I start controlnet reference_only and other renference_xx |
Beta Was this translation helpful? Give feedback.
-
|
Well, I tested it again, and it didn't work. My cn version: sd Startup parameters :--opt-channelslast --opt-sdp-no-mem-attention --api --device-id=1 --autolaunch --update-check --update-all-extensions txt2img resolution Settings: cn upload resolution: 540x960 What's the problem? Or am I using it the wrong way? Can I get some help? Thank you 好吧,我又测试了一次,还是不行. sd启动参数:--opt-channelslast --opt-sdp-no-mem-attention --api --device-id=1 --autolaunch --update-check --update-all-extensions cn upload resolution: 540x960 问题出现在哪里呢? 还是我用错方法了?? 有人帮忙一下吗,谢谢 |
Beta Was this translation helpful? Give feedback.
-
|
remove --opt-channelslast |
Beta Was this translation helpful? Give feedback.
-
|
hi, i remove, also the same error RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640 and this is my running environment: version: when i change the args to : --device-id=1 --autolaunch but the other controlnet preprocessor working welll just preprocessor reference_ not working :( |
Beta Was this translation helpful? Give feedback.
-
|
solve: |
Beta Was this translation helpful? Give feedback.
-
|
Guys, I see you have preprocessor params. Where does one set these? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I achieve a cross-image region drag based on the reference scheme. 1. Use inpaint controlnet to extract inpainted region feature from another image. 2. Use the segment-anything controlnet to keep reasonable pose. Code in https://github.com/sail-sg/EditAnything 
|
Beta Was this translation helpful? Give feedback.
-
|
Hey, can you give more mathematical details about how reference-only is implemented? I am very curious about it. |
Beta Was this translation helpful? Give feedback.
-
|
And what is reference_adain and style fidelity? |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if it's possible to use this method to restrict the Color scheme in the output, much like T2I-Adapter Color (but it's not available for SDXL) |
Beta Was this translation helpful? Give feedback.
-
|
Greetings 👋, I'm employing the Fooocus V2, Fooocus Photograph, Fooocus Negative model in replication to create realistic photos. However, despite providing four high-quality input photos, I'm encountering challenges in achieving accurate face matching in the AI-generated images, and the quality of the eyes is consistently subpar and very bad, not real like venom. Any suggestions on how to address this issue? |
Beta Was this translation helpful? Give feedback.
-
|
about reference_only，I can't reproduce it photos after May 16.Even with the same parameters. |
Beta Was this translation helpful? Give feedback.
-
|
Even if I go back to the version, I can't reproduce it |
Beta Was this translation helpful? Give feedback.
and others
-
V1.1.171 adds two new reference preprocessors:
reference_adain
AdaIn (Adaptive Instance Normalization) from
Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
Xun Huang and Serge Belongie, Cornell University
https://arxiv.org/abs/1703.06868
reference_adain+attn
AdaIn (Adaptive Instance Normalization) + Attention link same as "reference_only"
Comparison
Input image (midjourney v5, https://twitter.com/kajikent/status/1654409097041817601)
meta
woman in street, masterpiece, best quality,
Negative prompt: lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 12345, Size: 768x512, Model hash: c0d1994c73, Model: realisticVisionV20_v20, Version: v1.2.0, ControlNet 0: "preprocessor: reference_????, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 0.5, 64)"
reference_only (style fidelity=0.5):
reference_adain (style fidelity=0.5):
reference_adain+attn (style fidelity=0.5):
without CN (all controlnets disabled):
You need at least 1.1.171 to use them.
Beta Was this translation helpful? Give feedback.
All reactions