Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Marigold depth - high-quality diffusion-based monocular depth estimation #2330

Closed
toshas opened this issue Dec 15, 2023 · 7 comments
Assignees
Labels
enhancement New feature or request

Comments

@toshas
Copy link

toshas commented Dec 15, 2023

Improved monocular depth estimation algorithm based on fine-tuned stable diffusion. Expecting better guidance for depth-to-image and other models employing monocular depth.

https://marigoldmonodepth.github.io/
https://twitter.com/AntonObukhov1/status/1732946419663667464

@sdbds sdbds added the enhancement New feature or request label Dec 15, 2023
@sdbds sdbds self-assigned this Dec 15, 2023
@sdbds
Copy link
Collaborator

sdbds commented Dec 15, 2023

It looks like it could be combined with Patchfusion, according to their paper, is a high-definition approach to depth maps that is technically independent of the base model.

@toshas
Copy link
Author

toshas commented Dec 15, 2023

Great; Marigold produces depth maps of the same revolution as input

@huchenlei
Copy link
Collaborator

Wow, Marigold inference is using a diffusion pipeline to produce depth maps? I think this is heavy enough that we make it an optional component. (Or we want a component management system for ControlNet?)

@toshas
Copy link
Author

toshas commented Jan 23, 2024

Yes, but it can be integrated similarly to https://github.com/kijai/ComfyUI-Marigold by copying the code, so there is no need for extra dependencies. When we roll out an update for Marigold, we will notify dependency repositories about improvements upstream!

@sdbds
Copy link
Collaborator

sdbds commented Jan 23, 2024

Yes, but it can be integrated similarly to https://github.com/kijai/ComfyUI-Marigold by copying the code, so there is no need for extra dependencies. When we roll out an update for Marigold, we will notify dependency repositories about improvements upstream!

Apologies for the late reply, because while huggingface's range of packages are very good to replicate and implement, the iteration between versions makes breaking changes difficult to maintain.
For example diffusers 0.20 and 0.21 are not common between them, nor are 0.24 and 0.25.
We try not to use diffusers in webui, and we're trying to figure out how to read the diffusers config file to replicate the neural network constructs.

@sdbds
Copy link
Collaborator

sdbds commented Jan 23, 2024

Wow, Marigold inference is using a diffusion pipeline to produce depth maps? I think this is heavy enough that we make it an optional component. (Or we want a component management system for ControlNet?)

I think a generic config neural network structure for reading diffusers might be the way to go.

@huchenlei
Copy link
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants