Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebP: Automatically rotate the image when converting to JPEG #3342

Open
dessalines opened this issue Apr 17, 2023 · 14 comments
Open

WebP: Automatically rotate the image when converting to JPEG #3342

dessalines opened this issue Apr 17, 2023 · 14 comments
Labels
help wanted Well suited for external contributors! idea Feedback wanted / feature request metadata Related to Exif, XMP, IPTC & Co.

Comments

@dessalines
Copy link

dessalines commented Apr 17, 2023

#464

1. What is not working as documented?

When I upload a webp file, if its orientation is sideways, photoprism ignores the photo orientation.

Example: https://demo.photoprism.app/library/folders/art9w3jma2muz7xn/view#&gid=1&pid=1

The orientation field is correct below, but is ignored in the software.

Screenshot_2023-04-17-15-25-51-893_net slions fulguris full fdroid

2. How can we reproduce it?

Create a webp photo. Rotate it. Save it.

3. What behavior do you expect?

Showing the photo in the correct orientation.

4. What could be the cause of your problem?

Not sure exactly.

5. Can you provide us with example files for testing, error logs, or screenshots?

https://demo.photoprism.app/library/folders/art9w3jma2muz7xn/view#&gid=1&pid=1

6. Which software versions do you use?

Its broken on photoprism:latest, see the demo linked above

@dessalines dessalines added the bug Something isn't working label Apr 17, 2023
@lastzero
Copy link
Member

Please also explain the use case and how you have created and rotated the WebP image. From what I know, there are currently no cameras that natively create WebP files. At best Google might do this? If no vendor is doing this, I'd consider it a nice to have feature as WebP images are generally created with the correct orientation with the main use case of speeding up websites.

@lastzero lastzero changed the title Photoprism ignores webp orientation WebP: Automatically rotate the image when converting to JPEG Apr 17, 2023
@lastzero lastzero added idea Feedback wanted / feature request metadata Related to Exif, XMP, IPTC & Co. help wanted Well suited for external contributors! and removed bug Something isn't working labels Apr 17, 2023
@dessalines
Copy link
Author

dessalines commented Apr 17, 2023

My use case:

  • I have a massive folder of images, which I've converted into webp files using a script I wrote.. This saves ~40% disk space over the original files.
  • Use photoprism to scan that folder.

You can take any file, and run cwebp FILE -mt -metadata all , and it correctly copies the metadata (including orientation), from any jpeg / png file.

Photoprism is clearly reading the webp metadata because it displays the correct orientation in the details, but both the image, and thumbnails aren't shown right-side-up.

@lastzero
Copy link
Member

As PhotoPrism will create JPEG files for all WebP images, this approach ultimately requires more, not less storage. Since you already did custom development, you should modify your convert script in a way which auto rotates the images so you don't need an additional oritentation flag in the first place. This is what we do when we convert the WebP back to JPEG.

@lastzero
Copy link
Member

lastzero commented Apr 17, 2023

See also:

Exif Orientation is not common for WebP based on the intended use csse. It could also be automatically rotated and the metadata rotates it again, also resulting in a bad oritentation. Best is to auto rotate and then to set the Exif oritentation to 1 or even remove it completely.

@dessalines
Copy link
Author

I just tried to convert this webp file using magick FILE.webp -auto-orient test.jpg, and unfortunately it ignored the orientation. So this is likely an issue with ImageMagick.

Are you using imagemagick to do the webp -> jpeg conversion? If so, then I should open up this bug over there.

As PhotoPrism will create JPEG files for all WebP images, this approach ultimately requires more, not less storage.

Dang, that's unfortunate. That's just for the thumbnail images tho correct? I'd assumed the originals folder can be in any format, the jpeg conversion is only for generated thumbnails.

@lastzero
Copy link
Member

lastzero commented Apr 18, 2023

I just tried to convert this webp file using magick FILE.webp -auto-orient test.jpg, and unfortunately it ignored the orientation. So this is likely an issue with ImageMagick.
Are you using imagemagick to do the webp -> jpeg conversion? If so, then I should open up this bug over there.

(a) It's probably a problem with many applications, as WebP simply wasn't designed for what you're using it for. This is not a support forum and we are just preparing our next release. So unfortunately I have limited time to go into the full history of the file format and all the technical details. Essentially, WebP was developed by Google to speed up the loading of websites. Anything that slows this down - including the need to interpret Exif data to display it correctly - runs counter to this. Also, WebP has a 16K limitation (for 16K VIDEO projectors), while even the old JPEG supports at least 30K pixels (since it's a native image file format and users can only see part of a panorama, which isn't necessary for video since pixels usually move when you want to show more). Last but not least, WebP support is only partially implemented, even in Google's own WebP library, so sometimes we have to resort to imagemagick, for example for WebP image sequences (multiple images in the same container). IMHO this is total nonsense, because an image sequence is a video, so you could have used WebM in the first place, but yeah. It's a mess. You can report a bug, but it won't help.

Dang, that's unfortunate. That's just for the thumbnail images tho correct? I'd assumed the originals folder can be in any format, the jpeg conversion is only for generated thumbnails.

(b) The native formats are currently JPEG and PNG (if you use our development preview):

https://docs.photoprism.app/getting-started/faq/#what-media-file-types-are-supported

  • the reason for this is interoperability with other software e.g. for sharing, compatibility with ALL browsers and (a) so PhotoPrism can create thumbnails from all originals that even Google's own WebP library may not be able to read without crashing
  • reading Exif from WebP can also result in a brute-force search, essentially reading the entire file to find the header, as proper Exif header parsers only exist for common formats like JPEG, etc.
  • this is unfortunate and Google won't tell you about all these problems and drawbacks of WebP, but you can't really blame independent developers like us either

@lastzero
Copy link
Member

The Google team said Auto-Rotate is not implemented in their cwebp tool to "keep the scope of this [mess] manageable" and marked the issue as "WontFix (Closed)":

Just as implementing their own image sequence support in their own Go library seems "too much work":

"Animated WebP has depressingly poor support, even on Android"

"I doubt that anybody is opposed to doing this, but somebody has to do the work."

Lo and behold! I'm speechless 👀

@lastzero
Copy link
Member

lastzero commented Apr 18, 2023

My bet is that this can be fixed with like 10 lines of code in the cwebp tool, if just someone would do the work. Common JPEG libraries have an auto-rotate flag when you open an image, so there's not much to do.

@lastzero
Copy link
Member

While I understand that WebP is open source and therefore you "shouldn't complain", note that Google's management wants developers to use this format instead of competing (and much more suitable) alternatives like JPEG XL. So not only is the file format less capable, but the tools are broken.

Ultimately, this leads to bug reports for every single app that supports WebP. Instead, Google should do its homework and not close important issues with "Happy Weekend!". That would help WebP a lot.

@dessalines
Copy link
Author

I apologize, I didn't realize webp had so many issues... I'd just thought it was a better image format in terms of quality and storage space.

I'm not sure if I should leave this open or not, but I'll have to find a way to convert my library back to jpeg, while trying to solve these orientations issues.

@lastzero
Copy link
Member

No need to apologize! I only found this all out while implementing WebP support for PhotoPrism. Google seems very interested in making it the standard for everything for whatever reason, maybe because they think they can or to maximize shareholder value. Who knows?

From a technical perspective, I'd either go for AVIF if you need a solution now, or hope for the adoption of JPEG XL (which Google is actively working against, unfortunately):

https://avif.io/blog/comparisons/avif-vs-jpegxl/

With PhotoPrism, simple JPEG, PNG, or AVC (for video) files currently consume the least amount of storage, since they don't need to be converted before being displayed in a browser or used with TensorFlow or other libraries. This may change in a future release, but we'll wait for the dust to settle.

That being said, there are ways to work around the WebP issues and make sure the image is always rotated correctly (if you can accept the extra storage requirements for a JPEG sidecar file). I'd be reluctant to work on this though, as we have a lot of other requests whose implementation will bring real value to our users.

@dessalines
Copy link
Author

For anyone who got stuck in the mess I'm in, and wants to go back from webp -> jpg, here's a script to do it that correctly handles the rotation. Since a normal imagemagick convert can't handle the webp orientation headers, its unfortunately a 4-step process.

https://github.com/dessalines/webp-convert-dir/blob/main/jpg_convert_dir.sh#L10

@lastzero
Copy link
Member

Keep in mind that rotating and reencoding JPEG and WebP images causes a slight loss in quality, so this should be limited to what is absolutely necessary. That's one of the reasons why there is a header for photos taken by cameras, so you don't lose quality. Obviously there is absolutely no need to rotate via Exif if you reencode an image anyway.

@lastzero lastzero moved this to Ideas 💭 in Roadmap 🚀✨ Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Well suited for external contributors! idea Feedback wanted / feature request metadata Related to Exif, XMP, IPTC & Co.
Projects
Status: Ideas 💭
Development

No branches or pull requests

2 participants