Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux/macOS/BSD chromium cookies could not be decrypted; failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed #6564

Closed
9 of 10 tasks
finete opened this issue Mar 19, 2023 · 29 comments · Fixed by #11425
Labels
bug Bug that is not site-specific core:cookies Related to core cookie handling (e.g. --cookies or --cookies-from-browser) external issue Issue with an external tool or site

Comments

@finete
Copy link

finete commented Mar 19, 2023

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

  • I'm reporting a bug unrelated to a specific site
  • I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
  • I've checked that all provided URLs are playable in a browser with the same IP and same login details
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Trying to download a video from panopto doesn't seem to work as it doesnt seem to decrypt the cookies.
Very similar to #1073 which seems to be resolved
However I have the the most updated version but still seems to be not working.

I am running this on linux mint

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt-dlp -vU --cookies-from-browser chromium https://huji.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=64d725f2-972e-42f7-9eeb-afbf0067727a[debug] Command-line config: ['-vU', '--cookies-from-browser', 'chromium', 'https://huji.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=64d725f2-972e-42f7-9eeb-afbf0067727a']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d]
[debug] Lazy loading extractors is disabled
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.15.0-60-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2019.11.28, mutagen-1.46.0, pyxattr-0.6.1, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.4
[Cookies] Extracting cookies from chromium
[debug] Extracting cookies from: "/home/ratio/.config/chromium/Default/Cookies"
[Cookies] Loading cookie      0/   317[debug] detected desktop environment: CINNAMON
[debug] Chosen keyring: GNOMEKEYRING
WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?
[Cookies] Extracted 245 cookies from chromium (72 could not be decrypted)
[debug] cookie version breakdown: {'v10': 0, 'v11': 317, 'other': 0, 'unencrypted': 0}
[debug] Proxy map: {}
[debug] Loaded 1788 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[Panopto] Extracting URL: https://huji.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=64d725f2-972e-42f7-9eeb-afbf0067727a
[Panopto] 64d725f2-972e-42f7-9eeb-afbf0067727a: Downloading JSON metadata
ERROR: [Panopto] 64d725f2-972e-42f7-9eeb-afbf0067727a: This video is only available for registered users. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies
  File "/home/ratio/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "/home/ratio/.local/lib/python3.8/site-packages/yt_dlp/extractor/panopto.py", line 379, in _real_extract
    delivery_info = self._call_api(
  File "/home/ratio/.local/lib/python3.8/site-packages/yt_dlp/extractor/panopto.py", line 62, in _call_api
    self.raise_login_required(method='cookies')
  File "/home/ratio/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 1154, in raise_login_required
    raise ExtractorError(msg, expected=True)
@finete finete added bug Bug that is not site-specific triage Untriaged issue labels Mar 19, 2023
@pukkandan
Copy link
Member

[debug] detected desktop environment: CINNAMON
[debug] Chosen keyring: GNOMEKEYRING

Are these detected correctly?


cc @mbway

@coletdjnz
Copy link
Member

(also note that --cookies-from-browser doesn't work with panopto iirc due to the use of session cookies)

@finete
Copy link
Author

finete commented Mar 20, 2023

@pukkandan Yes about Cinnamon, not sure about GNOMEKEYRING how do i check?

@coletdjnz How do i download form panopto than?

@coletdjnz
Copy link
Member

@pukkandan Yes about Cinnamon, not sure about GNOMEKEYRING how do i check?

@coletdjnz How do i download form panopto than?

You will need to manually export the cookies from your browser. Note that in my experience this has to be done every so often for panopto.

https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp

@mbway
Copy link
Contributor

mbway commented Mar 25, 2023

As for why some cookies aren't decrypting properly, this looks like another case where the password is not obtained correctly, but the reason why isn't obvious. Like with previous cases, some cookies are reported successful decrypted. I think this just means decryption finished without error but the result could still be wrong.

A basic (but not foolproof) check for whether chromium is using the gnome keyring would be to run seahorse and see if you see a Chromium Safe Storage entry.

Since you're concerned with Panopto you may not be interested in troubleshooting this issue further, but if you are:

If you could install another chromium based browser that you haven't used before (eg brave), log into a service like youtube and attempt --cookies-from-browser brave I wonder if that would work. If it doesn't I may install mint for myself and see if I can reproduce the issue.

@Tatsh
Copy link
Contributor

Tatsh commented Aug 1, 2023

If you are on Linux, things like tmux may somehow affect how the decryption key is grabbed (in my case via KWallet 5). Most often I have to use yt-dlp without tmux.

@mbway
Copy link
Contributor

mbway commented Aug 1, 2023

that is bizarre if that's the case. I can't reproduce the issue. I tried the following situations:

  • Arch
  • Plasma 5.27 + KWallet
  • tmux 3.3
  • Konsole and Terminator
  • Brave, Chromium and Firefox

I had no issues. If you can find a situation that repeatably breaks I can take a look

@Despruk
Copy link

Despruk commented Sep 1, 2023

I was having the same issue with chromium browser (Gnome desktop)

Using seahorse I saw that I had multiple Chromium Safe Storage entries.
In the details of these entries there is a key application, which was set to application: chromium for one of the entries, but other ones were application: discord and application: slack.
Once I deleted the other entries, leaving only application: chromium one, the cookie extraction started working.

@bashonly bashonly removed the triage Untriaged issue label Sep 2, 2023
@protomors
Copy link

@Despruk Thank you. This helped me to solve the same problem. In my case it was Discord and VSCode that created another Chromium Safe Storage.
But it would be preferable for yt-dlp to check application field to find the right entry.

@Tatsh
Copy link
Contributor

Tatsh commented Oct 1, 2024

After a system and likely Chrome update, getting this error again. Chrome Beta 130.0.6723.6

[debug] Extracting cookies from: "/home/tatsh/.config/google-chrome/Default/Cookies"
[Cookies] Loading cookie      0/  2967[debug] detected desktop environment: KDE6
[debug] Chosen keyring: KWALLET6
[debug] using kwallet-query to obtain password from KWALLET6
[debug] NetworkWallet = "kdewallet"
[debug] password found
[Cookies] Loading cookie      1/  2967WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?

@peterwillcn
Copy link

peterwillcn commented Oct 1, 2024

Any news ?
Use MacOS sequoia 15.0 darwin_x64 chromium 130.0.6690.0

[Cookies] Loading cookie      0/   582WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?

@mbway
Copy link
Contributor

mbway commented Oct 1, 2024

Most likely it's the same issue as this
thewh1teagle/rookie#50

@Tatsh
Copy link
Contributor

Tatsh commented Oct 1, 2024

Most likely it's the same issue as this thewh1teagle/rookie#50

That issue looks like it only applies to Windows.

@mbway
Copy link
Contributor

mbway commented Oct 1, 2024

sorry, with the rookie issue being posted so recently I assumed it had to be related without looking carefully.

Although the symptom is the same as the original post in this issue thread, this probably warrants a new issue being opened since the cause of these new decryption issues is clear (caused by chrome upgrade).

I have been able to reproduce the error with chrome 130.0.6723.19 on Arch/KDE6.

I read through the changelog for recent changes to cookie storage in chromium but didn't find anything obvious that could be causing the breakage.

https://chromium.googlesource.com/chromium/src/+log/refs/heads/main/components/os_crypt

I tried playing around with some data as well and have some notes:

  • a hand-crafted example with values manually extracted from kwallet and the database still fail, so it isn't that the password is not being detected properly or something.
  • google-chrome-beta is definitely still using kwallet (new folder created on first launch)
  • looks like a fixed IV is still being used because many cookies had the same 32 bytes of ciphertext after the v11 prefix

@Tatsh
Copy link
Contributor

Tatsh commented Oct 2, 2024

This morning I was on an older Chrome beta (not updated machine, but not horribly out of date) and password decryption was still working. So it's definitely a very recent change.

@mbway
Copy link
Contributor

mbway commented Oct 2, 2024

What was the version of the working chrome? If it comes to bisecting commits it might narrow it down

@Tatsh
Copy link
Contributor

Tatsh commented Oct 2, 2024

The working version of beta was 128.0.6613.36 from about 2 months ago. Non-working likely 130.0.6723.6 and beyond (or perhaps 130.anything and beyond).

I also would say probably most or all of the 129 versions work but I cannot confirm. This is just based on that ever since about my update to 130.0.6723.6 the decryption stopped working.

https://github.com/gentoo/gentoo/commits/master/www-client/google-chrome-beta for versions I've been using.

@digitalsignalperson
Copy link

I'm running into this after updating my system

Arch with Plasma 6.2.1 with kwallet disabled and --password-store=basic in chromium flags, chromum 130.0.6723.58-1

~/.config/kwalletrc contains

[Wallet]
Enabled=false

For yt-dlp using --cookies-from-browser chromium:/path/to/custom/profile and seeing

[debug] Extracting cookies from: "/path/to/custom/profile/Default/Cookies"
[Cookies] Loading cookie      0/  1154WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?
Extracted 705 cookies from chromium (434 could not be decrypted)
[debug] cookie version breakdown: {'v10': 1154, 'v11': 0, 'other': 0, 'unencrypted': 0}

where previously no issues.

@digitalsignalperson
Copy link

and to help bisect this was the last version of chromium I was using where it was working: chromium-129.0.6668.58-1 from ~ Sept 17

@kesor
Copy link
Contributor

kesor commented Oct 22, 2024

I fixed it by hacking around the problem.

I tracked the "source" of this message:

WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?

The decryption of cookies is decoding the decrypted "plaintext" using UTF-8. And for some reason, I didn't yet track why exactly, there are 32 extra bytes of garbage at the beginning of the string. As others noted, this started very recently.

https://github.com/yt-dlp/yt-dlp/blob/2a24674/yt_dlp/cookies.py#L1013-L1021

The "return plaintext.decode()" is failing. After comparing values from different steps of the process against my actual cookies, I noticed there were simply extra bytes (garbage? idk...)

I changed it to cut off these extra bytes "return plaintext[32:].decode()" and it stopped failing (for all the cookies).

Hopefully this clue leads to someone solving the issue without hacking around it.

@ownrepo1
Copy link

Maybe check the sources of Chromium?

@digitalsignalperson
Copy link

the hack works here

chromium/chromium@129.0.6668.58...130.0.6723.58 has a bazillion changes. might need to bisect unless knowing where to look

just looking at the diff between those tags searching for keywords like AES CBC, may be some hints in various changes in chrome/browser/ash/platform_keys/platform_keys_service_* files. Like in platform_keys_service_nss.cc

-// Default signature length for signing with symmetric keys.
+// Default constants for symmetric keys.
+const int kDefaultSymKeySize = 32;
 const int kDefaultSymSignatureLength = 32;

that was just replacing a hard coded value, but maybe the 32 bytes is a key or signature idk

@abhranil26
Copy link

happening same with gallery-dl too!
I am on mac 15.0.1 (24A348) and latest chrome stable

@Tatsh
Copy link
Contributor

Tatsh commented Oct 28, 2024

I fixed it by hacking around the problem.

I tracked the "source" of this message:

WARNING: failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?

The decryption of cookies is decoding the decrypted "plaintext" using UTF-8. And for some reason, I didn't yet track why exactly, there are 32 extra bytes of garbage at the beginning of the string. As others noted, this started very recently.

https://github.com/yt-dlp/yt-dlp/blob/2a24674/yt_dlp/cookies.py#L1013-L1021

The "return plaintext.decode()" is failing. After comparing values from different steps of the process against my actual cookies, I noticed there were simply extra bytes (garbage? idk...)

I changed it to cut off these extra bytes "return plaintext[32:].decode()" and it stopped failing (for all the cookies).

Hopefully this clue leads to someone solving the issue without hacking around it.

This works great! I think a PR should be made with this change as it will keep BC:

diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
index 4a69c576b..5a5dbe941 100644
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -1016,7 +1016,10 @@ def _decrypt_aes_cbc_multi(ciphertext, keys, logger, initialization_vector=b' '
         try:
             return plaintext.decode()
         except UnicodeDecodeError:
-            pass
+            try:
+                return plaintext[32:].decode()
+            except UnicodeDecodeError:
+                pass
     logger.warning('failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
     return None

@fharper
Copy link

fharper commented Nov 1, 2024

This change is working well for me also, thanks 🎉

@bashonly
Copy link
Member

bashonly commented Nov 1, 2024

@kesor @Tatsh do one of you want to open a PR with that patch?

@mbway
Copy link
Contributor

mbway commented Nov 1, 2024

Backwards compatibility might be an issue since it will break for older versions and browsers using an older chromium base

@bashonly
Copy link
Member

bashonly commented Nov 1, 2024

@mbway If we try first with the full plaintext before slicing, that should be backwards compatible, yeah?

diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
index 4a69c576b..6ecc354cc 100644
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -1013,10 +1013,11 @@ def pbkdf2_sha1(password, salt, iterations, key_length):
 def _decrypt_aes_cbc_multi(ciphertext, keys, logger, initialization_vector=b' ' * 16):
     for key in keys:
         plaintext = unpad_pkcs7(aes_cbc_decrypt_bytes(ciphertext, key, initialization_vector))
-        try:
-            return plaintext.decode()
-        except UnicodeDecodeError:
-            pass
+        for cookie in (plaintext, plaintext[32:]):
+            try:
+                return cookie.decode()
+            except UnicodeDecodeError:
+                pass
     logger.warning('failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
     return None
 

@bashonly bashonly added the external issue Issue with an external tool or site label Nov 4, 2024
@bashonly
Copy link
Member

bashonly commented Nov 4, 2024

The issue that was being discussed since 2024.10.01 was resolved by 4613096, and this issue has been closed.

As mbway said upthread though, the original post in this thread was about a different issue with merely the same symptom. Since mbway could not reproduce the issue, and since Despruk found a workaround for an external issue that may have been the same thing as the OP was experiencing, I am keeping this issue closed for now.

If the original author of this issue or anyone else experiences a problem similar to the original post using the latest version, then they can share a new complete verbose log and this issue can be reopened.

@bashonly bashonly changed the title Extracted 245 cookies from chromium (72 could not be decrypted). failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong? Linux/macOS/BSD chromium cookies could not be decrypted; failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed Nov 4, 2024
@coletdjnz coletdjnz added the core:cookies Related to core cookie handling (e.g. --cookies or --cookies-from-browser) label Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug that is not site-specific core:cookies Related to core cookie handling (e.g. --cookies or --cookies-from-browser) external issue Issue with an external tool or site
Projects
None yet
Development

Successfully merging a pull request may close this issue.