Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CipherError #23

Closed
devistest opened this issue Jun 15, 2021 · 26 comments
Closed

CipherError #23

devistest opened this issue Jun 15, 2021 · 26 comments
Labels

Comments

@devistest
Copy link

I'm getting an error while parsing the notes from my backup. The password is correct in the file, and all unencrypted notes can be displayed (tho any with images do not). The error for the cipher is:

Apple Decrypter: Apple Note: 251 Attempting decryption.
Apple Decrypter: Apple Note: 251 caught a CipherError while trying final decrypt, likely the unwrapped key is not correct.

Could be user error, but I'm not sure what I'm doing incorrectly.

@threeplanetssoftware
Copy link
Owner

To make sure I understand, all of the notes that were encrypted threw a CipherError consistently? That would most likely be the password being incorrect. You might want to check for any additional whitespace on the line, remove any quotes if you have it quoted, etc. It should just be the bare password on a line.

For example, this is my test:

[notta@cuppa apple_cloud_notes_parser]$ cat passwords.txt
password
[notta@cuppa apple_cloud_notes_parser]$ ruby notes_cloud_ripper.rb -i ~/iphone/ -w passwords.txt --show-password-successes
...
------------------------------
Successfully decrypted notes using passwords: password
These are NOT logged, note it down now if you need it.
------------------------------

The images would likely be a separate issue, were you running this just from a NoteStore.sqlite file, or did you point it to a backup of some sort (Mac or iDevice)?

@devistest
Copy link
Author

devistest commented Jun 16, 2021

unfortunately, yes, I've made sure this is the correct password and there is no whitespace. I can copy/paste this into the icloud site and unlock the notes, but it fails still.

Failed to decrypt Apple Note: 251, unwrapped key likely isn't right.
Skipping Note ID 270 due to a missing folder or account, check the debug log for more details.
Updated AppleNoteStore object with 16 AppleNotes in 3 folders belonging to 1 accounts.
Updated AppleNoteStore object with 0 AppleNotes in 1 folders belonging to 1 accounts.
Adding the ZICNOTEDATA.ZPLAINTEXT and ZICNOTEDATA.ZDECOMPRESSEDDATA columns, this takes a few seconds
------------------------------
Successfully decrypted notes using passwords: <<REDACTED>>
These are NOT logged, note it down now if you need it.
------------------------------

As for the images, I've both from just NoteStore.sqlite and the full backup with the same results.

@threeplanetssoftware
Copy link
Owner

What I find interesting is that the log appears to indicate that you did successfully open at least one locked object. Are you able to see anything in the debug_log.txt file that indicates which one was successful? Maybe looking at that Note will help explain why the other fails.

[notta@cuppa apple_cloud_notes_parser]$ cat output/2021_06_15-22_17_07/debug_log.txt | grep "generated a decrypt" 
D, [2021-06-15T22:17:07.728370 #407832] DEBUG -- : Apple Decrypter: Apple Note: 65 generated a decrypt
D, [2021-06-15T22:17:07.740646 #407832] DEBUG -- : Apple Decrypter: Apple Note: 77 generated a decrypt
D, [2021-06-15T22:17:07.747491 #407832] DEBUG -- : Apple Decrypter: Apple Backup encrypted file generated a decrypt
D, [2021-06-15T22:17:07.754231 #407832] DEBUG -- : Apple Decrypter: Apple Backup encrypted file generated a decrypt

I'd also be interested to make sure that note was not deleted. I see there are only 16 at you're up at number 251, which makes me think there is some turnover and I have observed odd behaviors with notes after they have passed the time period wherein garbage collection should occur. These two SQLite queries might be interesting, especially if you look at the difference between whatever did open up and note 251. The first of which I would be making sure the result isn't NULL and the second of which I would be making sure there are values in each column (which there should be since we didn't fail to generate the key):

SELECT ZDATA FROM ZICNOTEDATA WHERE ZNOTE=251
SELECT ZCRYPTOSALT, ZCRYPTOTAG, ZCRYPTOWRAPPEDKEY, ZCRYPTOINITIALIZATIONVECTOR 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE Z_PK=251

Finally, if you wanted to check on what this code is doing, I have code in this blog entry that explains each step of the process and you could plug in the results of the above two SQLite queries to make sure something isn't just broken in this program without having to give me any of that information.

@devistest
Copy link
Author

so for the images, I think my issue has something to do with this from the debug_log:

Note 46: Created a new Embedded Object of type public.jpeg
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil
AppleNotesEmbeddedPublicJpeg 40DF4BF7-B927-4C5D-B491-6497076749FA: Filename is IMG_1158.JPG
Can't call back_up_file with filepath_on_disk that is nil
AppleNote: Note 46 replacing attachment 5556EA59-CD84-4303-9168-255EBEC3F955
Note 46: Created a new Embedded Object of type public.heic
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil

The note was definitely not deleted, actually there are 3 locked notes in the db and none of them are recovered.
I'm beginning to think maybe there is an issue with the database....

sqlite> SELECT ZDATA FROM ZICNOTEDATA WHERE ZNOTE=251;
 ▒▒۷▒9▒kQ-?▒d▒▒&h▒
▒oC
   B▒▒
4𲨟Q▒a▒U/▒@q▒▒&▒Xe"▒^
▒$▒VL▒6>6=▒▒▒&▒`^3F"▒▒J▒!GXV1▒▒[Ƞ=;▒ҏ$-!▒
                                         ▒s▒~h▒▒▒q`<▒~MQW▒▒1▒㧡▒▒c▒▒P y_lgq▒hõ▒▒▒T▒▒I▒F▒▒

@threeplanetssoftware
Copy link
Owner

threeplanetssoftware commented Jun 16, 2021

Thanks, those are interesting observations. So the note clearly has data, clearly is not being decrypted, but (based on the first log) something was or the password should not have been listed at the end. Because you are seeing the Can't call back_up_file with filepath_on_disk that is nil errors it clearly is a backup that isn't just a single file. May I ask what type (Mac, iTunes, or Physical) and if it is a full backup or just partial?

If you go to the folder you are pointing to, can you try to find one of the files that this is claiming can't be found? From your snippet, I would suggest:

find [folder you pointed at, ~/iphone/ in my example] -name 40DF4BF7-B927-4C5D-B491-6497076749FA

This won't work directly if you're looking at an iTunes backup, wherein the filenames are hashed. In that case you would open the Manifest.db file in the folder you pointed to (~/iphone/Manifest.db for me) and run this query, then use the fileID result in your find command:

SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%40DF4BF7-B927-4C5D-B491-6497076749FA%'

@threeplanetssoftware threeplanetssoftware added the need-to-reproduce For things that can't yet be reliably reproduced in testing label Jun 16, 2021
@devistest
Copy link
Author

It is an itunes backup (should be full), here is the query result on Manifest.db

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%40DF4BF7-B927-4C5D-B491-6497076749FA%';
bd6085165d412ccd822be2ebd99d0105de49173f|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-288x384-0.png
9807ae55b59a499c85d49da30f8ce391e390b370|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-288x384-0-oriented.png
6f9f363abaf7171f81b7d8f658644c2aae487fb4|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-144x192-0.png
de82a993755cd24e65a4633f62508364e6070819|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA
5abcbffbc86fed35eb9dd96c3336fc168d4764c8|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA/1593200297
624a906d4f313ab0ce6629fbb135bdd8963bc4ca|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA/1593200297/Electrical Hazards.JPG

@threeplanetssoftware
Copy link
Owner

Thanks, so the files should be there. What happens when you try to look at one, such as:

find [your path, such as ~/iphone/ in my example] -name bd6085165d412ccd822be2ebd99d0105de49173f
find [your path] -name 9807ae55b59a499c85d49da30f8ce391e390b370
find [your path] -name 6f9f363abaf7171f81b7d8f658644c2aae487fb4

Also, it would be worth looking for the original filename in Manifest.db, and doing the same sort of find command on the resulting fileID:
SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%IMG_1158.JPG%'

Thanks for helping to troubleshoot this!

@devistest
Copy link
Author

I can search for and find all of these ids in the files and can even view the images by pulling them into VLC player.

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%IMG_1158  .JPG%';
aa9682b53a7ca7b73ef8cedfcec22fb01cb81a18|Media/C07A1BC9-3E92-4C0D-942A-ED20E8F9B  ECB/IMG_1158.JPG
32cfae382e8f193c5602bdf64091251c5297ad1e|Media/PhotoData/Thumbnails/V2/DCIM/101A  PPLE/IMG_1158.JPG
998dbb485313f4543740759529c25bb676defb11|Media/PhotoData/Thumbnails/V2/DCIM/101A  PPLE/IMG_1158.JPG/5005.JPG
d39e2a9e5ffd4f4ca8ed388311d6570ffbf28e50|Media/DCIM/101APPLE/IMG_1158.JPG

I searched for these files and was able to view them as well.

@threeplanetssoftware
Copy link
Owner

threeplanetssoftware commented Jun 17, 2021

Thanks for working through this, it is really helpful and I'm getting a much better idea of what the problem is. If you look at the source for AppleNotesEmbeddedPublicJpeg, this is the function that gets the filepath we search for the media:

def get_media_filepath
return "Accounts/#{@note.account.identifier}/Media/#{get_media_uuid}/#{get_media_uuid}" if @is_password_protected
return "Accounts/#{@note.account.identifier}/Media/#{get_media_uuid}/#{@filename}"
end

You'll notice it is expecting Accounts/[UUID]/Media/[UUID]/[filename]. For example, a random one in my test data is

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE "%D533089A-83BF-44B1-A515-5DB4E157A1F2%";
2251e845879d544d85d866357797ec28b0274360|Accounts/D34714F2-F2F7-4AD0-8EA5-A54E31A74D72/Media/D533089A-83BF-44B1-A515-5DB4E157A1F2
41da368bed81d5061bb71f91828610b2221daaa4|Accounts/D34714F2-F2F7-4AD0-8EA5-A54E31A74D72/Media/D533089A-83BF-44B1-A515-5DB4E157A1F2/IMG_0014.png

However, in your data, the Account's UUID is not present and the path is starting down at the Media folder, so when get_media_filepath computes the filepath, it will be looking for something that doesn't exist. That is likely what is causing us to hit line 78 in this function, it fails to pass the File.exist? check:

def back_up_file(filepath_on_phone, filename_on_phone, filepath_on_disk,
is_password_protected=false, password=nil, salt=nil, iterations=nil, key=nil,
iv=nil, tag=nil, debug_text=nil)
if !filepath_on_disk or !File.exist?(filepath_on_disk)
@logger.error("Can't call back_up_file with filepath_on_disk that is nil") if @type != SINGLE_FILE_BACKUP_TYPE
return
end

So I think I understand what is happening, but I don't understand why. Can you tell me which version of iOS you are running and the type of device? Also, you mentioned iCloud, are these notes created on the iDevice, on your Mac, or in a browser? I hope those details will let me zero in on being able to recreate it myself.

Edit: One other thing to make sure, include the domain in your query against Manifest.db. Some of your entries are likely the CameraRollDomain and not what Notes uses, AppDomainGroup-group.com.apple.notes.

@devistest
Copy link
Author

No, thanks for working with me on this. I'll happily see if we can iron this out. The device is an iphone 11 running 14.6 iOS. The notes were all created on this iphone. I mentioned iCloud because it was the easiest way to copy/paste the password to ensure it was correct.

@threeplanetssoftware
Copy link
Owner

I haven't yet reproduced this, but I have seen at least one oddity when running this on my Mac. Could you try running this on the actual Mac version of Notes and see if it gives you the same errors as you get from the iTunes backup?

ruby notes_cloud_ripper.rb -m ~/Library/Application\ Support/MobileSync/Backup/[device UDID] -w passwords.txt --show-password-successes

What I'm finding is I have one note that I can unlock on my phone with the right password as all the other notes which does not unlock when I run the Mac version. When I run the iTunes backup, it opens just fine. As yet undecided about exactly why, but I'm curious to see how reliably reproducible yours is in your environment. Just rmember the Note IDs will change, my problem child (117) became 84 on the Mac.

@devistest
Copy link
Author

I don't have a mac to run this on, I'm running on a linux box.

@threeplanetssoftware
Copy link
Owner

Oh, sorry, I made an assumption based on the iTunes backup. How are you getting the iTunes rip? My test device is an "older" iPhone on iOS 14.6 and I'm ripping it down to a Mac running Big Sur. Then I check the results on my Linux test environment and the Mac, both the iPhones version and Mac version. As I said, on Mac I see a potential discrepancy, but not sure it is your issue or not.

@devistest
Copy link
Author

well the rip is being done on windows, but I'm doing the work with your script on my linux box

@threeplanetssoftware
Copy link
Owner

threeplanetssoftware commented Jun 18, 2021

I have a theory that this might be related to how and when the iCloud changes sync. I have been able to find a case on my Mac backup that seems to be the same as yours and believe it is tied to this code:

# Set values initially from the expected columns
crypto_iv = row["ZCRYPTOINITIALIZATIONVECTOR"]
crypto_tag = row["ZCRYPTOTAG"]
crypto_salt = row["ZCRYPTOSALT"]
crypto_iterations = row["ZCRYPTOITERATIONCOUNT"]
crypto_verifier = row["ZCRYPTOVERIFIER"]
crypto_wrapped_key = row["ZCRYPTOWRAPPEDKEY"]
# If they aren't there, we need to use the ZUNAPPLIEDENCRYPTEDRECORD
if row["ZUNAPPLIEDENCRYPTEDRECORD"]
keyed_archive = KeyedArchive.new(:data => row["ZUNAPPLIEDENCRYPTEDRECORD"])
unpacked_top = keyed_archive.unpacked_top()
ns_keys = unpacked_top["root"]["ValueStore"]["RecordValues"]["NS.keys"]
ns_values = unpacked_top["root"]["ValueStore"]["RecordValues"]["NS.objects"]
crypto_iv = ns_values[ns_keys.index("CryptoInitializationVector")]
crypto_tag = ns_values[ns_keys.index("CryptoTag")]
crypto_salt = ns_values[ns_keys.index("CryptoSalt")]
crypto_iterations = ns_values[ns_keys.index("CryptoIterationCount")]
crypto_key = ns_values[ns_keys.index("CryptoWrappedKey")]
end
tmp_note.add_cryptographic_settings(crypto_iv,
crypto_tag,
crypto_salt,
crypto_iterations,
crypto_verifier,
crypto_wrapped_key)

Ignoring the obvious bug of crypto_wrapped_key v crypto_key which is now fixed in the main branch, when I make the if block always fail the note of mine that won't decrypt, decrypts. It seems there are values "stuck" in the ZUNAPPLIEDENCRYPTEDRECORD and even after trying to force the phone and my Mac to sync a few times, including unlocking that note, it persists.

Would you please confirm there is a value in ZUNAPPLIEDENCRYPTEDRECORD on your problematic note? Please run this (note that I'm using "length" to avoid printing the actual contents if you want to compare the values in the two tables you could remove the length and look at the hex to compare):

SELECT ZICCLOUDSYNCINGOBJECT.Z_PK as NoteID,
	   length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOSALT)) as CloudSalt,
	   length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOTAG)) as CloudTag, 
	   length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOWRAPPEDKEY)) as CloudUnwrappedKey, 
	   length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOINITIALIZATIONVECTOR)) as CloudIV,
	   length(hex(ZICCLOUDSYNCINGOBJECT.ZUNAPPLIEDENCRYPTEDRECORD)) as CloudUnapplied,
	   length(hex(ZICNOTEDATA.ZCRYPTOINITIALIZATIONVECTOR)) as NoteIV,
	   length(hex(ZICNOTEDATA.ZCRYPTOTAG)) as NoteTag
FROM ZICCLOUDSYNCINGOBJECT, ZICNOTEDATA
WHERE ZICCLOUDSYNCINGOBJECT.ZUNAPPLIEDENCRYPTEDRECORD is not NULL AND ZICNOTEDATA.ZNOTE=ZICCLOUDSYNCINGOBJECT.Z_PK

In my case, I have the note I'm having issues with (117), as well as two others that I don't have issues with, likely because what is in the ZUNAPPLIEDENCRYPTEDRECORD is properly synced on notes 11 and 22. All three of these same notes are just fine on my iTunes backup.

NoteID CloudSalt CloudTag CloudUnwrappedKey CloudIV CloudUnapplied NoteIV NoteTag
11 32 32 48 32 9956 32 32
22 32 32 48 32 9790 32 32
117 32 32 48 32 9634 32 32

I need to think about the best way to catch this case, but would appreciate your confirmation that this is likely the right direction by running the above SQLite query.

@devistest
Copy link
Author

Yeah there is definitely data there:

66|32|32|48|32|9960|32|32
108|32|32|48|32|9614|32|32
109|32|32|48|32|9646|32|32

The contents without the length parameter included is massive.

@threeplanetssoftware
Copy link
Owner

What's interesting about that output is I don't see Note 251 (just 66, 108, and 109), the one you have with the CipherError listed in the original report, included. That would make me think it is not actually the right track. As a bit of a long shot, could you check out branch 21-wal-copy and see if that makes any difference at all?

@devistest
Copy link
Author

ran that version, and the result looks identical

@threeplanetssoftware
Copy link
Owner

Ok, well let's see if this works. I just pushed branch 23-bug-encryption-debug (7849da1), which simply bypasses the ZUNAPPLIEDENCRYPTEDRECORD field. Based on your output I don't think it should fix it, but please try and let me know if there are any differences.

@devistest
Copy link
Author

some success!... no longer get the encryption issue, but still no love on the images due to the Can't call back_up_file with filepath_on_disk that is nil error.

@threeplanetssoftware threeplanetssoftware added bug and removed need-to-reproduce For things that can't yet be reliably reproduced in testing labels Jun 18, 2021
@threeplanetssoftware
Copy link
Owner

Ok, in that case this is a bug that I can reproduce and I'll work on a solution. I have broken the images off into a separate issue because it is unrelated to the CipherError that spawned this ticket and I'd prefer to clearly track the distinct problems. Otherwise things get lost.

@devistest
Copy link
Author

understood, appreciate the help!

@threeplanetssoftware
Copy link
Owner

This isn't as quick of a change because the case I think I have to catch means that there are two sets of "valid" settings, as far as the database is concerned. Those in ZUNAPPLIEDENCRYPTEDRECORD and those in the database row itself. I've added a bit more smarts to how I handle that in 17b340a on the 23-bug-encryption-debug branch and pulled in the other changes.

This is a pure stopgap for you, not intended as production code and it may work for you, or this may continue to not work. If it does not work, I'd suggest looking at the files changed in the listed commit and try changing back to and false the lines I expanded. The reason I'm not doing that as a solution is in my test database I had objects that had the right settings in the ZUNAPPLIEDENCRYPTEDRECORD and some that did not, so I could open the note, but then not the object within the note. You may run into the same issue until I solve this a bit more permanently.

@devistest
Copy link
Author

Excellent. You're a saint for patiently delving into this niche case for me that will probably never happen to anyone else. I was able to recover everything.

@threeplanetssoftware
Copy link
Owner

That's great to hear! I will probably take a bit to figure out the best method to actually solve this. My hope is I can compare the timestamps on the ZUNAPPLIEDENCRYPTEDRECORD and not just try two sets of variables every time.

@threeplanetssoftware
Copy link
Owner

Sorry for the delay in a real solution to this issue. I just pushed d4a0d86 which will now check if a given set of cryptographic settings is correct when multiple are available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants