gphotos import + albums migration #1340
Replies: 36 comments 83 replies
-
|
A comparison for takeout helper tools: mattwilson1024/google-photos-exif:
https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper v3
https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper v2
https://github.com/m1rkwood/google-photos-takeout-scripts
https://github.com/laurentlbm/google-photos-takeout-date-fixer
Prior to running this script the file |
Beta Was this translation helpful? Give feedback.
-
|
May I mention my own project: https://github.com/simulot/immich-go
|
Beta Was this translation helpful? Give feedback.
-
|
This script is currently broken. Will un-strike-out when it's fixed. https://github.com/lukehmcc/simple-gphotos-immich-migration The script really is quite simple, just a combination of a lot of work that's been done here so far. It utilizes a forked version of MrYakobo/exif-wrapper which utilizes the new and improved https://github.com/garzj/google-photos-migrate over mattwilson1024/google-photos-exif. |
Beta Was this translation helpful? Give feedback.
-
|
is your script working now mate? |
Beta Was this translation helpful? Give feedback.
-
|
As the README says, the functionality has been moved to google-photos-migrate. This was completed a couple months ago. I still have plans to integrate this library into immich-cli, but have been time strapped. Happy migrating! |
Beta Was this translation helpful? Give feedback.
-
|
As I import my last remaining user, here are some notes: https://github.com/starmessage/CrcCheckCopy is useful if your user isn't very organized and has some old copies lying around for some reason, and has duplicates, but one folder is newer than the other, or some files are corrupted because they 😱 keep buying SD cards to store photos instead of moving them over to a hard drive https://github.com/mrishab/google-photos-delete-tool/blob/master/delete_photos.js is useful to delete all gphotos after you're done migrating |
Beta Was this translation helpful? Give feedback.
-
|
About step 5, running So, all my "unorganized" photos just end up in those "Photos from [Year]" albums, I guess! Update: For anyone reading this in the future, upload the "Photos from..." albums last. Any album that starts alphabetically later than "Photos..." will be messed up! |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that seems to fix it. This could be a bit of manual work... The date is encoded in the filename, so I will probably remove the prefix of these
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @Snuupy! I'll check out that utility.
|
Beta Was this translation helpful? Give feedback.
-
|
What's in the json file that it doesn't apply the date correctly? Strange. |
Beta Was this translation helpful? Give feedback.
-
|
It's pretty regular looking: {
"title": "PXL_20211210_001012327.NIGHT.mp4",
"description": "",
"imageViews": "18",
"creationTime": {
"timestamp": "1639095244",
"formatted": "Dec 10, 2021, 12:14:04 AM UTC"
},
"photoTakenTime": {
"timestamp": "1639095231",
"formatted": "Dec 10, 2021, 12:13:51 AM UTC"
},
"geoData": {
"redacted"
},
"geoDataExif": {
"redacted"
},
"url": "redacted",
"googlePhotosOrigin": {
"mobileUpload": {
"deviceFolder": {
"localFolderName": ""
},
"deviceType": "ANDROID_PHONE"
}
},
"photoLastModifiedTime": {
"timestamp": "1660935461",
"formatted": "Aug 19, 2022, 6:57:41 PM UTC"
}
}My best guess is the way that my Pixel 6 takes these pictures. This one is a "astrophotography" mode picture that is processed into a timelapse video shortly after it's finished processing the main photo. Based on the 113 other files in this album with the same issue, I believe that the Google Camera app is writing a bunch of zeros in the metadata field, or, the Takeout process adds them in there, but only to these processed videos. |
Beta Was this translation helpful? Give feedback.
-
|
That's certainly strange but I don't have the brainpower to work through why this is the case right now. Happy that you're able to set it manually instead in the meantime. Good thing we only have to do the migration once 😅 |
Beta Was this translation helpful? Give feedback.
-
|
I'm trying to run this but running in to the same error regardless of whether I run
|
Beta Was this translation helpful? Give feedback.
-
|
run |
Beta Was this translation helpful? Give feedback.
-
|
Thanks very much for this, it got me through it all. After running the exif fixes and the import (used the docker run version of the import as I was performing it from the same device as the server) I found that ffmpeg was running full throttle long after I had finished - 80-90% core utilization - and I had no idea why. Is this this anything you encountered? |
Beta Was this translation helpful? Give feedback.
-
|
Go to If there are files still being processed due to the import process queue then of course it will still be using high CPU. Just because Immich CLI has completed does not mean the files are finished processing. You can run |
Beta Was this translation helpful? Give feedback.
-
|
This is a project I have not yet seen: https://github.com/gilesknap/gphotos-sync |
Beta Was this translation helpful? Give feedback.
-
|
read: https://github.com/gilesknap/gphotos-sync#warning-google-api-issues |
Beta Was this translation helpful? Give feedback.
-
|
Ah yeah that'll do it |
Beta Was this translation helpful? Give feedback.
-
|
I wrote a python script that codifies steps 1-6, so you don't have to do it manually. https://github.com/MrYakobo/exif-wrapper After running it, inspect the error folders for debugging. |
Beta Was this translation helpful? Give feedback.
-
|
It seems like you're using @lukehmcc fork; please follow the instructions from his readme... 👍 |
Beta Was this translation helpful? Give feedback.
-
|
There is an issue in the fork where replacing a submodule didn't end up in git completely. https://github.com/jeverling/exif-wrapper Just to see if that works better for you. |
Beta Was this translation helpful? Give feedback.
-
|
@jeverling Tried it, got this error: |
Beta Was this translation helpful? Give feedback.
-
|
I'd appreciate a PR when any issues are found (: |
Beta Was this translation helpful? Give feedback.
-
|
Before trying to use @jeverling's fix, make sure to delete old docker containers + images: Seems obvious but it took me a while before i remembered i needed to rebuild the image. |
Beta Was this translation helpful? Give feedback.
-
|
It looks like this doesn't quite work specifically for a Pixel 7. My guess is it effects any other Pixels that do all of the image post-processing. Looking into the code of the What this means is: after uploading all of the images to immich, when trying to sync with existing files in the mobile app it will see the two files as "different". When doing a backup, it doesn't upload those files (assumingly because they have the same name), but you will see a local and remote copy of every image taken by the Pixel 7. Trying to work through how to resolve this currently |
Beta Was this translation helpful? Give feedback.
-
|
Update: I was able to come up with a solution. Basically, since server hashing is working properly, I ended up deleting every image taken since getting the Pixel 7 from the server (it was about 1200 images). After some uninstalling/reinstalling and clearing memory in the app, I was able to then back them all up from the phone directly. After that, everything was synced with no duplicates. After that, I used the CLI steps to only upload the albums, which I still had saved, and that linked the now-correctly-uploaded items to the proper albums. |
Beta Was this translation helpful? Give feedback.
-
|
Want to add on to this. So after running MrYakobo's lovely python script. (I couldn't get the docker container working, but just using the script by itself as the Dockerfile describes works great). There's a good way to optimize uploading the albums. BE CAREFUL HERE. You have to check that the python script was successful before using a script to automate uploads to make sure that you don't lose files. An easy way to do this is by running the command So it is safe to move forward if the find command looks like this: Now for the goods. #!/bin/bash
KEY="YOURKEYHERE"
SERVER="https://photos.yourdomain.com/api"
BASE_DIR="/foo/bar/Takeout/Google Photos/AlbumsProcessed"
# Change the working directory to BASE_DIR
cd "$BASE_DIR"
for album in "$BASE_DIR"/*; do
if [ -d "$album" ]; then
echo "Uploading album: $album"
immich upload \
-y \
--key "$KEY" \
--server "$SERVER" \
-d "$album" -t 16 -al
fi
doneVoila! You can skip manually uploading every album! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks this guide is awesome! By the way, here is a convenient way to do the albums manually when you have a lot of them. for each album
|
Beta Was this translation helpful? Give feedback.
-
|
This process worked really well for me! Out of about 60,000 photos/videos, about 200 of them got upload as today's date and 15 of them (all being videos) got upload as Jan 1, 2036. That means over 59,500 photos/videos were successfully transferred from Google photos while retaining the correct dates. The 200 listed as today's date doesn't bother me too much, but the 15 listed as Jan 1, 2036 are going to be really annoying because they'll stay up at the top of the photo feed until 2036. Does anyone know how I can manually fix their dates? Should I just download them, delete them from Immich, use some tool to modify their created date, then re-upload? Btw I used the docker version of the Immich CLI when importing. |
Beta Was this translation helpful? Give feedback.
-
That is probably the best approach, yes. I recommend |
Beta Was this translation helpful? Give feedback.
-
|
Wow thanks for that software recommendation. |
Beta Was this translation helpful? Give feedback.
-
|
google-photos-exif didn't quite do the job for me, so I wrote my own little tool google-photos-migrate, but use at your own risk |
Beta Was this translation helpful? Give feedback.
-
|
Tried your scripts and it works like a charm and also fixes many extension error of my pictures, thanks a lot for your work! |
Beta Was this translation helpful? Give feedback.
-
|
lol |
Beta Was this translation helpful? Give feedback.
-
|
Maybe related to this issue garzj/google-photos-migrate#3 |
Beta Was this translation helpful? Give feedback.
-
|
For those who are lazy, I used some Python snippets that others can use in A little setup: import os
import subprocessingStep 4
takeout = "/mnt/f/user google takeout/takeout"
albums = next(os.walk(os.path.join(takeout, "Albums/")))[1]
for album in albums:
subprocess.run(["yarn", "start", "--inputDir", os.path.join(takeout, "Albums/", album), "--outputDir", os.path.join(takeout, "AlbumsProcessed/", album), "--errorDir", os.path.join(takeout, "AlbumsError/", album)])Step 7
takeout = "/mnt/f/user google takeout/takeout"
albums = next(os.walk(os.path.join(takeout, "AlbumsProcessed/")))[1]
key = "APIKEY"
server = "https://immich.domain.tld/api"
for album in albums:
subprocess.run(["immich", "upload", "--yes", "--key", key, "--server", server, "-t", "16", "-al", "--recursive", os.path.join(takeout, "AlbumsProcessed/", album)]) |
Beta Was this translation helpful? Give feedback.
-
|
Here's @lukehmcc's #1340 (comment) script for those using Docker that creates the albums and then uploads the photos |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the script , unfortunately there are some updates to immich-cli image and the above script does not work as expected for me with the below changed script: Does anyone know the correct upload command or please share the documentation for upload the photos using immich-cli. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Figured out the correct command from other post, Incase if anyone is looking for an update: docker run -it --rm -v "/mnt/hdd2_tshb_abd/gtakeout_folder/albums/:/import" -e IMMICH_INSTANCE_URL=$SERVER -e IMMICH_API_KEY=$API_KEY -e IMMICH_RECURSIVE=true -e IMMICH_AUTO_CREATE_ALBUM=true ghcr.io/immich-app/immich-cli:latest upload -r /import |
Beta Was this translation helpful? Give feedback.
-
|
This script is currently broken, working on fixing it. When this command is run it prompts the user for the API address, API key, base dir (where the takeout zips are), and docker dir (a directory that can safely run docker containers via compose, as you cannot do this on cifs mounts). Once those values are input it unzips, cleans metadata, and automatically uploads all albums and photos. Here is the repo(lukehmcc/simple-gphotos-immich-migration). The script really is quite simple, just a combination of a lot of work that's been done here so far. It utilizes a forked version of MrYakobo/exif-wrapper which utilizes the new and improved https://github.com/garzj/google-photos-migrate over mattwilson1024/google-photos-exif. Lmk where you guys would like to bring this next. Could this be something that could be implemented into immich directly? If so how do you guys envision that? |
Beta Was this translation helpful? Give feedback.
-
|
I got this error: It only did this for one album. It then went on to connect to my Immich server and supposedly upload everything, but when I looked there was no content uploaded (albums or photos): |
Beta Was this translation helpful? Give feedback.
-
|
I found the issue and am working on it. Sorry for late reply. |
Beta Was this translation helpful? Give feedback.
-
|
When you download the albums, do you do this manually by going to google photos and clicking |
Beta Was this translation helpful? Give feedback.
-
|
The first. I wasn't sure if the takeout zips had the same album hierarchy but they might. |
Beta Was this translation helpful? Give feedback.
-
|
I get a similar error and the script exits. All takeout zip files were unzipped and folders updated… that’s as far as it got. @lukehmcc, any thoughts or suggestions to fix? |
Beta Was this translation helpful? Give feedback.
-
|
The script was humming along until this happened: I started it up again but it errored out with the same message on a different album. Do I need to pass any parameters when running the sudo docker compose up ? |
Beta Was this translation helpful? Give feedback.
-
|
afaik no, but you should clear out your output and error dirs first |
Beta Was this translation helpful? Give feedback.
-
|
I do see the yarn.lock file |
Beta Was this translation helpful? Give feedback.
-
yes, after you re-cloned the repo it was likely not there before |
Beta Was this translation helpful? Give feedback.
-
|
See, this is the problem with AIO (all-in-one) scripts, people don't know how to fix things when they hit an error because they need to do it the manual way first. The AIO scripts are generally not robust unless they've been tried/tested by many people (one is not sufficient). If you do it the manual way first, you can ensure each step is done correctly before proceeding onto the next, fixing any errors and knowing where the error stemmed from instead of playing guess the error. |
Beta Was this translation helpful? Give feedback.
-
What do you mean? Should I be deleting subfolders of AlbumsProcessed and AlbumsError that are empty?
|
Beta Was this translation helpful? Give feedback.
-
Yes. No files nor folders should exist under |
Beta Was this translation helpful? Give feedback.
-
|
Got it... and I'm starting to appreciate your advice of going back to your initial post and doing each step one at a time. Before I go that route, I'd like to try the AIO script one more time. So, even though some of the albums processed the first time I ran the script (I see just the image files in the AlbumsProcessed album folders), your saying I should just delete all of them and start again? |
Beta Was this translation helpful? Give feedback.
-
|
unless you're able to resume it from the point of failure instead of starting it fresh again |
Beta Was this translation helpful? Give feedback.
-
|
I must be stupid. I'm trying to do step 4 and I just get this output: I'm running the command in the google-photos-exif cloned repo directory. Solution: I had to install |
Beta Was this translation helpful? Give feedback.
-
|
what is the command you're using? should be |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
are you running the command from the working directory you cloned mattwilson1024/google-photos-exif into? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. I googled the error and am currently installing npm |
Beta Was this translation helpful? Give feedback.
-
|
Figured it out and edited my reply with the solution. |
Beta Was this translation helpful? Give feedback.
-
|
On step 4 I get this response even though there are json files that correspond to each media file and a Edit: I uploaded the album to the server just to test if everything had the correct date and it did. Not sure why but I'll take it. |
Beta Was this translation helpful? Give feedback.
-
Yes, all your files already have existing EXIF data. There is no need for the tool to add them. If you have some files from other devices (say, older phones/cameras/devices) that did NOT have EXIF data, this will add that. For me I had ~300+ which would show up at the beginning of the Immich timeline so things would be out of order. However, if you didn't run this tool, you would not know this for sure/guarantee the state of the files. When this tool traverses a file that already has EXIF data, it will not touch it (this is a good thing). If it is missing, it will read from the json file to see what dates to add to it so it fixes its position in the timeline. This tool does not correct for files that have the wrong EXIF date data. For example, I had 1 file that was malformed and had some date in the future (like the year 2060 or something). That one I had to correct myself manually and was immediately obvious because it showed up at the top of the immich timeline (lol)
Exactly. If you get
Explanation above should cover it. I'm glad you figured it out! :) |
Beta Was this translation helpful? Give feedback.
-
|
Any idea why some of my Google Photos didn't come with json files and don't have a DateTimeOriginal field but show up with the correct data on Google Photos? Should I request another takeout? |
Beta Was this translation helpful? Give feedback.
-
|
Apparently your archive is big, and it should be split into several zip files, like for mine. I found some JSON in another zip than the jpg. |
Beta Was this translation helpful? Give feedback.
-
|
Ah yup you're right. I'm just going to combine the two takeouts into one. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have discovered this discussion a bit late... Anyway, I wrote a tool to import google photos takeout files with albums when present, directly from the zipped files. It handles albums, handles duplicates, discard field videos, and many other options. No manual work is required. Easy installation Details here: https://github.com/simulot/immich-go#readme Reading this discussion, I have checked the timeline and the map. Sure enough, I have some pictured taken on 2036, and some others located in the gulf of Guinea. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, your tool has saved me a lot of time. |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome. I'm glad to read that! |
Beta Was this translation helpful? Give feedback.
-
|
It seems like the problem space of taking care of Takeout.zip was bigger than anybody thought; that's why we now have a million different repos (including my shitty one) that promise to import your Google Photos into immich. Maybe we should make a wiki page and populate it with all possible invariants. The situation right now is confusing for everybody I think... 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Google takeouts are a mess, and as you correctly note there have been many attempts at dealing with it. I think the main challenge here is that a person only ever needs to deal with this once, and so nobody is motivated to maintain a piece of tooling for the longer term. If a way past that issue is found, I think dealing with the actual takeout files in an effective way won't be a big problem comparatively. It would be awesome if this could be done in a way that is not tied to Immich specifically, so that anybody can make use of it :D |
Beta Was this translation helpful? Give feedback.
-
|
feel free to add tools here #1340 (comment) |
Beta Was this translation helpful? Give feedback.
-
done |
Beta Was this translation helpful? Give feedback.
-
|
This doesn't appear to work on live video which I have taken. There are tools which are capable of merging the videos into the photos to create a live photo, but this seems an unnecessarily complex solution. |
Beta Was this translation helpful? Give feedback.
-
|
I've successfully ran the google-photos-migrate tool on 1.16tb of google takeout photos (took 10hrs on my underpowered NAS but made it through). Can anyone help me understand the folder structure after everything is complete? Both AlbumsProcessed and PhotosProcessed seem to be close to the same size (~500gb a piece). Are these after deduplication scan? Do both directories have the same photos in them but is the photos directory just a flat directory and the albums directory has the same content but in their original directories? Id love to see a results.json file that lists all the stats once completed that explains things a bit. I need to know what directory i need to upload to immich and i dont want to do another dedupe scan. Any help would be great. |
Beta Was this translation helpful? Give feedback.
-
|
Everything in the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @lukehmcc. If i dont plan on using the cli to upload and plan on mounting an existing readonly library with these photos, would i be better off just mounting the AlbumsProcessed or should i stick with the PhotosProcessed? |
Beta Was this translation helpful? Give feedback.
-
|
Ah okay. I'm honestly not sure how dedup works if it's externally mounted. I presume that it will still hash each image and only put it on the timeline once but I can't say for sure. There likely are photos in |
Beta Was this translation helpful? Give feedback.
-
|
All my photos were in separate directories so i presume they would be located in both. |
Beta Was this translation helpful? Give feedback.
-
|
I would try mounting both and seeing how it goes. I'd like to know. |
Beta Was this translation helpful? Give feedback.
-
|
I prefer to keep the original quality/format of my photos/videos and not degrade to the reduced Google Photos upload/download quality. So I would like to apply the Google Albums metadata to recover those albums in my Immich originals. |
Beta Was this translation helpful? Give feedback.
-
That's an use case I haven't envisaged. Should I add a feature that recreate albums from Google Photos archives or why not using the google apis to avoid the dully process of takeout data? |
Beta Was this translation helpful? Give feedback.
-
|
Yes that is conceptually speaking what I want: just extract the 'album metadata' from Google Photos (via apis?) and not having to extract the media themselves (because useless) |
Beta Was this translation helpful? Give feedback.
-
|
Added to the roadmap. |
Beta Was this translation helpful? Give feedback.
-
|
I don't understand why enable photo compression and make your life more complicated after? It's your own fault, deal with it |
Beta Was this translation helpful? Give feedback.
-
|
Hi @eil047, try using czkawka or czkawka docker and click similar images, it may find many duplicates and then you can just batch delete the lower quality copies before importing to immich. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Be aware of the \ at the end of each path. |
Beta Was this translation helpful? Give feedback.
-
|
The entire discussion above focusses on date taken, but do any of the scripts also take care of migrating face data? Did quite some work on tagging all of my 150K photos with relevant gps and face recognition data and would not like to see it getting lost... |
Beta Was this translation helpful? Give feedback.
-
|
Hopefully this is useful - I used the command line in the posts from this forum thread (exiftool) to fix mine: https://exiftool.org/forum/index.php?topic=12602.0 and this https://exiftool.org/forum/index.php?topic=11064.45 This updated the original files instead of creating copies. |
Beta Was this translation helpful? Give feedback.
-
|
Useful links, but all of them only focus on datetime and gps metadata. If I look in my JSON files I am especially interested in the "people" and "name" tags... |
Beta Was this translation helpful? Give feedback.
-
|
Think you could probably get exiftool to read in those (if there is a metadata exif tag that matches those). Not sure if they exist though. Don't want to divert the topic but this is what I found on the above: https://exiftool.org/TagNames/EXIF.html |
Beta Was this translation helpful? Give feedback.
-
|
There is a thread on this but I have not gotten it to work yet. |
Beta Was this translation helpful? Give feedback.
-
|
I currently use gphotos-cdp through a container image I packaged with a script that by default adds the specific custom logic I need. |
Beta Was this translation helpful? Give feedback.
and others
-
gphotos import + albums
With this guide, you
STILL have to add images that are in multiple albums manually (except for the first instance of that image, so hopefully this isn’t too many)can get albums imported now as of https://github.com/immich-app/CLI/pull/49prereqs:
mattwilson1024/google-photos-exif,exiftool,ffmpegInstructions
download:
extract:
note: your files that are in albums will be duplicated, but immich will handle the deduplication and not upload it twice
create directory structure to:
run
mattwilson1024/google-photos-exifon EACH album to add missing EXIF data:run
mattwilson1024/google-photos-exifonGooglePhotosto add missing EXIF data:if any have errored out in
AlbumsProcessedorerror, fix metadata by running:exiftool -all= -tagsfromfile @ -all:all -unsafe -icc_profile .then run
mattwilson1024/google-photos-exifagain on the (fixed) error directory (2nd pass), they should all pass now. move all output files to their corresponding output foldersupload ALBUMS to immich FIRST
so media gets put into albums (otherwise it will not be put inside their respective albums and you’ll have to manually select each photo manually to add to the album, and you can’t re-upload it because of deduplication which rejects files with the same hash)Edit: I prefer doing it this way, but this no longer /has/ to be done first as the CLI will still associate images with albums if they are in the folder structure.Upload each album INDIVIDUALLY and VERIFY file counts. If there are any discrepancies, figure out why. For example, if a file is not supported, some files are missing their corresponding
jsonfiles so metadata was not added, or the file is a duplicate. If there were any errors, rerunmattwilson1024/google-photos-exifon them using the commands above.for unsupported files, you can convert the format using:
ffmpeg -i example.MPG example.mp4orffmpeg -i filename.mov -c copy filename.mp4for missing json files, you can find the nearest neighbor, such as
longfilename.(1).jpnote the missing “g” injp(as opposed tojpg)you can compare whether images have the same hash with:
sha1sum file.extor find duplicates with https://github.com/qarmin/czkawkafor duplicate files in multiple albums, add the corresponding missing images back to their albums.
Once all albums have been processed and verified, upload the rest that are not in albums:
immich upload \ --key APIKEY \ --server https://immich.domain.tld/api \ -d '/mnt/f/user google takeout/Takeout/output' -t 16Verify you don’t have any files that show up as current day upon refreshing immich. Repeat any of the above steps if they are showing up without proper metadata.
Rejoice as you are now free from gphotos.
Beta Was this translation helpful? Give feedback.
All reactions