Skip to content
This repository has been archived by the owner on Apr 19, 2023. It is now read-only.

Delete all files within folder #422

Closed
chris-mann-uk opened this issue Jan 15, 2019 · 9 comments
Closed

Delete all files within folder #422

chris-mann-uk opened this issue Jan 15, 2019 · 9 comments

Comments

@chris-mann-uk
Copy link

Is there a way to delete all files within a particular folder?

I'm running a cron on my server that creates a backup every 24 hours, with the file that is created being synced with Google Drive.

I wish to delete the backups after a few days, however, and am unsure of the command required to delete all files within my allocated backup folder.

I've found a way that would allow me to delete the folder but I don't want that...just the files located within.

@chris-mann-uk
Copy link
Author

chris-mann-uk commented Feb 3, 2019

I'm still no closer to resolving this, unfortunately.

Having used the help facility, I know I can use a command such as the one below:
gdrive delete <fileID>

-r will ensure files are deleted recursively, but my goal isn't to delete a folder - just the contents within it.

Commands such as this will list files from a particular folder, but is there something similar that would allow me to delete these files but keep the folder intact?

gdrive list --query "'<folderID>' in parents

@mbenlioglu
Copy link
Contributor

What you want can be achieved through batch request to API, but as of v2.1.0 gdrive doesn't support batch queries yet. Similar issue mentioned in #381. Until a fix is introduced in a future version, you can use 2 alternative ways to bypass it.

  1. You can remove the directory along with it's contents and recreate directory with same name in a single function. (Note: this will change the folder ID, if you're statically storing them for later use you need to update it)

  2. You can call multiple delete queries using xargs or similar e.g.

    $ gdrive list -q "'FOLDER_ID' in parents" --no-header --max 0 | cut -d" " -f1 - | xargs -L 1 gdrive delete -r
    

What this does is basically listing all files within the folder, getting file IDs from it using cut and executing gdrive delete -r <FILE_ID> for each of them

@chris-mann-uk
Copy link
Author

Solution number 2 is along the lines of what I'm looking for and seems to work.

It is preferable because solution number 1 would involve a different folder ID on each occasion, which only further complicates things as I intend to use this script as a cron.

One further question, however, is would it be possible to remove all files that are older than (let's say) 24 hours old?

I was thinking something along the lines of this, but it returns an error 400 message so clearly doesn't work.

$ gdrive list -q "('FOLDER_ID' in parents) and modifiedTime < date -d yesterday" --no-header --max 0 | cut -d" " -f1 - | xargs -L 1 gdrive delete -r

@mbenlioglu
Copy link
Contributor

@ChrisMann89 Drive API expects date formats given in queries to be in ISO 8601 format, for that you need to use -I, --iso-8601 option of date. So the command you want would be:

$ gdrive list -q "('FOLDER_ID' in parents) and modifiedTime < '$(date -d yesterday -Iseconds)'" --no-header --max 0 | cut -d" " -f1 - | xargs -L 1 gdrive delete -r

(Notice the single quotes and sub-shell call in the query)

Before I finish, I want to point out 2 things in this kind of query.

  1. By doing this you send separate commands to API for every file in the folder, which means if you have many files within the folder this will flood the API and will take a lot of time. So keep that in mind. (And hopefully you won't need to do this at all, once batch query support is introduced)
  2. Sometimes queries might get lost, because of unstable network connection or problems in the API etc. Which might cause some files to be not deleted or not getting the file IDs at all. So I would suggest a "safe" method that will persistently send the query if it fails. For example, if we define following bash functions:
# Echo error message to stderr
errcho () {
    >&2 echo $@
}

# Send list request to drive with retry mechanism against failures (same parameters as original)
safe_list () {
    local RESPONSE
    while true; do
        RESPONSE=$(eval gdrive list $@)
        echo "${RESPONSE}" | grep Failed &> /dev/null && errcho "Failed to get information from cloud. Trying again in 5 seconds" \
            && sleep 5 || break
    done
    echo "$RESPONSE"
}

# Send delete request to drive with retry mechanism against failures (same parameters as original)
safe_delete () {
    local RESPONSE
    while true; do
        RESPONSE=$(eval gdrive delete $@)
        echo "${RESPONSE}" | grep Failed &> /dev/null && errcho "Failed delete from cloud. Trying again in 5 seconds" \
            && sleep 5 || break
    done
    echo "$RESPONSE"
}

Your query would become

$ safe_list -q "\"('FOLDER_ID' in parents) and modifiedTime < '$(date -d yesterday -Iseconds)'\"" --no-header --max 0 | cut -d" " -f1 - | xargs -L 1 safe_delete -r

(Notice the extra quotes with backslash)

Now in this form, using the above functions, if your query gets a "Failed" message from gdrive it will try again in 5 seconds. (And if you check your logs you'll realize you're getting failed messages often)

@chris-mann-uk
Copy link
Author

Fantastic! Problem solved.

To summarise, I have a cron that creates a backup of files on my server each night and sends a .tar.gz over to Google Drive.

Over time, these build up so this new script - added via another line in the cron list - will automatically remove any backups older than one week old and free up space in Google Drive.

To do this, I have changed 'yesterday' to 'last-week' in the script, meaning it now looks as follows:
$ safe_list -q "\"('FOLDER_ID' in parents) and modifiedTime < '$(date -d last-week -Iseconds)'\"" --no-header --max 0 | cut -d" " -f1 - | xargs -L 1 safe_delete -r

Thank you for your help, @mbenlioglu. I now consider this issue closed.

@mbenlioglu
Copy link
Contributor

Glad to be of help

@thearabbit
Copy link

I would like to delete all file in 1 month ago.
But I trie to list by your code about to test

gdrive list -q "\"('j7Ru1********JSeeT' in parents) and modifiedTime < '$(date -d last-week -Iseconds)'\"" --no-header --max 0

Show eeror

usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Failed to list files: googleapi: Error 400: Invalid Value, invalid

Please help me

@thearabbit
Copy link

I test on Mac!

@hlnull
Copy link

hlnull commented Feb 2, 2021

Deleting files with Colab might be an alternative solution.

from google.colab import drive
drive.mount('/content/gdrive')

!rm *.jpg

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants