FML: accidentally trashed everything


#1

Okay, so long story short, through my own stupidity, the use of many variables, and my own lack of foresight to review scripts after variable changes, I managed to do an rclone move dir:/ dir:/. Yeah… stupid stupid stupid.

Worse yet, I let it run the full drive because I didn’t take the time to review what was going on.

So, I now have hundreds of thousands of files in the trash, waiting to be restored.

How can I accomplish restoring them back to their original directory structure (encrypted)?

I was thinking of something like using rclone mount dir:/ /mnt/dirtrash—use-trash-only and rclone mount dir:/ /mnt/dir and doing a mv /mnt/dirtrash /mnt/dir between the two, but would that download then re-upload everything?

I’ve got fiber at 1gb, so it’s not out of the question, but that 750gb limit is going to be hit for sure, like daily for the next 4 months…

I need some help.

Thank you all in advance.


#2

Oh, I’m using Google drive.


#3

The good thing about Drive is, that the folder structure is remained in the trash, although it does not look like it in the Webinterface.

How many files and folders do you need to untrash? If it’s less than 10000 you could use the Webinterface to restore 400 to 500 items at a time.

If there are a lot more files i may be able to help with a little go program that restores all files and folders.


#4

Hundreds of thousands of files, unfortunately. Maybe even over a million. I tried to do the restore by web interface, but my browsers keep crashing when I reach a few hundred, and anything less, and I’ll be at that longer than reasonable.


#5

I’ve also thought about doing a full delete, then a full restore… but that requires admin and I only have an edu account from an old school that I’m not enrolled in any longer, so I couldn’t count on any assistance from them.


#6

I’ve been able to get into the drive api, so I could probably run something like:

GET https://www.googleapis.com/drive/v3/files?corpus=user&orderBy=quotaBytesUsed&pageSize=1000&q=trashed+%3D+true&fields=files%2Fid&key={YOUR_API_KEY}
PATCH https://www.googleapis.com/drive/v3/files/files%3Fcorpus%3Duser%26orderBy%3DquotaBytesUsed%26pageSize%3D10%26q%3Dtrashed%2B%3D%2Btrue%26fields%3Dfiles%2Fid?fields=trashed&key={YOUR_API_KEY}

{
“trashed”: false
}

if I could just figure out where to do so…


#7

I would have thought that would do nothing at all.

I tried a quick test and that appeared to be the case…

$ rclone mkdir drive:test/mvtest
$ echo "hello" | rclone rcat drive:test/mvtest/hello.txt
$ rclone lsf drive:test/mvtest
hello.txt
$ rclone -v move drive:test/mvtest drive:test/mvtest
2018/02/08 21:49:16 INFO  : Google drive root 'test/mvtest': Modify window is 1ms
2018/02/08 21:49:16 ERROR : Google drive root 'test/mvtest': Nothing to do as source and destination are the same
2018/02/08 21:49:16 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.6s

$ rclone lsf drive:test/mvtest
hello.txt

#8

I cleaned up my code and put it on https://gitlab.com/B4dM4n/drive-untrash

It pretty much uses the API calls you found, but also handles the authentication. I hope this helps you.


#9

Correct, ncw. It had to do with the way the variables were in use, not the program. That’s why I went out of my way to say it was my fault alone.

I didn’t want to go into the long story, and I don’t expect any new development on the issue I experienced.

So, here’s the long story:
I had a setup with plexdrive. I’m moving to rclone with caching now that it is working very well.
I have a drive mounted with rclone mount GDriveCacheCrypt:/ /mnt/disks/GDriveCacheCrypt/
I have another set of scripts with variables which call /mnt/disks/GDriveCacheCrypt to move things in and out of a staging directory at /mnt/disks/Staging.
I have these directories mounted with a unionfs to /mnt/disks/UnionFS/
I had a script that ran rclone move /mnt/disks/Staging/ GDriveCrypt:/
Unfortunately, my script called a variable for $Staging, which I neglected to notice…

I changed the staging variable to GDriveCacheCrypt:/ Things did not go well after that. I had the rclone move setup so it would check the file if it existed and was newer, do nothing, but delete the file from $Staging…

This is the actual script that ran:
rclone move $Staging/Media/ GDriveCrypt:/Media/ --transfers=12 --checkers=24 --min-age 1m --buffer-size 150M --no-traverse --bwlimit=“06:00,3M 00:00,3M” --fast-list --tpslimit 1 --tpslimit-burst 2 -v

So, you see, $Staging/Media became /mnt/disks/GDriveCacheCrypt/Media/, and the rest is history. It tried to move the file, only to find that when it deleted to overwrite the file, the original was mysteriously gone.

Again, I can only blame myself for this. There is nothing that could have been expected of rclone to protect me in this instance. Garbage in, garbage out.


#10

Dude… awesome… I’m running it right now. We’ll see how I fare afterwards. I can’t tell if it’s running or not, but the cursor is flashing, and I got a debug or two about retries. I’ll take that as a sign it’s running.


#11

Oh, I see :frowning:

I’m always into putting safety checks into rclone to prevent data loss so if you can think of something I’ll put it in!


#12

Just an idea:

--max-allowed-deletions x

Require user confirmation if more than x files are to be deleted at end of transfer. 
x's default value is 100. 

If x is set to 0, no deletions are allowed. 

Set x to -1 to allow unlimited deletions. This can cause massive data loss if used improperly. 
Use --dry-run to test runs first for safety.

#13

There is an issue about that already here - can you add your support there please.


#14

personally I think the default should be -1
it’s rare for someone to use the move command and not want things deleted, because well, you have the copy command.
I pretty much always use the move command as if it were a purge command (only much safer because it checks everything first).


#15

I forgot to update. This worked beautifully.

Thank you so much.


#16

I came across someone on reddit that did something similar, and pointed him here. I’m wondering if there could be an “RClone Related Tools” section, where this, browser, and any others could be stickied?


#17

Partially complete.