Google Shared Drive to DropBox

[Groggles (Buchberger)]

What is the problem you are having with rclone?

• sync will resync files because the destination does not reflect the source mod time, and
• large *.pst file fails and rclone is stuck

By adding the --size-only flag the first problem is overcome, but there may inaccuracy.

The static hashes for these two cloud media are different, so using --checksum ( size and checksum ) would effectively be size anyway. Is that correct ?

I read in this post, https://forum.rclone.org/t/how-to-compare-md5sum-hashes-after-copying-files-from-dropbox-to-google-drive/7457 that there is/was a --download flag. I cannot find a reference in /flags. I would be happy to avoid size and modtime if a byte comparison is performed. Is it possible?

...

For the 'getting stuck problem', should i just exclude that file. Given that it is one of the largest we are transferring - at 47.5 GB - could that be the source of the issue?

What is your rclone version (output from rclone version)

1.53.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18

Which cloud storage system are you using? (eg Google Drive)

Google Shared Drives + DropBox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync ADMIN:/ CADropBox:/Shared\ Drives\ Backup/ADMIN --drive-chunk-size 512M --max-backlog 999999 --fast-list -P -vv 

The rclone config contents with secrets removed.

[CADropBox]
type = dropbox
token = {"access_token":" ... ","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

[ADMIN]
type = drive
client_id = ...
client_secret =...
scope = drive.readonly
token = {"access_token":" ... ","token_type":"Bearer","refresh_token":"1//...":"2020-11-11T02:45:49.877313349Z"}
team_drive = 0ACuh5ix5N44VUk9PVA
root_folder_id =

A log from the command with the -vv flag

Transferred:   	    1.596G / 47.516 GBytes, 3%, 2.471 MBytes/s, ETA 5h17m12s
Errors:                17 (retrying may help)
Checks:             58420 / 58420, 100%
Transferred:            1 / 2, 50%
Elapsed time:     13m24.6s
Transferring:
 * someclientname - John.pst:  3% /47.514G, 0/s, 0s

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

@Groggles

getting stuck problem

you would look into the debug log file for the reason.

[ncw (Nick Craig-Wood)]

@Groggles

sync will resync files because the destination does not reflect the source mod time, and

Both dropbox and drive support modification times. To set a modification time on dropbox you have to upload the file again, alas.

Once everything is in sync it shouldn't require any more uploading though.

@Groggles

By adding the --size-only flag the first problem is overcome, but there may inaccuracy.

Indeed. It shouldn't be necessary though...

@Groggles

The static hashes for these two cloud media are different, so using --checksum ( size and checksum ) would effectively be size anyway. Is that correct ?

That is correct. Drive supports MD5 - dropbox have their own hash.

@Groggles

I read in this post, How to compare md5sum hashes after copying files from Dropbox to Google Drive? that there is/was a --download flag. I cannot find a reference in /flags. I would be happy to avoid size and modtime if a byte comparison is performed. Is it possible?

You can use rclone check --download - that isn't a global flag so it doesn't exist in the /flags docs.

@Groggles

For the 'getting stuck problem', should i just exclude that file. Given that it is one of the largest we are transferring - at 47.5 GB - could that be the source of the issue?

A debug log (with -vv) would be useful to see why it is getting stuck.

[Groggles (Buchberger)]

Thank you for the very prompt and clear reply.

Yes I think I am guilty of mis-pasting logs for the file in question, it did give a few extra lines stating 'disallowed path'. Here below is a similar failure. As it happens another .pst file, this one is about 600MB - though size and type are probably not relevant.

I would also note that in the Drive dirs containing the failing pst file, there are *.tmp files prefixed with a tilda - screenshot attached.

(I have had no issue using rsync on this 1.27TB dir on our internal storage)

Any advice is appreciated, and I should say that the app is very impressive and has done most of what I set out to do. Thanks.

2020-11-20 21:23:56 DEBUG : Path_to_file/seems_to_always_involve_pstfiles.pst: Uploading chunk 6/6
2020-11-20 21:24:09 INFO : Path_to_file/seems_to_always_involve_pstfiles.pst: Copied (replaced existing)
2020-11-20 21:24:09 ERROR : Dropbox root 'Shared Drives Backup/CLIENTS': not deleting files as there were IO errors
2020-11-20 21:24:09 ERROR : Dropbox root 'Shared Drives Backup/CLIENTS: not deleting directories as there were IO errors
2020-11-20 21:24:09 ERROR : Attempt 3/3 failed with 16 errors and: upload failed: path/disallowed_name/
Transferred: 923.136M / 923.136 MBytes, 100%, 2.383 MBytes/s, ETA 0s
Errors: 16 (retrying may help)
Checks: 176964 / 176964, 100%
Transferred: 30 / 30, 100%
Elapsed time: 19m26.2s
2020/11/20 21:24:09 INFO :
Transferred: 923.136M / 923.136 MBytes, 100%, 2.383 MBytes/s, ETA 0s
Errors: 16 (retrying may help)
Checks: 176964 / 176964, 100%
Transferred: 30 / 30, 100%
Elapsed time: 19m26.2s

2020/11/20 21:24:09 DEBUG : 7 go routines active
2020/11/20 21:24:09 Failed to sync with 16 errors: last error was: upload failed: path/disallowed_name/![Screen Shot 2020-11-21 at 09.05.07|690x118]

[asdffdsa (jojothehumanmonkey)]

.pst, that is a format used by outlook, and the .tmp means that the .pst file is open and inuse by some program.
is that correct, what program is using the .pst file?

[Groggles (Buchberger)]

Yes correct, Outlook. There may also be another app that converts PST to MBOX, but I don't believe these write tmp files.

That said, these tmp files prefixed with a tilda are disallowed by rclone and that's a good thing. These files are not in use, so the tmp files must remain after Outlook is quit.

[asdffdsa (jojothehumanmonkey)]

if the .tmp are not in use, then why are they disallowed by rclone?

to be clear, outlook has the .pst open at the same time you try to copy it up with rclone?

[Groggles (Buchberger)]

Outlook is not in use. These files were opened using Outlook quite some time ago and I'm assuming those tmp files are 'left behind'.

I think rclone disallows anything with a tilda prefix, nothing to do with pst ? An assumption I will test.

The issue though is not the tmp files, but error ( 2020-11-20 21:24:09 ERROR : Attempt 3/3 failed with 16 errors and: upload failed: path/disallowed_name/ ) of syncing a plain pst file.

[Krandor (Brian Hartsfield)]

Here are my experiecnes. I signed up for dropbox business but because I mainly wanted to see how they handled the paying side and didn't want to risk having extra licenses skipped the trial and went right to paid.

So on my upload/copy from my VPS I have just passed the initial 3TB mark. However so far I am still able to copy and have not been blocked. On my Plex server (where I did also mount the DB as well) which is ubuntu the mount currently shows Size 3TB used -16EB (yes negative) and available 16EB. The dropbox GUI only shows 1TB in use so it clearly hasn't updated yet.

So far my sync is still running and no errors from DB on the rclone sync to dare.

[Harry]

Db is very costly man...

[Krandor (Brian Hartsfield)]

It is more expensive then gsuite for sure but for the amount of storage I have it is still cheap.

[Harry]

How much are we talking about? I have 900TB and it’s not movies but backups of work.

[Krandor (Brian Hartsfield)]

I'm around 500TB and I know of no other place you can store that much for $75/month.

[Harry]

Yeah.. that is true sadly. Can you let me know whether the storage limit increases automatically or by request?

[Krandor (Brian Hartsfield)]

Yes I will. That is why I posted my initial results. To let people know. Plan to keep doing that. So far I'm beyond the 3TB initial "limit" but rclone sync is still running. I will definitely keep this forum informed of my results going forward good or bad.

[Krandor (Brian Hartsfield)]

I just got an email that they automatically added 2TB for me. I should hit that tomorrow and we'll see what happens then.

[ncw (Nick Craig-Wood)]

@Groggles

Yes I think I am guilty of mis-pasting logs for the file in question, it did give a few extra lines stating 'disallowed path'.

That is trying to upload files with names that dropbox doesn't allow most likely.

The ERROR lines from your log would be useful to see but if you have a dropbox disallowed file name , eg one of these, that would explain it

• desktop.ini
• thumbs.db
• .ds_store
• icon\r
• .dropbox
• .dropbox.attr

If so then you'll have to exclude them from the sync.

If the PST file is open and being written to then you might have to use VSS which there is a great writeup here by our very own @asdffdsa

I want to share how to enable vss for rclone

hello to all, rlcone is great but since it is written to be comptible with so many operating systems, it is lacking a cirtical feature for Micro$oft window users. as per wikipedia, https://en.wikipedia.org/wiki/Shadow_Copy " VSS operates at the block level of volumes. A snapshot is a read-only point-in-time copy of the volume. Snapshots allow the creation of consistent backups of a volume, ensuring that the contents do not change and are not locked while the backup is being made." when i fi…

[asdffdsa (jojothehumanmonkey)]

the OP has written that the .pst file is not in use.
anyhoo, here is the wiki i wrote about using VSS with rclone or any command line app.

How to enable VSS for rclone · rclone/rclone Wiki · GitHub

[Groggles (Buchberger)]

@ncw & @asdffdsa

Thanks for the help. I ran sync excluding *.pst and all worked pretty well.

I ensured all users had closed their Office apps, and included *.pst and it worked. Perhaps there is some Windows weirdness and locked files.

So before attempting to enable vss for rclone, this worked:

• close all MS Office processes ( not just Outlook )

And, fwiw:

• if DBox destination then use the excludes ncw suggests
• I excluded *.tmp files as well

Thanks again.

[asdffdsa (jojothehumanmonkey)]

good to know,

so you have the .pst files on a server?

if so, running a vss snapshot on the server to backup the .pst, is not the way to go.
no way would i trust that.

for that to work reliably, the snapshot has to be run on the client.
this is due to the way vss interacts compliant applications such as m$office.

if you need a free backup solution for client and server machines, i use the community editions of veeam agent and veeam backup and replication.
then i use rclone sync --backup-dir with a date.time stamp, to copy those backup files to cheap aws s3 storage.

or run vss on client, use rclone sync --backup-dir to copy .pst to cloud,