Dropbox sync keeps deleting the files on Dropbox

Wayne_Spivak · February 12, 2025, 2:24am

STOP and READ USE THIS TEMPLATE **NO

What is the problem you are having with rclone?

sync to dropbox sends files up, and some time later they automatically delete. There is no change in the source folder

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.69.0

os/version: raspbian 12.9 (64 bit)
os/kernel: 6.6.74+rpt-rpi-2712 (aarch64)
os/type: linux
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /mnt/raid10/BACKUP/ALLSTARbackup/Allstar Dropbox:Backups --log-file=/home/wbs/logfile.txt  --log-level INFO

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

rclone config redacted
[Dropbox]
type = dropbox
token = XXX

A log from the command that you were trying to run with the `-vv` flag

The Log file is 517 lines long.... I also can't figure out how to add the "-vv" to the log file.

025/02/11 19:10:36 INFO  : NAS@debian12-tgz-backup-20250206-050054/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:10:36 INFO  : NAS@debian12-tgz-backup-20250206-050054/NAS-backup.mbr: Copied (new)
2025/02/11 19:10:37 INFO  : NAS@debian12-tgz-backup-20250206-050054/raspiBackup.log: Copied (new)
2025/02/11 19:10:38 INFO  : NAS@debian12-tgz-backup-20250206-050054/raspiBackup.msg: Copied (new)
2025/02/11 19:10:39 INFO  : NAS@debian12-tgz-backup-20250208-050047/NAS-backup.mbr: Copied (new)
2025/02/11 19:10:40 INFO  : NAS@debian12-tgz-backup-20250208-050047/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:11:01 INFO  : NAS@debian12-tgz-backup-20250208-050047/NAS-backup.img: Copied (new)
2025/02/11 19:11:02 INFO  : NAS@debian12-tgz-backup-20250206-050054/NAS-backup.img: Copied (new)
2025/02/11 19:11:02 INFO  : NAS@debian12-tgz-backup-20250208-050047/raspiBackup.msg: Copied (new)
2025/02/11 19:11:02 INFO  : NAS@debian12-tgz-backup-20250208-050047/raspiBackup.log: Copied (new)
2025/02/11 19:11:03 INFO  : NAS@debian12-tgz-backup-20250209-065943/NAS-backup.mbr: Copied (new)
2025/02/11 19:11:04 INFO  : NAS@debian12-tgz-backup-20250209-065943/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:11:30 INFO  : NAS@debian12-tgz-backup-20250209-065943/NAS-backup.img: Copied (new)
2025/02/11 19:11:31 INFO  : NAS@debian12-tgz-backup-20250209-065943/raspiBackup.log: Copied (new)
2025/02/11 19:11:32 INFO  : NAS@debian12-tgz-backup-20250209-065943/raspiBackup.msg: Copied (new)
2025/02/11 19:11:35 INFO  : 
Transferred:   	    4.220 GiB / 20.898 GiB, 20%, 72.418 MiB/s, ETA 3m55s
Transferred:           15 / 72, 21%
Elapsed time:       1m0.0s
Transferring:
 * NAS@debian12-tgz-backu…up-20250206-050054.tgz: 92% /1.119Gi, 16.887Mi/s, 5s
 * NAS@debian12-tgz-backu…up-20250208-050047.tgz: 93% /1.120Gi, 19.953Mi/s, 3s
 * NAS@debian12-tgz-backu…up-20250209-065943.tgz: 52% /1.121Gi, 19.599Mi/s, 27s
 * NAS@debian12-tgz-backu…-050046/NAS-backup.img:  9% /512Mi, 24.000Mi/s, 19s

2025/02/11 19:11:38 INFO  : NAS@debian12-tgz-backup-20250208-050047/NAS@debian12-tgz-backup-20250208-050047.tgz: Copied (new)
2025/02/11 19:11:39 INFO  : NAS@debian12-tgz-backup-20250120-050046/NAS-backup.mbr: Copied (new)
2025/02/11 19:11:41 INFO  : NAS@debian12-tgz-backup-20250120-050046/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:11:42 INFO  : NAS@debian12-tgz-backup-20250206-050054/NAS@debian12-tgz-backup-20250206-050054.tgz: Copied (new)
2025/02/11 19:11:43 INFO  : NAS@debian12-tgz-backup-20250120-050046/raspiBackup.log: Copied (new)
2025/02/11 19:11:44 INFO  : NAS@debian12-tgz-backup-20250120-050046/raspiBackup.msg: Copied (new)
2025/02/11 19:11:54 INFO  : NAS@debian12-tgz-backup-20250120-050046/NAS-backup.img: Copied (new)
2025/02/11 19:11:57 INFO  : NAS@debian12-tgz-backup-20241203-050044/NAS-backup.mbr: Copied (new)
2025/02/11 19:11:57 INFO  : NAS@debian12-tgz-backup-20241203-050044/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:12:03 INFO  : NAS@debian12-tgz-backup-20250209-065943/NAS@debian12-tgz-backup-20250209-065943.tgz: Copied (new)
2025/02/11 19:12:04 INFO  : NAS@debian12-tgz-backup-20241203-050044/raspiBackup.log: Copied (new)
2025/02/11 19:12:05 INFO  : NAS@debian12-tgz-backup-20241203-050044/raspiBackup.msg: Copied (new)
2025/02/11 19:12:10 INFO  : NAS@debian12-tgz-backup-20241203-050044/NAS-backup.img: Copied (new)
2025/02/11 19:12:10 INFO  : NAS@debian12-tgz-backup-20250205-050103/NAS-backup.mbr: Copied (new)
2025/02/11 19:12:12 INFO  : NAS@debian12-tgz-backup-20250205-050103/NAS-backup.sfdisk: Copied (new)
2025/02/11 19:12:31 INFO  : NAS@debian12-tgz-backup-20250205-050103/NAS-backup.img: Copied (new)
2025/02/11 19:12:32 INFO  : NAS@debian12-tgz-backup-20250205-050103/raspiBackup.log: Copied (new)
2025/02/11 19:12:33 INFO  : NAS@debian12-tgz-backup-20250205-050103/raspiBackup.msg: Copied (new)
2025/02/11 19:12:35 INFO  : 
Transferred:   	    8.654 GiB / 20.898 GiB, 41%, 75.847 MiB/s, ETA 2m45s
Transferred:           33 / 72, 46%
Elapsed time:       2m0.0s
Transferring:
 * NAS@debian12-tgz-backu…up-20250120-050046.tgz: 82% /1.370Gi, 21.462Mi/s, 11s
 * NAS@debian12-tgz-backu…up-20241203-050044.tgz: 54% /1.279Gi, 18.543Mi/s, 32s
 * NAS@debian12-tgz-backu…up-20250205-050103.tgz: 37% /1.119Gi, 18.590Mi/s, 38s
 * NAS@debian12-tgz-backu…-050047/NAS-backup.img:  9% /512Mi, 23.999Mi/s, 19s

SNIP
025/02/11 20:06:48 INFO  : NAS@debian12-tgz-backup-20250120-050046/NAS-backup.sfdisk: Deleted
2025/02/11 20:06:49 INFO  : Tar-backup/2025-02-05_ahead_www.tar.gz: Deleted
2025/02/11 20:06:49 INFO  : Tar-backup/2025-02-07_sbanetweb_www.tar.gz: Deleted
2025/02/11 20:06:50 INFO  : Tar-backup/2025-02-08_diamonds_www.tar.gz: Deleted
2025/02/11 20:06:50 INFO  : Tar-backup/2025-02-09_ahead_www.tar.gz: Deleted
2025/02/11 20:06:51 INFO  : Tar-backup/eggcream-www.sh: Deleted
2025/02/11 20:06:51 INFO  : Tar-backup/full_backup_sbaconsult.2025-02.tar.gz: Deleted
2025/02/11 20:06:52 INFO  : NAS@debian12-tgz-backup-20250206-050054/raspiBackup.log: Deleted
2025/02/11 20:06:52 INFO  : NAS@debian12-tgz-backup-20250206-050054/NAS@debian12-tgz-backup-20250206-050054.tgz: Deleted
2025/02/11 20:06:54 INFO  : NAS@debian12-tgz-backup-20250211-050050/NAS-backup.sfdisk: Deleted
2025/02/11 20:06:54 INFO  : Tar-backup/2025-02-05_treasurer.shomrim_home.tar.gz: Deleted
2025/02/11 20:06:56 INFO  : Tar-backup/2025-02-08_panynjshomrim_home.tar.gz: Deleted
2025/02/11 20:06:57 INFO  : Tar-backup/dataset-sba.sh: Deleted
2025/02/11 20:06:58 INFO  : NAS@debian12-tgz-backup-20241203-050044/NAS@debian12-tgz-backup-20241203-050044.tgz: Deleted
2025/02/11 20:06:59 INFO  : Tar-backup/2025-02-09_postfix_etc.tar.gz: Deleted
2025/02/11 20:07:00 INFO  : Tar-backup/2025-02-03_diamonds_www.tar.gz: Deleted
2025/02/11 20:07:01 INFO  : Tar-backup/2025-02-10_anthony.zuaro_home.tar.gz: Deleted
2025/02/11 20:07:01 INFO  : NAS@debian12-tgz-backup-20250211-050050/NAS-backup.img: Deleted
2025/02/11 20:07:03 INFO  : Tar-backup/2025-02-05_CGribbons_www.tar.gz: Deleted
2025/02/11 20:07:04 INFO  : Tar-backup/2025-02-02_panynjshomrim_www.tar.gz: Deleted
2025/02/11 20:07:04 INFO  : Tar-backup/wayne.spivak.rescue.sh: Deleted
2025/02/11 20:07:05 INFO  : Tar-backup/2025-02-04_CGribbons_www.tar.gz: Deleted
2025/02/11 20:07:06 INFO  : Tar-backup/2025-02-07_pashomrim.shomrim_home.tar.gz: Deleted
2025/02/11 20:07:06 INFO  : NAS@debian12-tgz-backup-20241203-050044/NAS-backup.mbr: Deleted
2025/02/11 20:07:06 INFO  : NAS@debian12-tgz-backup-20250209-065943/raspiBackup.msg: Deleted

kapitainsky · February 12, 2025, 6:12am

So far what you posted looks perfectly fine.

We need full debug log to see what is going on. Run:

rclone sync /mnt/raid10/BACKUP/ALLSTARbackup/Allstar Dropbox:Backups --log-file=/home/wbs/logfile.txt  --log-level DEBUG

and post logfile content - does not have to be here. You can share it on for example github or pastebin

Wayne_Spivak · February 12, 2025, 11:33am

Sent via Gist. with notes.

BTW, from another cron job I got these errors:

2025/02/12 06:36:08 ERROR : Dropbox root 'Backups/alphabackup/crontabs': sync batch commit: failed to commit batch length 3: batch had 3 errors: last error: upload failed: too_many_write_operations

2025/02/12 06:36:08 ERROR : php: Failed to copy: upload failed: batch upload failed: upload failed: too_many_write_operations

2025/02/12 06:36:08 ERROR : SkywarnPlus: Failed to copy: upload failed: batch upload failed: upload failed: too_many_write_operations

2025/02/12 06:36:08 ERROR : mdadm: Failed to copy: upload failed: batch upload failed: upload failed: too_many_write_operations

2025/02/12 06:36:08 ERROR : Dropbox root 'Backups/alphabackup/crontabs': not deleting files as there were IO errors

2025/02/12 06:36:08 ERROR : Dropbox root 'Backups/alphabackup/crontabs': not deleting directories as there were IO errors

2025/02/12 06:36:08 ERROR : Attempt 1/3 failed with 3 errors and: upload failed: batch upload failed: upload failed: too_many_write_operations

2025/02/12 06:36:11 ERROR : Attempt 2/3 succeeded

A bunch of these that make perfect sense:

2025/02/12 06:37:01 NOTICE: sites-enabled/default-ssl.conf: Can't follow symlink without -L/--copy-links

Some of these:

2025/02/12 06:39:07 ERROR : Dropbox root 'Backups/alphabackup/local-etc': sync batch commit: failed to commit batch length 4: batch had 4 errors: last error: upload failed: too_many_write_operations

The cron jobs.

30 6 * * * rclone sync  /etc/allmon3 Dropbox:Backups/alphabackup/allmon3
31 6 * * * rclone sync  /etc/asterisk/ Dropbox:Backups/alphabackup/asterisk
32 6 * * * rclone sync  /opt Dropbox:Backups/alphabackup/dvswitch --exclude-from /opt/exclude.list
33 6 * * * rclone sync  /etc/init.d Dropbox:Backups/alphabackup/init.d
34 6 * * * rclone sync  /home/wbs Dropbox:Backups/alphabackup/wbs --exclude-from /home/wbs/exclude.list
35 6 * * * rclone sync  /var/www/html/ Dropbox:Backups/alphabackup/website
36 6 * * * rclone sync  /etc/cron.d Dropbox:Backups/alphabackup/crontabs

37 6 * * * rclone sync  /etc/apache2 Dropbox:Backups/alphabackup/apache

38 6 * * * rclone sync  /usr/local/bin Dropbox:Backups/alphabackup/local-bin
39 6 * * * rclone sync  /usr/local/etc Dropbox:Backups/alphabackup/local-etc
40 6 * * * rclone sync /usr/local/bin/SkywarnPlus  Dropbox:Backups/alphabackup/skywarn

kapitainsky · February 12, 2025, 11:34am

Good. It would also help if you share link to it here.

Wayne_Spivak · February 12, 2025, 7:07pm

Here is the link
Gist file

I also created and modified my config by getting a Dropbox ID... (hopefully done corectly)...

kapitainsky · February 12, 2025, 7:12pm

Now can you give me an example of a file which was uploaded and then deleted in the same sync run? As because rclone sync deletes files is not an issue but sync feature. Sync makes your destination the same as source, deleting what is not present in the source.

For throttling errors you should:

add client_id/secret - but you already did it - good
add --tpslimit 12 --tpslimit-burst 0 to your sync command
do not run multiple syncs at the same time (as it will easily trigger throttling). Instead of starting them in your cron pretty much simultaneously put them all in some script so they can run one by one. Or create one sync command with include filter. Even better use filter-from flag and put all your include and exclude in a file.

Wayne_Spivak · February 12, 2025, 8:48pm

I did, but did something wrong, so recreated the id. Per instructions I read, I used the generated access token from the Developers site as the token.

This is the error:

rclone -vv ls Dropbox:Backup
2025/02/12 15:44:14 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "-vv" "ls" "Dropbox:Backup"]
2025/02/12 15:44:14 DEBUG : Creating backend with remote "Dropbox:Backup"
2025/02/12 15:44:14 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2025/02/12 15:44:15 ERROR : : error listing: Error in call to API function "files/list_folder": This app is currently disabled.
2025/02/12 15:44:15 DEBUG : 6 go routines active
2025/02/12 15:44:15 INFO : Dropbox root 'Backup': Committing uploads - please wait...
2025/02/12 15:44:15 NOTICE: Failed to ls with 2 errors: last error was: Error in call to API function "files/list_folder": This app is currently disabled.

See image:

add --tpslimit 12 --tpslimit-burst 0 to your sync command
do not run multiple syncs at the same time (as it will easily trigger throttling). Instead of starting them in your cron pretty much simultaneously put them all in some script so they can run one by one. Or create one sync command with include filter. Even better use filter-from flag and put all your include and exclude in a file.
[/quote]
That makes sense...
But first need to solve the ID issue

UPDATE
Fixed the ClientID issue (I think)

Thanks

kapitainsky · February 12, 2025, 10:21pm

In case you just insert ID into your config you have to run rclone config reconnect Dropbox: in order to "activate" it. Otherwise it does nothing..

Wayne_Spivak · February 12, 2025, 10:38pm

Had another issue that I fixed, and it seems to have "taken" the ID. Killed the old ClientID and Dropbox config, started from scratch.

Uploaded a directory, looked good.
Uploaded a second directory, now its deleting what looks like everything (it will be a while until all the files/folders are gone...By everything in the Dropbox:Backups folder which is being populated by rclone.
Will send debug file when done.

Thx's again!@

Wayne_Spivak · February 12, 2025, 10:47pm

Deleted almost everything:

2nd Logfile

kapitainsky · February 12, 2025, 11:14pm

Can you post all three full sync commands you run. I strongly suspect that you overlap destinations and delete what you previously uploaded.

What you should do:

rclone sync dir1 Dropbox:Backup/dir1
rclone sync dir2 Dropbox:Backup/dir2
rclone sync dir3 Dropbox:Backup/dir3

When you do for example:

rclone sync dir1 Dropbox:Backup
rclone sync dir2 Dropbox:Backup/dir2

then your first command will delete all previously uploaded dir2 data.

You trip over your own legs pretty much.

Wayne_Spivak · February 12, 2025, 11:43pm

I see what I may be doing...

Maybe sync the entire directory tree and use exclude?

 rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP/ALLSTARbackup/Allstar Dropbox:Backups
 rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP/BetaSTARbackup/Beta.sbanetweb.com Dropbox:Backups
 rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP/nas/NAS Dropbox:Backups

 rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP/MCQ-backup/snapshots Dropbox:Backups/MCQ
 rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP/MCQ-tar/ Dropbox:Backups/Tar-backup

kapitainsky · February 12, 2025, 11:55pm

Good we've got to the bottom of it.

Yes - either sync all tree and use excludes or arrange everything manually explicately stating destination directories (different for every sync)

rclone sync dir1 Dropbox:Backup/dir1dest
rclone sync dir2 Dropbox:Backup/dir2dest
rclone sync dir3 Dropbox:Backup/dir3dest

Wayne_Spivak · February 13, 2025, 12:24am

How does this look?

The raid mount /mnt/raid10/ looks like this (dir in bold):

BACKUP
CFO
._.DS_Store
.DS_Store
exclude.list
HotspotAllStar
HotspotBeta
lost+found
website

rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/   Dropbox:CFO --exclude-from=/mnt/raid10/exclude.list

rclone rsync --tpslimit 12 --tpslimit-burst 0 /mnt/raid10/BACKUP Dropbox:BACKUPS

The exclude.list:
/BACKUP/**
/lost+found/**

kapitainsky · February 13, 2025, 6:18am

Nothing overlaps now. So this should work.

Wayne_Spivak · February 14, 2025, 3:51pm

Btw, 2 Q's:

a) it seems that it is taking forever to upload, where as the Windows Dropbox program would have uploaded the data by now.

I'm uploading 241GB on a FIOS business-class connection. 117.2 Mbps download speed and 309.5 Mbps upload speed. To a New York Server (depending on where Dropox is sending may be slower).

b) I lost my connection from my winbox to the NAS, so had to restart the process (it is running in foreground) and It looks like it is uploading files it already uploaded (or may not be).

There seems to be

no duplication,
most of these files have never been touched, so the time stamps should not have changed, even if I rclone from my server(s) to the NAS, and the NAS to Dropbox,
they are saying "Copied (new)", so it is possible my assumption may be incorrect as these directories never made it up...

I have seen other messages in prior tests that say timestamp wrong and recopied. Not sure of the message of "no-change".

Thanks again... and I realize "time factor"

kapitainsky · February 14, 2025, 4:54pm

re a)
This is some wild speculation only. Could you be more specific? Maybe some numbers and details? As e.g. uploading small files is always much slower than big file.

re b)
Not sure what you expect others to comment when you finish your story with:

If you are not sure how anybody else can be? Especially that you do not provide any evidence.

If there is some issue please post some data. Logfile would be a good start...

Wayne_Spivak · February 14, 2025, 5:07pm

Still uploading (at least part "A")...

I will see when messages from the 2nd half of the rclone sync commands (see prior message). Currently uploading 2GB and running 2hrs...

Wayne_Spivak · February 15, 2025, 1:46am

It too a little over 8 hrs for 2GB to upload. Part "B" is showing files that have already been uploaded, so rclone/Dropbox seems to be working properly....

So the issue is still speed....

kapitainsky · February 15, 2025, 5:53am

Indeed it seems a bit slow... But maybe it is normal under your specific configuration and transfer profile.

I am not Dropbox user so have nothing to compare it to.

It would be great if somebody with Dropbox experience could help here.