File checks with Dropbox on MacOS takes forever

VincentP · December 21, 2021, 3:35pm

What is the problem you are having with rclone?

When using the "sync" command to refresh my backup, on a folder structure which has around 45k files, rclone is not done after 8h, having checked more than 100k files. I looked at the -vv log, and I see that files are being checked 3 times (at least), and yet the sync never ends. I do not know whether all files have been checked though.

This computer is running macOS 11.6.1, Intel processor.

The same command on my Windows 10 PC completes in reasonable times, log files show that each file is only ever checked once. However, my PC does not have as many files, the most I have in a "batch" is 18k files, that completes in 1 to 5 minutes, if there are not too many modified or new files to upload.

What is your rclone version (output from `rclone version`)

rclone v1.57.0

os/version: darwin 11.6.1 (64 bit)
os/kernel: 20.6.0 (x86_64)
os/type: darwin
os/arch: amd64
go/version: go1.17.2
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Dropbox, together with the crypt system (so saving to crypt, which is a specific folder on my Dropbox)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync $SOURCE_PATH $DESTINATION_PATH -P \
   --backup-dir $ARCHIVE_DESTINATION_PATH \
   --exclude-from ~/rclone/exclude.txt \
   --skip-links \
   --dropbox-batch-mode sync \
   --tpslimit 12 --tpslimit-burst 12 --log-file testlog.txt --log-level DEBUG

The rclone config contents with secrets removed.

[qtdropbox]
type = dropbox
token = {"access_token":"xxxx","token_type":"bearer","refresh_token":"xxxx","expiry":"2021-12-21T08:21:01.469992-05:00"}
client_id = xxxx (my own ID)
client_secret = xxxx (the associated secret)

[qtcrypt]
type = crypt
remote = qtdropbox:/xBackups/macbook/
filename_encryption = off
directory_name_encryption = false
password = xxxx

A log from the command with the `-vv` flag

This is a very truncated log, but the log only has entries of this type anyways.

2021/12/20 23:39:19 INFO  : Starting transaction limiter: max 12 transactions/s with burst 12
2021/12/20 23:39:23 INFO  : rclone/backup.sh: Deleted
2021/12/20 23:39:25 INFO  : rclone/backup.sh: Moved (server-side)
2021/12/20 23:39:31 INFO  : rclone/backup.sh: Copied (new)
2021/12/20 23:40:01 INFO  : Signal received: interrupt
2021/12/20 23:40:01 INFO  : Dropbox root 'xBackups/macbook/home.bin': Commiting uploads - please wait...
2021/12/20 23:40:01 INFO  : Dropbox root 'xBackups/macbook/home': Commiting uploads - please wait...
2021/12/20 23:40:01 INFO  : Dropbox root 'xBackups/macbook/MacBook_archive.bin': Commiting uploads - please wait...
2021/12/20 23:40:01 INFO  : Dropbox root 'xBackups/macbook/MacBook_archive': Commiting uploads - please wait...
2021/12/20 23:40:01 INFO  : Exiting...
2021/12/20 23:41:00 INFO  : Starting transaction limiter: max 12 transactions/s with burst 12
2021/12/20 23:41:04 INFO  : rclone/exclude.txt: Deleted
2021/12/20 23:41:06 INFO  : rclone/exclude.txt: Moved (server-side)
2021/12/20 23:41:13 INFO  : rclone/exclude.txt: Copied (new)
2021/12/20 23:43:19 INFO  : Signal received: interrupt
2021/12/20 23:43:19 INFO  : Dropbox root 'xBackups/macbook/MacBook_archive.bin': Commiting uploads - please wait...
2021/12/20 23:43:19 INFO  : Dropbox root 'xBackups/macbook/MacBook_archive': Commiting uploads - please wait...
2021/12/20 23:43:19 INFO  : Dropbox root 'xBackups/macbook/home.bin': Commiting uploads - please wait...
2021/12/20 23:43:19 INFO  : Dropbox root 'xBackups/macbook/home': Commiting uploads - please wait...
2021/12/20 23:43:19 INFO  : Exiting...
2021/12/20 23:43:47 INFO  : Starting transaction limiter: max 12 transactions/s with burst 12
2021/12/20 23:43:47 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "sync" "/Users/myname/" "qtcrypt:home" "-P" "--backup-dir" "qtcrypt:MacBook_archive" "--exclude-from" "/Users/myname/rclone/exclude.txt" "--skip-links" "--dropbox-batch-mode" "sync" "--tpslimit" "12" "--tpslimit-burst" "12" "--log-file" "testlog.txt" "--log-level" "DEBUG"]
2021/12/20 23:43:47 DEBUG : Creating backend with remote "/Users/myname/"
2021/12/20 23:43:47 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2021/12/20 23:43:47 DEBUG : Using config file from "/Users/vincentpelletier/.config/rclone/rclone.conf"
2021/12/20 23:43:47 DEBUG : fs cache: renaming cache item "/Users/vincentpelletier/" to be canonical "local{HK82T}:/Users/vincentpelletier/"
2021/12/20 23:43:47 DEBUG : Creating backend with remote "qtcrypt:home"
2021/12/20 23:43:47 DEBUG : Creating backend with remote "qtdropbox:/xBackups/macbook/home.bin"
2021/12/20 23:43:47 DEBUG : qtdropbox: detected overridden config - adding "{bBI1F}" suffix to name
2021/12/20 23:43:48 DEBUG : Dropbox root '': Using root namespace "45943168"
2021/12/20 23:43:48 DEBUG : fs cache: renaming cache item "qtdropbox:/xBackups/macbook/home.bin" to be canonical "qtdropbox{bBI1F}:xBackups/macbook/home.bin"
2021/12/20 23:43:48 DEBUG : Creating backend with remote "qtdropbox:/xBackups/macbook/home"
2021/12/20 23:43:48 DEBUG : qtdropbox: detected overridden config - adding "{bBI1F}" suffix to name
2021/12/20 23:43:48 DEBUG : Dropbox root '': Using root namespace "45943168"
2021/12/20 23:43:49 DEBUG : fs cache: renaming cache item "qtdropbox:/xBackups/macbook/home" to be canonical "qtdropbox{bBI1F}:xBackups/macbook/home"
2021/12/20 23:43:49 DEBUG : Creating backend with remote "qtcrypt:MacBook_archive"
2021/12/20 23:43:49 DEBUG : Creating backend with remote "qtdropbox:/xBackups/macbook/MacBook_archive.bin"
2021/12/20 23:43:49 DEBUG : qtdropbox: detected overridden config - adding "{bBI1F}" suffix to name
2021/12/20 23:43:49 DEBUG : Dropbox root '': Using root namespace "45943168"
2021/12/20 23:43:49 DEBUG : fs cache: renaming cache item "qtdropbox:/xBackups/macbook/MacBook_archive.bin" to be canonical "qtdropbox{bBI1F}:xBackups/macbook/MacBook_archive.bin"
2021/12/20 23:43:49 DEBUG : Creating backend with remote "qtdropbox:/xBackups/macbook/MacBook_archive"
2021/12/20 23:43:49 DEBUG : qtdropbox: detected overridden config - adding "{bBI1F}" suffix to name
2021/12/20 23:43:50 DEBUG : Dropbox root '': Using root namespace "45943168"
2021/12/20 23:43:50 DEBUG : fs cache: renaming cache item "qtdropbox:/xBackups/macbook/MacBook_archive" to be canonical "qtdropbox{bBI1F}:xBackups/macbook/MacBook_archive"
2021/12/20 23:43:50 DEBUG : testlog.txt: Excluded
2021/12/20 23:43:50 DEBUG : .config: Excluded
2021/12/20 23:43:50 DEBUG : Temp: Excluded
2021/12/20 23:43:50 DEBUG : .DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : .CFUserTextEncoding: Excluded
2021/12/20 23:43:50 DEBUG : .local: Excluded
2021/12/20 23:43:50 DEBUG : .webex: Excluded
2021/12/20 23:43:50 DEBUG : .zsh_history: Excluded
2021/12/20 23:43:50 DEBUG : Library: Excluded
2021/12/20 23:43:50 DEBUG : .oracle_jre_usage: Excluded
2021/12/20 23:43:50 DEBUG : .cups: Excluded
2021/12/20 23:43:50 DEBUG : .bash_sessions: Excluded
2021/12/20 23:43:50 DEBUG : .dropbox: Excluded
2021/12/20 23:43:50 DEBUG : .Trash: Excluded
2021/12/20 23:43:50 DEBUG : .anydesk: Excluded
2021/12/20 23:43:50 DEBUG : .mono: Excluded
2021/12/20 23:43:50 DEBUG : .imacro: Excluded
2021/12/20 23:43:50 DEBUG : .Xauthority: Excluded
2021/12/20 23:43:50 DEBUG : Downloads: Excluded
2021/12/20 23:43:50 DEBUG : .bash_history: Excluded
2021/12/20 23:43:50 DEBUG : .viminfo: Excluded
2021/12/20 23:43:50 DEBUG : .zsh_sessions: Excluded
2021/12/20 23:43:50 DEBUG : .islonline: Excluded
2021/12/20 23:43:50 DEBUG : Library: Excluded
2021/12/20 23:43:50 DEBUG : CDM v2.08.30 WHQL Certified.exe: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : CDM v2.08.30 WHQL Certified.exe: Unchanged skipping
2021/12/20 23:43:50 DEBUG : huygens_msg.txt: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : huygens_msg.txt: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Desktop/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Desktop/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Music/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Music/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Music/iTunes: Excluded
2021/12/20 23:43:50 DEBUG : Movies/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Movies/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Documents/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Documents/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Documents/temp test: Excluded
2021/12/20 23:43:50 DEBUG : Pictures/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Pictures/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Pictures/Photos Library.photoslibrary: Excluded
2021/12/20 23:43:50 DEBUG : Desktop/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Google Drive/Icon␍: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Google Drive/late_stage.tif: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Google Drive/late_stage.tif: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Google Drive/Icon␍: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Desktop/anydesk00000.png: Size and modification time the same (differ by -296.229919ms, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Desktop/anydesk00000.png: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Public/.com.apple.timemachine.supported: Excluded
2021/12/20 23:43:50 DEBUG : Public/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Public/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Mail/PersistenceInfo.plist: Excluded
2021/12/20 23:43:50 DEBUG : Mail/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Music/iTunes: Excluded
2021/12/20 23:43:50 DEBUG : Music/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Sites/.localized: Excluded
2021/12/20 23:43:50 DEBUG : Thunderbird/.DS_Store: Excluded
2021/12/20 23:43:50 DEBUG : Documents/.localized: Excluded
(...)
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Documents/License_destripe.rtf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Documents/License_destripe.rtf: Unchanged skipping
(...)
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A.pdf: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Unchanged skipping
2021/12/20 23:43:50 DEBUG : Documents/VT-iSIM System installation procedure v1.8.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/20 23:43:50 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Unchanged skipping

Then much later in the same log:

2021/12/21 02:47:51 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/21 02:47:51 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Unchanged skipping
2021/12/21 02:47:51 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/21 02:47:51 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Unchanged skipping
2021/12/21 02:47:51 DEBUG : Documents/License_destripe.rtf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/21 02:47:51 DEBUG : Documents/License_destripe.rtf: Unchanged skipping

Then again much later in the same log:

2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A.pdf: Unchanged skipping
(...)
2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A-3.pdf: Unchanged skipping
2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
(...)
2021/12/21 05:51:52 DEBUG : Documents/Contrat_Fondatechnique_213252-A-2.pdf: Unchanged skipping

Animosity022 · December 21, 2021, 3:49pm

Without a full log, it's really just guessing so not sure what else to offer.

VincentP · December 21, 2021, 3:57pm

Thank you for your input. The log is 40MB, 200k lines of the same as above, but I am reluctant to include them all because the file names could contain identifiable information for me, my contacts, customers, etc. So I do not really want to share the full log on a public forum.

The start of the log is all there in my post. After that, all I am seeing are line pairs saying "size and modification time the same", followed by "unchanged, skipping", with the occasional "excluded" item.

I do understand that the key could be the one line in the 200k lines which is different. I will have a look and post anything strange I find.

Animosity022 · December 21, 2021, 4:00pm

Sounds good. At this point then, nothing to offer as it is in your hands.

asdffdsa · December 21, 2021, 4:06pm

hello and welcome to the forum,

in the debug log, might search for ERROR

Ole · December 21, 2021, 4:51pm

Hi VincentP,

I few quick observations:

My guess that there are ERRORs in your log. They will then make rclone attempt the entire sync up to 3 retries before giving up. That will result in 3 x 45k = 135 k checks. You can change the number of --retries before giving up.

You have requested a drastically reduced speed by using --tps-limit=12, what is the reason for this? (The default speed is 100 TPS)

How many files are you excluding with by --exclude-from? (the more the slower your sync)

Animosity022 · December 21, 2021, 4:51pm

Dropbox has quota limits as this is the recommended value based on testing.

Ole · December 21, 2021, 4:59pm

@Animosity022 Ok, thanks!

VincentP · December 21, 2021, 5:09pm

I am guessing the best way to troubleshoot is to go deep enough that I can explain the situation. There are some strange items in the giant log, and I believe I understand too why the "checked" counter sometimes crawl to a stop:

I was trying to backup the entire "home" folder (~/) on my mac, but excluding some folders such as the hidden "~/Library/" folder. However, for some reason, I think rclone is checking every single file in every folder, including the excluded ones. I think my exclusion is built to ignore every file which has that folder as its root, instead or skipping the folder (and its subfolder) in one quick go.

So instead I restructured my backup to sync, one by one, the folders I wanted included. I figure that I have the wrong way to exclude folders in my command.

Here is the content of my old exclusion file:

.**
Temp/**
Library/**
Downloads/**
Photos Library.photoslibrary
temp test/**
*.plist
iTunes/**
Pictures/Photos Library**

The command is:

rclone sync $SOURCE_PATH $DESTINATION_PATH -P \
   --backup-dir $ARCHIVE_DESTINATION_PATH \
   --exclude-from ~/rclone/exclude.txt \
   --skip-links \
   --dropbox-batch-mode sync \
   --tpslimit 12 --tpslimit-burst 12 --log-file testlog.txt --log-level DEBUG

In the log file, there is one major error event, repeated 3 times:

2021/12/21 01:03:57 ERROR : Mail/V8/4F74EEA5-4FD7-43A3-B34C-0F02FD5B0845/Archive 2020.mbox/Sent 2020.mbox/FB501670-1F10-45B8-9DF8-4F4337FB0955/Data/9/3/Attachments/39257/2.2: error reading destination directory: <!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Dropbox - 5xx</title>
<link href="https://cfl.dropboxstatic.com/static/css/error.css" rel="stylesheet" type="text/css"/>
<link rel="shortcut icon" href="https://cfl.dropboxstatic.com/static/images/favicon.ico"/>

</head>
<body>
<div class="figure">
<img src="https://cfl.dropboxstatic.com/static/images/illustration_catalog/sickbox-illo_m1.png" srcset="https://cfl.dropboxstatic.com/static/images/illustration_catalog/sickbox-illo_m1@2x.png 2x" alt="Error: 5xx"/>
</div>
<div id="errorbox">
<h1>Error (5xx)</h1>Something went wrong. Don't worry, your files are still safe and the Dropboxers have been notified. Check out our <a href="https://www.dropbox.com/help">Help Center</a> and <a href="https://forums.dropbox.com">forums</a> for help, or head back to <a href="https://www.dropbox.com/home">home</a>.
</div>

<script>
message = {"ru": "\u003ch1\u003e\u041e\u0448\u0438\u0431\u043a\u0430 (5xx)\u003c/h1\u003e\u041f\u0440\u043e\u0438\u0437\u043e\u0448\u043b\u0430 \u043d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430. \u041d\u0435 \u0431\u0435\u0441\u043f\u043e\u043a\u043e\u0439\u0442\u0435\u0441\u044c, \u0432\u0430\u0448\u0438 \u0444\u0430\u0439\u043b\u044b \u043d\u0435 \u043f\u043e\u0441\u0442\u0440\u0430\u0434\u0430\u043b\u0438, \u0430 \u0441\u043e\u0442\u0440\u0443\u0434\u043d\u0438\u043a\u0438 Dropbox \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0435. \u041f\u0435\u0440\u0435\u0439\u0434\u0438\u0442\u0435 \u0432 \u003ca href=\"https://www.dropbox.com/help\"\u003e\u0441\u043f\u0440\u0430\u0432\u043e\u0447\u043d\u044b\u0439 \u0446\u0435\u043d\u0442\u0440\u003c/a\u003e \u0438\u043b\u0438 \u043d\u0430 \u003ca href=\"https://forums.dropbox.com\"\u003e\u0444\u043e\u0440\u0443\u043c\u044b\u003c/a\u003e \u0434\u043b\u044f \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u044f \u043f\u043e\u043c\u043e\u0449\u0438 \u0438\u043b\u0438 \u0432\u0435\u0440\u043d\u0438\u0442\u0435\u0441\u044c \u043d\u0430 \u003ca href=\"https://www.dropbox.com/home\"\u003e\u0434\u043e\u043c\u0430\u0448\u043d\u044e\u044e \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0443\u003c/a\u003e.", "fr": "\u003ch1\u003eErreur (5xx)\u003c/h1\u003eUn probl\u00e8me est survenu. Ne vous inqui\u00e9tez pas, vos fichiers sont toujours en s\u00e9curit\u00e9 et les Dropboxeurs ont \u00e9t\u00e9 avertis. Consultez notre \u003ca href=\"https://www.dropbox.com/help\"\u003ecentre d'assistance\u003c/a\u003e et nos \u003ca href=\"https://forums.dropbox.com\"\u003eforums\u003c/a\u003e pour obtenir de l'aide, ou retournez sur la \u003ca href=\"https://www.dropbox.com/home\"\u003epage d'accueil\u003c/a\u003e.", "zh_TW": "\u003ch1\u003e\u932f\u8aa4 (5xx)\u003c/h1\u003e\u767c\u751f\u932f\u8aa4\u3002\u5225\u64d4\u5fc3\uff0c\u60a8\u7684\u6a94\u6848\u5b89\u5168\u7121\u865e\uff0c\u5df2\u901a\u77e5 Dropbox \u54e1\u5de5\u3002\u8acb\u5230\u6211\u5011\u7684\u003ca href=\"https://www.dropbox.com/help\"\u003e\u8aaa\u660e\u4e2d\u5fc3\u003c/a\u003e\u548c\u003ca href=\"https://forums.dropbox.com\"\u003e\u8a0e\u8ad6\u5340\u003c/a\u003e\u53d6\u5f97\u5354\u52a9\uff0c\u6216\u662f\u56de\u5230\u003ca href=\"https://www.dropbox.com/home\"\u003e\u9996\u9801\u003c/a\u003e\u3002", "th_TH": "\u003ch1\u003e\u0e02\u0e49\u0e2d\u0e1c\u0e34\u0e14\u0e1e\u0e25\u0e32\u0e14 5xx\u003c/h1\u003e\u0e21\u0e35\u0e1a\u0e32\u0e07\u0e2d\u0e22\u0e48\u0e32\u0e07\u0e17\u0e35\u0e48\u0e44\u0e21\u0e48\u0e16\u0e39\u0e01\u0e15\u0e49\u0e2d\u0e07 \u0e44\u0e21\u0e48\u0e15\u0e49\u0e2d\u0e07\u0e40\u0e1b\u0e47\u0e19\u0e2b\u0e48\u0e27\u0e07 \u0e44\u0e1f\u0e25\u0e4c\u0e02\u0e2d\u0e07\u0e04\u0e38\u0e13\u0e22\u0e31\u0e07\u0e1b\u0e25\u0e2d\u0e14\u0e20\u0e31\u0e22\u0e41\u0e25\u0e30 Dropboxer \u0e44\u0e14\u0e49\u0e23\u0e31\u0e1a\u0e01\u0e32\u0e23\u0e41\u0e08\u0e49\u0e07\u0e40\u0e15\u0e37\u0e2d\u0e19\u0e41\u0e25\u0e49\u0e27 \u0e42\u0e1b\u0e23\u0e14\u0e14\u0e39\u0e23\u0e32\u0e22\u0e25\u0e30\u0e40\u0e2d\u0e35\u0e22\u0e14\u0e17\u0e35\u0e48 \u003ca href=\"https://www.dropbox.com/help\"\u003e\u0e28\u0e39\u0e19\u0e22\u0e4c\u0e04\u0e27\u0e32\u0e21\u0e0a\u0e48\u0e27\u0e22\u0e40\u0e2b\u0e25\u0e37\u0e2d\u003c/a\u003e \u0e41\u0e25\u0e30 \u003ca href=\"https://forums.dropbox.com\"\u003e\u0e1f\u0e2d\u0e23\u0e31\u0e21\u003c/a\u003e \u0e40\u0e1e\u0e37\u0e48\u0e2d\u0e02\u0e2d\u0e04\u0e27\u0e32\u0e21\u0e0a\u0e48\u0e27\u0e22\u0e40\u0e2b\u0e25\u0e37\u0e2d \u0e2b\u0e23\u0e37\u0e2d\u0e01\u0e25\u0e31\u0e1a\u0e44\u0e1b\u0e17\u0e35\u0e48 \u003ca href=\"https://www.dropbox.com/home\"\u003e\u0e2b\u0e19\u0e49\u0e32\u0e41\u0e23\u0e01\u003c/a\u003e", "pt_BR": "\u003ch1\u003eErro ( 5xx ).\u003c/h1\u003eDeu algo errado, mas n\u00e3o se preocupe. Os seus arquivos permanecem em seguran\u00e7a, e a equipe do Dropbox j\u00e1 foi notificada. Confira nossa \u003ca href=\"https://www.dropbox.com/help\"\u003eCentral de ajuda\u003c/a\u003e e \u003ca href=\"https://forums.dropbox.com\"\u003enossos f\u00f3runs\u003c/a\u003e para obter ajuda ou volte para a \u003ca href=\"https://www.dropbox.com/home\"\u003ep\u00e1gina inicial\u003c/a\u003e.", "zh_CN": "\u003ch1\u003e\u9519\u8bef (5xx)\u003c/h1\u003e \u51fa\u73b0\u95ee\u9898\u3002\u522b\u62c5\u5fc3\uff0c\u60a8\u7684\u6587\u4ef6\u4ecd\u5b89\u5168\uff0c\u800c\u4e14 Dropbox \u5458\u5de5\u5df2\u6536\u5230\u901a\u77e5\u3002\u8bf7\u67e5\u9605\u6211\u4eec\u7684\u003ca href=\"https://www.dropbox.com/help\"\u003e\u5e2e\u52a9\u4e2d\u5fc3\u003c/a\u003e\u548c \u003ca href=\"https://forums.dropbox.com\"\u003e\u8bba\u575b\u003c/a\u003e\u83b7\u53d6\u5e2e\u52a9\uff0c\u6216\u8fd4\u56de \u003ca href=\"https://www.dropbox.com/home\"\u003e\u4e3b\u9875\u003c/a\u003e\u3002", "da_DK": "\u003ch1\u003eFejl (5xx)\u003c/h1\u003e Noget gik galt. Men ingen grund til bekymring, dine filer er sikre, og Dropboxerne har f\u00e5et besked. Tjek vores \u003ca href=\"https://www.dropbox.com/help\"\u003eHj\u00e6lpecenter\u003c/a\u003e og \u003ca href=\"https://forums.dropbox.com\"\u003eforummer\u003c/a\u003e for hj\u00e6lp, eller g\u00e5 tilbage til \u003ca href=\"https://www.dropbox.com/home\"\u003estart\u003c/a\u003e.", "de": "\u003ch1\u003eFehler (5xx)\u003c/h1\u003eEtwas ist schiefgelaufen. Ihre Dateien sind aber immer noch sicher und Dropbox wurde benachrichtigt. Schauen Sie sich im \u003ca href=\"https://www.dropbox.com/help\"\u003eHilfecenter\u003c/a\u003e und in den \u003ca href=\"https://forums.dropbox.com\"\u003eForen\u003c/a\u003e um, wenn Sie Hilfe ben\u00f6tigen, oder kehren Sie zur\u00fcck auf die \u003ca href=\"https://www.dropbox.com/home\"\u003eStartseite\u003c/a\u003e.", "ko": "\u003ch1\u003e\uc624\ub958(5xx)\u003c/h1\u003e\ubb38\uc81c\uac00 \ubc1c\uc0dd\ud588\uc2b5\ub2c8\ub2e4. \ud30c\uc77c\uc740 \uc548\uc804\ud558\uac8c \ubcf4\uad00\ub418\uc5b4 \uc788\uc73c\ub2c8 \uc548\uc2ec\ud558\uc138\uc694. \uc774 \ubb38\uc81c\ub294 Dropbox \ub2f4\ub2f9\uc790\uc5d0\uac8c \uc804\ub2ec\ub418\uc5c8\uc73c\uba70, \ub3c4\uc6c0\uc774 \ud544\uc694\ud55c \uacbd\uc6b0 Dropbox\uc758 \u003ca href=\"https://www.dropbox.com/help\"\u003e\ub3c4\uc6c0\ub9d0 \uc13c\ud130\u003c/a\u003e \ubc0f \u003ca href=\"https://forums.dropbox.com\"\u003e\ud3ec\ub7fc\u003c/a\u003e\uc744 \ucc38\uc870\ud558\uac70\ub098 \u003ca href=\"https://www.dropbox.com/home\"\u003e\ud648\u003c/a\u003e\uc73c\ub85c \ub3cc\uc544\uac00\uc138\uc694.", "it": "\u003ch1\u003eErrore (5xx)\u003c/h1\u003eQualcosa \u00e8 andato storto. Non ti preoccupare i tuoi file sono comunque al sicuro e i Dropboxer sono stati notificati. Consulta il \u003ca href=\"https://www.dropbox.com/help\"\u003eCentro assistenza\u003c/a\u003e e \u003ca href=\"https://forums.dropbox.com\"\u003ei forum\u003c/a\u003e per assistenza oppure torna alla \u003ca href=\"https://www.dropbox.com/home\"\u003ehomepage\u003c/a\u003e.", "ja": "\u003ch1\u003e\u30a8\u30e9\u30fc\uff085xx\uff09\u304c\u767a\u751f\u3057\u307e\u3057\u305f\u3002\u003c/h1\u003e\u30d5\u30a1\u30a4\u30eb\u306f\u5b89\u5168\u3067\u3059\u306e\u3067\u3054\u5b89\u5fc3\u304f\u3060\u3055\u3044\u3002Dropbox \u306b\u306f\u65e2\u306b\u901a\u77e5\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u003ca href=\"https://www.dropbox.com/help\"\u003e\u30d8\u30eb\u30d7\u30bb\u30f3\u30bf\u30fc\u003c/a\u003e\u3084\u003ca href=\"https://forums.dropbox.com\"\u003e\u30d5\u30a9\u30fc\u30e9\u30e0\u003c/a\u003e\u3092\u53c2\u7167\u3059\u308b\u304b\u3001[\u003ca href=\"https://www.dropbox.com/home\"\u003e\u30db\u30fc\u30e0\u003c/a\u003e] \u30da\u30fc\u30b8\u306b\u623b\u3063\u3066\u304f\u3060\u3055\u3044\u3002", "uk_UA": "\u003ch1\u003e\u041f\u043e\u043c\u0438\u043b\u043a\u0430 (5xx)\u003c/h1\u003e\u0429\u043e\u0441\u044c \u043f\u0456\u0448\u043b\u043e \u043d\u0435 \u0442\u0430\u043a. \u041d\u0435 \u0445\u0432\u0438\u043b\u044e\u0439\u0442\u0435\u0441\u044f, \u0432\u0430\u0448\u0456 \u0444\u0430\u0439\u043b\u0438 \u0432 \u0431\u0435\u0437\u043f\u0435\u0446\u0456, \u0430 \u043f\u0440\u0430\u0446\u0456\u0432\u043d\u0438\u043a\u0438 Dropbox \u0443\u0436\u0435 \u0432 \u043a\u0443\u0440\u0441\u0456 \u043f\u043e\u0434\u0456\u0439. \u041f\u0435\u0440\u0435\u0433\u043b\u044f\u043d\u044c\u0442\u0435 \u003ca href=\"https://www.dropbox.com/help\"\u003e\u0426\u0435\u043d\u0442\u0440 \u0434\u043e\u0432\u0456\u0434\u043a\u0438\u003c/a\u003e \u0442\u0430 \u003ca href=\"https://forums.dropbox.com\"\u003e\u0444\u043e\u0440\u0443\u043c\u0438\u003c/a\u003e \u0430\u0431\u043e \u043f\u043e\u0432\u0435\u0440\u043d\u0456\u0442\u044c\u0441\u044f \u043d\u0430 \u003ca href=\"https://www.dropbox.com/home\"\u003e\u0433\u043e\u043b\u043e\u0432\u043d\u0443 \u0441\u0442\u043e\u0440\u0456\u043d\u043a\u0443\u003c/a\u003e.", "nl_NL": "\u003ch1\u003eFout (5xx).\u003c/h1\u003eEr is iets misgegaan. Maak je geen zorgen, je bestanden zijn veilig en de Dropboxers zijn op de hoogte gesteld. Kijk in ons \u003ca href=\"https://www.dropbox.com/help\"\u003eHelpcentrum\u003c/a\u003e en de \u003ca href=\"https://forums.dropbox.com\"\u003eforums\u003c/a\u003e voor hulp of ga terug naar de \u003ca href=\"https://www.dropbox.com/home\"\u003estartpagina\u003c/a\u003e.", "pl": "\u003ch1\u003eB\u0142\u0105d (5xx)\u003c/h1\u003eWyst\u0105pi\u0142y problemy. Nie martw si\u0119, Twoje pliki s\u0105 nadal bezpieczne, a\u00a0pracownicy us\u0142ugi Dropbox zostali powiadomieni. Odwied\u017a \u003ca href=\"https://www.dropbox.com/help\"\u003eCentrum pomocy\u003c/a\u003e i\u00a0\u003ca href=\"https://forums.dropbox.com\"\u003efora\u003c/a\u003e, aby uzyska\u0107 pomoc, lub przejd\u017a do \u003ca href=\"https://www.dropbox.com/home\"\u003estrony g\u0142\u00f3wnej\u003c/a\u003e.", "sv_SE": "\u003ch1\u003eFel (5xx)\u003c/h1\u003e N\u00e5got gick fel. Oroa dig inte. Dina filer \u00e4r fortfarande s\u00e4kra och Dropboxare har meddelats. Kolla in v\u00e5rt \u003ca href=\"https://www.dropbox.com/help\"\u003eHj\u00e4lpcenter\u003c/a\u003e och \u003ca href=\"https://forums.dropbox.com\"\u003ev\u00e5ra forum\u003c/a\u003e f\u00f6r att f\u00e5 hj\u00e4lp eller g\u00e5 tillbaka till \u003ca href=\"https://www.dropbox.com/home\"\u003estartsidan.\u003c/a\u003e", "en_GB": "\u003ch1\u003eError (5xx)\u003c/h1\u003eSomething went wrong. Don't worry, your files are still safe and the Dropboxers have been notified. Check out our \u003ca href=\"https://www.dropbox.com/help\"\u003eHelp Centre\u003c/a\u003e and \u003ca href=\"https://forums.dropbox.com\"\u003eforums\u003c/a\u003e for help, or head back \u003ca href=\"https://www.dropbox.com/home\"\u003ehome\u003c/a\u003e.", "ms": "\u003ch1\u003eRalat (5xx)\u003c/h1\u003eAda sesuatu yang tidak kena. Jangan risau, fail masih selamat dan Dropboxer telah dimaklumkan. Lihat \u003ca href=\"https://www.dropbox.com/help\"\u003ePusat Bantuan\u003c/a\u003e dan \u003ca href=\"https://forums.dropbox.com\"\u003eforum\u003c/a\u003e kami untuk bantuan, atau kembali ke \u003ca href=\"https://www.dropbox.com/home\"\u003eutama\u003c/a\u003e.", "es_ES": "\u003ch1\u003eError (5xx)\u003c/h1\u003eSe ha producido un error. No te preocupes, los archivos est\u00e1n a salvo y los Dropboxers est\u00e1n al tanto. Visita nuestro \u003ca href=\"https://www.dropbox.com/help\"\u003eCentro de ayuda\u003c/a\u003e y los \u003ca href=\"https://forums.dropbox.com\"\u003eforos\u003c/a\u003e si necesitas que te echen una mano, o bien vuelve a la \u003ca href=\"https://www.dropbox.com/home\"\u003ep\u00e1gina principal\u003c/a\u003e.", "nb_NO": "\u003ch1\u003eFeilmelding (5xx)\u003c/h1\u003eNoe gikk galt. Ikke bekymre deg, filene dine er trygge og Dropbox er varslet. Sjekk ut v\u00e5rt \u003ca href=\"https://www.dropbox.com/help\"\u003ehjelpesenter\u003c/a\u003e og \u003ca href=\"https://forums.dropbox.com\"\u003efora\u003c/a\u003e for hjelp, eller g\u00e5 tilbake til \u003ca href=\"https://www.dropbox.com/home\"\u003ehjemmesiden\u003c/a\u003e.", "id": "\u003ch1\u003eKesalahan(5xx)\u003c/h1\u003eAda yang salah. Jangan cemas, file Anda tetap aman dan Dropboxer sudah diberi tahu. Kunjungi \u003ca href=\"https://www.dropbox.com/help\"\u003ePusat Bantuan\u003c/a\u003e dan \u003ca href=\"https://forums.dropbox.com\"\u003eforum\u003c/a\u003e kami untuk mendapatkan bantuan, atau kembalilah ke \u003ca href=\"https://www.dropbox.com/home\"\u003ehalaman awal\u003c/a\u003e.", "es": "\u003ch1\u003eError (5xx)\u003c/h1\u003eSe ha producido un error. No te preocupes, los archivos est\u00e1n a salvo y los Dropboxers est\u00e1n al tanto. Visita nuestro \u003ca href=\"https://www.dropbox.com/help\"\u003eCentro de ayuda\u003c/a\u003e y los \u003ca href=\"https://forums.dropbox.com\"\u003eforos\u003c/a\u003e si necesitas que te echen una mano, o bien vuelve a la \u003ca href=\"https://www.dropbox.com/home\"\u003ep\u00e1gina principal\u003c/a\u003e."};
function read_cookie (name) {
        var nameEQ = name + "=";
        var ca = document.cookie.split(';');
        for (var i = 0; i < ca.length; i++) {
            var c = ca[i];
            while (c.charAt(0) == ' ') {
                c = c.substring(1, c.length);
            }
            if (c.indexOf(nameEQ) === 0) {
                return c.substring(nameEQ.length, c.length);
            }
        }
        return null;
}
function localize() {
    var locale = read_cookie('locale');
    if (locale) {
       var msg = message[locale];
       if (msg) {
           var elem = document.getElementById('errorbox');
           if (elem) {
               elem.innerHTML = msg;
           }
       }
    }
}
localize();
</script>

</body>
</html>

then there is a gap of 1h in the log with no entries, then a few files checked and skipped. Then the error repeats, but with an "attempt 1/3" line, and the whle procedure starts again.

2021/12/21 02:47:50 DEBUG : Encrypted drive 'qtcrypt:home': Waiting for checks to finish
2021/12/21 02:47:50 DEBUG : Encrypted drive 'qtcrypt:home': Waiting for transfers to finish
2021/12/21 02:47:50 ERROR : Encrypted drive 'qtcrypt:home': not deleting files as there were IO errors
2021/12/21 02:47:50 ERROR : Encrypted drive 'qtcrypt:home': not deleting directories as there were IO errors
2021/12/21 02:47:50 ERROR : Attempt 1/3 failed with 1 errors and: <!DOCTYPE html>

(snip html code, prob. same as above)

</html>
2021/12/21 02:47:50 DEBUG : Creating backend with remote "qtcrypt:MacBook_archive"
2021/12/21 02:47:50 DEBUG : fs cache: switching user supplied name "qtdropbox:/xBackups/macbook/MacBook_archive.bin" for canonical name "qtdropbox{bBI1F}:xBackups/macbook/MacBook_archive.bin"
2021/12/21 02:47:50 DEBUG : Creating backend with remote "qtdropbox:/xBackups/macbook/MacBook_archive.bin"
2021/12/21 02:47:50 DEBUG : qtdropbox: detected overridden config - adding "{bBI1F}" suffix to name
2021/12/21 02:47:51 DEBUG : Dropbox root '': Using root namespace "45943168"
2021/12/21 02:47:51 DEBUG : fs cache: switching user supplied name "qtdropbox:/xBackups/macbook/MacBook_archive" for canonical name "qtdropbox{bBI1F}:xBackups/macbook/MacBook_archive"
2021/12/21 02:47:51 DEBUG : testlog.txt: Excluded
2021/12/21 02:47:51 DEBUG : .config: Excluded
2021/12/21 02:47:51 DEBUG : Temp: Excluded
2021/12/21 02:47:51 DEBUG : .DS_Store: Excluded
2021/12/21 02:47:51 DEBUG : .CFUserTextEncoding: Excluded
2021/12/21 02:47:51 DEBUG : .local: Excluded
2021/12/21 02:47:51 DEBUG : .webex: Excluded
2021/12/21 02:47:51 DEBUG : .zsh_history: Excluded
2021/12/21 02:47:51 DEBUG : Library: Excluded
2021/12/21 02:47:51 DEBUG : .oracle_jre_usage: Excluded
2021/12/21 02:47:51 DEBUG : .cups: Excluded
2021/12/21 02:47:51 DEBUG : .bash_sessions: Excluded
2021/12/21 02:47:51 DEBUG : .dropbox: Excluded
2021/12/21 02:47:51 DEBUG : .Trash: Excluded
2021/12/21 02:47:51 DEBUG : .anydesk: Excluded
2021/12/21 02:47:51 DEBUG : .mono: Excluded
2021/12/21 02:47:51 DEBUG : .imacro: Excluded
2021/12/21 02:47:51 DEBUG : .Xauthority: Excluded
2021/12/21 02:47:51 DEBUG : Downloads: Excluded
2021/12/21 02:47:51 DEBUG : .bash_history: Excluded

The really odd thing here is that the error occurs on a file which is in an excluded folder! I would have expected that file to just not even be part of the check.

So, for now, things are working, though if you have a trick to better exclude folders, I would like that, I prefer to add exclusions than to miss something which should have been included.

VincentP · December 21, 2021, 5:17pm

Hi all,

I made a lengthy post which is now hidden for moderation. I wrote that as new posts appeared. Ole's observation appears to be correct.

there is an error which results in tons to HTML data, and following a long break the whole sync tries again (1 of 3, then later 2 of 3).
I have a huge number of excluded files (meaning, I am excluding a few folder, but somehow that appears to eaxclude every files one by one instead of the folder at once), so things are incredibly slow after a while
the error occurs with a file which is in an excluded folder, so it is completely avoidable.
restructuring the backup to go folder by folder rather than backup all of home (~/) and excluding folders (such as ~/Library/) works, but I would prefer to learn a better way to exclude folders instead, I would not want to miss something important which should have been included (I prefer to get the occasional file included which should have been excluded)

Exclusion file:

.**
Temp/**
Photos Library.photoslibrary/**
temp test/**
*.plist
iTunes/**
Pictures/Photos Library**
Downloads/**
Library/**

asdffdsa · December 21, 2021, 5:19pm

Pictures/Photos Library**

is that correct or did you intend
Pictures/Photos Library/**

not sure, perhaps Pictures/Photos Library** is a file filter, which would be slower than a dir filter

VincentP · December 21, 2021, 5:23pm

It looks like a mistake on my part, a duplicate of that 3rd line, but which does not work.

The file causing the error is deep in the Library folder. So I would like to exclude this folder:
~/Library/
all at once, but it appears like

Library/**

actually excludes all files in Library one by one, so all have to be checked.

asdffdsa · December 21, 2021, 5:27pm

so there is a single file that is the problem?

and not knowing your exact use-case, and if you run the sync on a schedule
might consider --max-age=24h, tho it has some caveats.

VincentP · December 21, 2021, 6:07pm

Actually, looking at the log in more details, there are 4 files which fail one time each throughout the repeats (until I stopped the transfer this morning). They are all variations of the following:

2021/12/21 05:45:48 ERROR : Mail/V8/4F74EEA5-4FD7-43A3-B34C-0F02FD5B0845/Archive 2011.mbox/Sent 2011.mbox/FB501670-1F10-45B8-9DF8-4F4337FB0955/Data/7/8/1/Attachments/187768/2.6: error reading destination directory: <!DOCTYPE html>

(meaning not all from my 2011 archive, not same "Data" subfolder)

The full path starts with ~/Library/Mail/V8/
so it is in my excluded list.

Not on a schedule, more like "whenever I think of it, it has been a while". But since yesterday I finally solved the "Too many requests or write" problem (by creating my own client id), and now I have a solution which will complete in a matter of minutes instead of hours, I may start to do it more frequently, so noted.

Unless there is something I can do to have better exclusion filtering, that will be it, thanks all!

VincentP · December 22, 2021, 4:11pm

One last update, as I finally figured out all my problems. This could be helpful for someone stumbling on this topic later on. Also, everything seems fine with rclone, no worries.

The reason why file checks was taking so long was my own fault. Somehow I thought it would be a good idea to sync all of my home folder (with exceptions) into a folder on my Dropbox, then do a second sync of another folder into that same destination. Works well the first time, but the second time around, the sync command sees all these extra files which need to be removed, and there are 300k+ of them (that second folder is littered with tiny files).

So I moved my second folder elsewhere in my Dropbox, and all of a sudden my sync takes 8 minutes instead of 8 hours.

Thanks again to all who replied, I could not have done it without bouncing ideas and being forced to clarify the situation in my mind to explain it. I don't suppose I can close my topic? Anyways, I won't touch it again, it will close on its own.

Ole · December 22, 2021, 8:58pm

Thanks of behalf of us all!

Glad you found a solution and very good explanation.

I can add a little (and somewhat dangerous) trick to use in situation like yours, that is the --delete-excluded flag – it will automatically delete all the destination files excluded from your sync. This will help if you later change your exclusion/filter but be extremely careful – always use --dry-run after adding the flag or changing the filters.

system · January 21, 2022, 8:58pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.