1Fichier - Rate limit (Flood detected: IP Locked)

Fluffy · December 9, 2021, 8:22am

Hi!

I'm having issues with syncing files from Google Drive to 1Fichier. Especially 1Fichier is giving me rate limit issues:

2021/12/09 09:18:51 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")

The above error I get a lot during my sync.

The Rclone command I use to do the sync is:

rclone sync -vv --transfers=1 sa: 1f-crypt:

I already set the amount of transfers to 1, but this doesn't solve the problem. Maybe the problem lies in the amount of small files <10MB I need to have synced.

My Rclone version is:

rclone v1.57.0
- os/version: debian 10.11 (64 bit)
- os/kernel: 4.19.0-13-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

My Rclone.conf looks like:

[1f]
type = fichier
api_key = 6T5xxxxxxx

[1f-crypt]
type = crypt
remote = 1f:
password = FOExxxxxxx
password2 = ZAHxxxxxxx

[sa]
type = drive
scope = drive
service_account_file = /root/.config/rclone/sa/cxxxxxxxxxxx.json
impersonate = jxxxxxxx@xxxxx.xx
client_id = 203xxxxxxx
client_secret = GOCxxxxxx

Log, when having the --transfers=4 set:

2021/12/09 13:57:25 DEBUG : pacer: Reducing sleep to 500.564574ms
2021/12/09 13:57:25 DEBUG : pacer: Reducing sleep to 400ms
2021/12/09 13:57:58 DEBUG : backups/xxxxxxxxxx/xxxxxx-nl-10-05-2021.tar: whirlpool = 03f1210c122b1a8740e0f9xxxxxxxxxxxxxxxxxxxd0bf73ea1eaf9d92116b02741df45fcdec0f18e563d94d OK
2021/12/09 13:57:58 INFO  : backups/xxxxxxxxxx/xxxxxx-nl10-05-2021.tar: Copied (new)
2021/12/09 13:58:08 DEBUG : backups/xxxxxxxxxx/xxxxxx-nl-10-06-2021.tar: whirlpool = 29ad209c675c0b8c8e3xxxxxxx9501729874ba94894b57347ede5abf9c134016d811a4fdf48 OK
2021/12/09 13:58:08 INFO  : backups/xxxxxxxxxx/xxxxxx-nl-10-06-2021.tar: Copied (new)
2021/12/09 13:58:09 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 13:58:10 DEBUG : backups/xxxxxxxxxx/xxxxxx-nl-10-07-2021.tar: whirlpool = f17400e259edb5adxxxxxxx165d7595f2aa571d68f2be02b2d30c5b82aaf9859ce50be9e81652c9a619ec3e97db2b9d OK
2021/12/09 13:58:10 INFO  : backups/xxxxxxxxxx/xxxxxx-nl-10-07-2021.tar: Copied (new)
2021/12/09 13:58:10 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 13:58:11 DEBUG : backups/xxxxxxxxxx/xxxxxx-nl-10-08-2021.tar: whirlpool = e829080fa604270aee846daexxxxxxx0772bbddef22bd4b2641d84af360236f7c411d9d1c076368b0298be9998aee7dc2eebf28d2 OK
2021/12/09 13:58:11 INFO  : backups/xxxxxxxxxx/xxxxxx-nl-10-08-2021.tar: Copied (new)
2021/12/09 13:58:11 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/09 13:58:20 DEBUG : backups/xxxxxxxxxx/xxxxxx-nl-10-09-2021.tar: whirlpool = 863428e3f910e262b302262079cbf385c9xxxxxxxd56de31274f839683e367d3df0ac9e17c383ddc9f9df31ecb5cb3d9178e60f6 OK
2021/12/09 13:58:20 INFO  : backups/xxxxxxxxxx/xxxxxx-nl-10-09-2021.tar: Copied (new)
2021/12/09 13:58:20 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/09 13:58:22 INFO  :
Transferred:        7.253 GiB / 195.021 GiB, 4%, 14.519 MiB/s, ETA 3h40m43s
Checks:              1740 / 1740, 100%
Transferred:          125 / 10137, 1%
Elapsed time:      37m0.3s
Transferring:
 * backups/xxxxxxxxxx/xxxxxx1-nl-10-10-2021.tar:  0% /218.047Mi, 0/s, -
 * backups/xxxxxxxxxx/xxxxxx2-nl-10-11-2021.tar:  0% /218.018Mi, 0/s, -
 * backups//xxxxxxxxxx/xxxxxx3-nl-10-12-2021.tar:  0% /224.502Mi, 0/s, -
 * backups/xxxxxxxxxx/xxxxxx4-nl-10-13-2021.tar:  0% /239.248Mi, 0/s, -

2021/12/09 13:58:39 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
2021/12/09 13:58:39 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/09 13:58:39 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/09 13:58:40 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
2021/12/09 13:58:41 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/09 13:58:44 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/09 13:58:49 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 13:58:50 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/09 13:58:54 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 13:59:09 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/09 13:59:09 DEBUG : pacer: Reducing sleep to 3.75s
2021/12/09 13:59:14 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/09 13:59:14 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/09 13:59:14 DEBUG : pacer: Reducing sleep to 3.75s
2021/12/09 13:59:19 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
2021/12/09 13:59:19 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/09 13:59:19 DEBUG : pacer: Reducing sleep to 3.75s
2021/12/09 13:59:22 INFO  :
Transferred:        7.253 GiB / 195.021 GiB, 4%, 309.393 KiB/s, ETA 1w8h46m11s
Checks:              1740 / 1740, 100%
Transferred:          125 / 10137, 1%
Elapsed time:      38m0.3s
Transferring:
 * backups/xxxxxxxxxx/xxxxxx1-nl-10-10-2021.tar:  0% /218.047Mi, 0/s, -
 * backups/xxxxxxxxxx/xxxxxx2-nl-10-11-2021.tar:  0% /218.018Mi, 0/s, -
 * backups/xxxxxxxxxx/xxxxxx3-nl10-12-2021.tar:  0% /224.502Mi, 0/s, -
 * backups/xxxxxxxxxx/xxxxxx4-nl-10-13-2021.tar:  0% /239.248Mi, 0/s, -

Log, when having the --transfers=1 set:

2021/12/09 14:51:09 DEBUG : pacer: Reducing sleep to 2.109375s
2021/12/09 14:51:11 DEBUG : pacer: Reducing sleep to 1.58203125s
2021/12/09 14:51:11 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-12-2021.sql.tar: whirlpool = 96cxxxxxxxxxxxxac0526fa85c8f1665dab5b1cd53569fbf6c705bfca47 OK
2021/12/09 14:51:11 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-12-2021.sql.tar: Copied (new)
2021/12/09 14:51:14 DEBUG : pacer: Reducing sleep to 1.186523437s
2021/12/09 14:51:16 DEBUG : pacer: Reducing sleep to 889.892577ms
2021/12/09 14:51:18 DEBUG : pacer: Reducing sleep to 667.419432ms
2021/12/09 14:51:19 DEBUG : pacer: Reducing sleep to 500.564574ms
2021/12/09 14:51:19 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-13-2021.sql.tar: whirlpool = 0f2b19a0daf320d4caxxxxxxxxx4183bc184f7b8bd9exxxxxxxxxx87bd5 OK
2021/12/09 14:51:19 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-13-2021.sql.tar: Copied (new)
2021/12/09 14:51:23 DEBUG : pacer: Reducing sleep to 400ms
2021/12/09 14:51:25 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-14-2021.sql.tar: xxxad6c3dd047bf9a449fa7dca527a4c423bac OK
2021/12/09 14:51:25 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx4611-14-2021.sql.tar: Copied (new)
2021/12/09 14:51:32 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-15-2021.sql.tar: whirlpool = 1250c97778ae2a020291xxxxxxxxxxxd208d84ec0xxxx9185afb09d579e3430eff3d00 OK
2021/12/09 14:51:32 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx4611-15-2021.sql.tar: Copied (new)
2021/12/09 14:51:38 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx4611-16-2021.sql.tar: whirlpool = 2926fecead88xxxxxxxxxxxxff82c6xxxx4919b688 OK
2021/12/09 14:51:38 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46=11-16-2021.sql.tar: Copied (new)
2021/12/09 14:51:45 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-17-2021.sql.tar: whirlpool = 211b769exxxxxxxxx774b2b1dd417e4bd70a2xxxxxxxcbd4 OK
2021/12/09 14:51:45 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-17-2021.sql.tar: Copied (new)
2021/12/09 14:51:48 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/09 14:52:09 INFO  :
Transferred:        9.186 GiB / 182.597 GiB, 5%, 15.686 MiB/s, ETA 3h8m40s
Checks:              1891 / 1891, 100%
Transferred:          119 / 10128, 1%
Elapsed time:      42m0.4s
Transferring:
 * backups/xxxxxx/xxxxxxx_16xxxxx46-11-18-2021.sql.tar:  0% /6.396Mi, 0/s, -

2021/12/09 14:52:18 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/09 14:52:18 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/09 14:52:18 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 14:52:48 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
2021/12/09 14:52:48 DEBUG : pacer: Reducing sleep to 3.75s
2021/12/09 14:52:53 DEBUG : pacer: Reducing sleep to 2.8125s
2021/12/09 14:52:58 DEBUG : pacer: Reducing sleep to 2.109375s
2021/12/09 14:53:01 DEBUG : pacer: Reducing sleep to 1.58203125s
2021/12/09 14:53:01 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-18-2021.sql.tar: whirlpool = 5e672208d2fc825dxxxxxxxx48bfd44d1907bcff0xxxxxxxxc5ce0c OK
2021/12/09 14:53:01 INFO  :backups/xxxxxx/xxxxxxx_16xxxxx46-11-18-2021.sql.tar: Copied (new)
2021/12/09 14:53:04 DEBUG : pacer: Reducing sleep to 1.186523437s
2021/12/09 14:53:05 DEBUG : pacer: Reducing sleep to 889.892577ms
2021/12/09 14:53:08 DEBUG : pacer: Reducing sleep to 667.419432ms
2021/12/09 14:53:08 DEBUG : pacer: Reducing sleep to 500.564574ms
2021/12/09 14:53:08 DEBUG :backups/xxxxxx/xxxxxxx_16xxxxx46-11-19-2021.sql.tar: whirlpool = 2737218cxxxxxxxxx4a451464e5cb33d28c90bxxxxxxxxxe6b95 OK
2021/12/09 14:53:08 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-19-2021.sql.tar: Copied (new)
2021/12/09 14:53:09 INFO  :
Transferred:        9.199 GiB / 182.591 GiB, 5%, 15.686 MiB/s, ETA 3h8m39s
Checks:              1891 / 1891, 100%
Transferred:          121 / 10130, 1%
Elapsed time:      43m0.4s
Transferring:
 * backups/xxxxxx/xxxxxxx_16xxxxx46-11-20-2021.sql.tar: transferring

2021/12/09 14:53:12 DEBUG : pacer: Reducing sleep to 400ms
2021/12/09 14:53:15 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-20-2021.sql.tar: whirlpool = 70exxxxxxxxxd4bfc58382d4f5b5f6dc78daaed5e43213a876da1720d1c2cbb6bf7be851 OK
2021/12/09 14:53:15 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-20-2021.sql.tar: Copied (new)
2021/12/09 14:53:21 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-21-2021.sql.tar: whirlpool = d9bxxxxxxxxxx597c41dbe91cc15dcd3fa32xxxxxx OK
2021/12/09 14:53:21 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-21-2021.sql.tar: Copied (new)
2021/12/09 14:53:27 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-22-2021.sql.tar: whirlpool = 10276e3a63xxxxxxxxxxfabe356476c66d1745eb639b937dd4f45xxxxxxxx OK
2021/12/09 14:53:27 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-22-2021.sql.tar: Copied (new)
2021/12/09 14:53:33 DEBUG : backups/xxxxxx/xxxxxxx_16xxxxx46-11-23-2021.sql.tar: whirlpool xxxxxxxxxxx5c8603fa9b80fed6bacb74253e812e0ad467b66233527ea33fd3e3edd2b189bcc669c07ea1b12cxxxxxxxxxx OK
2021/12/09 14:53:33 INFO  : backups/xxxxxx/xxxxxxx_16xxxxx46-11-23-2021.sql.tar: Copied (new)
2021/12/09 14:53:36 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
^[[B^[[B^[[B^[[B2021/12/09 14:54:06 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
2021/12/09 14:54:06 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/09 14:54:06 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}"
2021/12/09 14:54:09 INFO  : 
Transferred:        9.224 GiB / 182.608 GiB, 5%, 15.686 MiB/s, ETA 3h8m39s
Checks:              1891 / 1891, 100%
Transferred:          125 / 10134, 1%
Elapsed time:      44m0.4s
Transferring:
 * backups/xxxxxx/xxxxxxx_16xxxxx46-11-24-2021.sql.tar:  0% /6.387Mi, 0/s, -

Thanks for helping!

Ole · December 9, 2021, 10:59am

Hi Fluffy,

I would first try reducing --checkers to 4, 2 or 1 while having --transfers at the half, that is 2, 1 or 1.

If this doesn’t help then I would try reducing --tps-limit to 4, 2 or 1.

Fluffy · December 9, 2021, 11:16am

I'm trying this at the moment. Still with a --tpslimit of 0.5 I still get these errors. Somehow the API of 1Fichier is really limit in the amount of calls you can do in a short period, probably to prevent abuse.

Ole · December 9, 2021, 12:12pm

Did you see this:

github.com/rclone/rclone

An error occurred while using 1fichier, Flood detected: IP Locked

opened 09:51AM - 24 Apr 20 UTC

closed 08:12PM - 26 Apr 20 UTC

Hubray

bug Remote: Fichier

#### What is the problem you are having with rclone? An error occurred during c…opy and sync `Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" ` #### What is your rclone version (output from `rclone version`) rclone v1.51.0 - os/arch: linux/amd64 - go version: go1.13.7 #### Which OS you are using and how many bits (eg Windows 7, 64 bit) CentOS 7.7.1908 (64 Bit) KVM #### Which cloud storage system are you using? (eg Google Drive) 1fichier #### The command you were trying to run (eg `rclone copy /tmp remote:tmp`) rclone sync /www/backup test1:test003 -P rclone sync /www/backup test1:test003 -P --bwlimit 10M rclone copy /www/backup test1:test005 -P rclone copy /www/backup test1:test005 -P --bwlimit 10M #### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`) Sorry, I do n’t know which part of the log to present I copied some information about running errors `2020-04-24 17:14:31 ERROR : panel/2020-04-11/data/***.json: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" 2020-04-24 17:14:32 ERROR : panel/2020-04-11/data/***.conf: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" 2020-04-24 17:14:32 ERROR : panel/2020-04-11/data/***.pl: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" 2020-04-24 17:14:33 ERROR : panel/2020-04-11/data/***.json: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" 2020-04-24 17:14:33 ERROR : panel/2020-04-11/data/***.txt: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" 2020-04-24 17:14:33 ERROR : panel/2020-04-11/data/***.conf: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" 2020-04-24 17:14:34 ERROR : panel/2020-04-11/data/***.pl: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" 2020-04-24 17:14:34 ERROR : panel/2020-04-11/data/***.conf: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" 2020-04-24 17:14:35 ERROR : panel/2020-04-11/data/***.json: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" 2020-04-24 17:14:35 ERROR : panel/2020-04-11/data/******.pl: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}" 2020-04-24 17:14:35 ERROR : panel/2020-04-11/data/******.json: Failed to copy: couldn't list files: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}" `

Are you using an rclone version where this has been fixed?
(What is your output from rclone version?)

If so, then I suggest you wait 24 hours for 1Fichier to lift the block while waiting for somebody with experience in 1Fichier to give you some more specific advice - I don't know 1Fichier.

Fluffy · December 9, 2021, 12:15pm

I'm using the latest stable version 1.57.0 of Rclone

Animosity022 · December 9, 2021, 12:55pm

We don't know what your exact setup is because you deleted the template so key pieces are missing

So time is wasted going back and forth asking the same questions.

Can you share your rclone version?
rclone.conf?
A full log with debug.

Fluffy · December 9, 2021, 1:04pm

I updated my original post

Hope this clears thing up. Thank you

Animosity022 · December 9, 2021, 1:07pm

Closer.

You have:

But the log is from 4 transfers?

Can you share the full log with 1 transfer?

Fluffy · December 9, 2021, 1:59pm

Sure, added it to my original post!

Animosity022 · December 9, 2021, 2:04pm

That looks like a snippet again and not a full log.

Can you just reply rather than repeat editing as it's tough to keep track of what changed.

I think you need to wait a bit if you are locked as they throttle hard. Not sure what the time is.

@buengese might be able to add some details

ncw · December 9, 2021, 5:26pm

The API for 1fichier is really stingy on transactions per second.

The backend defaults to 1 transaction every 400mS so --tpslimit 2.5.

--tpslimit should help once the 24hrs has expired.

Fluffy · December 9, 2021, 7:56pm

Thanks! In 24 hours I will try the --tpslimit 2.5.

Should I leave the --transfers on 1 or can I increase this to for example the default 4?

Ole · December 10, 2021, 8:37am

I would start a bit to the safe side with a balanced command like this:

rclone sync ... --checkers 2 --transfers 1 --tps-limit 2

based on my experience with throttling/rate limiting on OneDrive and Google Drive and the fact that your previous attempt using default settings (with 1 transfer) caused "Flood detected: IP locked".

Fluffy · December 10, 2021, 10:41am

Thank you Ole! I will try your settings and let you all know what the results are

Fluffy · December 11, 2021, 8:18pm

It seems like it's working now using

rclone sync ... --checkers 2 --transfers 1 --tps-limit 2

But sadly I still get this pattern when uploading:

2021/12/11 21:11:15 INFO  : Pictures/2020/03/02/IMG_7114.JPG: Copied (new)
2021/12/11 21:11:16 DEBUG : pacer: Reducing sleep to 400ms
2021/12/11 21:11:19 DEBUG : Pictures/2020/03/02/IMG_7115.JPG: whirlpool = 4e1682cc8206xxxxx3c786f37520f67832228607d14d694207607975c2e142a5e0a01b67 OK
2021/12/11 21:11:19 INFO  :  Pictures/2020/03/02/IMG_7115.JPG: Copied (new)
2021/12/11 21:11:23 DEBUG : Pictures/2020/03/02/IMG_7116.JPG: whirlpool = e0a50xxxxxxx77ebab5df8dea9c119f69b7ce3bcfb4fd90b994588a3a92b0890a27e3559514224fdb2 OK
2021/12/11 21:11:23 INFO  : Pictures/2020/03/02/IMG_7116.JPG: Copied (new)
2021/12/11 21:11:26 DEBUG : Pictures/2020/03/02/IMG_7117.JPG: whirlpool = 97efc9cc44xxxxxx367ae0f7b7a2f790f0b50277363ab0110f37a51f1402f8d8c01e8076a471a259793 OK
2021/12/11 21:11:26 INFO  : Pictures/2020/03/02/IMG_7117.JPG: Copied (new)
2021/12/11 21:11:30 DEBUG : Pictures/2020/03/02/IMG_7118.JPG: whirlpool = 70f55bb7cf4b831d97e0xxxxxee8d97005b7f85f69894cdfb2c9f1f16322ca9db8890a092f5eef1f37ceb98 OK
2021/12/11 21:11:30 INFO  :  Pictures/2020/03/02/IMG_7118.JPG: Copied (new)
2021/12/11 21:11:31 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/11 21:11:34 INFO  :
Transferred:       34.634 GiB / 119.077 GiB, 29%, 4.019 KiB/s, ETA 36w3d3m53s
Checks:                52 / 52, 100%
Transferred:         5305 / 15308, 35%
Elapsed time:   23h20m0.7s
Transferring:
 *      Pictures/2020/03/02/IMG_7119.JPG:  0% /1.782Mi, 0/s, -

2021/12/11 21:12:01 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
2021/12/11 21:12:01 DEBUG : pacer: Rate limited, increasing sleep to 5s
2021/12/11 21:12:01 DEBUG : Sleeping for 30 seconds due to: HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
2021/12/11 21:12:31 DEBUG : pacer: low level retry 2/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}")
^[[B^[[B2021/12/11 21:12:31 DEBUG : pacer: Reducing sleep to 3.75s
^[[B2021/12/11 21:12:34 INFO  : 
Transferred:       34.634 GiB / 119.077 GiB, 29%, 4.019 KiB/s, ETA 36w3d3m53s
Checks:                52 / 52, 100%
Transferred:         5305 / 15308, 35%
Elapsed time:   23h21m0.7s
Transferring:
 *      Pictures/2020/03/02/IMG_7119.JPG:  0% /1.782Mi, 0/s, -

2021/12/11 21:12:36 DEBUG : pacer: Reducing sleep to 2.8125s
2021/12/11 21:12:41 DEBUG : pacer: Reducing sleep to 2.109375s
2021/12/11 21:12:43 DEBUG : pacer: Reducing sleep to 1.58203125s
2021/12/11 21:12:43 DEBUG :  Pictures/2020/03/02/IMG_7119.JPG: whirlpool = 5b2fb0df6b535cxxxxx23f0e5a56440609984e1239bef789b5670f635bdf0074498c44a5bdee96e93 OK
2021/12/11 21:12:43 INFO  : Pictures/2020/03/02/IMG_7119.JPG: Copied (new)
2021/12/11 21:12:45 DEBUG : pacer: Reducing sleep to 1.186523437s
2021/12/11 21:12:46 DEBUG : pacer: Reducing sleep to 889.892577ms

It manages to upload around 5 files and then hits the IP Locked error. It will take a minute and then begins uploading again. I think the --tpslimit has to be set even lower to not hit this error.

I leave the sync for now because it works (besides hitting the limit over and over), but it seems like their API is very strict and more suited for bigger files that take a few minutes to upload instead of pictures that take 2 seconds or some.

Ole · December 11, 2021, 9:32pm

I am happy to hear that

I fully agree, probably 0.25 or lower.

This incredible slow compared to other alternatives such as OneDrive and Google Drive where I see TPS around 5-15 and photos syncing around 1-5 per second (on my paid accounts) – just to give you a rough benchmark (this is not a guarantee).

You may want to check the market for a better balance between price and performance.

Fluffy · December 11, 2021, 9:42pm

At this moment I'm using Google Workspace and never have any of these issues. 1Fichier is an encrypted back-up for me at this moment, so if it's very slow it's not that big of a deal.

I have to see what my exact usage will be the coming month / year and if I will go beyond the 2 TB I might need to look for other solution that don't break the bank. First because of the cold storage limit of 1Fichier and second because of the restore that I might have to do in the future.

I don't want to solely rely on Google Workspace for my important personal files (I also have a lot of files that don't have much personal value to me).

Thanks for looking into this with me/us!

So for people that might consider 1Fichier as a back-up or primary (encrypted) cloud storage solution using Rclone. The API is very strict and limited, so uploading and downloading a lot of files in a short amount of time will be limited. If you have not much but very large files, it might be a fine option.

system · January 10, 2022, 9:43pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.