HTTP sync errors?

Hello,

I'm trying to download a archive.org repository, but since its slow and I don't want to overload its servers, I'm trying to transfer one file at a time.

I believe rclone is not correctly parsing the arguments for the following reasons:
1 - I have transfers=1 and multi-thread-streams=0, but it will still start downloading more than 1 file
2 - I have local-no-sparse but it still uses sparse files
3 - I have multi-thread-streams disabled but it still uses multi-threads
4 - Also I'm using max-depth=1 but it still tries to download one additional level (archive.org has a link for each file which lists its contents, but it effectively is one additional level - see error "error reading source directory: error listing "Gradius II - X68k FM+MT-32 OST (Hack) v1 protogem.7z/": directory not found" below)

Is it because of these combinations of arguments? Or is it because of the :http: argument?

Thanks,

Mike

Operating system: Windows 10
rclone version v1.52.0-032-g7458d37d-beta (but this also happens with the non-beta)

C:\rclone>rclone -vvv sync --http-url "https://archive.org/download/everdrivepack/Project%20Peacock/" :http: "X:\mirrors\Everdrive\Project Peacock\" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10

2020/06/09 09:45:55 DEBUG : rclone: Version "v1.52.0-032-g7458d37d-beta" starting with parameters ["rclone" "-vvv" "sync" "--http-url" "https://archive.org/download/everdrivepack/Project%20Peacock/" ":http:" "X:\\mirrors\\Everdrive\\Project Peacock\" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10 "]
2020/06/09 09:45:55 DEBUG : Using config file from "C:\\Users\\Mike\\.config\\rclone\\rclone.conf"
2020/06/09 09:45:55 DEBUG : fs cache: renaming cache item "X:\\mirrors\\Everdrive\\Project Peacock\" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10 " to be canonical "//?/X:/mirrors/Everdrive/Project Peacock\" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10 "
2020/06/09 09:46:55 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2020/06/09 09:47:17 DEBUG : Ai Chou Aniki (Japan) (SADS).7z: Sizes differ (src 485161861 vs dst 243580494)
2020/06/09 09:47:17 DEBUG : Avenger (Japan).7z: Sizes differ (src 265694666 vs dst 133774927)
2020/06/09 09:47:17 DEBUG : CD Denjin - Rockabilly Tengoku (Japan).7z: Sizes differ (src 448394290 vs dst 224771662)
2020/06/09 09:47:17 DEBUG : Chou Aniki (Japan) (FAAT, FACT).7z: Sizes differ (src 228641267 vs dst 126976)
2020/06/09 09:47:17 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2020/06/09 09:47:17 DEBUG : Daisenpuu Custom (Japan).7z: Starting multi-thread copy with 2 parts of size 169.500M
2020/06/09 09:47:17 DEBUG : Daisenpuu Custom (Japan).7z: multi-thread copy: stream 2/2 (177733632-355349463) size 169.388M starting
2020/06/09 09:47:17 DEBUG : Chou Jikuu Yousai Macross 2036 (Japan).7z: Starting multi-thread copy with 2 parts of size 224.750M
2020/06/09 09:47:17 DEBUG : Chou Jikuu Yousai Macross 2036 (Japan).7z: multi-thread copy: stream 2/2 (235667456-471322533) size 224.738M starting
2020/06/09 09:47:17 DEBUG : Chou Jikuu Yousai Macross 2036 (Japan).7z: multi-thread copy: stream 1/2 (0-235667456) size 224.750M starting
2020/06/09 09:47:17 DEBUG : Daisenpuu Custom (Japan).7z: multi-thread copy: stream 1/2 (0-177733632) size 169.500M starting
2020/06/09 09:47:33 ERROR : Gradius II - X68k FM+MT-32 OST (Hack) v1 protogem.7z: error reading source directory: error listing "Gradius II - X68k FM+MT-32 OST (Hack) v1 protogem.7z/": directory not found
2020/06/09 09:47:55 INFO  :
Transferred:       13.116M / 28.062 GBytes, 0%, 351.872 kBytes/s, ETA 23h13m7s
Errors:                 1 (retrying may help)
Checks:                 4 / 4, 100%
Transferred:            0 / 85, 0%
Elapsed time:        38.1s
Transferring:
 *                        Down Load 2 (Japan).7z:  1% /225.871M, 110.465k/s, 34m26s
 *                   Daisenpuu Custom (Japan).7z:  1% /338.888M, 173.437k/s, 32m51s
 *     Chou Jikuu Yousai Macross 2036 (Japan).7z:  0% /449.488M, 135.523k/s, 56m4s
 * Cycho Rider (from Seiy…th Fantasy Stories).7z:  2% /39.806M, 32.343k/s, 20m28s

2020/06/09 09:47:57 ERROR : @Shooter Elements (CD)/Xak III - The Eternal Recurrence - English (Japan) T+Eng v1 X3 De-Japanization 2004.7z: error reading source directory: error listing "@Shooter Elements (CD)/Xak III - The Eternal Recurrence - English (Japan) T+Eng v1 X3 De-Japanization 2004.7z/": directory not found
2020/06/09 09:48:08 DEBUG : Local file system at //?/X:/mirrors/Everdrive/Project Peacock" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10 : Waiting for checks to finish
2020/06/09 09:48:08 DEBUG : Local file system at //?/X:/mirrors/Everdrive/Project Peacock" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1 --local-no-sparse --multi-thread-streams 0 --low-level-retries 32 --retries 10 : Waiting for transfers to finish
2020/06/09 09:48:55 INFO  :
Transferred:       34.178M / 28.062 GBytes, 0%, 356.510 kBytes/s, ETA 22h53m59s
Errors:                 2 (retrying may help)
Checks:                 4 / 4, 100%
Transferred:            0 / 85, 0%
Elapsed time:      1m38.1s
Transferring:
 *                        Down Load 2 (Japan).7z:  3% /225.871M, 76.293k/s, 48m44s
 *                   Daisenpuu Custom (Japan).7z:  4% /338.888M, 148.474k/s, 37m16s
 *     Chou Jikuu Yousai Macross 2036 (Japan).7z:  1% /449.488M, 93.808k/s, 1h20m12s
 * Cycho Rider (from Seiy…th Fantasy Stories).7z:  7% /39.806M, 26.579k/s, 23m38s

2020/06/09 09:49:55 INFO  :
Transferred:       62.647M / 28.062 GBytes, 0%, 405.569 kBytes/s, ETA 20h6m35s
Errors:                 2 (retrying may help)
Checks:                 4 / 4, 100%
Transferred:            0 / 85, 0%
Elapsed time:      2m38.1s
Transferring:
 *                        Down Load 2 (Japan).7z:  6% /225.871M, 121.659k/s, 29m43s
 *                   Daisenpuu Custom (Japan).7z:  7% /338.888M, 128.984k/s, 41m29s
 *     Chou Jikuu Yousai Macross 2036 (Japan).7z:  3% /449.488M, 91.364k/s, 1h21m16s
 * Cycho Rider (from Seiy…th Fantasy Stories).7z: 22% /39.806M, 85.730k/s, 6m8s

2020/06/09 09:50:55 INFO  :
Transferred:      101.069M / 28.062 GBytes, 0%, 474.378 kBytes/s, ETA 17h10m11s
Errors:                 2 (retrying may help)
Checks:                 4 / 4, 100%
Transferred:            0 / 85, 0%
Elapsed time:      3m38.1s
Transferring:
 *                        Down Load 2 (Japan).7z:  8% /225.871M, 119.659k/s, 29m21s
 *                   Daisenpuu Custom (Japan).7z: 11% /338.888M, 207.949k/s, 24m35s
 *     Chou Jikuu Yousai Macross 2036 (Japan).7z:  6% /449.488M, 214.881k/s, 33m29s
 * Cycho Rider (from Seiy…th Fantasy Stories).7z: 35% /39.806M, 128.802k/s, 3m25s

2020/06/09 09:51:55 INFO  :
Transferred:      122.928M / 28.062 GBytes, 0%, 452.525 kBytes/s, ETA 17h59m6s
Errors:                 2 (retrying may help)
Checks:                 4 / 4, 100%
Transferred:            0 / 85, 0%
Elapsed time:      4m38.1s
Transferring:
 *                        Down Load 2 (Japan).7z:  9% /225.871M, 14.431k/s, 4h1m6s
 *                   Daisenpuu Custom (Japan).7z: 14% /338.888M, 94.190k/s, 52m26s
 *     Chou Jikuu Yousai Macross 2036 (Japan).7z:  7% /449.488M, 125.852k/s, 56m16s
 * Cycho Rider (from Seiy…th Fantasy Stories).7z: 42% /39.806M, 44.656k/s, 8m43s

Just found out that rclone was creating a directory called "Project Peacock" --checkers 1 --transfers=1 --progress --delete-during --max-depth 1"...

The error is that rclone cannot have the last slash before the double quote in the local path. So the local path must be:

"X:\mirrors\Everdrive\Project Peacock"

instead of:

"X:\mirrors\Everdrive\Project Peacock\"

Thanks!

Mike

have you seen this flag?
https://rclone.org/docs/#bwlimit-bandwidth-spec

Yes, thank you. The archive.org servers are so slow I've never had to use it, so I use only one connection so as not to overload the server connections.

By the way, is there a Patreon created for the rclone creators? I googled it but didn't find anything...

yes, i have donated a few times.
the main author is trying to focus on open-source rclone full time.
https://rclone.org/donate/

Oh, that is a Windows cmd gotcha.

The backslash "escapes" the quote sign. I make that mistake too fairly often!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.