What is the problem you are having with rclone?
rclone is re-downloading files that already exist locally when mirroring the Fedora EPEL archive via the http backend.
Even when:
- The file name is identical
- The file exists in the destination
- The file size is the same
- The timestamp appears unchanged
rclone logs still show:
"Need to transfer - File not found at Destination"
and the same file gets downloaded again as "Copied (new)".
Run the command 'rclone version' and share the full output of the command.
rclone v1.71.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-86-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Not using cloud storage. I am using the HTTP backend
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy :http: "${LOCAL_MIRROR_BASE}/" \
--http-url=https://archives.fedoraproject.org/pub/archive/epel \
--fast-list \
--log-file="${LOG_DIR}/${LOG_FILE}" \
--ignore-existing \
--stats=99999m \
--stats-log-level=NOTICE \
--bwlimit=0 \
--transfers=8 \
--checkers=16 \
I also tried the following options (separately and together), but the same behavior occurs:
rclone copy :http: "${LOCAL_MIRROR_BASE}/" \
--http-url=https://archives.fedoraproject.org/pub/archive/epel \
-vv \
--log-file="${LOG_DIR}/${LOG_FILE}" \
--stats=99999m \
--stats-log-level=NOTICE \
--bwlimit=0 \
--transfers=4 \
--checkers=8 \
--no-gzip-encoding \
--header="Accept-Encoding:identity" \
--timeout=300s \
--contimeout=60s \
--low-level-retries=10 \
--retries=3 \
--retries-sleep=10s \
--update \
--use-server-modtime
I have already tried:
-
--ignore-existing
-
--update
-
--use-server-modtime
However, rclone still reports Copied (new) and re-downloads files that already exist.
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
rclone config redacted
2025/11/24 21:44:56 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
; empty config
### Double check the config for sensitive info before posting publicly
A log from the command that you were trying to run with the -vv flag
2025/11/24 22:02:30 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" ":http:" "/mirror/repo/epel/" "--http-url=https://archives.fedoraproject.org/pub/archive/epel" "-vv" "--log-file=/var/log/epel/epel.log" "--stats-log-level=NOTICE" "--stats=99999m" "--bwlimit=0" "--transfers=4" "--checkers=8" "--no-gzip-encoding" "--header=Accept-Encoding:identity" "--timeout=300s" "--contimeout=60s" "--low-level-retries=10" "--retries=3" "--retries-sleep=10s" "--update" "--use-server-modtime" "--exclude=4*/**" "--exclude=5*/**" "--exclude=playground/**" "--exclude=testing/**" "--exclude=**/aarch64/**" "--exclude=**/armhfp/**" "--exclude=**/ppc64/**" "--exclude=**/ppc64le/**" "--exclude=**/s390x/**" "--exclude=**/i386/**" "--exclude=**/SRPMS/**" "--exclude=**/source/**" "--exclude=**/debug/**"]
2025/11/24 22:02:30 DEBUG : Creating backend with remote ":http:"
2025/11/24 22:02:30 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2025/11/24 22:02:30 DEBUG : :http: detected overridden config - adding "{tgQ79}" suffix to name
2025/11/24 22:02:30 DEBUG : Root: https://archives.fedoraproject.org/pub/archive/epel/
2025/11/24 22:02:30 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{tgQ79}:"
2025/11/24 22:02:30 DEBUG : Creating backend with remote "/mirror/repo/epel/"
2025/11/24 22:02:30 DEBUG : fs cache: renaming cache item "/mirror/repo/epel/" to be canonical "/mirror/repo/epel"
---
2025/11/24 22:07:41 DEBUG : 8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm: Need to transfer - File not found at Destination
2025/11/24 22:12:25 DEBUG : 8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm.7edc9a03.partial: renamed to: 8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm
2025/11/24 22:12:25 INFO : 8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm: Copied (new)
---
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/Packages/y: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/Packages/x: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/Packages/z: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/drpms: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/Packages: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64/repodata: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything/x86_64: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8/Everything: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 INFO : 8.8: Set directory modification time (using DirSetModTime)
2025/11/24 22:34:03 NOTICE:
Transferred: 17.117 GiB / 17.117 GiB, 100%, 7.701 MiB/s, ETA 0s
Checks: 496399 / 496399, 100%, Listed 1006038
Transferred: 10017 / 10017, 100%
Elapsed time: 31m33.0s
Run summary (previous run, same content):
2025/11/24 18:10:55 INFO : 8.8/Everything/x86_64/Packages: Set directory modification time (using DirSetModTime)
2025/11/24 18:10:55 INFO : 8.8/Everything: Set directory modification time
2025/11/24 18:10:55 NOTICE:
Transferred: 17.117 GiB / 17.117 GiB, 100%, 17.8 MiB/s, ETA 0s
Checks: 496399 / 496399, 100%, Listed 1006038
Transferred: 10017 / 10017, 100%
Elapsed time: 15m26.1s
Additional Information / Verification
To confirm that the file already exists on the destination, I checked it directly on the local filesystem:
ls -lah /mirror/repo/epel/8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm
Output:
-rw-r--r-- 1 root root 114K Aug 23 2023 /mirror/repo/epel/8.8/Everything/x86_64/Packages/i/ImageMagick-6.9.12.93-1.el8.x86_64.rpm
Even though this file clearly exists on the destination, rclone still reports:
Need to transfer - File not found at Destination
and then downloads the same file again as:
Copied (new)