Mirror Ubuntu Repo

Help and Support
1

What is the problem you are having with rclone?

I'm just wondering, is it possible to use rclone to mirror Ubuntu from here? https://launchpad.net/ubuntu/+archivemirrors. If so does anyone have any working examples? In an ideal situation what I would like to do is just mirror the 20.04 and 22.04 versions.

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.60.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-53-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: none
2

here is a command that you can tweak

+ rclone sync --http-url http://mirror.enzu.com/ubuntu/ :http: /home/user01/dest -vv --dry-run
2022/11/20 14:13:09 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "sync" "--http-url" "http://mirror.enzu.com/ubuntu/" ":http:" "/home/user01/dest" "-vv" "--dry-run"]
2022/11/20 14:13:09 DEBUG : Creating backend with remote ":http:"
2022/11/20 14:13:09 DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
2022/11/20 14:13:09 DEBUG : :http: detected overridden config - adding "{DtBEM}" suffix to name
2022/11/20 14:13:09 DEBUG : Root: http://mirror.enzu.com/ubuntu/
2022/11/20 14:13:09 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{DtBEM}:"
2022/11/20 14:13:09 DEBUG : Creating backend with remote "/home/user01/dest"
2022/11/20 14:13:09 NOTICE: ls-lR.gz: Skipped copy as --dry-run is set (size 21.469Mi)
2022/11/20 14:13:09 NOTICE: ubuntu/ls-lR.gz: Skipped copy as --dry-run is set (size 21.469Mi)
2022/11/20 14:13:09 NOTICE: project/ubuntu-archive-keyring.gpg: Skipped copy as --dry-run is set (size 6.556Ki)
3

Thanks @asdffdsa . I think what I'm looking for is more the part on how to filter based on certain Ubuntu versions.

4

then find the dir you want and use that as the source
rclone lsd --http-url http://mirror.enzu.com/ubuntu/ :http: -vv -R
or use --include

or provide more details.

5

For the most part the sync seems to be working of the ubuntu mirrors, but I'm getting these errors, even though the user has full access on the destination. Any ideas?

INFO  : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/pool/universe/e/exfatprogs/exfatprogs_1.2.0.orig.tar.xz.asc: Copied (new)
ERROR : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/indices/override.kinetic-proposed.universe.debian-installer: Failed to copy: Update Create failed: permission denied
INFO  : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/pool/universe/i/indiv-screenlets/screenlets-pack-all_0.1.6-0ubuntu1_all.deb: Copied (new)
INFO  : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/pool/main/l/language-pack-gnome-wa/language-pack-gnome-wa_14.04+20140410_all.deb: Copied (new)
INFO  : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/pool/universe/h/haskell-haddock/libghc-haddock-dev_2.13.1-3build1_i386.deb: Copied (new)
INFO  : ubuntu/ubuntu/ubuntu/pool/universe/l/linux-euclid/linux-headers-4.4.0-9026-euclid_4.4.0-9026.28_amd64.deb: Copied (new)
ERROR : ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/indices/override.kinetic-proposed.universe.src: Failed to copy: Update Create failed: permission denied
6

as a test, try to manually download the file using the website.

post the top 20 lines of the debug log.

7

I tend to try to use the easiest tool to get the job done.

Why not use apt-mirror ?

8

@asdffdsa Yes I can get those files from the website manually. Seems to work fine. I can try and get debug logs for you thanks. @Animosity022 . That is a good point. I was wondering if it may be better/more reliable to use apt-mirror. Has this worked well for you in the past? The only downside of this for me, is I would have to apt-mirror to local storage in a DMZ, and then rclone from there to the network. I could potentially use a rclone mount, or sshfs here. I though it might be easier to just use a remote source, and remote destination using rclone and get the full mirror so that is where I'm at now.

9

Here are some more logs in debug:

rclone sync -vv --log-file rclone-ubuntu-archive.log --bwlimit 0 --http-url http://mirror.it.ubc.ca :http:/ubuntu/ remote:/nfs/repos/ubuntu/
 
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1.dsc: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1.dsc: Unchanged skipping
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1.tar.gz: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1.tar.gz: Unchanged skipping
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1_all.deb: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.54ubuntu1_all.deb: Unchanged skipping
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1.dsc: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1.dsc: Unchanged skipping
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1.tar.xz: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1.tar.xz: Unchanged skipping
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1_all.deb: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/pool/main/d/d-shlibs/d-shlibs_0.66ubuntu1_all.deb: Unchanged skipping
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 ERROR : indices/override.focal-proposed.universe.debian-installer: Failed to copy: Update Create failed: permission denied
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 ERROR : indices/override.focal-proposed.universe.src: Failed to copy: Update Create failed: permission denied
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 ERROR : indices/override.focal-security.extra.main: Failed to copy: Update Create failed: permission denied
2022/11/22 18:44:30 DEBUG : ubuntu/ubuntu/ubuntu/ubuntu/ls-lR.gz: Sizes differ (src 23509728 vs dst 17000493)
2022/11/22 18:44:30 DEBUG : sftp://sshuser@192.168.1.23:22//nfs/repos/ubuntu/: Connection OK after error: permission denied
2022/11/22 18:44:30 ERROR : indices/override.focal-proposed.main: Failed to copy: Update Create failed: permission denied
2022/11/22 18:44:30 ERROR : indices/override.focal-security.extra.multiverse: Failed to copy: Update Create failed: permission denied
10

are using http or sftp?

when you post a debug log, need to include the top 20+ lines

11

The source is http, the destination remote is sftp://

12

I never had the need/urge to replicate a mirrored repo as there are tons of them already.

13

No. I'm new to rclone, so haven't had it working yet.

14

in this case, i agree with @Animosity022

if you want to make rclone work, i suggest that you focus on copying a single file that is causing trouble.
and might try --dump=headers

15

Ok. Thanks for the advice. I have apt-mirror working well, so will just stick with that for now. Use rclone for other things.

1 Like
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.