Copy: Checking files is taking forever

Help and Support
1

Hi there,

I just set up rclone and wanted to move my files to my new encrypted mount. Used rsync first but got some errors so I decided to use rclone copy:

# rclone copy -P /home/plx/movies /mnt/gdrive/movies

While copying itself doesn’t take long, checking seems to run forever. Copying files took about 1,5h, checking files about 12h. I canceled the command. I tried the beta with same result. I even got files that weren’t copied completely. The actual size should be above 10G but it was only ~300M.
My guess is that I configured something wrong. Any hints?

# rclone version
rclone v1.46.0-027-g8f4ea77c-beta
- os/arch: linux/amd64
- go version: go1.12rc1

# cat /root/.config/rclone/rclone.conf
[gdrive]
type = drive
scope = drive
token = {..}

[cache]
type = cache
remote = gdrive:/gdrive
plex_url = http://..:32400
plex_username = ..
plex_password = ..
chunk_size = 5M
info_age = 1d
chunk_total_size = 10G

[gcrypt]
type = crypt
remote = cache:/crypt
filename_encryption = standard
directory_name_encryption = true
password = ..
password2 = ..

# cat /etc/systemd/system/rclone-systemd.service 
[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/root/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount gcrypt: /mnt/gdrive \
   --allow-other \
   --dir-cache-time 72h \
   --cache-chunk-path=/root/.cache/rclone \
   --fast-list \
   --cache-db-path=/root/.cache/rclone \
   --log-level INFO \
   --log-file /root/logs/rclone.log
ExecStop=/bin/fusermount -uz /mnt/gdrive
Restart=on-failure
User=root
Group=root

[Install]
WantedBy=multi-user.target
2

Did you make your own client ID?

https://rclone.org/drive/#making-your-own-client-id

The copy defaults are a bit high for GD so I’m guessing your log would show rate limits. You may want to turn down to 4 transfers and maybe 4 checkers and see if that helps.

I wouldn’t use the beta version unless you are testing a particular feature.

Using the Cache backend with the plex integration also severely slows anything going directly to the mount as it only uses 1 worker.

I’d copy using an encrypted GD config from one to the other. Also, you can only copy 750GB at most per day.

1 Like
3

I didn’t but will do.

I followed this topic to set up rclone.


What would you recommend instead?

4

Is copying from one drive to another plex? :slight_smile: That’s a good config for Plex but very poor if you want to copy from a mount.

I’d setup an encrypted remote and copy directly from remote to remote as that will be faster.

5

Moving files will only happens one time. I want to move the files from the local hdd to gdrive. Plex then is using gdrive as a library.

6

Copying files to the cache backend with Plex integration is slow as it only uses 1 worker since it isn’t plex traffic.

Replace remote with local drive and it’s the same concept.

Crete an encrypted remote and just copy directly to that and don’t use the cache.

1 Like
7

Okay, I create another encrypted remote that doesn’t use cache in between, correct?
It copies files to encrypted remote -> gdrive. So extend rclone config with something like this:

[gcrypt2]
type = crypt
remote = gdrive:/gdrive/crypt
filename_encryption = standard
directory_name_encryption = true
password = ..
password2 = ..

?

Thanks for your time. Rclone is a bit confusing to me

8

Yes, that looks good!

You can copy from your local HDD to gcrypt2:

Anything you copy would also show up on the cache mount in ~1 minute as that’s normal GD polling time.

1 Like
9

Your suggestion is working wonderful! I have another question though: would it be useful to mount the 2nd encrypted remote as well for copying new files and have the encrypted cache mounted for plex only? If yes what mount options would you recommend? Thanks!

10

There is no need to mount as that would only add overhead.

11

So it’s fine for radarr/sonarr to use the encrypted cache mount? Won’t it face the same problems I experienced with copying manually?

12

I don’t think it would be a problem other than your chunk size is super tiny.

I’d make that 32M or 64M.

I find that the plex integration makes thing much slower imo as it takes some time to notice it’s being played and than speed up so slower start times.

1 Like