Problems copying files with rclone

talesam · October 13, 2022, 5:05pm

First I will explain my problem. I have 30T to copy from one Google Drive to another Google Drive, there are 15G, 10G, 20G and 50k files.
The problem is that Google limits to 750G of daily transfer, when you reach the limit it's like the file doesn't exist anymore.
I tried to use pure rsync at first, it was very slow, reaching about 11 hours to copy a 4G file from one drive to another. I found a project called msrsync, with it I managed to copy about 600G in 10 minutes, and here the problem started, because when the limit runs out, it gets stuck in the program and doesn't come out. Already the rclone he keeps trying to copy even giving error, and I want that, if he tried to copy and gave an error, he doesn't try anymore and exit the command.
I made a script to leave it running to copy, but I still haven't had success, because in both that I tested, they are stuck in the system trying to copy something that is stuck.

In short:

I need to copy 30T from one Drive to another Drive
When it reaches 750 transfer, the Drive will block and it will be released in 24h, so I need that when I can't copy, it gives an error, exit the rclone copy and continue executing the rest of my script.
You need to resume, return the copy where you left off, not copying files that have already been copied and are intact.

What is the problem you are having with rclone?

It keeps trying to copy files that are not accessible, forever.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-1017-oracle (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy --update --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s $DIR1/ $DIR #>/dev/ null 2>&1

The rclone config contents with secrets removed.

[jelly]
type = drive
client_id = xxxxxx
client_secret = xxxxxx
scope = drive
token = xxxxxx
team_drive = xxxxxx
root_folder_id = 

[jelly-crypt]
type = crypt
remote = jelly:/
password = xxxxxx
[mirabel]
type = drive
client_id = xxxxxx
client_secret = xxxxxx
scope = drive
token = xxxxxx
team_drive = xxxxxx
root_folder_id = 

[mirabel-crypt]
type = crypt
remote = mirabel:/
password = xxxxxx 

[mirabel-bkp]
type = drive
client_id = xxxxxx
client_secret = xxxxxx
scope = drive
token = xxxxxx
root_folder_id =

A log from the command with the `-vv` flag

10/13/2022 17:03:30 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv"]
Usage:
  rclone [flags]
  rclone [command]

My Script:

#!/bin/bash

# Script fazer bkp do drive
# v0.0.3

# Tabela de cores
# Cores degrade
RED001='\e[38;5;1m'             # Vermelho 1
RED009='\e[38;5;9m'             # Vermelho 9
CYA122='\e[38;5;122m'           # Ciano 122
CYA044='\e[38;5;44m'            # Ciano 44
ROX063='\e[38;5;63m'            # Roxo 63
ROX027='\e[38;5;27m'            # Roxo 27
GRE046='\e[38;5;46m'            # Verde 46
GRY247='\e[38;5;247m'           # Cinza 247
LAR208='\e[38;5;208m'           # Laranja 208
LAR214='\e[38;5;214m'           # Laranja 214
AMA226='\e[38;5;226m'           # Amarelo 226
BLU039='\e[38;5;44m'            # Azul 39
MAR094='\e[38;5;94m'            # Marrom 94
MAR136='\e[38;5;136m'           # Marrom 136

# Cores chapadas
CIN='\e[30;1m'                  # Cinza
RED='\e[31;1m'                  # Vermelho
GRE='\e[32;1m'                  # Verde
YEL='\e[33;1m'                  # Amarelo
BLU='\e[34;1m'                  # Azul
ROS='\e[35;1m'                  # Rosa
CYA='\e[36;1m'                  # Ciano
NEG='\e[37;1m'                  # Negrito
CUI='\e[40;31;5m'               # Vermelho pisacando, aviso!
STD='\e[m'                      # Fechamento de cor


# Variáveis

DATA=$(date '+%d-%m-%Y %H:%M')
DIR="/home/jelly/Jellyfin/BKP"
DIR1="/home/jelly/Mirabel"
TESTE='/home/jelly/Mirabel/Filmes/The Matrix (1999)/clearlogo.png'
#DIR2="/home/jelly/Mirabel-bkp"

# ---
while : ; do
    clear
    # Testa se o servidor está acessível para copiar arquivos
    cp -f "$TESTE" /tmp >/dev/null 2>&1

    # Checa se a cópia foi feita
    if [ "$?" -eq "0" ]; then
        echo -e "${GRE}*${STD} ${NEG}Iniciando cópia dos arquivos...${STD}" && sleep 1
        echo ""
        #msrsync -P -p 8 --rsync "-ravzP" $DIR1/ $DIR #>/dev/null 2>&1
        rclone copy --update --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s $DIR1/ $DIR #>/dev/null 2>&1
        if [ "$?" -eq "0" ]; then
            echo -e "${CYA}*${STD} ${NEG}Cópia gerada com sucesso em $DATA${STD}" && sleep 1
            echo  "-> Cópia gerada com sucesso em $DATA (UTC)" >> $DIR/log-rsync.txt
        else
            echo -e "${RED}*${STD} ${NEG}ERRO ao gerar cópia em $DATA${STD}" && sleep 1
            echo "*ERRO em ao gerar cópia em $DATA" >> $DIR/log-rsync-erro.txt
        fi
    else
        # Tenta copiar um arquivo, se não conseguir aguarda até conseguir
        echo ""
        echo -e " ${NEG}Aguardando a boa vontade do${STD} ${BLU}G${STD}${RED}o${STD}${YEL}o${STD}${BLU}g${STD}${GRE}l${STD}${RED}e${STD} ${NEG}liberar o limite do Drive...${STD}${ROS}😴${STD}"
        until $(cp -f "$TESTE" /tmp >/dev/null 2>&1); do
            # Aguardando liberar o limite do Drive
            sleep 60
            echo -ne "."
        done
    fi
done

Animosity022 · October 13, 2022, 5:09pm

It is though so we can identify if we are testing, we have the same setup.

You'd have to share a log file with the actual error to troubleshoot so we have something to look at.

talesam · October 13, 2022, 5:14pm

Where is the rclone error log generated?

Animosity022 · October 13, 2022, 5:16pm

You'd have to add -vv --log-file /some/location/rclone.log to your rclone command and share the full file.

talesam · October 13, 2022, 6:14pm

rclone copy -vv --log-file $DIR/rclone.log --update --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s $DIR1/ $DIR

I'm running like this, but it's not showing the output of the command, I have to wait for the 750G limit to end, then it will start to give an error, it keeps trying to copy, I want it to have an error and not be able to copy, give a break

Animosity022 · October 13, 2022, 7:05pm

What happens with that command is that if you hit your quota, it'll keep trying 3 times for every failing file.

If the goal is stop the copy once you hit your quota, you can add:

https://rclone.org/drive/#drive-stop-on-upload-limit

and I'd probably just remove this:

and just run at the defaults and validate things are working as you'd expect. Generally, though that first flag would be helpful and stop once you hit an error.

talesam · October 13, 2022, 8:34pm

I switched to this..

rclone sync --drive-stop-on-upload-limit --update --verbose --transfers 10 --stats 1s $DIR1/ $DIR -vv --log-file $DIR/rclone.log

I saw that the sync, the goal is to keep the destination identical to the source, that's what I need. and the --transfers 10 is for me to get 10 simultaneous connections to make the copy faster.

talesam · October 13, 2022, 9:45pm

LOG: Cloud T4l35

It didn't work, it gets stuck in this command and doesn't come out when the error.

rclone sync --drive-stop-on-upload-limit --update --verbose --transfers 10 --stats 1s $DIR1/ $DIR -vv --log-file $DIRL/rclone.log

talesam · October 13, 2022, 9:54pm

I saw that rclone has a rc flag that something can be executed during the execution of rclone, if you check the log, the error it gives is "input/output error", I can put something for when rclone gives this error stop running?

talesam · October 14, 2022, 3:43am

It's really, the parameter "--drive-stop-on-upload-limit" doesn't work, when the limit ends, it keeps giving error: input/output error and it doesn't stop copying, it should stop using this parameter, someone can help me?

I tested with sync and copy, same problem.

and if I use the rc command and pass something like this..

rc[ "$?" -eq "1" ]; break

but I don't know how to use this rc inside rclone.

Animosity022 · October 14, 2022, 3:50am

Are you copying on a mount as well not directly a remote?

If you have mounts, you'd have to share your mount commands.

Best to test without any script and just run a command.

The flag I shared works with a google drive remote to copy to, not a local file system or a mount.

talesam · October 14, 2022, 3:54am

Yes, the 2 drives are mounted.
I just need a break for when the error appears :-/

[Install]
WantedBy=multi-user.target

[Unit]
Description=rClone montar drive criptografado
AssertPathIsDirectory=/home/jelly/Mirabel
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
  --config=/home/jelly/.config/rclone/rclone.conf \
  --fast-list \
        --drive-skip-gdocs \
        --vfs-read-chunk-size=128M \
        --vfs-read-chunk-size-limit=2048M \
        --buffer-size=128M \
        --max-read-ahead=256M \
        --poll-interval=1m \
        --dir-cache-time=12h \
        --timeout=10m \
        --transfers=16 \
        --checkers=8 \
        --drive-chunk-size=128M \
        --allow-non-empty \
        --allow-other \
        --umask=027 \
        -vP \
        mirabel-crypt:/ /home/jelly/Mirabel
ExecStop=/usr/bin/fusermount -u /home/jelly/Mirabel
Restart=on-failure
RestartSec=10
User=jelly

[Install]
WantedBy=multi-user.target

Animosity022 · October 14, 2022, 4:08am

Looks to be the source from the log:

/home/jelly/Mirabel/

and what is

/home/jelly/Jellyfin/BKP - local disk? mount? what?

ncw · October 14, 2022, 10:40am

Are you syncing to or from an rclone mount?

You will get much better performance if you sync the underlying remotes directly

talesam · October 14, 2022, 2:35pm

Looks to be the source from the log:

/home/jelly/Mirabel/
and what is

A Mounted Google Drive

/home/jelly/Jellyfin/BKP - local disk? mount? what?

Another Google Drive Mounted

talesam · October 14, 2022, 2:42pm

Are you syncing to or from an rclone mount?

Yes, Both 2 drives are mounted

You will get much better performance if you sync the underlying remotes directly

I have no idea how to do this and the performance with this command is not bad, the average transfer rate is 100mbits/s which ends the 750G limit very quickly.

I have 2 Google Drives mounted with TeamDrive and crypt on top, copying the content from one to the other, my only problem, as I said at the beginning of the topic, is it stops trying to copy when it gives an error, but it doesn't happen, it keeps trying to copy forever, and is stuck.

Pure rsyn is unfeasible, 11 hours to copy 4G, msrsync managed to do 600G in 10 minutes, but it crashes just like rclone and doesn't come out, but it seems that rclone has more resource and msrsync has been abandoned for years, that's why I'm trying with rclone now. Solving this problem of giving I/O error it stops trying to copy and exiting the command is enough.

ncw · October 14, 2022, 3:29pm

You would do

rclone copy --dry-run source: dest: -v --drive-stop-on-upload-limit

where source: and dest: are jelly-crypt:/mirabel-crypt:/mirabel-bkp: - I'm not sure exact which is the source and the dest from the above.

This will be a lot more efficient and reliable than syncing two mounts and the --drive-stop-on-upload-limit will work (it won't work between two mounts).

Try it first with --dry-run then remove when happy

talesam · October 14, 2022, 3:52pm

jelly@jellyfin:~$ rclone config
Current remotes:

Name                 Type
====                 ====
jelly                drive
jelly-crypt          crypt
mirabel              drive
mirabel-bkp          drive
mirabel-crypt        crypt

mirabel-crypt = origin

jelly@jellyfin:~$ rclone lsd mirabel-crypt:
          -1 2021-10-15 20:20:03        -1 Filmes
          -1 2021-10-15 20:20:05        -1 Livros
          -1 2021-10-15 20:20:05        -1 Musicas
          -1 2021-10-15 20:20:06        -1 Series
          -1 2021-10-15 20:20:08        -1 Temp

jelly-crypt:/BKP = destiny

jelly@jellyfin:~$ rclone lsd jelly-crypt:/BKP
          -1 2022-10-13 21:41:55        -1 Filmes
          -1 2022-10-13 18:10:25        -1 Musicas
          -1 2022-10-13 18:19:33        -1 Series

I want to speed up the copy so I put 10 simultaneous transfers (--transfers 10) but I can lower it to 8 or 5.

Another detail, what's the difference between using sync and copy? the source drive is read-only, and I want whatever is on the source drive to be identical to the destination drive.

I think that would be it, correct?
rclone copy mirabel-crypt: jelly-crypt:/BKP -v --drive-stop-on-upload-limit

Animosity022 · October 14, 2022, 3:57pm

Sync makes the destination identical to the source and deletes things not found in the source so it's a 'mirror'.

Copy makes destination look like the source, but does not delete anything so if you have an extra file there, it will remain.

talesam · October 15, 2022, 1:36am

This is the log being generated, is that right?

2022/10/15 01:32:45 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/ujkjpmmgkpgru2dpec3t0teqs2ru6rrupp332b1sfith66p5p69ru8j0oulqecgj3k35b1qsq1s4u4slurtru32g0nddroot8b5n3034eflc318sicfp0uh7rj40urtfs1la185r8k9vnk65ov8pqcf376iiib8va5d3qr6c5kvb1nq8gkm0: Sending chunk 394264576 length 8388608
2022/10/15 01:32:45 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/nrcqhesqvcabk62nnllfdnttalqhehm49naco75r2sdljcvpmut95pimaoh2pucqqulh6losrs2gerb3g7n2d9mp6ot8mr6f210g5h32ab3fadjtpgduhob0avmt8vd2vpqoc3e5j9ueddig4l4r6iljem1traj0vfodb5lbf5frhbr6iqag: Sending chunk 536870912 length 8388608
2022/10/15 01:32:45 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/9dei8sms6mbfeem2ran07djlrpikt9hdfudaovj1209vilrl2rmp4vuavbte6j7pupctb1ngfigeveskmu9qmmnohmp0phkhhdb1bp8m4atf792vrccvhd25r31n52rmqk5ahrc5aub9vkijsqvo7uih6l3rtku5sio6s9hhpjsogkej8tpg: Sending chunk 469762048 length 8388608
2022/10/15 01:32:45 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/63c9rtu6jlfsqp1cr4is8ot4gushj4amc56a8quorkm8hug9rdvvocnn78ujg7ed6211g68qh25t4s2v0jsqpraoql0sk0garogdu1d9p7fuf5i7f780osn0bb64nco19c9v2d70o23kof49f5p93nqippcdt7jc6jvo7qogsdjl5ngorr00: Sending chunk 595591168 length 8388608
2022/10/15 01:32:46 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/e0jrevlap7a3dir776klqrdqcig34kan9p5d209n4jcolo7h1s356f4i2cvi2g81pgg1bcul1n3o8o0klf9usctimj2gaf4sn4p0ot4qlv2203l93egej2h0eh2v1pqov3o7frj38h3io2c4l56one8cdfjbq6lpb9tihhp06ag0fla6bmp0: Sending chunk 603979776 length 8388608
2022/10/15 01:32:46 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/9quka8ntcncbijpt6npej22qqfgp7ag8otra0d8spsbrdiuue51c3m8gq7e7dekm6fljqic7rg33r0moneqr4qge2inub516876thbliifr6stmmif0olpvobt640i23erafvp70eiso23n4u69m6dfq5gqndig7tqcdrd5qovo6f5mjfvj0: Sending chunk 562036736 length 8388608
2022/10/15 01:32:46 DEBUG : se3u18sr0u92jm6vif23d21jpc/mj1oajpak0hkq1vsfgc8djv4daai01g8v316k8crqn4lborbgbng/i7klr9gpoh2mf9thg48ngsitv0/6autjo7r80s297fsvfbjl9qp7saae4tlmt9tgbm8j61nbha7aon32k9hjcajr7cs9kr62flhepg275umk0oafqqgf049i7p5dogdprh5377bd5qn4ho6jscijnpmdv1hu1toshl8ojg0jvqfiqah6cfcbo: Sending chunk 2785017856 length 8388608
2022/10/15 01:32:46 DEBUG : se3u18sr0u92jm6vif23d21jpc/n3tmcanejqlat18qjemmh2ddbq2g4det4rqt9tejblsflg9f9e10/i7klr9gpoh2mf9thg48ngsitv0/a8qgj0mg1spsme0l7gihocadvo286bov9a67kop4kios2dmhn8j36ic47f10vhsfttj65k3rnsv5f07d1e3bauo0i80e34hcsnioukq308pf1tajeihhcpmq23bjua2sjnplv96dcq7121ecn4r31cgfe8om3nlsae57np488n1shrc7uso0: Sending chunk 268435456 length 8388608