What is the problem you are having with rclone?
I am using rclone copy in scrip using loop "for i " method.
copying multiple pattern of files multiple path from Linux remote server using ssh.
now issue is while I am copying files and I am given pattern via script for loop .
suppose I am copying A file from pending Directory and B file from incoming directory
for loop looking for A file and pending and incoming both path and copied
same for B file it is looking B incoming and Pending both directory and copying same file again .
What is your rclone version (output from rclone version)
Which cloud storage system are you using? (eg Google Drive)
I am copying data to Minio object storage from Linux machine using rclone copy
The command you were trying to run (eg rclone copy /tmp remote:tmp)
have multiple patterns define in below file
/tmp/file {A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}
B.2MMO.CHARGE.????????.A901*
and then we have multiple directory information file
/tmp/direname
The rclone config contents with secrets removed.
this is the script for rclone using loop
FILE=`cat /tmp/file`
DIRNAME=`cat /tmp/direname`
for i in $DIRNAME
do
for j in $FILE
do
rclone copy --max-depth=1 --include=$j $i /home/miniousr/rclone_script/dynamo/test2/$i -vvv -P
done
done
A log from the command with the -vv flag
logs from rclone script
sh -x test.sh
++ cat /tmp/file
+ FILE='{A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}
B.2MMO.CHARGE.????????.A901*'
++ cat /tmp/direname
+ DIRNAME='/home/miniousr/rclone_script/dynamo/incoming
/home/miniousr/rclone_script/dynamo/pending'
+ for i in '$DIRNAME'
+ for j in '$FILE'
+ rclone copy --max-depth=1 '--include={A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}' /home/miniousr/rclone_script/dynamo/incoming /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming -vvv -P
2021/11/29 11:48:36 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "--max-depth=1" "--include={A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}" "/home/miniousr/rclone_script/dynamo/incoming" "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming" "-vvv" "-P"]
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/incoming"
2021/11/29 11:48:36 DEBUG : Using config file from "/home/miniousr/.config/rclone/rclone.conf"
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming"
2021-11-29 11:48:36 DEBUG : B.2MMO.CHARGE.123456789.A901: Excluded
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming: Waiting for checks to finish
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming: Waiting for transfers to finish
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2021-11-29 11:48:36 INFO : A.SERVICE_AGREEMENT.12345678.A901: Copied (new)
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.0s
2021/11/29 11:48:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.0s
2021/11/29 11:48:36 DEBUG : 3 go routines active
+ for j in '$FILE'
+ rclone copy --max-depth=1 '--include=B.2MMO.CHARGE.????????.A901*' /home/miniousr/rclone_script/dynamo/incoming /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming -vvv -P
2021/11/29 11:48:36 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "--max-depth=1" "--include=B.2MMO.CHARGE.????????.A901*" "/home/miniousr/rclone_script/dynamo/incoming" "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming" "-vvv" "-P"]
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/incoming"
2021/11/29 11:48:36 DEBUG : Using config file from "/home/miniousr/.config/rclone/rclone.conf"
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming"
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : B.2MMO.CHARGE.123456789.A901: Excluded
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming: Waiting for checks to finish
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/incoming: Waiting for transfers to finish
2021-11-29 11:48:36 INFO : There was nothing to transfer
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 0.0s
2021/11/29 11:48:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 0.0s
2021/11/29 11:48:36 DEBUG : 3 go routines active
+ for i in '$DIRNAME'
+ for j in '$FILE'
+ rclone copy --max-depth=1 '--include={A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}' /home/miniousr/rclone_script/dynamo/pending /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending -vvv -P
2021/11/29 11:48:36 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "--max-depth=1" "--include={A.SERVICE_AGREEMENT.????????.A901*,C.CONTRACT_TYPE.????????.A901*}" "/home/miniousr/rclone_script/dynamo/pending" "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending" "-vvv" "-P"]
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/pending"
2021/11/29 11:48:36 DEBUG : Using config file from "/home/miniousr/.config/rclone/rclone.conf"
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending"
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending: Waiting for checks to finish
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending: Waiting for transfers to finish
2021-11-29 11:48:36 DEBUG : C.CONTRACT_TYPE.12345678.A901: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2021-11-29 11:48:36 INFO : C.CONTRACT_TYPE.12345678.A901: Copied (new)
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2021-11-29 11:48:36 INFO : A.SERVICE_AGREEMENT.12345678.A901: Copied (new)
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Transferred: 2 / 2, 100%
Elapsed time: 0.0s
2021/11/29 11:48:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Transferred: 2 / 2, 100%
Elapsed time: 0.0s
2021/11/29 11:48:36 DEBUG : 3 go routines active
+ for j in '$FILE'
+ rclone copy --max-depth=1 '--include=B.2MMO.CHARGE.????????.A901*' /home/miniousr/rclone_script/dynamo/pending /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending -vvv -P
2021/11/29 11:48:36 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "--max-depth=1" "--include=B.2MMO.CHARGE.????????.A901*" "/home/miniousr/rclone_script/dynamo/pending" "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending" "-vvv" "-P"]
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/pending"
2021/11/29 11:48:36 DEBUG : Using config file from "/home/miniousr/.config/rclone/rclone.conf"
2021/11/29 11:48:36 DEBUG : Creating backend with remote "/home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending"
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : C.CONTRACT_TYPE.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : A.SERVICE_AGREEMENT.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : C.CONTRACT_TYPE.12345678.A901: Excluded
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending: Waiting for checks to finish
2021-11-29 11:48:36 DEBUG : Local file system at /home/miniousr/rclone_script/dynamo/test2//home/miniousr/rclone_script/dynamo/pending: Waiting for transfers to finish
2021-11-29 11:48:36 INFO : There was nothing to transfer
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 0.0s
2021/11/29 11:48:36 INFO :
Transferred: 0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time: 0.0s
2021/11/29 11:48:36 DEBUG : 2 go routines active
he is the copy file result . we can see "A.SERVICE_AGREEMENT.12345678.A901" this file available in both "incoming and pending" and we can see the destination path where the files copied , from this name "A.SERVICE_AGREEMENT.12345678.A901" 2 files available .
miniousr:/home/miniousr/rclone_script/dynamo $ ll test2/home/miniousr/rclone_script/dynamo/incoming
total 0
-rw-------. 1 miniousr miniousr 0 Nov 29 11:44 A.SERVICE_AGREEMENT.12345678.A901
/home/miniousr/rclone_script/dynamo $ ll test2/home/miniousr/rclone_script/dynamo/pending
total 0
-rw-------. 1 miniousr miniousr 0 Nov 29 11:42 A.SERVICE_AGREEMENT.12345678.A901
-rw-------. 1 miniousr miniousr 0 Nov 29 11:43 C.CONTRACT_TYPE.12345678.A901
/home/miniousr/rclone_script/dynamo $ ll pending/
total 0
-rw-------. 1 miniousr miniousr 0 Nov 29 11:42 A.SERVICE_AGREEMENT.12345678.A901
-rw-------. 1 miniousr miniousr 0 Nov 29 11:43 C.CONTRACT_TYPE.12345678.A901
/home/miniousr/rclone_script/dynamo $ ll incoming/
total 0
-rw-------. 1 miniousr miniousr 0 Nov 29 11:44 A.SERVICE_AGREEMENT.12345678.A901
-rw-------. 1 miniousr miniousr 0 Nov 29 11:43 B.2MMO.CHARGE.123456789.A901
Pls let me know if you understand what I am trying to explain in this post.