Copy from-file list with UNC paths to a UNC path

Help and Support
1

What is the problem you are having with rclone?

unable to copy files from a list to a UNC path

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

not using cloud storage - copying from a UNC path to a new UNC path from a file-list

The command you were trying to run (eg rclone copy /tmp remote:tmp)

C:\rclone\rclone.exe -Pv copy --files-from C:\rclone\prd3-rubys.txt / '\\corp\apps$\ECM\LiveLink\dev3'

The rclone config contents with secrets removed.

[default]

type = local

nounc = false

A log from the command with the -vv flag

2022-05-04 16:31:55 INFO  : There was nothing to transfer
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.5s
C:\rclone\rclone.exe : 2022/05/04 16:31:55 INFO  : 
At line:1 char:1
+ C:\rclone\rclone.exe -Pv copy --files-from C:\rclone\prd3-rubys.txt / ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022/05/04 16:31:55 INFO  : :String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.5s
2

hello and welcome to the forum,

please run the command with -vv` for debug output and then post log

3 

C:\rclone\rclone.exe : 2022/05/04 17:02:36 DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\rclone\\rclone.exe" "-Pvvv" "copy" "--files-from" 
"C:\\rclone\\prd3-rubys.txt" "\\\\corp\\apps$\\ECM\\LiveLink\\prd3\\" "\\\\corp\\apps$\\ECM\\LiveLink\\dev3" "--multi-thread-streams" "32" "--fast-list"]
At line:1 char:1
+ C:\rclone\rclone.exe -Pvvv copy --files-from C:\rclone\prd3-rubys.txt ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022/05/04 17:0... "--fast-list"]:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2022/05/04 17:02:36 DEBUG : Creating backend with remote "\\\\corp\\apps$\\ECM\\LiveLink\\prd3\\"
2022/05/04 17:02:36 DEBUG : Using config file from "C:\\Users\\xadmrubys\\AppData\\Roaming\\rclone\\rclone.conf"
2022/05/04 17:02:36 DEBUG : fs cache: renaming cache item "\\\\corp\\apps$\\ECM\\LiveLink\\prd3\\" to be canonical "//?/UNC/corp/apps$/ECM/LiveLink/prd3/"
2022/05/04 17:02:36 DEBUG : Creating backend with remote "\\\\corp\\apps$\\ECM\\LiveLink\\dev3"
2022/05/04 17:02:36 DEBUG : fs cache: renaming cache item "\\\\corp\\apps$\\ECM\\LiveLink\\dev3" to be canonical "//?/UNC/corp/apps$/ECM/LiveLink/dev3"
2022/05/04 17:02:36 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022/05/04 17:02:36 DEBUG : 2 go routines active

2022-05-04 17:02:36 DEBUG : 0091: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022-05-04 17:02:36 DEBUG : 0092: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022-05-04 17:02:36 DEBUG : 0093: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022-05-04 17:02:36 DEBUG : Local file system at //?/UNC/corp/apps$/ECM/LiveLink/dev3: Waiting for checks to finish
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022-05-04 17:02:36 DEBUG : Local file system at //?/UNC/corp/apps$/ECM/LiveLink/dev3: Waiting for transfers to finish
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
2022-05-04 17:02:36 INFO  : There was nothing to transfer
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s
4

what does the contents C:\rclone\prd3-rubys.txt look like

is it possible to run the rclone commands direct on the command line?
to post the exact command, not just its debug output.

5 
"0091/866/91866781.dat"
"0091/879/91879577.dat"
"0092/023/92023359.dat"
"0092/029/92029607.dat"
"0092/042/92042329.dat"
"0092/043/92043913.dat"
"0092/044/92044457.dat"
"0092/097/92097121.dat"
"0092/097/92097122.dat"
"0092/097/92097124.dat"
"0092/130/92130363.dat"
"0092/342/92342284.dat"

a bunch of lines like that. i have random files from a directory with over 12TB of data i need to copy

here is the command i am using

C:\rclone\rclone.exe -Pvvv copy --files-from C:\rclone\prd3-rubys.txt "\\corp\apps$\ECM\LiveLink\prd3\" '\\corp\apps$\ECM\LiveLink\dev3' --multi-thread-streams 32 --fast-list
6

but what is the exact problem?

as per the log you posted,
INFO : There was nothing to transfer

7

i don't know what the problem is for sure..

is it syntax of my command or what is in the --files-from ?

the source files are all there and the destination is empty

8

for testing filters, try rclone ls
the output should be a list of all the source files based on --files-from

9

correct rclone ls works and it takes about 6 minutes to complete running as there are thousands for files. however when i run it with the --files-from it excludes the top level directories

PS Microsoft.PowerShell.Core\FileSystem::\\corp\apps$\ECM\LiveLink\prd3> C:\rclone\rclone.exe -vvv ls --files-from C:\rclone\prd3-rubys.txt '\\corp\apps$\ECM\LiveLink\prd3'
C:\rclone\rclone.exe : 2022/05/04 17:43:14 DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\rclone\\rclone.exe" "-vvv" "ls" "--files-from" 
"C:\\rclone\\prd3-rubys.txt" "\\\\corp\\apps$\\ECM\\LiveLink\\prd3"]
At line:1 char:1
+ C:\rclone\rclone.exe -vvv ls --files-from C:\rclone\prd3-rubys.txt '\ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022/05/04 17:4...iveLink\\prd3"]:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2022/05/04 17:43:14 DEBUG : Creating backend with remote "\\\\corp\\apps$\\ECM\\LiveLink\\prd3"
2022/05/04 17:43:14 DEBUG : Using config file from "C:\\Users\\xadmrubys\\AppData\\Roaming\\rclone\\rclone.conf"
2022/05/04 17:43:14 DEBUG : fs cache: renaming cache item "\\\\corp\\apps$\\ECM\\LiveLink\\prd3" to be canonical "//?/UNC/corp/apps$/ECM/LiveLink/prd3"
2022/05/04 17:43:14 DEBUG : 0091: Excluded
2022/05/04 17:43:14 DEBUG : 0092: Excluded
2022/05/04 17:43:14 DEBUG : 0093: Excluded
2022/05/04 17:43:14 DEBUG : 2 go routines active
10

--- are you running rclone direct on the command line, remotely using powershell or what?
--- should remove the quotes from the .txt file, unless the quotes are part of the filename

imho, to speed up testing,
--- for the source, pick a single subfolder
\\corp\apps$\ECM\LiveLink\prd3\0091\866
--- create a new test file with a single entry
/91866781.dat

11

i have tried from CLI (both CMD and pshell)

removed the quotes from the txt

as i test i will try just 1 file... and get back to you

12

just curious what is hosting the smb share? windows, synology, or what?

13

NetApp

if it wasnt a 60TB volume i would just snapmirror this and call it done
but the app needs a "dev" copy of only a slightly smaller subset of this data
i have tried robocopy/rsync/xcopy none of which really works elegantly .. i've had much better luck with rclone in these types of scenarios (until now)

15

You don't want the " on the lines so remove those

What are you trying to acheive here? Copy files called 0091/866/91866781.dat anywhere in the file systems? If so that is the right syntax.

If you want it just at the root of the transfer then you want /0091/866/91866781.dat - this will be much more efficient as rclone doesn't have to look through the whole disk.

Check it is working with

rclone lsf --files-from prd3-rubys.txt \path\to\test
16

ok i have updated the from file and made sure all the syntax in command is correct, it is now working.

C:\rclone\rclone.exe -Pvvv copy --files-from-raw C:\rclone\prd5.txt '\\corp\apps$\ECM\Livelink\prd5\' '\\corp\apps$\ECM\LiveLink\dev5' --multi-thread-streams 32 --fast-list

thanks everyone for the tips and helping get the wheels rolling in the right direction

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.