Copy files from multiple links

Rishi_Vora · December 28, 2021, 3:18pm

What is the problem you are having with rclone?

Hello, with reference to this post, I tried using command something like this

rclone copyurl --files-from urls2.txt --http-url https://index.txcloud.ga :http: TX:/ -a

but I get error

2021/12/28 20:26:13 ERROR : Attempt 1/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 20:26:13 ERROR : Attempt 2/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 20:26:13 ERROR : Attempt 3/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 20:26:13 Failed to copyurl: parse ":http:": missing protocol scheme

The rclone config contents with secrets removed

[TX]
type = drive
client_id = removed
client_secret = removed
scope = drive
service_account_file = .\accounts\0.json
service_account_file_path = .\accounts\
team_drive = removed
root_folder_id = 

[txcloud]
type = http
url = https://index.txcloud.ga

A log from the command with the `-vv` flag

2021/12/28 21:26:51 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copyurl" "--files-from" "urls2.txt" "--http-url" "https://index.txcloud.ga" ":http:" "TX:/" "-a" "-vv"]
2021/12/28 21:26:51 DEBUG : Creating backend with remote "TX:/"
2021/12/28 21:26:51 DEBUG : Using config file from "C:\\Users\\Admin\\AppData\\Roaming\\rclone\\rclone.conf"
2021/12/28 21:26:51 DEBUG : fs cache: renaming cache item "TX:/" to be canonical "TX:"
2021/12/28 21:26:51 ERROR : Attempt 1/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 21:26:51 ERROR : Attempt 2/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 21:26:51 ERROR : Attempt 3/3 failed with 1 errors and: parse ":http:": missing protocol scheme
2021/12/28 21:26:51 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s

2021/12/28 21:26:51 DEBUG : 2 go routines active
2021/12/28 21:26:51 Failed to copyurl: parse ":http:": missing protocol scheme

What is your rclone version (output from `rclone version`)

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

https to Google Drive

asdffdsa · December 28, 2021, 3:24pm

hello and welcome to the forum,

when you posted, there was a template of questions., none of which you answered.
help us to help you and answer all the questions.

Rishi_Vora · December 28, 2021, 4:00pm

So sorry for that, I have updated it now.

asdffdsa · December 28, 2021, 4:04pm

in the command, you use a remote named TX: but i do not see that in the config file?

Rishi_Vora · December 28, 2021, 4:07pm

I was using 2 different configs, might have sent wrong, corrected that, but issue is still the same.

asdffdsa · December 28, 2021, 4:10pm

i think that problem is that the file path contains a colon and windows does not like that.
on my windows laptop, using wsl2 linux, i can download the file.

Failed to copyurl: parse "'https://index.txcloud.ga/0:/Movie%20Bot/%20Memoria%20(2021)/Memoria.2021.WEB.1080p.Dream.mkv'": first path segment in URL cannot contain colon

Rishi_Vora · December 28, 2021, 4:30pm

Try this one, it seems to work

rclone copyurl "https://fkdup.sharepoint.workers.dev/0:/x265lk.com/%5BeXvoiy%5Dwwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv" TX:/ -a -P

asdffdsa · December 28, 2021, 4:31pm

please add -vv for debug log, run the command and post the output

Rishi_Vora · December 28, 2021, 4:32pm

2021/12/28 22:01:56 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copyurl" "https://fkdup.sharepoint.workers.dev/0:/x265lk.com/%5BeXvoiy%5Dwwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv" "TX:/" "-a" "-P" "-vv"]
2021/12/28 22:01:56 DEBUG : Creating backend with remote "TX:/"
2021/12/28 22:01:56 DEBUG : Using config file from "C:\\Users\\Admin\\AppData\\Roaming\\rclone\\rclone.conf"
2021/12/28 22:01:56 DEBUG : fs cache: renaming cache item "TX:/" to be canonical "TX:"
2021-12-28 22:01:59 DEBUG : [eXvoiy]wwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv: File name found in url
2021-12-28 22:02:01 DEBUG : [eXvoiy]wwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv: Sending chunk 0 length 8388608
2021-12-28 22:02:02 DEBUG : [eXvoiy]wwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv: Sending chunk 8388608 length 8388608
Transferred:       14.844 MiB / 657.883 MiB, 2%, 3.336 MiB/s, ETA 3m12s
Transferred:            0 / 1, 0%
Elapsed time:         8.0s
Transferring:
 * [eXvoiy]wwwx265lk.com.…p.BluRay.x265.HEVC.mkv:  2% /657.883Mi, 3.344Mi/s, 3m12s^C

asdffdsa · December 28, 2021, 4:35pm

did not work for me

rclone.exe copyurl "https://fkdup.sharepoint.workers.dev/0:/x265lk.com/BeXvoiyDwwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv" D:\test -a -vv 
2021/12/28 11:34:49 DEBUG : Setting --config "C:\\data\\rclone\\scripts\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\scripts\\rclone.conf"
2021/12/28 11:34:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copyurl" "https://fkdup.sharepoint.workers.dev/0:/x265lk.com/BeXvoiyDwwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv" "D:\\test" "-a" "-vv"]
2021/12/28 11:34:49 DEBUG : Creating backend with remote "D:\\test"
2021/12/28 11:34:49 DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
2021/12/28 11:34:49 DEBUG : fs cache: renaming cache item "D:\\test" to be canonical "//?/D:/test"
2021/12/28 11:34:49 ERROR : Attempt 1/3 failed with 1 errors and: CopyURL failed: 404 Not Found
2021/12/28 11:34:50 ERROR : Attempt 2/3 failed with 1 errors and: CopyURL failed: 404 Not Found
2021/12/28 11:34:50 ERROR : Attempt 3/3 failed with 1 errors and: CopyURL failed: 404 Not Found

Rishi_Vora · December 28, 2021, 4:40pm

you have the parsed url, which is ofc incorrect, this is the one

https://fkdup.sharepoint.workers.dev/0:/x265lk.com/%5BeXvoiy%5Dwwwx265lk.com.Ant-Man.and.the.Wasp.2018.720p.BluRay.x265.HEVC.mkv

Could this be a bug, because it worked here ig?

Rishi_Vora · December 29, 2021, 12:49pm

So basically, is there any other way to copy from a text file having urls (base url is same) to any remote (like gdrive)?

asdffdsa · December 29, 2021, 4:30pm

hi, i wrote that script mentioned in that link.

did a quick test, and that script worked.

rclone copyurl https://index.txcloud.ga/0%3A/Movie%20Bot/%20Memoria%20(2021)/Memoria.2021.WEB.1080p.Dream.mkv D:\test -a -vv --stats-one-line --progress
DEBUG : Setting --config "C:\\data\\rclone\\scripts\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copyurl" "https://index.txcloud.ga/0%3A/Movie%20Bot/%20Memoria%20(2021)/Memoria.2021.WEB.1080p.Dream.mkv" "D:\\test" "-a" "-vv" "--stats-one-line" "--progress"]
DEBUG : Creating backend with remote "D:\\test"
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : fs cache: renaming cache item "D:\\test" to be canonical "//?/D:/test"
DEBUG : Memoria.2021.WEB.1080p.Dream.mkv: File name found in url
1.612 GiB / 2.168 GiB, 74%, 87.360 MiB/s, ETA 6s

Rishi_Vora · December 30, 2021, 12:13pm

single urls do work, but I want to pass them from a file because I have many of them.

asdffdsa · December 30, 2021, 1:47pm

yes, i understand that.

that script i wrote in https://forum.rclone.org/t/copy-files-from-multiple-links-to-my-drive/18641/5
works just fine. it will copy multiple files, which is what you want.

that debug log, was from that script.
did you try the scripts, it works well.
did you try the script, it works......

Rishi_Vora · December 30, 2021, 2:26pm

ran this on WSL

for /f %%u in (urls2.txt) do rclone copyurl %%u TX: -a

Got this error

-bash: syntax error near unexpected token `%%u'

and this is urls2.txt

https://index.txcloud.ga/2:/MR.KITTY%20-%20Discography%20(2008-2014)/Remixes/Mr.Kitty%20-%20R.M.X%20(2008)/1.%20Estrella%20(Mr.Kitty%20Remix).flac
https://index.txcloud.ga/2:/MR.KITTY%20-%20Discography%20(2008-2014)/Remixes/Mr.Kitty%20-%20R.M.X%20(2008)/2.%20All%20at%20Once%20(Mr.Kitty%20Remix).flac

asdffdsa · December 30, 2021, 2:47pm

that is for cmd.exe , not wsl2

Rishi_Vora · December 30, 2021, 3:03pm

D:\rclone>for /f %%u in (urls2.txt) do rclone copyurl %%u TX: -a
%%u was unexpected at this time.

asdffdsa · December 30, 2021, 3:08pm

run it from a .cmd batch file as that will not work direct from command line without modification.

if you want from the command line try
for /f %u in (urls2.txt) do rclone copyurl %u TX: -a

https://ss64.com/nt/syntax-args.html
"This is unlike regular variables which have both leading and trailing %'s such as %variable%, or FOR command variables which use a single leading % on the command line or a double leading %% when used in a batch file."

Rishi_Vora · December 30, 2021, 3:22pm

thanks a lot bro. but if this would have worked then it possible to do parallel transfers, right?