Rclone copyurl problem

[haris_semeon]

What is the problem you are having with rclone?

I cant consistently upload link to my remote,sometime it work,sometime not.
This one im using service account,using normal remote also facing same problem.
Sometime it failed 1st,then it will pass 2nd time, sometime not at all.
For context,this is debrid link

Run the command 'rclone version' and share the full output of the command.

rclone v1.56.0

• os/version: Microsoft Windows 10 Pro 2009 (64 bit)
• os/kernel: 10.0.19042.1466 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.16.5
• go/linking: dynamic
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google team drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
```rclone copyurl "link" leech: -a -P -vv```


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

Paste config here

type = drive
scope = drive
service_account_file = C:/Users/Anon\Downloads\program/rclone-v1.56.0-windows-amd64/rclone-v1.56.0-windows-amd64/accounts/39f74922c2d6e8a2da5ca7681c24d769a55b3164.json
team_drive = 0AKzP8dZl046uUk9PVA

[hsemeon]
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
token = {}
team_drive = 0APp59WvGl6ObUk9PVA



#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

2022/02/01 00:15:58 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copyurl" "https://of8b14.dl/2c0p1auc191/3246.part2.rar" "leech:" "-a" "-P" "-vv"]
2022/02/01 00:15:58 DEBUG : Creating backend with remote "leech:"
2022/02/01 00:15:58 DEBUG : Using config file from "C:\\Users\\Anon\\AppData\\Roaming\\rclone\\rclone.conf"
2022-02-01 00:16:00 DEBUG : 3246.part2.rar: File name found in url
2022-02-01 00:17:21 DEBUG : pacer: low level retry 1/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:21 DEBUG : pacer: Rate limited, increasing sleep to 1.496430461s
2022-02-01 00:17:22 DEBUG : 3246.part2.rar: Sending chunk 687865856 length 8388608
2022-02-01 00:17:22 DEBUG : pacer: low level retry 2/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:22 DEBUG : pacer: Rate limited, increasing sleep to 2.546591491s
2022-02-01 00:17:23 DEBUG : 3246.part2.rar: Sending chunk 687865856 length 8388608
2022-02-01 00:17:24 DEBUG : pacer: low level retry 3/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:24 DEBUG : pacer: Rate limited, increasing sleep to 4.063951959s
2022-02-01 00:17:26 DEBUG : 3246.part2.rar: Sending chunk 687865856 length 8388608
2022-02-01 00:17:26 DEBUG : pacer: low level retry 4/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:26 DEBUG : pacer: Rate limited, increasing sleep to 8.749204975s
2022-02-01 00:17:30 DEBUG : 3246.part2.rar: Sending chunk 687865856 length 8388608
2022-02-01 00:17:30 DEBUG : pacer: low level retry 5/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:30 DEBUG : pacer: Rate limited, increasing sleep to 16.267293519s
2022-02-01 00:17:38 DEBUG : 3246.part2.rar: Sending chunk 687865856 length 8388608
2022-02-01 00:17:39 DEBUG : pacer: low level retry 6/10 (error Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdt2XnxAmjx0gY6IH_Q9Pi1oqaLDmNwDckjxc9EMbh2u3dl27Tk4wifulAIe6RtEbqOpOFYD83oche986bPD69-YPN6UZA": unexpected EOF)
2022-02-01 00:17:39 DEBUG : pacer: Rate limited, increasing sleep to 16.698418183s

[asdffdsa (jojothehumanmonkey)]

hi,
--- the redacted config file is missing?
--- the complete log file, or at least the top lines, are missing?

make sure to create a client id for gdrive.

[haris_semeon]

yes already create id,in fact i make it two for service account.
im using service account,for this one,so only that available in config.
i attached some other config for other remote,but idk it will help.
forgot to mention,the problem occur when it nearly finish,speed become slow suddently n that error log occur

[ncw (Nick Craig-Wood)]

The error unexpected EOF likely means that rclone failed to download the entire file for some reason - perhaps rate limiting on the source...

If you run with -vv --dump headers --low-level-retries 1 --retries 1 then you can get rclone to dump the HTTP transactions which might help.

[haris_semeon]

ok will try later. any option command can i do next if it still happen?

[asdffdsa (jojothehumanmonkey)]

@ncw

-vv --dump headers --low-level-retries 1 --retries 1

might be better to save the output to a log log file
--dump headers --low-level-retries 1 --retries 1 --log-level DEBUG --log-file rclone.log

[haris_semeon]

@asdffdsa

--dump headers

2022/02/02 04:57:55 DEBUG : 3 go routines active
2022/02/02 04:57:55 Failed to copyurl with 2 errors: last error was: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdtIRfQ_HSBk4xW14CUed0OitGsJ7yBZaAGLBcdN4sue0l-y21K085nfAawIA3yt7MvQ-txkAK0hFDMy923E8Ve149ehkQ": unexpected EOF

any option i still have?

[asdffdsa (jojothehumanmonkey)]

can you post the full debug log, enclose it with three backticks so it looks like

2022/02/02 04:57:55 DEBUG : 3 go routines active
2022/02/02 04:57:55 Failed to copyurl with 2 errors: last error was: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ADPycdtIRfQ_HSBk4xW14CUed0OitGsJ7yBZaAGLBcdN4sue0l-y21K085nfAawIA3yt7MvQ-txkAK0hFDMy923E8Ve149ehkQ": unexpected EOF

and what is debrid link?

[haris_semeon]

alldebrid link.link dont have problem. i trying download with idm 1st

[haris_semeon]

try using this method,but got error too many request. any improvement can i do with this command to work

Speed slow compared to IDM

Download managers tend to use multiple streams which copyurl does not do. You can do something like this and up the streams if you want. Here is how it works to figure out number of streams: https://rclone.org/docs/#multi-thread-cutoff-size felix@gemini:~$ rclone copy --http-url http://ipv4.download.thinkbroadband.com :http:1GB.zip /home/felix -vv --multi-thread-streams 8 --multi-thread-cutoff 128M 2021/03/13 09:06:21 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "copy"…