Trying to copy single file via http and url changes to directory

What is the problem you are having with rclone?

Trying to copy a single from via http backend (no config)

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: debian 9.13
  • os/kernel: 4.19.75-v7l+ (armv7l)
  • os/type: linux
  • os/arch: arm
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

HTTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --http-url=https://somedomain.com/ :http:/path/subpath/myfile.txt

The rclone config contents with secrets removed.

NONE

A log from the command with the -vv flag

I see in the error its changing the url and putting a / after the txt and says it fails to read source root directory. but... i pointed to a file (no ending /) and just want to copy a single file.

thanks

hello and welcome to the forum,

need to see a debug log, so add -vv to the rclone command and post the output.
enclose it with three backticks so it is formatted like so.

rclone version -vv
2022/01/18 13:14:08 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "version" "-vv"]
rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.60.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none
2022/01/18 13:14:08 DEBUG : rclone: Version "v1.57.0" finishing with parameters ["rclone" "version" "-vv"]

I've updated it a bit to remove sensitive stuff... tell me if it helps. thanks


rclone copy --no-check-certificate=true "--http-url=https://somedomain.com" ":http:/public/root.txt" "./" -vv
2022/01/18 18:17:51 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "copy" "--no-check-certificate=true" "--http-url=https://somedomain.com" ":http:/public/root.txt" "./" "-vv"]
2022/01/18 18:17:51 DEBUG : Creating backend with remote ":http:/public/root.txt"
2022/01/18 18:17:51 DEBUG : :http: detected overridden config - adding "{nh4pO}" suffix to name
2022/01/18 18:17:51 NOTICE: Config file "/home/pi/.config/rclone/rclone.conf" not found - using defaults
2022/01/18 18:17:51 DEBUG : fs cache: renaming cache item ":http:/public/root.txt" to be canonical ":http{nh4pO}:/public/root.txt"
2022/01/18 18:17:51 DEBUG : Creating backend with remote "./"
2022/01/18 18:17:51 DEBUG : fs cache: renaming cache item "./" to be canonical "/home/pi/workspace"
2022/01/18 18:17:51 ERROR : https://somedomain.com/public/root.txt/: error reading source root directory: error listing "": directory not found
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for checks to finish
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for transfers to finish
2022/01/18 18:17:51 ERROR : Attempt 1/3 failed with 1 errors and: error listing "": directory not found
2022/01/18 18:17:51 ERROR : https://somedomain.com/public/root.txt/: error reading source root directory: error listing "": directory not found
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for checks to finish
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for transfers to finish
2022/01/18 18:17:51 ERROR : Attempt 2/3 failed with 1 errors and: error listing "": directory not found
2022/01/18 18:17:51 ERROR : https://somedomain.com/public/root.txt/: error reading source root directory: error listing "": directory not found
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for checks to finish
2022/01/18 18:17:51 DEBUG : Local file system at /home/pi/workspace: Waiting for transfers to finish
2022/01/18 18:17:51 ERROR : Attempt 3/3 failed with 1 errors and: error listing "": directory not found
2022/01/18 18:17:51 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.4s

2022/01/18 18:17:51 DEBUG : 2 go routines active
2022/01/18 18:17:51 Failed to copy: error listing "": directory not found

not sure the need to use quotes.

rclone copy --http-url=https://downloads.rclone.org :http:/v1.57.0/version.txt ./ -vv
DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "--http-url=https://downloads.rclone.org" ":http:/v1.57.0/version.txt" "./" "-vv"]
DEBUG : Creating backend with remote ":http:/v1.57.0/version.txt"
DEBUG : :http: detected overridden config - adding "{Zhl7M}" suffix to name
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : fs cache: adding new entry for parent of ":http:/v1.57.0/version.txt", ":http{Zhl7M}:/v1.57.0/version.txt"
DEBUG : Creating backend with remote "./"
DEBUG : fs cache: renaming cache item "./" to be canonical "/mnt/c/data/rclone/scripts/rr/other"
DEBUG : version.txt: Need to transfer - File not found at Destination
INFO  : version.txt: Copied (new)

mot sure rclone sees the quotes but I'll give it a try and update as soon as i can. thanks.

ok, found that the issue is that head request is failing due to security which impacts the GET afterwards and turns it into a directory (adds a /).
i didn't find a way to prevent the head request from going (--http-no-head isn't working for copy command).

Not all HTTP servers play nice as some use javascript and odd things which make standard commands that scrape not work.

Without knowing what site you are hitting, it's really tough to tell what's going on and why as we can't inspect anything.

--http-no-head will turn off all the HTTP HEAD requests, except for the first one which checks to see whether the root is a file or a directory.

That is affecting you because you are trying to copy individual files.

You might be better off using

rclone copyurl https://somedomain.com//path/subpath/myfile.txt myfile.txt

This just does a single GET request.

$ rclone -vv --dump bodies copyurl https://beta.rclone.org/version.txt /tmp/version.txt
2022/01/20 12:21:27 DEBUG : rclone: Version "v1.58.0-beta.5935.48d55d6fa.fix-5917-pcloud-mtime" starting with parameters ["rclone" "-vv" "--dump" "bodies" "copyurl" "https://beta.rclone.org/version.txt" "/tmp/version.txt"]
2022/01/20 12:21:27 DEBUG : Creating backend with remote "/tmp/"
2022/01/20 12:21:27 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2022/01/20 12:21:27 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/01/20 12:21:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/01/20 12:21:27 DEBUG : HTTP REQUEST (req 0xc0005c9d00)
2022/01/20 12:21:27 DEBUG : GET /version.txt HTTP/1.1
Host: beta.rclone.org
User-Agent: rclone/v1.58.0-beta.5935.48d55d6fa.fix-5917-pcloud-mtime
Accept-Encoding: gzip

2022/01/20 12:21:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/01/20 12:21:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/01/20 12:21:27 DEBUG : HTTP RESPONSE (req 0xc0005c9d00)
2022/01/20 12:21:27 DEBUG : HTTP/2.0 200 OK
Content-Length: 35
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Thu, 20 Jan 2022 12:21:27 GMT
Etag: "r5x7urz"
Last-Modified: Tue, 18 Jan 2022 19:37:39 GMT
Server: Caddy

rclone v1.58.0-beta.5948.c504d9701
2022/01/20 12:21:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/01/20 12:21:27 INFO  : 
Transferred:   	         35 B / 35 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2022/01/20 12:21:27 DEBUG : 5 go routines active
1 Like

thanks. there is no documentation on copyurl at Documentation

Or, in general, as I mention in the linked issue ticket, could make sure the root is a directory path (which matches the fallback assumption when the initial head request fails), and then use filtering to pick the file(s):

rclone copy --http-url=http://somedomain.com/ :http:/path/subpath --include /myfile.txt

But yes, copyurl is probably the perfect fit in this case.

1 Like

(You have to go via "Overview", the drop-down only shows the most common)

1 Like

Maybe it should show all commands? Or at least the top level ones. Though currently they are sorted in order of popularity.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.