Downloading large files from GoogleDrive direct in iDrive cloud storage using AWS VM

Chuckero · June 27, 2021, 4:20pm

Hello everyone,
I'm new with rclone, just discovering how ease is to install and use it, awesome! Also this is my first project using rclone and I'm facing some difficulties to make it to work.

What is the problem you are having with rclone?

I'm trying do download a large file (100GB) from a GoogleDrive link directly to my iDrive Cloud storage (www.idrive.com/cloud/) using an AWS virtual machine. I mounted a virtual drive unit (H:) in that VM, but rclone is using disk storage as a cache, and as that VM have only 11.6GB free disk there is no way to make a copy of this file. At least, for me this seems to be the problem.
I'm downloading the file using FireFox browser, also tried Chrome and Free Download Manager.
I can copy small files from VM and my local computer with no problems.
So, which is a good command line to make this task works?
Thanks in advance!!

What is your rclone version (output from `rclone version`)

rclone v1.55.1

os/type: windows
os/arch: amd64
go/version: go1.16.3
go/linking: dynamic
go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

I have an AWS Free Tier virtual machine, created just to test some cloud storage, this machine uses Windows Server 2016 Datacenter x64 V10.0.14393, 1 vCore, 1 GB RAM, 30GB hard drive (11.6GB free).

Which cloud storage system are you using? (eg Google Drive)

I'm using iDrive Cloud (www.idrive.com/cloud/).
Reference: Setting RCLONE for idrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Full cache:

rclone mount --vfs-cache-mode writes --log-level INFO --config "C:\Users\Administrator\.config\rclone\rclone.conf" --log-file "C:\rclone\rcloneH.log" iDrive: H:

Minimum cache:

rclone mount --vfs-cache-mode minimal --log-level INFO --config "C:\Users\Administrator\.config\rclone\rclone.conf" --log-file "C:\rclone\rcloneH.log" iDrive: H:

No cache:

rclone mount --vfs-cache-mode off --log-level INFO --config "C:\Users\Administrator\.config\rclone\rclone.conf" --log-file "C:\rclone\rcloneH.log" iDrive: H:

The rclone config contents with secrets removed.

[iDrive]
type = s3
provider = Other
env_auth = false
access_key_id = xxxxxxxxxxxxxxxxxxx
secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
region = us-east-1
endpoint = s3.us-west-1.idrivecloud.io
location_constraint = ""
acl = private

A log from the command with the `-vv` flag

Full cache:

2021/06/27 01:20:08 INFO  : S3 root: poll-interval is not supported by this remote
2021/06/27 01:20:08 ERROR : Local file system at //?/C:/Users/Administrator/AppData/Local/rclone/vfs/iDrive: Failed to list "RemoteStore1": directory not found
2021/06/27 01:20:08 ERROR : RemoteStore1: Failed to rmdir: remove \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1: The system cannot find the file specified.
2021/06/27 01:20:08 ERROR : Local file system at //?/C:/Users/Administrator/AppData/Local/rclone/vfs/iDrive: vfs cache: failed to remove empty directories from metadata cache path "": remove \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1: The system cannot find the file specified.
2021/06/27 01:20:08 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 0 (was 0)
The service rclone has been started.
2021/06/27 01:21:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/06/27 01:21:35 INFO  : RemoteStore1/TestFile.tst: vfs cache: queuing for upload in 5s
2021/06/27 01:21:35 INFO  : RemoteStore1/TestFile.tst: vfs cache: removed cache file as file deleted
2021/06/27 01:21:36 INFO  : RemoteStore1/TestFile.tst: vfs cache: queuing for upload in 5s
2021/06/27 01:21:39 INFO  : RemoteStore1/TestFile.tst.part: vfs cache: queuing for upload in 5s
2021/06/27 01:21:41 INFO  : RemoteStore1/TestFile.tst: Copied (new)
2021/06/27 01:21:41 INFO  : RemoteStore1/TestFile.tst: vfs cache: upload succeeded try #1
2021/06/27 01:22:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 700.372M (was 700.372M)
2021/06/27 01:23:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 1.763G (was 1.763G)
2021/06/27 01:24:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 2.827G (was 2.827G)
2021/06/27 01:25:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 3.985G (was 3.985G)
2021/06/27 01:26:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 5.037G (was 5.037G)
2021/06/27 01:27:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 6.016G (was 6.016G)
2021/06/27 01:28:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 7.167G (was 7.167G)
2021/06/27 01:29:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 8.227G (was 8.227G)
2021/06/27 01:30:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 9.381G (was 9.381G)
2021/06/27 01:31:08 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 10.452G (was 10.452G)
2021/06/27 01:32:09 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 11.524G (was 11.524G)
2021/06/27 01:32:18 ERROR : IO error: write \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1\TestFile.tst.part: There is not enough space on the disk.
2021/06/27 01:32:18 ERROR : IO error: write \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1\TestFile.tst.part: There is not enough space on the disk.
The service rclone has been stopped.

Minimum cache:

2021/06/26 22:09:55 INFO  : S3 root: poll-interval is not supported by this remote
2021/06/26 22:09:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 0 (was 0)
The service rclone has been started.
2021/06/26 22:10:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/06/26 22:11:19 INFO  : RemoteStore1/FileTest.tst: vfs cache: queuing for upload in 5s
2021/06/26 22:11:19 INFO  : RemoteStore1/FileTest.tst: vfs cache: removed cache file as file deleted
2021/06/26 22:11:20 INFO  : RemoteStore1/FileTest.tst: vfs cache: queuing for upload in 5s
2021/06/26 22:11:21 INFO  : RemoteStore1/FileTest.tst.part: vfs cache: queuing for upload in 5s
2021/06/26 22:11:25 INFO  : RemoteStore1/FileTest.tst: Copied (new)
2021/06/26 22:11:25 INFO  : RemoteStore1/FileTest.tst: vfs cache: upload succeeded try #1
2021/06/26 22:11:56 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 721.862M (was 721.862M)
2021/06/26 22:12:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 1.645G (was 1.645G)
2021/06/26 22:13:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 2.603G (was 2.603G)
2021/06/26 22:14:56 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 3.557G (was 3.557G)
2021/06/26 22:15:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 4.498G (was 4.498G)
2021/06/26 22:16:56 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 5.456G (was 5.456G)
2021/06/26 22:17:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 6.397G (was 6.397G)
2021/06/26 22:18:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 7.339G (was 7.339G)
2021/06/26 22:19:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 8.295G (was 8.295G)
2021/06/26 22:20:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 9.234G (was 9.234G)
2021/06/26 22:21:56 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 10.202G (was 10.202G)
2021/06/26 22:22:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 0, total size 11.184G (was 11.184G)
2021/06/26 22:23:18 ERROR : IO error: write \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1\FileTest.tst.part: There is not enough space on the disk.
2021/06/26 22:23:18 ERROR : IO error: write \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1\FileTest.tst.part: There is not enough space on the disk.
2021/06/26 22:23:18 ERROR : IO error: write \\?\C:\Users\Administrator\AppData\Local\rclone\vfs\iDrive\RemoteStore1\FileTest.tst.part: There is not enough space on the disk.
2021/06/26 22:23:18 INFO  : RemoteStore1/FileTest.tst.part: vfs cache: queuing for upload in 5s
2021/06/26 22:23:55 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:53:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:54:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:55:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:56:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:57:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:58:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 00:59:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:00:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:01:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:02:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:03:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:04:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:05:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:06:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:07:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:08:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:09:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:10:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:11:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:12:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:13:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
2021/06/27 01:14:55 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 0, uploading 1, total size 11.629G (was 11.629G)
The service rclone has been stopped.

No cache:

2021/06/26 22:07:12 INFO  : S3 root: poll-interval is not supported by this remote
The service rclone has been started.
2021/06/26 22:08:00 ERROR : RemoteStore1/TestFile.tst.part: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2021/06/26 22:08:00 NOTICE: S3 root: Streaming uploads using chunk size 5M will have maximum file size of 48.828G
2021/06/26 22:08:35 ERROR : RemoteStore1/TestFile.tst.part: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/06/26 22:08:36 ERROR : RemoteStore1/TestFile.tst.part: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/06/26 22:08:36 ERROR : RemoteStore1/TestFile.tst.part: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
The service rclone has been stopped.

Animosity022 · June 27, 2021, 4:23pm

The logs indicate whatever program you are using to write requires write mode on so that's what you need to use. That means items are written locally first and uploaded.

asdffdsa · June 27, 2021, 4:41pm

hello and welcome to the forum,

why not use rclone copy as that does not cache files or use local storage of the VM.

Chuckero · June 30, 2021, 5:15pm

I'm trying to use the copy command, but I think I didn't understand how to use it. All the options I try, always I got some error.

For example, a simple download from Google Drive to a local folder. Why rclone is using the remote config?

rclone copy https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg C:\temp -vv

2021/06/30 13:11:26 DEBUG : Using config file from "C:\\Users\\admin\\.config\\rclone\\rclone.conf"
2021/06/30 13:11:26 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg" "C:\\temp" "-vv"]
2021/06/30 13:11:26 DEBUG : Creating backend with remote "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg"
2021/06/30 13:11:26 Failed to create file system for "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg": didn't find section in config file

Now, I'm trying to download that same file directly to my remote storage:

rclone copy https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive: -vv

2021/06/30 13:14:32 DEBUG : Using config file from "C:\\Users\\admin\\.config\\rclone\\rclone.conf"
2021/06/30 13:14:32 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg" "iDrive:" "-vv"]
2021/06/30 13:14:32 DEBUG : Creating backend with remote "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg"
2021/06/30 13:14:32 Failed to create file system for "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg": didn't find section in config file

For sure, I did not understand the logic behind rclone.

asdffdsa · June 30, 2021, 6:05pm

try
rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg C:\temp\0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg -vv

Chuckero · June 30, 2021, 9:40pm

Thanks @asdffdsa, that command worked well. Now is just missing a way to copy that file directly to the remote storage, something like this:

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive:0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg -vv

2021/06/30 17:35:24 DEBUG : Using config file from "C:\\Users\\admin\\.config\\rclone\\rclone.conf"
2021/06/30 17:35:24 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copyurl" "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg" "iDrive:0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg" "-vv"]
2021/06/30 17:35:24 DEBUG : Creating backend with remote "iDrive:"
2021/06/30 17:35:26 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: txdedc12a86fee498f9e9dc-0060dce39c, host id: txdedc12a86fee498f9e9dc-0060dce39c
2021/06/30 17:35:26 ERROR : Attempt 1/3 failed with 2 errors and: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: txdedc12a86fee498f9e9dc-0060dce39c, host id: txdedc12a86fee498f9e9dc-0060dce39c
2021/06/30 17:35:26 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: tx946fbac2d194467389d78-0060dce39d, host id: tx946fbac2d194467389d78-0060dce39d
2021/06/30 17:35:26 ERROR : Attempt 2/3 failed with 2 errors and: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: tx946fbac2d194467389d78-0060dce39d, host id: tx946fbac2d194467389d78-0060dce39d
2021/06/30 17:35:27 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: txf549b10ae72343f882c89-0060dce39d, host id: txf549b10ae72343f882c89-0060dce39d
2021/06/30 17:35:27 ERROR : Attempt 3/3 failed with 2 errors and: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: txf549b10ae72343f882c89-0060dce39d, host id: txf549b10ae72343f882c89-0060dce39d
2021/06/30 17:35:27 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         2.2s

2021/06/30 17:35:27 DEBUG : 6 go routines active
2021/06/30 17:35:27 Failed to copyurl with 2 errors: last error was: InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: txf549b10ae72343f882c89-0060dce39d, host id: txf549b10ae72343f882c89-0060dce39d

With full path to the storage:

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive:/store1/0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg -vv

2021/06/30 17:37:20 DEBUG : Using config file from "C:\\Users\\admin\\.config\\rclone\\rclone.conf"
2021/06/30 17:37:20 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copyurl" "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg" "iDrive:/store1/0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg" "-vv"]
2021/06/30 17:37:20 DEBUG : Creating backend with remote "iDrive:/store1/"
2021/06/30 17:37:20 DEBUG : fs cache: renaming cache item "iDrive:/store1/" to be canonical "iDrive:store1"
2021/06/30 17:37:21 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx1306cc0468be4b15b8da6-0060dce410, host id: tx1306cc0468be4b15b8da6-0060dce410
2021/06/30 17:37:21 ERROR : Attempt 1/3 failed with 2 errors and: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx1306cc0468be4b15b8da6-0060dce410, host id: tx1306cc0468be4b15b8da6-0060dce410
2021/06/30 17:37:22 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx1cbf161b014f4168a815e-0060dce410, host id: tx1cbf161b014f4168a815e-0060dce410
2021/06/30 17:37:22 ERROR : Attempt 2/3 failed with 2 errors and: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx1cbf161b014f4168a815e-0060dce410, host id: tx1cbf161b014f4168a815e-0060dce410
2021/06/30 17:37:22 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx3c0115c2bb0240bb9fb60-0060dce411, host id: tx3c0115c2bb0240bb9fb60-0060dce411
2021/06/30 17:37:22 ERROR : Attempt 3/3 failed with 2 errors and: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx3c0115c2bb0240bb9fb60-0060dce411, host id: tx3c0115c2bb0240bb9fb60-0060dce411
2021/06/30 17:37:22 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         2.3s

2021/06/30 17:37:22 DEBUG : 6 go routines active
2021/06/30 17:37:22 Failed to copyurl with 2 errors: last error was: InvalidLocationConstraint: The specified location constraint is not valid.
        status code: 400, request id: tx3c0115c2bb0240bb9fb60-0060dce411, host id: tx3c0115c2bb0240bb9fb60-0060dce411

asdffdsa · June 30, 2021, 10:13pm

based on the error The specified location constraint is not valid. there is something wrong with the config file.

perhaps remove and/or set the correct value for location_constraint
i use s3 storage and never had that in any of my config files.

perhaps, the region and/or the endpoint are not correct.
one is us-east-1 and the other is us-west-1
that does not look correct to me.

i ran these commands and they worked

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg  wasabi01:urlcopytest/0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg -vv 
DEBUG : rclone: Version "v1.55.1" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copyurl" "https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg" "wasabi01:urlcopytest/0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg" "-vv"]
DEBUG : Creating backend with remote "wasabi01:urlcopytest/"
DEBUG : fs cache: renaming cache item "wasabi01:urlcopytest/" to be canonical "wasabi01:urlcopytest"
INFO  : S3 bucket urlcopytest: Bucket "urlcopytest" created with ACL "private"
INFO  : 
Transferred:   	   49.290k / 49.290 kBytes, 100%, 64.058 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.3s

rclone ls wasabi01:urlcopytest 
    50473 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg

Chuckero · July 1, 2021, 1:18am

Thanks again @asdffdsa, but when I mount the virtual drive everything works well, no problems at all:

rclone mount --vfs-cache-mode writes --log-level INFO --log-file "C:\rclone\rclone.log" iDrive: H:

In the config file, I tried to change the

region = us-east-1

to

region = us-west-1

But i got an error saying to use "us-east-1":

2021/06/30 21:12:23 ERROR : 0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg: Post request put error: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-west-1' is wrong; expecting 'us-east-1'
        status code: 400, request id: txce37a85e6d96467ab0557-0060dd1677, host id: txce37a85e6d96467ab0557-0060dd1677

In the iDrive docs also indicates to use "us-east-1" (Basic Concepts - IDrive Cloud Developer Guide). I read almost everything in the guides and no clue about what to use in the "location_constraint".

I tried different settings but always the same error:

location_constraint = ""
location_constraint = "us-east-1"
location_constraint = "us-west-1"

status code: 400, request id: tx79a1bd8911824564a85b8-0060dd17ab, host id: tx79a1bd8911824564a85b8-0060dd17ab
2021/06/30 21:17:31 ERROR : Attempt 1/3 failed with 2 errors and: InvalidLocationConstraint: The specified location constraint is not valid.

asdffdsa · July 1, 2021, 12:25pm

this works for me

[idrive]
type = s3
provider = Other
access_key_id = 
secret_access_key = 
endpoint = s3.us-west-1.idrivecloud.io

Chuckero · July 1, 2021, 4:18pm

FINALLY!

It's working with "us-east-1" without double quotes:

location_constraint = us-east-1

This is my final config file:

[iDrive]
type = s3
provider = Other
env_auth = false
access_key_id = xxxxxxxxxxxxxxxxxxxx
secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
region = us-east-1
endpoint = s3.us-west-1.idrivecloud.io
location_constraint = us-east-1
acl = private

And this is the command line:

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive:store1/0B9o1MNFt5ld1N3k1cm9tVnZxQjg.jpg -vv

This also works:

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive:\store1 --auto-filename --print-filename

And a command line for large files with progress info:

rclone copyurl https://drive.google.com/uc?id=0B9o1MNFt5ld1N3k1cm9tVnZxQjg iDrive:store1 --auto-filename -vv --progress

NOTE: it always take some seconds before update the remote storage.

Thank you all for the help!

asdffdsa · July 1, 2021, 4:51pm

good we got it working.

system · July 4, 2021, 4:52pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.