Improving rclone usability for simple downloading operations

Feature
1

rclone is a great tool, especially when it's set up once for long-term usage on a single machine.

However, I think there is an opportunity to improve it's usability and accessibility for some groups of users.

There are some tasks where the current setup overhead might be excessive.

On example of such task is downloading files/directories from public locations or locations that the current user already has access to.
Some concrete examples.

Example 1

I have 200 public Google drive URLs like this: https://drive.google.com/file/d/XXXXXXX/view?usp=sharing
I wish I could do this: rclone copy https://drive.google.com/file/d/XXXXXXX/view?usp=sharing ~/local_dir/
or rclone copy :google_drive:https://drive.google.com/file/d/XXXXXXX/view?usp=sharing ~/local_dir/

I've found this answer about downloading from Google Drive on StackOverflov and it's pretty disheartening: How can I download the large file from google drive using rclone? - Stack Overflow

It looks like for every one of those 200 Google Drive links I'll have to create a new remote using rclone config and then use that remote. This runs opposite to the idea of automation. At this rate, it would be faster for me to manually download those 200 files using a mouse and a browser compared to setting up rclone 200 times.

Example 2

I have a public Google Cloud Storage bucket. I can access it anonymously using gsutil ls gs://my-bucket
I want to mount this bucket.
I wish I could just do this: rclone mount gs://my-bucket or rclone mount :google_cloud_storage:my-bucket

Example 3

I have a non-public Google Cloud Storage bucket, but the cloud machine is set in a way that tools can access that bucket (GOOGLE_APPLICATION_CREDENTIALS or Workflow Identity). I can access the bucket using gsutil ls gs://my-bucket
I want to sync this bucket. I wish I could just do this: rclone copy gs://my-bucket ~/local_dir or rclone copy :google_cloud_storage:my-bucket ~/local_dir

Unfortunately, rclone does not seem to understand such syntax.

I see three issues here:

  1. No obvious support for for downloading available data without setting up remotes. This hurts programmability.
  2. Lack of support for canonical storage URIs (gs://, s3:// etc) and service URLs (e.g. https://drive.google.com/...).
  3. No ability to specify remote information as part of the command-line. rclone copy :google_cloud_storage:my-bucket ~/local_dir --remote-config={...}

In contrast, tools like gallery-dl support many dozens of "backends" that each handle a diverse set of URLs and the user can start downloading data without any setup by just running gallery-dl https://some/url.

Proposals

Proposal 1: Each supported backend should have a default remote available.

If rclone supports Google Drive, then there should be a google-drive backend.
The default backend remote configuration should use the default system credentials (if available) and fall back to anonymous access.

Proposal 2: Support canonical URI formats.

rclone should recognize URIs like gs://... or s3://... and map them to corresponding backends.

Proposal 3: Support service URL formats.

rclone should recognize URIs like https://drive.google.com/... and map them to corresponding backends and paths.

2

Create a single google drive remote.

Use the rclone backend copyid command to copy each ID assuming they are files. If the source is a folder set root_folder_id. Let's say you called your remote gdrive: you'd then override folder_id using gdrive,root_folder_id=XXXXX:

You can use :gcs: to refer to the uncofigured google cloud storage. I think the backend will complain about no credentials if you do that so you'll need to set the anonymous flag, eg :gcs,anonymous:bucket - that should do it hopefully.

Use the latest beta (since this is a new feature), and use :gcs,env_auth:my-bucket

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.