Rclone copy shared Box folders/files to local filesystem

enslay · February 26, 2021, 4:09pm

What is the problem you are having with rclone?

Thank you for developing this tool! I am happy to report that Rclone works great for files and folders relative to my own Box folder. However, I want to use rclone to copy files shared by someone relative to their folder. I've seen backend commands for Google Drive, but these don't work for Box.

For example:
Sorry, I am not able to properly share links even from this forum.

https://forum.rclone.org/t/how-do-i-copy-files-from-public-gdrive-links-to-my-gdrive-account/20872

This unfortunately doesn't work on Box as Box doesn't support backend commands.

I've also found this solution that copies from a shared link to a destination:

https://forum.rclone.org/t/copy-shared-link-from-onedrive-to-local/20669

If I use the shared link from Box, I get 3 errors about enumerating files/folders (although these are visible and downloadable through the browser).

I've tried to play with folder IDs and truncated portions of the shared URL. For example, /s/longID/folder/ID form... I've tried using longID as in remote:longID ... or I tried to use --box-root-folder-id with the ID from one of the shared files. No such luck.

What is your rclone version (output from `rclone version`)

rclone v1.53.1

os/arch: linux/amd64
go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

RedHat Enterprise 7.9
Linux 3.10.0-1160.11.1.el7.x86_64

Which cloud storage system are you using? (eg Google Drive)

Box

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

This is what I am trying to figure out.

Something like this doesn't work:

rclone copy remote:https://box.shared.folder.from.other.account/ temp/

Copying a link of the files/folders into my folder and then sharing this publicly doesn't work either.

The rclone config contents with secrets removed.

Paste config here

I'm sorry, I think a lot this might be sensitive information. Which bits of the config can I safely provide that are not sensitive?

A log from the command with the `-vv` flag

I'm not sure what command I should run to begin with. Even still, I need to be careful what I actually share.
Here's an example of trying to copy shared folders/files using my credentials from remote:.

2021/02/26 10:53:48 DEBUG : Using config file from "/home/XXX/.config/rclone/rclone.conf"
2021/02/26 10:53:48 DEBUG : Creating backend with remote "remote:https://box.shared.folder.link"
2021/02/26 10:53:48 DEBUG : Creating backend with remote "PartA"
2021/02/26 10:53:48 DEBUG : fs cache: renaming cache item "PartA" to be canonical "XXX"
2021/02/26 10:53:49 ERROR : : error reading source directory: directory not found
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for checks to finish
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for transfers to finish
2021/02/26 10:53:49 INFO  : There was nothing to transfer
2021/02/26 10:53:49 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021/02/26 10:53:49 ERROR : : error reading source directory: directory not found
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for checks to finish
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for transfers to finish
2021/02/26 10:53:49 INFO  : There was nothing to transfer
2021/02/26 10:53:49 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021/02/26 10:53:49 ERROR : : error reading source directory: directory not found
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for checks to finish
2021/02/26 10:53:49 DEBUG : Local file system at XXX: Waiting for transfers to finish
2021/02/26 10:53:49 INFO  : There was nothing to transfer
2021/02/26 10:53:49 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2021/02/26 10:53:49 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         4.4s

asdffdsa · February 26, 2021, 4:23pm

hello and welcome to the forum,

for that debug log, you did not include the most important part, the top lines with the exact command.
if you do not want to share the exact http url, then just change a few lettters.
the debug log does not contain any ID and passwords.

can you post the config file, just change any ID and passwors to random info.

about the shared link, is the access public?
if so, this might help.
https://rclone.org/http/#usage-without-a-config-file

enslay · February 26, 2021, 4:31pm

Thank you for the prompt response. Here's the config with (from rclone config dump)... I removed the tokens and expiration date:

{
    "remote": {
        "box_sub_type": "user",
        "token": "{\"access_token\":\"XXX\",\"token_type\":\"bearer\",\"refresh_token\":\"XXX\",\"expiry\":\"XXX\"}",
        "type": "box"
    }
}

I will try that http option without a config file, but I don't think it will work. These are shared within the organization but not publicly. They are perfectly visible and accessible from the browser... after I login. I can further share a link and make it public (I think). I will definitely try!

asdffdsa · February 26, 2021, 4:34pm

just an idea,

https://rclone.org/box/#root-folder-id
if you have permission to the folder and set the root folder, you might be able to access it.

enslay · February 26, 2021, 4:44pm

I think is a great idea! Although, I tried using --box-root-folder-id already. Here's what happens with rclone lsd:

rclone lsd --box-root-folder-id NNNNNN remote
Enter configuration password:
password:
2021/02/26 11:40:57 ERROR : : error listing: directory not found
2021/02/26 11:40:57 Failed to lsd with 2 errors: last error was: directory not found

Where I got NNNNNN from the long number "ID" (last token) in this scheme "boxUrl/s/LongID/folder/ID"

I feel like Box is just terrible if you want to grab files/folders from other accounts! I wanted to try boxcli, but I'm not likely to get authorization for a custom app to let boxcli even work (plus I'm a newbie at this stuff).

EDIT: Oh I see... need the colon (: ) after remote... similar result though...

2021/02/26 11:40:57 Failed to lsd with 2 errors: last error was: directory not found
[layns@helix temp]$ rclone lsd --box-root-folder-id NNNNNNN remote:
Enter configuration password:
password:
2021/02/26 11:55:50 ERROR : : error listing: couldn't list files: Error "not_found" (404): Not Found
2021/02/26 11:55:50 Failed to lsd with 2 errors: last error was: couldn't list files: Error "not_found" (404): Not Found

enslay · February 26, 2021, 5:03pm

Here's an interesting observation with -vv.

If you use the Browser to visit the shared folder. If you press "Save Link", a web_link ends up in my home folder. Now rclone does not seem aware of this. But if you use -vv, rclone will see a web_link and have no idea what to do.
For example:
rclone copy remote: PartA/

2021/02/26 12:01:55 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "-vv" "remote:" "PartA/"]
Enter configuration password:
password:
2021/02/26 12:01:57 DEBUG : Using config file from "/home/XXX/.config/rclone/rclone.conf"
2021/02/26 12:01:57 DEBUG : Creating backend with remote "remote:"
2021/02/26 12:01:57 DEBUG : Creating backend with remote "PartA/"
2021/02/26 12:01:57 DEBUG : fs cache: renaming cache item "PartA/" to be canonical "XXX/PartA"
2021/02/26 12:01:57 DEBUG : box root '': Ignoring "Shared Folder - PartA" - unknown type "web_link"
2021/02/26 12:01:57 DEBUG : Local file system at XXX/PartA: Waiting for checks to finish
2021/02/26 12:01:57 DEBUG : Local file system at XXX/PartA: Waiting for transfers to finish
2021/02/26 12:01:57 INFO  : There was nothing to transfer
2021/02/26 12:01:57 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         2.3s

2021/02/26 12:01:57 DEBUG : 5 go routines active

It seems like rclone just needs to be taught what "web_link" is for Box.

enslay · February 26, 2021, 8:15pm

I finally got in contact with a super Box Admin/Guru in our organization. Here's what I learned about shared folders to make rclone or ftp/ftps work properly.

The sharing party needs to share with one of the following permissions: Editor, Viewer Uploader, Previewer Uploader, or Uploader.
I think they need to explicitly list you as a collaborator. Then you get a notification in your account and the shared folder is visible in your Box account's file tree.

And then I can at least use FTPS to access the shared folder... if FTPS can do it, so too can rclone. I will probably use rclone for this very large transfer!

So it's a permission error and a newbie error on my part. So I will click this as "solved." I guess Box is not as bad as I thought it was!

EDIT: If you use FTPS and your organization uses some kind of smart card authentication for the Box service, you may need to visit your account settings and create a password (something I wasn't aware of).

asdffdsa · February 26, 2021, 8:39pm

the problem with FTP is it is unreliable, and with rclone, does not support checksums or mod-times.
how would you know if a file was corrupted on transfer?

enslay · February 26, 2021, 9:24pm

Yeah, I don't know. I'm finding LFTP to be a great FTPS client and its 'mirror' command keeps track of progress to a special logging file.

lftp -p 990 -u BoxUserID ftps://ftp.box.com
cd path/to/folder
mirror --parallel=4

I started with rclone but it was transferring at 300 KB/s over 4 threads. LFTP is pulling at 20 MB/s. I don't know why there is difference in transfer speeds.

system · March 1, 2021, 9:24pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.