Box getting 403 errors when using chunker (working fine without it)

[vashp2029 (Vash Patel)]

What is the problem you are having with rclone?

Rclone is able to upload to my Box remote fine if I attempt to push it directly to the the remote without the chunker layer. However, when attempting to copy something through the chunker layer, it gives me 403 errors regardless of filesize being greater or smaller than the threshold (4.9GB).

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0-beta.7174.3ed4a2e96
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.15.0-78-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21rc3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Box > Chunker

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The following fails, when using a file of 20 kb:

rclone copy "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt" "box_chunker:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/" -vv
----

2023/07/27 21:04:50 DEBUG : rclone: Version "v1.64.0-beta.7174.3ed4a2e96" starting with parameters ["rclone" "copy" "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt" "box_chunker:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/" "-vv"]
2023/07/27 21:04:50 DEBUG : Creating backend with remote "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt"
2023/07/27 21:04:50 DEBUG : Using config file from "/home/vash/.config/rclone/rclone.conf"
2023/07/27 21:04:50 DEBUG : fs cache: adding new entry for parent of "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt", "/home/vash"
2023/07/27 21:04:50 DEBUG : Creating backend with remote "box_chunker:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/"
2023/07/27 21:04:50 DEBUG : Creating backend with remote "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32"
2023/07/27 21:04:53 DEBUG : Reset feature "ListR"
2023/07/27 21:04:53 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Need to transfer - File not found at Destination
2023/07/27 21:04:54 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: skip slow MD5 on source file, hashing in-transit
2023/07/27 21:05:00 DEBUG : Couldn't decode error response: EOF
2023/07/27 21:05:00 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn7s86: Couldn't move: Error "403 Forbidden" (403)
2023/07/27 21:05:01 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn7s86: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 56 49 52 55 50 49 54 56 49 55 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 56 49 52 55 50 49 54 56 49 55 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/27 21:05:01 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Failed to copy: Error "403 Forbidden" (403)
2023/07/27 21:05:01 ERROR : Attempt 1/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/27 21:05:02 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Need to transfer - File not found at Destination
2023/07/27 21:05:02 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: skip slow MD5 on source file, hashing in-transit
2023/07/27 21:05:06 DEBUG : Couldn't decode error response: EOF
2023/07/27 21:05:06 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn80i1: Couldn't move: Error "403 Forbidden" (403)
2023/07/27 21:05:08 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn80i1: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 56 49 50 56 49 57 48 50 52 55 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 56 49 50 56 49 57 48 50 52 55 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/27 21:05:08 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Failed to copy: Error "403 Forbidden" (403)
2023/07/27 21:05:08 ERROR : Attempt 2/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/27 21:05:08 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Need to transfer - File not found at Destination
2023/07/27 21:05:09 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: skip slow MD5 on source file, hashing in-transit
2023/07/27 21:05:12 DEBUG : Couldn't decode error response: EOF
2023/07/27 21:05:12 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn87we: Couldn't move: Error "403 Forbidden" (403)
2023/07/27 21:05:14 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt - [Chunk 00001]_hn87we: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 56 49 53 50 54 51 57 48 48 53 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 56 49 53 50 54 51 57 48 48 53 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/27 21:05:14 ERROR : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Failed to copy: Error "403 Forbidden" (403)
2023/07/27 21:05:14 ERROR : Attempt 3/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/27 21:05:14 INFO  : 
Transferred:       60.158 KiB / 60.158 KiB, 100%, 2.969 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:        23.2s

2023/07/27 21:05:14 DEBUG : 8 go routines active
2023/07/27 21:05:14 Failed to copy: Error "403 Forbidden" (403)

The following works, using the same 20 kb file:

rclone copy "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt" "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/" -vv
----

2023/07/27 21:05:57 DEBUG : rclone: Version "v1.64.0-beta.7174.3ed4a2e96" starting with parameters ["rclone" "copy" "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt" "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/" "-vv"]
2023/07/27 21:05:57 DEBUG : Creating backend with remote "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt"
2023/07/27 21:05:57 DEBUG : Using config file from "/home/vash/.config/rclone/rclone.conf"
2023/07/27 21:05:57 DEBUG : fs cache: adding new entry for parent of "./Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt", "/home/vash"
2023/07/27 21:05:57 DEBUG : Creating backend with remote "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/"
2023/07/27 21:06:00 DEBUG : fs cache: renaming cache item "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32/" to be canonical "box:Media/TV/Hindi_HD/Devon Ke Dev. Mahadev (2011) [imdb-tt2147999]/Season 32"
2023/07/27 21:06:00 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Need to transfer - File not found at Destination
2023/07/27 21:06:02 DEBUG : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: sha1 = b058bd701acc17ad57a277d5c6fed23a06f6a0bd OK
2023/07/27 21:06:02 INFO  : Devon Ke Dev. Mahadev (2011) - S32E02 - Kartikey vows to fight Lohitang [HDTV-720p][AAC 2.0][h264].srt: Copied (new)
2023/07/27 21:06:02 INFO  : 
Transferred:       20.053 KiB / 20.053 KiB, 100%, 20.049 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         5.2s

2023/07/27 21:06:02 DEBUG : 8 go routines active

The following fails, using a file that is 4.95 gb:

rclone copy ./tautulli.tar box_chunker: -vv
----

2023/07/27 21:23:21 DEBUG : rclone: Version "v1.64.0-beta.7174.3ed4a2e96" starting with parameters ["rclone" "copy" "./tautulli.tar" "box_chunker:" "-vv"]
2023/07/27 21:23:21 DEBUG : Creating backend with remote "./tautulli.tar"
2023/07/27 21:23:21 DEBUG : Using config file from "/home/vash/.config/rclone/rclone.conf"
2023/07/27 21:23:21 DEBUG : fs cache: adding new entry for parent of "./tautulli.tar", "/mnt/local/Backups/Saltbox/opt"
2023/07/27 21:23:21 DEBUG : Creating backend with remote "box_chunker:"
2023/07/27 21:23:21 DEBUG : Creating backend with remote "box:"
2023/07/27 21:23:21 DEBUG : Reset feature "ListR"
2023/07/27 21:23:21 DEBUG : tautulli.tar: Need to transfer - File not found at Destination
2023/07/27 21:23:22 DEBUG : tautulli.tar: skip slow MD5 on source file, hashing in-transit
2023/07/27 21:23:23 DEBUG : tautulli.tar - [Chunk 00001]_ho2khm: Multipart upload session started for 79 parts of size 64Mi
2023/07/27 21:23:23 DEBUG : tautulli.tar - [Chunk 00001]_ho2khm: Uploading part 1/79 offset 0/4.900Gi part size 64Mi
...
2023/07/27 21:14:02 DEBUG : tautulli.tar - [Chunk 00001]_hnj9c1: Uploading part 79/79 offset 4.875Gi/4.900Gi part size 64Mi
2023/07/27 21:14:07 DEBUG : tautulli.tar - [Chunk 00001]_hnj9c1: commit multipart upload failed 1/100 - trying again in 9 seconds (not ready yet)
2023/07/27 21:14:19 DEBUG : tautulli.tar - [Chunk 00002]_hnj9c1: Multipart upload session started for 8 parts of size 8Mi
2023/07/27 21:14:19 DEBUG : tautulli.tar - [Chunk 00002]_hnj9c1: Uploading part 1/8 offset 0/56.541Mi part size 8Mi
...
2023/07/27 21:14:21 DEBUG : tautulli.tar - [Chunk 00002]_hnj9c1: Uploading part 8/8 offset 56Mi/56.541Mi part size 8Mi
2023/07/27 21:14:29 DEBUG : Couldn't decode error response: EOF
2023/07/27 21:14:29 ERROR : tautulli.tar - [Chunk 00001]_hnj9c1: Couldn't move: Error "403 Forbidden" (403)
2023/07/27 21:14:29 DEBUG : Couldn't decode error response: EOF
2023/07/27 21:14:29 ERROR : tautulli.tar - [Chunk 00001]_hnj9c1: Couldn't move: Error "403 Forbidden" (403)
2023/07/27 21:14:31 ERROR : tautulli.tar: Failed to copy: Error "403 Forbidden" (403)
2023/07/27 21:14:31 ERROR : Attempt 3/3 failed with 2 errors and: Error "403 Forbidden" (403)
2023/07/27 21:14:31 INFO  : 
Transferred:       14.866 GiB / 14.866 GiB, 100%, 40.413 MiB/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:                 6 / 6, 100%
Elapsed time:      7m52.4s

2023/07/27 21:14:31 DEBUG : 7 go routines active
2023/07/27 21:14:31 Failed to copy with 2 errors: last error was: Error "403 Forbidden" (403)

The following works, using the same 4.95 gb file:

rclone copy ./tautulli.tar box: -vv
----

2023/07/27 21:24:09 DEBUG : rclone: Version "v1.64.0-beta.7174.3ed4a2e96" starting with parameters ["rclone" "copy" "./tautulli.tar" "box:" "-vv"]
2023/07/27 21:24:09 DEBUG : Creating backend with remote "./tautulli.tar"
2023/07/27 21:24:09 DEBUG : Using config file from "/home/vash/.config/rclone/rclone.conf"
2023/07/27 21:24:09 DEBUG : fs cache: adding new entry for parent of "./tautulli.tar", "/mnt/local/Backups/Saltbox/opt"
2023/07/27 21:24:09 DEBUG : Creating backend with remote "box:"
2023/07/27 21:24:10 DEBUG : tautulli.tar: Need to transfer - File not found at Destination
2023/07/27 21:24:11 DEBUG : tautulli.tar: Multipart upload session started for 80 parts of size 64Mi
2023/07/27 21:24:11 DEBUG : tautulli.tar: Uploading part 1/80 offset 0/4.955Gi part size 64Mi
...
2023/07/27 21:26:21 DEBUG : tautulli.tar: Uploading part 80/80 offset 4.938Gi/4.955Gi part size 64Mi
2023/07/27 21:26:26 DEBUG : tautulli.tar: commit multipart upload failed 1/100 - trying again in 5 seconds (not ready yet)
2023/07/27 21:26:32 DEBUG : tautulli.tar: sha1 = 527bc68af05c663f33fc16bc7d175e635faf2995 OK
2023/07/27 21:26:32 INFO  : tautulli.tar: Copied (new)
2023/07/27 21:26:32 INFO  : 
Transferred:        4.955 GiB / 4.955 GiB, 100%, 18.681 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      2m22.8s

2023/07/27 21:26:32 DEBUG : 8 go routines active

The rclone config contents with secrets removed.

[box]
type = box
client_id = [redacted]
access_token = [redacted]
box_sub_type = enterprise
root_folder_id = [redacted]

[box_chunker]
type = chunker
remote = box:
chunk_size = 4.900Gi
name_format = * - [Chunk #####]

[kapitainsky]

most likely reason is use of spaces in chunker name_format, change:

@vashp2029

name_format = * - [Chunk #####]

to e.g.:

name_format = *.rcc###

I would also suggest you add encryption to your remotes. Not only it will keep your data safe but will allow to use longer path than by default - I have noticed you use extremely long file names:)

See all example here:

Recommended Box remote setup

in response to: Box has max file size limitation for most of their plans so it is required to use chunker: rclone.config: [box_remote] type = box token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} [crypt_remote] type = crypt remote = box_remote:data password = XXXX password2 = XXXX filename_encoding = base32768 [chunker_remote] type = chunker remote = crypt_remote: chunk_size = 49G name_format = *.rcc### hash_type = sha1all Create data folder on you…

[vashp2029 (Vash Patel)]

That was a good point about the filename length so I did what you suggested and added an encryption layer. I also changed the name_format so that my new config is:

[box]
type = box
client_id = [redacted]
access_token = [redacted]
box_sub_type = enterprise
root_folder_id = [redacted]

[box_chunker]
type = chunker
remote = box:
chunk_size = 4.900Gi
name_format = *.rcc#####

[box_crypt]
type = crypt
remote = box_chunker:
password = [redacted]

However, I'm getting the same issue still:

> rclone copy test_file.txt box_crypt: -vv

2023/07/29 14:33:47 DEBUG : rclone: Version "v1.64.0-beta.7174.3ed4a2e96" starting with parameters ["rclone" "copy" "test_file.txt" "box_crypt:" "-vv"]
2023/07/29 14:33:47 DEBUG : Creating backend with remote "test_file.txt"
2023/07/29 14:33:47 DEBUG : Using config file from "/home/vash/.config/rclone/rclone.conf"
2023/07/29 14:33:47 DEBUG : fs cache: adding new entry for parent of "test_file.txt", "/home/vash"
2023/07/29 14:33:47 DEBUG : Creating backend with remote "box_crypt:"
2023/07/29 14:33:47 DEBUG : Creating backend with remote "box_chunker:"
2023/07/29 14:33:47 DEBUG : Creating backend with remote "box:"
2023/07/29 14:33:47 DEBUG : Reset feature "ListR"
2023/07/29 14:33:48 DEBUG : test_file.txt: Need to transfer - File not found at Destination
2023/07/29 14:33:48 DEBUG : test_file.txt: Computing md5 hash of encrypted source
2023/07/29 14:33:51 DEBUG : Couldn't decode error response: EOF
2023/07/29 14:33:51 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kufyii: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 14:33:52 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kufyii: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 52 55 55 49 50 56 56 53 48 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 52 55 55 49 50 56 56 53 48 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 14:33:52 ERROR : test_file.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 14:33:52 ERROR : Attempt 1/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 14:33:53 DEBUG : test_file.txt: Need to transfer - File not found at Destination
2023/07/29 14:33:53 DEBUG : test_file.txt: Computing md5 hash of encrypted source
2023/07/29 14:33:56 DEBUG : Couldn't decode error response: EOF
2023/07/29 14:33:56 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kug3uy: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 14:33:57 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kug3uy: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 52 55 55 50 52 54 48 48 55 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 52 55 55 50 52 54 48 48 55 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 14:33:57 ERROR : test_file.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 14:33:57 ERROR : Attempt 2/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 14:33:58 DEBUG : test_file.txt: Need to transfer - File not found at Destination
2023/07/29 14:33:58 DEBUG : test_file.txt: Computing md5 hash of encrypted source
2023/07/29 14:34:01 DEBUG : Couldn't decode error response: EOF
2023/07/29 14:34:01 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kug8vv: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 14:34:02 ERROR : rn0unjl2dt9rfdlrtkpnlvr0ik.rcc00001_kug8vv: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 52 55 56 50 52 57 50 52 56 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 52 55 56 50 52 57 50 52 56 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 14:34:02 ERROR : test_file.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 14:34:02 ERROR : Attempt 3/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 14:34:02 INFO  : 
Transferred:       60.299 KiB / 60.299 KiB, 100%, 4.307 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:        14.8s

2023/07/29 14:34:02 DEBUG : 8 go routines active
2023/07/29 14:34:02 Failed to copy: Error "403 Forbidden" (403)

Another thing I tested was to setup the encryption layer right on top of the box remote, skipping the chunker for now--and that resolves my error. However, re-introducing the chunker layer either before or after the encryption layer gives me the same EOF error as above. So both encryption > chunker > remote and chunker > encryption > remote give me errors. However, encryption > remote works fine, as does simply running commands on the remote itself. Therefore, I think its safe to assume that the problem is occurring with the chunker layer for whatever reason.

This is the alternative config I tried in which both box: and box_crypt are fully functional, but NOT box_chunker.

[box]
type = box
client_id = [redacted]
access_token = [redacted]
box_sub_type = enterprise
root_folder_id = [redacted]

[box_chunker]
type = chunker
remote = box_crypt:
chunk_size = 4.900Gi
name_format = *.rcc#####

[box_crypt]
type = crypt
remote = box:
password = [redacted]

[ncw (Nick Craig-Wood)]

If you want to make the filenames shorter then try filename_encoding = base32768 which works well with box. See docs for filename_encoding.

[kapitainsky]

Yes default encoding will actually make filenames longer:) base32768 recommendation was mentioned in link I sent.

But where Couldn't move: Error "403 Forbidden" (403) is coming from for chunker is puzzling.

[kapitainsky]

could you try

chunk_size = 4G

instead of:

chunk_size = 4.900Gi

As per docs I think your 4.900Gi is not something rclone understands.

[vashp2029 (Vash Patel)]

Thank you (and @ncw) for the note on changing filename encoding, did that. With regards to the chunker, I changed the chunk_size to 4G and tried again with no success. I'm getting the same results as above for both configurations (chunker on top of encryption layer and vice versa).

> rclone copy new.txt box_crypt: -vv

2023/07/29 19:37:25 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "copy" "new.txt" "box_crypt:" "-vv"]
2023/07/29 19:37:25 DEBUG : Creating backend with remote "new.txt"
2023/07/29 19:37:25 DEBUG : Using config file from "c:\\portable\\rclone\\rclone.conf"
2023/07/29 19:37:25 DEBUG : fs cache: adding new entry for parent of "new.txt", "//?/C:/Users/vpate/Desktop"
2023/07/29 19:37:25 DEBUG : Creating backend with remote "box_crypt:"
2023/07/29 19:37:25 DEBUG : Creating backend with remote "box_chunker:"
2023/07/29 19:37:25 DEBUG : Creating backend with remote "box:"
2023/07/29 19:37:25 DEBUG : Reset feature "ListR"
2023/07/29 19:37:25 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:37:26 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:37:28 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:37:28 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q077: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:37:29 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q077: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 52 51 50 55 53 53 56 55 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 52 51 50 55 53 53 56 55 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:37:29 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:37:29 ERROR : Attempt 1/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:37:29 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:37:29 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:37:32 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:37:32 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q3wn: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:37:33 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q3wn: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 53 48 50 56 56 48 52 57 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 53 48 50 56 56 48 52 57 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:37:33 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:37:33 ERROR : Attempt 2/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:37:33 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:37:33 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:37:35 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:37:35 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q7sg: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:37:36 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l5q7sg: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 53 48 55 51 52 53 51 50 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 53 48 55 51 52 53 51 50 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:37:36 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:37:36 ERROR : Attempt 3/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:37:36 INFO  :
Transferred:            408 B / 408 B, 100%, 37 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:        11.6s

2023/07/29 19:37:36 DEBUG : 5 go routines active
2023/07/29 19:37:36 Failed to copy: Error "403 Forbidden" (403)

I had the box WebUI up while doing this and I noticed that it shows a message stating that the file was successfully uploaded, but rclone (rightfully) tells me it failed. When I refresh the WebUI, the file is not present--nor is it in the trash.

[kapitainsky]

could you try the same with v1.63.1 of rclone instead of the latest beta? Maybe there is some regression bug you discovered.

[vashp2029 (Vash Patel)]

Sure, I tried that as well:

> rclone --version

rclone v1.63.1
- os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
- os/kernel: 10.0.22000.2176 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: cmount

But same errors:

rclone copy new.txt box_crypt: -vv
2023/07/29 19:43:56 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "copy" "new.txt" "box_crypt:" "-vv"]
2023/07/29 19:43:56 DEBUG : Creating backend with remote "new.txt"
2023/07/29 19:43:56 DEBUG : Using config file from "c:\\portable\\rclone\\rclone.conf"
2023/07/29 19:43:56 DEBUG : fs cache: adding new entry for parent of "new.txt", "//?/C:/Users/vpate/Desktop"
2023/07/29 19:43:56 DEBUG : Creating backend with remote "box_crypt:"
2023/07/29 19:43:56 DEBUG : Creating backend with remote "box_chunker:"
2023/07/29 19:43:56 DEBUG : Creating backend with remote "box:"
2023/07/29 19:43:56 DEBUG : Reset feature "ListR"
2023/07/29 19:43:57 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:43:57 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:43:59 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:43:59 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l60v1k: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:44:00 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l60v1k: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 53 50 57 48 56 52 54 55 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 53 50 57 48 56 52 54 55 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:44:00 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:44:00 ERROR : Attempt 1/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:44:00 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:44:01 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:44:03 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:44:03 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l60z8h: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:44:04 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l60z8h: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 52 57 53 48 55 54 51 48 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 52 57 53 48 55 54 51 48 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:44:04 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:44:04 ERROR : Attempt 2/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:44:04 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:44:04 DEBUG : new.txt: Computing md5 hash of encrypted source
2023/07/29 19:44:07 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:44:07 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l612wa: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:44:07 ERROR : 捂嘔穛䝟睵鷝⏒ဇ佟.rcc00001_l612wa: Failed to remove temporary chunk: Error "not_found" (404): Not Found ([123 34 101 114 114 111 114 115 34 58 91 123 34 114 101 97 115 111 110 34 58 34 105 110 118 97 108 105 100 95 112 97 114 97 109 101 116 101 114 34 44 34 110 97 109 101 34 58 34 102 105 108 101 34 44 34 109 101 115 115 97 103 101 34 58 34 73 110 118 97 108 105 100 32 118 97 108 117 101 32 39 49 50 54 57 53 51 57 50 57 56 50 57 56 39 46 32 39 102 105 108 101 39 32 119 105 116 104 32 118 97 108 117 101 32 39 49 50 54 57 53 51 57 50 57 56 50 57 56 39 32 110 111 116 32 102 111 117 110 100 34 125 93 125])
2023/07/29 19:44:07 ERROR : new.txt: Failed to copy: Error "403 Forbidden" (403)
2023/07/29 19:44:07 ERROR : Attempt 3/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:44:07 INFO  :
Transferred:            408 B / 408 B, 100%, 40 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:        11.0s

2023/07/29 19:44:07 DEBUG : 5 go routines active
2023/07/29 19:44:07 Failed to copy: Error "403 Forbidden" (403)

Another interesting note, if I run the random group of numbers that box returns in the error message through this translator, I get the following human-legible message:

{
  "errors": [
    {
      "reason": "invalid_parameter",
      "name": "file",
      "message": "Invalid value '1269539298298'. 'file' with value '1269539298298' not found"
    }
  ]
}

This makes me think that on upload, the chunk is being assigned some file ID but I'm not sure what occurs afterwards since the file is not visible either in the WebUI or via rclone. Box at least intermittently believes the file has uploaded given the WebUI toast message I linked in the last post. I really have no idea what's going on or how to test this further.

[kapitainsky]

Second potential issue I see is that error:

Couldn't move: Error "403 Forbidden" (403)

indicates that your:

client_id = [redacted]

does not have required privileges for moving files. You can create new file but can not move/rename (it would explain why things work without chunker as chunker uploads file and then renames it). I am not familiar with box client_id configuration but you should double check what this client_is is allowed to do.

[kapitainsky]

Thank you for testing it. At least we know that there is no new bug.

[kapitainsky]

you could test it by uploading some test file directly to box and then trying to rclone move it

[vashp2029 (Vash Patel)]

You're right about the move permissions. To test it, I tried uploading the test file directly to the box remote (which has been working fine) and then tried to move to a different directory:

> rclone move box:new.txt box:test_folder -vv

2023/07/29 19:52:37 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "move" "box:new.txt" "box:test_folder" "-vv"]
2023/07/29 19:52:37 DEBUG : Creating backend with remote "box:new.txt"
2023/07/29 19:52:37 DEBUG : Using config file from "c:\\portable\\rclone\\rclone.conf"
2023/07/29 19:52:38 DEBUG : fs cache: adding new entry for parent of "box:new.txt", "box:"
2023/07/29 19:52:38 DEBUG : Creating backend with remote "box:test_folder"
2023/07/29 19:52:39 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:52:39 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:52:39 ERROR : new.txt: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:52:39 ERROR : Attempt 1/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:52:40 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:52:41 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:52:41 ERROR : new.txt: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:52:41 ERROR : Attempt 2/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:52:41 DEBUG : new.txt: Need to transfer - File not found at Destination
2023/07/29 19:52:42 DEBUG : Couldn't decode error response: EOF
2023/07/29 19:52:42 ERROR : new.txt: Couldn't move: Error "403 Forbidden" (403)
2023/07/29 19:52:42 ERROR : Attempt 3/3 failed with 1 errors and: Error "403 Forbidden" (403)
2023/07/29 19:52:42 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         4.6s

2023/07/29 19:52:42 DEBUG : 3 go routines active
2023/07/29 19:52:42 Failed to move: Error "403 Forbidden" (403)

So you're right that there does seem to be an issue with the rename/move privileges. I'm not exactly sure why since the service account attached to this client ID has Co-Owner privileges for the entire root directory. Also, if the issue is truly with the renaming of the file, shouldn't I still see the un-renamed chunk either in the directory or in the trash?

[kapitainsky]

Your post is not the first one showing strange box behaviour when client_id is used:

Box remote won't auto-refresh access token

This means that the token you got from box didn't have a refresh token so rclone couldn't refresh it. This might be a setting in your client_id that needs fixing. Google drive does this when your client_id isn't set to public but I'm not sure about box. That should give you an area to look at.

@vashp2029

Also, if the issue is truly with the renaming of the file, shouldn't I still see the un-renamed chunk either in the directory or in the trash?

as chunker can't rename the file it cleans temporary uploads - this is expected. I would assume it does not use thrash - which would be right thing to do. It would require looking into source code to check, Either way your test with rclone move confirms what the issue is. Your client_id can not rename files. Why? I have no idea:)

[vashp2029 (Vash Patel)]

The post you linked is actually my post from when I was attempting to set this account up

[vashp2029 (Vash Patel)]

I'm having trouble getting any of the other authorization methods to work. There's no documentation on Box's end that shows how to create an auto-refreshing access token when using either of the server authentication methods (JWT or client credentials grant) and when using the OAuth method, rclone isn't able to start the webserver for some reason.

[kapitainsky]

hahaha. At the same time I see posts from other people using box with chunker and not having any issues. And some I think used client_id too.... Hard for me to comment as I have no knowledge of this box aspect. But it is almost certain that when you remove client_id all will work.

[kapitainsky]

We also can not also rule out yet an issue on rclone side - maybe client_id is not used 100% as it should be.

[vashp2029 (Vash Patel)]

Okay, so I set up a new box remote as follows:

[box2]
type = box
box_config_file = box.json
root_folder_id = [redacted]
box_sub_type = enterprise
token = {"access_token":"[redacted]","token_type":"bearer","expiry":"2023-07-29T20:30:25.4796582-04:00"}

However, then I have the issue that rclone does not auto-refresh the access token and when it expires in 60 minutes, I have to manually run rclone config reconnect box2: to get another hour of access. This is actually what I initially posted about and you linked to that thread in your message above.

> rclone lsd box2: -vv

2023/07/29 20:31:06 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "lsd" "box2:" "-vv"]
2023/07/29 20:31:06 DEBUG : Creating backend with remote "box2:"
2023/07/29 20:31:06 DEBUG : Using config file from "c:\\portable\\rclone\\rclone.conf"
2023/07/29 20:31:06 DEBUG : box root '': Token expired - 1 uploads in progress - refreshing
2023/07/29 20:31:06 DEBUG : box2: Loaded invalid token from config file - ignoring
2023/07/29 20:31:06 ERROR : : error listing: couldn't list files: Get "https://api.box.com/2.0/folders/218658551144/items?fields=type%2Cid%2Csequence_id%2Cetag%2Csha1%2Cname%2Csize%2Ccreated_at%2Cmodified_at%2Ccontent_created_at%2Ccontent_modified_at%2Citem_status%2Cshared_link%2Cowned_by&limit=1000&usemarker=true": token expired and there's no refresh token - manually refresh with "rclone config reconnect box2:"
2023/07/29 20:31:06 DEBUG : 4 go routines active
2023/07/29 20:31:06 Failed to lsd with 2 errors: last error was: couldn't list files: Get "https://api.box.com/2.0/folders/218658551144/items?fields=type%2Cid%2Csequence_id%2Cetag%2Csha1%2Cname%2Csize%2Ccreated_at%2Cmodified_at%2Ccontent_created_at%2Ccontent_modified_at%2Citem_status%2Cshared_link%2Cowned_by&limit=1000&usemarker=true": token expired and there's no refresh token - manually refresh with "rclone config reconnect box2:"

[kapitainsky]

definitely we have some issue here - either on box or rclone end