Chunker issue in windows

Help and Support
1

I'm having issues with accessing chunker files on Windows through File Stream that were created in Linux.

On Linux:

rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none
[gdrive]
type = drive
service_account_file = /example.json
team_drive = xxxxxx
root_folder_id =

[gchunk]
type = chunker   
remote = gdrive:chunks
chunk_size = 1G

On Windows

rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none
[gdrive]
type = local

[gchunk]
type = chunker   
remote = gdrive:G:/Shared drives/xxx/chunks
chunk_size = 1G

All it does is lists all the files in the chunk folder (i.e. all chunk pieces).

2021/06/25 20:11:55 DEBUG : Using config file from "C:\\Users\\zzz\\.config\\rclone\\rclone.conf"
2021/06/25 20:11:55 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vv" "ls" "gchunk:"]
2021/06/25 20:11:55 DEBUG : Creating backend with remote "gchunk:"
2021/06/25 20:11:55 DEBUG : Creating backend with remote "gdrive:G:/Shared Drives/xxx/chunks/"
2021/06/25 20:11:55 DEBUG : fs cache: renaming cache item "gdrive:G:/Shared Drives/xxx/chunks/" to be canonical "gdrive://?/G:/Shared Drives/xxx/chunks/"
2021/06/25 20:11:55 DEBUG : Reset feature "ListR"

...followed by list of files including chunks...

Does anyone know why this is happening or what I'm doing wrong?

2

Can you post the full debug log? There isn't any information of value in the snippet.

3

The rest of the debug log contains the chunk files:

2021/06/25 20:11:55 DEBUG : Using config file from "C:\\Users\\zzz\\.config\\rclone\\rclone.conf"
2021/06/25 20:11:55 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vv" "ls" "gchunk:"]
2021/06/25 20:11:55 DEBUG : Creating backend with remote "gchunk:"
2021/06/25 20:11:55 DEBUG : Creating backend with remote "gdrive:G:/Shared Drives/xxx/chunks/"
2021/06/25 20:11:55 DEBUG : fs cache: renaming cache item "gdrive:G:/Shared Drives/xxx/chunks/" to be canonical "gdrive://?/G:/Shared Drives/xxx/chunks/"
2021/06/25 20:11:55 DEBUG : Reset feature "ListR"
       84 xxx.dat
1073741824 xxx.dat.rclone_chunk.001.001_o02xux
1073741824 xxx.dat.rclone_chunk.002.002_o02xux
1073741824 xxx.dat.rclone_chunk.003.003_o02xux
1073741824 xxx.dat.rclone_chunk.004.004_o02xux
1073741824 xxx.dat.rclone_chunk.005.005_o02xux
1073741824 xxx.dat.rclone_chunk.006.006_o02xux
385067411 xxx.dat.rclone_chunk.007.007_o02xux
2021/06/25 20:11:55 DEBUG : 2 go routines active

Any ideas?

4

Can you post the full log?

5

That is the full log.

6

And what does the log look like on the other machine that works?

7

I believe I have found the issue. In Google Drive, the chunk file names are not appearing correctly:

.rclone_chunk.001.001_o02xux

Should be:

.rclone_chunk.001

On the web browser the file names show up perfectly fine. I have no idea why this is happening or how to resolve this. So it is not an rclone issue.

8

While new file upload is in progress, chunker will name file parts like xxx.dat.rclone_chunk.003_o02xux, where the name segment after underscore is a random upload identifier. When all parts uploaded completely, it will rename them on server side to xxx.dat.rclone_chunk.003 (without random suffix) and create a small json named xxx.dat with total file size and optional checksums. If gdrive returned errors for rename operations, the whole upload would have failed.
The file names you see xxx.dat.rclone_chunk.003.003_o02xux have chunk number doubled, they were not created by chunker. Either google drive or a human or I don't know what must have tampered with the names.

1 Like
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.