Storj "ghost files" when syncing

TowerBR · October 2, 2023, 11:31am

What is the problem you are having with rclone?

I'm trying to migrate some files to storj, using sync. However, it appears that rclone is creating "ghost files" with another passphrase (or none at all) even though I specify the phrase in the sync command:

rclone sync source:path storj-......:bucket_name --transfers 1 --no-update-modtime --checksum --progress --filter-from /path/to/filter_file --storj-passphrase "............"

I've made several attempts, for existing storj buckets and new ones, and the same thing always happens. I already tried to copy just one folder with 7 files, and it created 5 "ghost files". The number of "ghost files" appears to be about 80% of the "real" files:

There is no other passphrase, and I use a password manager to save it, so it's not a typo or anything like that.

I've already tried with the default configuration for Storj and using the S3 gateway.

The "ghost files" appear to have no size or are very small, as the total space reported by Storj approximately matches the total size of the files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-1040-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Storj

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync source:path storj-......:bucket_name --transfers 1 --no-update-modtime --checksum --progress --filter-from /path/to/filter_file --storj-passphrase "............"

The rclone config contents with secrets removed.

[storj-.......]
type = storj
access_grant = 1LBi1y.......erNUsM

[storj-s3-........]
type = s3
provider = Storj
access_key_id = jwi.n.....5xuba
secret_access_key = j2v6a.......wkwjhy
endpoint = gateway.storjshare.io

A log from the command with the `-vv` flag

There are no errors in the logs

asdffdsa · October 2, 2023, 12:47pm

hi, best if you can:

test using latest version of rclone.
reproduce the issue using the simplest command possible, as few flags as possible, perhaps copy a single file.
post the full debug log

ncw · October 2, 2023, 3:39pm

The Storj library got an update in 1.64 so this is a very good idea.

TowerBR · October 2, 2023, 4:16pm

Ok, updated:

rclone v1.64.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-1040-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.1
- go/linking: static
- go/tags: none

I created a completely new account for testing, new access key, new bucket (I even left the default name) and copied a 19M PDF file and then a 168M MP4, without errors or "ghost files":

23-10-02_13-06-24__chrome

So I decided to delete one of the files using the web interface to redo the sync as a test, and the error appeared as soon as I deleted the file, before even running sync:

23-10-02_13-06-46__chrome

Even though it seems to be something related to Storj, does Rclone delete any temporary files or something similar during sync?

BTW, the logs of the two operations:

1st file - PDF

2023/10/02 14:15:12 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "sync" "gdrive-user-media-rcln-crpt:filmes/zz.test/PX3321-T1_001.pdf" "storj-test:demo-bucket" "--storj-passphrase" "skin example luxury cricket judge figure aspect nation math domain innocent question" "-vv"]
2023/10/02 14:15:12 DEBUG : Creating backend with remote "gdrive-user-media-rcln-crpt:filmes/zz.test/PX3321-T1_001.pdf"
2023/10/02 14:15:12 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2023/10/02 14:15:12 DEBUG : Creating backend with remote "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/jv8vbv2umtoqdm2rbbo4fpmc9bba884vjr5vhg5et0dqsk2ri5ug"
2023/10/02 14:15:12 DEBUG : Google drive root 'media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/jv8vbv2umtoqdm2rbbo4fpmc9bba884vjr5vhg5et0dqsk2ri5ug': 'root_folder_id = 0ANiDy4aLJMfOUk9PVA' - save this in the config to speed up startup
2023/10/02 14:15:14 DEBUG : fs cache: adding new entry for parent of "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/jv8vbv2umtoqdm2rbbo4fpmc9bba884vjr5vhg5et0dqsk2ri5ug", "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk"
2023/10/02 14:15:14 DEBUG : Creating backend with remote "storj-test:demo-bucket"
2023/10/02 14:15:14 DEBUG : storj-test: detected overridden config - adding "{W8wmJ}" suffix to name
2023/10/02 14:15:14 DEBUG : FS sj://demo-bucket: connecting...
2023/10/02 14:15:14 DEBUG : FS sj://demo-bucket: connected: <nil>
2023/10/02 14:15:14 DEBUG : fs cache: renaming cache item "storj-test:demo-bucket" to be canonical "storj-test{W8wmJ}:demo-bucket"
2023/10/02 14:15:14 DEBUG : FS sj://demo-bucket: stat ./PX3321-T1_001.pdf
2023/10/02 14:15:14 DEBUG : FS sj://demo-bucket: err: uplink: object not found ("PX3321-T1_001.pdf")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:514
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1982
        github.com/rclone/rclone/fs/operations.CopyFile:2076
        github.com/rclone/rclone/cmd/sync.glob..func1.1:73
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/sync.glob..func1:69
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/10/02 14:15:14 DEBUG : PX3321-T1_001.pdf: Need to transfer - File not found at Destination
2023/10/02 14:15:19 DEBUG : FS sj://demo-bucket: cp input ./PX3321-T1_001.pdf # [HashesOption([])] 19145514
2023/10/02 14:15:22 INFO  : PX3321-T1_001.pdf: Copied (new)
2023/10/02 14:15:22 INFO  :
Transferred:       18.259 MiB / 18.259 MiB, 100%, 2.607 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         9.7s

2023/10/02 14:15:22 DEBUG : 211 go routines active

2nd file - MP4

2023/10/02 14:30:12 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "sync" "gdrive-user-media-rcln-crpt:filmes/zz.test/Enigma.mp4" "storj-test:demo-bucket" "--storj-passphrase" "skin  ... question" "-vv"]
2023/10/02 14:30:12 DEBUG : Creating backend with remote "gdrive-user-media-rcln-crpt:filmes/zz.test/Enigma.mp4"
2023/10/02 14:30:12 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2023/10/02 14:30:12 DEBUG : Creating backend with remote "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/k5driugt9e3ag1tifntsd75r3k"
2023/10/02 14:30:12 DEBUG : Google drive root 'media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/k5driugt9e3ag1tifntsd75r3k': 'root_folder_id = 0ANiDy4aLJMfOUk9PVA' - save this in the config to speed up startup
2023/10/02 14:30:14 DEBUG : fs cache: adding new entry for parent of "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk/k5driugt9e3ag1tifntsd75r3k", "gdrive-user:media-rcln-crpt/duccjut1kvlpskmo1vbsmmomek/rvkop1rqq8it9cqmi8j9ru60pk"
2023/10/02 14:30:14 DEBUG : Creating backend with remote "storj-test:demo-bucket"
2023/10/02 14:30:14 DEBUG : storj-test: detected overridden config - adding "{W8wmJ}" suffix to name
2023/10/02 14:30:14 DEBUG : FS sj://demo-bucket: connecting...
2023/10/02 14:30:14 DEBUG : FS sj://demo-bucket: connected: <nil>
2023/10/02 14:30:14 DEBUG : fs cache: renaming cache item "storj-test:demo-bucket" to be canonical "storj-test{W8wmJ}:demo-bucket"
2023/10/02 14:30:14 DEBUG : FS sj://demo-bucket: stat ./Enigma.mp4
2023/10/02 14:30:14 DEBUG : FS sj://demo-bucket: err: uplink: object not found ("Enigma.mp4")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:514
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1982
        github.com/rclone/rclone/fs/operations.CopyFile:2076
        github.com/rclone/rclone/cmd/sync.glob..func1.1:73
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/sync.glob..func1:69
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/10/02 14:30:14 DEBUG : Enigma.mp4: Need to transfer - File not found at Destination
2023/10/02 14:30:15 DEBUG : FS sj://demo-bucket: cp input ./Enigma.mp4 # [HashesOption([])] 167965680
2023/10/02 14:30:15 DEBUG : FS sj://demo-bucket: cp input ./Enigma.mp4 # [HashesOption([])] 167965680
2023/10/02 14:30:33 INFO  : Enigma.mp4: Copied (new)
2023/10/02 14:30:33 INFO  :
Transferred:      160.185 MiB / 160.185 MiB, 100%, 8.839 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        20.5s

2023/10/02 14:30:33 DEBUG : 210 go routines active

Animosity022 · October 2, 2023, 4:42pm

Still a bug, but my brain is having a tough time processing how you are using sync.

Sync for me has always been directory to directory.

I've never done a 'file' to a directory.

Even locally, that really doesn't compute and do what I'd want.

[felix@gemini two]$ rclone sync /home/felix/test/one/hosts /home/felix/test/two -vv
2023/10/02 12:40:53 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/10/02 12:40:53 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "sync" "/home/felix/test/one/hosts" "/home/felix/test/two" "-vv"]
2023/10/02 12:40:53 DEBUG : Creating backend with remote "/home/felix/test/one/hosts"
2023/10/02 12:40:53 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/10/02 12:40:53 DEBUG : fs cache: adding new entry for parent of "/home/felix/test/one/hosts", "/home/felix/test/one"
2023/10/02 12:40:53 DEBUG : Creating backend with remote "/home/felix/test/two"
2023/10/02 12:40:53 DEBUG : hosts: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/10/02 12:40:53 DEBUG : hosts: Unchanged skipping
2023/10/02 12:40:53 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

2023/10/02 12:40:53 DEBUG : 4 go routines active
[felix@gemini two]$ ls
adskfjadfq  hosts

I'd imagine that's even more wacky as S3 really doesn't have directories.

ncw · October 2, 2023, 5:19pm

No rclone doesn't do anything with temporary files on a sync to Storj. It will overwrite existing files but I don't think you did that in your test.

My guess, is that this is a bug in Storj. It looks like the files are being uploaded OK so maybe it is a bug in the web interface. It would be worth reporting it to them I think.

Animosity022 · October 2, 2023, 5:23pm

Right, but what is supposed to happen if I sync a file to a directory?

My local example, the extra files remain.

TowerBR · October 2, 2023, 5:44pm

You are right. My orignal command syncs the entire folder. When I tested with just one file, I added the file name only to the source, by mistake. But the transfer was done correctly.

I'm using storage with the Storj pattern (type = storj).

TowerBR · October 2, 2023, 5:47pm

Yes, I already did this:

TowerBR · October 4, 2023, 10:45am

Bringing the feedback, Storj guys agree that it must be a bug in the web interface:

system · October 7, 2023, 10:45am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.