TestSyncWithTrackRenames failing with new pixeldrain backend

Dev Discussions
1

Hi everyone! I'm building a new backend for my new cloud storage system on https://pixeldrain.com (which is currently still in closed beta). It's going really well so far, I managed to get most of the functionality working in a single day.

All the operations tests are passing with flying colours, but there is one test in the sync test suite which I'm tripping over. That's TestSyncWithTrackRenames.

I added logging to each API endpoint to track what's going on:

=== RUN   TestSyncWithTrackRenames
    run.go:180: Remote "pixeldrain root 'rclone-test-wonehub3domubij9fuxafel8'", Local "Local file system at /tmp/rclone3652194870", Modify Window "1ms"
    sync_test.go:1039: Can track renames: true
List ''
Put 'potato'
Put 'yam'
List ''
List ''
NewObject 'yaml'
Object 'yaml' does not exist
Move 'yam' 'yaml'
Remove 'yaml'
List ''
    fstest.go:298: Sleeping for 1s for list eventual consistency: 1/3
List ''
    fstest.go:298: Sleeping for 2s for list eventual consistency: 2/3
List ''
    fstest.go:298: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:305: 
                Error Trace:    /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:305
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:338
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:350
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/run.go:340
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fs/sync/sync_test.go:1056
                Error:          Should be true
                Test:           TestSyncWithTrackRenames
                Messages:       listing wrong, want
                                  potato (14), yaml (11) got
                                  potato (14)
    fstest.go:203: Not found "yaml"
    fstest.go:206: 
                Error Trace:    /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:206
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:310
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:338
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/fstest.go:350
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fstest/run.go:340
                                                        /home/wim/Documents/workspace/go/src/github.com/rclone/fs/sync/sync_test.go:1056
                Error:          Not equal: 
                                expected: 0
                                actual  : 1
                Test:           TestSyncWithTrackRenames
                Messages:       1 objects not found
List ''
Remove 'potato'
List ''
--- FAIL: TestSyncWithTrackRenames (7.53s)

What happens here is that the test creates the files potato and yam. Then it checks if yaml exists, which doesn't. Then it moves yam to yaml and removes yaml. Then the test is surprised that yaml was removed.

It seems to me that pixeldrain's behaviour is correct, yaml should not exist because it was deleted in the instruction before it. What is going wrong here?

Thanks in advance,
Fornax.

2

Here's my code, if you'd like to snoop around: https://github.com/Fornaxian/rclone/blob/master/backend/pixeldrain/pixeldrain.go

3

Haven't taken a close look, but I don't think it is supposed to be removing yaml. The test renames yam to yaml on local and then checks that yam was server-side moved to yaml on the remote. It expects that after the sync, yaml should exist and yam should not.

So I think the key will be figuring out why it is trying to remove yaml.

4

I figured it out. My Move function was modifying the source object which confused rclone. Instead I need to fetch the updated object from the API with a second API call.

I'll see if I can make it more efficient by returning the updated information directly from the rename API.

5

Oh my, that took some doing.

image
image780×484 47.4 KB

Your test suite even discovered a couple of bugs on the server side. I think I'll run these tests every time I change something on the server side from now on.

6

Well done! That is quite an achievement.

Yes, the test suite has found quite a few bugs in cloud storage systems over the e years!

1 Like
7

Does rclone support file permissions? Pixeldrain supports getting and setting file permissions, but I don't see a feature interface in rclone to implement it.

Running chmod in my mounted directory does nothing. It would be nice if this was supported.

8

@ncw In this thread you say:

rclone doesn't maintain permissions or ownership as it isn't a concept which maps to cloud storage.

I don't really agree with that. File permissions are fundamental properties of a file, just like the file's name, create/mod-time and size. Every other sync tool for Linux can preserve file permissions. Omitting file permissions means that it is not possible to create a full backup of your system as all the executables will stop working. Copying a git repo to the cloud is also not possible as most git repos depend heavily on permissions. These are really important use cases!

9

In backends which support metadata you can preserve permissions. So if you sync with -M/--metadata from local to s3 you can preserve permissions/xattr/ownership/etc. Not all backends support metadata though...

10

Thanks, I'll see if I can get it working. Do the metadata permissions work in rclone mount?

11

Not yet, though that is fairly high on the priority list!

12

Would be lovely if that could be implemented. The pixeldrain backend supports file mode and birth time metadata now.

There is a pull request here: Add pixeldrain backend by Fornax96 · Pull Request #7460 · rclone/rclone · GitHub

Side note: If you are not comfortable with merging support for an unreleased cloud storage system then I don't have a problem with maintaining my own fork for a while until it gains more adoption.

13

I'll take a look.

At minimum I'd need an account for the integration tester.

I think I wouldn't want to merge this until it was publically available otherwise the rclone docs are going to point to things which aren't available.

You can also maintain a backend as a plugin. Though maintaining a fork with a single backend in is pretty easy as backends are pretty self contained.

14

At minimum I'd need an account for the integration tester.

No problem at all. Post a username and I'll load it up with credits.

I think I wouldn't want to merge this until it was publically available otherwise the rclone docs are going to point to things which aren't available.

In the merge request I put in the docs that pixeldrain is in closed beta. Though the beta is not really restricted except that it requires a subscription. I put all the information about how to access it in pixeldrain.md. It's only closed in that I don't advertise the feature on my front page.

You can also maintain a backend as a plugin. Though maintaining a fork with a single backend in is pretty easy as backends are pretty self contained.

Yea, I noticed that keeping the fork updated is as simple as clicking Sync on github once in a while.

Thanks for your time.

15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.