Working on new FTP backend - various issues

Dev Discussions
1

Hi All,

I’m trying to implement an FTP backend for rclone (cfr. issue #540) but I’m having a few issues. I admit i am a go newbie, so I might be doing something really really stupid…

However, what happens is that when I try to list a directory the program seems stuck. The List() function terminates (as far as I can see), but then the code seems stuck in some sort of deadlock (strace says it’s stuck in a futex() call).

I’ve also tried with delve but I couldn’t really see what happens after the List() is called.

If you want to check the code it’s in my github fork at https://github.com/arcimboldo/rclone

2

Uhm, turns out it works if I call Finished(). Strange, swift backend is not calling it…

3

Exciting :smile:

swift calls defer out.Finished() and you’ll need to do that too.

I had a quick look at your code - I suggest you move your mu into the Fs if it protects the ftp.ServeConn from concurrent use. Leaving it as a global var will mean trouble if the user decides to make two ftp remotes!

The concurrency aspects will need some thought…

I hope you saw the outline of how to make a new remote here: https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend

4

Thank you.

I think I have fixed most of the concurrency issues, but I cannot run tests. It seems that the “make test” target assumes there is a configuration somewhere, how can I run the tests?

5

Follow the instructions here: https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend

For the “unit tests” section, then the “integration tests” section.

make test is used to do a complete integration test before release so probably isn’t what you want.

6

Thank you, it should now be fixed. I’ll issue a new pull request

7

I have issues with the tests. The main issue I guess is that I don’t know how to add a dependency from github.com/jlaffaye/ftp
How can I add the dependency?

8

You need to use godep: https://github.com/tools/godep#update-a-dependency

Then commit the changes to the vendor directory in a separate commit.

I just had a quick look over your code - looks like you are getting there - well done!

How are you doing with the integration tests?

9

Integration tests work. I’m also using it to complete the sync of 300TB (1000-genomes project), so far it seems to work.

About godep: I’ve tried, but I’m not sure I’m doing it right. In case, would you mind giving me more detailed instructions? (again: go newbie here :))

10

That is great news

I have to say I don’t find godep easy to use!

There must be a better way than this but…

Edit Godep/Godeps.json and add this to the end (and also an extra comma to the line before)

                {
	                "ImportPath": "github.com/jlaffaye/ftp"
			"Comment": "",
			"Rev": ""
                }

Run

go get -u github.com/jlaffaye/ftp
godep update github.com/jlaffaye/ftp

Then commit vendor/github.com/jlaffaye/ and Godeps/Godeps.json in a single commit.

11

Is this working OK? I think I’m going to try it. :slight_smile:

12

I’m having some issues but didn’t have much time to work on it.

When dealing with a very big tree, apparently not all files are processed.

13

Thanks for the reply, will give feedback.

14

Off topic but how does one get access to that data. I was trying to locate the 10000 genomes project but can't actually find anything but publications.