Go modularization & getting rid of vendoring

Now that Go v1.13 is stable we have a much more powerful v2 modules that can fully replace GOPATH and vendoring while still providing security through checksumming.
cc @ncw

As such, let's use this thread to discuss the pros/cons of a complete switch to modules, removing vendoring. I'll start:

Pros:

  • Cleaner and smaller tree, dependency code not part of repo anymore.
  • Dependencies can be added+upgraded easier, single version change instead of large commits + PRs.
  • More modern, encouraged way of handling deps in Go.
  • Better tooling: Dependency + Security Vulnerability bots, etc.

Cons:

  • Much slower build if Go module support is disabled, since dependencies have to be git-cloned to GOPATH. Risk for malicious code injection since GOPATH doesn't do checksumming.
  • Module v2 builds only supported in Go 1.13+, which is not bundled in most system repositories yet.

I will start a few branches with different module layouts in the coming days. Let's see how it turns out.

Before we do this we need to deprecate building with go1.11 probably. Rclone's policy is to support 4 releases of go, currently this is go1.13, go1.12, go1.11 and go1.10.

It is my plan to do this eventually and you'll see in the CI that there is a fully go modules enabled build which ignores the vendor directory, so currently rclone is supporting both.

I don't think we can lose the vendor directory just yet though!

2 Likes

Before we do this we need to deprecate building with go1.11 probably. Rclone's policy is to support 4 releases of go, currently this is go1.13, go1.12, go1.11 and go1.10.

Sounds good, let's pick up the discussion about removing vendoring next year at Go 1.15 again :wink:

What do you think about using multiple Go modules? My idea was to use one for each backend, one for the core of rclone and one for the cmd frontend. That way we minimize third-party deps in rclone core.
We could keep them all in a single repo using replace directives.

If they could live in the same repo that would probably be liveable with!

I haven't committed to keeping the backend interface stable and I don't really want to at the moment which may be a problem with go's major version policy.

1 Like

Is it possible to structure the modules such that they would lend themselves to programmatic access of rclones functionality? I have a project where I would like to interface with rclone without having to make clunky subprocess calls.

You can use the backends without using the rest of rclone fairly easily. I haven't committed to a stable API for that, but I don't break it very often!

If you are willing to use rclone's config file then it is very easy, if you want to supply your own config then it is a bit harder but still possible.

This example uses rclone's config file

package main

import (
	"context"
	"fmt"
	"log"

	_ "github.com/rclone/rclone/backend/drive"
	"github.com/rclone/rclone/fs"
)

func main() {
	f, err := fs.NewFs("drive:")
	if err != nil {
		log.Fatal(err)
	}
	entries, err := f.List(context.Background(), "")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(entries)
	for i, o := range entries {
		fmt.Printf("object[%i] = (%T) %v\n", i, o, o)
	}
}