Install from source: go get errors

Ajaja · July 26, 2020, 8:29am

I've been compiling Rclone for myself using "go get" for some time, but lately starting to recieve errors (with a clean GOPATH enviroment):

M:\gopath>M:\go\bin\go get -u github.com/rclone/rclone
package github.com/putdotio/go-putio/putio: cannot find package "github.com/putdotio/go-putio/putio" in any of:
	M:\go\src\github.com\putdotio\go-putio\putio (from $GOROOT)
	M:\gopath\src\github.com\putdotio\go-putio\putio (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/config: cannot find package "github.com/yunify/qingstor-sdk-go/v3/config" in any of:
	M:\go\src\github.com\yunify\qingstor-sdk-go\v3\config (from $GOROOT)
	M:\gopath\src\github.com\yunify\qingstor-sdk-go\v3\config (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/request/errors: cannot find package "github.com/yunify/qingstor-sdk-go/v3/request/errors" in any of:
	M:\go\src\github.com\yunify\qingstor-sdk-go\v3\request\errors (from $GOROOT)
	M:\gopath\src\github.com\yunify\qingstor-sdk-go\v3\request\errors (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/service: cannot find package "github.com/yunify/qingstor-sdk-go/v3/service" in any of:
	M:\go\src\github.com\yunify\qingstor-sdk-go\v3\service (from $GOROOT)
	M:\gopath\src\github.com\yunify\qingstor-sdk-go\v3\service (from $GOPATH)
package github.com/spacemonkeygo/monkit: code in directory M:\gopath\src\github.com\spacemonkeygo\monkit expects import "github.com/spacemonkeygo/monkit/v3"
unrecognized import path "goftp.io/server/core": reading https://goftp.io/server/core?go-get=1: 404 Not Found

Go v1.14.6 (Windows, x86-64).

I had to switch to

go build

it works.

Any suggestions how to fix "go get"?

ncw · July 26, 2020, 9:53am

Rclone switched from using the vendor directory to go modules.

Try clearing your module cache with go clean -modcache and if that doesn't work try go clean -cache

Ajaja · July 26, 2020, 12:02pm

I got this error just after I updated Go to v1.14.6 and cleaned everything in GOPATH (mod, cache, everything). Then I returned back to v1.14.5 , cleaned everything again, got the same errors.

Could you look into this code?

github.com

rclone/rclone/blob/711736054fada7ee0044db04bc4feeed1f80c42e/cmd/serve/ftp/ftp.go#L29


	"github.com/rclone/rclone/cmd/serve/proxy/proxyflags"
	"github.com/rclone/rclone/fs"
	"github.com/rclone/rclone/fs/accounting"
	"github.com/rclone/rclone/fs/config/flags"
	"github.com/rclone/rclone/fs/log"
	"github.com/rclone/rclone/fs/rc"
	"github.com/rclone/rclone/vfs"
	"github.com/rclone/rclone/vfs/vfsflags"
	"github.com/spf13/cobra"
	"github.com/spf13/pflag"
	ftp "goftp.io/server/core"
)

// Options contains options for the http Server
type Options struct {
	//TODO add more options
	ListenAddr   string // Port to listen on
	PublicIP     string // Passive ports range
	PassivePorts string // Passive ports range
	BasicUser    string // single username for basic auth if not using Htpasswd
	BasicPass    string // password for BasicUser

Looks like goftp.io/server/core is 404 now

ncw · July 26, 2020, 12:51pm

That code is ok I think, it builds in the CI on 3 different platforms.

What happens if you do

go get goftp.io/server/core

On its own?

Ajaja · July 26, 2020, 2:55pm

I get:

unrecognized import path "goftp.io/server/core": reading https://goftp.io/server/core?go-get=1: 404 Not Found

Ajaja · July 26, 2020, 3:03pm

But

set GO111MODULE=on
go get goftp.io/server/core

works, BTW.
I'll try now to build Rclone with GO111MODULE...

ncw · July 26, 2020, 3:05pm

That is strange... Did you have GO111MODULE set before?

I think you shouldn't need it set

Ajaja · July 26, 2020, 3:24pm

I did not.

Yes, with GO111MODULE I get eventually:

# github.com/rclone/rclone/cmd/serve/ftp
pkg\mod\github.com\rclone\rclone@v1.52.2\cmd\serve\ftp\ftp.go:154:3: unknown field 'PublicIp' in struct literal of type core.ServerOpts (but does have PublicIP)

ncw · July 26, 2020, 4:12pm

I just tried this on a newly created user

GO111MODULE=on go get github.com/rclone/rclone

That means you updated to a newer version of the ftp package that is in go.mod somehow.

check there are no modifications to rclone's go.mod

Ajaja · July 26, 2020, 4:56pm

A diffrent platform, but the same problem:

$ go get github.com/rclone/rclone
package github.com/putdotio/go-putio/putio: cannot find package "github.com/putdotio/go-putio/putio" in any of:
        /data/data/com.termux/files/usr/lib/go/src/github.com/putdotio/go-putio/putio (from $GOROOT)
        /data/data/com.termux/files/home/go/src/github.com/putdotio/go-putio/putio (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/config: cannot find package "github.com/yunify/qingstor-sdk-go/v3/config" in any of:
        /data/data/com.termux/files/usr/lib/go/src/github.com/yunify/qingstor-sdk-go/v3/config (from $GOROOT)
        /data/data/com.termux/files/home/go/src/github.com/yunify/qingstor-sdk-go/v3/config (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/request/errors: cannot find package "github.com/yunify/qingstor-sdk-go/v3/request/errors" in any of:
        /data/data/com.termux/files/usr/lib/go/src/github.com/yunify/qingstor-sdk-go/v3/request/errors (from $GOROOT)
        /data/data/com.termux/files/home/go/src/github.com/yunify/qingstor-sdk-go/v3/request/errors (from $GOPATH)
package github.com/yunify/qingstor-sdk-go/v3/service: cannot find package "github.com/yunify/qingstor-sdk-go/v3/service" in any of:
        /data/data/com.termux/files/usr/lib/go/src/github.com/yunify/qingstor-sdk-go/v3/service (from $GOROOT)
        /data/data/com.termux/files/home/go/src/github.com/yunify/qingstor-sdk-go/v3/service (from $GOPATH)
unrecognized import path "goftp.io/server/core": reading https://goftp.io/server/core?go-get=1: 404 Not Found

I think it appears if the src folder is empty.
Did you try "go get" (without GO111MODULE ) under the newly created user?

Ajaja · July 26, 2020, 6:20pm

Yes, with a clean GOPATH folder it works.

Ajaja · July 26, 2020, 7:51pm

I found the issue. It was "-u" option.

In conclusion:
go build - OK
go get - fail
go get -u - fail
GO111MODULE=on go get -u - fail
GO111MODULE=on go get - OK

ncw · July 26, 2020, 9:20pm

Ah, yes, that will be it.

The goftp authors made a backwards incompatible change is the problem so the latest version doesn't build with 1.52.2

I've fixed this on master

Ajaja · July 27, 2020, 10:20am

GO111MODULE=on go get -u github.com/rclone/rclone@master - OK.

Thank you!

ncw · July 27, 2020, 10:34am

Except even there you shouldn't use the -u as you'll pull in stuff which is known to be broken (like for example bazil/fuse has dropped macOS support)

Using the -u flag updates the transitive dependencies according to the docs which is a fancy way of saying that it ignores what is in rclone's go.mod and updates all those things to the latest release. This is usually OK but in a big project like rclone there are usually a few dependencies which are broken at the latest release and rclone's go.mod.

I've realised that we have the -u on the rclone.org/install page - this was great advice in the past (pre modules) but no longer - I shall update it!

Ajaja · July 27, 2020, 10:55am

Yes, It was exactly my goal to build a latest rclone version with latest libraries. I realize that it may be unstable.

For a stable build I use now:

GO111MODULE=on go get github.com/rclone/rclone

ncw · July 27, 2020, 11:04am

Once every release cycle I run go get -u ./... to update all of rclone's dependencies. It then takes anything from an hour to several days to get everything working again!

The current release cycle was particularly troublesome and took several days to fix up:

goftp made a backwards incompatible change (PublicIp vs PublicIP)
goftp broke the auth proxy (complicated!)
azure libraries stopped compiling for go1.12 and earlier
ftp server libraries stopped compiling for go1.12 and earlier
xgo builds stopped working for macOS
bazil/fuse stopped supporting macOS

I ended up having to make two pull requests to goftp, loads of fixes to rclone and winding back the bazil/fuse version.

So updating the dependencies is not for the faint hearted!

system · September 26, 2020, 7:04am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.