FTP file transfer options & Logging level from Go

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

The 1st part is, I am trying to transfer some files from an ftp server to an s3 bucket. The files are gziped (eg file001.xml.gz). After copying the directory, I find that many of the files are corrupted and the md5 checksum is not the same as the one provided by the server.

I believe this might have something to do with the transfer mode. The server mentions it should happen in binary mode. The package used by rclone to implement the ftp backend has an option for binary mode, but I can't seem to find something relevant in the source code or the documentation of Rclone itself.

The 2nd part, I am trying to change the logging level from NOTICE to INFO. As already mentioned, I am trying to do this from inside Go. Since I'm not super familiar with the language, I'm finding it difficult to do. Is there any way to directly access and edit --log-level and --log-file from inside Go?

Run the command 'rclone version' and share the full output of the command.

I am using Rclone as part of a Go package I'm writing, so I'm getting it from github instead. According to go.mod, I am using github.com/rclone/rclone v1.62.2

Which cloud storage system are you using? (eg Google Drive)


The command you were trying to run (eg rclone copy /tmp remote:tmp)

        ctx := context.Background()


	var filterOpts = filter.DefaultOpt
	filterOpts.RulesOpt = filter.RulesOpt{
		IncludeRule: []string{filterString},

	firmFilter, err := filter.NewFilter(&filterOpts)
	if err != nil {
		return nil, err
	filteredCtx := filter.ReplaceConfig(ctx, firmFilter)

	fsource, err := fs.NewFs(filteredCtx, sourceRemote)
	if err != nil {
		return nil, fmt.Errorf("could not create source remote '%s': %w", sourceRemote, err)

	fdest, err := fs.NewFs(filteredCtx, targetRemote)
	if err != nil {
		return nil, fmt.Errorf("could not create target remote '%s': %w", targetRemote, err)

	entries, err := walkSourceRemote(filteredCtx, fsource) // result entries are not filtered
	if err != nil {
		return nil, fmt.Errorf("could not list source remote '%s': %w", sourceRemote, err)

	log.Printf("Found %d files to copy.\n", len(entries))

	err = sync.CopyDir(filteredCtx, fdest, fsource, true)
	if err != nil {
		return nil, fmt.Errorf("could not copy source remote '%s' to target remote '%s': %w", sourceRemote, targetRemote, err)
	log.Printf("Done copying files.\n")

	return entries, nil

#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

Paste config here

#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

Please provide all required info as per template. As saying only "I am trying" does not contain any info useful for troubleshooting:)

I cannot use the template because I am not using any CLI commands. I pasted the relevant code I'm using so I don't see what extra information I can add to the post.

I will show the output of rclone version, but I don't see how the knowledge of my OS would help anybody with the source code of Rclone.

rclone v1.62.2
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: dynamic
- go/tags: none

In addition, I am not asking for troubleshooting, just for knowledge/opinions on how easy it is, if at all possible, to set transfer mode to binary.

OK. All clear:)

I'd recommend testing with the rclone command line. You can use -vv --dump bodies and you'll see what rclone is sending to the ftp server.

You should see this

2023/06/07 09:54:04 DEBUG : FTP Tx: "TYPE I"
2023/06/07 09:54:04 DEBUG : FTP Rx: "200 Type set to I"

Which is how you set binary mode (Image mode) on FTP servers. Rclone does this automatically all the time so I doubt this is the problem.

The recommended way to use rclone is to use it via librclone this effectively runs an in-process API server so you can use all the commands in the api docs and in there is an explanation of how to change the logging level. Using librclone gives you much looser coupling and an inteface which won't break - we shuffle the internals of rclone quite regularly!

If you still want to do this within your code something like this will change the log level.

        ci := fs.GetConfig(context.Background())
        ci.LogLevel = fs.LogLevelInfo

However I'd try to replicate your problem using the command line and open a new topic with a full log and hopefully we'll be able to help you.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.