Is it possible to change the log file option when using the API?

Justin_R · February 9, 2022, 5:01pm

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
os/kernel: 10.0.17763.2510 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.2
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

N/A

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

curl --location --request POST 'https://localhost/options/local' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic cDpw' \
--data-raw '{
    "_config": {
        "Transfers": 64,
        "DryRun": true
    },
    "log": {
        "File": "C:\\rclone\\logs\\rclone.infpwfs1sa-apps.copy.log"
    }
}'

The rclone config contents with secrets removed.

[LocalDisk]
type = local

[aws]
type = s3
provider = AWS
access_key_id = 
secret_access_key = 
region = us-east-1
location_constraint = us-east-1

A log from the command with the `-vv` flag

Here are the results from the API call.

{
	"config": {
		"LogLevel": 6,
		"StatsLogLevel": 6,
		"UseJSONLog": false,
		"DryRun": true,
		"Interactive": false,
		"CheckSum": false,
		"SizeOnly": false,
		"IgnoreTimes": false,
		"IgnoreExisting": false,
		"IgnoreErrors": false,
		"ModifyWindow": 1,
		"Checkers": 8,
		"Transfers": 64,
		"ConnectTimeout": 60000000000,
		"Timeout": 300000000000,
		"ExpectContinueTimeout": 1000000000,
		"Dump": 0,
		"InsecureSkipVerify": false,
		"DeleteMode": 3,
		"MaxDelete": -1,
		"TrackRenames": false,
		"TrackRenamesStrategy": "hash",
		"LowLevelRetries": 10,
		"UpdateOlder": false,
		"NoGzip": false,
		"MaxDepth": -1,
		"IgnoreSize": false,
		"IgnoreChecksum": false,
		"IgnoreCaseSync": false,
		"NoTraverse": false,
		"CheckFirst": false,
		"NoCheckDest": false,
		"NoUnicodeNormalization": false,
		"NoUpdateModTime": false,
		"DataRateUnit": "bytes",
		"CompareDest": null,
		"CopyDest": null,
		"BackupDir": "",
		"Suffix": "",
		"SuffixKeepExtension": false,
		"UseListR": false,
		"BufferSize": 16777216,
		"BwLimit": "",
		"BwLimitFile": "",
		"TPSLimit": 0,
		"TPSLimitBurst": 1,
		"BindAddr": "",
		"DisableFeatures": null,
		"UserAgent": "rclone/v1.57.0",
		"Immutable": false,
		"AutoConfirm": false,
		"StreamingUploadCutoff": 102400,
		"StatsFileNameLength": 45,
		"AskPassword": true,
		"PasswordCommand": null,
		"UseServerModTime": false,
		"MaxTransfer": -1,
		"MaxDuration": 0,
		"CutoffMode": 0,
		"MaxBacklog": 10000,
		"MaxStatsGroups": 1000,
		"StatsOneLine": false,
		"StatsOneLineDate": false,
		"StatsOneLineDateFormat": "",
		"ErrorOnNoTransfer": false,
		"Progress": false,
		"ProgressTerminalTitle": false,
		"Cookie": false,
		"UseMmap": false,
		"CaCert": "",
		"ClientCert": "",
		"ClientKey": "",
		"MultiThreadCutoff": 262144000,
		"MultiThreadStreams": 4,
		"MultiThreadSet": false,
		"OrderBy": "",
		"UploadHeaders": null,
		"DownloadHeaders": null,
		"Headers": null,
		"RefreshTimes": false,
		"NoConsole": false,
		"TrafficClass": 0,
		"FsCacheExpireDuration": 300000000000,
		"FsCacheExpireInterval": 60000000000,
		"DisableHTTP2": false,
		"HumanReadable": false,
		"KvLockTime": 1000000000
	},
	"filter": {
		"DeleteExcluded": false,
		"FilterRule": null,
		"FilterFrom": null,
		"ExcludeRule": null,
		"ExcludeFrom": null,
		"ExcludeFile": "",
		"IncludeRule": null,
		"IncludeFrom": null,
		"FilesFrom": null,
		"FilesFromRaw": null,
		"MinAge": 9223372036854775807,
		"MaxAge": 9223372036854775807,
		"MinSize": -1,
		"MaxSize": -1,
		"IgnoreCase": false
	}
}

asdffdsa · February 9, 2022, 5:08pm

hello and welcome to the forum,

you want to set the location for the log file?

did you try logfile?

Justin_R · February 9, 2022, 5:19pm

Hello,

Thanks for your help. I did try logfile, but the result was the same. I believe that only the options returned from the API call can be set via the API call, but I'm hoping I'm mistaken.

Command

curl --location --request POST 'https://rclone.hsag.com/options/local' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic cDpw' \
--data-raw '{
    "_config": {
        "Transfers": 64,
        "DryRun": true,
        "logfile": "C:\\rclone\\logs\\rclone.infpwfs1sa-apps.copy.log"
    }
}'

Result

{
	"config": {
		"LogLevel": 6,
		"StatsLogLevel": 6,
		"UseJSONLog": false,
		"DryRun": true,
		"Interactive": false,
		"CheckSum": false,
		"SizeOnly": false,
		"IgnoreTimes": false,
		"IgnoreExisting": false,
		"IgnoreErrors": false,
		"ModifyWindow": 1,
		"Checkers": 8,
		"Transfers": 64,
		"ConnectTimeout": 60000000000,
		"Timeout": 300000000000,
		"ExpectContinueTimeout": 1000000000,
		"Dump": 0,
		"InsecureSkipVerify": false,
		"DeleteMode": 3,
		"MaxDelete": -1,
		"TrackRenames": false,
		"TrackRenamesStrategy": "hash",
		"LowLevelRetries": 10,
		"UpdateOlder": false,
		"NoGzip": false,
		"MaxDepth": -1,
		"IgnoreSize": false,
		"IgnoreChecksum": false,
		"IgnoreCaseSync": false,
		"NoTraverse": false,
		"CheckFirst": false,
		"NoCheckDest": false,
		"NoUnicodeNormalization": false,
		"NoUpdateModTime": false,
		"DataRateUnit": "bytes",
		"CompareDest": null,
		"CopyDest": null,
		"BackupDir": "",
		"Suffix": "",
		"SuffixKeepExtension": false,
		"UseListR": false,
		"BufferSize": 16777216,
		"BwLimit": "",
		"BwLimitFile": "",
		"TPSLimit": 0,
		"TPSLimitBurst": 1,
		"BindAddr": "",
		"DisableFeatures": null,
		"UserAgent": "rclone/v1.57.0",
		"Immutable": false,
		"AutoConfirm": false,
		"StreamingUploadCutoff": 102400,
		"StatsFileNameLength": 45,
		"AskPassword": true,
		"PasswordCommand": null,
		"UseServerModTime": false,
		"MaxTransfer": -1,
		"MaxDuration": 0,
		"CutoffMode": 0,
		"MaxBacklog": 10000,
		"MaxStatsGroups": 1000,
		"StatsOneLine": false,
		"StatsOneLineDate": false,
		"StatsOneLineDateFormat": "",
		"ErrorOnNoTransfer": false,
		"Progress": false,
		"ProgressTerminalTitle": false,
		"Cookie": false,
		"UseMmap": false,
		"CaCert": "",
		"ClientCert": "",
		"ClientKey": "",
		"MultiThreadCutoff": 262144000,
		"MultiThreadStreams": 4,
		"MultiThreadSet": false,
		"OrderBy": "",
		"UploadHeaders": null,
		"DownloadHeaders": null,
		"Headers": null,
		"RefreshTimes": false,
		"NoConsole": false,
		"TrafficClass": 0,
		"FsCacheExpireDuration": 300000000000,
		"FsCacheExpireInterval": 60000000000,
		"DisableHTTP2": false,
		"HumanReadable": false,
		"KvLockTime": 1000000000
	},
	"filter": {
		"DeleteExcluded": false,
		"FilterRule": null,
		"FilterFrom": null,
		"ExcludeRule": null,
		"ExcludeFrom": null,
		"ExcludeFile": "",
		"IncludeRule": null,
		"IncludeFrom": null,
		"FilesFrom": null,
		"FilesFromRaw": null,
		"MinAge": 9223372036854775807,
		"MaxAge": 9223372036854775807,
		"MinSize": -1,
		"MaxSize": -1,
		"IgnoreCase": false
	}
}

asdffdsa · February 9, 2022, 5:26pm

ok, did not know you already tried that, it was worth a shot.

"The response will be a JSON blob in the body of the response.
This is formatted to be reasonably human readable."

so i would try a simple command, such as the equivalent of rclone version and check the json output

and if that does not work, no worries, someone more knowledge about the api should stop by soon.

Animosity022 · February 9, 2022, 5:35pm

I don't think you can change the log file as there is an open item for log rotation that doesn't exist.

Implement logrotate solution · Issue #2259 · rclone/rclone (github.com)

Couple fixes in there as well.

ncw · February 9, 2022, 5:41pm

I think this should work, but it currently isn't implemented

Do you want to fix it? I'll give you a walk through if you do!

Justin_R · February 9, 2022, 7:44pm

100% yes, I would love to help. Point me in the right direction, and I'll get started (I'll try not to ask for too much handholding).

ncw · February 10, 2022, 5:02pm

What needs to happen is that this line

github.com

rclone/rclone/blob/9cc50a614b53e89d398d37f48db1d8fa2ec01560/fs/log/logflags/logflags.go#L13

      
        
            
            
import (
            	"github.com/rclone/rclone/fs/config/flags"
            	"github.com/rclone/rclone/fs/log"
            	"github.com/rclone/rclone/fs/rc"
            	"github.com/spf13/pflag"
            )
            
            
// AddFlags adds the log flags to the flagSet
            func AddFlags(flagSet *pflag.FlagSet) {
            	rc.AddOption("log", &log.Opt)
            
            
	flags.StringVarP(flagSet, &log.Opt.File, "log-file", "", log.Opt.File, "Log everything to this file")
            	flags.StringVarP(flagSet, &log.Opt.Format, "log-format", "", log.Opt.Format, "Comma separated list of log format options")
            	flags.BoolVarP(flagSet, &log.Opt.UseSyslog, "syslog", "", log.Opt.UseSyslog, "Use Syslog for logging")
            	flags.StringVarP(flagSet, &log.Opt.SyslogFacility, "syslog-facility", "", log.Opt.SyslogFacility, "Facility for syslog, e.g. KERN,USER,...")
            	flags.BoolVarP(flagSet, &log.Opt.LogSystemdSupport, "log-systemd", "", log.Opt.LogSystemdSupport, "Activate systemd integration for the logger")
            }

Needs to change to use rc.AddOptionReload which has a callback which is called when the option block is changed. There is an example of that here

github.com

rclone/rclone/blob/master/fs/filter/filterflags/filterflags.go

// Package filterflags implements command line flags to set up a filter
package filterflags

import (
	"context"

	"github.com/rclone/rclone/fs/config/flags"
	"github.com/rclone/rclone/fs/filter"
	"github.com/rclone/rclone/fs/rc"
	"github.com/spf13/pflag"
)

// Options set by command line flags
var (
	Opt = filter.DefaultOpt
)

// Reload the filters from the flags
func Reload(ctx context.Context) (err error) {
	fi := filter.GetConfig(ctx)

This file has been truncated. show original

The Reload function needs to call InitLogging to restart the logging.

github.com

rclone/rclone/blob/9cc50a614b53e89d398d37f48db1d8fa2ec01560/fs/log/log.go#L93

      
        
            		return
            	}
            	arr := [16 * 1024]byte{}
            	buf := arr[:]
            	n := runtime.Stack(buf, false)
            	buf = buf[:n]
            	fs.LogPrintf(fs.LogLevelDebug, o, "%s\nStack trace:\n%s", info, buf)
            }
            
            
// InitLogging start the logging as per the command line flags
            func InitLogging() {
            	flagsStr := "," + Opt.Format + ","
            	var flags int
            	if strings.Contains(flagsStr, ",date,") {
            		flags |= log.Ldate
            	}
            	if strings.Contains(flagsStr, ",time,") {
            		flags |= log.Ltime
            	}
            	if strings.Contains(flagsStr, ",microseconds,") {
            		flags |= log.Lmicroseconds

That's glossing over a few details like closing the old log file, but it should point you in the right direction!

That should also make things like changing the log format work etc.

system · March 12, 2022, 5:02pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Is it possible to change the log file option when using the API?

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

A log from the command with the -vv flag

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag