Rclone api /operations/check does not respect global size-only override

Gald · April 25, 2024, 1:40pm

What is the problem you are having with rclone?

I am trying to validate two folders, one local and one on my s3 bucket. I'm trying to automate this using the operations/check command.
Because these folders are upwards of 100 gigs, I think a size comparison would be fine.

I am trying to supply a json dict setting this, but the option does not seem to get respected because it still takes very long, even though the command is almost instant on the command line.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-105-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.21.6
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

s3

I send this via python, this is what it sends:

{'method': 'POST', 'url': 'url/operations/check', 'headers': {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-type': 'application/json', 'Content-Length': '562', 'Authorization': 'Basic Token'}, '_cookies': <RequestsCookieJar[]>, 'body': '{"srcFs": "/path-to-big-folder", "dstFs": "bucket:/path-to-big-folder", "oneWay": true, "size-only": true, "SizeOnly": true, "_main": {"SizeOnly": true, "size-only": true}, "main": {"SizeOnly": true, "size-only": true}, "config": {"SizeOnly": true, "size-only": true}, "_config": {"SizeOnly": true, "size-only": true}}', 'hooks': {'response': []}, '_body_position': None}

The rclone config contents with secrets removed.

{
	"dlna": {
		"ListenAddr": ":7879",
		"FriendlyName": "",
		"LogTrace": false,
		"InterfaceNames": [],
		"AnnounceInterval": 720000000000
	},
	"filter": {
		"DeleteExcluded": false,
		"FilterRule": null,
		"FilterFrom": null,
		"ExcludeRule": null,
		"ExcludeFrom": null,
		"IncludeRule": null,
		"IncludeFrom": null,
		"ExcludeFile": null,
		"FilesFrom": null,
		"FilesFromRaw": null,
		"MetaRules": {
			"FilterRule": null,
			"FilterFrom": null,
			"ExcludeRule": null,
			"ExcludeFrom": null,
			"IncludeRule": null,
			"IncludeFrom": null
		},
		"MinAge": 9223372036854775807,
		"MaxAge": 9223372036854775807,
		"MinSize": -1,
		"MaxSize": -1,
		"IgnoreCase": false
	},
	"ftp": {
		"ListenAddr": "localhost:2121",
		"PublicIP": "",
		"PassivePorts": "30000-32000",
		"BasicUser": "anonymous",
		"BasicPass": "",
		"TLSCert": "",
		"TLSKey": ""
	},
	"log": {
		"File": "/var/log/rclone/rclone.log",
		"Format": "date,time",
		"UseSyslog": false,
		"SyslogFacility": "DAEMON",
		"LogSystemdSupport": false
	},
	"main": {
		"LogLevel": "NOTICE",
		"StatsLogLevel": "INFO",
		"UseJSONLog": false,
		"DryRun": false,
		"Interactive": false,
		"CheckSum": false,
		"SizeOnly": false,
		"IgnoreTimes": false,
		"IgnoreExisting": false,
		"IgnoreErrors": false,
		"ModifyWindow": 1,
		"Checkers": 8,
		"Transfers": 4,
		"ConnectTimeout": 60000000000,
		"Timeout": 300000000000,
		"ExpectContinueTimeout": 1000000000,
		"Dump": "",
		"InsecureSkipVerify": false,
		"DeleteMode": 3,
		"MaxDelete": -1,
		"MaxDeleteSize": -1,
		"TrackRenames": false,
		"TrackRenamesStrategy": "hash",
		"LowLevelRetries": 10,
		"UpdateOlder": false,
		"NoGzip": false,
		"MaxDepth": -1,
		"IgnoreSize": false,
		"IgnoreChecksum": false,
		"IgnoreCaseSync": false,
		"NoTraverse": false,
		"CheckFirst": false,
		"NoCheckDest": false,
		"NoUnicodeNormalization": false,
		"NoUpdateModTime": false,
		"DataRateUnit": "bytes",
		"CompareDest": null,
		"CopyDest": null,
		"BackupDir": "",
		"Suffix": "",
		"SuffixKeepExtension": false,
		"UseListR": false,
		"BufferSize": 16777216,
		"BwLimit": "",
		"BwLimitFile": "",
		"TPSLimit": 0,
		"TPSLimitBurst": 1,
		"BindAddr": "",
		"DisableFeatures": null,
		"UserAgent": "rclone/v1.65.2",
		"Immutable": false,
		"AutoConfirm": false,
		"StreamingUploadCutoff": 102400,
		"StatsFileNameLength": 45,
		"AskPassword": true,
		"PasswordCommand": null,
		"UseServerModTime": false,
		"MaxTransfer": -1,
		"MaxDuration": 0,
		"CutoffMode": "HARD",
		"MaxBacklog": 10000,
		"MaxStatsGroups": 1000,
		"StatsOneLine": false,
		"StatsOneLineDate": false,
		"StatsOneLineDateFormat": "",
		"ErrorOnNoTransfer": false,
		"Progress": false,
		"ProgressTerminalTitle": false,
		"Cookie": false,
		"UseMmap": false,
		"CaCert": null,
		"ClientCert": "",
		"ClientKey": "",
		"MultiThreadCutoff": 268435456,
		"MultiThreadStreams": 4,
		"MultiThreadSet": false,
		"MultiThreadChunkSize": 67108864,
		"MultiThreadWriteBufferSize": 131072,
		"OrderBy": "",
		"UploadHeaders": null,
		"DownloadHeaders": null,
		"Headers": null,
		"MetadataSet": null,
		"RefreshTimes": false,
		"NoConsole": false,
		"TrafficClass": 0,
		"FsCacheExpireDuration": 300000000000,
		"FsCacheExpireInterval": 60000000000,
		"DisableHTTP2": false,
		"HumanReadable": false,
		"KvLockTime": 1000000000,
		"DisableHTTPKeepAlives": false,
		"Metadata": false,
		"ServerSideAcrossConfigs": false,
		"TerminalColorMode": "AUTO",
		"DefaultTime": "2000-01-01T00:00:00Z",
		"Inplace": false,
		"PartialSuffix": ".partial",
		"MetadataMapper": null
	},
	"mount": {
		"DebugFUSE": false,
		"AllowNonEmpty": false,
		"AllowRoot": false,
		"AllowOther": false,
		"DefaultPermissions": false,
		"WritebackCache": false,
		"Daemon": false,
		"DaemonWait": 60000000000,
		"MaxReadAhead": 131072,
		"ExtraOptions": [],
		"ExtraFlags": [],
		"AttrTimeout": 1000000000,
		"DeviceName": "",
		"VolumeName": "",
		"NoAppleDouble": true,
		"NoAppleXattr": false,
		"DaemonTimeout": 0,
		"AsyncRead": true,
		"NetworkMode": false,
		"CaseInsensitive": null
	},
	"nfs": {
		"ListenAddr": ""
	},
	"rc": {
		"HTTP": {
			"ListenAddr": [
				"0.0.0.0:5572"
			],
			"BaseURL": "",
			"ServerReadTimeout": 3600000000000,
			"ServerWriteTimeout": 3600000000000,
			"MaxHeaderBytes": 4096,
			"TLSCert": "",
			"TLSKey": "",
			"TLSCertBody": null,
			"TLSKeyBody": null,
			"ClientCA": "",
			"MinTLSVersion": "tls1.0",
			"AllowOrigin": ""
		},
		"Auth": {
			"HtPasswd": "",
			"Realm": "",
		},
		"Template": {
			"Path": ""
		},
		"Enabled": true,
		"Serve": true,
		"Files": "",
		"NoAuth": false,
		"WebUI": true,
		"WebGUIUpdate": false,
		"WebGUIForceUpdate": false,
		"WebGUINoOpenBrowser": false,
		"WebGUIFetchURL": "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest",
		"EnableMetrics": true,
		"JobExpireDuration": 300000000000,
		"JobExpireInterval": 10000000000
	},
	"sftp": {
		"ListenAddr": "localhost:2022",
		"HostKeys": null,
		"AuthorizedKeys": "~/.ssh/authorized_keys",
		"User": "",
		"Pass": "",
		"NoAuth": false,
		"Stdio": false
	},
	"vfs": {
		"NoSeek": false,
		"NoChecksum": false,
		"ReadOnly": false,
		"NoModTime": false,
		"DirCacheTime": 300000000000,
		"Refresh": false,
		"PollInterval": 60000000000,
		"Umask": 18,
		"UID": 0,
		"GID": 0,
		"DirPerms": 511,
		"FilePerms": 438,
		"ChunkSize": 134217728,
		"ChunkSizeLimit": -1,
		"CacheMode": "off",
		"CacheMaxAge": 3600000000000,
		"CacheMaxSize": -1,
		"CacheMinFreeSpace": -1,
		"CachePollInterval": 60000000000,
		"CaseInsensitive": false,
		"WriteWait": 1000000000,
		"ReadWait": 20000000,
		"WriteBack": 5000000000,
		"ReadAhead": 0,
		"UsedIsSize": false,
		"FastFingerprint": false,
		"DiskSpaceTotalSize": -1
	}
}

A log from the command with the `-vv` flag

Not possible afaik

asdffdsa · April 25, 2024, 1:48pm

welcome to the forum,

have you tested rclone check without using rc?

the correct syntax is remote:bucket/folder

that is an old version, so might rclone selfupdate and test again.

should use a debug log, the answer might be there

maybe i missed it, but did not see a s3 remote

Gald · April 25, 2024, 2:07pm

Hi,

1: Yes, that works just fine
2: Yes, my bad I redacted the actual bucket names, but in the wrong format it s eems
3: How can I get debug logs for api requests?
4:
[rclone-captured-development]
type = s3

ncw · April 27, 2024, 7:46am

I think you are setting the filter and config parameters wrong.

Check out this section in the docs about the _config parameter and the one below it about _filter

Gald · April 29, 2024, 7:58am

Hi, yea I already saw that configuration and I think I am following it correctly.

I now tried to do it via rclone rc so that hopefully it's a bit easier to read

rclone rc --rc-user="..." --rc-pass="..." operations/check srcFs="/local-path" dstFs="rclone-confiuguration:/bucket/path _config='{"DryRun":true, "SizeOnly":true, "dry-run":true, "size-only":true}' -vv

This ignores both the dry run and size only arguments, I am seeing those two options under the _main section in options/get. But no dice. The logs just print that they are starting the command:

2024/04/29 07:54:17 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "rc" "--rc-user=..." "--rc-pass=..." "operations/check" "srcFs=..." "dstFs=..." "_config={\"DryRun\":true, \"SizeOnly\":true, \"dry-run\":true, \"size-only\":true}" "-vv"]
{

but then they hang.. until it's done after which it prints this:

{
        "differ": [],
        "error": [],
        "hashType": "md5",
        "missingOnDst": [],
        "missingOnSrc": [],
        "status": "OK",
        "success": true
}