Chtimes Error On Local Mounted Copy

RCloneUser1 · July 9, 2020, 4:57pm

What is the problem you are having with rclone?

When copying to a cifs mounted directory, RClone Remote Control will successfully copy the file but return a 500 error with "error": "chtimes /mnt2/transfer/folder/1.txt: operation not permitted"

The permission on the directory is 775 with the rclone user being part of the group but not the owner

The error only occurs on first time copy. Using "--no-update-modtime" ensures that the error does not occur if the file is copied again (with the same data)

Hoping for an rclone solution rather than changing permissions on the mounted directory

What is your rclone version (output from `rclone version`)

1.51

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone rc operations/copyfile srcFs="/usr/local/redacted" srcRemote="1.txt" dstFs="test:/mnt2/transfer/folder" dstRemote="1.txt"

The rclone config contents with secrets removed.

[test]
type = local
set_modtime = false

A log from the command with the `-vv` flag

$ rclone rcd --rc-no-auth --no-update-modtime --config=/usr/local/redacted/configs/redacted/rclone.config -vv
2020/07/09 09:49:01 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "rcd" "--rc-no-auth" "--no-update-modtime" "--config=/usr/local/redacted/configs/redacted/rclone.config" "-vv"]
2020/07/09 09:49:01 NOTICE: Serving remote control on http://127.0.0.1:5572/
2020/07/09 09:49:16 DEBUG : rc: "operations/copyfile": with parameters map[dstFs:test:/mnt2/transfer/folder dstRemote:1.txt srcFs:/usr/local/redacted srcRemote:1.txt]
2020/07/09 09:49:16 DEBUG : Using config file from "/usr/local/redacted/configs/redacted/rclone.config"
2020/07/09 09:49:16 DEBUG : 1.txt: Need to transfer - File not found at Destination
2020/07/09 09:49:16 ERROR : 1.txt: Failed to copy: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 09:49:16 ERROR : rc: "operations/copyfile": error: chtimes /mnt2/transfer/folder/1.txt: operation not permitted

Animosity022 · July 9, 2020, 5:02pm

Those are normally permissions errors.

Can you share the ls -al on the file and id of the user running it?

RCloneUser1 · July 9, 2020, 5:13pm

[username@xxx-4 folder]$ ls -l /mnt2/transfer/folder/
total 116736
-rwxrwxr-x. 1 adminuser usersgroup        3 Jul  9 09:49 1.txt

[username@xxx-4 folder]$ groups rcloneuser
rcloneuser: xxxx usersgroup xxxx

Find the requested information above.The names have been changed but the data is correct.

Animosity022 · July 9, 2020, 5:17pm

Can you replicate the error just doing a normal file copy?

That's a permissions error on the CIFS side.

RCloneUser1 · July 9, 2020, 5:25pm

Unable to reproduce the error to a non-mounted local copy.

Changing the owner/permissions of the mount is a "last resort"solution (but not out of the question). Was mostly wondering if there was a way to have RClone not update the modification time at all (similiar to set_modtime = false which seems to be aimed directly at sftp)

Animosity022 · July 9, 2020, 5:29pm

Sorry as I wasn't clear.

Can you do a rclone move -vv from the command prompt without the rcd ?

RCloneUser1 · July 9, 2020, 10:15pm

Here is the result with the file already existing in the destination:

[rcloneuser@xxx-4 ~]$ rclone move /usr/local/redacted/1.txt /mnt2/transfer/folder/1.txt -vv
2020/07/09 15:08:51 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "/usr/local/redacted/1.txt" "/mnt2/transfer/folder/1.txt" "-vv"]
2020/07/09 15:08:51 DEBUG : Using config file from "/home/rcloneuser/.config/rclone/rclone.conf"
2020/07/09 15:08:51 Failed to create file system for "/mnt2/transfer/folder/1.txt": is a file not a directory
[rcloneuser@xxx-4 ~]$ rclone move /usr/local/redacted/1.txt /mnt2/transfer/folder/ -vv
2020/07/09 15:08:59 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "/usr/local/redacted/1.txt" "/mnt2/transfer/folder/" "-vv"]
2020/07/09 15:08:59 DEBUG : Using config file from "/home/rcloneuser/.config/rclone/rclone.conf"
2020/07/09 15:08:59 DEBUG : 1.txt: Modification times differ by 1177h26m56.971729223s: 2020-05-21 13:24:05.720178777 -0700 PDT, 2020-07-09 14:51:02.691908 -0700 PDT
2020/07/09 15:08:59 DEBUG : 1.txt: MD5 = e2665feebe35bc97aff1b329c87b87e7 OK
2020/07/09 15:08:59 ERROR : 1.txt: Failed to set modification time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:08:59 DEBUG : 1.txt: Unchanged skipping
2020/07/09 15:08:59 ERROR : 1.txt: Couldn't delete: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 ERROR : Attempt 1/3 failed with 2 errors and: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 DEBUG : 1.txt: Modification times differ by 1177h26m56.971729223s: 2020-05-21 13:24:05.720178777 -0700 PDT, 2020-07-09 14:51:02.691908 -0700 PDT
2020/07/09 15:08:59 DEBUG : 1.txt: MD5 = e2665feebe35bc97aff1b329c87b87e7 OK
2020/07/09 15:08:59 ERROR : 1.txt: Failed to set modification time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:08:59 DEBUG : 1.txt: Unchanged skipping
2020/07/09 15:08:59 ERROR : 1.txt: Couldn't delete: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 ERROR : Attempt 2/3 failed with 2 errors and: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 DEBUG : 1.txt: Modification times differ by 1177h26m56.971729223s: 2020-05-21 13:24:05.720178777 -0700 PDT, 2020-07-09 14:51:02.691908 -0700 PDT
2020/07/09 15:08:59 DEBUG : 1.txt: MD5 = e2665feebe35bc97aff1b329c87b87e7 OK
2020/07/09 15:08:59 ERROR : 1.txt: Failed to set modification time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:08:59 DEBUG : 1.txt: Unchanged skipping
2020/07/09 15:08:59 ERROR : 1.txt: Couldn't delete: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 ERROR : Attempt 3/3 failed with 2 errors and: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:08:59 Failed to move with 2 errors: last error was: remove /usr/local/redacted/1.txt: permission denied

Here is the result with the file not already existing in the destination. The file copied correctly but RClone still returned an error.

[rcloneuser@xxx-4 ~]$ rclone move /usr/local/redacted/1.txt /mnt2/transfer/folder/ -vv
2020/07/09 15:10:07 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "/usr/local/redacted/1.txt" "/mnt2/transfer/folder/" "-vv"]
2020/07/09 15:10:07 DEBUG : Using config file from "/home/rcloneuser/.config/rclone/rclone.conf"
2020/07/09 15:10:07 DEBUG : 1.txt: Need to transfer - File not found at Destination
2020/07/09 15:10:07 DEBUG : 1.txt: Can't move: rename /usr/local/redacted/1.txt /mnt2/transfer/folder/1.txt: invalid cross-device link: trying copy
2020/07/09 15:10:07 DEBUG : 1.txt: Can't move, switching to copy
2020/07/09 15:10:07 ERROR : 1.txt: Failed to copy: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:10:07 ERROR : 1.txt: Not deleting source as copy failed: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:10:07 ERROR : Attempt 1/3 failed with 1 errors and: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:10:07 DEBUG : 1.txt: Modification times differ by 1177h46m1.822324746s: 2020-05-21 13:24:05.720178777 -0700 PDT, 2020-07-09 15:10:07.542503523 -0700 PDT
2020/07/09 15:10:07 DEBUG : 1.txt: MD5 = e2665feebe35bc97aff1b329c87b87e7 OK
2020/07/09 15:10:07 ERROR : 1.txt: Failed to set modification time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:10:07 DEBUG : 1.txt: Unchanged skipping
2020/07/09 15:10:07 ERROR : 1.txt: Couldn't delete: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:10:07 ERROR : Attempt 2/3 failed with 2 errors and: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:10:07 DEBUG : 1.txt: Modification times differ by 1177h46m1.818593723s: 2020-05-21 13:24:05.720178777 -0700 PDT, 2020-07-09 15:10:07.5387725 -0700 PDT
2020/07/09 15:10:07 DEBUG : 1.txt: MD5 = e2665feebe35bc97aff1b329c87b87e7 OK
2020/07/09 15:10:07 ERROR : 1.txt: Failed to set modification time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/09 15:10:07 DEBUG : 1.txt: Unchanged skipping
2020/07/09 15:10:07 ERROR : 1.txt: Couldn't delete: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:10:07 ERROR : Attempt 3/3 failed with 2 errors and: remove /usr/local/redacted/1.txt: permission denied
2020/07/09 15:10:07 Failed to move with 2 errors: last error was: remove /usr/local/redacted/1.txt: permission denied

This functionality works correctly on a NFS mount (currently using CIFS). As mentioned before, this is obviously a permission-based issue but checking if there is a way to turn off modifications for RClone

Animosity022 · July 9, 2020, 10:27pm

From my understanding:

That's for the SFTP backend so it wouldn't do anything for local.

No update modtime is when you make a copy and two files are the same and the mod times are different and it won't update it.

I don't believe what you are asking exists but in your use case, I probably wouldn't even use rclone as I'd probably use rsync. I only use rclone when I have cloud needs.

ncw · July 10, 2020, 2:56pm

I'm understanding the problem to be that rclone can't set modtimes on your destination which is a cifs mounted directory.

There isn't a way of disabling the setting of modtimes on a write to the local backend at the moment.

To double check this is the problem try rclone touch to set a mod time, and compare that with what the unix command touch does both running as the rclone user which is part of the group but not the owner.

You need to have write access to the file to set the modification time

$ echo hello > file.txt
$ sudo chown root.root file.txt
$ ls -l --full-time
total 4
-rw-rw-r-- 1 root root 6 2020-07-10 15:51:31.721277902 +0100 file.txt
$ touch file.txt 
touch: cannot touch 'file.txt': Permission denied
$ sudo chgrp ncw file.txt
$ touch file.txt 
$ sudo chmod 644 file.txt 
$ ls -l --full-time
total 4
-rw-r--r-- 1 root ncw 6 2020-07-10 15:53:02.853866988 +0100 file.txt
$ touch file.txt 
touch: cannot touch 'file.txt': Permission denied
$ sudo chmod 664 file.txt 
$ ls -l --full-time
total 4
-rw-rw-r-- 1 root ncw 6 2020-07-10 15:53:02.853866988 +0100 file.txt
$ touch file.txt 
$

I can't think of a way of writing a file which then has the permissions meaning you can't touch it afterwards. So I suspect there is a setting in your CIFS which needs tweaking.

What permissions do the files have when rclone has finished writing them?

RCloneUser1 · July 10, 2020, 3:57pm

Here is the result of the rclone touch command:

[rcloneuser@xxx-4 ~]$ rclone touch /mnt2/transfer/folder/1.txt
2020/07/10 08:53:19 ERROR : Attempt 1/3 failed with 1 errors and: touch: couldn't set mod time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/10 08:53:19 ERROR : Attempt 2/3 failed with 1 errors and: touch: couldn't set mod time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/10 08:53:19 ERROR : Attempt 3/3 failed with 1 errors and: touch: couldn't set mod time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted
2020/07/10 08:53:19 Failed to touch: touch: couldn't set mod time: chtimes /mnt2/transfer/folder/1.txt: operation not permitted

After the file is copied (as it was above), the file permissions are: owner = adminuser, group = usersgroup, permissions = 0775. This matches the values on the mount setup:

sudo mount -t cifs //xxxx-xxxx/users/transfer -o username=redacted,password=redacted /mnt2/transfer -o file_mode=0775,dir_mode=0775,uid=1001,gid=1002

ncw · July 10, 2020, 10:34pm

Can you try the Unix touch command too?

RCloneUser1 · July 10, 2020, 10:59pm

The Unix touch command works as expected.

touch -m /mnt2/transfer/folder/1.txt
returns touch: setting times of ‘/mnt2/transfer/tslaughter/1.txt’: Operation not permitted

My assumption of what is happening:

RClone copies the file to the CIFs mount.
The CIFs mount updates the permissions of the file to admingroup/usersgroup
By the time RClone attempts to update the modification time, the RClone user no longer owns the file which causes the error.

Based on the man page for utime (which essentially is what Go calls from what I can tell), the user must be the owner of the file to update the modification time. (Tried to add the link to man page but is not allowed)

It does not seem like there will be a RClone related solution unless a config is added in the future to not update the modification time.

ncw · July 11, 2020, 7:30am

Ok that makes sense.

Adding a flag to rclone would be quite easy. It would be a local backend flag.

Do you want to have a go at adding this? Happy to talk you through it .

RCloneUser1 · July 21, 2020, 4:49pm

I might be able to take a crack at adding the new flag.

I am trying to use RClone (and this potential new feature) for a project with an upcoming deadline. If a pull request is submitted this week, when would the update be released (or when is the next scheduled release)?

ncw · July 21, 2020, 9:06pm

Great!

PR's go into the the latest beta first and then on into the next release. I think 1.53 would be 2-4 weeks away from now.

RCloneUser1 · July 21, 2020, 9:37pm

Are there any tips for getting started on this? I see there is a page on GitHub which I can follow: https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md . Are there any tips on where the new flag should live or things like that?

ncw · July 22, 2020, 8:44am

This needs to be a local backend flag, so you'll need to add another flag like this one

github.com

rclone/rclone/blob/d2efb4b29b3b0be4daf700ffd803267f07f635ea/backend/local/local.go#L137-L147


		}, {
			Name: "no_sparse",
			Help: `Disable sparse files for multi-thread downloads

On Windows platforms rclone will make sparse files when doing
multi-thread downloads. This avoids long pauses on large files where
the OS zeros the file. However sparse files may be undesirable as they
cause disk fragmentation and can be slow to work with.`,
			Default:  false,
			Advanced: true,
		}, {

And then add it to the Options structure

github.com

rclone/rclone/blob/d2efb4b29b3b0be4daf700ffd803267f07f635ea/backend/local/local.go#L168


type Options struct {
	FollowSymlinks    bool                 `config:"copy_links"`
	TranslateSymlinks bool                 `config:"links"`
	SkipSymlinks      bool                 `config:"skip_links"`
	NoUTFNorm         bool                 `config:"no_unicode_normalization"`
	NoCheckUpdated    bool                 `config:"no_check_updated"`
	NoUNC             bool                 `config:"nounc"`
	OneFileSystem     bool                 `config:"one_file_system"`
	CaseSensitive     bool                 `config:"case_sensitive"`
	CaseInsensitive   bool                 `config:"case_insensitive"`
	NoSparse          bool                 `config:"no_sparse"`
	Enc               encoder.MultiEncoder `config:"encoding"`
}

// Fs represents a local filesystem rooted at root
type Fs struct {
	name        string              // the name of the remote
	root        string              // The root directory (OS path)
	opt         Options             // parsed config options
	features    *fs.Features        // optional features
	dev         uint64              // device number of root node

And then check for it in SetModTime and don't set the modtime

github.com

rclone/rclone/blob/d2efb4b29b3b0be4daf700ffd803267f07f635ea/backend/local/local.go#L879-L892


// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
	var err error
	if o.translatedLink {
		err = lChtimes(o.path, modTime, modTime)
	} else {
		err = os.Chtimes(o.path, modTime, modTime)
	}
	if err != nil {
		return err
	}
	// Re-read metadata
	return o.lstat()
}

I suggest we call the option no_set_modtime which will make a flag --local-no-set-modtime. This will potentially cause a problem with syncing if rclone can't set modtimes so this flag may need to alter the Precision to return fs.ModTimeNotSupported to signal to the rest of rclone that time's can't be set so shouldn't be used for syncing

github.com

rclone/rclone/blob/d2efb4b29b3b0be4daf700ffd803267f07f635ea/backend/local/local.go#L543-L549


// Precision of the file system
func (f *Fs) Precision() (precision time.Duration) {
	f.precisionOk.Do(func() {
		f.precision = f.readPrecision()
	})
	return f.precision
}

You shouldn't need to edit any other files the docs will be auto generated from the option help.

RCloneUser1 · July 29, 2020, 10:25pm

Hello @ncw

Created a PR based on the suggestions you provided. The flag works on our local tests.

Do you happen to have a timeline for the next RClone release (hopefully including this fix)?

ncw · July 30, 2020, 3:45pm

I've merged this to master now - thank you! - which means it will be in the latest beta in 15-30 mins and released in v1.53.

1.53 should be about 2 weeks away

RCloneUser1 · July 30, 2020, 4:52pm

Thank you for the update and for all the help on this!

Chtimes Error On Local Mounted Copy

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

A log from the command with the -vv flag

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag