Using rclone touch recursively

Anyone got any advice tips on how I could use rclone touch recursively, it would be great if I could someone give it a directory and say change all the files in it, and below it to this date/time.

Maybe I could pipe output from rclone lsf to rclone touch maybe? Would that work?

hi,

using rclone lsf, i do not think that is supported as per this
i did a test

  • rclone did not touch the files
  • rclone did not complain.

so it would seem to require scripting, and for need to know what platform you are using?

on windows, i wrote this just now, very quickly as a prove on concept.
it is not very efficient.

if anyone has a tweaked version for any platform, please post it.
the best script will win a free virtual :beer:, well since it is virtual, make that a six-pack.

rclone lsf --files-only --recursive --absolute %source% > files.txt
for /F "tokens=*" %%A in (files.txt) do rclone touch --no-create  %source%/%%A"

I tried with a little help from your post, but sadly it doesn't work as there is a space between the remote and the path, I don't know how to get rid of that, it would work I believe without the space.

rclone lsf GD:'/TestFolder/' --fast-list --files-only --format p --recursive --absolute | xargs -t -L 1 rclone touch --dry-run --fast-list --localtime --log-level DEBUG --no-create GD:

I get the following output from the touch command.

rclone touch --dry-run --fast-list --localtime --log-level DEBUG --no-create GD: /rufus-2.0p.exe
2021/05/02 17:26:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/05/02 17:26:56 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "touch" "--dry-run" "--fast-list" "--localtime" "--log-level" "DEBUG" "--no-create" "GD:" "/rufus-2.0p.exe"]
Usage:
  rclone touch remote:path [flags]

Flags:
  -h, --help               help for touch
      --localtime          Use localtime for timestamp, not UTC.
  -C, --no-create          Do not create the file if it does not exist.
  -t, --timestamp string   Use specified time instead of the current time of day.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command touch needs 1 arguments maximum: you provided 2 non flag arguments: ["GD:" "/rufus-2.0p.exe"]
rclone lsf --files-only --recursive --absolute %source% > files.txt
for /F "tokens=*" %%A in (files.txt) do rclone touch --no-create  "%source%\%%A" -vv

or

rclone lsf $source --files-only --recursive --absolute | xargs -I file rclone touch "$source/file" --no-create -vv

1 Like

The problem with that I am having is that as it's only possible to do one file at a time, it is painfully slow. maybe @ncw can add --recursive to rclone touch maybe that would surely speed things up.

first, --magic as that would take care of touch --recursive and every other feature request :wink:

another option is rclone mount and use the linux command touch

What would --magic do ?

sorry, that is my lame attempt at rclone humor in the forum...
try it youself, not so easy to do :wink:

Ah conveying humour with just text is hard, when the other person doesn't get the context or the delivery of said joke, and instead just reads it as how it's written and left to make it up from there.

well, my ability to deliver a joke is off but at least my scripts work

I was going to suggest using the rclone api to an rclone rcd instance. However we seem to have left touch out of the API :blush:

A recursive flag for touch would be ideal I agree, and that would be even faster than using the api.

It would be fairly easy to add. If you wanted to have a go at it, I'd be happy to talk you through it.

Sure @ncw please talk me through it.

@AeroMaxx - first step is open a new issue on Github.

Further discussion should go there.

Here are some notes on how to implement.

The code for the touch command is here: rclone/touch.go at master · rclone/rclone · GitHub

This will need refactoring a bit, but in outline instead of calling cmd.NewFsDstFile if -R/--recursive is set you'll call cmd.NewFsSrc(args) instead and then call a new operations.TouchRecursive with the result.

This will do something like

// Recursively touch every file in f with time t
func TouchRecursive(ctx context.Context, f fs.Fs, t time.Time) error {
	return ListFn(ctx, f, func(o fs.Object) {
		err := o.SetModTime(ctx, t)
		if err != nil {
			err = fs.CountError(err)
			fs.Errorf(o, "Failed to touch: %v", err)
		}
	})
}

This needs a test in operations_test.go but we can talk about that in step 2.

The getting the timestamp to pass to TouchRecursive needs the time stamp calculation code factored out of Touch.

I don't think rclone touch -R should ever create files only touch existing ones so I think that is about it!

Here is how rclone lsf defines the -R flag

	flags.BoolVarP(cmdFlags, &recurse, "recursive", "R", false, "Recurse into the listing.")

@ncw Add `--recursive` to rclone touch. · Issue #5301 · rclone/rclone · GitHub

So I have done the first bit. The Go code you posted I didn't really understand that, but I'll try figure it out!

func TouchRecursive(ctx context.Context, f fs.Fs, t time.Time) error {
        timeAtr := time.Now()
        if timeAsArgument != "" {
                layout := defaultLayout
                if len(timeAsArgument) == len(layoutDateWithTime) {
                        layout = layoutDateWithTime
                } else if len(timeAsArgument) > len(layoutDateWithTime) {
                        layout = layoutDateWithTimeNano
                }
                var timeAtrFromFlags time.Time
                if localTime {
                        timeAtrFromFlags, err = time.ParseInLocation(layout, timeAsArgument, time.Local)
                } else {
                        timeAtrFromFlags, err = time.Parse(layout, timeAsArgument)
                }
                if err != nil {
                        return errors.Wrap(err, "failed to parse date/time argument")
                }
                timeAtr = timeAtrFromFlags
        }
        return ListFn(ctx, f, func(o fs.Object) {
                err := o.SetModTime(ctx, timeAtr)
                if err != nil {
                        err = fs.CountError(err)
                        fs.Errorf(o, "touch: couldn't set mod time %v", err)
                }
        })
}

The test functions...

func TestTouchRecursiveWithTimestamp(t *testing.T) {
        r := fstest.NewRun(t)
        defer r.Finalise()

        timeAsArgument = "060102"
        srcFileName := "oldFile"
        err := TouchRecursive(context.Background(), r.Fremote, srcFileName)
        require.NoError(t, err)
        checkFile(t, r.Fremote, srcFileName, "")
}

func TestTouchRecursiveWithLongerTimestamp(t *testing.T) {
        r := fstest.NewRun(t)
        defer r.Finalise()

        timeAsArgument = "2006-01-02T15:04:05"
        srcFileName := "oldFile"
        err := TouchRecursive(context.Background(), r.Fremote, srcFileName)
        require.NoError(t, err)
        checkFile(t, r.Fremote, srcFileName, "")
}

That looks like you are going in the right direction! It is a bit difficult to see without seeing all the code.

I suggest you factor out the time parsing code - this should live in cmd/touch wheras the actual time setting code plus tests should live in fs/operations.

Tests in fs/operations get run against all the backends by the integration tester.

Do you want to submit a PR then I can comment on the bits of code more easily?

Thank you :slight_smile:

Sure I have forked the repo, I will submit my changes so far and do a pull request, I'll need to do that later tho.

@ncw i have done a draft pull request.

1 Like

@ncw Could I get a response to my pull request?

@ncw nevermind I have seen my pull request build fail, as it all made nothing to me I don't think I am able to do this sadly, maybe someone else can pick up the can.

If you can explain the build process I could be tempted to have another go.

No worries! I'll pick up the pull request and fix it up in a bit :slight_smile: