Control log output of copy/copyto

[patakijv (John Pataki)]

When using rclone copy or rclone copyto, I want to have a one line output that includes the file that is being considered and if copy/copyto decides it needs to be updated or if it is the same and doesn't need to be updated.

I've tried adding --stats-one-line and it although it does seem to compress to one line of text, it seems to add a blank line (I discovered the blank line is for the operations.go line when the file is not needed to be copied - see below my code and output I added to this post) after each line of output so for each file there are two lines created in the log (one blank).

Also, how can I just show the filename of the file being worked on and not the .go filename where the code is executing? When I try to change --log-format shortfile,date,time ex: 2021/03/09 09:46:07 stats.go:368: INFO the "stats.go:368" (or "operations.go:477") is a reference to the code that is executing and has nothing to do with the file being worked on... and it is unexpected from the --log-format option description to see that code reference.

Maybe this is really two questions? Ultimately I would like to see output similar to rclone sync when using rclone copy or rclone copyto

Here is my command line: rclone copy -v --log-format shortfile,date,time --stats-one-line --log-file /srv/logs/${CONFIG_CODE}.log ${downloaded_path} ${fileserver_dir}

rclone --version
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

I notice that -stats-one-line I am using is actually documented as a flag to use with --progress even though I am not using --progress, maybe that is why I am getting two lines unexpectedly?

and below is a snippet of output... I don't need the "progress output as the second line, I just want the filename and the status of what happened (i.e. Copied (replaced existing))

2021/03/09 14:49:47 operations.go:477: INFO  : acme_discounts.zip: Copied (replaced existing)
2021/03/09 14:49:47 stats.go:368: INFO  :     5.540k / 5.540 kBytes, 100%, 8.265 MBytes/s, ETA 0s

2021/03/09 14:49:47 operations.go:477: INFO  : acme_blue_orders.zip: Copied (replaced existing)
2021/03/09 14:49:47 stats.go:368: INFO  :     1.983M / 1.983 MBytes, 100%, 183.183 MBytes/s, ETA 0s

2021/03/09 14:49:49 operations.go:477: INFO  : acme_inventory.zip: Copied (replaced existing)
2021/03/09 14:49:49 stats.go:368: INFO  :    28.131M / 28.131 MBytes, 100%, 399.907 MBytes/s, ETA 0s

2021/03/09 14:49:50 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:50 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:52 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:52 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:53 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:54 stats.go:368: INFO  :          0 / 0 Bytes, -, 0 Bytes/s, ETA -

2021/03/09 14:49:51 operations.go:477: INFO  : acme_green_orders.zip: Copied (replaced existing)
2021/03/09 14:49:51 stats.go:368: INFO  :     4.547M / 4.547 MBytes, 100%, 330.929 MBytes/s, ETA 0s

2021/03/09 14:49:52 operations.go:477: INFO  : acme_red_orders.zip: Copied (replaced existing)
2021/03/09 14:49:52 stats.go:368: INFO  :     7.817M / 7.817 MBytes, 100%, 35.990 MBytes/s, ETA 0s

[ncw (Nick Craig-Wood)]

@patakijv

When using rclone copy or rclone copyto, I want to have a one line output that includes the file that is being considered and if copy/copyto decides it needs to be updated or if it is the same and doesn't need to be updated.

Using -v will give you the files rclone decides to copy. Those are the INFO lines in your log above.

@patakijv

I've tried adding --stats-one-line and it although it does seem to compress to one line of text, it seems to add a blank line (I discovered the blank line is for the operations.go line when the file is not needed to be copied - see below my code and output I added to this post) after each line of output so for each file there are two lines created in the log (one blank).

The stats will show all the files that are being worked on if you don't use --stats-one-line. It will only do this every 1 minute by default.

--stats-one-line compresses this to a single line.

@patakijv

Also, how can I just show the filename of the file being worked on and not the .go filename where the code is executing? When I try to change --log-format shortfile,date,time ex: 2021/03/09 09:46:07 stats.go:368: INFO the "stats.go:368" (or "operations.go:477") is a reference to the code that is executing and has nothing to do with the file being worked on... and it is unexpected from the --log-format option description to see that code reference.

Yes the log file shortfile is for debugging the go program.

Rclone logs always show the file names they are operating on after the first :

@patakijv

rclone copy -v --log-format shortfile,date,time --stats-one-line --log-file /srv/logs/${CONFIG_CODE}.log ${downloaded_path} ${fileserver_dir}

Try removing shortfile and replacing -v with -vv - you'll get info on what files rclone is considering as DEBUG logs and what files it transfers as INFO logs.

@patakijv

and below is a snippet of output... I don't need the "progress output as the second line, I just want the filename and the status of what happened (i.e. Copied (replaced existing))

Using -v is probably what you want

rclone copy -v --log-file /srv/logs/${CONFIG_CODE}.log ${downloaded_path} ${fileserver_dir}

This will only show the files rclone copies though, not the ones that were the same.

[patakijv (John Pataki)]

Thanks Nick for your reply.

I see the following when I use rclone copy -v --log-file /srv/logs/${CONFIG_CODE}.log ${downloaded_path} ${fileserver_dir} - (note this is in a bash loop) - notice there are two outputs per file... one is the progress even though I haven't specified --progress and for the ones that are the same there is a blank output where it would normally have the file if it was being copied.

2021/03/10 13:00:13 INFO  : acme_red_orderss.zip: Copied (new)
2021/03/10 13:00:13 INFO  : 
Transferred:   	  389.834k / 389.834 kBytes, 100%, 76.344 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/10 13:00:13 INFO  : acme_blue_ordeers.zip: Copied (new)
2021/03/10 13:00:13 INFO  : 
Transferred:   	  171.939k / 171.939 kBytes, 100%, 64.800 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/10 13:00:13 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2021/03/10 13:00:13 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2021/03/10 13:00:13 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2021/03/10 13:00:13 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

How do I disable the progress and the output of the blank entry when files are the same?

[patakijv (John Pataki)]

I just found --stats=0 which does turn off the progress update that was happening. I realized that because it is occurring in a bash loop that progress is intended to be at the default interval or the end of the task and each iteration in the loop was a new task.

So that helps resolve the "too much output" issue... is there a way to control the filename output so it includes the full (or a relative) path or not vs just the filename - like it does with rclone sync?

[ncw (Nick Craig-Wood)]

@patakijv

So that helps resolve the "too much output" issue..

Great

@patakijv

is there a way to control the filename output so it includes the full (or a relative) path or not vs just the filename - like it does with rclone sync?

Rclone shows the full path relative to the root of the sync. If you use rclone copy on a file then the root of the sync is the directory the file is in, so for rclone copy on a file, you'll always just see the file names.