Empty dirs not wanted

What is the problem you are having with rclone?

i get empty dirs for metadata
i don't use the metadata flag but get metadata

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: unknown
  • os/kernel: 3.10.108 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./rclone copy    /volume1/cameras/ onedrive: -v --min-age=5s --max-age=1h

Please run 'rclone config redacted' and share the full output. If you get

[onedrive]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal
client_id = XXX
client_secret = XXX

A log from the command that you were trying to run with the -vv flag

2024/03/14 20:34:28 DEBUG : achter/20240314AM: Making directory with metadata
2024/03/14 20:34:28 DEBUG : voor/@SSRECMETA: Excluded
2024/03/14 20:34:29 INFO  : voor/20240314PM: Set directory modification time (using SetModTime)
2024/03/14 20:34:29 INFO  : achter/20240314AM: Made directory with metadata (mtime=2024-03-14T13:01:21.822780297+01:00)

might try --no-update-dir-modtime

still making dirs without metadata

I can reproduce it myself - starting with rclone v1.66 empty directories are always copied.

It seems to be a bug introduced together with ability to sync directory modification times.

At the moment I think only solution is to run rmdirs after rclone copy.

Hopefully in the next rclone version this will be fixed.

Thx for testing
Let’s wait for the next version

Before v1.66 rclone didn't sync directories at all.

With v1.66 we've introduced directory syncing code. This means that rclone tries to replicate exactly the directories on the source including the modification times.

These two log lines show the directories being created and updated with the correct modification times. The fact it says metadata there is irrelevant - it is just the mechanism used for setting the modification time.

What is new is getting the empty directories synced. Rclone v1.65.2 didn't used to do this which is arguably a long standing bug in rclone which has been fixed by v1.66.

Does this behaviour make sense? Do you think we need to fix something?

This is where the problem is. IMO it is hard to argue that it was a bug as rclone always offered --create-empty-src-dirs flag. So was always able to create empty directories but it was optional.

Of course it is design decision how to handle it but I think that by default empty dirs should continue not to be created - keeping default behaviour the same as always.

with the max-age=1h i would expect that only the dirs will be craeted with file less dan 1h old

is have 14 days of video recording in 14x4=56 folders. if i
use copy the last hour i woud expect to see only the folders for the last hour
after the copy i do a delete min-age=1h en rmdirs to remove the empty folders
now the job create en rmdirs the folder every run

If this is now intended behaviour I would love to see a new flag to prevent empty dirs from being created.

I use a --max-age=15min and then move the files inside manually, eventually.
This was working great and new directories normally meant that new files were inside it. But now I get a new directory every time t he command runs and it is empty.
Since I run the rclone command 24/7 using a cronjob to sync instantly to my local machine I cannot use the rmdirs command.

So I had to rollback to 1.65.2 unfortunately.

Note that there is an issue about this now

For rclone v1.66.1 I am going to make sure it obeys the --create-empty-src-dirs flag.

What the defaults for this for rclone sync, rclone copy and rclone move I have yet to decide. Currently they are all false. There is an argument for making this true for rclone sync but that might be a step too far for a point release.

Please try this fix

v1.67.0-beta.7855.13c72a94e.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

It should restore v1.65.2 behaviour

yes no empty dirs are created. that is good. the orignial scritps works now
but another script with a source with empty dirs now gives
Failed to update directory timestamp or metadata: directory not found
so its trying to write metadata to an non existing dir
with --create-empty-src-dirs is works fine :wink:

Can you have a go with this one please?

v1.67.0-beta.7901.7042810e8.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

still trying to change metadata on non existing folder

rclone v1.67.0-beta.7901.7042810e8.fix-7689-empty-dirs

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-102-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.2
  • go/linking: static
  • go/tags: none
    2024/04/17 08:22:04 ERROR : XXX/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865: Failed to update directory timestamp or metadata: directory not found
    2024/04/17 08:22:04 ERROR : XXX: Failed to update directory timestamp or metadata: directory not found
    2024/04/17 08:22:05 ERROR : OneDrive root 'XXX': not deleting directories as there were IO errors
    2024/04/17 08:22:05 ERROR : Attempt 1/3 failed with 3 errors and: failed to set directory modtime: 2 errors: last error: directory not found

Can you give me a way to replicate that please?

Managed to replicate it!

mkdir -p /tmp/src/a/b/c/d
echo hello > /tmp/src/a/b/c/d/file.txt
rclone copy -vv /tmp/src/ /tmp/dst --min-size=1k

on my Windows PC in have c:\temp
i create c:\temp\folder1
i create c:\temp\folder1\subfolder1
i have file New Text Document.txt in c:\temp


dir -r c:\temp\

Directory: C:\Temp

Mode LastWriteTime Length Name


d---- 17/04/2024 12:01 folder1
-a--- 17/04/2024 12:01 0 New Text Document.txt

Directory: C:\Temp\folder1

Mode LastWriteTime Length Name


d---- 17/04/2024 12:01 subfolder1


then i do
.\rclone.exe sync c:\temp onedrive:temp -P

on onedrive is only have the New Text Document.txt file


this happens

.\rclone.exe sync c:\temp onedrive:temp -P
2024/04/17 12:02:14 ERROR : folder1: Failed to update directory timestamp or metadata: directory not found
2024/04/17 12:02:14 ERROR : OneDrive root 'temp': not deleting directories as there were IO errors
2024/04/17 12:02:14 ERROR : Attempt 1/3 failed with 2 errors and: failed to set directory modtime: directory not found
2024/04/17 12:02:14 ERROR : folder1: Failed to update directory timestamp or metadata: directory not found
2024/04/17 12:02:14 ERROR : OneDrive root 'temp': not deleting directories as there were IO errors
2024/04/17 12:02:14 ERROR : Attempt 2/3 failed with 2 errors and: failed to set directory modtime: directory not found
2024/04/17 12:02:15 ERROR : folder1: Failed to update directory timestamp or metadata: directory not found
2024/04/17 12:02:15 ERROR : OneDrive root 'temp': not deleting directories as there were IO errors
2024/04/17 12:02:15 ERROR : Attempt 3/3 failed with 2 errors and: failed to set directory modtime: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 2 (retrying may help)
Checks: 2 / 2, 100%
Transferred: 1 / 1, 100%
Elapsed time: 1.9s
2024/04/17 12:02:15 Failed to sync with 2 errors: last error was: failed to set directory modtime: directory not found

I think I've figured it out!

Can you give this a go please

v1.67.0-beta.7906.91d78fd38.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

there is no windows version :wink:
and the ams64 linux doesn't run

Any update on this ?