On Linux, RClone ignores empty subdirectories when uploading to Google Drive.
However, uploading the same directory from a Windows computer to the same remote results in RClone uploading all subdirectories, empty or not:
2024/03/14 12:28:12 INFO : data/1c: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:13 INFO : keys/108b46d713cdb08939dbe7975c0a607abac198cc8f4e7279b18e024059c2614b: Copied (new)
2024/03/14 12:28:13 INFO : config: Copied (new)
2024/03/14 12:28:14 INFO : index/86766cb93cafce22d440701df466567e6f755db73b82bdcddd7fba07577b2aaf: Copied (new)
2024/03/14 12:28:14 INFO : data/1d: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:16 INFO : data/1e: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:18 INFO : data/1f: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:20 INFO : data/20: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:22 INFO : data/21: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:23 INFO : data/22: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:25 INFO : data/23: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:27 INFO : data/24: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:29 INFO : data/25: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:31 INFO : data/26: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:33 INFO : data/27: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:35 INFO : data/28: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:37 INFO : data/29: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:39 INFO : data/2a: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:40 INFO : data/2b: Made directory with metadata (mtime=2024-03-14T11:20:07.161475+01:00)
2024/03/14 12:28:42 INFO : data/2c: Made directory with metadata (mtime=2024-03-14T11:20:07.1771116+01:00)
2024/03/14 12:28:44 INFO : data/2d: Made directory with metadata (mtime=2024-03-14T11:20:07.1771116+01:00)
2024/03/14 12:28:46 INFO : data/2e: Made directory with metadata (mtime=2024-03-14T11:20:07.1771116+01:00)
These results appear with rclone ncdu:
rclone ncdu v1.65.2 - use the arrow keys to navigate, press ? for help
-- remote:/Linux/Backup/Repo/data ------------------------------------------------------------------------------------------------------
10.001Mi [##########] /ed
1.446Ki [ ] /33
Total usage: 10.002Mi, Objects: 2
rclone ncdu v1.65.2 - use the arrow keys to navigate, press ? for help
-- remote:/Windows/Backup/Repo/data ------------------------------------------------------------------------------------------------------
e 0 [ ] /00
e 0 [ ] /01
e 0 [ ] /02
e 0 [ ] /03
e 0 [ ] /04
e 0 [ ] /05
e 0 [ ] /06
e 0 [ ] /07
e 0 [ ] /08
e 0 [ ] /09
e 0 [ ] /0a
Total usage: 10.002Mi, Objects: 256
How can I avoid uploading empty directories from Windows?
might want to update rclone to latest stable, rclone selfupdate
and, for the copy command, can you post top lines of rclone debug log, or at least, a log showing the issue?
edit: i was able to reproduce the issue.
strange that rclone would behave differently on windows and linux with empty dirs?
maybe this is a bug?
This sounds very possibly related to what I was seeing here (Windows slashes causing trouble again). It's weird that the Github CI wouldn't be catching it though. Any chance you could run the same test on Windows and Linux and upload both debug logs for comparison? (both with v1.66, not earlier)
Does the Windows issue persist even with --no-update-dir-modtime?
Yesterday I was using 1.66 without --no-update-dir-modtime. Today I tried with the option on.
It immediately copied files from the parent dir, but the whole operation didn't end soon after. Instead, it hangs printing with ETA 0s every minute.
After 8 minutes, I bash "ENTER" and it finally tells me it completed the upload a couple of minute before. However, rclone ncdu shows that it also created the empty directories, even if it doesn't print them to the console (unlinke yesterday without specifying the option).
This is the console output, pay attention to the timestamps.
After 11:08:48 I received anything until I bashed ENTER 4 times at 11:12:00. I don't know why it did not print ETA 0s at 11.09.48:
No, just a guess. Today it is creating the same empty folders, but with --no-update-dir-modtime there is no output about it.
I am seeing the same problem on Linux, but downgrading fixed the issue (without using that flag):
Rclone version
Upload time
Empty dirs uploaded
Non-empty dirs uploaded
1.65.2
14s
0
2
1.66.0
512s
254
2
1.65.2 output:
2024/03/15 18:44:20 INFO : Encrypted drive 'remote{Db_Y9}:/Test': Running all checks before starting transfers
2024/03/15 18:44:20 INFO : Encrypted drive 'remote{Db_Y9}:/Test': Checks finished, now starting transfers
2024/03/15 18:44:24 INFO : config: Copied (new)
2024/03/15 18:44:27 INFO : keys/71c5547e3ac1fc04eb0648dda406d9cd4a5e7d3a70459aea4b3fad7495b68975: Copied (new)
2024/03/15 18:44:28 INFO : snapshots/6765dd167d997da78651020dd37aaaf4e5c224299836bc82d9b8a9130ac3f764: Copied (new)
2024/03/15 18:44:29 INFO : index/bb5e3b420fa61c1901251f6fad4041974d78fd3b1d5f9efe2cabd0f6356e243c: Copied (new)
2024/03/15 18:44:32 INFO : data/33/331af2bd7f155ceae1634231b94d439899e7544735fb9939df36dad3fca59d06: Copied (new)
2024/03/15 18:44:34 INFO : data/93/93983617e884088bea8b49fc641832774b4b7b28f9ecf3e072a13b6ba2d50447: Copied (new)
2024/03/15 18:44:34 INFO : 2.004 MiB / 2.004 MiB, 100%, 157.848 KiB/s, ETA 0s
1.66.0 output (aborted):
2024/03/15 18:38:39 INFO : Encrypted drive 'remote{Db_Y9}:/Test': Running all checks before starting transfers
2024/03/15 18:39:39 INFO : 0 B / 1.665 KiB, 0%, 0 B/s, ETA - (xfr#0/4)
2024/03/15 18:40:39 INFO : 0 B / 1.665 KiB, 0%, 0 B/s, ETA - (xfr#0/4)
2024/03/15 18:41:39 INFO : 0 B / 1.665 KiB, 0%, 0 B/s, ETA - (xfr#0/4)
2024/03/15 18:42:39 INFO : 0 B / 1.665 KiB, 0%, 0 B/s, ETA - (xfr#0/4)
The original issue with Windows is that I was using 1.66.0 there (slow, empty dirs), and 1.65.2 on Linux
I think the issue is that it is not reflected anywhere in documentation. --create-empty-src-dirs flag is still listed in rclone copy etc. without any caveats that it now applies only to "special" cases as otherwise it is "default"
With v1.66 suddenly things started working differently. It is fantastic news that dirs metadata can be preserved (for some remotes) but maybe it should not be default, breaking years old rclone behaviour?
Yes please, something like --dont-create-empty-src-dirs if now the default is to create empty dirs. I had to revert to 1.65.2 because backups started to take too long.
perhaps there could be a quick release of v1.66.1 and revert back to 1.65.2 behavior.
and if someone wants to use the new feature, modtime for dirs, then add explicit use flag --update-dir-modtime
imho, then there could be a discussion in the forum about breaking backwards compatibility for rclone 2.0
perhaps there could be a quick release of v1.66.1 and revert back to 1.65.2 behavior.
and if someone wants to use the new feature, modtime for dirs, then add explicit use flag
I'm unsure exactly how to make everyone happy though!
Here some ideas.
Default --create-empty-src-dirs to true and make --create-empty-src-dirs=false take the workaround action to restore the v1.65.2 behaviour. This would implicitly set --no-update-dir-modtime probably.
Make --no-update-dir-modtime restore the v1.65.2 behaviour. This is kind of what the release notes implied, but we see above that this doesn't actually work properly. The flag name isn't very relevant to creating empty directories though.
These options could be combined of course.
We could potentially make different choices for rclone move vs rclone copy vs rclone sync as I think a sync tool which doesn't sync empty directories is weird but there may be a different argument for rclone copy and rclone move.
My vote would be for default working like before in regards to empty dirs - for years people get used to it, incorporated rclone in countless scripts and automation workflows. Used --create-empty-src-dirs flag when needed and without it expect that empty ones are not created. Hence with v1.66 multiple voices here and on github asking for rclone to behave by default like it was before.
The problem is not that directories modtime is updated but that empty src directories are created. Without understanding development implications I would think that ideal solution would update modtime on dirs by default (new feature which is great that it finally works), but not create empty dirs (so this problematic aspect is unchanged)
I might be in the minority on this, but I think this comes down to a question of "how long are we obligated to stick with a prior unfortunate decision?"
If we were starting over and creating rclone from scratch, I think syncing empty dirs by default would unquestionably be the right decision. Sync is about "mirroring" the src to the dest, and it's not truly a mirror if directories are present on the src but missing on the dest -- simple as that. (Here's my longer rant about the importance of empty dirs, if anyone is interested!)
That said, I also acknowledge that for better or worse, that is not the decision that was made, and rclone has now existed for a long time with the opposite default. So, changing it now would certainly be a change, and any change is going to be unwelcome for some users. There are certainly good arguments for sticking with the prior default, just for the sake of not breaking people's workflows. (I could quibble about why such people are updating rclone without reading the release notes... but it is what it is!)
Three points to consider in favor of changing it:
Technically speaking, bug fixes are not "breaking changes" (for example, Golang's compatibility promise makes exceptions for bugs and unspecified behavior.) Users do sometimes become accustomed to buggy behavior (see this classic comic!), but that can't be a reason to never fix bugs. I think this one is a pretty close call -- there are good arguments that failing to sync empty dirs was a bug that was finally fixed, and also good arguments that it was an intended design choice.
For every existing user who will be upset about the change, there could be a future new user who will be confused about the unintuitive default behavior. (I specifically remember tripping over this when I was new to rclone -- it didn't even occur to me that empty dirs might not be copied!)
The "not breaking things" ship has maybe sailed a bit now, as v1.66.0 did in fact make this breaking change for better or worse, and the question before us now is whether to change it back... (anyone currently using v1.66.0 has already had their workflow break once...should we make them fix it twice?)
Whatever is decided, I do think that @kapitainsky is right that syncing empty dirs and syncing dir modtimes are two separate questions, and decoupling them is helpful. (i.e. even if we decide to revert the empty dir behavior, I think we could still keep syncing dir modtimes by default for non-empty dirs. Dir modtimes is a brand new feature, so there's no backward compatibility to worry about, and file modtimes are already synced by default so doing the same for dirs is consistent.)
TL;DR: syncing empty dirs by default makes more sense to me, but it's been the other way for a long time now, so I understand if the better design choice is outweighed by the priority of not breaking existing workflows. Ultimately, if I need to keep using --create-empty-src-dirs to get my empty dirs, I am fine with that outcome.
As for the rest of the argument for changing "empty dirs" behaviour is also very solid. I agree that it is more logical and probably would be default behaviour when building rclone from scratch again. But legacy sometimes is equally important. Better sometimes is enemy of good:)
If it was decided by voting I would need some time to think which vote to cast:) It is not black and white overall.
not seeing a fire that needed to be put out.
so, let's revert back to old behavior.
before breaking backwards compatibility, should take our time to discuss it in the forum.
sounds confusing and will create new, unforeseen problems.
