OneDrive _Changed folder

Help and Support
1

First of all, my apologies for categorizing this as 'Off Topic' - I'm not sure what other category to use..
Edit: I'm trying to move this to 'Help and Support'

I've been using OneDrive to store files via 'rclone crypt' for years. Some hours ago I noticed my Windows OneDrive taskbar messages scrolling like crazy, with hundreds of files with, for lack of a better term, 'rclone encrypted filenames' being listed as 'moved to _Changed files'.

I used 'rclone ncdu' on a non-Windows machine to check what this was, found the _Changed folder, with a file that didn't look like an encrypted file:

_OneDrive update details.csv

This file has a second line of:
‎Learn more at Why has my filename changed? - Microsoft Support

Going to this URL you get the following text:

OneDrive is being updated to bring you new AI features, enhanced photo and video technology, and a more personalized OneDrive experience. In some cases, OneDrive may need to rename your files and folders. This can happen if:

  • Your filenames are too long,
  • Your folder structure has too many layers (making the overall file name too long),

I think many of my rclone encrypted files are now in this _Changed folder.

Now I'm wondering what steps to take next..

2

I've noticed the same _Changed folder and the same behaviour, it seems to contain files and folders pulled from my rclone crypt and moved to this "_changed folder". Those files seem to have retained the same file names for all of them, just flattened the directory structure to shorten the overall folder/file name length.

Because the files have been moved to the "_changed folder". Rclone's next run is recopying them to the rclone crypt as "new" files.
It looks like onedrive is then also moving these to the _changed folder too. because I have copies of the same file with xxx(1), xxx(2), xxx(3) in that "_changed folder". So looks like that folder will continue to grow out until it blows my quota. Eak!

Unlear what the exact character limit that's triggering this is. But, "_OneDrive update details.csv" contains a column with the old path and file name and checking the characters in that column, the smallest is 373 characters. Could indicate that the character limit is 372??

3

Oh well... might be the end of using OneDrive as a general storage. MS services only... Unless there are some conditions which can prevent it - 372 characters observation is definitely good starting point.

@PepJr - does it mean that for path shorter than 372 files were not moved? Do you keep crypt files in dedicated directory? Or you use OneDrive root folder? Do you use by any chance OneDrive sync to move rclone crypt data to the cloud?

Would you mind sharing few full path of moved content? If names are encrypted there is nothing personal there.

4

Here are a few of the moved files/filepaths

Blockquote
‎My files‎/OneDrive_Backup_crypt/bd5jo77uga7jnft5avd4l2cooc/0p6jmv9930m2bj1ivbi2fan04o/o2442mvctn0ko4v8acl68cefco/e2jtjad5b05vv8rppqkisidpuk/dhait98jsi1e66pdrgrmlcrisc/82petqenc3ekn094j773fcmnu8/7kh7qgk341g7ujar14191rqfpk/4rbl7cprdc2madd78v324esg9o/au6ku1l29vrrfurr3723n5ls24/tdcpqpu7bpe7ndfbck1hrjmp1s/ai8pf8nhcgcbgeq4v6us8v3li4/kqq9vs14lt2i28l2ac68qcphuk/lgpaf8muedtcqlgk6jgsp1os2c
‎My files‎/OneDrive_Backup_crypt/bd5jo77uga7jnft5avd4l2cooc/0p6jmv9930m2bj1ivbi2fan04o/o2442mvctn0ko4v8acl68cefco/e2jtjad5b05vv8rppqkisidpuk/dhait98jsi1e66pdrgrmlcrisc/82petqenc3ekn094j773fcmnu8/7kh7qgk341g7ujar14191rqfpk/4rbl7cprdc2madd78v324esg9o/au6ku1l29vrrfurr3723n5ls24/tdcpqpu7bpe7ndfbck1hrjmp1s/ai8pf8nhcgcbgeq4v6us8v3li4/lt8cbeij8kvujrlgdg62ajdek3i82toqcjmkb9bcd26a5f1t5bmg
‎My files‎/OneDrive_Backup_crypt/bd5jo77uga7jnft5avd4l2cooc/0p6jmv9930m2bj1ivbi2fan04o/o2442mvctn0ko4v8acl68cefco/e2jtjad5b05vv8rppqkisidpuk/dhait98jsi1e66pdrgrmlcrisc/82petqenc3ekn094j773fcmnu8/7kh7qgk341g7ujar14191rqfpk/4rbl7cprdc2madd78v324esg9o/au6ku1l29vrrfurr3723n5ls24/tdcpqpu7bpe7ndfbck1hrjmp1s/ai8pf8nhcgcbgeq4v6us8v3li4/q5uatral19gdaq9e48fptphscc/bt5l1i1pav951l5ba0sisgeuss
‎My files‎/OneDrive_Backup_crypt/bd5jo77uga7jnft5avd4l2cooc/0p6jmv9930m2bj1ivbi2fan04o/o2442mvctn0ko4v8acl68cefco/e2jtjad5b05vv8rppqkisidpuk/dhait98jsi1e66pdrgrmlcrisc/82petqenc3ekn094j773fcmnu8/7kh7qgk341g7ujar14191rqfpk/4rbl7cprdc2madd78v324esg9o/au6ku1l29vrrfurr3723n5ls24/tdcpqpu7bpe7ndfbck1hrjmp1s/ai8pf8nhcgcbgeq4v6us8v3li4/q5uatral19gdaq9e48fptphscc/bt5l1i1pav951l5ba0sisgeuss/6ogi7nvhn6spqsmd50k07f50s8
‎My files‎/OneDrive_Backup_crypt/bd5jo77uga7jnft5avd4l2cooc/0p6jmv9930m2bj1ivbi2fan04o/o2442mvctn0ko4v8acl68cefco/e2jtjad5b05vv8rppqkisidpuk/dhait98jsi1e66pdrgrmlcrisc/82petqenc3ekn094j773fcmnu8/7kh7qgk341g7ujar14191rqfpk/4rbl7cprdc2madd78v324esg9o/au6ku1l29vrrfurr3723n5ls24/tdcpqpu7bpe7ndfbck1hrjmp1s/ai8pf8nhcgcbgeq4v6us8v3li4/q5uatral19gdaq9e48fptphscc/bt5l1i1pav951l5ba0sisgeuss/8c4cois23hib5490vt0q82ta20

1 Like
5

I've set up a quick experiment with some .Txt files in a folder, such that the total file and path lenghs are from 360 through 374 characters*

Will see what\if onedrive decides to move.

  • measuring as MS seem to be... so including "my files" at the start

This is outside of my crypt btw, just an experiment in onedrive

6

Good idea. Let's see what it brings.

7

According to OneDrive documentation max full path length limit is 400 characters so it is strange where 372 limit comes from. However they also mention that files can be moved when "Your folder structure has too many layers" - unfortunately they do not explain it in more details.

BTW - you should use basse32768 encoding for OneDrive. Your path would be significantly shorter. You are using the worst possible choice - base32.

base32 is 5 bits encoding so you can encode only 250 bytes using 400 characters. Switching to base32768 (15 bits) this is 750 bytes. All thanks to OneDrive counting 2 bytes UCS2 characters (used by base32768) the same way as one byte ones.

1 Like
8

Thanx for the info about base32768 - I haven't been keeping track of new rclone features (it just works..). I will investigate it!

I checked my 'OldPath' values, and found length varying between 364 and 416.

Note that the OldPath values I used stripped the first part ('enc' is the subdir I use for crypt-ed files). The MS provided CSV fields are prefixed with a \u200e (for 'lrm' I guess):

skipField4N = len('"<200e>My files<200e>/enc/') - 1;

I also did the following to get the decrypted file names from the MS supplied OldPath values in the CSV file.

  • create a local directory structure similar to the MS supplied OldPath values(sorry for the 'pseudo-code'):

     for $1 in OldPaths:  # $1 is 'touchTest/$oE'
   mktouch() { mkdir -p "$(dirname "$1")" && touch "$1"; } # https://superuser.com/q/139046/touch-non-existing-dirs

  • create an rclone config file with 'remote' pointing to the 'touchTest' dir (other values obtained from rclone config dump )

       [decryptNames]
   type = crypt
   remote = ./touchTest
   filename_encryption = standard
   directory_name_encryption = true
   password = password-value-from-rclone-config-dump-json
   password2 = password2-value-from-rclone-config-dump-json

  • execute rclone with this config file to get a JSON file containing, among other things, the decrypted names:

       rclone --config rccfg.2 lsjson decryptNames: --encrypted -R

Anyone know if there is an easier/alternative way to get the decrypted names?

9

Sorry but what exactly you are trying to achieve here?

10

I have a list of 10's of thousands of files that were moved by MS. This list is in the CSV file MS stuck in the '_Changed files' dir.

Most of the files have the same (basename, i.e., no dir1/dir2/, just the filename) encrypted name that rclone used, but others have a (1),(2),(3),etc suffix.

I want to be able to know what the original decrypted full path name (including dir1/dir2/dir3..etc) of those moved files were.

I already think I lost the creation-date-time of these files, as MS moving them means they have a $now creation-date-time.

11

Oups. What a pickle.

But if they were only moved I would think it should be enough to create new crypt pointing into _Changed/path/to/cryptroot directory and simply restore all from there.

12

Yes, it's an 'interesting' problem..

I believe I know how to restore them.

However, as PepJr pointed out, the directory structure was flattened when MS did the move.

I would like to recreate that dir structure (especially given base32768 ) to keep the same structure I had before MS moved them.

Also, some files are duplicates (MS put the '(1),(2),etc' suffix on them) because they were in a different subdirectory. Hence my needing to know the decrypted full path name.

Note: I've been able to get the decrtyped full path name using the (lengthy and probably confusing) reply I posted above. I'm just wondering if there was an easier way instead of creating the full subdirectories structure.

13

It is ridiculous what MS does. BIG red flag for OneDrive future in my setup.

I am checking if my other backups are up to date:)

1 Like
14

Yep, and thanks for your speedy replies!

15

This happened to me too a while back. Thankfully I only had 10 or so files moved, so it was not too difficult to figure out what moved from where with a bit of guesswork.

I agree it's shitty of MS to do such things, but I won't dismiss them as my storage of choice, as they are by far the cheapest per GB.

16

Did you figure out why these files were moved?, and not others. Path length?

17

@kapitainsky Thanks for the heads up on Base32768. I've moved over to it with a server side move.

MS have done nothing with the "experiment" files I created. Will check them in a few days... either it's a scheduled job on their end, I messed up somehow, or there is something about the way I set it up that skips past MS's cleanup. Who knows?

1 Like
18

Well I actually got emails that said the following:

===================================================
Starting on November 4, 2024, we’re making some system updates to bring you new AI features, enhanced photo and video technology, and a more personalized experience.

How this may impact your files

  • File names: Some file and folder names may be slightly altered with an added number or shortened if character count limit is exceeded (400 characters max).
  • File locations: If paths of your files and folders locations exceed character count limit (400 characters max), they may be moved to a new folder in your OneDrive.
    To see which files will be affected, sign in to OneDrive, go to My files, open the folder "_Changed files”, then open the CSV file “_OneDrive update details” .
    =================================================
1 Like
19

Is it possible to change this for existing crypt remotes?

20

Yes.

What you do is set up a new crypt remote by copying the old one in the config file, renaming it and adding filename_encoding = base32768. Choosing a different destination directory is a good idea here too so the old and the new don't overlap.

You can then do a server side move something like

rclone move --server-side-across-configs -vv old: new: