Files randomly not migrated since path is not found (Local to GDrive)

Vivien_VOILLOT · June 15, 2022, 4:11pm

What is the problem you are having with rclone?

Hello,
I am migrating from local to several Google Shared Drives.
We are talking about 500 to 600Go of data, and some files, irregardless to source emplacement or extension or else return an error that I can't really set here since it is in french, so I will try to translate at best :
Failed to copy: failed to open source object: the specified path is not found (?)
(Failed to copy: failed to open source object: Le chemin d’accès spécifié est introuvable.)
It shouldn't be related to the remotes or the command itself since it impacts only several files amongst a vast majority which is expectedly migrated.

What I checked :

remigrating returns the same result
remigrated specifically the folder of those files returns the same result
the path of the source folder is correct
rclone ls [source] find the files that are not migrated
the files which can't be migrated are visible and usable in the source directory by my user account
the files' paths' numbers of characters are not "that" long (I am not sure there is a limit of characters for loacl sources)
the files' path's numbers of sublevels are not "that" long (same uncertainty here)
moving the files form the source to the destination with a drag and drop works.

Since I have around 12k files to move, I really don't want to do it manually ^^

Would you happen to know how to solve the kind of issue I am facing here ?

Thanks

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

os/version: Microsoft Windows Server 2012 R2 Standard (64 bit)
os/kernel: 6.3.9600.19505 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.9
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local as source
GDrive at destination (~50 Shared Drives)

The command you were trying to run

As an example, amongst the 50-ish I made

rclone copyto local:"D:\OneDrive - Mairie des Allues\Services\Domaine skiable" Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-3evague.csv --log-level INFO -P --transfers 8

The rclone config contents with secrets removed.

[local]
type = local
nounc = true

[Mbl-05.6-DomaineSkiable]
type = drive
scope = drive
service_account_file = C:\RClone\test.json
team_drive = 0ACc9eDDCHugXUk9PVA
root_folder_id =

A log from the command with the `-vv` flag

https://docs.google.com/spreadsheets/d/1EeO0FEThDc4J0ySNwqf7Mxh5_ghrOXpcLgnTM-QaQwE/edit?usp=sharing

There are like 18k lines in it

Animosity022 · June 15, 2022, 4:21pm

There's been a few fixes related to that in the beta.

Install (rclone.org)

Can you test with the beta version and validate that solves the issue?

Vivien_VOILLOT · June 15, 2022, 4:52pm

Thanks.
After setting on a new folder, I've tried to do a rclone config for both :

v1.59.0-beta.6215.0279bf3ab
v1.59.0-beta.6211.ec117593f
and each time I get an access denied !

Is there a way to "update" to a beta an existing version ?

Animosity022 · June 15, 2022, 4:53pm

You can:

rclone selfupdate --beta

From where you normally have the binary.

ncw · June 15, 2022, 5:11pm

Can you make a list of the file names which fail

I picked out two file names at random, first a failing one, then a good one

BAD: CONVENTION TRIPARTITE/CONVENTIONS TRIPARTITES/ANCIENNES CONVENTIONS - écoles + 6 moniteurs/CONTENTIEUX 2012/CONTENTIEUX CONVENTIONS 2012/lettre du fin Avril - commentaires sur les pistes de Meribel Parallel lines 11 07 12.msg
OK:  CONVENTION TRIPARTITE/CONVENTIONS TRIPARTITES/ANCIENNES CONVENTIONS - écoles + 6 moniteurs/CONTENTIEUX 2012/CONTENTIEUX CONVENTIONS 2012/ANALYSE ADAMAS.pdf

I don't see anything obviously the matter with them.

While file system are you copying from?

Can you copy that file if you CD into that directory first?

Trying the beta is a good idea.

Vivien_VOILLOT · June 15, 2022, 9:01pm

Thanks both of you.

I am now in the latest beta version.

I retried a command targeting a folder which contains only not-migrated files (all of them are .jpg) and the result is the same
The command being :
rclone copyto local:"D:\OneDrive - Mairie des Allues\Services\Domaine skiable\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE" Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-test.csv --log-level INFO -P --transfers 8
And the log file is :

Here is the list of all the files which were not migrated Log - 3rd wave - Google Sheets
You will find in it around 1k3 lines in yellow : those are other aberrations : there is no error given, and the path has been brutally cut !

@ncw if I CD ?
The ls command is effective :
C:\RClone>rclone ls local:"D:\OneDrive - Mairie des Allues\Services\Domaine skia
ble\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE"
188595 Alexis PINTURAULT (FRA).jpg
318333 Andrew WEIBRECHT (USA).jpg
74034 Christof INNERHOFER (ITA).jpg
28138 Didier DEFAGO (SUI).jpg
60617 Dustin COOK (CAN).jpg
1533977 Georg STREITBERGER (AUT).jpg
48472 Johan CLAREY (FRA).jpg
724182 Matteo MARSAGLIA (ITA).jpg
62729 Max FRANZ (AUT).jpg
161466 Patrick KUENG (SUI).jpg
90368 Peter FILL (ITA).jpg
126334 Romed BAUMANN (AUT).jpg
20992 Thumbs.db
95747 Vincent KRIECHMAYR (AUT).jpg

Edit :
can a comma in the files' paths cut the command ?
It seems the ones which are cut contain a comma at this precise moment !

Edit 2 nah that was a stupid take : some files of those folders have been migrated. But the error messages of those which weren't are cut...

ncw · June 16, 2022, 11:51am

How about

cd "D:\OneDrive - Mairie des Allues\Services\Domaine skiable\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE"

Then

rclone copyto . Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-test.csv --log-level INFO -P --transfers 8

Does that work?

Also why are you using --ignore-size - you shouldn't need that - maybe that is part of the problem?

Vivien_VOILLOT · June 16, 2022, 3:08pm

Ah ok, thanks.

I haven't done it yet but I think me and my colleague found what happens !
On the same folder, of 5 items, of which only 1 has been migrated, when we go on the files' setting here are the path :

migrated file :
D:\OneDrive - Mairie des Allues\Services\Services techniques\Bâtiments\Accessibilité PMR\BATIMENTS\Parc Olympique\2017_Mise aux normes accessibilité sanitaires PO\Marché travaux\03_Exécution\Lot 4_Revolta Blaudeau_Peinture placo
not migrated file :
D:\ONEDRI~2\Services\SERVIC~1\BTIMEN~1\ACCESS~1\BATIME~1\PARCOL~1\2017_M~1\MARCHT~1\03_EXC~1\LOT4_R~1
So, I don't know how users can manage to get this outcome while working on their files, but it will explain that RClone can't find the "true" path of those files.

We then copied the folder at root, and migrated it : successfully for all the 5 items.

We will do a broader test.

ncw · June 16, 2022, 5:06pm

Ah, that is interesting - well spotted.

Try adding --local-nounc=true to your command line - that might help.

Vivien_VOILLOT · June 16, 2022, 8:45pm

I just tried the command with the addition you proposed, but sadly we got the same result.

We also made a larger test copying a top folder, it worked since any paths are now "new and well formed".
That might be a workaround, for what I supsect to be Windows specificities

ncw · June 17, 2022, 9:46am

Those paths are old style Windows 95 paths by the look of it. I don't really understand how they got there and why the OS doesn't just find the path.

Maybe @albertony or @asdffdsa might know as both know a lot more about Windows than me.

asdffdsa · June 17, 2022, 1:01pm

not sure how it exactly got there but i understand the logic.

D:\ONEDRI~2 is old DOS style 8.3 filenames
OneDrive is eight characters, the max for 8.3 style and there cannot be two filenames spelled the same way.
so windows has to rename the D:\OneDrive to D:\ONEDRI~2

what program created those files, maybe something very old, that might use 8.3?

for example,

>dir /x /AD
 Volume in drive C is EN08CDRIVE
 Volume Serial Number is D694-2C0A

 Directory of C:\

05/17/2022  10:02 AM    <DIR>                       $RECYCLE.BIN
04/20/2022  01:20 PM    <DIR>                       $Windows.~WS
06/14/2022  04:41 PM    <DIR>          $WINRE~1     $WinREAgent
02/23/2022  09:37 AM    <DIR>                       apps
05/30/2022  11:50 AM    <DIR>                       data
06/27/2021  03:08 PM    <JUNCTION>     DOCUME~1     Documents and Settings [C:\Users]
04/20/2022  01:55 PM    <DIR>                       ESD
05/17/2022  10:45 AM    <DIR>                       export
11/22/2021  12:33 PM    <DIR>                       inetpub
05/30/2022  10:30 AM    <DIR>                       Intel
05/17/2022  01:37 PM    <DIR>                       PerfLogs
06/14/2022  04:40 PM    <DIR>          PROGRA~1     Program Files
06/16/2022  01:34 PM    <DIR>          PROGRA~2     Program Files (x86)
06/16/2022  01:42 PM    <DIR>          PROGRA~3     ProgramData
03/12/2022  12:08 PM    <DIR>                       Recovery
06/16/2022  08:40 PM    <DIR>          SYSTEM~1     System Volume Information
11/30/2021  12:22 PM    <DIR>                       Users
10/21/2021  03:03 PM    <DIR>                       VeeamFLR
08/07/2021  05:45 PM    <DIR>          VEEAMR~1     VeeamRepository
05/25/2022  03:54 PM    <DIR>                       Windows

asdffdsa · June 17, 2022, 1:41pm

if this is a one time migration,
might try to create a list of files to be copied and feed that to rclone copy/sync --files-from

Vivien_VOILLOT · June 19, 2022, 8:29pm

That was our first crisis-solving solution and it worked

But the good ol'time path is a weird issue. As far as I understood, there are some pre-2000 files here. I guess they kept literally everything that ahs been created since they got computers !

Thanks everyone.
I hope that if someone got the same error s/he could find this topic.
Even if it seems really unlikely that happens.

asdffdsa · June 19, 2022, 10:19pm

some questions might be:
--- on what machine were the files originally created?
--- what are the timestamps of the trouble files?
--- what app created those files?

ncw · June 20, 2022, 9:22am

I would guess that Windows 95 is the problem. It used to create those sort of files I seem to remember. It wasn't until Windows XP that NTFS became standard and those 8.3 compatible names went away (all hazy memories - take with pinch of salt!).

I found an article which shows how to disable the short names

Vivien_VOILLOT · June 20, 2022, 11:20am

@afdffdsa
Most of the details are lost in time, I'm afraid.
There are (e.g.) .msg from 2016 so Outlook, I guess but which one ? Almost any MSOffice format with or without x, .pdf and so on...

@ncw
Most likely yes. It is the IT system of french municipalities, those are not known to be up to date regarding IT stuff ^^. It is not excluded they kept Win95 longer than expected.

system · July 20, 2022, 11:21am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.