Files randomly not migrated since path is not found (Local to GDrive)

What is the problem you are having with rclone?

Hello,
I am migrating from local to several Google Shared Drives.
We are talking about 500 to 600Go of data, and some files, irregardless to source emplacement or extension or else return an error that I can't really set here since it is in french, so I will try to translate at best :
Failed to copy: failed to open source object: the specified path is not found (?)
(Failed to copy: failed to open source object: Le chemin d’accès spécifié est introuvable.)
It shouldn't be related to the remotes or the command itself since it impacts only several files amongst a vast majority which is expectedly migrated.

What I checked :

  • remigrating returns the same result
  • remigrated specifically the folder of those files returns the same result
  • the path of the source folder is correct
  • rclone ls [source] find the files that are not migrated
  • the files which can't be migrated are visible and usable in the source directory by my user account
  • the files' paths' numbers of characters are not "that" long (I am not sure there is a limit of characters for loacl sources)
  • the files' path's numbers of sublevels are not "that" long (same uncertainty here)
  • moving the files form the source to the destination with a drag and drop works.

Since I have around 12k files to move, I really don't want to do it manually ^^

Would you happen to know how to solve the kind of issue I am facing here ?

Thanks :slight_smile:

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: Microsoft Windows Server 2012 R2 Standard (64 bit)
  • os/kernel: 6.3.9600.19505 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local as source
GDrive at destination (~50 Shared Drives)

The command you were trying to run

As an example, amongst the 50-ish I made

rclone copyto local:"D:\OneDrive - Mairie des Allues\Services\Domaine skiable" Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-3evague.csv --log-level INFO -P --transfers 8

The rclone config contents with secrets removed.

[local]
type = local
nounc = true
[Mbl-05.6-DomaineSkiable]
type = drive
scope = drive
service_account_file = C:\RClone\test.json
team_drive = 0ACc9eDDCHugXUk9PVA
root_folder_id =

A log from the command with the -vv flag

https://docs.google.com/spreadsheets/d/1EeO0FEThDc4J0ySNwqf7Mxh5_ghrOXpcLgnTM-QaQwE/edit?usp=sharing

There are like 18k lines in it :expressionless:

There's been a few fixes related to that in the beta.

Install (rclone.org)

Can you test with the beta version and validate that solves the issue?

Thanks.
After setting on a new folder, I've tried to do a rclone config for both :

  • v1.59.0-beta.6215.0279bf3ab
  • v1.59.0-beta.6211.ec117593f
    and each time I get an access denied !

Is there a way to "update" to a beta an existing version ?

You can:

rclone selfupdate --beta

From where you normally have the binary.

Can you make a list of the file names which fail

I picked out two file names at random, first a failing one, then a good one

BAD: CONVENTION TRIPARTITE/CONVENTIONS TRIPARTITES/ANCIENNES CONVENTIONS - écoles + 6 moniteurs/CONTENTIEUX 2012/CONTENTIEUX CONVENTIONS 2012/lettre du fin Avril - commentaires sur les pistes de Meribel Parallel lines 11 07 12.msg
OK:  CONVENTION TRIPARTITE/CONVENTIONS TRIPARTITES/ANCIENNES CONVENTIONS - écoles + 6 moniteurs/CONTENTIEUX 2012/CONTENTIEUX CONVENTIONS 2012/ANALYSE ADAMAS.pdf

I don't see anything obviously the matter with them.

While file system are you copying from?

Can you copy that file if you CD into that directory first?

Trying the beta is a good idea.

Thanks both of you.

I am now in the latest beta version.

I retried a command targeting a folder which contains only not-migrated files (all of them are .jpg) and the result is the same :confused:
The command being :
rclone copyto local:"D:\OneDrive - Mairie des Allues\Services\Domaine skiable\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE" Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-test.csv --log-level INFO -P --transfers 8
And the log file is :

Here is the list of all the files which were not migrated :confused: Log - 3rd wave - Google Sheets
You will find in it around 1k3 lines in yellow : those are other aberrations : there is no error given, and the path has been brutally cut !

@ncw if I CD ?
The ls command is effective :
C:\RClone>rclone ls local:"D:\OneDrive - Mairie des Allues\Services\Domaine skia
ble\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE"
188595 Alexis PINTURAULT (FRA).jpg
318333 Andrew WEIBRECHT (USA).jpg
74034 Christof INNERHOFER (ITA).jpg
28138 Didier DEFAGO (SUI).jpg
60617 Dustin COOK (CAN).jpg
1533977 Georg STREITBERGER (AUT).jpg
48472 Johan CLAREY (FRA).jpg
724182 Matteo MARSAGLIA (ITA).jpg
62729 Max FRANZ (AUT).jpg
161466 Patrick KUENG (SUI).jpg
90368 Peter FILL (ITA).jpg
126334 Romed BAUMANN (AUT).jpg
20992 Thumbs.db
95747 Vincent KRIECHMAYR (AUT).jpg

Edit :
can a comma in the files' paths cut the command ?
It seems the ones which are cut contain a comma at this precise moment !

Edit 2 :slight_smile: nah that was a stupid take : some files of those folders have been migrated. But the error messages of those which weren't are cut...

How about

cd "D:\OneDrive - Mairie des Allues\Services\Domaine skiable\2023\Championnats du monde 2023\FINALE COUPE DU MONDE 2015\PARC OLYMPIQUE\TIRAGE AU SORT DES DOSSARDS\TIRAGE AU SORT DES DOSSARDS_VISUELS ATHLETES\MERCREDI 18\SUPER G\HOMMES\2nd GROUPE"

Then

rclone copyto . Mbl-05.6-DomaineSkiable: --ignore-size --log-file C:\RClone\logs-test.csv --log-level INFO -P --transfers 8

Does that work?

Also why are you using --ignore-size - you shouldn't need that - maybe that is part of the problem?

Ah ok, thanks.

I haven't done it yet but I think me and my colleague found what happens !
On the same folder, of 5 items, of which only 1 has been migrated, when we go on the files' setting here are the path :

  • migrated file :
    D:\OneDrive - Mairie des Allues\Services\Services techniques\Bâtiments\Accessibilité PMR\BATIMENTS\Parc Olympique\2017_Mise aux normes accessibilité sanitaires PO\Marché travaux\03_Exécution\Lot 4_Revolta Blaudeau_Peinture placo
  • not migrated file :
    D:\ONEDRI~2\Services\SERVIC~1\BTIMEN~1\ACCESS~1\BATIME~1\PARCOL~1\2017_M~1\MARCHT~1\03_EXC~1\LOT4_R~1
    So, I don't know how users can manage to get this outcome while working on their files, but it will explain that RClone can't find the "true" path of those files.

We then copied the folder at root, and migrated it : successfully for all the 5 items.

We will do a broader test.

Ah, that is interesting - well spotted.

Try adding --local-nounc=true to your command line - that might help.

I just tried the command with the addition you proposed, but sadly we got the same result.

We also made a larger test copying a top folder, it worked since any paths are now "new and well formed".
That might be a workaround, for what I supsect to be Windows specificities :confused:

Those paths are old style Windows 95 paths by the look of it. I don't really understand how they got there and why the OS doesn't just find the path.

Maybe @albertony or @asdffdsa might know as both know a lot more about Windows than me.

not sure how it exactly got there but i understand the logic.

D:\ONEDRI~2 is old DOS style 8.3 filenames
OneDrive is eight characters, the max for 8.3 style and there cannot be two filenames spelled the same way.
so windows has to rename the D:\OneDrive to D:\ONEDRI~2

what program created those files, maybe something very old, that might use 8.3?

for example,

>dir /x /AD
 Volume in drive C is EN08CDRIVE
 Volume Serial Number is D694-2C0A

 Directory of C:\

05/17/2022  10:02 AM    <DIR>                       $RECYCLE.BIN
04/20/2022  01:20 PM    <DIR>                       $Windows.~WS
06/14/2022  04:41 PM    <DIR>          $WINRE~1     $WinREAgent
02/23/2022  09:37 AM    <DIR>                       apps
05/30/2022  11:50 AM    <DIR>                       data
06/27/2021  03:08 PM    <JUNCTION>     DOCUME~1     Documents and Settings [C:\Users]
04/20/2022  01:55 PM    <DIR>                       ESD
05/17/2022  10:45 AM    <DIR>                       export
11/22/2021  12:33 PM    <DIR>                       inetpub
05/30/2022  10:30 AM    <DIR>                       Intel
05/17/2022  01:37 PM    <DIR>                       PerfLogs
06/14/2022  04:40 PM    <DIR>          PROGRA~1     Program Files
06/16/2022  01:34 PM    <DIR>          PROGRA~2     Program Files (x86)
06/16/2022  01:42 PM    <DIR>          PROGRA~3     ProgramData
03/12/2022  12:08 PM    <DIR>                       Recovery
06/16/2022  08:40 PM    <DIR>          SYSTEM~1     System Volume Information
11/30/2021  12:22 PM    <DIR>                       Users
10/21/2021  03:03 PM    <DIR>                       VeeamFLR
08/07/2021  05:45 PM    <DIR>          VEEAMR~1     VeeamRepository
05/25/2022  03:54 PM    <DIR>                       Windows

if this is a one time migration,
might try to create a list of files to be copied and feed that to rclone copy/sync --files-from

That was our first crisis-solving solution and it worked :slight_smile:

But the good ol'time path is a weird issue. As far as I understood, there are some pre-2000 files here. I guess they kept literally everything that ahs been created since they got computers !

Thanks everyone.
I hope that if someone got the same error s/he could find this topic.
Even if it seems really unlikely that happens.

some questions might be:
--- on what machine were the files originally created?
--- what are the timestamps of the trouble files?
--- what app created those files?

I would guess that Windows 95 is the problem. It used to create those sort of files I seem to remember. It wasn't until Windows XP that NTFS became standard and those 8.3 compatible names went away (all hazy memories - take with pinch of salt!).

I found an article which shows how to disable the short names

@afdffdsa
Most of the details are lost in time, I'm afraid.
There are (e.g.) .msg from 2016 so Outlook, I guess but which one ? Almost any MSOffice format with or without x, .pdf and so on...

@ncw
Most likely yes. It is the IT system of french municipalities, those are not known to be up to date regarding IT stuff ^^. It is not excluded they kept Win95 longer than expected.