I am trying to consolidate locally my backup on a Ubuntu box then sync it to ACD.
There is a directory on the Ubuntu box that is skipped from the syncing process (RClone sync /media/discoA bck:).
Really I am fighting with stupid folder… can not understand why it is skipped /media/discoA/Volume_1.
I made some test and really I can not figure out how to solve.
a) on the local Linux box there is /media/discoA that consolidate all the backups of local PC
b) so, inside /media/discoA I have /media/discoA/foleder1, /mdia/discoA/folder2 and so on
c) on the Amazon Cloud Drive ACD I have a folder called “bck” that is the encrypted target of RClone
d) I run the command "rclone sync /media/discoA bck: and everything works fine… alla the folder1, folder2, etc are created and synced correctly, BUT one of them /media/discoA/folderX no… this one is not created and so not synced even if I do not see errors…
e) running "rclone sync /media/discoA/folderX bck:folderX runs perfectly
I hope someone can help me in this troubleshootiong, please.
Thanks a lot.
Is it a symlink to somewhere? rclone doesn’t copy symlinks at the moment.
No, it is not a link… it is a regular folder. 
Can you run stat on it and post the result?
Volume_1 is the one I can not sync
FILM is correctly synced
stefano@MSG8:/media/discoA$ stat Volume_1/
File: Volume_1/' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 801h/2049d Inode: 66494465 Links: 3 Access: (0777/drwxrwxrwx) Uid: ( 1001/ UNKNOWN) Gid: ( 1001/ UNKNOWN) Access: 2017-01-19 22:40:04.000000000 +0100 Modify: 2013-12-25 00:34:36.000000000 +0100 Change: 2013-12-25 00:34:36.000000000 +0100 Birth: - stefano@MSG8:/media/discoA$ stat FILM/ File:FILM/’
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 801h/2049d Inode: 240828417 Links: 7
Access: (0777/drwxrwxrwx) Uid: ( 1001/ UNKNOWN) Gid: ( 1001/ UNKNOWN)
Access: 2017-01-19 21:50:42.000000000 +0100
Modify: 2016-10-26 01:02:01.000000000 +0200
Change: 2016-10-26 01:02:01.000000000 +0200
Birth: -
stefano@MSG8:/media/discoA$
Looks normal enough…
What if you do
rclone lsd Volume_1 --max-depth 100 -v
And the same with ls instead of lsd
Do you see directories and files?
Any warning messages?
stefano@MSG8:/media/discoA$ rclone lsd Volume_1 --max-depth 100 -v
2017/01/20 23:06:50 rclone: Version “v1.33” starting with parameters [“rclone” “lsd” “Volume_1” “–max-depth” “100” “-v”]
2017/01/20 23:06:50 Local file system at /media/discoA/Volume_1: Modify window is 1s
0 2013-12-27 18:44:45 0 share
0 2013-12-27 18:43:38 0 share/ffp
0 2013-12-23 18:28:44 0 share/.systemfile
2017/01/20 23:06:50 share/ffp/sbin/init: Can’t transfer non file/directory
0 2013-12-27 18:45:00 0 share/ffp/lib
0 2013-12-27 18:45:01 0 share/ffp/share
0 2013-12-27 18:43:41 0 share/ffp/bin
0 2013-12-27 18:43:43 0 share/ffp/sbin
0 2013-12-27 18:43:31 0 share/ffp/share/man
0 2013-12-27 18:43:29 0 share/ffp/share/man/man3
0 2013-12-27 18:43:30 0 share/ffp/share/man/man1
0 2013-12-27 18:43:32 0 share/ffp/share/man/man8
2017/01/20 23:06:50 share/ffp/sbin/mkfs.minix: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/sbin/setlogcons: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/sbin/nameif: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/sbin/sulogin: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/share/man/man8/rpc.sm-notify.8: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/share/man/man8/rpc.statd.8: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/share/man/man1/reset.1: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/share/man/man1/slogin.1: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/bin/basename: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/bin/ipcalc: Can’t transfer non file/directory
2017/01/20 23:06:50 share/ffp/bin/cal: Can’t transfer non file/directory
…
…
…
and continues with these messages
stefano@MSG8:/media/discoA$ rclone ls Volume_1 --max-depth 100 -v
2017/01/20 23:07:56 rclone: Version “v1.33” starting with parameters [“rclone” “ls” “Volume_1” “–max-depth” “100” “-v”]
2017/01/20 23:07:56 Local file system at /media/discoA/Volume_1: Modify window is 1s
1778 share/fun_plug
119414 share/usb-storage.ko
181440 share/fdisk
435 share/VOLUME_1.txt
715 share/DNS-323.commands.txt
411750 share/ffp.log
242 share/rsyncd.conf
2017/01/20 23:07:56 share/ffp/sbin/init: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/share/man/man8/rpc.sm-notify.8: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/share/man/man8/rpc.statd.8: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/mkfs.minix: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/setlogcons: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/nameif: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/sulogin: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/watchdog: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/inetd: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/route: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/mkswap: Can’t transfer non file/directory
2017/01/20 23:07:56 share/ffp/sbin/pivot_root: Can’t transfer non file/directory
…
…
…
and so on
OK that shows rclone can see some directories and files in Volume_1.
For instance it can see share/fun_plug - does that get transferred?
The files which say share/ffp/sbin/init: Can't transfer non file/directory - could you stat one of those? rclone won’t transfer those files - stat should show us why.
The directory /media/discoA/Volume_1 is not transferred at all and so nothing inside it is transferred.
The file DNS-323.commands.txt does exist and is perfectly readable and is not transferred.
stefano@MSG8:/media/discoA/Volume_1/share$ stat DNS-323.commands.txt
File: `DNS-323.commands.txt’
Size: 715 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 66494554 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-01-20 23:04:49.000000000 +0100
Modify: 2013-10-17 00:49:58.000000000 +0200
Change: 2013-12-25 00:34:38.000000000 +0100
Birth: -
init is a link and of course is not transferred
stefano@MSG8:/media/discoA/Volume_1/share/ffp/sbin$ stat init
File: init' ->…/bin/busybox’
Size: 14 Blocks: 0 IO Block: 4096 symbolic link
Device: 801h/2049d Inode: 243957781 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-01-20 23:04:23.000000000 +0100
Modify: 2009-03-06 04:50:40.000000000 +0100
Change: 2013-12-25 16:23:37.000000000 +0100
Birth: -
My question is why regular files (and not links) are not trasferred.
Is the directory which won’t transfer a volume mounted from elsewhere?
The -x flag would stop it being transferred. Are you using that?
If not I wonder if there is a bug in that code…
no.
/media/discoA is a 4 TB HDD in an HP Proliant \ Ubuntu.
It is shared with SAMBA and contains some folders mounted from client PC that write their backup into it, then the entire disk is synced and encrypted into the cloud (Amazon)
RClone sync /media/discoA bck:
No, I do not use -x but anyway everything is local and not mounted from elsewhere.
Many thanks for your support, I appreciate a lot.
P.S.: it is not so much important to sync this folder… I can survive without… but I do not want to think that there are similar problems elsewhere.
If I try to sync the content of this folder, it works (rclone sync /media/discoA/Volume_1 bck:)
If I copy this folder elsewhere in the root of the HDD, even the copy is not synced
if i copy this folder into another folder, it is synced
It must be something about that folder - I’m puzzled as to what though!
What happens if you do
rclone lsd -v /media/discoA
Do you see Volume_1 in the listing? Any interesting debug messages when you do that?
Of course, I can see the Volume_1 !
You can see the lost+found folder, well this is the only folder that produces errors when synced to the cloud (permission denied).
stefano@MSG8:/$ rclone lsd -v /media/discoA
2017/01/21 22:46:25 rclone: Version “v1.33” starting with parameters [“rclone” “lsd” “-v” “/media/discoA”]
2017/01/21 22:46:25 Local file system at /media/discoA: Modify window is 1s
0 2016-10-15 22:13:29 0 FOTO
0 2017-01-17 17:10:13 0 Archivio
0 2016-10-26 01:02:01 0 FILM
0 2017-01-19 22:40:03 0 nuova1
0 2016-11-04 00:00:38 0 MUSICA
0 2014-05-14 00:12:28 0 Varie
0 2013-12-25 00:34:36 0 Volume_1
0 2013-12-10 00:39:41 0 lost+found
0 2017-01-19 01:04:17 0 CARTONI
0 2015-11-15 22:21:46 0 media
2017/01/21 22:46:25
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 0s
2017/01/21 22:46:25 Go routines at exit 1
Ah, ha! The permission denied on lost+found will cause the directory listing to stop and it won’t get to volume 1.
Add an --exclude “/lost+found/**” and all should be well.
OK!
Now it works!!! I excluded the lost+found folder and rclone is now able to sync what was missing before.
Why this behaviour? lost+found is in the root and all the other folders in the root are synced properly, but just /Volume_1
Why just /Volume_1 is not synced?
Why rclone can not simply skip lost+found and go on?
Now the sync an check performances are much better… just 11 minutes to sync or to check about 3 TB data
many many thanks for your support.
Some updates on this subject.
Excluding “/media/discoA/lost+found” folder from the sync process, not only allowed me to sync the “/media/disco/Volume_1”, but now Rclone is syncing MANY MANY more files to the cloud located everywhere on “media/discoA”.
Great
rclone does lots of stuff asynchronously so when things happen is a little random.
Doing it with a flag would probably be a good idea. In general you don’t want it though as if you were syncing a directory, then it became unreadable for some reason, it would get deleted on the remote.
Great.
I understand what you say, but I become aware of this issue just because this missed directrory was in the root level!
as I told, many other files aroud were missed and now thay are synced excluding the lost+found.
I personally vote for a flag to skip what rclone is not able to sync\copy\check\etc. maybe with outputting some warnings.
anyway, RClone is a very great piece of software. before rclone I evaluated other commercial tools and they have been removed from my machines!
regards!
Do you want to make an issue about this on github?