Google Drive (Team Drive) JPEG Corruption on rclone copy

Ben_Wilson · January 15, 2021, 5:44pm

What is the problem you are having with rclone?

JPEG corruption on copy from Linux to Google Drive using rclone copy.

We are moving from a local NAS to Google Drive (Team Drive), using rclone copy. All files transfer successfully, but many of our JPEGs are corrupted in the process.

Original "Good" JPEG: https://drive.google.com/file/d/1xtIm9JRoto7WpzLgQvnwZud5UnXZvOSU/view?usp=sharing

Resulting "Bad" JPEG in Google Drive:

I have researched the Google Photos drive issues circa 2019, and it doesn't appear to me that this is the same issue. Transfers are successful, but resulting files are corrupted.

What is your rclone version (output from `rclone version`)

rclone v1.53.3

os/arch: linux/amd64
go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

(see above for arch/bits)

Which cloud storage system are you using? (eg Google Drive)

Google Drive (Team Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -vv -c --exclude=".DS_Store" /mnt/VIA/\!\ clients\ \(ONLY\ ARCHIVED\)/\!\ active\ \(DEPRECATED\!\ USE\ GOOGLE\ SHARED\ DRIVE\!\)/VIA\ Studio/_Library/Photography/2020.01\ Holiday\ Socks/_N8A7290.JPG  activeclients:VIA\ Studio/_Library/Photography/2020.01\ Holiday\ Socks/

The rclone config contents with secrets removed.

[activeclients]
type = drive
scope = drive
token = REDACTED
team_drive = 0AKcsC_RLAzI8Uk9PVA
root_folder_id =

A log from the command with the `-vv` flag

2021/01/15 12:42:55 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "-vv" "-c" "--exclude=.DS_Store" "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks/_N8A7290.JPG" "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks/"]
2021/01/15 12:42:55 DEBUG : Creating backend with remote "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks/_N8A7290.JPG"
2021/01/15 12:42:55 DEBUG : Using config file from "/home/viaadmin/.config/rclone/rclone.conf"
2021/01/15 12:42:55 DEBUG : fs cache: adding new entry for parent of "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks/_N8A7290.JPG", "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks"
2021/01/15 12:42:55 DEBUG : Creating backend with remote "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks/"
2021/01/15 12:42:58 DEBUG : fs cache: renaming cache item "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks/" to be canonical "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks"
2021/01/15 12:42:58 DEBUG : _N8A7290.JPG: MD5 = c9701e9e08823c9c97566804eb58be6c (Local file system at /mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks)
2021/01/15 12:42:58 DEBUG : _N8A7290.JPG: MD5 = 1165ef0e9a68b9642e4ec432593dce2f (Google drive root 'VIA Studio/_Library/Photography/2020.01 Holiday Socks')
2021/01/15 12:42:58 DEBUG : _N8A7290.JPG: MD5 differ
2021/01/15 12:43:05 DEBUG : _N8A7290.JPG: MD5 = 2dfb5e9d8e74b379842c13dab8636aa0 OK
2021/01/15 12:43:05 INFO  : _N8A7290.JPG: Copied (replaced existing)
2021/01/15 12:43:05 INFO  : 
Transferred:        6.770M / 6.770 MBytes, 100%, 963.970 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         9.8s

Ben_Wilson · January 15, 2021, 5:46pm

Oh, and FWIW, the corruption differs each time, as near as I can tell. (I'm not a JPEG file format expert)

Animosity022 · January 15, 2021, 5:59pm

Do you have a source file and a file on the drive that is different MD5SUMs?

The log you shared doesn't show much other than a file is changed so it updated it.

Ben_Wilson · January 15, 2021, 7:26pm

Yes - if I am understanding you correctly?

The Source file is the GOOD file I linked above. The "file on the drive" is BAD file I linked above. They should have the same MD5, but as the log shows, they do not.

Animosity022 · January 15, 2021, 7:30pm

Right, but that just tells me the files are different but not what caused them to be different. Sure someone didn't upload a different file with the same name?

(I'm not saying you are wrong btw but no way to reproduce or show what changed so far)

Ben_Wilson · January 15, 2021, 7:42pm

Oh yes, absolutely. I've tried this with multiple files, both individually and as an entire directory. This is not the result of just a single manual overwrite.

What I'm giving you above is the source file and the resulting file from the rclone command snippet I've provided.

EDIT: I've also tried this with every flag and suggestion I've found in the docs and forums for related issues and always have the same outcome. I'm at a loss!

Ben_Wilson · January 15, 2021, 7:57pm

I've also just tested using a renamed copy of the "Good" file, pushing to the GDrive as a new file. Same result.

Animosity022 · January 15, 2021, 8:06pm

Does someone have a Google Drive app? That does optimization of photos and could they be uploading?

I upload the same file over and over again and it never changes:

felix@flux:/mnt/c/users/earlt/Downloads$ rclone copy testimage.jpg GD: -vv
2021/01/15 15:06:29 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "testimage.jpg" "GD:" "-vv"]
2021/01/15 15:06:29 DEBUG : Creating backend with remote "testimage.jpg"
2021/01/15 15:06:29 DEBUG : Using config file from "/home/felix/.config/rclone/rclone.conf"
2021/01/15 15:06:29 DEBUG : fs cache: adding new entry for parent of "testimage.jpg", "/mnt/c/users/earlt/Downloads"
2021/01/15 15:06:29 DEBUG : Creating backend with remote "GD:"
2021/01/15 15:06:29 DEBUG : testimage.jpg: Size and modification time the same (differ by -176.8µs, within tolerance 1ms)
2021/01/15 15:06:29 DEBUG : testimage.jpg: Unchanged skipping
2021/01/15 15:06:29 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.3s

2021/01/15 15:06:29 DEBUG : 4 go routines active
felix@flux:/mnt/c/users/earlt/Downloads$ rclone copy testimage.jpg GD: -vv
2021/01/15 15:06:30 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "testimage.jpg" "GD:" "-vv"]
2021/01/15 15:06:30 DEBUG : Creating backend with remote "testimage.jpg"
2021/01/15 15:06:30 DEBUG : Using config file from "/home/felix/.config/rclone/rclone.conf"
2021/01/15 15:06:30 DEBUG : fs cache: adding new entry for parent of "testimage.jpg", "/mnt/c/users/earlt/Downloads"
2021/01/15 15:06:30 DEBUG : Creating backend with remote "GD:"
2021/01/15 15:06:31 DEBUG : testimage.jpg: Size and modification time the same (differ by -176.8µs, within tolerance 1ms)
2021/01/15 15:06:31 DEBUG : testimage.jpg: Unchanged skipping
2021/01/15 15:06:31 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.9s

2021/01/15 15:06:31 DEBUG : 4 go routines active

Ben_Wilson · January 15, 2021, 8:31pm

I will note that I started using -c because the size and modification time was always the same. I do see the same things you do in your log, without -c.

Animosity022 · January 15, 2021, 8:37pm

If you upload it, rclone md5sum the file, does it change if you keep checking the md5sum after the upload?

Or are these more random and not reproducible?

The checksum never changes on me:

felix@flux:/mnt/c/users/earlt/Downloads$ rclone copy testimage.jpg GD: -vv -c
2021/01/15 15:35:32 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "testimage.jpg" "GD:" "-vv" "-c"]
2021/01/15 15:35:32 DEBUG : Creating backend with remote "testimage.jpg"
2021/01/15 15:35:32 DEBUG : Using config file from "/home/felix/.config/rclone/rclone.conf"
2021/01/15 15:35:32 DEBUG : fs cache: adding new entry for parent of "testimage.jpg", "/mnt/c/users/earlt/Downloads"
2021/01/15 15:35:32 DEBUG : Creating backend with remote "GD:"
2021/01/15 15:35:32 DEBUG : testimage.jpg: MD5 = b9f2af5bc3e885b0dd03c1b26388719c OK
2021/01/15 15:35:32 DEBUG : testimage.jpg: Size and MD5 of src and dst objects identical
2021/01/15 15:35:32 DEBUG : testimage.jpg: Unchanged skipping
2021/01/15 15:35:32 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.3s

2021/01/15 15:35:32 DEBUG : 4 go routines active
felix@flux:/mnt/c/users/earlt/Downloads$ rclone copy testimage.jpg GD: -vv -c
2021/01/15 15:35:33 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "testimage.jpg" "GD:" "-vv" "-c"]
2021/01/15 15:35:33 DEBUG : Creating backend with remote "testimage.jpg"
2021/01/15 15:35:33 DEBUG : Using config file from "/home/felix/.config/rclone/rclone.conf"
2021/01/15 15:35:33 DEBUG : fs cache: adding new entry for parent of "testimage.jpg", "/mnt/c/users/earlt/Downloads"
2021/01/15 15:35:33 DEBUG : Creating backend with remote "GD:"
2021/01/15 15:35:34 DEBUG : testimage.jpg: MD5 = b9f2af5bc3e885b0dd03c1b26388719c OK
2021/01/15 15:35:34 DEBUG : testimage.jpg: Size and MD5 of src and dst objects identical
2021/01/15 15:35:34 DEBUG : testimage.jpg: Unchanged skipping
2021/01/15 15:35:34 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.3s

2021/01/15 15:35:34 DEBUG : 4 go routines active

calisro · January 15, 2021, 9:45pm

I'm fairly certain Google will modify these files EXIF data. The magic behind the scenes and the fact taht you can't redownload them in full size from the API makes me not use this as a photo backup.

EDIT: Sorry. I read this quickly as 'google photos'.

Ben_Wilson · January 15, 2021, 9:47pm

UPDATE:

I believe there is something happening on the local end.

the local files are actually mounted from a CIFS drive.
I did an md5sum on the file on the CIFS mount, and it changes each time (wtf?)
I copied the file to be truly local, and the md5sum there is stable, and running my rclone copy command resulted in the expected outcome (md5 sums are the same on both sides)

So... something is up with my mounted source.

(However, that doesn't exactly explain to me why the files end up as they do in Google Drive... but I'll work on my mount issue and get back to you.)

Animosity022 · January 15, 2021, 9:53pm

That would be freaky! I'd love to hear what causes that as that seems super bizarre to me. It's been quite a number of years since I've used CIFS for anything though.

Ben_Wilson · January 15, 2021, 10:18pm

Yes, bizarre to me as well. We are migrating from an old NAS, and using a linux machine as the mount point.

At this point, I'm copying all files from the NAS to the linux machine locally, and will attempt to recreate this issue then.

(meanwhile)
I did try the following, but it still failed to solve the md5sum issue:

mounting the CIFS share readonly (md5sum changed each time locally)
mounting the share via NFS, readonly (md5sum stayed the same locally...until rclone accessed it)

I'm not an expert, but I suspect some sort of caching / file locking thing is happening there... but that's not rclone's problem exactly

ncw · January 17, 2021, 5:14pm

I did a binary diff of the files

$ diff -u _N8A7290.JPG.hex _N8A7290\ \(1\).JPG.hex 
--- _N8A7290.JPG.hex	2021-01-17 17:07:10.603886192 +0000
+++ "_N8A7290 (1).JPG.hex"	2021-01-17 17:07:28.000012503 +0000
@@ -16411,9 +16411,9 @@
 00041ac0  7c cc 92 71 98 d8 0e 32  3f ad 13 4a 10 6e 2b 63  ||..q...2?..J.n+c|
 00041ad0  a9 fe ee 2e 68 c3 bf d2  12 4c b3 7c e3 68 27 6e  |....h....L.|.h'n|
 00041ae0  4e 3d aa 9a 68 aa ea 65  91 23 53 9c ec 8d 01 56  |N=..h..e.#S....V|
-00041af0  5c f7 c6 2b 4a 78 c5 1a  5b bb 9c 4b 1d 6d 6e 59  |\..+Jx..[..K.mnY|
-00041b00  9f c3 30 6c 95 42 a8 93  ca fd df 96 46 d0 3a fb  |..0l.B......F.:.|
-00041b10  f3 59 ef e1 44 4c 2c 31  07 0a 3c d4 69 09 19 f5  |.Y..DL,1..<.i...|
+00041af0  5c f7 c6 2b 4a 78 ff 00  ff 00 00 00 00 00 00 00  |\..+Jx..........|
+00041b00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00041b10  00 00 00 00 00 00 2c 31  07 0a 3c d4 69 09 19 f5  |......,1..<.i...|
 00041b20  fc 3a d4 4b 1d ec e1 cd  29 1a ac ce aa 56 6f 43  |.:.K....)....VoC|
 00041b30  4a d7 41 b4 67 2b 72 aa  23 91 43 11 1e 42 a9 f4  |J.A.g+r.#.C..B..|
 00041b40  ae 86 df c3 76 f3 1f 9a  32 5b 0b 1a aa 7d d0 83  |....v...2[...}..|
@@ -39393,9 +39393,9 @@
 0009b720  71 b7 df e8 68 eb 40 5d  5e c2 0f d7 db d2 93 9f  |q...h.@]^.......|
 0009b730  d7 f4 a0 4d 5a d6 11 8e  09 fa 70 7d 29 aa b8 39  |...MZ.....p})..9|
 0009b740  f5 5e 68 1a 69 ec 30 8f  4e 78 a4 fe 54 03 5b 8d  |.^h.i.0.Nx..T.[.|
-0009b750  21 71 d3 a7 1c 54 45 48  c7 27 1c f0 7b 13 49 c5  |!q...TEH.'..{.I.|
-0009b760  3d d1 29 bb 6c 46 50 76  eb 8e b5 1b 23 1e e7 ea  |=.).lFPv....#...|
-0009b770  2b 86 b5 24 db ba 35 4e  4a 29 dc af b0 92 01 1f  |+..$..5NJ)......|
+0009b750  21 71 ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |!q..............|
+0009b760  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0009b770  00 00 b5 24 db ba 35 4e  4a 29 dc af b0 92 01 1f  |...$..5NJ)......|
 0009b780  9d 41 22 2f f7 7a 83 9e  38 35 e6 57 a0 a5 75 63  |.A"/.z..85.W..uc|
 0009b790  68 54 b6 c5 56 8b 27 a6  33 9f ca a0 f2 41 04 11  |hT..V.'.3....A..|
 0009b7a0  83 b8 82 4f a5 7c f6 37  0a 92 6d a3 b6 95 4f 32  |...O.|.7..m...O2|
@@ -68368,9 +68368,9 @@
 0010ca10  fd 1f f6 7e f1 63 49 0e  ed 36 f1 0a ed 40 a6 12  |...~.cI..6...@..|
 0010ca20  54 9e e7 80 70 2b e9 f0  fc 3b 88 9d a3 34 7c de  |T...p+...;...4|.|
 0010ca30  63 c4 f9 7e 16 ed 55 4d  9e bf a2 fe cc fe 2b ba  |c..~..UM......+.|
-0010ca40  6f b3 be 97 74 37 b9 58  a4 86 3c 02 be a4 9f 4e  |o...t7.X..<....N|
-0010ca50  bd 2b d3 34 bf d8 d7 5d  bc 92 3b 87 b7 98 86 6f  |.+.4...]..;....o|
-0010ca60  2c 14 6f 94 11 fe f0 c8  cd 7d 0e 1b 85 69 c2 ce  |,.o......}...i..|
+0010ca40  6f b3 be 97 74 37 b9 58  a4 86 3c 02 be a4 ff 00  |o...t7.X..<.....|
+0010ca50  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0010ca60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 c2 ce  |................|
 0010ca70  4a e7 c6 66 be 28 52 a1  0e 4c 3c 8f 47 d3 3f 62  |J..f.(R..L<.G.?b|
 0010ca80  0b 97 f3 0e a6 1f cb 38  67 81 10 31 1e 9f 36 30  |.......8g..1..60|
 0010ca90  7e 95 e8 da 17 ec 59 a0  45 29 f3 ad 25 52 c7 0c  |~.....Y.E)..%R..|
@@ -91355,9 +91355,9 @@
 001666c0  19 39 39 eb 51 90 a0 11  cf 18 c9 eb 5f 8f 54 a2  |.99.Q......._.T.|
 001666d0  f0 f2 78 79 c7 53 d1 4e  eb 9a e3 7c b2 c7 3b b8  |..xy.S.N...|..;.|
 001666e0  2b 82 29 9b 57 81 96 38  23 a8 ae 29 e1 a4 dc 9b  |+.).W..8#..)....|
-001666f0  e8 5a 52 d3 51 72 09 38  ed 9f 5e 29 9d 01 27 e6  |.ZR.Qr.8..^)..'.|
-00166700  fa e6 95 3a d1 82 56 5a  75 2a 50 8c b5 b8 e2 01  |...:..VZu*P.....|
-00166710  c9 c0 20 fa 75 a6 e3 03  ab 11 c6 00 1d eb 8e b4  |.. .u...........|
+001666f0  e8 5a 52 d3 51 72 09 38  ed 9f 5e 29 9d 01 ff 00  |.ZR.Qr.8..^)....|
+00166700  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00166710  00 00 00 00 00 00 00 00  00 00 00 00 00 00 8e b4  |................|
 00166720  a5 36 f5 34 8c 54 76 1f  50 91 8c 82 3b 8d bc f5  |.6.4.Tv.P...;...|
 00166730  15 bd 26 a1 69 37 bf 41  35 66 dd c5 3b 72 46 31  |..&.i7.A5f..;rF1|
 00166740  d3 d3 ad 29 5c 74 c1 38  c7 cd de a6 9d 24 e7 29  |...)\t.8.....$.)|
@@ -114161,9 +114161,9 @@
 001bf820  38 38 19 ea 38 18 03 a1  a0 96 ec 9b b1 2a 82 4f  |88..8........*.O|
 001bf830  07 04 7c dc d4 bc 10 48  03 2d f3 72 31 cd 68 9e  |..|....H.-.r1.h.|
 001bf840  9a 33 20 1d 32 70 29 9d  77 63 6e 73 d7 39 c5 5c  |.3 .2p).wcns.9.\|
-001bf850  b5 56 1b 93 6a c2 05 ce  49 39 eb f8 1a 93 38 c0  |.V..j...I9....8.|
-001bf860  3c f4 15 9b d6 c9 a1 2d  d5 87 6e c1 24 fc c4 e0  |<......-..n.$...|
-001bf870  1c e4 9c 51 9f d3 35 2d  59 d8 7c ae f6 13 ff 00  |...Q..5-Y.|.....|
+001bf850  b5 56 1b 93 6a c2 05 ce  49 39 eb f8 1a 93 ff 00  |.V..j...I9......|
+001bf860  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+001bf870  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff 00  |................|
 001bf880  d7 4c 39 1f ef 12 79 ef  8a 13 b5 f4 29 37 ca c4  |.L9...y.....)7..|
 001bf890  0d db 00 7a e4 e3 9a 5c  ee e7 38 2b db da 8b 69  |...z...\..8+...i|
 001bf8a0  72 5c 9b 00 dc 92 71 8f  d6 9d 9c 63 3c e7 00 67  |r\....q....c<..g|
@@ -137053,9 +137053,9 @@
 00218ee0  1c f3 81 d0 e3 18 06 a3  50 a4 93 9a 2c f7 b1 49  |........P...,..I|
 00218ef0  29 3b dc b1 f2 93 c7 04  12 78 a0 06 e3 18 20 02  |);.......x.... .|
 00218f00  4e 4e 0e 69 04 9c 93 d0  67 24 90 7f fd 46 90 e0  |NN.i....g$...F..|
-00218f10  72 0f 4c fd 28 1c b6 b1  0b 4a 88 ac cc c0 6d 19  |r.L.(....J....m.|
-00218f20  c6 46 4d 40 fa 95 b4 38  56 9c 03 c6 13 6f ca 7f  |.FM@...8V....o..|
-00218f30  1a 05 18 6e da 2d c5 75  04 ce f1 a4 f0 bc a8 85  |...n.-.u........|
+00218f10  72 0f ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |r...............|
+00218f20  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00218f30  00 00 18 6e da 2d c5 75  04 ce f1 a4 f0 bc a8 85  |...n.-.u........|
 00218f40  dc 21 38 07 d2 94 b6 ee  70 07 f5 a6 9d ae 29 a6  |.!8.....p.....).|
 00218f50  be 14 4a 0f 51 c0 e9 8e  72 69 ac 47 2b 8e 77 1f  |..J.Q...ri.G+.w.|
 00218f60  9b 34 85 f0 df cc 15 63  c7 53 bb 81 48 09 0c 07  |.4.....c.S..H...|
@@ -160040,9 +160040,9 @@
 00272b90  9e 07 3c e0 7b 52 7d ee  48 c6 0f 42 68 2f 4d 05  |..<.{R}.H..Bh/M.|
 00272ba0  66 5c 1c 00 38 e1 46 30  4d 31 7e 62 49 ed 81 8a  |f\..8.F0M1~bI...|
 00272bb0  04 38 c6 b8 c9 c7 3e 9d  85 35 8a 2e 44 79 23 a7  |.8....>..5..Dy#.|
-00272bc0  23 07 14 ee ed 62 79 57  37 31 11 8d 9f 95 e9 f5  |#....byW71......|
-00272bd0  a4 0b 81 d0 67 a8 a4 3d  6e 44 5d b3 93 92 40 f4  |....g..=nD]...@.|
-00272be0  e2 9d bc f0 48 00 e3 18  c0 c6 28 12 77 6c 7c 65  |....H.....(.wl|e|
+00272bc0  23 07 ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |#...............|
+00272bd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00272be0  00 00 bc f0 48 00 e3 18  c0 c6 28 12 77 6c 7c 65  |....H.....(.wl|e|
 00272bf0  41 39 c9 e0 e3 eb 52 99  76 e7 b1 c1 03 14 14 43  |A9....R.v......C|
 00272c00  b8 9c 9c e3 bf 3c 9a 72  ee 38 e3 04 f6 ef 40 9c  |.....<.r.8....@.|
 00272c10  92 dc 3e bd b3 9e dc 53  37 29 c9 e7 24 60 fd 28  |..>....S7)..$`.(|
@@ -183027,9 +183027,9 @@
 002cc840  71 8f 97 d7 35 e9 1a 46  a0 1a 38 9c c8 cc 49 ea  |q...5..F..8...I.|
 002cc850  e3 07 1f 5a f8 cc d3 0f  68 ca a4 51 f3 38 a5 cd  |...Z....h..Q.8..|
 002cc860  2d 4e d6 09 4b 9d e9 80  81 f1 c3 73 9a e8 11 43  |-N..K......s...C|
-002cc870  46 57 9d ec 00 23 3c 11  5f 09 8c 82 51 e7 6f 53  |FW...#<._...Q.oS|
-002cc880  c5 c4 d3 93 a8 9c 19 cc  6a d6 db 92 46 c6 58 a9  |........j...F.X.|
-002cc890  d9 92 06 0d 78 e6 b1 6b  cc 8e ca 63 95 87 23 ef  |....x..k...c..#.|
+002cc870  46 57 ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |FW..............|
+002cc880  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+002cc890  00 00 06 0d 78 e6 b1 6b  cc 8e ca 63 95 87 23 ef  |....x..k...c..#.|
 002cc8a0  02 7b 91 e9 5e ce 45 56  5c c7 a5 95 4f d9 57 b7  |.{..^.EV\...O.W.|
 002cc8b0  73 cc 35 08 36 3b 7c ce  c7 27 27 15 cf bc 72 6d  |s.5.6;|..''...rm|
 002cc8c0  2e 5f 00 f0 31 c9 02 bf  51 c1 4a 12 a7 73 ec 14  |._..1...Q.J..s..|
@@ -205827,9 +205827,9 @@
 00325940  9a 27 60 5b 04 04 38 00  f1 5a d3 aa f6 91 e6 d7  |.'`[..8..Z......|
 00325950  c3 be 69 3a 65 02 0a 67  82 72 dc 8c 11 83 4a d0  |..i:e..g.r....J.|
 00325960  09 b6 60 e3 cb 3b 80 3d  09 f5 ae a5 2b 7b c8 f3  |..`..;.=....+{..|
-00325970  e7 09 55 87 24 d1 34 76  ca 58 1d 99 6d c3 3c 66  |..U.$.4v.X..m.<f|
-00325980  b6 a1 b6 8d 58 90 a3 69  e9 cf 20 d6 75 aa e9 a1  |....X..i.. .u...|
-00325990  54 f0 71 b5 da 27 30 b8  63 18 43 cf 20 9e 00 1d  |T.q..'0.c.C. ...|
+00325970  e7 09 55 87 24 d1 34 76  ca 58 1d 99 6d c3 ff 00  |..U.$.4v.X..m...|
+00325980  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00325990  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 1d  |................|
 003259a0  a9 ff 00 65 0e 80 b0 f9  f9 04 13 81 59 c6 a2 8d  |...e........Y...|
 003259b0  b5 32 ab 46 13 52 49 5c  c6 be b1 52 c7 60 24 0c  |.2.F.RI\...R.`$.|
 003259c0  7c a7 b1 f6 f7 ae 62 fe  c1 5d 49 68 f2 1b 39 0d  ||.....b..]Ih..9.|
@@ -228814,9 +228814,9 @@
 0037f5f0  4c 7d 08 23 07 39 03 a8  a4 c7 cc 38 e0 71 d7 3c  |L}.#.9.....8.q.<|
 0037f600  d7 65 7a ad c5 b5 1b 98  f2 49 34 ee 3d be f0 fe  |.ez......I4.=...|
 0037f610  11 c8 38 e6 8e 84 f2 48  ed ed 5c ee 4a 69 a3 49  |..8....H..\.Ji.I|
-0037f620  3b 59 89 9c 7a e4 1e 0e  0f 5a 32 3d 3d fa 56 0d  |;Y..z....Z2==.V.|
-0037f630  d4 56 57 17 32 95 f4 1c  d9 c8 c7 4f 7f 4a 50 b9  |.VW.2......O.JP.|
-0037f640  38 ce 06 33 b8 8e 2b 45  89 72 b5 39 3b b1 38 b7  |8..3..+E.r.9;.8.|
+0037f620  3b 59 89 9c 7a e4 1e 0e  0f 5a 32 3d 3d fa ff 00  |;Y..z....Z2==...|
+0037f630  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0037f640  00 00 00 00 00 00 00 00  00 00 00 00 00 00 38 b7  |..............8.|
 0037f650  2b 24 38 80 15 48 fb d9  24 e7 b8 ed 4d 24 83 b8  |+$8..H..$...M$..|
 0037f660  e3 3f a5 6f 53 9e 56 8a  5b 15 65 0b ab 89 b8 31  |.?.oS.V.[.e....1|
 0037f670  ce 01 1d 31 d2 95 32 49  3e 9c 0f 42 6b 92 4a d2  |...1..2I>..Bk.J.|
@@ -251796,9 +251796,9 @@
 003d9250  d6 80 40 e3 27 1d 49 c7  34 dd 7f 67 3e 5b 89 27  |..@.'.I.4..g>[.'|
 003d9260  7b b0 c9 6c 7a 67 be 09  a7 6d 20 11 9c 2f e1 93  |{..lzg...m ../..|
 003d9270  57 39 2a 95 13 8e 96 1a  bb dd 08 cf 81 c6 33 8c  |W9*...........3.|
-003d9280  e4 73 4f 1c fe 3d 2b 3a  e9 d4 f8 55 bb 83 e5 7b  |.sO..=+:...U...{|
-003d9290  0a 18 10 40 3c e4 60 52  60 a8 c6 01 39 f5 e8 2a  |...@<.`R`...9..*|
-003d92a0  e3 83 94 23 04 99 9b 93  dd 07 39 1c f1 dc 62 94  |...#......9...b.|
+003d9280  e4 73 4f 1c fe 3d 2b 3a  e9 d4 ff 00 ff 00 00 00  |.sO..=+:........|
+003d9290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+003d92a0  00 00 00 00 00 00 00 00  00 00 39 1c f1 dc 62 94  |..........9...b.|
 003d92b0  af 23 8e a7 f1 15 9c ea  29 25 cc af 66 68 e2 f7  |.#......)%..fh..|
 003d92c0  b8 a5 71 ce 0e 71 e9 8a  66 ed b8 eb c7 7a 72 ab  |..q..q..f....zr.|
 003d92d0  18 cd 7b 35 a0 ac d2 d5  8f 04 1c 7e 79 1d 69 5d  |..{5.......~y.i]|
@@ -274693,9 +274693,9 @@
 00432960  64 72 23 7c ac dd cf d2  ad 68 da 04 cd 3f 9a 19  |dr#|.....h...?..|
 00432970  e4 45 0a ec 57 80 3d 45  7f 34 e2 b1 0e 94 ea a6  |.E..W.=E.4......|
 00432980  b7 3f 26 c5 29 52 94 a4  d5 8f 41 d4 34 bb c9 f4  |.?&.)R....A.4...|
-00432990  d2 f3 22 b5 ac 98 81 f7  6d ea 39 c7 af e3 5f 02  |..".....m.9..._.|
-004329a0  fc 5b f0 92 cb aa 5e 34  30 30 06 62 c5 06 32 18  |.[....^400.b..2.|
-004329b0  7f 21 da bd 9e 06 cc 2a  61 b3 58 b7 a7 fc 39 ed  |.!.....*a.X...9.|
+00432990  d2 f3 ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |................|
+004329a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+004329b0  00 00 da bd 9e 06 cc 2a  61 b3 58 b7 a7 fc 39 ed  |.......*a.X...9.|
 004329c0  f0 ee 2f 93 11 49 2e e7  90 e9 76 63 4b 8e 48 9e  |../..I....vcK.H.|
 004329d0  02 09 70 72 e0 f0 7f ad  79 5f 8f 34 eb 8b fd 52  |..pr....y_.4...R|
 004329e0  69 6d 99 65 1b c3 39 44  28 aa 0f a0 af e9 3c a3  |im.e..9D(.....<.|
@@ -297589,9 +297589,9 @@
 0048c060  45 52 a2 9c 91 a1 67 a6  59 cf 13 49 71 04 32 3b  |ER....g.Y..Iq.2;|
 0048c070  28 3f 38 27 24 76 e3 9a  a9 a8 08 e1 6f b3 05 85  |(?8'$v......o...|
 0048c080  1a 3f 97 82 09 c7 a6 7d  a8 95 7f 6b 1e 54 cd e1  |.?.....}...k.T..|
-0048c090  5e 5c fc 96 d8 e6 6f ac  6e 70 66 8c b3 27 92 5f  |^\....o.npf..'._|
-0048c0a0  00 64 81 ea 29 9a 34 b7  96 f7 6c cc 1d 77 05 22  |.d..).4...l..w."|
-0048c0b0  66 6c 72 38 c5 74 47 93  d8 b8 49 6a 77 46 71 a9  |flr8.tG...IjwFq.|
+0048c090  5e 5c fc 96 d8 e6 6f ac  6e 70 ff 00 ff 00 00 00  |^\....o.np......|
+0048c0a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0048c0b0  00 00 00 00 00 00 00 00  00 00 49 6a 77 46 71 a9  |..........IjwFq.|
 0048c0c0  49 c5 6e 7a 1d fe af 2d  d4 1e 5a dc 10 f2 46 23  |I.nz...-..Z...F#|
 0048c0d0  97 28 18 85 ef d4 7e 46  b8 e7 b0 95 98 e1 f6 0e  |.(....~F........|
 0048c0e0  a5 8f 04 9a 78 4a cd f3  46 47 95 45 ba 6e 4a da  |....xJ..FG.E.nJ.|
@@ -320480,9 +320480,9 @@
 004e5710  46 b3 8d f4 5f 89 c8 e3  a5 99 20 40 39 ea 1b 81  |F..._..... @9...|
 004e5720  4f 38 04 05 cf b7 d6 a2  8c 79 e2 ea c5 05 dc 6c  |O8.......y.....l|
 004e5730  98 9d 7d b1 e9 de 9c 46  3a f1 f4 39 c9 ae 6a 90  |..}....F:..9..j.|
-004e5740  a8 e6 dc 55 d1 69 36 dd  90 d6 3d 02 13 9c f3 9e  |...U.i6...=.....|
-004e5750  05 2f 23 af 07 1f 98 a7  46 9c aa 54 bd f4 07 28  |./#.....F..T...(|
-004e5760  a7 67 b0 a1 49 e4 31 04  76 14 e0 ad bb 3c 02 01  |.g..I.1.v....<..|
+004e5740  a8 e6 dc 55 d1 69 36 dd  90 d6 3d 02 13 9c ff 00  |...U.i6...=.....|
+004e5750  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+004e5760  00 00 00 00 00 00 00 00  00 00 00 00 00 00 02 01  |................|
 004e5770  6e 7f cf 7a eb f6 6e 11  70 4e d7 25 ca 09 ed 71  |n..z..n.pN.%...q|
 004e5780  84 6e 20 9e 39 1c 8a 1b  e5 cf 7e bd fa 0a 99 46  |.n .9.....~....F|
 004e5790  2a 8a a7 39 68 10 93 96  a8 08 39 24 f4 3c 8c 53  |*..9h.....9$.<.S|
@@ -343467,9 +343467,9 @@
 0053f3c0  af e0 6c dc a7 b9 13 60  9c b7 e5 9e 29 14 1d f8  |..l....`....)...|
 0053f3d0  c7 de e0 7d 68 85 68 b6  ee b5 1b b6 a8 7e 0a 05  |...}h.h......~..|
 0053f3e0  5c 83 9f bc 33 c8 34 06  5d db 42 fc bd 09 f4 35  |\...3.4.].B....5|
-0053f3f0  cb 3b 51 ac db df a1 6a  57 49 13 0c 01 81 8c 73  |.;Q....jWI.....s|
-0053f400  8e 99 34 a3 39 1e bd 0e  0d 75 c1 39 46 31 9a d1  |..4.9....u.9F1..|
-0053f410  8d a4 dd a2 0c 76 1e 41  3c 53 4e e6 24 96 c9 ce  |.....v.A<SN.$...|
+0053f3f0  cb 3b 51 ac db df a1 6a  57 49 13 0c 01 81 ff 00  |.;Q....jWI......|
+0053f400  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0053f410  00 00 00 00 00 00 00 00  00 00 00 00 00 00 c9 ce  |................|
 0053f420  47 60 07 ff 00 5e 95 4a  d4 62 e5 16 b5 e8 47 25  |G`...^.J.b....G%|
 0053f430  fe 26 07 07 26 42 5c 80  4f 1e b4 9b b3 d0 75 e0  |.&..&B\.O.....u.|
 0053f440  77 af 3e 34 e5 69 56 6f  40 b3 be a4 bc 63 90 db  |w.>4.iVo@....c..|
@@ -366273,9 +366273,9 @@
 00598520  8c d1 cd cc b7 09 2e 5b  f2 92 14 0a a0 17 63 ba  |.......[......c.|
 00598530  31 9c 74 ff 00 39 a6 bb  2e 51 71 85 45 db b4 77  |1.t..9...Qq.E..w|
 00598540  3d cd 43 92 72 b0 be 15  7b 08 dc 29 c1 c2 90 06  |=.C.r...{..)....|
-00598550  3d aa 61 1e 08 dc 07 3c  9f 7f 6a 39 ad b9 32 94  |=.a....<..j9..2.|
-00598560  e0 d6 9a 0a c9 f2 ed 55  5c e4 b0 3c e7 14 aa ab  |.......U\..<....|
-00598570  95 dc 33 83 96 ec 68 73  4a cd 1d 12 71 ba e5 1a  |..3...hsJ...q...|
+00598550  3d aa 61 1e 08 dc 07 3c  9f 7f 6a 39 ad b9 ff 00  |=.a....<..j9....|
+00598560  ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+00598570  00 00 00 00 00 00 00 00  00 00 00 00 00 00 e5 1a  |................|
 00598580  23 1f 3e 17 e5 56 27 eb  4e c2 90 3a 01 d7 8e b4  |#.>..V'.N..:....|
 00598590  a7 29 37 a1 93 96 ac 94  26 51 5b 9c 75 5f 42 3d  |.)7.....&Q[.u_B=|
 005985a0  6a 35 c3 64 e0 e7 3c 00  78 06 85 7e 56 fb 19 37  |j5.d..<.x..~V..7|
@@ -387513,9 +387513,9 @@
 005eb4a0  a9 28 8f 03 18 2d 97 e7  1d 00 a9 3c bd c7 69 03  |.(...-.....<..i.|
 005eb4b0  23 a7 24 91 d6 b1 9f 32  96 e2 a7 27 35 b0 30 23  |#.$....2...'5.0#|
 005eb4c0  a7 2d d3 20 80 29 40 ce  37 1c fb 71 81 59 bb 25  |.-. .)@.7..q.Y.%|
-005eb4d0  70 7a b4 89 55 38 c9 24  1c fe 38 a9 19 06 57 8e  |pz..U8.$..8...W.|
-005eb4e0  ab 93 9e c2 b3 9b 56 b5  c4 d4 af 74 00 0e c3 d0  |......V....t....|
-005eb4f0  7d 6a 65 4c 91 cf 5c 7c  bc 74 ac 6f be a3 71 bb  |}jeL..\|.t.o..q.|
+005eb4d0  70 7a ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |pz..............|
+005eb4e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+005eb4f0  00 00 65 4c 91 cf 5c 7c  bc 74 ac 6f be a3 71 bb  |..eL..\|.t.o..q.|
 005eb500  4d 8f 31 f5 c0 3d 73 d3  93 52 05 5d ac 31 9e 78  |M.1..=s..R.].1.x|
 005eb510  f5 c5 47 b4 b7 42 25 7e  a4 aa 85 40 e0 71 9c e4  |..G..B%~...@.q..|
 005eb520  80 05 3f 60 0c 43 71 d7  02 b2 6e ed 90 e4 92 4a  |..?`.Cq...n....J|
@@ -410500,9 +410500,9 @@
 00645150  1d 48 88 6d dd d3 9e 9c  75 ae 48 7c 6a c4 52 a7  |.H.m....u.H|j.R.|
 00645160  25 3b a7 a1 4e de d8 a7  bb 85 05 ba 80 6b 72 20  |%;..N........kr |
 00645170  c0 27 50 78 20 03 c0 15  be 32 70 94 55 98 62 6d  |.'Px ....2p.U.bm|
-00645180  36 e2 8b 66 50 72 cd 9c  20 f5 1d 7f cf f9 e6 a5  |6..fPr.. .......|
-00645190  59 86 47 cc 01 65 ce 70  71 9f 4a f2 64 9b 7c a7  |Y.G..e.pq.J.d.|.|
-006451a0  2c 62 9a 77 23 21 03 48  d2 22 c8 54 8e 39 00 55  |,b.w#!.H.".T.9.U|
+00645180  36 e2 ff 00 ff 00 00 00  00 00 00 00 00 00 00 00  |6...............|
+00645190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+006451a0  00 00 9a 77 23 21 03 48  d2 22 c8 54 8e 39 00 55  |...w#!.H.".T.9.U|
 006451b0  7b e9 63 54 62 ca 0c 8c  a7 12 31 24 fd 2b aa 92  |{.cTb.....1$.+..|
 006451c0  93 94 54 42 11 8f 3a e5  38 b9 27 0c cb d1 59 b2  |..TB..:.8.'...Y.|
 006451d0  40 1e b5 5f cd 2e 42 9c  92 4e 7b ff 00 9f ff 00  |@.._..B..N{.....|
@@ -433396,9 +433396,9 @@
 0069e850  c9 dd b4 88 fc c0 5c 14  23 76 36 fc c4 62 ad ac  |......\.#v6..b..|
 0069e860  d8 05 1c fc cb 8e 31 83  4e 50 ba 57 e8 52 a7 66  |......1.NP.W.R.f|
 0069e870  ae c9 d8 c7 20 32 1c 07  38 1c 0c 0f f2 6a a4 a3  |.... 2..8....j..|
-0069e880  ef 6c 54 19 90 b3 65 6a  69 dd 34 a4 c1 ee e3 61  |.lT...eji.4....a|
-0069e890  91 c3 11 97 ee 6e dd c0  24 e3 07 d6 af aa aa 2e  |.....n..$.......|
-0069e8a0  72 c1 84 84 63 a0 06 8a  b5 26 e2 92 22 51 70 5c  |r...c....&.."Qp\|
+0069e880  ef 6c 54 19 90 b3 65 6a  69 dd ff 00 ff 00 00 00  |.lT...eji.......|
+0069e890  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
+0069e8a0  00 00 00 00 00 00 00 00  00 00 e2 92 22 51 70 5c  |............"Qp\|
 0069e8b0  cd 90 4c 77 31 21 b7 1e  7e f1 eb 55 d9 49 60 8d  |..Lw1!..~..U.I`.|
 0069e8c0  f2 ec c8 74 c6 33 fe 7f  c2 ae 0e c9 69 b1 b3 6a  |...t.3......i..j|
 0069e8d0  10 4c a1 22 aa b3 10 85  57 27 1c 83 8a 62 82 49  |.L."....W'...b.I|

That shows that what is happening is that chunks of 32 bytes are being overwritten, mostly with 0s but with some FFs

So definite data corruption.

This is ringing alarm bells. There has been several problems with the go runtime and CIFS recently which are due to bugs in the Linux kernel.

These can be mitigated with setting the environment variable GODEBUG=asyncpreemptoff=1. I haven't heard of these causing data corruption but there is always a first time...

So can you try with that environment variable?

Or maybe the problem is bigger than that You could try with rclone md5sum and the above variable and see if it helps.

Note that if you were to run rclone check after an apparently successful transfer, I'm pretty sure it would fail.

Ben_Wilson · January 18, 2021, 2:18pm

Set GODEBUG and re-ran the copy command below. Still failed - differing MD5 sums.
Ran rclone md5sum twice, same checksums both times.

rclone copy -vv -c -P --exclude=".DS_Store" /mnt/VIA/\!\ clients\ \(ONLY\ ARCHIVED\)/\!\ active\ \(DEPRECATED\!\ USE\ GOOGLE\ SHARED\ DRIVE\!\)/VIA\ Studio/_Library/Photography/2020.01\ Holiday\ Socks/_N8A7290.JPG  activeclients:VIA\ Studio/_Library/Photography/2020.01\ Holiday\ Socks/

2021/01/18 09:13:21 DEBUG : Creating backend with remote "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks/_N8A7290.JPG"
2021/01/18 09:13:21 DEBUG : Using config file from "/home/viaadmin/.config/rclone/rclone.conf"
2021/01/18 09:13:21 DEBUG : fs cache: adding new entry for parent of "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks/_N8A7290.JPG", "/mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks"
2021/01/18 09:13:21 DEBUG : Creating backend with remote "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks/"
2021/01/18 09:13:23 DEBUG : fs cache: renaming cache item "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks/" to be canonical "activeclients:VIA Studio/_Library/Photography/2020.01 Holiday Socks"
2021-01-18 09:13:23 DEBUG : _N8A7290.JPG: MD5 = 5fd57d14d347f3702a675eb57eb81063 (Local file system at /mnt/VIA/! clients (ONLY ARCHIVED)/! active (DEPRECATED! USE GOOGLE SHARED DRIVE!)/VIA Studio/_Library/Photography/2020.01 Holiday Socks)
2021-01-18 09:13:23 DEBUG : _N8A7290.JPG: MD5 = 827bf0662fda9c907f81f0469a907609 (Google drive root 'VIA Studio/_Library/Photography/2020.01 Holiday Socks')
2021-01-18 09:13:23 DEBUG : _N8A7290.JPG: MD5 differ
2021-01-18 09:13:30 DEBUG : _N8A7290.JPG: MD5 = b2dcadb4348fae0b599549926e708494 OK
2021-01-18 09:13:30 INFO  : _N8A7290.JPG: Copied (replaced existing)
Transferred:        6.770M / 6.770 MBytes, 100%, 1021.606 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         8.3s
2021/01/18 09:13:30 INFO  : 
Transferred:        6.770M / 6.770 MBytes, 100%, 1021.606 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         8.3s

rclone md5sum activeclients:VIA\ Studio/_Library/Photography/2020.01\ Holiday\ Socks/_N8A7290.JPG

b2dcadb4348fae0b599549926e708494  _N8A7290.JPG                                                                                            
b2dcadb4348fae0b599549926e708494  _N8A7290.JPG

ncw · January 18, 2021, 3:26pm

That isn't too suprising, alas.

I would guess the file got into your OS cache so it was the same for each run.

You can clear the cache easily enough, but I think solving the data corruption when reading from CIFS is the thing to work on and I don't think that is an rclone problem

Ben_Wilson · January 18, 2021, 3:51pm

Thank you @ncw and @Animosity022 for the help!

I'm proceeding by copying files locally off the CIFS share, and then running rclone copy from there.

Ben_Wilson · January 19, 2021, 10:31pm

Following up here.

Summary: this is not rclone's fault. Appears to be an issue with linux/CIFS/NFS.

Details:
I ran through a number of other tests - all of which led me to see that the copy from the Synology DS410 NAS over both a CIFS or NFS mount resulted in a corrupted file.

I am now using rclone on a WINDOWS machine to rclone copy files to GDrive from Synology DS410 network share, and it works fine.

So... there's something up with the network file share stuff on that linux machine.

Thanks for your help and patience here!

ncw · January 19, 2021, 11:02pm

Well done for getting to the bottom of it. Data corruption is very bad