Split folder using move --max-transfer doesn't work at same drive

Suspected Bug
1

What is the problem you are having with rclone?

I'm trying split a huge folder to smaller folder size using move --max-transfer, but when I try to do it at the same drive, every file is moved, it's like --max-transfer was ignored, but when I change the drive it works

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.19044.1566 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local Files

The command you were trying to run (eg rclone copy /tmp remote:tmp)

C:\***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads
Total objects: 1.184k (1184)
Total size: 12.261 GiB (13165043855 Byte)

C:\***\rclone-v1.57.0-windows-amd64>rclone move F:\Downloads\ F:\Downloads_1\ -P --max-transfer 1G
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

C:\***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads
2022/03/08 22:46:56 ERROR : : error listing: directory not found
2022/03/08 22:46:56 Failed to size with 2 errors: last error was: directory not found

C:\***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads_1
Total objects: 1.184k (1184)
Total size: 12.261 GiB (13165043855 Byte)

C:\***\rclone-v1.57.0-windows-amd64>rclone move F:\Downloads_1\ G:\Downloads_1\ -P --max-transfer 1G
2022-03-08 22:54:55 ERROR : ****.pdf: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-08 22:54:55 ERROR : Cancelling sync due to fatal error: Max transfer limit reached as set by --max-transfer
2022-03-08 22:54:55 ERROR : ***.zip: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-08 22:54:56 ERROR : ***.bat: Failed to copy: context canceled
2022-03-08 22:54:56 ERROR : ***.bat: Not deleting source as copy failed: context canceled
2022-03-08 22:55:37 NOTICE: ***.zip: Removing partially written file on error: Max transfer limit reached as set by --max-transfer
2022-03-08 22:55:37 ERROR : ***.zip: Failed to copy: Max transfer limit reached as set by --max-transfer
2022-03-08 22:55:37 ERROR : ***.zip: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-08 22:55:37 ERROR : Fatal error received - not attempting retries
Transferred:        1.000 GiB / 11.411 GiB, 9%, 3.823 MiB/s, ETA 46m28s
Errors:                 6 (fatal error encountered)
Checks:               758 / 758, 100%
Deleted:              377 (files), 0 (dirs)
Renamed:              377
Transferred:          377 / 1180, 32%
Elapsed time:       8m2.6s
2022/03/08 22:55:38 Failed to move with 6 errors: last error was: Max transfer limit reached as set by --max-transfer

C:\***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads_1
Total objects: 807 (807)
Total size: 11.550 GiB (12402028431 Byte)

C:\***\rclone-v1.57.0-windows-amd64>rclone size G:\Downloads_1
Total objects: 378 (378)
Total size: 727.670 MiB (763017693 Byte)

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
2

This looked like it worked to me there is a bit less than 1GB of files in G:. Rclone got to the cutoff while it was transferring some files and it couldn't finish those ones off so it gave errors for those.

You can try --cutoff-mode soft or --cutoff-mode cautious for different effects.

3

OK. I understand about the error in command "rclone move F:\Downloads_1\ G:\Downloads_1\ -P --max-transfer 1G" the error is in "rclone move F:\Downloads\ F:\Downloads_1\ -P --max-transfer 1G" command. It doesn't return any error, but it moves every file, and to prove it I ran the command "rclone size F:\Downloads_1", it returns "Total size: 12.261 GiB", it isn't correct, it should be something near 1GiB

I put the command "rclone move F:\Downloads_1\ G:\Downloads_1\ -P --max-transfer 1G" to ensure that command can run normally when I move files to another drive (F: to G:) but don't work when I move to same drive in another fold (F:\Downloads\ to F:\Downloads_1)

4

Ah I see.

What rclone did was rename the directory rather than transfer any files...

Try with --disable DirMove that will stop rclone renaming the directory and it will be forced to move the files.

5

Still didn't work, but now it isn't 'automatic' (Transferred: 0B/0B and Elapsed time: 0.0s), it show the number of files and Elapsed Time:

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads
Total objects: 1.197k (1197)
Total size: 8.528 GiB (9157265650 Byte)

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads_1
Total objects: 0 (0)
Total size: 0 B (0 Byte)

C:***\rclone-v1.57.0-windows-amd64>rclone move F:\Downloads\ F:\Downloads_1\ -P --max-transfer 1G --disable DirMove
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 1197 / 1197, 100%
Renamed: 1197
Elapsed time: 6.4s

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads
Total objects: 0 (0)
Total size: 0 B (0 Byte)

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads_1
Total objects: 1.197k (1197)
Total size: 8.528 GiB (9157265650 Byte)

6

Now it is renaming the files instead of the directory.

Try --disable DirMove,Move

7

Finally works, thanks!

C:***\rclone-v1.57.0-windows-amd64>rclone move F:\Downloads\ F:\Downloads_1\ -P --max-transfer 1G --disable DirMove,Move
2022-03-09 17:44:24 NOTICE: ***.zip: Removing partially written file on error: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 NOTICE: ***.img: Removing partially written file on error: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 NOTICE: ***.png: Removing partially written file on error: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 NOTICE: ***.exe: Removing partially written file on error: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 ERROR : ***.png: Failed to copy: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 ERROR : ***.img: Failed to copy: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 ERROR : ***.png: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 ERROR : ***.img: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:24 ERROR : Cancelling sync due to fatal error: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:25 ERROR : ***.exe: Failed to copy: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:25 ERROR : ***.exe: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:25 ERROR : ***.zip: Failed to copy: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:25 ERROR : ***.zip: Not deleting source as copy failed: Max transfer limit reached as set by --max-transfer
2022-03-09 17:44:25 ERROR : Fatal error received - not attempting retries
Transferred: 1.000 GiB / 7.347 GiB, 14%, 29.283 MiB/s, ETA 3m41s
Errors: 4 (fatal error encountered)
Checks: 874 / 874, 100%
Deleted: 435 (files), 0 (dirs)
Renamed: 435
Transferred: 435 / 1193, 36%
Elapsed time: 33.2s
2022/03/09 17:44:25 Failed to move with 4 errors: last error was: Max transfer limit reached as set by --max-transfer

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads
Total objects: 762 (762)
Total size: 7.658 GiB (8222681621 Byte)

C:***\rclone-v1.57.0-windows-amd64>rclone size F:\Downloads_1
Total objects: 435 (435)
Total size: 891.289 MiB (934584029 Byte)

Do you have documetation about this commands? When I run "rclone --disable help", it shows only a list of commands.

C:***\rclone-v1.57.0-windows-amd64>rclone --disable help
2022/03/09 17:46:25 Possible backend features are: CaseInsensitive, DuplicateFiles, ReadMimeType, WriteMimeType, CanHaveEmptyDirectories, BucketBased, BucketBasedRootOK, SetTier, GetTier, ServerSideAcrossConfigs, IsLocal, SlowModTime, SlowHash, Purge, Copy, Move, DirMove, ChangeNotify, UnWrap, WrapFs, SetWrapper, DirCacheFlush, PublicLink, PutUnchecked, PutStream, MergeDirs, CleanUp, ListR, About, OpenWriterAt, UserInfo, Disconnect, Command, Shutdown

8

Great!

It isn't as efficient as it could be, but glad it is working.