Copying Files Via Rclone Mount

Help and Support
1

What is the problem you are having with rclone?

I copy/cut a little over 100 GBs of GoPro video files I have from scuba diving on vacation into my S3 drive via Rclone Mount. It took about 15 minutes to copy from the computer's desktop into the folder I created via Windows Explorer. I did low bandwidth things like email for the next 45 minutes. About 30 minutes after the copy/cut looked to be complete, I went browsing in the Rclone Mount drive for an item. The computer froze. I hard booted the computer. Once I brought the computer and drive back up, only about 10 GB or 10% of the video files were there, the rest seemingly lost. I'm guessing all of the video files were copied to a temporary folder on my computer. I still don't understand why this would take 15 minutes on a copy/cut when files just changing directories on the same drive do so almost instantly. I have fast internet, so I assumed this time was all upload time. It appears the uploading was happening in the background, crashed my computer, and now I cannot find the files. Is there a way to recover these files that seemingly disappeared?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: Microsoft Windows 10 Home 21H2 (64 bit)
  • os/kernel: 10.0.19044.1889 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

iDrive S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Windows GUI (via Rclone Mount). Copy/Cut/Paste Folder from desktop on computer into folder on iDrive S3 via Rclone Mount.

The rclone config contents with secrets removed.

[idrive]
type = s3
provider = Other
access_key_id =
secret_access_key =
region = us-east-1
endpoint = s3.us-east-1.idrivecloud.io
location_constraint = us-east-1
max_upload_parts = 1000

A log from the command with the -vv flag

No log - as no command line was completed.

2

Hi Christopher,

Sorry to hear about your issue.

Perhaps this post can help you find a (possible) rclone (write) cache containing files that haven't been uploaded yet:

https://forum.rclone.org/t/cant-find-cache-folder-on-windows/15186/7

If not, then please share the exact mount command used, that is something like:

rclone mount myS3: ...

How many percent of the files do you see in S3 when using a command something like this:

rclone lsl myS3:
3

@Ole I appreciate your quick reply. It led to successfully getting the video files back. I still have some questions that either you or another community member maybe able to answer regarding rclone:

  1. Resource Management of the rclone mount: During copies and when I am actively browsing the mounted drive in Windows Explorer, rclone mount routinely freezes Windows Explorer and my computer overall. During this case of a large copy/cut of 31 files totaling 105 GB, it took approximately 15 minutes for the Windows Explorer copy dialog to complete cutting the folder from the desktop into the rclone mounted folder. I assumed since the files disappeared from the desktop that the upload was complete. Only four files, totaling about 8GB, were completely uploaded into the S3, while a few others were partially uploaded. The remaining files were in the local cache directory (thank you so much for the assistance on this). Is there something that tells rclone to resume copying from local cache if interrupted? Can the progress be reported via the Windows Explorer copy dialog, so the user knows how long the upload is taking? I have started uploading the remaining files (95 GB) via the command line so I can see the progress of each file. It is only uploading about 4 GB/hour, so I have no idea what took Windows Explorer 15 minutes on its copy/cut dialog, when usually a folder change on the same drive (desktop to local cache) is instantaneous. It seems like rclone mount’s browsing or copying takes up most of the computer’s resources. Can this be scaled back so it doesn’t lock up the computer? I understand my mount command may need adjusting, so I included it below in case it needs to be tweaked.

  2. Any tips for uploading large files? Through the command line, GUI, or another preferred method?

  3. Any tips for viewing large files, such as videos? For example, I took a 35-second video on my iPhone, uploaded the 75MB video onto my S3 Drive, and attempted to view the video on my computer using VLC through the rclone mount in Windows Explorer. The 75MB file took about 15-20 seconds to start playing. It stopped after only a few seconds and would not resume. I copied the entire file onto my local drive in less than 10 seconds and viewed it without issue. Is there a way to stream video files so there isn’t much lag/chop?

  4. Deleted folders/directories keep reappearing. There are folders I have deleted probably 50 times through the rclone mount Windows Explorer GUI. I don’t know if they aren’t getting deleted on the local cache or the S3 drive itself, but after about 5 minutes or so usually they are popping back up and not deleted. I have also tried using the command prompt with the commands of delete, purge, rmdir, or rmdirs. Has anyone else run into problems deleting folders from a mounted drive? How have you solved this issue?

The post you shared directed me to the cache directory where the video files still remained. I am very grateful! I use NSSM to mount the S3 drive automatically each time the computer starts. I have done so for about a year. The command I use is:


mount idrive:idrive/ S: --vfs-cache-mode writes --file-perms 0777

Is there a better command that will help rclone operate smoother?

4

Glad to hear :sweat_smile:

I use the Microsoft OneDrive client on Windows and only use rclone mount now and then to access my OneDrive on Linux, so I really can't comment on your experiences.

I do see similar experiences once in a while in the forum, but the vast majority of users seem happy. I think part of the trick is to disable any display of previews and thumbnails (of photos, videos etc.) in Windows Explorer - as they probably require a download/scan of the file to be generated. The same would apply to meta data such as "date taken". Try searching the forum for tips around this.

Is your WinFSP up-to date?

rclone should automaticly restart any pending uploads when the mount is restarted.

No, but check my proposal at the end - specifically the PowerShell command to check current status.

This isn't my experience from Linux, nor the many posts I have seen in this forum.

I suggest you try my proposal at the end and then checks the log the next time it happens. If unable to troubleshoot yourself, then try replicating with debug logging -vv and then make a new support post with the debug log and all the other requested information.

I use rclone copy/sync on the command line (or from automated PowerShell scripts).

Perhaps @asdffdsa has some tips.

I think we discussed this a year ago; no changes since as far as I can see:

Here is my proposal for an all-round mount based on my experiences mounting OneDrive on Linux:

rclone mount onedrive: o: --vfs-cache-mode=full --vfs-cache-max-size=100G --vfs-cache-max-age=720h --log-file=onedrive_mount.log --log-level=INFO --stats=1m

I use --vfs-cache-mode=full --vfs-cache-max-size=100G --vfs-cache-max-age=720h to get quick access to the most frequently used files, that is my "working set". I keep the files used in the past month up to a maximum of 100G.

I use --log-file=onedrive_mount.log --log-level=INFO --stats=1m to record the activities of the mount. This is very useful to understand, tune and troubleshoot. The current status can quickly be checked with this PowerShell command: Get-Content onedrive_mount.log -Tail 10 -Wait

You probably want to add --dir-perms and --file-perms to make it fit your setup and usage.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.