Extracting to ACD mount possible?

Hello guys,

I have two questions:

  1. I was trying to extract files from my server directly to the ACD mount but unfortunately I just got an error.
    Is extracting to a ACD mount even possible or is there a workaround?

  2. I tried to move files directly to the mounted folder on my server and it even the encryption seemed to work perfectly fine. The files are shown in the mount and if I go the ACD WebUI I can only see the encrypted files.

So my question is: Do I need to use the rclone move command to move files to ACD or can I just use something like WinSCP to do it?

rclone copy or rclone move if you don’t want originals left behind is the most reliable way of doing the transfers. The rclone mount isn’t as reliable writing files.

So, if I understood you correctly files which are written using rclone mount are as well encrypted as they would be using rclone move, correct?

What about extracting files to a ACD mount, is that possible? Do you have experiences?

If you copied them to an crypted remote, then yes.

That will depend on what is doing the extracting. I recommend extracting them locally then using rclone move for the moment.

It are .RAR files with a couple of files inside I am trying to extract to the mount.

Besides the workaround you do not have an idea how a could solve this issue, right?

Btw, thanks a lot for the excellent support! :slight_smile:

I will solve this issue in the next release or two. It isn’t a trivial fix and it will involve temporary files on the disk.

Nice to hear that you are planning on tackling this issue! Looking forward to get to beta test it.

Once again, thanks a lot! :slight_smile:

@ncw, after a couple of months and moving to Gdrive I gave it another try.

Extraction is working pretty well but unfortunately there are some… lets call it “leakages”.
Although the extraction log does not show any error some files have 0 bytes after extraction.

Already tried it without and with vfs-cache-mode writes and the mount is encrypted but pointed directly to Gdrive without a cache layer.

Why does this happen?
Could I use the --cache-writes flag even though I am not using the cache?
What else would you recommend?

I have a log file of the rclone mount but it is about 400MB and my laptop isn’t powerful enough to handle it to be honest.

This depends how you are extracting your files. I’m using unrar with --vfs-cache-mode writes and got no empty files so far.
It is possible to get empty file by using touch on files with --vfs-cache-mode writes, which is probably not intended (i will open a issue about this behavior). This is only a visual bug, but could confuse the extracting program (see #2021).

The documentation explains the use of the --vfs-cache-... flags pretty well.
In short, yes you can use it without a cache remote. It’s completely independent feature.

Well, I do use the integrated archive extractor in JDownloader with no change in the default settings. May it be a problem with the extractor of JDownloader?
During extraction I do not do anything with the files or the folder they are being extracted to.

I am not sure if you understood me correctly in that point. As alternative to the --vfs-cache-mode there is also the flag --cache-writes…

–cache-writes : Will cache file data on writes through the FS

… but I just heard about it being used together with a cache remote. Is that flag also usable without a cache remote?

Thanks for your feedback! :slight_smile:

I’m sorry, I misread the flag you are trying to understand.

--cache-writes is indeed a option for the cache backend and documented here. When using the cache backend this flag will keep uploaded data in the local cache for reading. Without this flag uploaded data will not be in the cache after the transfer is done.

Is your JDownloader using a rclone mount as it’s download folder? I can try that and see how it works for me. It should in theory work fine.

Actually, I was using a cache remote (gdrive -> cache -> crypt) with that flag in the first place but then I had the same problem and that was the reason for me to try it without the cache remote in between (gdrive -> crypt). Unfortunately, the problem still persists.

No, JDownloader is loading to a local folder (/tmp) and extracts then to the mount.
Would appreciate if you could do that and report back.

Just saw that you edited your previous post. I actually already had that problem as well. The files seemed to be empty but actually they were complete.
Unfortunately, in my case this isn’t just a visual bug, the files are empty for real.

Can you reproduce this without the cache remote but with --vfs-cache-mode writes? So just gdrive -> crypt ?

That will give us an idea as to where the problem lies.

Yes, I already tried it in the following three ways (chronologically):

  1. With cache remote (gdrive -> cache -> crypt) and the flag --cache-writes
  2. Without the cache remote (gdrive -> crypt) and no additional flag.
  3. Without the cache remote (gdrive -> crypt) but with the --vfs-cache-mode writes flag.

All these ways lead to the same problem.
I mean, it doesn’t happen every time sometimes it extracts the files correctly but unfortunately that doesn’t happen too often.

If you need any help, just ask would be glad to help as far as I can! :slight_smile:

I tried both scenarios: download + extract on rclone mount and just extract on rclone mount.
Both tries were on GDrive > crypt > mount --vfs-cache-mode writes
download + extract failed after download with a crc error and the download file was empty (log excerpt)

Just extracting on rlcone mount worked, but there are some errors in the log

I will try some more files to see if extraction can fail for me.

@ncw, @B4dM4n,

I shortened the logfile to the essentiell part:

2018/01/26 21:16:13 DEBUG : rclone: Version “v1.39β” starting with parameters [“rclone_1.39” “mount” “–log-level” “DEBUG” “–log-file” “/tmp/log.txt” “–allow-non-empty” “–allow-other” “–buffer-size” “32M” “–dir-cache-time” “300h” “–poll-interval” “8h” “–vfs-cache-mode” “writes” “upload:” “/media/cry”]
2018/01/26 21:16:14 INFO : Encrypted drive ‘upload:’: Modify window is 1ms
2018/01/26 21:16:14 DEBUG : Encrypted drive ‘upload:’: Mounting on “/media/cry”
2018/01/26 21:16:14 DEBUG : vfs cache root is “/root/.cache/rclone/vfs/upload”
2018/01/26 21:16:14 DEBUG : : Root:
2018/01/26 21:16:14 DEBUG : : >Root: node=/, err=
2018/01/26 21:16:14 DEBUG : Google drive root ‘’: Checking for changes on remote
2018/01/26 21:16:15 DEBUG : Google drive root ‘’: All changes were processed. Waiting for more.
2018/01/26 21:16:36 DEBUG : : forgetting directory cache
2018/01/26 21:16:36 DEBUG : : forgetting directory cache
2018/01/26 21:17:32 DEBUG : /: Attr:
2018/01/26 21:17:32 DEBUG : /: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:17:32 DEBUG : : Statfs:
2018/01/26 21:17:32 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2018/01/26 21:17:32 DEBUG : : Statfs:
2018/01/26 21:17:32 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2018/01/26 21:19:44 DEBUG : /: Attr:
2018/01/26 21:19:44 DEBUG : /: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:19:44 DEBUG : : Statfs:
2018/01/26 21:19:44 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2018/01/26 21:19:44 DEBUG : : Statfs:
2018/01/26 21:19:44 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2018/01/26 21:20:53 DEBUG : /: Lookup: name=“upload”
2018/01/26 21:20:53 DEBUG : /: >Lookup: node=upload/, err=
2018/01/26 21:20:53 DEBUG : upload/: Attr:
2018/01/26 21:20:53 DEBUG : upload/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:20:53 DEBUG : /: Attr:
2018/01/26 21:20:53 DEBUG : /: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:20:53 DEBUG : : Statfs:
2018/01/26 21:20:53 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:20:53 DEBUG : upload/: >Lookup: node=, err=no such file or directory
2018/01/26 21:20:53 DEBUG : upload/: Mkdir: name=“testfile”
2018/01/26 21:20:54 DEBUG : upload/: >Mkdir: node=upload/testfile/, err=
2018/01/26 21:21:21 DEBUG : upload/testfile/: Lookup: name=“testfile.txt”
2018/01/26 21:21:21 DEBUG : upload/testfile/: >Lookup: node=, err=no such file or directory
2018/01/26 21:21:21 DEBUG : upload/testfile/: Create: name=“testfile.txt”
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Open: flags=O_RDWR|O_CREATE|O_EXCL
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: >Open: fd=upload/testfile/testfile.txt (rw), err=
2018/01/26 21:21:21 DEBUG : upload/testfile/: >Create: node=upload/testfile/testfile.txt, handle=&{upload/testfile/testfile.txt (rw)}, err=
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Attr:
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: >Attr: a=valid=1m0s ino=0 size=0 mode=-rw-r–r--, err=
2018/01/26 21:21:21 DEBUG : &{upload/testfile/testfile.txt (rw)}: Flush:
2018/01/26 21:21:21 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Flush: err=
2018/01/26 21:21:21 DEBUG : &{upload/testfile/testfile.txt (rw)}: Release:
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: RWFileHandle.Release closing
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: close:
2018/01/26 21:21:21 DEBUG : upload/testfile/: Remove: name=“testfile.txt”
2018/01/26 21:21:21 ERROR : upload/testfile/: Dir.Remove error: file does not exist
2018/01/26 21:21:21 DEBUG : upload/testfile/: >Remove: err=no such file or directory
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Open: flags=OpenReadWrite
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Open: flags=O_RDWR|O_CREATE
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: >Open: fd=upload/testfile/testfile.txt (rw), err=
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Opening cached copy with flags=O_RDWR|O_CREATE|O_EXCL
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Opening cached copy with flags=O_RDWR|O_CREATE
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: >Open: fh=&{upload/testfile/testfile.txt (rw)}, err=
2018/01/26 21:21:21 DEBUG : upload/testfile/testfile.txt: Couldn’t find file - need to transfer
2018/01/26 21:21:22 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=0
2018/01/26 21:21:22 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:21:22 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=131072
2018/01/26 21:21:22 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=

…many writes, why does it say here that it is copied, although it’s not?..

2018/01/26 21:21:23 INFO : upload/testfile/testfile.txt: Copied (new)
2018/01/26 21:21:23 INFO : upload/testfile/testfile.txt: Deleted

…many writes, here the same, it says transferred to remote but it is not!..

2018/01/26 21:21:24 DEBUG : upload/testfile/testfile.txt: transferred to remote
2018/01/26 21:21:24 DEBUG : upload/testfile/testfile.txt: >close: err=
2018/01/26 21:21:24 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Release: err=

…many writes, here the length begins to differ quite a lot, why is it so?

2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=934805504
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=934936576
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=935067648
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=51698, offset=935198720
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=51698, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=128526, offset=935250418
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=128526, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=2546, offset=935378944
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=2546, err=
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=128526, offset=935381490
2018/01/26 21:22:06 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=128526, err=
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=2546, offset=1159249920
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=2546, err=
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=128526, offset=1159252466
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=128526, err=
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=1159380992
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=1159512064
2018/01/26 21:22:14 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=

…is it normal, that the cache got cleaned while still transferring?..

2018/01/26 21:22:14 DEBUG : Cleaning the cache

…many writes, another cache clean…

2018/01/26 21:23:14 DEBUG : Cleaning the cache

…many writes…

2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=3552, offset=3199897600
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=3552, err=
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=127520, offset=3199901152
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=127520, err=
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=3200028672
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=131072, offset=3200159744
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=131072, err=
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Write: len=22941, offset=3200290816
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Write: written=22941, err=
2018/01/26 21:23:49 DEBUG : upload/testfile/testfile.txt: Fsync:
2018/01/26 21:23:49 DEBUG : upload/testfile/testfile.txt: >Fsync: err=
2018/01/26 21:23:49 DEBUG : &{upload/testfile/testfile.txt (rw)}: Flush:
2018/01/26 21:23:49 DEBUG : upload/testfile/testfile.txt: close:

…what happened here? It seems to me that the source was deleted before the transfer was completeted?..

2018/01/26 21:23:50 ERROR : upload/testfile/testfile.txt: failed to transfer file from cache to remote: object not found
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: >close: err=failed to transfer file from cache to remote: object not found
2018/01/26 21:23:50 ERROR : upload/testfile/testfile.txt: RWFileHandle.Flush error: failed to transfer file from cache to remote: object not found
2018/01/26 21:23:50 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Flush: err=failed to transfer file from cache to remote: object not found
2018/01/26 21:23:50 DEBUG : &{upload/testfile/testfile.txt (rw)}: Release:
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: RWFileHandle.Release nothing to do
2018/01/26 21:23:50 DEBUG : &{upload/testfile/testfile.txt (rw)}: >Release: err=
2018/01/26 21:23:50 DEBUG : /: Lookup: name=“upload”
2018/01/26 21:23:50 DEBUG : /: >Lookup: node=upload/, err=
2018/01/26 21:23:50 DEBUG : upload/: Attr:
2018/01/26 21:23:50 DEBUG : upload/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:23:50 DEBUG : /: Lookup: name=“upload”
2018/01/26 21:23:50 DEBUG : /: >Lookup: node=upload/, err=
2018/01/26 21:23:50 DEBUG : upload/: Attr:
2018/01/26 21:23:50 DEBUG : upload/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:23:50 DEBUG : upload/: Lookup: name=“testfile”
2018/01/26 21:23:50 DEBUG : upload/: >Lookup: node=upload/testfile/, err=
2018/01/26 21:23:50 DEBUG : upload/testfile/: Attr:
2018/01/26 21:23:50 DEBUG : upload/testfile/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxr-xr-x, err=
2018/01/26 21:23:50 DEBUG : upload/testfile/: Lookup: name=“testfile.txt”
2018/01/26 21:23:50 DEBUG : upload/testfile/: >Lookup: node=upload/testfile/testfile.txt, err=
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: Attr:
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: >Attr: a=valid=1m0s ino=0 size=0 mode=-rw-r–r--, err=
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: Setattr: a=Setattr [ID=0xb510 Node=0x4 Uid=0 Gid=0 Pid=31426] atime=2018-01-26 21:21:21 +0000 UTC mtime=2013-12-17 12:37:14 +0000 UTC handle=INVALID-0x0
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: File.applyPendingModTime OK
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: >Setattr: err=
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: Attr:
2018/01/26 21:23:50 DEBUG : upload/testfile/testfile.txt: >Attr: a=valid=1m0s ino=0 size=0 mode=-rw-r–r--, err=
2018/01/26 21:24:15 DEBUG : upload/testfile: Removed empty directory

It seems to me that the file gets deleted while it is still being transferred. Am I right?

What version are you using @B4dM4n? I am on v 1.39 (no beta)

PS: Sorry, for putting the log here in plaintext but didn’t know a why around it.

I’m running a rclone master build. This one is a few days old.

The next file i extracted didn’t get uploaded, it got size 0.
Now i have a log of a successful and an failed upload: failed log

Let’s see how the two bad ones compare to the good one.

Well, one difference is the cache being cleaned while the file is still being transferred which seems to be the reason for the error entries at the end of the file (failed to transfer to remote)
That also happened to me.

Besides that I don’t see any difference.

€dit: I just tried the latest beta available for AMD64-Linux on beta.rclone.org which is 1.39.045 (the following one are just available for windows somehow) but the problem persists there as well and btw. log function
seems to be broken in that particular version.

€dit2: @B4dM4n, you said you’re using unrar to extract your files and it is working flawlessly, right? A workaround could be to instead of using the archive extractor of JDownloader one could install a cronjob to check a specific folder for rar-files and extract them to the mount.
How does that sound to you?

Hmm, that doesn’t look good.

Any ideas what is happening there?

I’m currently not using JD for most tasks. The tests i ran were just to collect logs.
You could probably use the “Event Script” plugin to extract your files, but i assume it would be hard to get it working right.
A simpler approach could be to use the JD integrated extractor and move the extracted files to a rclone mount afterwards.

Not right now. I currently think it happens because of #2021, but i couldn’t find a cause jet.
I will try to debug #2021 and see if fixing it resolves the extraction issue.