Files disappearing after upload

[retched1 (Aaron)]

I upload an 8Gb file… I can see the file… and minutes later it is gone… I watched the 8Gb file upload, i saw it in the folder both in Putty and Filezilla… And now it’s gone…

root@rapidseedbox:/home/user/gdrive# cd Movies
root@rapidseedbox:/home/user/gdrive/Movies# ls
— (This folder did have the movie file, now its gone)
root@rapidseedbox:/home/user/gdrive/Movies# 2018/07/10 16:20:07 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 16:20:09 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 16:20:09 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found

version 1.42
Basically followed this -
https://bytesized-hosting.com/pages/rclone-gdrive

Mounted with this -
rclone mount --allow-other --allow-non-empty --cache-db-purge gcrypt: /home/user/gdrive &

[gdrive]
type = drive
client_id =
client_secret =
scope = drive
root_folder_id =
service_account_file =
token = removed

[gcache]
type = cache
remote = gdrive:/gdrive
plex_url = http://public-ip:32400
plex_username = plexusername
plex_password = removed
chunk_size = 5M
info_age = 24h
chunk_total_size = 32G
(I dont have a plex token on this conig, i do on another without this issue)

[gcrypt]
type = crypt
remote = gcache:/gcrypt
filename_encryption = standard
directory_name_encryption = true
password =
password2 =

Worth mentioning
I do not have this issue on another host, with the same config (i believe)

[retched1 (Aaron)]

Just uploaded it again - file - “Blockers (2018) [Bluray-1080p].mkv”

root@rapidseedbox:/home/user/gdrive# cd Movies
root@rapidseedbox:/home/user/gdrive/Movies# ls
2018/07/10 16:58:49 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 16:58:49 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 16:58:49 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
Blockers (2018) [Bluray-1080p].mkv
root@rapidseedbox:/home/user/gdrive/Movies#

During testing i DID have 4 files uploaded to this gdrive, which were deleted… I’m guessing those four not found items are them… But did purge the cache so not sure why they are still “missing”

I do not see this file on gdrive yet (in browser)

And now the file is gone from the local file system…

root@rapidseedbox:/home/user/gdrive/Movies# 2018/07/10 17:04:18 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:20 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:21 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:21 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:21 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:21 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:04:21 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found

[retched1 (Aaron)]

and now it shows again via putty but filezilla shows a 0 byte size…

root@rapidseedbox:/home/user/gdrive# ls
Movies MoviesUHD Music TVShows
root@rapidseedbox:/home/user/gdrive# cd Movies
root@rapidseedbox:/home/user/gdrive/Movies# ls
2018/07/10 17:12:57 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:12:58 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
2018/07/10 17:12:58 ERROR : 3sn2p7km7msr7dbpgpmj3296k0/chnf3gmdc99inm5jnmrr1nkm3dt4obu3b8dm2nlcpb98bi28motekvgmftmmf6ofon7mbdgc4u0ee: error refreshing object in : in cache fs Google drive root ‘gdrive/gcrypt’: object not found
Blockers (2018) [Bluray-1080p].mkv
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies# ls -l
total 0
root@rapidseedbox:/home/user/gdrive/Movies# ls
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies#
root@rapidseedbox:/home/user/gdrive/Movies#

[Animosity022]

How are you uploading it? What’s your mount command look like?

Is something else removing it elsewhere?

[retched1 (Aaron)]

i am uploading with filezilla to the mounted gdrive folder (same as i do to the working host)

df-h shows
gcrypt: 1.0P 212G 1.0P 1% /home/user/gdrive
pwd shows
/home/user/gdrive/Movies

the mount command is -
rclone mount --allow-other --allow-non-empty --cache-db-purge gcrypt: /home/user/gdrive &

[Animosity022]

Just to make sure I get the flow.

You have a laptop or some other device.
You have the file on that device.
You are connecting to a server with the rclone mount above.
You are using FTP to transfer the file from laptop/some other device to the server with the GD mount.

Is that the right flow? If so, can you maybe capture some debug type logs and see what happens once it finishes? You can do that with a -vv or log level on DEBUG.

[retched1 (Aaron)]

That is my flow

pc with file, uploaded with filezilla via ftp to the gdrive mount point.

how do i enable debug on rclone, not sure where to apply the “-vv”

[retched1 (Aaron)]

this is a ubuntu 16.04 host

[retched1 (Aaron)]

I deleted it all and started over… line by line…

I also did it as “user” and not as root

I did notice that when looking at file permissions there was files (before they were deleted) that were owned as root under the gdrive…

So trying another upload…

[retched1 (Aaron)]

And now I got this on upload of the file -

error uploading Eror 403 Rate Limit Exceeded
googleapi Error 403 rate limit exceeded

guessing i hit a ban for 24 hrs on this gsuite account…

[Animosity022]

That could be the case.

I was going to suggest rather than uploading via FTP, you can just do a rclone copy as you can . just copy your rclone config to the machine.

To get debug, you can just as an example change:

rclone mount --allow-other --allow-non-empty gcrypt: ~/mnt/gdrive &

to

rclone mount --allow-other --allow-non-empty gcrypt: ~/mnt/gdrive -vv -log-file /home/user/log/rclone.log &

[retched1 (Aaron)]

Still happening… The files just delete…

Some of the logs -

2018/07/10 21:41:41 DEBUG : rclone: Version “v1.42” starting with parameters [“rclone” “mount” “–allow-other” “–allow-non-empty” “–cache-db-purge” “gcrypt:” “/home/user/gdrive” “-vv” “–log-file” “/home/user/log/rclone2.log”]
2018/07/10 21:41:41 DEBUG : Using config file from “/home/user/.config/rclone/rclone.conf”
2018/07/10 21:41:41 DEBUG : gcache: wrapped gdrive:gdrive/crypt at root crypt
2018/07/10 21:41:41 DEBUG : gcache: Purging the DB
2018/07/10 21:41:41 INFO : gcache: Cache DB path: /home/user/.cache/rclone/cache-backend/gcache.db
2018/07/10 21:41:41 INFO : gcache: Cache chunk path: /home/user/.cache/rclone/cache-backend/gcache
2018/07/10 21:41:41 INFO : gcache: Chunk Memory: true
2018/07/10 21:41:41 INFO : gcache: Chunk Size: 5M
2018/07/10 21:41:41 INFO : gcache: Chunk Total Size: 32G
2018/07/10 21:41:41 INFO : gcache: Chunk Clean Interval: 1m0s
2018/07/10 21:41:41 INFO : gcache: Workers: 4
2018/07/10 21:41:41 INFO : gcache: File Age: 24h0m0s
2018/07/10 21:41:41 DEBUG : Adding path “cache/expire” to remote control registry
2018/07/10 21:41:41 DEBUG : Adding path “cache/stats” to remote control registry
2018/07/10 21:41:42 DEBUG : Google drive root ‘gdrive/crypt’: Checking for changes on remote
2018/07/10 21:41:42 DEBUG : gcache: wrapped gdrive:gdrive/crypt at root crypt
2018/07/10 21:41:42 DEBUG : gcache: Purging the DB
2018/07/10 21:41:42 INFO : gcache: Cache DB path: /home/user/.cache/rclone/cache-backend/gcache.db
2018/07/10 21:41:42 INFO : gcache: Cache chunk path: /home/user/.cache/rclone/cache-backend/gcache
2018/07/10 21:41:42 INFO : gcache: Chunk Memory: true
2018/07/10 21:41:42 INFO : gcache: Chunk Size: 5M
2018/07/10 21:41:42 INFO : gcache: Chunk Total Size: 32G
2018/07/10 21:41:42 INFO : gcache: Chunk Clean Interval: 1m0s
2018/07/10 21:41:42 INFO : gcache: Workers: 4
2018/07/10 21:41:42 INFO : gcache: File Age: 24h0m0s
2018/07/10 21:41:42 DEBUG : Adding path “cache/expire” to remote control registry
2018/07/10 21:41:42 DEBUG : Adding path “cache/stats” to remote control registry
2018/07/10 21:41:42 DEBUG : Encrypted drive ‘gcrypt:’: Mounting on “/home/user/gdrive”
2018/07/10 21:41:42 DEBUG : Google drive root ‘gdrive/crypt’: All changes were processed. Waiting for more.
2018/07/10 21:41:42 DEBUG : Cache remote gcache:crypt: subscribing to ChangeNotify
2018/07/10 21:41:42 DEBUG : Adding path “vfs/forget” to remote control registry
2018/07/10 21:41:42 DEBUG : : Root:
2018/07/10 21:41:42 DEBUG : : >Root: node=/, err=
2018/07/10 21:41:42 DEBUG : /: Lookup: name=".Trash"
2018/07/10 21:41:42 DEBUG : Cache remote gcache:crypt: list ‘’
2018/07/10 21:41:42 DEBUG : /: Attr:
2018/07/10 21:41:42 DEBUG : : list: error: couldn’t open bucket (crypt)
2018/07/10 21:41:42 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=
2018/07/10 21:41:42 DEBUG : Google drive root ‘gdrive/crypt’: Checking for changes on remote
2018/07/10 21:41:42 DEBUG : : list: read 5 from source
2018/07/10 21:41:42 DEBUG : : list: source entries: [dhpcrsmv02t85703qf4mmacius ih0o392h0cikjvnb53mdvsaobk 1ulp415vivfpqpgi14iun10psc 87qbbuq226ab6jhk5i7gguhmsc 6cksr6pc7cbmu0ctcoik4g7fhg]
2018/07/10 21:41:42 DEBUG : Google drive root ‘gdrive/crypt’: All changes were processed. Waiting for more.
2018/07/10 21:41:42 DEBUG : : list: cached directories: 5
2018/07/10 21:41:42 DEBUG : : list: cached dir: ‘crypt’, cache ts: 2018-07-10 21:41:42.638554897 +0000 UTC m=+1.291568579
2018/07/10 21:41:42 DEBUG : /: >Lookup: node=, err=no such file or directory
2018/07/10 21:41:42 DEBUG : /: Lookup: name=".xdg-volume-info"
2018/07/10 21:41:42 DEBUG : /: >Lookup: node=, err=no such file or directory
2018/07/10 21:41:42 DEBUG : /: Lookup: name=“autorun.inf”
2018/07/10 21:41:42 DEBUG : /: >Lookup: node=, err=no such file or directory
2018/07/10 21:41:42 DEBUG : /: ReadDirAll:
2018/07/10 21:41:42 DEBUG : /: >ReadDirAll: item=5, err=
2018/07/10 21:41:42 DEBUG : /: Lookup: name=".Trash-1000"
2018/07/10 21:41:42 DEBUG : /: >Lookup: node=, err=no such file or directory
2018/07/10 21:41:42 DEBUG : /: Attr:
2018/07/10 21:41:42 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=
2018/07/10 21:41:42 DEBUG : /: ReadDirAll:
2018/07/10 21:41:42 DEBUG : /: >ReadDirAll: item=5, err=

[Animosity022]

2018/07/10 21:41:42 DEBUG : : list: error: couldn’t open bucket (crypt)

That seems like something not quite right. Can you post your rclone.conf with the keys and such masked out?

[retched1 (Aaron)]

Will do when I get home, it should be above though in the first post

[Animosity022]

Oh, I’m sorry as I see it now.

Not having the plex token won’t really impact much as that means it has not authenticated to plex to get a token so something is wrong with the user name or password. I found that you have to do that via the rclone config for it to work.

That being said, if you stop the cache, you can manually clean up the cache files and cache.db as those are right here:

2018/07/10 21:41:41 INFO : gcache: Cache DB path: /home/user/.cache/rclone/cache-backend/gcache.db
2018/07/10 21:41:41 INFO : gcache: Cache chunk path: /home/user/.cache/rclone/cache-backend/gcache

If you do that, before starting it, can you just do a rclone ls gcrypt: and validate it actually spits out all the right info?

If you want to mount without the cache to test if that’s the issue, you can create another entry with the same passwords and run a mount like:

rclone mount noncachecrypt: /gmedia \
   --allow-other \
   --dir-cache-time 48h \
   --vfs-read-chunk-size 32M \
   --vfs-read-chunk-size-limit 2G \
   --log-file /home/someone/logs/somelog.log \
   --umask 002 \
   --log-level INFO

and check to see if you see it there.

[retched1 (Aaron)]

Very similar to this -

New rclone cache feature - files randomly disappear

Same here, although it seems to be much more predictible in my case. I’ve created a new cache remote for my existing Gdrive remote that is wrapped into a crypting remote, so the order is GDrive -> GDCache -> GDCrypt. With nothing currently mounted, i can run the usual commands like ‘ls’ and ‘lsd’ which would deliver expected results. but after 5 minutes (the interval seems to be pretty exact as i’ve done it three times now) all files and folders disappear except one single file that is located…

[Animosity022]

Could be related.

I’d just lose the cache for now and use a vfs-read-chunk config.

[retched1 (Aaron)]

So i reloaded my seedbox
I connected via SSH as my user, not root… (I setup rclone as root last time, this time i installed with sudo user)

• thinking maybe the user and plex didnt have access to the cache or crypt…???

re-setup rclone, new keys etc…
Deleted all the files (manually) from GDrive

mounted rclone

Uploaded one file…

Seems fine so far…