Massive amount of storage on remote used by invisible temp files

[theUnlikely]

What is the problem you are having with rclone?

I seem to be having a weird problem with rclone. Apparently it was having some problems uploading to one of my remotes for a little while and left around 10TB of temp files on the remote, but they can't be seen by ncdu -e nor can I find them with ls -a (I didn't think that was possible). ncdu shows 1TiB and 779 files, but the uptobox web interface says 11.6TB and 3,455 files. The files don't actually show up in the web interface though. Any idea how to get rid of these temp files? I only know about their existence because uptobox support showed me a screenshot of four of the temp files in the root directory.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

• os/version: debian 10.13 (64 bit)
• os/kernel: 5.10.0-0.bpo.7-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.18.6
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Uptobox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=RClone Uploader

[Service]
Type=simple

ExecStart=%h/bin/rclone move %h/Stuff/Local utbcrypt: \
    --config=%h/.config/rclone/rclone.conf \
    --drive-chunk-size 64M \
    --tpslimit 10 \
    -vvv \
    --delete-empty-src-dirs \
    --fast-list \
    --use-mmap \
    --bwlimit 30M \
    --transfers=2 \
    --checkers=4 \
    --drive-stop-on-upload-limit \
    --exclude "files/**" \
    --exclude "media/**"

StandardOutput=file:%h/scripts/rclone_uploader.log
Restart=on-failure

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = *********
client_secret = *********
scope = drive
token = *********

[uptobox]
type = uptobox
access_token = **********

A log from the command with the -vv flag

The log file is absolutely massive with mostly no errors. I will paste one of the errors below. It happens 3 times in the log file. Even one error is too big to fit in pastebin's 512KB size limit so I removed most of it as it just seems to repeat.
https://pastebin.com/pZ9uVCyc

[kapitainsky]

I think it is not really rclone problem but uptobox.

Their API is very basic and does not give any special control over temp files (which I guess are unfinished uploads). When rclone uploads file it gets upload URL from uptobox and sends file there - it is success or failure. That's it.

Now it looks like their system does not clean failed uploads and leaves garbage behind which counts against your storage space. They either have to clean it or at least make visible in their web interface.

BTW - you should upgrade your rclone to the latest version. Things are constantly fixed and improved with every new release.

[theUnlikely]

Is there no way for me to see these files myself through the mount?

[kapitainsky]

There is some possibility that these are files without correct names encryption.

you can try:

rclone ls uptobox: --include "rcloneTemp*"

and if by any chance they are there just delete them with:

rclone delete uptobox: --include "rcloneTemp*"

run first with --dry-run -vv to make sure you don't delete too much:)

But I doubt it will work as you would see these files using uptobox web interface. You have to talk to uptobox and ask them to delete them for you - or fix their system.

[ncw (Nick Craig-Wood)]

There were some fixes for uptobox recently so it is worth trying v1.63.0

[theUnlikely]

Thanks, just updated yesterday so hopefully no more leftover files get stuck there. Still waiting on a response from their support for removing the files.