~tmp file issue

this is in regards to a previous thread, here, where i wanna share additional thoughts.

This is just to say that at large scale, the above issue is becoming a bit of a concern. with a large remote (approximately 1 million files) there are getting to be 30-50 directories that can't be removed, some of them quite long, because of the ~tmp file issue.

recently, i have started a sweep of the remote (not the crypt wrap) with

rclone delete remote:path --include ~tmp* 

to have rclone sweep the remote and remove the issue files, at which point the directories can be removed. not sure if this is a good idea and will inhibit the healthy function of the remote, but i'm testing it. on a million-file remote :stuck_out_tongue: i'll let the rclone team decide where to take this. cheers on the wonderful product :slight_smile:

I'm not sure what rclone would do in this situation as you have some 3rd party leaving temp files around and not cleaning them up.

If you are using Windows, you'd probably want a snapshot and then copy I'd imagine. The goal being to have a 'quiet' / non changing source to copy from or you get odd results.

i'm not familiar enough with the inner workings of OneDrive Business/Sharepoint to know who is leaving the temp files scattered about. i do know that nobody but rclone is using the remote.

my thoughts are rclone should be able to delete the temp file as i can easily delete it with a browser, or even with rclone when not going through the crypt wrap.

Rclone should never delete anything on its own as there are way too many permutations that would create havoc.

You'd really want to 'quiet' the source and do a copy rather than trying to clean up after the fact.

I'm sure @asdffdsa would chime in on how to properly copy from Windows.

well by delete i mean rclone doing a check, seeing the file is not in the source, and deleting it.

That's a 'sync' from source to destination and that's how it works already.

in this case, what's keeping rclone from deleting a file that's not in the source?

That's been answered in the other post:

If you have a directory and something else dropped something there, rclone isn't going to delete because it's not empty and something else dropped a file in there i.e your tmp files.

That's so rclone doesn't destroy something it wasn't instructed to do.

yeah. manually uploading a random .pdf file into a crypt remote using a browser results in the following:

2022/06/30 14:03:14 DEBUG : A-AF31-201635-TT/OE-28275.pdf: Skipping undecryptable file name: illegal base32 data at input byte 2

i don't see anything about the temp files in the logs for issue directories, but i assume it's a similar situation.

well, i'll leave it to you guys. for now, i'll just use an rclone filter to sweep the directory for temp files and delete them. cheers.

jared

That error means you are mixing a crypt remote and non crypted remote and data is mixed together.

You'd have to share more details.

well yeah. my thinking is rclone has the same reaction to the temp files, because they were put there by something other than rclone's crypt.

What else would be writing data into the crypt location?

that's the million dollar question here. we are sir walter raleigh and the source of those temp files is el dorado. you are a developer at rclone. i highly assume if rclone's crypt was depositing ~tmp files by standard procedure you'd know about it.

I'm not a developer. I don't work for rclone.

Rclone doesn't make "tmp" files and drop them in your crypt.

Rclone only copies what you ask it so I really don't know why you'd have non crypt files in your crypt location as that's not rclone doing that.

you certainly contribute a lot :slight_smile:

all i can think of is the inner workings of sharepoint then. and as i mentioned, i'm not familiar with the inner plumbing over there. it must be something about how they process files through microsoft graph.

Your example though isn't about a temp file.

That's looks to be a normal PDF file that is in your crypt remote location.

You'd want to figure out how you are mixing things up to get that in there and fix what is causing it.

hi jared, hope all is well with that super server you are building.

https://forum.rclone.org/t/tmp-files-on-sharepoint-remote/25385

whenever i backup on windows, using rclone, 7zip, fastcopy, i always use VSS as the source.

in the past, i used a simple four line .cmd script to create a temporary snapshot
as detailed at
https://forum.rclone.org/t/i-want-to-share-how-to-enable-vss-for-rclone/11644

and since then i have moved on to a 140+ lines of python code
create a permanent snapshot, mount that snapshot as a local dir, backup from that snapshot and then delete that snapshot.

i know. because as previously mentioned, there is no log of the actual ~tmp file. however, below is a log of the occasion. what happens is, the temp files get created over time in directories. once the directories are deleted from natural causes on the file server, rclone dumps the files but gets stuck on the temp file and therefore can't delete the directory.

not good. got a BIOS issue that i'm waiting for a reply on from the company :confused: it's the last step though and i'm free of the harbor and sailing :slight_smile:

jeepers. that's pretty extensive.

2022/06/30 15:35:13 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "sync" "--verbose" "-vvv" "--log-file=TBAccount.txt" "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"]
2022/06/30 15:35:13 DEBUG : Creating backend with remote "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"
2022/06/30 15:35:13 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/30 15:35:13 DEBUG : fs cache: renaming cache item "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" to be canonical "//?/E:/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:13 DEBUG : Creating backend with remote "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"
2022/06/30 15:35:14 DEBUG : Creating backend with remote "OneDriveWy:APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/v0ancu2jmn60gds35k1237mdoc"
2022/06/30 15:35:14 DEBUG : One drive root 'APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/v0ancu2jmn60gds35k1237mdoc': Token expired but no uploads in progress - doing nothing
2022/06/30 15:35:14 DEBUG : OneDriveWy: Loaded invalid token from config file - ignoring
2022/06/30 15:35:15 DEBUG : Saving config "token" in section "OneDriveWy" of the config file
2022/06/30 15:35:15 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2022/06/30 15:35:15 DEBUG : OneDriveWy: Saved new token in config file
2022/06/30 15:35:21 DEBUG : Creating backend with remote "OneDriveWy:APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:24 DEBUG : fs cache: renaming cache item "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" to be canonical "OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:25 DEBUG : bottom_bg.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : bottom_bg.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : button_bg.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : search_btn_hover2.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : search_btn_hover2.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : no-icon-available.png: Unchanged skipping
2022/06/30 15:35:25 DEBUG : shortcuts_bg.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : shortcuts_bg.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : sort-order-descending.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : top-nav-bg.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : top-nav-bg.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : up.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : up.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Icons/~tmp94_prboap0fdamm43ms9m89n9l77g: Skipping undecryptable file name: illegal base32 data at input byte 0
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex_m.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex_m.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid_amex.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid_amex.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp_m.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp_m.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa_m.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa_m.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_00.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_00.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_05.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_01.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_01.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_09.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_03.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_03.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_06.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_06.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_02.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_02.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_04.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_04.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_08.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_08.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_05.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': Waiting for checks to finish
2022/06/30 15:35:25 DEBUG : ratings/rate_07.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_07.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_09.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_10.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_10.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': Waiting for transfers to finish
2022/06/30 15:35:25 DEBUG : Waiting for deletions to finish
2022/06/30 15:35:25 INFO  : Icons: Removing directory
2022/06/30 15:35:26 DEBUG : Icons: Failed to Rmdir: directory not empty
2022/06/30 15:35:26 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': failed to delete 1 directories
2022/06/30 15:35:26 INFO  : There was nothing to transfer
2022/06/30 15:35:26 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                59 / 59, 100%
Deleted:                0 (files), 1 (dirs)
Elapsed time:        12.1s

2022/06/30 15:35:26 DEBUG : 6 go routines active

That's what I'm trying to get at though, rclone isn't doing that as that's not a function of rclone.

That's not a function of Sharepoint/Onedrive either "naturally". You'd have some other app writing into the area.

If rclone was writing it, you'd get a log entry somewhere as rclone is great, but doesn't make up files :slight_smile:

Cant' you find that file and see what created it? There should be an audit trail of the file / owner / etc. I can see all that stuff when I look at OneDrive or Sharepoint and you can see the audit logs. The compliance portal I believe is where you can look but I'm not sure what level of access you have.

You should be able to see the file and who created it at a minimum.

I think those ~tmp files might be something to do with failed mulipart uploads on OneDrive.

I think there might be more forum posts or issues about it but I'm on the move at the moment and can't search.