this is in regards to a previous thread, here, where i wanna share additional thoughts.
This is just to say that at large scale, the above issue is becoming a bit of a concern. with a large remote (approximately 1 million files) there are getting to be 30-50 directories that can't be removed, some of them quite long, because of the ~tmp file issue.
recently, i have started a sweep of the remote (not the crypt wrap) with
rclone delete remote:path --include ~tmp*
to have rclone sweep the remote and remove the issue files, at which point the directories can be removed. not sure if this is a good idea and will inhibit the healthy function of the remote, but i'm testing it. on a million-file remote i'll let the rclone team decide where to take this. cheers on the wonderful product
I'm not sure what rclone would do in this situation as you have some 3rd party leaving temp files around and not cleaning them up.
If you are using Windows, you'd probably want a snapshot and then copy I'd imagine. The goal being to have a 'quiet' / non changing source to copy from or you get odd results.
i'm not familiar enough with the inner workings of OneDrive Business/Sharepoint to know who is leaving the temp files scattered about. i do know that nobody but rclone is using the remote.
my thoughts are rclone should be able to delete the temp file as i can easily delete it with a browser, or even with rclone when not going through the crypt wrap.
If you have a directory and something else dropped something there, rclone isn't going to delete because it's not empty and something else dropped a file in there i.e your tmp files.
That's so rclone doesn't destroy something it wasn't instructed to do.
that's the million dollar question here. we are sir walter raleigh and the source of those temp files is el dorado. you are a developer at rclone. i highly assume if rclone's crypt was depositing ~tmp files by standard procedure you'd know about it.
all i can think of is the inner workings of sharepoint then. and as i mentioned, i'm not familiar with the inner plumbing over there. it must be something about how they process files through microsoft graph.
and since then i have moved on to a 140+ lines of python code
create a permanent snapshot, mount that snapshot as a local dir, backup from that snapshot and then delete that snapshot.
i know. because as previously mentioned, there is no log of the actual ~tmp file. however, below is a log of the occasion. what happens is, the temp files get created over time in directories. once the directories are deleted from natural causes on the file server, rclone dumps the files but gets stuck on the temp file and therefore can't delete the directory.
not good. got a BIOS issue that i'm waiting for a reply on from the company it's the last step though and i'm free of the harbor and sailing
jeepers. that's pretty extensive.
2022/06/30 15:35:13 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "sync" "--verbose" "-vvv" "--log-file=TBAccount.txt" "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"]
2022/06/30 15:35:13 DEBUG : Creating backend with remote "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"
2022/06/30 15:35:13 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/30 15:35:13 DEBUG : fs cache: renaming cache item "E:\\TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" to be canonical "//?/E:/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:13 DEBUG : Creating backend with remote "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images"
2022/06/30 15:35:14 DEBUG : Creating backend with remote "OneDriveWy:APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/v0ancu2jmn60gds35k1237mdoc"
2022/06/30 15:35:14 DEBUG : One drive root 'APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/v0ancu2jmn60gds35k1237mdoc': Token expired but no uploads in progress - doing nothing
2022/06/30 15:35:14 DEBUG : OneDriveWy: Loaded invalid token from config file - ignoring
2022/06/30 15:35:15 DEBUG : Saving config "token" in section "OneDriveWy" of the config file
2022/06/30 15:35:15 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2022/06/30 15:35:15 DEBUG : OneDriveWy: Saved new token in config file
2022/06/30 15:35:21 DEBUG : Creating backend with remote "OneDriveWy:APT Server/TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:24 DEBUG : fs cache: renaming cache item "OneDriveWyEncryptednew:TBAccount\\Costech Update 08-07-2013\\Website\\App_Themes\\Computer\\images" to be canonical "OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images"
2022/06/30 15:35:25 DEBUG : bottom_bg.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : bottom_bg.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : button_bg.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : search_btn_hover2.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : search_btn_hover2.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : no-icon-available.png: Unchanged skipping
2022/06/30 15:35:25 DEBUG : shortcuts_bg.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : shortcuts_bg.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : sort-order-descending.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : top-nav-bg.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : top-nav-bg.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : up.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : up.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Icons/~tmp94_prboap0fdamm43ms9m89n9l77g: Skipping undecryptable file name: illegal base32 data at input byte 0
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex_m.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex_m.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid_amex.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid_amex.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cid.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_amex.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp_m.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/checkhelp_m.jpg: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa_m.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : PaymentInstruments/cvv_visa_m.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_00.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_00.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_05.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_01.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_01.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_09.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_03.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_03.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_06.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_06.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_02.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_02.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_04.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_04.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_08.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_08.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_05.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': Waiting for checks to finish
2022/06/30 15:35:25 DEBUG : ratings/rate_07.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_07.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_09.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : ratings/rate_10.gif: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/06/30 15:35:25 DEBUG : ratings/rate_10.gif: Unchanged skipping
2022/06/30 15:35:25 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': Waiting for transfers to finish
2022/06/30 15:35:25 DEBUG : Waiting for deletions to finish
2022/06/30 15:35:25 INFO : Icons: Removing directory
2022/06/30 15:35:26 DEBUG : Icons: Failed to Rmdir: directory not empty
2022/06/30 15:35:26 DEBUG : Encrypted drive 'OneDriveWyEncryptednew:TBAccount/Costech Update 08-07-2013/Website/App_Themes/Computer/images': failed to delete 1 directories
2022/06/30 15:35:26 INFO : There was nothing to transfer
2022/06/30 15:35:26 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 59 / 59, 100%
Deleted: 0 (files), 1 (dirs)
Elapsed time: 12.1s
2022/06/30 15:35:26 DEBUG : 6 go routines active
That's what I'm trying to get at though, rclone isn't doing that as that's not a function of rclone.
That's not a function of Sharepoint/Onedrive either "naturally". You'd have some other app writing into the area.
If rclone was writing it, you'd get a log entry somewhere as rclone is great, but doesn't make up files
Cant' you find that file and see what created it? There should be an audit trail of the file / owner / etc. I can see all that stuff when I look at OneDrive or Sharepoint and you can see the audit logs. The compliance portal I believe is where you can look but I'm not sure what level of access you have.
You should be able to see the file and who created it at a minimum.