Rclone is giving this message in the logs. This is fine, but the problem is that when this happens, the timestamps are no longer handled as usual. The files get the local time instead of the file timestamp.
When files are not spooled the timestamps are correct.
Aug 15 15:23:04 Debian-104-buster-64-LAMP rclone[15253]: mega root '': Target remote doesn't support streaming uploads, creating temporary local FS to spool file
Help anyone?
This looks like a bug...
Try working around it with --vfs-cache-mode writes
Yes, I thought it might be. I'll post a bug. One note though. Since the files are rather large the spooling mechinism gets rid of the file as soon as it's uploaded. I can't have the files sitting around in the cache.
With vfs-cache-mode write ON, (I always have vfs-cache-mode OFF) how can I set it up so that as soon as the file is uploaded, it's purged?
Thanks
Just set the cache expiry time really low. You can increase the frequency of the cache expirer too. You could make it disappear in any time you liked with that.
Ok, I'll try that.
Thanks
Well, I reconfigured to use --vfs-cache-mode writes. The files in the cache path have the correct timestamps, the files uploaded to "Mega" have the current upload time.
Any other suggestions I could try?
Will do!
Thanks
I tried the latest beta rclone-beta-latest-linux-amd64.deb.
It's actually worse than before. Now, even the files in the cache don't have the correct timestamps.
I did notice however that initially the files in the cache would have the correct timestamp for a moment,
but then the timestamp would change to the current time. On upload to "mega" I saw the same behavior.
Here is the mega and Google configs. Pretty basic.
Transfer direction. Google -> Mega
Mega
--config=/root/.config/rclone/rclone.conf
--vfs-cache-mode writes
--cache-db-purge
--log-level DEBUG
--syslog
--fast-list "Mega Drive Root Drive"
"/mnt/Mega Drive Root Drive Not Encrypted"
Google
--config=/root/.config/rclone/rclone.conf
--vfs-cache-mode=off
--log-level DEBUG
--syslog
--fast-list "G Suite Google Drive Root Drive" "/mnt/G Suite Google Drive Root Drive Not Encrypted"
NOTE: I'm going to run some test and I'll post in detail what's happening.
Thanks
OK,
This is crazy weird!
From Google -> To Mega (Google files have various timestamps)
|_
| vfs Folder = Files have correct Timestamp and some don't.
| vfsMeta Folder = Files have correct Timestamp and some don't.
| Mega Folder = Files have correct Timestamp and some don't.
|
|_ Once all the files have been uploaded then refresh all the folders.
|
| vfs Folder = Files have upload time instead & date not correct Timestamp.
| vfsMeta Folder = Files have upload time instead & date not correct Timestamp.
| Mega Folder = Files have upload time instead & date not correct Timestamp.
(Notice not just mega but vfs & vfsMeta cache folders too are wrong) using latest beta
Is there any thing else I can do/check?
By the way, I really appreciate all the help you've given!!\
K.C
The timestamps of the files in the VFS cache probably aren't too relevant.
What is important is what times the mount show and what times you see on the mega web interface.
Do these agree?
Yes, they agree. But they're wrong. The original timestamps of the files are lost. Now all the files show the same date in the mount and Mega web interface. They have the current date(uploading date Aug 19). Slightly different time by minutes or seconds depending on the file size because of uploading.
K.C.
Last night I fixed a but with the VFS upload code and timestamps. It is in the right area to be a fix for you too. So if you try the latest beta and --vfs-cache-mode writes it may work.
Thanks
I'll get it.
I really thought I had good news for you.
This time all the timestamps were correct, both in the cache and on Mega.
However, after a few minutes, the timestamps of the files in the cache "changed" back to the current time by rlocne, and then rclone changed the timestamps on Mega too back to the current time too.
It's seems like there is something rclone is doing with the timestamps after it's finished uploading.
K.C.
Hmm it sounds like when the update came back from mega it had the wrong times.
That is a bit strange.
I wonder if mega is changing the timestamps for some reason.
If you just try rclone copy do the files arrive with the correct timestamps and do they stay that way?
Same problem using rclone copy.
K.C.
I just had a look through the docs. The mega backend doesn't support modification times. This is probably because there isn't a way of setting them when you upload a file.
So without changes to the backend this isn't going to work.
I don't know if this is relevant, but it seems like the MEGA API "DOES" support modtimes.
Please let me know if this is relevant to this issue.
Thanks
K. Carter
| void mega::MegaApi::startUpload | ( | const char * | localPath , |
|---|---|---|---|
| MegaNode * | parent , | ||
| int64_t | mtime , | ||
| MegaTransferListener * |
listener = NULL
|
||
| ) |
Upload a file or a folder with a custom modification time.
If the status of the business account is expired, onTransferFinish will be called with the error code MegaError::API_EBUSINESSPASTDUE. In this case, apps should show a warning message similar to "Your business account is overdue, please contact your administrator."
Parameters
| localPath | Local path of the file |
|---|---|
| parent | Parent node for the file in the MEGA account |
| mtime | Custom modification time for the file in MEGA (in seconds since the epoch) |
| listener | MegaTransferListener to track this transfer |
Note
The custom modification time will be only applied for file transfers. If a folder is transferred using this function, the custom modification time won't have any effect,
| void mega::MegaApi::startUpload | ( | const char * | localPath , |
|---|---|---|---|
| MegaNode * | parent , | ||
| int64_t | mtime , | ||
| bool | isSourceTemporary , | ||
| MegaTransferListener * |
listener = NULL
|
||
| ) |
Upload a file or a folder with a custom modification time.
If the status of the business account is expired, onTransferFinish will be called with the error code MegaError::API_EBUSINESSPASTDUE. In this case, apps should show a warning message similar to "Your business account is overdue, please contact your administrator."
Parameters
| localPath | Local path of the file |
|---|---|
| parent | Parent node for the file in the MEGA account |
| mtime | Custom modification time for the file in MEGA (in seconds since the epoch) |
| isSourceTemporary | Pass the ownership of the file to the SDK, that will DELETE it when the upload finishes. This parameter is intended to automatically delete temporary files that are only created to be uploaded. |
| listener | MegaTransferListener to track this transfer |
| void mega::MegaApi::startUpload | ( | const char * | localPath , |
|---|---|---|---|
| MegaNode * | parent , | ||
| const char * | fileName , | ||
| MegaTransferListener * |
listener = NULL
|
||
| ) |
Upload a file or folder with a custom name.
If the status of the business account is expired, onTransferFinish will be called with the error code MegaError::API_EBUSINESSPASTDUE. In this case, apps should show a warning message similar to "Your business account is overdue, please contact your administrator."
Parameters
| localPath | Local path of the file or folder |
|---|---|
| parent | Parent node for the file or folder in the MEGA account |
| fileName | Custom file name for the file or folder in MEGA |
| listener | MegaTransferListener to track this transfer |
| void mega::MegaApi::startUpload | ( | const char * | localPath , |
|---|---|---|---|
| MegaNode * | parent , | ||
| const char * | fileName , | ||
| int64_t | mtime , | ||
| MegaTransferListener * |
listener = NULL
|
||
| ) |
Upload a file or a folder with a custom name and a custom modification time.
If the status of the business account is expired, onTransferFinish will be called with the error code MegaError::API_EBUSINESSPASTDUE. In this case, apps should show a warning message similar to "Your business account is overdue, please contact your administrator."
Parameters
| localPath | Local path of the file |
|---|---|
| parent | Parent node for the file in the MEGA account |
| fileName | Custom file name for the file in MEGA |
| mtime | Custom modification time for the file in MEGA (in seconds since the epoch) |
| listener | MegaTransferListener to track this transfer |
The custom modification time will be only applied for file transfers. If a folder is transferred using this function, the custom modification time won't have any effect
| void mega::MegaApi::startUpload | ( | const char * | localPath , |
|---|---|---|---|
| MegaNode * | parent , | ||
| const char * | appData , | ||
| const char * | fileName , | ||
| int64_t | mtime , | ||
| MegaTransferListener * |
listener = NULL
|
||
| ) |
Upload a file or a folder with a custom name and a custom modification time.
If the status of the business account is expired, onTransferFinish will be called with the error code MegaError::API_EBUSINESSPASTDUE. In this case, apps should show a warning message similar to "Your business account is overdue, please contact your administrator."
Parameters
| localPath | Local path of the file |
|---|---|
| parent | Parent node for the file in the MEGA account |
| appData | Custom app data to save in the MegaTransfer object The data in this parameter can be accessed using MegaTransfer::getAppData in callbacks related to the transfer. If a transfer is started with exactly the same data (local path and target parent) as another one in the transfer queue, the new transfer fails with the error API_EEXISTS and the appData of the new transfer is appended to the appData of the old transfer, using a '!' separator if the old transfer had already appData. |
| fileName | Custom file name for the file in MEGA |
| mtime | Custom modification time for the file in MEGA (in seconds since the epoch) |
| listener | MegaTransferListener to track this transfer |
The custom modification time will be only applied for file transfers. If a folder is transferred using this function, the custom modification time won't have any effect
that will work