How to back up only new files to onedrive from local folder and then delete?

[batman]

Hi all, this is a wonderful tool and I'm glad it exists!

I have some questions; let me start with what I want to do and then what I've tried.

What I want to do

I want to back up folders from my local machine to onedrive. The local folders have a mix of new and already backed up files, I want to backup/move/copy only the new files and delete everything locally (all files) once the backup is complete.

The items to be backed up are coming from my phone via syncthing. The flow I have in mind is the following:

1. Phone > PC (via syncthing; already set up)
2. PC > onedrive (check and only upload new files; files deleted locally shouldn't affect anything)
3. Delete from PC
4. Do this operation automatically overnight after 12am

Note that step 3 is mostly because I don't currently have enough storage to keep files on my PC but I plan to in the future, so delete for now but it will be nice to have the option to not delete later.

What I've tried

I experimented a little and this is the "ideal" command I ended up with. Ideal meaning I thought this had everything I needed

Command:

rclone move "local\windows\path"  onedrive-rclone:"onedrive/path" --dry-run --log-file="logs.txt" --exclude-from "exclude.txt" --metadata --checksum -vv

Notes:

• Using --checksum because I saw that onedrive can have issues with file sizes and I would prefer not to have renamed duplicates.
• Doing a move so local files are deleted after the operation.
• --metadata seems useful and I thought better to have than to not have.
• exclude.txt is to prevent deletion of syncthing [hidden] files.

exclude.txt:

# https://forum.rclone.org/t/exclude-hidden-files/268/5

.*
.*/**

Version

rclone v1.59.2
- os/version: Microsoft Windows 10 Home 21H2 (64 bit)
- os/kernel: 10.0.19044.2006 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: cmount

Cloud: onedrive

Config file

[onedrive]
type = onedrive
token = {"access_token":"banana","token_type":"Bearer","refresh_token":"another banana","expiry":"queen's 100th birthday"}
drive_id = 69420
drive_type = personal

-vv log

Snippet:

2022/09/30 14:30:24 DEBUG : Screenshot_20220522-193237.png: sha1 = d026a13e370e22f79e4ef240a556e81400951072 OK
2022/09/30 14:30:24 DEBUG : Screenshot_20220522-193237.png: Size and sha1 of src and dst objects identical
2022/09/30 14:30:24 DEBUG : Screenshot_20220522-193237.png: Unchanged skipping
2022/09/30 14:30:24 NOTICE: Screenshot_20220522-193237.png: Skipped delete as --dry-run is set (size 1.294Mi)
2022/09/30 14:30:24 DEBUG : One drive root 'onedrive/path': Waiting for transfers to finish
2022/09/30 14:30:24 INFO  : There was nothing to transfer
2022/09/30 14:30:24 NOTICE: 
Transferred:   	    5.154 GiB / 5.154 GiB, 100%, 0 B/s, ETA -
Checks:             25827 / 25827, 100%
Deleted:             8768 (files), 0 (dirs)
Renamed:             8291
Elapsed time:     12m16.7s

2022/09/30 14:30:24 DEBUG : 3 go routines active

Questions

1. Why are there so many renamed files? Is this like appending "(2)" to a file name? I tried to ctrl+f with "rename" to find specific files that were renamed but it just finds the "Renamed: 8291" portion at the end of the log.
2. What mistakes have I made?
3. Am I doing anything unecessary?
4. Is there a better way to do this? Simpler the better!
5. What's the DEBUG : 3 go routines active all about?

[Animosity022]

You'd have to share the full log as you've asked some questions, but there isn't anything the log to really explain them as you've got a snippet of a log.

[Ole (Ole Frost)]

@batman

• Am I doing anything unecessary?
• Is there a better way to do this? Simpler the better!

What specific need(s) made you choose not to use the built-in OneDrive client of Windows?
and not to use the photo upload of the OneDrive app for your phone?

[batman]

Apologies, I tried to use pastebin but I got an error because it was over 512Kb. Do you know where I can paste the whole log?

[batman]

Ah good question!

I want to upload to a specific path. Doesn't seem like I can do this on desktop.

And on mobile, everything gets uploaded to the camera roll with the automatic backup. For example, both screenshots and photos get uploaded to the same folder and I want it better organized than that.

[Ole (Ole Frost)]

@batman

I want to upload to a specific path. Doesn't seem like I can do this on desktop.

Not sure I fully understand. How about making syncthing copy the phone content to

C:\Users\batman\OneDrive\some\folder\in\onedrive\

I think you can make it more robust by finding the OneDrive folder in an environment variable, haven't checked (yet).

Do you have other needs not supported by OneDrive on Windows?

@batman

And on mobile, everything gets uploaded to the camera roll with the automatic backup. For example, both screenshots and photos get uploaded to the same folder and I want it better organized than that.

I don't know syncthing, so really can't compare it to the phone app for OneDrive - just trying to ask some good questions to help you see any blind spots.

Perhaps, you could do the organization directly in OneDrive using rclone after the phone app has uploaded them (by a nightly script)? If so, then there will be no need for them to hit the PC.

[batman]

Not sure I fully understand. How about making syncthing copy the phone content to
C:\Users\batman\OneDrive\some\folder\in\onedrive\

When I tried last, it seems I will have to add the onedrive folder locally first, meaning it will have to be downloaded to the PC

I don't know syncthing, so really can't compare it to the phone app for OneDrive - just trying to ask some good questions to help you see any blind spots.

Thank you! I appreciate the help

Syncthing is basically like rclone but doesn't connect to cloud services. I can install syncthing on two pr more devices and keep folders in sync and/or backup

Perhaps, you could do the organization directly in OneDrive using rclone after the phone app has uploaded them (by a nightly script)? If so, then there will be no need for them to hit the PC.

Good idea. This sounds appealing but might be a mess especially considering I want to back up things within various subfolders

[batman]

@Animosity022 I tired with a smaller upload but that was still too big for pastebin and similar services I tried. I cannot paste it here either, I get Body is limited to 32000 characters; you entered 673410.

[Animosity022]

In the template, we offer some advice / options:

You should use 3 backticks to begin and end your paste to make it readable. Or use a service such as https://pastebin.com or https://gist.github.com/

[Ole (Ole Frost)]

@batman

When I tried last, it seems I will have to add the onedrive folder locally first, meaning it will have to be downloaded to the PC

This isn't the case anymore, now the OneDrive client for Windows is using On-demand downloads by default:
Save disk space with OneDrive Files On-Demand for Windows
Sync files with OneDrive Files on Demand

and you can optionally activate Storage Sense, to automatically free up space of unused locally downloaded cloud content:
Use OneDrive and Storage Sense in Windows 10 to manage disk space

@batman

especially considering I want to back up things within various subfolders

Sounds like your need is more than the photo upload/backup offered in the OneDrive Phone app, so you probably need syncthing and the PC.

@batman

Thank you! I appreciate the help

Thanks!

[batman]

Forgot about the gist!

Here it is: rclone logs using -vv · GitHub

[batman]

Wow on demand sync is nice! Didn't know about that thank you

Will read into that, sounds good.

One more thing; do you know if onedrive will only backup new items? A few years ago things got duplicated when using the android app

[Ole (Ole Frost)]

@batman

One more thing; do you know if onedrive will only backup new items? A few years ago things got duplicated when using the android app

Very good question!

It only copies new items, but not sure exactly how the app compares to decide.

Seems like it compares all the photos on the phone (mine is iOS) to OneDrive on initial start - it takes quite some time (hours) so make sure have the phone plugged in and OneDrive in foreground to speed things up - I also remove the cover to maximize cooling (to reduce battery wear).

I am not sure whether it compares to everything in Pictures or Camera Roll and whether it compares on folder level, but I am sure it uploads to year (and optionally month) folders in Pictures/Camera Roll.

I would make a backup of Camera Roll before activating; to allow for easy roll back in case of unexpected behavior.

@batman

Forgot about the gist!

Here is is: rclone logs using -vv · GitHub

I can see that the debug log has a lot of photos from WhatApp, I doubt the OneDrive photo upload will find and upload those. It probably only looks into the CameraRoll folder of your phone. Seems like you should stick to syncthing!

Does syncthing copy everything from the phone or just deltas?

If deltas, then sending directly into the local OneDrive folder is worth considering, to avoid the nightly upload.

If full, then you are probably better of with the move and nightly upload outlined - otherwise the OneDrive client might upload everything each time. You could test to see, I am not sure.

@batman

Using --checksum because I saw that onedrive can have issues with file sizes and I would prefer not to have renamed duplicates.

I doubt you will see them on upload, I only remember seeing them on downloads - and doubt --checksum will help in that case, so I would remove to keep things simple and as close to default as possible (at least until an issue arises).

@batman

--metadata seems useful and I thought better to have than to not have.

OneDrive doesn't support this, so I suggest you remove to make things simple.
(https://rclone.org/overview/#features)

@batman

Why are there so many renamed files? Is this like appending "(2)" to a file name? I tried to ctrl+f with "rename" to find specific files that were renamed but it just finds the "Renamed: 8291" portion at the end of the log.

The renamed in the end of your log are the files moved, try searching for "skipped move" in the log.

@batman

What's the DEBUG : 3 go routines active all about?

Low level debug info telling how many Go routines were active at program end; only rclone programmers can use this info. 3 go routines is fine in your situation.

[batman]

@Ole

Seems like it compares all the photos on the phone (mine is iOS) to OneDrive on initial start - it takes quite some time (hours) so make sure have the phone plugged in and OneDrive in foreground to speed things up - I also remove the cover to maximize cooling (to reduce battery wear).

Thanks this is useful. Haha yes good move trying to keep it cool!

@Ole

Does syncthing copy everything from the phone or just deltas?

If deltas, then sending directly into the local OneDrive folder is worth considering, to avoid the nightly upload.

Ah so it's deltas, but I've moved to a new device. Previously I was using an old android phone with syncthing + an app called autosync by metactrl (this uploaded deltas to onedrive and deleted locally), that phone suddenly died and I came to rclone to replace autosync.

Basically it is deltas but since I've moved to a new device (old android > PC) it will have to, for the first run, look over every file. Once this is done, it'll just be new files/deltas.

@Ole

I doubt you will see them on upload, I only remember seeing them on download

Oh my bad I thought this was what was used to check onedrive if the file existed or not. So if there was a.jpg locally and on onedrive with slightly different sizes, checksum would allow rclone to know it already exists.

@Ole

OneDrive doesn't support this, so I suggest you remove to make things simple.

Thank you!

@Ole

The renamed in the end of your log are the files moved, try searching for "skipped move" in the log.

The moved files are logged as renamed? That's a little unexpected haha. But if so, then I think my move and delete were successful! Minus --metadata and --checksum of course

@Ole

Low level debug info telling how many Go routines were active at program end

Thanks

[Ole (Ole Frost)]

@batman

@Ole

I doubt you will see them on upload, I only remember seeing them on download

Oh my bad I thought this was what was used to check onedrive if the file existed or not. So if there was a.jpg locally and on onedrive with slightly different sizes, checksum would allow rclone to know it already exists.

rclone by default (conceptually) performs these two commands to determine if a file needs to be transferred:

rclone lsl source:some/folder/somefile
rclone lsl target:some/folder/somefile

if they both show the same output, then the file is already up-to-date and skipped (and deleted if using move).

If you add --checksum then it also performs these commands (when source=Windows and target=OneDrive):

rclone sha1sum source:some/folder/somefile
rclone sha1sum target:some/folder/somefile

If the size from the first set of commands and the hashsum from the second set are the same, then the file is already up-to-date and skipped (and deleted if using move). Note: that this option ignores the modification time of the file.

More details here: https://rclone.org/docs/#c-checksum

@batman

The moved files are logged as renamed? That's a little unexpected haha

Haha, this is the typical Windows versus Linux terminology confusion

Windows users are used to rename filename newfilename
Linux users are used to mv filename newfilename

I hardly notice the difference anymore, except when typing

[batman]

@Ole

If the size from the first set of commands and the hashsum from the second set are the same, then the file is already up-to-date and skipped (and deleted if using move). Note: that this option ignores the modification time of the file.

Thank you for the summary, it makes a lot more sense now! Modification times I don't think are a big deal (although do correct me if they are!), the main thing for me are that timestamps (created at or photo taken at) are preserved.

@Ole

Windows users are used to rename filename newfilename
Linux users are used to mv filename newfilename

Omg this was the biggest point of confusion for me

I thought my file names were being updated lol.

Thank you so much! This is great, means the simple move and exclude-from commands are all I need!

[batman]

@Ole little update: first chunk of files has backed up successfully! Thank you again for your help