RClone - Powershell Variables

programmerpilot · May 21, 2021, 11:43pm

Hello Everyone, Happy Friday!!

I'm using Powershell to upload to an AWS s3 bucket using the rclone command copyto source dest. command.

All works perfectly if I specify the directory for the source (eg. copyto c:\FolderCopy AWS:******).

The problem I run into is when I try and replace rclone source with a PowerShell variable
$FileName = c:\FolderCopy

copyto $FileName AWS:*******

I receive an Access Denied error message. The folder directory is shared and "Everyone" set as full control over the entire directory and subdirectories.

Any advice is greatly appreciated.

Thanks,

programmerpilot · May 21, 2021, 11:45pm

I'm also running the latest version of RClone 1.55.0 on Windows 10 Enterprise 1909 (Sorry I forgot to add in my original post)

asdffdsa · May 21, 2021, 11:49pm

hello and welcome to the forum,

you did not post the most important parts of the help and support template.

please post

the powershell script
rclone config file
full rclone debug log by adding -vv to the command.

programmerpilot · May 22, 2021, 12:00am

*Debug Log*
C:\users\...\Downloads\rclone-v1.55.0-windows-amd64\rclone-v1.55.0-windows-amd64\rclone.exe : 2021/05/21 18:52:20 DEBUG : Using config file from "C:\\Users\\...\\.config\\rclone\\rclone.conf"
At line:1 char:1
+ C:\users\...\Downloads\rclone-v1.55.0-windows-amd64\rclone-v1.5 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2021/05/21 18:5...e\\rclone.conf":String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2021/05/21 18:52:20 DEBUG : rclone: Version "v1.55.0" starting with parameters ["C:\\users\\...\\Downloads\\rclone-v1.55.0-windows-amd64\\rclone-v1.55.0-windows-amd64\\rclone.exe" "-vv" "copyto" "C:\\test\\Original_1\\Hello_World.txt" 
"aws:***-testwksbucket\\***"]
2021/05/21 18:52:20 DEBUG : Creating backend with remote "C:\\test\\Original_1\\Hello_World.txt"
2021/05/21 18:52:20 DEBUG : fs cache: adding new entry for parent of "C:\\test\\Original_1\\Hello_World.txt", "//?/C:/test/Original_1"
2021/05/21 18:52:20 DEBUG : Creating backend with remote "aws:***-testwksbucket/"
2021/05/21 18:52:20 DEBUG : fs cache: renaming cache item "aws:***-testwksbucket/" to be canonical "aws:***-testwksbucket"
2021/05/21 18:52:20 DEBUG : Hello_World.txt: Need to transfer - File not found at Destination
2021/05/21 18:52:20 ERROR : Hello_World.txt: Failed to copy: AccessDenied: Access Denied
	status code: 403, request id: QC7A9TTCGKP4MFBP, host id: +8Xf12bqYITQHa+LoSzWBtJ44fg75tucrcBPDIeGBLXoA9rW/iey+oAARTJBR7RuRs8o8XAxZEE=
2021/05/21 18:52:20 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: QC7A9TTCGKP4MFBP, host id: +8Xf12bqYITQHa+LoSzWBtJ44fg75tucrcBPDIeGBLXoA9rW/iey+oAARTJBR7RuRs8o8XAxZEE=
2021/05/21 18:52:20 DEBUG : Hello_World.txt: Need to transfer - File not found at Destination
2021/05/21 18:52:20 ERROR : Hello_World.txt: Failed to copy: AccessDenied: Access Denied
	status code: 403, request id: QC75JD3W40B36194, host id: +oFLSunA0ZeGmjm50JZ13VKS8JwNSRFPpHAy5JDygMUrFHfGP4ExdPztqa2r0zCMC2KN6OWmkNU=
2021/05/21 18:52:20 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: QC75JD3W40B36194, host id: +oFLSunA0ZeGmjm50JZ13VKS8JwNSRFPpHAy5JDygMUrFHfGP4ExdPztqa2r0zCMC2KN6OWmkNU=
2021/05/21 18:52:21 DEBUG : Hello_World.txt: Need to transfer - File not found at Destination
2021/05/21 18:52:21 ERROR : Hello_World.txt: Failed to copy: AccessDenied: Access Denied
	status code: 403, request id: 0CAR7AQD6ZHF60MT, host id: D0iu3aUhpSFAXNPkNGkD7i3pF+jn3QzHQog50NKmLOZBuWomhZj8tT4N6Ntfwlt1j0olESjQlig=
2021/05/21 18:52:21 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
	status code: 403, request id: 0CAR7AQD6ZHF60MT, host id: D0iu3aUhpSFAXNPkNGkD7i3pF+jn3QzHQog50NKmLOZBuWomhZj8tT4N6Ntfwlt1j0olESjQlig=
2021/05/21 18:52:21 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.6s
2021/05/21 18:52:21 DEBUG : 4 go routines active
2021/05/21 18:52:21 Failed to copyto: AccessDenied: Access Denied
	status code: 403, request id: 0CAR7AQD6ZHF60MT, host id: D0iu3aUhpSFAXNPkNGkD7i3pF+jn3QzHQog50NKmLOZBuWomhZj8tT4N6Ntfwlt1j0olESjQlig=

*RClone Config File*
[aws]
type = s3
provider = AWS
env_auth = false
access_key_id = *****
secret_access_key = ****
region = us-east-2
acl = private
server_side_encryption = AES256

*Powershell Script*
         elseif (($File).LastWriteTime.Date -eq (get-date).Date)
         {
          #Check to see if it has already been uploaded
          $HashLog = Get-Content $XMLHash
          $SingleFile =  (Get-FileHash -Algorithm MD5 -LiteralPath $File.FullName).Hash
            IF($HashLog -match $SingleFile){
                
                
                 "Duplicate: " + $SingleFile + " " + ($File).FullName  | Add-Content $XMLHash
                  C:\users\...\Downloads\rclone-v1.55.0-windows-amd64\rclone-v1.55.0-windows-amd64\rclone.exe copyto $File.FullName aws:****-testwksbucket\FolderA
            }

asdffdsa · May 22, 2021, 12:18am

now that we have that info, perhaps @albertony might have an idea

seems to be an issue with powershell syntax not rclone.

this works for me

$source=	"C:\data\rclone\scripts\new\test.txt"
$dest=		"wasabi01:\en07\test"
$command=	"C:\data\rclone\scripts\rclone.exe"
$config=	"--config=C:\data\rclone\scripts\wasabi.conf"

&"$command" copyto $source $dest $config  -vv

powershell.exe -ExecutionPolicy Unrestricted -NoLogo -File "C:\data\rclone\scripts\doit.ps1"
Process started (PID=10300) >>>
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\wasabi.conf"
DEBUG : rclone: Version "v1.55.1" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "copyto" "C:\\data\\rclone\\scripts\\new\\test.txt" "wasabi01:\\bucket\\test" "--config=C:\\data\\rclone\\scripts\\wasabi.conf" "-vv"]
DEBUG : Creating backend with remote "C:\\data\\rclone\\scripts\\new\\test.txt"
DEBUG : fs cache: adding new entry for parent of "C:\\data\\rclone\\scripts\\new\\test.txt", "//?/C:/data/rclone/scripts/new"
DEBUG : Creating backend with remote "wasabi01:/bucket/"
DEBUG : fs cache: renaming cache item "wasabi01:/bucket/" to be canonical "wasabi01:bucket"
DEBUG : test.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
DEBUG : test.txt: Unchanged skipping
INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.3s

albertony · May 22, 2021, 8:08am

Without testing it myself (can try to do that in a while), I think you need to add parens around the $File.FullName reference: ($File.FullName)

rclone.exe copyto ($File.FullName) aws:****-testwksbucket\FolderA

Or, if that does not work, then perhaps try something like:

rclone.exe copyto "$($File.FullName)" "aws:****-testwksbucket\FolderA"

programmerpilot · May 24, 2021, 3:03pm

Thanks for all the help. Unfortunately, I'm still getting an access denied error message.

This is the powershell line that is throwing the error message:

C:\users\***\Downloads\rclone-v1.55.0-windows-amd64\rclone-v1.55.0-windows-amd64\rclone.exe copyto ($File.FullName) aws:****-testwksbucket\****_LT11 -vv

If I use this command everything works just fine, the only difference is the Powershell variable.

C:\users\****\Downloads\rclone-v1.55.0-windows-amd64\rclone-v1.55.0-windows-amd64\rclone.exe copyto C:\test\Staging aws:****-testwksbucket\****_LT11

asdffdsa · May 24, 2021, 5:11pm

for the script i posted, i changed
copyto $source $dest $config
to
copyto ($source) ($dest) ($config) ($flags) and to copyto "$($source)" "$($dest)" "$($config)" "$($flags)"`

in both cases, the command ran without problems

i am not a powershell expert but i noticed that in your script both

$File.FullName
($File).FullName

programmerpilot · May 24, 2021, 5:52pm

Thanks @asdffdsa for the quick response.

What I have determined is that if I try and copy a file I get the access denied error message, but if I copy the directory where the file is at it uploads to aws without any issues.

I even tried your method above and got the access denied error message when trying to upload the file.

system · July 24, 2021, 1:52pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.