Premiumize can't move files

ivandeex · October 28, 2021, 2:47am

@ncw
premiumize might have been affected by

or by

ingep · October 28, 2021, 8:56pm

@asdffdsa is right - this used to work for me until around 5th-7th October. I did not change anything at that time, it just stopped working. So that points to something having changed on the Premiumize side, but then again, using the REST API does work, which points to the issue being in rclone.

I just tried "mv test.txt ../NewDir" (using a folder as destination) in my rclone-mounted folder, but got the same error.

ncw · October 31, 2021, 12:37pm

I think the premiumizeme API has changed somehow.

In particular when I use the folder paste API it returns a successful response but then doesn't move anything

https://app.swaggerhub.com/apis-docs/premiumize.me/api/1.6.7#/folder/folderPaste

Here is what the message looked like

2021/10/23 22:06:56 DEBUG : HTTP REQUEST (req 0xc00064db00)
2021/10/23 22:06:56 DEBUG : POST /api/folder/paste?apikey=XXX HTTP/1.1
Host: www.premiumize.me
User-Agent: rclone/v1.57.0-beta.5800.a542ddf60
Transfer-Encoding: chunked
Content-Type: multipart/form-data;
boundary=b0ca803bd4c403d1ef508c0b6cb812d842e73541e3147d66cb8a253a0841
Accept-Encoding: gzip

151
--b0ca803bd4c403d1ef508c0b6cb812d842e73541e3147d66cb8a253a0841
Content-Disposition: form-data; name="id"

t6PPZhL09VB0B0WuHHRtXg
--b0ca803bd4c403d1ef508c0b6cb812d842e73541e3147d66cb8a253a0841
Content-Disposition: form-data; name="files[]"

dyP0CNotvyhY_gQJo-CAKw
--b0ca803bd4c403d1ef508c0b6cb812d842e73541e3147d66cb8a253a0841--

0

And here is the response - apparently successful, but the file did not move into the folder.

2021/10/23 22:06:57 DEBUG : HTTP RESPONSE (req 0xc00064db00)
2021/10/23 22:06:57 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443";
ma=86400, h3-27=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 6a2deea2199b4096-LHR
Content-Type: application/json
Date: Sat, 23 Oct 2021 21:06:57 GMT
Expect-Ct: max-age=604800,
report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To:
{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GKtVD84f4Bc82QGZG5EgSMDXuq4db%2BprdQmSviYkzFlGITvkyk0X4%2BMfYs29Swl%2BEy4kmAaxr7kxiw6MIdczRbvKdw5IqC9pkJNG0cTWtY
gwz%2FwiEn1mrjYpHdF%2FqCjzo0RV7Q%3D%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Set-Cookie: xss-token=1689fa9eedc120f877373349e0ffde72; expires=Mon,
22-Nov-2021 21:06:56 GMT; Max-Age=2592000; path=/
Strict-Transport-Security: max-age=604800
X-Frame-Options: sameorigin

{"status":"success"}

So @ingep @mithrial I think this is the cause of the broken mv/moveto

I did email them to get the integration test account working again and they said they've done a major overhaul of the API so there might be broken stuff.

The bit of rclone code which does this is

github.com

rclone/rclone/blob/a8059b8a90ec980b9c2112d571a9eb3e3a638526/backend/premiumizeme/premiumizeme.go#L643-L672

    
      
          	// Move the object to a new directory (with the existing name)
          	// if required
          	if doMove {
          		opts := rest.Opts{
          			Method:     "POST",
          			Path:       "/folder/paste",
          			Parameters: f.baseParams(),
          			MultipartParams: url.Values{
          				"id": {newDirectoryID},
          			},
          		}
          		if isFile {
          			opts.MultipartParams.Set("files[]", id)
          		} else {
          			opts.MultipartParams.Set("folders[]", id)
          		}
          		//replacedLeaf := enc.FromStandardName(leaf)
          		var resp *http.Response
          		var result api.Response
          		err = f.pacer.Call(func() (bool, error) {

This file has been truncated. show original

If anyone can see any mistakes in the code, that would be much appreciated, because I've stared at it for ages now

asdffdsa · October 31, 2021, 2:32pm

hi,
is this correct, that rclone is getting confused?

jwink3101 · October 31, 2021, 3:51pm

Not your main issue but...:

You need to use moveto or copyto when you're specifying a file name destination.

$ rclone copy remote:sub/myfile.txt remote:newsub/ # copy -- specify dest directory
$ rclone copyto remote:sub/myfile.txt remote:newsub/myfile.txt # copyto -- specify dest location

asdffdsa · October 31, 2021, 3:55pm

hi,
i did a rclone moveto and it did not work. is there something wrong with that command?

https://forum.rclone.org/t/premiumize-cant-move-files/27169/19?u=asdffdsa

jwink3101 · October 31, 2021, 3:57pm

No. I was answering OP's original comment about the moves. Not the underlying API issue

ncw · October 31, 2021, 4:29pm

I think I figured out what changed in the premiumizeme API

Can anyone having trouble moving files on premiumizeme try this beta please

v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme on branch fix-premiumizeme (uploaded in 15-30 mins)

There may be more things which need fixing...

asdffdsa · October 31, 2021, 4:46pm

good point, in all the back and forth, that was never make explicitly clear.

asdffdsa · October 31, 2021, 5:30pm

for the beta, the windows 64bit was not uploaded, so i tested using the 32bit.
i tried these two commands and both worked
--- rclone moveto premiumize:test/OldDir/OldFile.bin premiumize:test/NewDir/NewFile.bin
--- rclone move premiumize:test/OldDir/OldFile.bin premiumize:test/NewDir

d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone delete premiumize:test/OldDir -v 

d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone delete premiumize:test/NewDir -v 

d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone copy C:\data\rclone\scripts\rr\other\newtest\pme\OldFile.bin premiumize:test/OldDir -v 
2021/10/31 13:05:19 INFO  : OldFile.bin: Copied (new)
2021/10/31 13:05:19 INFO  : 
Transferred:   	          1 B / 1 B, 100%, 0 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.9s


d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone moveto premiumize:test/OldDir/OldFile.bin premiumize:test/NewDir/NewFile.bin -vv 
2021/10/31 13:05:20 DEBUG : Setting --config "C:\\data\\rclone\\scripts\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\scripts\\rclone.conf"
2021/10/31 13:05:20 DEBUG : rclone: Version "v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme" starting with parameters ["d:\\data\\rclone\\exe\\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\\rclone.exe" "moveto" "premiumize:test/OldDir/OldFile.bin" "premiumize:test/NewDir/NewFile.bin" "-vv"]
2021/10/31 13:05:20 DEBUG : Creating backend with remote "premiumize:test/OldDir/OldFile.bin"
2021/10/31 13:05:20 DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
2021/10/31 13:05:21 DEBUG : fs cache: adding new entry for parent of "premiumize:test/OldDir/OldFile.bin", "premiumize:test/OldDir"
2021/10/31 13:05:21 DEBUG : Creating backend with remote "premiumize:test/NewDir/"
2021/10/31 13:05:21 DEBUG : fs cache: renaming cache item "premiumize:test/NewDir/" to be canonical "premiumize:test/NewDir"
2021/10/31 13:05:21 DEBUG : OldFile.bin: Need to transfer - File not found at Destination
2021/10/31 13:05:22 INFO  : OldFile.bin: Moved (server-side) to: NewFile.bin
2021/10/31 13:05:22 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         2.1s

2021/10/31 13:05:22 DEBUG : 5 go routines active

C:\data\rclone\scripts\rr\other\newtest\pme>d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone copy C:\data\rclone\scripts\rr\other\newtest\pme\OldFile.bin premiumize:test/OldDir -v 
2021/10/31 13:05:24 INFO  : OldFile.bin: Copied (new)
2021/10/31 13:05:24 INFO  : 
Transferred:   	          1 B / 1 B, 100%, 0 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.0s

C:\data\rclone\scripts\rr\other\newtest\pme>d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone move premiumize:test/OldDir/OldFile.bin premiumize:test/NewDir -vv 
2021/10/31 13:05:24 DEBUG : Setting --config "C:\\data\\rclone\\scripts\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\scripts\\rclone.conf"
2021/10/31 13:05:24 DEBUG : rclone: Version "v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme" starting with parameters ["d:\\data\\rclone\\exe\\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\\rclone.exe" "move" "premiumize:test/OldDir/OldFile.bin" "premiumize:test/NewDir" "-vv"]
2021/10/31 13:05:24 DEBUG : Creating backend with remote "premiumize:test/OldDir/OldFile.bin"
2021/10/31 13:05:24 DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
2021/10/31 13:05:25 DEBUG : fs cache: adding new entry for parent of "premiumize:test/OldDir/OldFile.bin", "premiumize:test/OldDir"
2021/10/31 13:05:25 DEBUG : Creating backend with remote "premiumize:test/NewDir"
2021/10/31 13:05:26 DEBUG : OldFile.bin: Need to transfer - File not found at Destination
2021/10/31 13:05:26 INFO  : OldFile.bin: Moved (server-side)
2021/10/31 13:05:26 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         1.9s

2021/10/31 13:05:26 DEBUG : 5 go routines active

d:\data\rclone\exe\rclone-v1.57.0-beta.5811.9a5cf0509.fix-premiumizeme-windows-386\rclone ls premiumize:test 
        1 NewDir/NewFile.bin
        1 NewDir/OldFile.bin

ncw · October 31, 2021, 7:21pm

I think I've tracked the remaining API changes down now and all the integration tests pass again

I've merged this to master now which means it will be in v1.57.0-beta.5816.f7764a0c9 (or later) in 15-30 minutes and released in v1.57

I think this is the last showstopper for 1.57 so we should be able to get that out tomorrow

asdffdsa · October 31, 2021, 7:29pm

just tested the exact same commands using latest rclone-v1.57.0-beta.5816.f7764a0c9-windows-amd64
all commands worked. i can post the debug logs if requested.

are there any other command that you would want me to run?

ncw · October 31, 2021, 8:25pm

I think that is great testing thank you

The next test will be to see if the integration tests run clean - these run tonight and the results will appear here: FAIL: 20 tests failed in 11h9m0.344735968s

ingep · November 1, 2021, 1:49am

I just did a test too, moving a file in a folder that was mounted with rclone, and great success! Thank you very much for the quick fix!

mithrial · November 1, 2021, 3:56pm

Thank you very much. This beta solves my issue fully.

system · November 4, 2021, 3:56pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.