pCloud keeps partial interrupted uploads

signal9 · March 18, 2024, 8:01am

First and foremost - Hi all and thank you for your time and hard work!

What is the problem you are having with rclone?

When an uncached file transfer to pCloud is interrupted due to losing internet, pCloud seems to keep the partially uploaded file around (it can take a few minutes to appear in pCloud).

First off I'd like to ask if this is the intended behavior or if an "all or nothing" is expected?

If this is indeed intended, is there any way of avoiding leaving half-uploaded files around with the same name as the final file, e.g. by forcing rclone to upload to a .partial file, as it does with a number of other destinations, like sftp?

(I can't seem to find an opposite flag to --inplace in the documentation and --partial-suffix seems to only affect destinations that already use temporary files.)

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: darwin 14.2.1 (64 bit)
- os/kernel: 23.2.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

pCloud

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Have a 1 GiB file named file

$ rclone lsl file
1073741824 2024-03-17 22:16:40.358814994 file

Start a copy of file to pCloud.

$ rclone copy -v -P file pcloud:
Transferred:   	   10.871 MiB / 1 GiB, 1%, 652.236 KiB/s, ETA 26m30s
Transferred:            0 / 1, 0%
Elapsed time:        18.0s
Transferring:
 *                                          file:  1% /1Gi, 652.204Ki/s, 26m30s^C

Interrupt internet connection before transfer is complete.
Interrupt rclone (CTRL+C).
Restore internet connection.
Observe pCloud partially uploaded file.

$ rclone lsl -v pcloud:file
 11087872 2024-03-17 22:16:40.000000000 file

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[pcloud]
type = pcloud
hostname = eapi.pcloud.com
token = XXX

A log from the command that you were trying to run with the `-vv` flag

(I omitted the -vv only to avoid noise from the discussion. I'd be happy to supply it if it is relevant!)

Thank you for taking the time!

kapitainsky · March 18, 2024, 9:19am

I can reproduce it. It is definitely not typical cloud storage behaviour.

Yes I would expect "all or nothing" too. But clearly this is not the case here.

It would be good workaround. Should not be difficult to add. Have a look at original --inplace PR

github.com/rclone/rclone

Implement 'atomic downloads' with copy-to-temp + rename

rclone:master ← nurpax:copy-rename

opened 03:30PM - 04 Apr 23 UTC

nurpax

+179 -10

Download a local copy into a temporary file first. Once the download is complet…e, rename the temp file to the real destination. This prevents other processes from seeing partially downloaded copies of files being downloaded. for #3770 #### What is the purpose of this change? Support "atomic downloads" by copy-to-temp/rename. This prevents other processes from seeing partially downloaded files while rclone is downloading. #### Was the change discussed in an issue or in the forum before? https://github.com/rclone/rclone/issues/3770 #### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [ ] I have added tests for all changes in this PR if appropriate. - [ ] I have added documentation for the changes if appropriate. - [ ] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [ ] I'm done, this Pull Request is ready for review :-) Checklist incomplete, pushing PR for review. If the approach looks sensible, I'll update the PR with tests & other required items.

To add it to pCloud would require only adding PartialUploads: true feature to pCloud remote here:

github.com

rclone/rclone/blob/7803b4ed6c3256f25270e3b3024011f8f1e77ca0/backend/pcloud/pcloud.go#L326C1-L328C33


      
          	f.features = (&fs.Features{
          		CaseInsensitive:         false,
          		CanHaveEmptyDirectories: true,

and updating documentation including adding " Limitations" section to pCloud one with explanation why --inplace is used.

You can try to have a go and propose PR maybe?

asdffdsa · March 18, 2024, 2:00pm

hi.
yes, i think it is intended behavior.
pcloud supports resumable uploads, so it makes sense the partial uploads would remain.

pcloud has an api parameter to prevent partial uploads.
If nopartial is set, partially uploaded files will not be saved (that is when the connection breaks

in that case, if the internet goes down, --inplace would not make a practical difference.

tho, if rclone did support --inplace for pcloud, then could list and/or delete the partial uploads
rclone ls --include='*.partial'
rclone delete --include='*.partial'

if all is true, then best for rclone to use the api parameter.
tho, @kapitainsky has shown, could easily add --inplace to pcloud.

kapitainsky · March 18, 2024, 2:13pm

Problem is that rclone does not support any form of resumable uploads and in case of pCloud simply starts uploading files again...

And without using PartialUploads feature set it is impossible to tell if upload finished or not.

It would be ideal if pCloud API could be leveraged here but it is probably something depending on (which is general framework to support resume):

github.com/rclone/rclone

fs: add Resumer interface

rclone:master ← mcalman:add-resume-interface

opened 02:58PM - 01 Sep 20 UTC

mcalman

+910 -85

Added an interface for resuming failed uploads. Later on this interface can be …implemented by any backend where resumes are possible. #87  #### What is the purpose of this change? Adds an optional interface for resuming uploads that can later be implemented by any backend where resuming uploads is possible. Once implemented by a backend, this will allow uploads to continue after a failure, even when restarting Rclone operations. #### Was the change discussed in an issue or in the forum before? https://github.com/rclone/rclone/issues/87 https://forum.rclone.org/t/drive-resume-resumable-upload-implementation/18289/4 https://forum.rclone.org/t/chunker-resume-upload-contribution/18571 #### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-)

Which after 4 years and 100+ comments is nowhere:)

In the meantime I suggest whoever needs PartialUploads for pCloud creates PR. This one is very easy.

asdffdsa · March 18, 2024, 2:21pm

i agree, as you have shown, rclone could easily add --inplace for pcloud.

but not a solution to the OP issue.
for that, rclone would need to set nopartial parameter for the api call.

kapitainsky · March 18, 2024, 2:22pm

Yeap - agree. PartialUploads is workaround only.

signal9 · March 18, 2024, 2:24pm

Thank you both for you comments. This is actually quite illuminating.

Thanks @asdffdsa for the link. Seems like pCloud will discard partial uploads if nopartial is set.

And thanks @kapitainsky for your link, as it looks like rclone does set that flag.

github.com

rclone/rclone/blob/7803b4ed6c3256f25270e3b3024011f8f1e77ca0/backend/pcloud/pcloud.go#L1210


      
          		Body:             in,
          		ContentType:      fs.MimeType(ctx, src),
          		ContentLength:    &size,
          		Parameters:       url.Values{},
          		TransferEncoding: []string{"identity"}, // pcloud doesn't like chunked encoding
          		Options:          options,
          	}
          	leaf = o.fs.opt.Enc.FromStandardName(leaf)
          	opts.Parameters.Set("filename", leaf)
          	opts.Parameters.Set("folderid", dirIDtoNumber(directoryID))
          	opts.Parameters.Set("nopartial", "1")
          	opts.Parameters.Set("mtime", fmt.Sprintf("%d", uint64(modTime.Unix())))
          
          	// Special treatment for a 0 length upload.  This doesn't work
          	// with PUT even with Content-Length set (by setting
          	// opts.Body=0), so upload it as a multipart form POST with
          	// Content-Length set.
          	if size == 0 {
          		formReader, contentType, overhead, err := rest.MultipartUpload(ctx, in, opts.Parameters, "content", leaf)
          		if err != nil {
          			return fmt.Errorf("failed to make multipart upload for 0 length file: %w", err)

Hmm, is this a bug I should report to pCloud?

Thanks!

asdffdsa · March 18, 2024, 2:26pm

ok, so rclone already uses that paramter.

as for being a bug, not sure yet if the issue is with rclone, pcloud or both.
would have to run a test, copy a single file using --dump=headers, killing the internet before upload completes.
from that, we could see what rclone and pcloud are doing.

kapitainsky · March 18, 2024, 2:26pm

From pCloud API docs:

If nopartial is set, partially uploaded files will not be saved (that is when the connection breaks before file is read in full).

which exactly means that rclone does not resume anything.

pCloud should not save it - but it does. So indeed looks like a bug on their end.

Or their understanding is very different to ours:)

signal9 · March 18, 2024, 3:12pm

Looks to me like rclone did set the nopartial flag.

--dump=headers gave me this snippet:

...
2024/03/18 08:04:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/03/18 08:04:45 DEBUG : file: Need to transfer - File not found at Destination
2024/03/18 08:04:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/03/18 08:04:46 DEBUG : HTTP REQUEST (req 0x140005a7e60)
2024/03/18 08:04:46 DEBUG : PUT /uploadfile?filename=file&folderid=0&mtime=1710739000&nopartial=1 HTTP/1.1
...
Transferred:   	    6.621 MiB / 1 GiB, 1%, 1.656 MiB/s, ETA 10m14s
Transferred:            0 / 1, 0%
Elapsed time:         5.5s
Transferring:
 *                                          file:  0% /1Gi, 1.656Mi/s, 10m14s^C

Also, looks like the partial file remains even if I issue a CTRL-C with internet connected.

$ rclone lsl pcloud:file
  6926336 2024-03-17 22:16:40.000000000 file

Seems like contacting pCloud support is my next step?

Thanks!

asdffdsa · March 18, 2024, 3:16pm

yeah, let's us know how pcloud responds...

signal9 · March 19, 2024, 3:48pm

Hi all,

pCloud Support got back to me and very politely informed me that they don't offer technical support on the developer API.

I did formulate my email as a question, asking wether what I'm seeing is expected behavior, so I guess that's fair.

I wrote back, asking if I could file a bug report with the developers. We'll see what they say.

Thanks,
Alex

signal9 · March 20, 2024, 5:41pm

Bug report filed.

Support was friendly and said they forwarded it to the devs but did not specifically say they'd get back to me. I'll update here if they do.

system · April 19, 2024, 5:42pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.