"Failed to set sparse: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function." errors

What is the problem you are having with rclone?

Copying files from encrypted Google Drive to local hard drive causes Failed to set sparse: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function. errors. Hard drive audibly clicks every time this error occurs (it sounds like it abruptly stops).

It's the same issue as discussed in this post: Getting errors: "Failed to set sparse: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function."

The hard drive is freshly formatted with exFAT, completely empty. Vanilla, not ran through any translation layer like VeraCrypt. 8TB WD My Book WDC WD80EDAZ-11TA3A0 connected via USB.

I tried the --local-no-sparse flag but it made transfers unbearably slow. Files were stuck at 0% transfer for minutes and when they started downloading it was unbearably slow, much slower than what I get without the flag. Not sure why. I would prefer to avoid this flag.

Do you have any idea how to remedy this issue? I really appreciate your help.

Also I'm not sure why rclone is reporting being rate limited, I created my own API key and according to Google's cloud console metrics the key is well below the quota of 1 billion queries per day and 20k queries per user and in total per 100 seconds.

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: Microsoft Windows 10 Enterprise LTSC 2021 2009 (64 bit)
  • os/kernel: 10.0.19044.1415 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive using crypt wrapper

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vv --progress gcrypt:Topic/Source K:/Destination

A log from the command with the -vv flag

PS C:\Users\user> rclone copy -vv --progress gcrypt:Topic/Source K:/Destination
2022/01/03 01:59:33 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\Tools\\rclone.exe" "copy" "-vv" "--progress" "gcrypt:Topic/Source" "K:/Destination"]
2022/01/03 01:59:33 DEBUG : Creating backend with remote "gcrypt:Topic/Source"
Enter configuration password:
password:
2022/01/03 01:59:43 DEBUG : Using config file from "C:\\Users\\user\\.config\\rclone\\rclone.conf"
2022/01/03 01:59:43 DEBUG : Creating backend with remote "gdrive:/vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84"
2022/01/03 01:59:43 DEBUG : Google drive root 'vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84': 'root_folder_id = 0AC_bvH4HuiKUUk9PVA' - save this in the config to speed up startup
2022/01/03 01:59:44 DEBUG : fs cache: renaming cache item "gdrive:/vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84" to be canonical "gdrive:vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84"
2022/01/03 01:59:44 DEBUG : fs cache: switching user supplied name "gdrive:/vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84" for canonical name "gdrive:vault/nhr5ku1cq09t59vk4fe9sls05o/6dmodgtfa0eudtdh2h8n1jtn84"
2022/01/03 01:59:44 DEBUG : Creating backend with remote "K:/Destination"
2022/01/03 01:59:44 DEBUG : fs cache: renaming cache item "K:/Destination" to be canonical "//?/K:/Destination"
2022-01-03 01:59:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:45 DEBUG : pacer: Rate limited, increasing sleep to 1.565765643s
2022-01-03 01:59:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:45 DEBUG : pacer: Rate limited, increasing sleep to 2.031188438s
2022-01-03 01:59:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:45 DEBUG : pacer: Rate limited, increasing sleep to 4.670338484s
2022-01-03 01:59:45 DEBUG : pacer: Rate limited, increasing sleep to 8.168316403s
2022-01-03 01:59:45 DEBUG : pacer: Rate limited, increasing sleep to 16.623417513s
2022-01-03 01:59:45 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 1.065860901s
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 2.883284718s
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 4.848075898s
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 8.991370616s
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 16.441888023s
2022-01-03 01:59:47 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:47 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 1.804144925s
2022-01-03 01:59:47 ERROR : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: Failed to set sparse: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function.
2022-01-03 01:59:47 DEBUG : pacer: Rate limited, increasing sleep to 2.726190418s
2022-01-03 01:59:48 DEBUG : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: Starting multi-thread copy with 4 parts of size 5.274Gi
2022-01-03 01:59:48 DEBUG : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: multi-thread copy: stream 2/4 (5663358976-11326717952) size 5.274Gi starting
2022-01-03 01:59:48 DEBUG : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: multi-thread copy: stream 1/4 (0-5663358976) size 5.274Gi starting
2022-01-03 01:59:48 DEBUG : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: multi-thread copy: stream 4/4 (16990076928-22653316442) size 5.274Gi starting
2022-01-03 01:59:48 DEBUG : Dark Shadows 1080p BluRay REMUX/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON/Dark.Shadows.2012.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON.mkv: multi-thread copy: stream 3/4 (11326717952-16990076928) size 5.274Gi starting
2022-01-03 01:59:48 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 1.921596186s
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 2.204618864s
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 4.16141832s
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 8.726700103s
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 16.837967096s
2022-01-03 01:59:49 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:49 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 1.97956892s
2022-01-03 01:59:49 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:49 DEBUG : pacer: Rate limited, increasing sleep to 1.216079652s
2022-01-03 01:59:49 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 1.768424442s
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: redacted, userRateLimitExceeded)
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 2.302874108s
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 4.877629458s
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 8.474786357s
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 16.886990819s
2022-01-03 01:59:51 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:51 DEBUG : pacer: Rate limited, increasing sleep to 1.851133209s
2022-01-03 01:59:51 DEBUG : pacer: Reducing sleep to 0s
2022-01-03 01:59:52 ERROR : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: Failed to set sparse: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function.
2022-01-03 01:59:52 DEBUG : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: Starting multi-thread copy with 4 parts of size 4.031Gi
2022-01-03 01:59:52 DEBUG : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: multi-thread copy: stream 4/4 (12983205888-17310708838) size 4.030Gi starting
2022-01-03 01:59:52 DEBUG : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: multi-thread copy: stream 2/4 (4327735296-8655470592) size 4.031Gi starting
2022-01-03 01:59:52 DEBUG : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: multi-thread copy: stream 3/4 (8655470592-12983205888) size 4.031Gi starting
2022-01-03 01:59:52 DEBUG : Allied 2160p BluRay/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn/Allied 2016 2160p UHD HDR10+ BluRay x265 DTS-HD MA 5.1-UnKn0wn.mkv: multi-thread copy: stream 1/4 (0-4327735296) size 4.031Gi starting

hi,

perhaps re-format the hard drive using a file system that supports sparse files.
as per the documentation,
" IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not"

thank you, that was a silly mistake on my side.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.