Uploading to Internet Archive, problems with metadata (as far as I understand)

riccardo · August 24, 2023, 7:09am

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

os/version: Microsoft Windows 10 Home 22H2 (64 bit)
os/kernel: 10.0.19045.3324 Build 19045.3324.3324 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Internet Archive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

C:\rclone-v1.62.2-windows-amd64\rclone.exe copy 'C:\Users\ricca\OneDrive\Documenti\Mixes\Rogie_VinylSundays_010608\Rogie_VinylSundays_010608.wav' 'archive.org:Rogie_VinylSundays_010608.wav' `
--metadata-set title="The Vinyl Sundays - In The Mix" `
--metadata-set identifier="Rogie_VinylSundays_010608" `
--metadata-set mediatype=audio `
--metadata-set collection=opensource_audio `
--metadata-set date=1998 `
--metadata-set description="Rogie in the mix" `
--metadata-set creator="Rogie Raregrooves" `
--metadata-set language=English `
--metadata-set licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/ `
--metadata-set noindex=noindex `
--transfers=4 `
--fast-list `
--buffer-size=256M `
--progress

The rclone config contents with secrets removed.

[Backblaze B2]
type = b2
account = xxxxxxxxxxxxxxx  
key = xxxxxxxxxxxxxxx

[archive.org]
type = internetarchive
access_key_id = xxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxx

A log from the command with the `-vv` flag

https://pastebin.com/daWTwC0D

kapitainsky · August 24, 2023, 8:19am

It does not look like rclone issue.

Your logfile indicates that it is your operating system problem not able to execute your command. Must be something Windows specific you should seek help on Windows forums.

Eccezione:
System.ArgumentOutOfRangeException: Il valore deve essere maggiore o uguale a zero e minore della grandezza del buffer della console in tale dimensione.   
Nome parametro: top
Valore effettivo: -1.
   in System.Console.SetCursorPosition(Int32 left, Int32 top)
   in Microsoft.PowerShell.PSConsoleReadLine.ReallyRender(RenderData renderData, String defaultColor)
   in Microsoft.PowerShell.PSConsoleReadLine.ForceRender()
   in Microsoft.PowerShell.PSConsoleReadLine.Insert(Char c)
   in Microsoft.PowerShell.PSConsoleReadLine.SelfInsert(Nullable`1 key, Object arg)
   in Microsoft.PowerShell.PSConsoleReadLine.ProcessOneKey(ConsoleKeyInfo key, Dictionary`2 dispatchTable, Boolean ignoreIfNoAction, Object arg)
   in Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   in Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics)

It even suggest to report an issue to:

Si è verificato un problema. Segnala il bug con i dettagli riportati di seguito.
Segnala su GitHub: https://github.com/lzybkr/PSReadLine/issues/new

riccardo · August 24, 2023, 8:21am

I see. Thanks @kapitainsky

kapitainsky · August 24, 2023, 8:23am

Something is too long or some characters are illegal? I am not windows user so can't really tell.

I would try to quote all metadata values without exception e.g.:

--metadata-set creator="Rogie Raregrooves"
--metadata-set language="English"
--metadata-set licenseurl="https://creativecommons.org/licenses/by-nc-nd/4.0/"

Then I would try to put all command in some shell script - not command line - maybe there are some limits in Windows and all command is too long?

riccardo · August 24, 2023, 8:29am

Yes, I'm going to try your suggestions. Thanks!

kapitainsky · August 24, 2023, 12:54pm

@asdffdsa you are Windows guru. Do you see here any Power Shell related problem in executing this rclone command?

asdffdsa · August 24, 2023, 1:34pm

runs fine via powershell, script and command line

this is the script

c:\data\rclone\rclone.exe copy 'c:\data\rclone\other\iarchive\Rogie_VinylSundays_010608.wav' 'iarchive:zorkzork/Rogie_VinylSundays_0106082.wav' -vv --retries=1 `
--metadata-set title="The Vinyl Sundays - In The Mix" `
--metadata-set identifier="Rogie_VinylSundays_010608" `
--metadata-set mediatype=audio `
--metadata-set collection=opensource_audio `
--metadata-set date=1998 `
--metadata-set description="Rogie in the mix" `
--metadata-set creator="Rogie Raregrooves" `
--metadata-set language=English `
--metadata-set licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/ `
--metadata-set noindex=noindex

this is the output of that script as run from notepad++

NPP_SAVE: C:\data\rclone\other\iarchive\iarchive.ps1
powershell.exe -ExecutionPolicy Unrestricted -NoLogo -File "C:\data\rclone\other\iarchive\iarchive.ps1"
Process started (PID=27476) >>>
2023/08/24 09:42:22 DEBUG : MetadataUpload map[collection:opensource_audio creator:Rogie Raregrooves date:1998 description:Rogie in the mix identifier:Rogie_VinylSundays_010608 language:English licenseurl:https://creativecommons.org/licenses/by-nc-nd/4.0/ mediatype:audio noindex:noindex title:The Vinyl Sundays - In The Mix]
2023/08/24 09:42:22 DEBUG : rclone: Version "v1.63.0" starting with parameters ["C:\\data\\rclone\\rclone.exe" "copy" "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav" "iarchive:zorkzork/Rogie_VinylSundays_0106082.wav" "-vv" "--retries=1" "--metadata-set" "title=The Vinyl Sundays - In The Mix" "--metadata-set" "identifier=Rogie_VinylSundays_010608" "--metadata-set" "mediatype=audio" "--metadata-set" "collection=opensource_audio" "--metadata-set" "date=1998" "--metadata-set" "description=Rogie in the mix" "--metadata-set" "creator=Rogie Raregrooves" "--metadata-set" "language=English" "--metadata-set" "licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/" "--metadata-set" "noindex=noindex"]
2023/08/24 09:42:22 DEBUG : Creating backend with remote "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav"
2023/08/24 09:42:22 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2023/08/24 09:42:22 DEBUG : fs cache: adding new entry for parent of "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav", "//?/c:/data/rclone/other/iarchive"
2023/08/24 09:42:22 DEBUG : Creating backend with remote "iarchive:zorkzork/Rogie_VinylSundays_0106082.wav"
2023/08/24 09:42:22 DEBUG : Rogie_VinylSundays_010608.wav: Need to transfer - File not found at Destination
2023/08/24 09:42:24 DEBUG : Rogie_VinylSundays_010608.wav: Dst hash empty - aborting Src hash check
2023/08/24 09:42:24 INFO  : Rogie_VinylSundays_010608.wav: Copied (new)
2023/08/24 09:42:24 INFO  : 
Transferred:   	    2.303 KiB / 2.303 KiB, 100%, 2.272 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.2s

2023/08/24 09:42:24 DEBUG : 4 go routines active
<<< Process finished (PID=27476). (Exit code 0)
================ READY ================

and this is output from command line

PS C:\data\rclone> c:\data\rclone\rclone.exe copy 'c:\data\rclone\other\iarchive\Rogie_VinylSundays_010608.wav' 'iarchive:zorkzork/Rogie_VinylSundays_0106083.wav' -vv --retries=1 --tpslimit=1 --tpslimit-burst=0 `
>> --metadata-set title="The Vinyl Sundays - In The Mix" `
>> --metadata-set identifier="Rogie_VinylSundays_010608" `
>> --metadata-set mediatype=audio `
>> --metadata-set collection=opensource_audio `
>> --metadata-set date=1998 `
>> --metadata-set description="Rogie in the mix" `
>> --metadata-set creator="Rogie Raregrooves" `
>> --metadata-set language=English `
>> --metadata-set licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/ `
>> --metadata-set noindex=noindex
2023/08/24 09:48:20 DEBUG : MetadataUpload map[collection:opensource_audio creator:Rogie Raregrooves date:1998 description:Rogie in the mix identifier:Rogie_VinylSundays_010608 language:English licenseurl:https://creativecommons.org/licenses/by-nc-nd/4.0/ mediatype:audio noindex:noindex title:The Vinyl Sundays - In The Mix]
2023/08/24 09:48:20 INFO  : Starting transaction limiter: max 1 transactions/s with burst 1
2023/08/24 09:48:20 DEBUG : rclone: Version "v1.63.0" starting with parameters ["C:\\data\\rclone\\rclone.exe" "copy" "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav" "iarchive:zorkzork/Rogie_VinylSundays_0106083.wav" "-vv" "--retries=1" "--tpslimit=1" "--tpslimit-burst=0" "--metadata-set" "title=The Vinyl Sundays - In The Mix" "--metadata-set" "identifier=Rogie_VinylSundays_010608" "--metadata-set" "mediatype=audio" "--metadata-set" "collection=opensource_audio" "--metadata-set" "date=1998" "--metadata-set" "description=Rogie in the mix" "--metadata-set" "creator=Rogie Raregrooves" "--metadata-set" "language=English" "--metadata-set" "licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/" "--metadata-set" "noindex=noindex"]
2023/08/24 09:48:20 DEBUG : Creating backend with remote "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav"
2023/08/24 09:48:20 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2023/08/24 09:48:20 DEBUG : fs cache: adding new entry for parent of "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav", "//?/c:/data/rclone/other/iarchive"
2023/08/24 09:48:20 DEBUG : Creating backend with remote "iarchive:zorkzork/Rogie_VinylSundays_0106083.wav"
2023/08/24 09:48:21 DEBUG : Rogie_VinylSundays_010608.wav: Need to transfer - File not found at Destination
2023/08/24 09:48:24 DEBUG : Rogie_VinylSundays_010608.wav: Dst hash empty - aborting Src hash check
2023/08/24 09:48:24 INFO  : Rogie_VinylSundays_010608.wav: Copied (new)
2023/08/24 09:48:24 INFO  :
Transferred:        2.303 KiB / 2.303 KiB, 100%, 1.141 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         4.0s

and works fine from dos .cmd batch

C:\data\rclone\other\iarchive\iarchive.cmd
Process started (PID=25964) >>>

C:\data\rclone\other\iarchive>c:\data\rclone\rclone.exe copy c:\data\rclone\other\iarchive\Rogie_VinylSundays_010608.wav iarchive:zorkzork/Rogie_VinylSundays_0106081.wav -vv --retries=1 --metadata-set title="The Vinyl Sundays - In The Mix" --metadata-set identifier="Rogie_VinylSundays_010608" --metadata-set mediatype=audio --metadata-set collection=opensource_audio --metadata-set date=1998 --metadata-set description="Rogie in the mix" --metadata-set creator="Rogie Raregrooves" --metadata-set language=English --metadata-set licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/ --metadata-set noindex=noindex 
2023/08/24 09:39:22 DEBUG : MetadataUpload map[collection:opensource_audio creator:Rogie Raregrooves date:1998 description:Rogie in the mix identifier:Rogie_VinylSundays_010608 language:English licenseurl:https://creativecommons.org/licenses/by-nc-nd/4.0/ mediatype:audio noindex:noindex title:The Vinyl Sundays - In The Mix]
2023/08/24 09:39:22 DEBUG : rclone: Version "v1.63.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav" "iarchive:zorkzork/Rogie_VinylSundays_0106081.wav" "-vv" "--retries=1" "--metadata-set" "title=The Vinyl Sundays - In The Mix" "--metadata-set" "identifier=Rogie_VinylSundays_010608" "--metadata-set" "mediatype=audio" "--metadata-set" "collection=opensource_audio" "--metadata-set" "date=1998" "--metadata-set" "description=Rogie in the mix" "--metadata-set" "creator=Rogie Raregrooves" "--metadata-set" "language=English" "--metadata-set" "licenseurl=https://creativecommons.org/licenses/by-nc-nd/4.0/" "--metadata-set" "noindex=noindex"]
2023/08/24 09:39:22 DEBUG : Creating backend with remote "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav"
2023/08/24 09:39:22 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2023/08/24 09:39:22 DEBUG : fs cache: adding new entry for parent of "c:\\data\\rclone\\other\\iarchive\\Rogie_VinylSundays_010608.wav", "//?/c:/data/rclone/other/iarchive"
2023/08/24 09:39:22 DEBUG : Creating backend with remote "iarchive:zorkzork/Rogie_VinylSundays_0106081.wav"
2023/08/24 09:39:22 DEBUG : Rogie_VinylSundays_010608.wav: Need to transfer - File not found at Destination
2023/08/24 09:39:25 DEBUG : Rogie_VinylSundays_010608.wav: Dst hash empty - aborting Src hash check
2023/08/24 09:39:25 INFO  : Rogie_VinylSundays_010608.wav: Copied (new)
2023/08/24 09:39:25 INFO  : 
Transferred:   	    2.303 KiB / 2.303 KiB, 100%, 1.144 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         3.1s

2023/08/24 09:39:25 DEBUG : 4 go routines active
<<< Process finished (PID=25964). (Exit code 0)
================ READY ================

kapitainsky · August 24, 2023, 2:17pm

@riccardo - as you can see above it looks that there is some Windows specific problem you are experiencing. It is not rclone. But I am afraid I have no clue what it is in your particular case.

asdffdsa · August 24, 2023, 2:43pm

did more testing on the command line, works fine using
PSVersion 5.1.22621.1778
and
PSVersion 7.3.6

riccardo · August 24, 2023, 2:50pm

@kapitainsky I see. I've tried quotes, Windows Command shell, PS as admin, and still I get those errors.
@asdffdsa I'm going to try PSVersion 7.3.6 tomorrow.
If these problems with my Windows persist, I will upload the files to the Internet Archive from their GUI, and I will use rclone for Backblaze where it works very well.
Thanks for your kind help @kapitainsky @asdffdsa

asdffdsa · August 24, 2023, 3:09pm

test the simplest command possible and do so on a single-line.

rclone copy C:\Users\ricca\OneDrive\Documenti\Mixes\Rogie_VinylSundays_010608\Rogie_VinylSundays_010608.wav archive.org:Rogie_VinylSundays_010608.wav -vv -dry-run

run this command, again on a single-line.

rclone copy C:\Users\ricca\OneDrive\Documenti\Mixes\Rogie_VinylSundays_010608\Rogie_VinylSundays_010608.wav archive.org:Rogie_VinylSundays_010608.wav -vv --dry-run --metadata-set title=test

now, take that same command and use multi-line

rclone copy C:\Users\ricca\OneDrive\Documenti\Mixes\Rogie_VinylSundays_010608\Rogie_VinylSundays_010608.wav archive.org:Rogie_VinylSundays_010608.wav -vv --dry-run `
--metadata-set title=test

riccardo · August 24, 2023, 5:26pm

PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image> C:\rclone-v1.62.2-windows-amd64\rclone.exe copy C:\Users\ricca\OneDrive\Documenti\Websites\mixtapes\rogie\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 -vv -dry-run
Error: unknown shorthand flag: 'd' in -dry-run
Usage:
  rclone copy source:path dest:path [flags]

Flags:
      --create-empty-src-dirs   Create empty source dirs on destination after copy
  -h, --help                    help for copy

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2023/08/24 19:10:01 Fatal error: unknown shorthand flag: 'd' in -dry-run
PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image>

PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image> C:\rclone-v1.62.2-windows-amd64\rclone.exe copy C:\Users\ricca\OneDrive\Documenti\Websites\mixtapes\rogie\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 -vv --dry-run --metadata-set title=test
2023/08/24 19:12:40 DEBUG : MetadataUpload map[title:test]
2023/08/24 19:12:40 DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\rclone-v1.62.2-windows-amd64\\rclone.exe" "copy" "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "-vv" "--dry-run" "--metadata-set" "title=test"]
2023/08/24 19:12:40 DEBUG : Creating backend with remote "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:12:40 DEBUG : Using config file from "C:\\Users\\ricca\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/24 19:12:40 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3", "//?/C:/Users/ricca/OneDrive/Documenti/Websites/mixtapes/rogie"
2023/08/24 19:12:40 DEBUG : Creating backend with remote "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:12:43 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:12:43 NOTICE: Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Skipped copy as --dry-run is set (size 140.063Mi)
2023/08/24 19:12:43 NOTICE:
Transferred:      140.063 MiB / 140.063 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         3.6s

2023/08/24 19:12:43 DEBUG : 4 go routines active
PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image>

PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image> C:\rclone-v1.62.2-windows-amd64\rclone.exe copy C:\Users\ricca\OneDrive\Documenti\Websites\mixtapes\rogie\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 -vv --dry-run `
>> --metadata-set title=test
2023/08/24 19:16:45 DEBUG : MetadataUpload map[title:test]
2023/08/24 19:16:45 DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\rclone-v1.62.2-windows-amd64\\rclone.exe" "copy" "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "-vv" "--dry-run" "--metadata-set" "title=test"]
2023/08/24 19:16:45 DEBUG : Creating backend with remote "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:16:45 DEBUG : Using config file from "C:\\Users\\ricca\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/24 19:16:45 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3", "//?/C:/Users/ricca/OneDrive/Documenti/Websites/mixtapes/rogie"
2023/08/24 19:16:45 DEBUG : Creating backend with remote "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:16:47 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:16:47 NOTICE: Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Skipped copy as --dry-run is set (size 140.063Mi)
2023/08/24 19:16:47 NOTICE:
Transferred:      140.063 MiB / 140.063 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.0s

2023/08/24 19:16:47 DEBUG : 4 go routines active
PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image>

asdffdsa · August 24, 2023, 5:29pm

ok, good, now remove --dry-run

if that works, then add one flag at a time until you get an error.

riccardo · August 24, 2023, 5:31pm

Sorry, here's the complete log:

PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image> C:\rclone-v1.62.2-windows-amd64\rclone.exe copy C:\Users\ricca\OneDrive\Documenti\Websites\mixtapes\rogie\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 -vv
2023/08/24 19:30:48 DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\rclone-v1.62.2-windows-amd64\\rclone.exe" "copy" "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "-vv"]      
2023/08/24 19:30:48 DEBUG : Creating backend with remote "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:30:48 DEBUG : Using config file from "C:\\Users\\ricca\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/24 19:30:48 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3", "//?/C:/Users/ricca/OneDrive/Documenti/Websites/mixtapes/rogie"
2023/08/24 19:30:48 DEBUG : Creating backend with remote "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:30:52 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:31:35 ERROR : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Failed to copy: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>d6d3d247-415f-4d5b-b197-cfe364303a2f</RequestId></Error>"
2023/08/24 19:31:35 ERROR : Attempt 1/3 failed with 1 errors and: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>d6d3d247-415f-4d5b-b197-cfe364303a2f</RequestId></Error>"
2023/08/24 19:31:36 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:31:50 INFO  : 
Transferred:       13.117 MiB / 144.840 MiB, 9%, 300.488 KiB/s, ETA 7m28s
Transferred:            0 / 1, 0%
Elapsed time:       1m2.8s
Transferring:
 * Rogie-Raregrooves_Funk…ebels_mixtape_1998.mp3:  5% /140.063Mi, 610.714Ki/s, 3m40s

2023/08/24 19:32:19 ERROR : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Failed to copy: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>800d1200-44d1-4b94-87b6-4eda1f613673</RequestId></Error>"
2023/08/24 19:32:19 ERROR : Attempt 2/3 failed with 1 errors and: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>800d1200-44d1-4b94-87b6-4eda1f613673</RequestId></Error>"
2023/08/24 19:32:21 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:32:50 INFO  : 
Transferred:       27.113 MiB / 153.180 MiB, 18%, 593.131 KiB/s, ETA 3m37s
Transferred:            0 / 1, 0%
Elapsed time:       2m2.8s
Transferring:
 * Rogie-Raregrooves_Funk…ebels_mixtape_1998.mp3:  9% /140.063Mi, 386.948Ki/s, 5m33s

2023/08/24 19:33:04 ERROR : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Failed to copy: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>4037c353-2f1d-49c1-adbc-91714f498ac7</RequestId></Error>"
2023/08/24 19:33:04 ERROR : Attempt 3/3 failed with 1 errors and: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>4037c353-2f1d-49c1-adbc-91714f498ac7</RequestId></Error>"
2023/08/24 19:33:04 INFO  :
Transferred:       27.113 MiB / 27.113 MiB, 100%, 593.131 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:      2m16.7s

2023/08/24 19:33:04 DEBUG : 2 go routines active
2023/08/24 19:33:04 Failed to copy: HTTP error 403 (403 Forbidden) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>This item has been taken offline</Resource><RequestId>4037c353-2f1d-49c1-adbc-91714f498ac7</RequestId></Error>"
PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image>

asdffdsa · August 24, 2023, 5:39pm

that is not a rclone issue.

riccardo · August 24, 2023, 5:42pm

I don't understand. The file is in my local disc and its path is right.

asdffdsa · August 24, 2023, 5:45pm

the server rejected that file, nothing rclone can do about that.

so far, nothing you posted seems to be an issue with rclone.
if you have an actual rclone quesion, please ask.

kapitainsky · August 24, 2023, 5:47pm

this should be --dry-run

kapitainsky · August 24, 2023, 5:50pm

Double check credentials in your config:

as it looks as server rejects rclone requests

riccardo · August 24, 2023, 5:51pm

@asdffdsa Ok. Maybe because I've previously uploaded such file on Internet Archive and then deleted it. Internet Archive does not allow such file to be uploaded anymore? I don't know. I will try with another file tomorrow.

@kapitainsky Here's with --dry-run

PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image> C:\rclone-v1.62.2-windows-amd64\rclone.exe copy C:\Users\ricca\OneDrive\Documenti\Websites\mixtapes\rogie\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3 -vv --dry-run
2023/08/24 19:49:48 DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\rclone-v1.62.2-windows-amd64\\rclone.exe" "copy" "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3" "-vv" "--dry-run"]
2023/08/24 19:49:48 DEBUG : Creating backend with remote "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:49:48 DEBUG : Using config file from "C:\\Users\\ricca\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/24 19:49:48 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\ricca\\OneDrive\\Documenti\\Websites\\mixtapes\\rogie\\Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3", "//?/C:/Users/ricca/OneDrive/Documenti/Websites/mixtapes/rogie"
2023/08/24 19:49:48 DEBUG : Creating backend with remote "archive.org:Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3"
2023/08/24 19:49:49 DEBUG : Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Need to transfer - File not found at Destination
2023/08/24 19:49:49 NOTICE: Rogie-Raregrooves_Funk-Soul-Rebels_mixtape_1998.mp3: Skipped copy as --dry-run is set (size 140.063Mi)
2023/08/24 19:49:49 NOTICE: 
Transferred:      140.063 MiB / 140.063 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.0s

2023/08/24 19:49:49 DEBUG : 4 go routines active
PS C:\Users\ricca\OneDrive\Documenti\Websites\sharp-image>

Also checked credentials in config and are alright.

Uploading to Internet Archive, problems with metadata (as far as I understand)

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

A log from the command with the -vv flag

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag