Platform independent use of librclone (Python API)

Dev Discussions
1

We are developing an open-source tool for managing bioscience projects that needs to be platform independent. Thanks to all here for your work and support of rclone, this is a fantastic tool that suits our needs perfectly.

We plan to use rclone within Python and would like to leverage librclone (e.g. rclone/rclone.py at master · rclone/rclone · GitHub) to use rclone through the API rather than use subprocess calls. However setting this up locally, use of librclone requires go installation to compile a librclone shared object (or deployment of pre-compiled librclone.so / .dll) which (as far as I understand) will reduce the flexibility around platform independence.

Could anyone advise on the best way to proceed? Happy to help contribute as would like to use the API where possible rather than subprocess calls. Many Thanks.

2

Hi Joseph,

I guess you are targeting the mainstream Linux platforms (used in bioscience) and therefore can find a rclone executable for all your main platforms on the download page: https://rclone.org/downloads/

It would therefore be relatively easy to install/update rclone with a single OS call from Python using the installation script:
https://rclone.org/downloads/#script-download-and-install and it will install the correct executable for the platform.

I would therefore start by investigating an approach where your tool:

  1. Executes the installation script (at least at first execution)
  2. Starts the remote control demon: rclone rcd
  3. Makes HTTP/REST calls to the remote control API: Remote Control / API
3

Do you think we should be distributing a librclone.so or librclone.a for the architectures we support?

What architectures do you wish to support?

It might be you only need to build a few librclone.so's as librclone.so only has a few symbols it imports and these are pretty standard

$ ldd librclone.so 
	linux-vdso.so.1 (0x00007ffec73f4000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0e4c168000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0e4fd4b000)

$ nm -D librclone.so  | grep ' U ' | cut -d@ -f2 | sort | uniq
GLIBC_2.2.5
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.32
GLIBC_2.3.4
GLIBC_2.34
GLIBC_2.4

So librclone.so should work with glibc 2.34 or newer as I've just compiled it.

It would probably be better to compile it against an older glibc.

4

Thanks both, we are looking to support Windows, macOS and linux as we have users on all these systems. That is a good idea for the package to install the relevant rclone version based on current operating system (had initially considered including all binaries and selecing the relevant one from within code, which would blow up the install size).

I think it could be useful to provide compiled librclone, just in the interest of avoiding duplication of effort across developers. I do not have alot of experience compilling binaries across platforms (am in the process of doing more research). Will .dll compiled on windows 10 have backwards / forwards compatability? I presume .so compiled on older macOS will not be compatable with M1 chip?

Many Thanks,
Joe

5

Windows .dll will work with any version of Windows I think.

I'm not sure about macOS versions, but M1 (arm) .so won't work with the older amd64 macs.

Linux is a potential compatibility nightmare, but compiling the .so against the oldest version of glibc in use will make it compatible with all I think.

If you can, just using the binary will save you headaches I think! Compiling shared libraries for many systems is hard work.

6

Fully agree!

Thinking a bit more about your situation you may also want to avoid a version headache, where your users could encounter (different difficult to reproduce) errors due to different versions of rclone being used with the same version of your tool.

This can be avoided by hardcoding each version of your tool to download a specific version of rclone from https://downloads.rclone.org/ - but then you probably need to also do some of the things in the install scripts (e.g. setting the execute permission).

7

Thanks both that's very useful, I'll use the API as mentioned above and avoid the hassle!

Thanks also for advice on distributing the rclone binary - that's a good point about versioning, will definately want to keep that consistent.

Thanks for the help!

Best WIshes,
Joe

2 Likes
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.