Added support for Conan as a dependency manager

[jellespijker]

This PR is part of the following PRs:

• libArcus -> Added support for Conan as a dependency manager libArcus#122
• libSavitar -> Added support for Conan as a dependency manager libSavitar#36
• libCharon -> Added support for Conan as a dependency manager libCharon#33
• libnest2d -> Added support for Conan as a dependency manager libnest2d#2
• pynest2d -> Added support for Conan as a dependency manager pynest2d#11
• Uranium -> Added support for Conan as a dependency manager Uranium#720
• CuraEngine -> Added support for Conan as a dependency manager CuraEngine#1479

The purpose of these changes is to set up a dependency manager for Cura and her repositories. Cura uses both third-party and Ultimaker maintained dependencies, written in both Python and C++ (or mixtures of both). Not all of these dependencies can be downloaded with the help of a dependency manager such as pip. This makes setting up Cura from source, a pain in the $%^#$$%^. See the graph below for the current dependencies. Adding to the complexity is the way how we're currently consuming third-party dependencies; Some have to be present on the system/provided by the user, some are shipped within the repo, while others are downloaded by CMake.

All of the above-mentioned PRs and this one, add a conanfile.py to the root of this project. This is a recipe written in Python which instructs Conan (https://docs.conan.io/en/latest/) how to build and package the repository in such a way that it can be reused by other dependencies. If a required dependency has no binary for your OS and compiler it will build that dependency from scratch and store it in the cache.

Installing Conan

Installing Conan clean

pip install conan
conan config install https://github.com/jellespijker/conan-config.git

Reinstalling Conan

If you have already played with Conan it is good to clean it up and make sure you have the right profiles installed

pip install conan --upgrade
rm ~/.conan/profile/default
rm -rf ~/.conan/data/
conan profile new default --detect
conan config install https://github.com/jellespijker/conan-config.git

Installing Cura

git clone https://github.com/ultimaker/Cura.git
git checkout conan
conan install . -pr cura_release.jinja -if venv --build=missing

For a more detailed description see the README.md in this repository https://github.com/jellespijker/conan-um

For testing purposes, I have set up a small home artifactory server that can be used by Ultimaker employees. The url for this server was automatically added in the conan config install ... step. The artifactory server contains a recipe for SIP and Python

You can use your own profiles for this, but I have personally tested and developed them with my own jinja template profiles on Linux Manjaro with a GCC compiler, Mac OS Big Sur with a Clang compiler and Windows 11 with a Visual Studio 2019 compiler.
These profiles can be installed with the conan config install https://github.com/jellespijker/conan-config. Make sure you add -pr cura_release.jinja to your install instructions

Conan allows for multiple ways of working. Either the exiting package can be used from the cache, or if you want to work on multiple repositories you can put that repo in editable mode such that the xxx-config.cmake in the project that is depending on the other, will point to the paths of your repo, see https://docs.conan.io/en/latest/developing_packages/editable_packages.html

Because the best practice method to use Conan, which is also the preferred way in Conan 2.0, is to use ( https://docs.conan.io/en/latest/reference/conanfile/tools/cmake.html ) With the tools CMakeDeps, CMakeToolchain, and CMake. The CMakeDeps class will generate xxx-config.cmake files per dependency, while the CMakeToolchain will generate a toolchain to be passed to CMake -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake These two prepare the way for CMake to actually build the project such that it won't need to change the CMakeLists.txt. Allowing Conan to be optional and not mandatory. Unfortunately, our repositories have organically grown CMakeLists.txt mixing old and modern syntax and a build system that uses CMake 3.13.

Todo:

• Install vitrual venv automatically using the compiled Python recipe
• Add scm_to_conandata to the conan configuration repo

Still, WIP at the moment, since I'm finalizing some last changes across all repos

[MatthewGentoo]

Hi, requiring Conan to build doesn't work very well for distros so none of them have updated to Cura 5 yet.

Would it be possible to re-arrange the Conan files and CMake scripts so that Conan is optional, and we can still manage dependencies by hand? (Or alternatively - would you merge pull requests that do this?) I had a play with libArcus before the pyArcus split and it didn't seem to require too many changes.

Thanks,
Matthew

[richfelker]

@MatthewGentoo I have an open ticket related to this, #12941. It has a partial solution that allows building with preinstalled deps, but I haven't tested it with Arcus because I build without Arcus. It should be easy to adapt though. I'd welcome input from distro folks on how to transform my hack into something Cura upstream might be able to accept.

[jellespijker]

Hi @MatthewGentoo,

TLDR; For the direct future I don't see us switching strategy. Ultimaker can't spare the resources to set up and maintain multiple build methods, with the current budget and teams size.

Us switching to Conan and going all in on both Conan and JFrog Artifactory stems from the fact that it is a huge cost reduction (saving a yearly salary of 1fte developer) on our build servers alone.

We also have reproducable builds now. This goes for all our supported OSes Windows, Linux and Mac. No longer a dependency hell, but downgraded to a dependency purgatory.

But a bigger advancement is the fact that the whole process isn't one big overly complex sequential build script anymore. It is now setup in small(er) single responsibility modulair scripts, were each of our own dependencies is responsible for its own output.

By deciding to go all in on Conan, we could simplify our CMakeLists.txt a lot. This ensures that more people in the team can work on build related tickets instead of 1 or 2 senior devs.

You mentioned that Cura 5.x aren't distributed with the system package manager anymore. I can understand why. I worked on the switch to PyQt6 for Cura 5.0, we first tried to implement that in our "old" setup. It nearly gave me PTSD. Getting it up and running on good old Linux (or even Windows) was relatively easy, but setting it up for that #&€@&& Apple device was a huge undertaking (this statement doesn't reflect the company view and is purely personal 😉)

The only way to finally get it to work was to partially switch to Conan, which ensured that all compiler, linker flags, pre process definitions, rpaths handling, the whole shebang, was taken care of in a uniform way.

All of the above mentioned reasons are great but this switch also has a downside, apart from some so called teething problems and that is that we messed up workflows for all of you wonderful Linux package maintainers. For that I'm sorry.

Although we won't switch our build proces to allow Conan as an optional build method ourself, we will look seriously into any PR that would help us both out. Provided that the maintenance and complexity for such a system is relatively low, and it works on all three OSes withe the same toolchains we are currently using on our GitHub runners.

On a personal note I want to say that I believe Conan will be adopted by more and more projects. It has the potential to become an industry accepted "standard" such as CMake.
I also think this switch will help the Cura community and will attract new Cura developers. When our recipes and workflow grow more robust, setting up a development environment will take up minutes in stead of hours.

I'm tagging our new community manager @MariMakes so she is aware of you guys needs and wants as well.

[richfelker]

TLDR; For the direct future I don't see us switching strategy. Ultimaker can't spare the resources to set up and maintain multiple build methods, with the current budget and teams size.

I don't think anyone is asking you to setup and maintain another build method. What folks are asking for is just a way to bypass the Conan dependency fetching and building, and instead be able to build with these libraries already available in the search path the compiler toolchain on its own, or as instructed by CMake, is searching. If you see the patch I used in the linked issue, this is not any big ask. It's basically just doing target_link_libraries instead of find_package.

[jellespijker]

@richfelker I understand that and we'll look into any PR that would facilitate that request without creating an extra burden on our resources.