Q: how to build cmake module without downloading cmake source code and building it?

[kloczek]

IMO current build procedure is a bit odd because after testing that cmake is available it tries to download cmake cource code to build it

Details

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
running egg_info
creating src/cmake.egg-info
writing src/cmake.egg-info/PKG-INFO
writing dependency_links to src/cmake.egg-info/dependency_links.txt
writing entry points to src/cmake.egg-info/entry_points.txt
writing requirements to src/cmake.egg-info/requires.txt
writing top-level names to src/cmake.egg-info/top_level.txt
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
reading manifest file 'src/cmake.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory '_skbuild'
warning: no previously-included files matching '*' found under directory 'CMake-src'
adding license file 'LICENSE_Apache_20'
adding license file 'LICENSE_BSD_3'
adding license file 'AUTHORS.rst'
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
* Building wheel...


--------------------------------------------------------------------------------
-- Trying 'Ninja' generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is GNU 13.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 13.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Ninja' generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    /usr/bin/cmake /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5 -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-install/src/cmake/data -DPYTHON_VERSION_STRING:STRING=3.8.18 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/lib/python3.8/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DCMAKE_BUILD_TYPE:STRING=Release

Not searching for unused variables given on the command line.
-- The CXX compiler identification is GNU 13.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- ***************************************************
-- Build CMake from source: ON
-- ***************************************************
CMake Warning (dev) at /usr/share/cmake/Modules/ExternalProject.cmake:3136 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake/Modules/ExternalProject.cmake:4345 (_ep_add_download_command)
  CMakeLists.txt:138 (ExternalProject_add)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- SuperBuild - CMakeProject-src-download
-- SuperBuild - CMakeProject-src-download - URL: https://github.com/Kitware/CMake/releases/download/v3.27.5/cmake-3.27.5.tar.gz
-- SuperBuild - CMakeProject-src-download - CMakeProject_SOURCE_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/CMake-src
-- Looking for __GLIBC__
-- Looking for __GLIBC__ - found
-- Looking for __GLIBC_MINOR__
-- Looking for __GLIBC_MINOR__ - found
-- SuperBuild -   CMakeProject-build
-- SuperBuild -   CMakeProject-build - CMakeProject_BINARY_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build
-- SuperBuild -   CMakePythonDistributions
-- Configuring done (1.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build
[1/27] Performing download step (download, verify and extract) for 'CMakeProject-src-download'
-- Downloading...
   dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.27.5.tar.gz'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/Kitware/CMake/releases/download/v3.27.5/cmake-3.27.5.tar.gz'
-- [download 0% complete]
-- [download 1% complete]
-- [download 2% complete]
-- [download 3% complete]
-- [download 4% complete]
-- [download 5% complete]
-- [download 6% complete]
-- [download 7% complete]
-- [download 8% complete]
-- [download 9% complete]
-- [download 10% complete]
-- [download 11% complete]
-- [download 12% complete]
-- [download 13% complete]
-- [download 14% complete]
-- [download 15% complete]
-- [download 16% complete]
-- [download 17% complete]
-- [download 18% complete]
-- [download 19% complete]
-- [download 20% complete]
-- [download 21% complete]
-- [download 22% complete]
-- [download 23% complete]
-- [download 24% complete]
-- [download 25% complete]
-- [download 26% complete]
-- [download 27% complete]
-- [download 28% complete]
-- [download 29% complete]
-- [download 30% complete]
-- [download 31% complete]
-- [download 32% complete]
-- [download 33% complete]
-- [download 34% complete]
-- [download 35% complete]
-- [download 36% complete]
-- [download 37% complete]
-- [download 38% complete]
-- [download 39% complete]
-- [download 40% complete]
-- [download 41% complete]
-- [download 42% complete]
-- [download 43% complete]
-- [download 44% complete]
-- [download 45% complete]
-- [download 46% complete]
-- [download 47% complete]
-- [download 48% complete]
-- [download 49% complete]
-- [download 50% complete]
-- [download 51% complete]
-- [download 52% complete]
-- [download 53% complete]
-- [download 54% complete]
-- [download 55% complete]
-- [download 56% complete]
-- [download 57% complete]
-- [download 58% complete]
-- [download 59% complete]
-- [download 60% complete]
-- [download 61% complete]
-- [download 62% complete]
-- [download 63% complete]
-- [download 64% complete]
-- [download 65% complete]
-- [download 66% complete]
-- [download 67% complete]
-- [download 68% complete]
-- [download 69% complete]
-- [download 70% complete]
-- [download 71% complete]
-- [download 72% complete]
-- [download 73% complete]
-- [download 74% complete]
-- [download 75% complete]
-- [download 76% complete]
-- [download 77% complete]
-- [download 78% complete]
-- [download 79% complete]
-- [download 80% complete]
-- [download 81% complete]
-- [download 82% complete]
-- [download 83% complete]
-- [download 84% complete]
-- [download 85% complete]
-- [download 86% complete]
-- [download 87% complete]
-- [download 88% complete]
-- [download 89% complete]
-- [download 90% complete]
-- [download 91% complete]
-- [download 92% complete]
-- [download 93% complete]
-- [download 94% complete]
-- [download 95% complete]
-- [download 96% complete]
-- [download 97% complete]
-- [download 98% complete]
-- [download 99% complete]
-- [download 100% complete]
-- verifying file...
       file='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.27.5.tar.gz'
-- Downloading... done
-- extracting...
     src='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.27.5.tar.gz'
     dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/CMake-src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[12/27] Performing configure step for 'CMakeProject-build'
loading initial cache file /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/initial-cache.txt
loading initial cache file /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build-prefix/tmp/CMakeProject-build-cache-Release.cmake
-- The C compiler identification is GNU 13.2.1
-- The CXX compiler identification is GNU 13.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking if compiler supports C11 _Thread_local
-- Checking if compiler supports C11 _Thread_local - yes
-- Checking if compiler supports needed C++17 constructs
-- Checking if compiler supports needed C++17 constructs - yes
-- Checking if compiler supports C++ make_unique
-- Checking if compiler supports C++ make_unique - yes
-- Checking if compiler supports C++ filesystem
-- Checking if compiler supports C++ filesystem - yes
-- Looking for unsetenv
-- Looking for unsetenv - found
-- Looking for environ
-- Looking for environ - not found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Checking whether wstring is available
-- Checking whether wstring is available - yes
-- Checking whether C compiler has ptrdiff_t in stddef.h
-- Checking whether C compiler has ptrdiff_t in stddef.h - yes
-- Checking whether C compiler has ssize_t in unistd.h
-- Checking whether C compiler has ssize_t in unistd.h - yes
-- Checking whether C compiler has clock_gettime
-- Checking whether C compiler has clock_gettime - yes
-- Checking whether CXX compiler has setenv
-- Checking whether CXX compiler has setenv - yes
-- Checking whether CXX compiler has unsetenv
-- Checking whether CXX compiler has unsetenv - yes
-- Checking whether CXX compiler has environ in stdlib.h
-- Checking whether CXX compiler has environ in stdlib.h - no
-- Checking whether CXX compiler has utimes
-- Checking whether CXX compiler has utimes - yes
-- Checking whether CXX compiler has utimensat
-- Checking whether CXX compiler has utimensat - yes
-- Checking whether CXX compiler struct stat has st_mtim member
-- Checking whether CXX compiler struct stat has st_mtim member - yes
-- Checking whether CXX compiler struct stat has st_mtimespec member
-- Checking whether CXX compiler struct stat has st_mtimespec member - no
-- Looking for include files sys/types.h, ifaddrs.h
-- Looking for include files sys/types.h, ifaddrs.h - found
-- Checking whether CXX compiler has rlimit64
-- Checking whether CXX compiler has rlimit64 - yes
-- Looking for C++ include execinfo.h
-- Looking for C++ include execinfo.h - found
-- Checking whether backtrace works with this C++ compiler
-- Checking whether backtrace works with this C++ compiler - yes
-- Looking for C++ include dlfcn.h
-- Looking for C++ include dlfcn.h - found
-- Checking whether dladdr works with this C++ compiler
-- Checking whether dladdr works with this C++ compiler - yes
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - found
-- Checking whether cxxabi works with this C++ compiler
-- Checking whether cxxabi works with this C++ compiler - yes
-- Checking whether CXX compiler has getloadavg
-- Checking whether CXX compiler has getloadavg - yes
-- Checking whether <ext/stdio_filebuf.h> is available
-- Checking whether <ext/stdio_filebuf.h> is available - yes
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR - Success
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID - Success
-- Looking for connect in socket;
-- Looking for connect in socket; - not found
-- Looking for gethostname
-- Looking for gethostname - found
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)
CMake Error at Utilities/cmcurl/CMakeLists.txt:587 (message):
  Could not find OpenSSL.  Install an OpenSSL development package or
  configure CMake with -DCMAKE_USE_OPENSSL=OFF to build without OpenSSL.


-- Configuring incomplete, errors occurred!
FAILED: CMakeProject-build-prefix/src/CMakeProject-build-stamp/CMakeProject-build-configure /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build-prefix/src/CMakeProject-build-stamp/CMakeProject-build-configure
cd /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build && /usr/bin/cmake -C /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/initial-cache.txt -GNinja -C/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build-prefix/tmp/CMakeProject-build-cache-Release.cmake -S /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/CMake-src -B /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build && /usr/bin/cmake -E touch /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build-prefix/src/CMakeProject-build-stamp/CMakeProject-build-configure
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
    cmkr.make(make_args, install_target=cmake_install_target, env=env)
  File "/usr/lib/python3.8/site-packages/skbuild/cmaker.py", line 697, in make
    self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
  File "/usr/lib/python3.8/site-packages/skbuild/cmaker.py", line 742, in make_impl
    raise SKBuildError(msg)


    =============================DEBUG ASSISTANCE=============================
    If you are seeing a compilation error please try the following steps to
    successfully install cmake:
    1) Upgrade to the latest pip and try again. This will fix errors for most
       users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
    2) If on Linux, with glibc < 2.12, you can set PIP_ONLY_BINARY=cmake in
       order to retrieve the last manylinux1 compatible wheel.
    3) If on Linux, with glibc < 2.12, you can cap "cmake<3.23" in your
       requirements in order to retrieve the last manylinux1 compatible wheel.
    4) Open an issue with the debug information that follows at
       https://github.com/scikit-build/cmake-python-distributions/issues

    Python: 3.8.18
    platform: Linux-6.4.8-200.fc38.x86_64-x86_64-with-glibc2.34
    glibc: glibc 2.38.9000
    machine: x86_64
    bits: 64
    pip: n/a
    setuptools: 68.2.2
    scikit-build: 0.17.6
    PEP517_BUILD_BACKEND=setuptools.build_meta
    =============================DEBUG ASSISTANCE=============================

An error occurred while building with CMake.
  Command:
    /usr/bin/cmake --build . --target install --config Release --
  Install target:
    install
  Source directory:
    /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5
  Working directory:
    /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.27.5/_skbuild/linux-x86_64-3.8/cmake-build
Please check the install target is valid and see CMake's output for more information.


ERROR Backend subprocess exited when trying to invoke build_wheel

Q: what neds to be passed to pep517 build procedure params execution to build cmake module with reuse installed cmake binary?

[henryiii]

It uses cmake to build the version of cmake you've requested.

If using PEP 517, you can use the dynamic build requirements to only add cmake if required (same with ninja, etc). Scikit-build-core does this for you, or you can manually implement it (simple example in https://scikit-build.readthedocs.io/en/latest/usage.html#adding-cmake-as-building-requirement-only-if-not-installed-or-too-low-a-version).

[kloczek]

But cmake in this actual case is already present in build env!! 🤔
Why not just check is cmake/ninjja/etc if they are present? 🤔
Issue is that other modules do not check availability of the cmake command but they want this module and this module bundles own cmake executable.

[henryiii]

See #451 (comment). The problem is with those other modules - they should implement get_requires_for_build_wheel() or use scikit-build-core. If you request the Python cmake distribution, you should get the Python cmake distribution!

[kloczek]

Still it is not explanation why cmake python module cannot be wrapper around system installed cmake command.
Using your logic none of the python modules should use gcc/clang compilers and cython should have own copy of those compilers.

[henryiii]

That would be awesome, but I'm not sure anyone is up to providing them. GLIBC hacks and all - good luck. By your logic, there should be a "gcc" module that they depend on that is actually a dummy module. And since the entire point of cmake is to provide cmake, your imaginary "gcc" should provide it in some cases too, and not just be a wrapper! You absolute don't need the cmake module if you don't want cmake provided. You can just find and use the CMake command. Just like we do with the GCC command today.

The way modern packaging works, you can't make decisions when installing the wheel (eggs supported this, but not wheels). And installing packages always goes through wheels. So only the wheel-making step could make this decision, which is not the install step!

[jcfr]

why cmake python module cannot be wrapper around system installed cmake command.

If a "valid" version of cmake is available on the system, by using scikit-build-core none will be installed. See https://scikit-build-core.readthedocs.io/en/latest/configuration.html#cmake-and-ninja-minimum-versions

The cmake python module is a convenient wrapper that ends up installing a launcher executable¹ only used to lookup and run the specific cmake executable packaged in the wheel.

Footnotes

1. https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#creating-executable-scripts ↩

[kloczek]

That would be awesom

Awesome?
Rally awesome to have module which NO ONE currently needs? 🤔

And since the entire point of cmake is to provide cmake,

Issue is that it is not.
cmake as module wit is own copy of entire cmake noto only binaries but all documentation and modules is REQUIRED my other modules.
It is not possible to build cmake python module which is able to use already system installed cmake.

All because in _version.py files HARDCODED cmake version. Instead that it should be short code which obtain that version from the output of the cmake --version command output.
The same in init.py there is no code which is checking cmake/ctest binaries availability. That code simple returns paths from 'cmake` python module installed tree.

[kloczek]

If a "valid" version of cmake is available on the system, by using scikit-build-core none will be installed.

So that would be the bug because in my build procedure (as rpm spec file) I have such valid version in build env and it still tries download source coulde and build it.
Nevertheless it is not any bug because in current pep517 based build procedure there is no even SINGLE line which is trying to check is cmake already installed in build env.