cross-compiling on x64_64 when target is of same arch as host fails

[th0ma7]

Really long story short, Our build framework starts by building a minimal python38 and includes the necessary wheels, including crossenv. It happens that all version >=1.0 of crossenv makes PyNaCl to fail to build in our cross-compiling environment, only for x86_64 architecture.

I presume that crossenv has a bug that gets triggered in cross-compiling environment where if the target is of the same arch of the host it goes outside of its boundary instead of using the passed library paths of our toolchain.

Our PR where we hit the issue is: SynoCommunity/spksrc#4902
Issue opened on PyNaCl (although problem seems to be with crossenv): pyca/pynacl#688

[benfogle]

Thanks for waiting while I got to this.

I am having trouble reproducing this. I created separate Python installs, each for x86_64 and tried to build PyNaCl using crossenv:

$ ./py8-build/bin/python3.8 -m crossenv ./py8-host/bin/python3.8 cross-null
WARNING: CC is a compound command (['gcc', '-pthread'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CC='gcc' and CFLAGS='-pthread'
WARNING: CXX is a compound command (['g++', '-pthread'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CXX='g++' and CXXFLAGS='-pthread'
$ . cross-null/bin/activate
(cross) $ pip install pynacl
Collecting pynacl
  Using cached https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting cffi>=1.4.1 (from pynacl)
  Using cached https://files.pythonhosted.org/packages/00/9e/92de7e1217ccc3d5f352ba21e52398372525765b2e0c4530e6eb2ba9282a/cffi-1.15.0.tar.gz
Collecting six (from pynacl)
  Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl)
  Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Building wheels for collected packages: pynacl
  Building wheel for pynacl (PEP 517) ... done
  Created wheel for pynacl: filename=PyNaCl-1.4.0-cp38-cp38-linux_x86_64.whl size=1161477 sha256=1b4abf86db705050ecc4c1b71990a201e2d0460205773a36b92a9529f9e62aeb
  Stored in directory: /home/user/.cache/pip/wheels/c2/d9/ca/81e002fb0cee770c198715dace9edab919daba3ae49bd287a0
Successfully built pynacl
Installing collected packages: pycparser, cffi, six, pynacl
  Running setup.py install for cffi ... done
Successfully installed cffi-1.15.0 pycparser-2.20 pynacl-1.4.0 six-1.16.0

Can you let me know:

• What is the precise version(s) of Python that you are using.
• How are you creating crossenv? (It was unclear to me from the issues you linked.)
• What commands are you using to build?

[th0ma7]

• I'm using python 3.8.12
• Crossenv is using spksrc framework that download Synology's DSM linux toolchain for multiple archs
• That's embeded within the framework. It ends-up calling https://github.com/SynoCommunity/spksrc/blob/master/mk/spksrc.wheel.mk which in turns calls:

pip wheel --no-binary :all: --cache-dir $(PIP_DIR) --no-deps --wheel-dir $(WHEELHOUSE) --no-build-isolation --requirement requirements.txt

EDIT: I've sort of translated most variables to give a general idea of the pip command being called.

[benfogle]

I have still been unable to reproduce this, and I can't think of a likely source of the error. This is a complete shot in the dark, but does adding something like --manylinux manylinux2010 when creating crossenv do anything? Probably the biggest gotcha for 1.1 is that crossenv no longer builds manylinux wheels by default, so the cffi module will depend on the system libffi rather than a bundled libffi like the pre-compiled wheel does. (This would still be odd: build-python can still use pre-build wheels, and it should be build-python's version of cffi that gets imported.)

[th0ma7]

So if I understand correctly (please correct me if I'm wrong), when building our host python (to be used later in the build process) we are currently including:

@$(MSG) Installing pip, setuptools, cffi and cross env
@$(RUN) wget https://bootstrap.pypa.io/get-pip.py
@$(RUN) $(PYTHON) get-pip.py "pip==21.3.1"
@$(PIP) install "setuptools==58.3.0" "cffi==1.15" "crossenv==1.0"

After cross-compiling the target python, we call-up crossenv and install the basic necessary wheels such as:

@$(RUN) $(PYTHON_NATIVE) -m crossenv $(STAGING_INSTALL_PREFIX)/bin/python$(PKG_VERS_MAJOR_MINOR) $(WORK_DIR)/crossenv/
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) wget https://bootstrap.pypa.io/get-pip.py
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) build-python get-pip.py "pip==21.3.1"
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) python get-pip.py "pip==21.3.1"
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) build-pip install "setuptools==58.3.0"
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) pip install "setuptools==58.3.0"
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) build-pip install "wheel==0.37.0"
. $(WORK_DIR)/crossenv/bin/activate && $(RUN) pip install "wheel==0.37.0"
@echo CROSSENV=$(WORK_DIR)/crossenv/bin/activate >> $@

Later-on we are calling this host python to build the pure and cross python wheels as needed using pip wheel with all the necessary CFLAGS and LDFLAGS and all.

Although, interestingly, in the case of PyNaCl wheel we use a legacy build method such as (and now that I notice this, it makes me wonder why isn't it using our regular pip wheel installation instead, something to look at):

# Python 3 case: using crossenv helper
	@. $(CROSSENV) && $(RUN) PYTHONPATH=$(PYTHONPATH) python -c "import setuptools;__file__='setup.py';exec(compile(open(__file__).read().r
eplace('\r\n', '\n'), __file__, 'exec'))" $(BUILD_ARGS) bdist_wheel -d $(WHEELHOUSE)
endif

So if I get this right, you're shot in the dark would be to add --manylinux manylinux2010 somewhere in this call I guess?

EDIT: In parallel I'm testing out the feasibility to use our regular cross-compilation pip wheel call against crossenv==1.1.4 on our host python to see if it solves the issue (adding @hgy59 for his awareness).

[benfogle]

It should be added as a parameter when setting up the crossenv helper, something like:

@$(RUN) $(PYTHON_NATIVE) -m crossenv --manylinux manylinux2010 $(STAGING_INSTALL_PREFIX)/bin/python$(PKG_VERS_MAJOR_MINOR) $(WORK_DIR)/crossenv/

[th0ma7]

I ended-up solving the issue by using our regular pip wheel cross-compilation environment instead of the legacy call to setup.py. Using that instead I can now build all my dependencies using latest version of crossenv.

I haven't tried the --manylinux manylinux2010 in our legacy wheel cross-building environment. But I'll keep it in mind if a new issue of that same kind arise.

Thnx for your help.

[th0ma7]

@benfogle Interestingly I'm facing once more this problem. This time as pip cannot help me for my issue pyca/cryptography#7671 I have to run it using python setup.py bdist_wheel --py-limited-api=cp36 method. As such my first attempt was to try out --manylinux manylinux2010 when generating my cross-environment (also tried manylinux1 and manylinux2014, no luck).

But I think I've partially found where the issue is: no LDFLAGS are being set into the crossenv when host == target (and perhaps a few other as well).

In every case my build environment contains CFLAGS, CPPFLAGS, ... , LDFLAGS. This is true for all archs, including x86_64 using the arch specific toolchain (being no different).

Here's a comparison between aarch64 and x86_64 related to the LDFLAGS using a quick grep. Below is a working aarch64:

work-aarch64-6.1/crossenv/cross/bin$ ./python -m sysconfig | grep LDFLAGS
	CONFIGURE_LDFLAGS = "-L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib"
	CONFIGURE_LDFLAGS_NODIST = ""
	CONFIG_ARGS = "'--host=aarch64-unknown-linux-gnueabi' '--build=i686-pc-linux' '--prefix=/var/packages/python310/target' '--enable-shared' '--without-static-libpython' '--enable-ipv6' '--without-ensurepip' '--enable-loadable-sqlite-extensions' '--with-computed-gotos=yes' '--disable-test-modules' 'ac_cv_buggy_getaddrinfo=no' 'ac_cv_file__dev_ptmx=no' 'ac_cv_file__dev_ptc=no' 'ac_cv_have_long_long_format=yes' '--with-ssl-default-suites=openssl' '--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' 'build_alias=i686-pc-linux' 'host_alias=aarch64-unknown-linux-gnueabi' 'CC=/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-gcc' 'CFLAGS=-I/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/usr/include -I/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install/var/packages/python310/target/lib -I /home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install/var/packages/python310/target/include' 'LDFLAGS=-L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib ' 'CPPFLAGS=-I/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/usr/include -I/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/include ' 'CPP=/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-cpp' 'PKG_CONFIG_LIBDIR=/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib/pkgconfig'"
	LDFLAGS = "-L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib -L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib"
	LDFLAGS_NODIST = ""
	OPENSSL_LDFLAGS = "-L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install/var/packages/python310/target/lib"
	PY_CORE_LDFLAGS = "-L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib -L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib"
	PY_LDFLAGS = "-L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib -L/home/spksrc/rustc2/spksrc/toolchain/syno-aarch64-6.1/work/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/lib -L/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/rustc2/spksrc/spk/python310/work-aarch64-6.1/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib"
	PY_LDFLAGS_NODIST = ""

And now the equivalent, broken x86_64:

work-x64-6.1/crossenv/cross/bin$ ./python -m sysconfig | grep LDFLAGS
	CONFIGURE_LDFLAGS = ""
	CONFIGURE_LDFLAGS_NODIST = ""
	CONFIG_ARGS = "'--prefix=/usr/local' '--enable-ipv6' '--without-static-libpython' '--without-ensurepip' '--disable-test-modules' '--with-system-ffi' '--with-ssl-default-suites=openssl' 'CC=' 'CFLAGS=' 'LDFLAGS=' 'CPPFLAGS=' 'CPP='"
	LDFLAGS = ""
	LDFLAGS_NODIST = ""
	OPENSSL_LDFLAGS = ""
	PY_CORE_LDFLAGS = ""
	PY_LDFLAGS = ""
	PY_LDFLAGS_NODIST = ""