Building on PyPy 2.6 (official docker image) breaks with CompileError

[npiganeau]

Building on PyPy 2.6 (with cffi 1.1) breaks with ValidationError whereas it works perfectly on PyPy 2.5. Tests were made with official PyPy docker images (tags '2-2.6' vs '2-2.5').

PyPy 2.6:

npiganeau@ndp-host:~$ docker run -t -i --name pypy pypy:2-2.6 bash
root@8557c3209fe5:/# pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
  Cloning git://github.com/amauryfa/lxml.git (to cffi) to /src/lxml-cffi
    Complete output from command python setup.py egg_info:
    src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c: In function ‘_cffi_check_struct__xmlOutputBuffer’:
    src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c:1624:24: warning: initialization from incompatible pointer type
       { xmlBuffer * *tmp = &p->buffer; (void)tmp; }
                            ^
    src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c:1625:24: warning: initialization from incompatible pointer type
       { xmlBuffer * *tmp = &p->conv; (void)tmp; }
                            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c: In function ‘_cffi_e__xmlRelaxNGValidErr’:
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7845:8: error: ‘XML_RELAXNG_OK’ undeclared (first use in this function)
       if ((XML_RELAXNG_OK) > 0 || (long)(XML_RELAXNG_OK) != 0L) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7845:8: note: each undeclared identifier is reported only once for each function it appears in
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7855:8: error: ‘XML_RELAXNG_ERR_MEMORY’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_MEMORY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_MEMORY) != 1UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7865:8: error: ‘XML_RELAXNG_ERR_TYPE’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_TYPE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPE) != 2UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7875:8: error: ‘XML_RELAXNG_ERR_TYPEVAL’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_TYPEVAL) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPEVAL) != 3UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7885:8: error: ‘XML_RELAXNG_ERR_DUPID’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_DUPID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DUPID) != 4UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7895:8: error: ‘XML_RELAXNG_ERR_TYPECMP’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_TYPECMP) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPECMP) != 5UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7905:8: error: ‘XML_RELAXNG_ERR_NOSTATE’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_NOSTATE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOSTATE) != 6UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7915:8: error: ‘XML_RELAXNG_ERR_NODEFINE’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_NODEFINE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NODEFINE) != 7UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7925:8: error: ‘XML_RELAXNG_ERR_LISTEXTRA’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_LISTEXTRA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTEXTRA) != 8UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7935:8: error: ‘XML_RELAXNG_ERR_LISTEMPTY’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_LISTEMPTY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTEMPTY) != 9UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7945:8: error: ‘XML_RELAXNG_ERR_INTERNODATA’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_INTERNODATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERNODATA) != 10UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7955:8: error: ‘XML_RELAXNG_ERR_INTERSEQ’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_INTERSEQ) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERSEQ) != 11UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7965:8: error: ‘XML_RELAXNG_ERR_INTEREXTRA’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_INTEREXTRA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTEREXTRA) != 12UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7975:8: error: ‘XML_RELAXNG_ERR_ELEMNAME’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMNAME) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNAME) != 13UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7985:8: error: ‘XML_RELAXNG_ERR_ATTRNAME’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ATTRNAME) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRNAME) != 14UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7995:8: error: ‘XML_RELAXNG_ERR_ELEMNONS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMNONS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNONS) != 15UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8005:8: error: ‘XML_RELAXNG_ERR_ATTRNONS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ATTRNONS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRNONS) != 16UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8015:8: error: ‘XML_RELAXNG_ERR_ELEMWRONGNS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMWRONGNS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMWRONGNS) != 17UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8025:8: error: ‘XML_RELAXNG_ERR_ATTRWRONGNS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ATTRWRONGNS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRWRONGNS) != 18UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8035:8: error: ‘XML_RELAXNG_ERR_ELEMEXTRANS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMEXTRANS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMEXTRANS) != 19UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8045:8: error: ‘XML_RELAXNG_ERR_ATTREXTRANS’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ATTREXTRANS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTREXTRANS) != 20UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8055:8: error: ‘XML_RELAXNG_ERR_ELEMNOTEMPTY’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMNOTEMPTY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNOTEMPTY) != 21UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8065:8: error: ‘XML_RELAXNG_ERR_NOELEM’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_NOELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOELEM) != 22UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8075:8: error: ‘XML_RELAXNG_ERR_NOTELEM’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_NOTELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOTELEM) != 23UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8085:8: error: ‘XML_RELAXNG_ERR_ATTRVALID’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ATTRVALID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRVALID) != 24UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8095:8: error: ‘XML_RELAXNG_ERR_CONTENTVALID’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_CONTENTVALID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_CONTENTVALID) != 25UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8105:8: error: ‘XML_RELAXNG_ERR_EXTRACONTENT’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_EXTRACONTENT) <= 0 || (unsigned long)(XML_RELAXNG_ERR_EXTRACONTENT) != 26UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8115:8: error: ‘XML_RELAXNG_ERR_INVALIDATTR’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_INVALIDATTR) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INVALIDATTR) != 27UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8125:8: error: ‘XML_RELAXNG_ERR_DATAELEM’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_DATAELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DATAELEM) != 28UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8135:8: error: ‘XML_RELAXNG_ERR_VALELEM’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_VALELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_VALELEM) != 29UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8145:8: error: ‘XML_RELAXNG_ERR_LISTELEM’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_LISTELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTELEM) != 30UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8155:8: error: ‘XML_RELAXNG_ERR_DATATYPE’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_DATATYPE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DATATYPE) != 31UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8165:8: error: ‘XML_RELAXNG_ERR_VALUE’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_VALUE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_VALUE) != 32UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8175:8: error: ‘XML_RELAXNG_ERR_LIST’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_LIST) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LIST) != 33UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8185:8: error: ‘XML_RELAXNG_ERR_NOGRAMMAR’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_NOGRAMMAR) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOGRAMMAR) != 34UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8195:8: error: ‘XML_RELAXNG_ERR_EXTRADATA’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_EXTRADATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_EXTRADATA) != 35UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8205:8: error: ‘XML_RELAXNG_ERR_LACKDATA’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_LACKDATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LACKDATA) != 36UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8215:8: error: ‘XML_RELAXNG_ERR_INTERNAL’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_INTERNAL) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERNAL) != 37UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8225:8: error: ‘XML_RELAXNG_ERR_ELEMWRONG’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_ELEMWRONG) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMWRONG) != 38UL) {
            ^
    src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8235:8: error: ‘XML_RELAXNG_ERR_TEXTWRONG’ undeclared (first use in this function)
       if ((XML_RELAXNG_ERR_TEXTWRONG) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TEXTWRONG) != 39UL) {
            ^
    Building lxml version 3.4.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.28
    Traceback (most recent call last):
      File "<builtin>/app_main.py", line 75, in run_toplevel
      File "<builtin>/app_main.py", line 601, in run_it
      File "<string>", line 20, in <module>
      File "/src/lxml-cffi/setup.py", line 232, in <module>
        **setup_extra_options()
      File "/src/lxml-cffi/setup.py", line 145, in setup_extra_options
        STATIC_CFLAGS, STATIC_BINARIES)
      File "setupinfo.py", line 167, in ext_modules
        import lxml.etree
      File "src/lxml/../lxml-cffi/etree.py", line 26, in <module>
        from .xmlerror import _initThreadLogging, clear_error_log
      File "src/lxml/../lxml-cffi/xmlerror.py", line 150, in <module>
        libraries=['xml2'])
      File "/usr/local/lib_pypy/cffi/api.py", line 373, in verify
        lib = self.verifier.load_library()
      File "/usr/local/lib_pypy/cffi/verifier.py", line 96, in load_library
        self._compile_module()
      File "/usr/local/lib_pypy/cffi/verifier.py", line 192, in _compile_module
        outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
      File "/usr/local/lib_pypy/cffi/ffiplatform.py", line 38, in compile
        outputfilename = _build(tmpdir, ext)
      File "/usr/local/lib_pypy/cffi/ffiplatform.py", line 65, in _build
        raise VerificationError('%s: %s' % (e.__class__.__name__, e))
    VerificationError: CompileError: command 'cc' failed with exit status 1

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /src/lxml-cffi

PyPy 2.5

npiganeau@ndp-host:~$ docker run -t -i --name pypy pypy:2-2.5 bash
root@6b84fa93cfd9:/# pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
You are using pip version 7.0.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
  Cloning git://github.com/amauryfa/lxml.git (to cffi) to /src/lxml-cffi
Installing collected packages: lxml-cffi
  Running setup.py develop for lxml-cffi
Successfully installed lxml-cffi

[aglyzov]

I believe these bindings are outdated and cannot be used with today's PyPy.
After much fiddling with the sources I was able to build it on OSX 10.9.5 (Maverics) / macports + PyPy 2.6.
However it was super unstable, spat out RuntimeError when querying an elem for its children (in random places). Totally unusable IMO. So I deleted it all together.

[amauryfa]

The error messages seem to indicate a different version of libxml, PyPy's version is not relevant here.
Can you check which version of libxml you are using?

[aglyzov]

2.9.2

[npiganeau]

@amauryfa Actually it is the same version of libxml in both docker instances: 2.9.1.

root@6689ce5271c5:/# pypy --version
Python 2.7.9 (9c4588d731b7, Mar 23 2015, 16:30:30)
[PyPy 2.5.1 with GCC 4.6.3]
root@6689ce5271c5:/# ldconfig -v|grep xml
        libxml2.so.2 -> libxml2.so.2.9.1

root@3b5593dee3af:/# pypy --version
Python 2.7.9 (295ee98b6928, May 31 2015, 07:29:04)
[PyPy 2.6.0 with GCC 4.8.2]
root@3b5593dee3af:/# ldconfig -v |grep xml
        libxml2.so.2 -> libxml2.so.2.9.1

[aglyzov]

Then perhaps it's an OSX thing

[aglyzov]

And while we're at it, in my tests I noticed it took the cffi+pypy lxml ~3-5 times more time than cpython lxml to parse the same HTML page. I wasn't expecting that at all, thought cffi would be faster.
Have you seen anything like it?

[amauryfa]

2015-06-21 22:41 GMT+02:00 Aleksandr Glyzov notifications@github.com:

And while we're at it, in my tests I noticed it took the cffi+pypy lxml
~3-5 times more time than cpython lxml to parse the same HTML page. I
wasn't expecting that at all, thought cffi would be faster.
Have you seen anything like it?

I remember doing some benchmarks, and the results were in the same range as
CPython (some were 40% faster, others 40% slower...)
But it's very possible that some paths are still much slower.

Also, be sure to warm up pypy's JIT by running 1000 loops of your function
before doing any timings.

Amaury Forgeot d'Arc

[nichochar]

+1 on this error. What is the suggested solution here?

[aldarund]

+1. same error here on debian without any docker. Any solutions?

[aglyzov]

@nichochar and @aldarund guys, try this fork: https://github.com/aglyzov/lxml.git
I have added explicit libxml/relaxng.h includes everywhere and it started to build ok.