mamba brings pypy instead of CPython when creating env

[gabrielcnr]

I have here what it seems to be related to the issue #81

When using mamba to create an environment, it keeps pulling pypy, which is not part of the given specs. When using pure conda that does not happen.

I tested this on Mac and Linux, and both presented the same behaviour.

Creating an env with:

mamba create -n foobar "cachetools" "dataclasses" "diskcache>=4,<5" "humanize" "matplotlib>=3.1.3,<3.2" "numpy>=1.12.1" "pandas<1" "pydantic " "pytest" "pytest-mock" "pytest-datadir" "python>=3.6.10,<3.7" "python-dateutil" "pytz" "requests-mock>=1.7,<1.8" "ujson" "lxml>=3.8.0" "beautifulsoup4>=4.8 .0,<4.9"

Gives:

Looking for: ['cachetools', 'dataclasses', "diskcache[version='>=4,<5']", 'humanize', "matplotlib[version='>=3.1.3,<3.2']", "numpy[version='>=1.12.1']", "pandas[version='<1']", 'pydantic', 'pytest', 'pytest-mock', 'pytest-datadir', "python[version='>=3.6.10,<3.7']", 'python-dateutil', 'pytz', "requests-mock[version='>=1.7,<1.8']", 'ujson', "lxml[version='>=3.8.0']", "beautifulsoup4[version='>=4.8.0,<4.9']"]

conda-forge/osx-64       Using cache
conda-forge/noarch       Using cache
main/noarch              [====================] (00m:00s) No change
main/osx-64              [====================] (00m:00s) No change
Transaction

  Prefix: /Users/greis/conda/envs/boa-dev/envs/foobar

  Updating specs:

   - cachetools
   - dataclasses
   - diskcache[version='>=4,<5']
   - humanize
   - matplotlib[version='>=3.1.3,<3.2']
   - numpy[version='>=1.12.1']
   - pandas[version='<1']
   - pydantic
   - pytest
   - pytest-mock
   - pytest-datadir
   - python[version='>=3.6.10,<3.7']
   - python-dateutil
   - pytz
   - requests-mock[version='>=1.7,<1.8']
   - ujson
   - lxml[version='>=3.8.0']
   - beautifulsoup4[version='>=4.8.0,<4.9']


  Package               Version  Build              Channel                  Size
───────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────

  attrs                  20.3.0  pyhd3eb1b0_0       main/noarch             42 KB
  beautifulsoup4          4.8.2  py36_0             main/osx-64            161 KB
  brotlipy                0.7.0  py36h9ed2024_1003  main/osx-64            359 KB
  bzip2                   1.0.8  h1de35cc_0         main/osx-64            104 KB
  ca-certificates     2020.12.8  hecd8cb5_0         main/osx-64            Cached
  cachetools              4.2.0  pyhd3eb1b0_0       main/noarch             13 KB
  certifi             2020.12.5  py36hecd8cb5_0     main/osx-64            143 KB
  cffi                   1.14.4  py36h2125817_0     main/osx-64            217 KB
  chardet                 4.0.0  py36hecd8cb5_1003  main/osx-64            219 KB
  cryptography            3.3.1  py36hbcfaee0_0     main/osx-64            615 KB
  cycler                 0.10.0  py36hfc81398_0     main/osx-64             13 KB
  dataclasses               0.7  py36_0             main/osx-64             30 KB
  diskcache               4.1.0  pyh9f0ad1d_0       conda-forge/noarch      37 KB
  expat                  2.2.10  hb1e8313_2         main/osx-64            Cached
  freetype               2.10.4  ha233b18_0         main/osx-64            891 KB
  gdbm                     1.18  h8a0c380_2         conda-forge/osx-64     131 KB
  humanize                3.2.0  pyhd3eb1b0_0       main/noarch             39 KB
  icu                      58.2  h0a44026_3         main/osx-64             23 MB
  idna                     2.10  py_0               main/noarch            Cached
  importlib-metadata      3.4.0  py36h5dafb3c_0     conda-forge/osx-64      20 KB
  importlib_metadata      3.4.0  hd8ed1ab_0         conda-forge/noarch       3 KB
  iniconfig               1.1.1  py_0               main/noarch              8 KB
  kiwisolver              1.3.1  py36hb084ce4_1     conda-forge/osx-64      57 KB
  libblas                 3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libcblas                3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libcxx                 11.0.0  h4c3b8ed_1         conda-forge/osx-64     Cached
  libffi                    3.3  hb1e8313_2         main/osx-64            Cached
  libgfortran             5.0.0  9_3_0_h6c81a4c_16  conda-forge/osx-64      19 KB
  libgfortran5            9.3.0  h6c81a4c_16        conda-forge/osx-64       2 MB
  libiconv                 1.16  h1de35cc_0         main/osx-64            Cached
  liblapack               3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libopenblas            0.3.12  openmp_h54245bb_1  conda-forge/osx-64       9 MB
  libpng                 1.6.37  ha441bb4_0         main/osx-64            325 KB
  libxml2                2.9.10  h7cdb67c_3         main/osx-64              1 MB
  libxslt                1.1.34  h83b36ba_0         main/osx-64            534 KB
  llvm-openmp            11.0.0  h73239a0_1         conda-forge/osx-64     268 KB
  lxml                    4.6.2  py36h26b266a_0     main/osx-64              1 MB
  matplotlib              3.1.3  py36_0             main/osx-64             20 KB
  matplotlib-base         3.1.3  py36h9aa3819_0     main/osx-64              7 MB
  more-itertools          8.6.0  pyhd3eb1b0_0       main/noarch             38 KB
  ncurses                   6.2  h0a44026_1         main/osx-64            Cached
  numpy                  1.19.5  py36h9faff70_1     conda-forge/osx-64       5 MB
  openssl                1.1.1i  h9ed2024_0         main/osx-64            Cached
  packaging                20.8  pyhd3eb1b0_0       main/noarch            Cached
  pandas                 0.25.3  py36h0a44026_0     main/osx-64             10 MB
  pip                    20.3.3  py36hecd8cb5_0     main/osx-64              2 MB
  pluggy                 0.13.1  py36_0             main/osx-64             32 KB
  py                     1.10.0  pyhd3eb1b0_0       main/noarch             74 KB
  pycparser                2.20  py_2               main/noarch            Cached
  pydantic                1.7.3  py36h9ed2024_0     main/osx-64              2 MB
  pyopenssl              20.0.1  pyhd3eb1b0_1       main/noarch            Cached
  pyparsing               2.4.7  py_0               main/noarch            Cached
  pypy3.6                 7.3.3  h41cf889_3         conda-forge/osx-64      26 MB
  pysocks                 1.7.1  py36_0             main/osx-64             30 KB
  pytest                  6.2.1  py36hecd8cb5_0     main/osx-64            429 KB
  pytest-datadir          1.3.1  py_0               conda-forge/noarch       7 KB
  pytest-mock             3.5.1  pyhd3eb1b0_0       main/noarch             20 KB
  python                 3.6.12  5_73_pypy          conda-forge/osx-64       5 KB
  python-dateutil         2.8.1  py_0               main/noarch            224 KB
  python_abi                3.6  1_pypy36_pp73      conda-forge/osx-64       4 KB
  pytz                   2020.5  pyhd3eb1b0_0       main/noarch            Cached
  readline                  8.0  h1de35cc_0         main/osx-64            Cached
  requests               2.25.1  pyhd3eb1b0_0       main/noarch            Cached
  requests-mock           1.7.0  py_0               conda-forge/noarch      20 KB
  setuptools             51.1.2  py36hecd8cb5_3     main/osx-64            906 KB
  six                    1.15.0  py36hecd8cb5_0     main/osx-64             27 KB
  soupsieve                 2.1  pyhd3eb1b0_0       main/noarch             30 KB
  sqlite                 3.34.0  h17101e1_0         conda-forge/osx-64     Cached
  tk                     8.6.10  hb0a8c7a_0         main/osx-64            Cached
  toml                   0.10.2  pyhd8ed1ab_0       conda-forge/noarch      18 KB
  tornado                   6.1  py36h9ed2024_0     main/osx-64            642 KB
  typing_extensions     3.7.4.3  py_0               main/noarch             29 KB
  ujson                   4.0.1  py36hb1e8313_0     main/osx-64             47 KB
  urllib3                1.26.2  pyhd3eb1b0_0       main/noarch            Cached
  wheel                  0.36.2  pyhd3eb1b0_0       main/noarch            Cached
  xz                      5.2.5  h1de35cc_0         main/osx-64            Cached
  zipp                    3.4.0  pyhd3eb1b0_0       main/noarch             15 KB
  zlib                   1.2.11  h1de35cc_3         main/osx-64            Cached

  Summary:

  Install: 78 packages

  Total download: 94 MB

As you can see from above, the pypy-related packages:

pypy3.6                 7.3.3  h41cf889_3         conda-forge/osx-64      26 MB
python                 3.6.12  5_73_pypy          conda-forge/osx-64       5 KB

environment

My environment is (truncated below):

boa                       0.3.3                     dev_0    <develop>
conda                     4.9.2            py38hecd8cb5_0    main
conda-build               3.21.3           py38hecd8cb5_0    main
libsolv                   0.7.16               hab81aa3_0    conda-forge
mamba                     0.7.6            py38he6f4b85_0    conda-forge
python                    3.8.5                h26836e1_1    main

@wolfv @SylvainCorlay - I tried to debug/investigate the problem but I confess I got a bit lost and realised that I wouldn't have the time now., so I'm sorry I can't give more details here at the moment. But I was wondering if there could be an easy workaround for this problem? For example, maybe some kind of version constraint hint we can give to instruct the solver to ignore the *pypy* build strings?

[marcelotrevisani]

I tested it on my local and it works fine for Linux and mamba=0.4.4

[marcelotrevisani]

Well, I also tried with 0.7.8 and it worked fine... that is weird

[marcelotrevisani]

As a workaround you can write python as:

"python=3.6.*=*_cpython"

[gabrielcnr]

I reinstalled mamba and now I'm on mamba=0.7.8 and I continue to see the same problem on both Mac and Linux.

[gabrielcnr]

I have further information here, after a bit more investigation.
I noticed if I change the specs that I pass to mamba to consider only python instead of python>=3.6.10,<3.7", then it correctly picks up the CPython python. If try to pass python=3.7.* or python=3.8.* it also works fine. But if I try to pass python=3.6.* then it picks up the pypy build. So, I guess it's something related to python 3.6.

The only way that I and @marcelotrevisani managed to make mamba to pick up the CPython version for Python 3.6 was by specifying "python>=3.6.10,<3.7=*_cpython"

[SylvainCorlay]

@gabrielcnr, could you please report the version of libsolv that you have?
This is related to openSUSE/libsolv#395 which was only included in libsolv 0.7.15.

[gabrielcnr]

@SylvainCorlay , I've got libsolv=0.7.16

More specifically:

libsolv                   0.7.16               hab81aa3_0    conda-forge

[wolfv]

@gabrielcnr I suppose you're running natively on OSX, right?

When testing on Linux, using

CONDA_SUBDIR=osx-64 mamba create -n foobar "cachetools" "dataclasses" "diskcache>=4,<5" "humanize" "matplotlib>=3.1.3,<3.2" "numpy>=1.12.1" "pandas<1" "pydantic " "pytest" "pytest-mock" "pytest-datadir" "python>=3.6.10,<3.7" "python-dateutil" "pytz" "requests-mock>=1.7,<1.8" "ujson" "lxml>=3.8.0" "beautifulsoup4>=4.8 .0,<4.9"

mamba correctly resolves to CPython. I wonder if this is an issue on the OS X build of libsolv ... that would be interesting :)

[wolfv]

Hmm, I tried locally on OS X (intel) and your command also properly takes cpython.

What happens if you do not use main as additional channel? It's discouraged, and they never did the python_abi patches that conda-forge has to support both pypy and cpython.

Looking for: ['cachetools', 'dataclasses', "diskcache[version='>=4,<5']", 'humanize', "matplotlib[version='>=3.1.3,<3.2']", "numpy[version='>=1.12.1']", "pandas[version='<1']", 'pydantic', 'pytest', 'pytest-mock', 'pytest-datadir', "python[version='>=3.6.10,<3.7']", 'python-dateutil', 'pytz', "requests-mock[version='>=1.7,<1.8']", 'ujson', "lxml[version='>=3.8.0']", "beautifulsoup4[version='>=4.8.0,<4.9']"]

conda-forge/osx-64       Using cache
conda-forge/noarch       Using cache
Users/wolfv/mini38/conda [====================] (00m:00s) Done
Users/wolfv/mini38/conda [====================] (00m:00s) Done
robostack/osx-64         [====================] (00m:00s) No change
robostack/noarch         [====================] (00m:00s) No change
Transaction

  Prefix: /Users/wolfv/mini38/envs/foobar

  Updating specs:

   - cachetools
   - dataclasses
   - diskcache[version='>=4,<5']
   - humanize
   - matplotlib[version='>=3.1.3,<3.2']
   - numpy[version='>=1.12.1']
   - pandas[version='<1']
   - pydantic
   - pytest
   - pytest-mock
   - pytest-datadir
   - python[version='>=3.6.10,<3.7']
   - python-dateutil
   - pytz
   - requests-mock[version='>=1.7,<1.8']
   - ujson
   - lxml[version='>=3.8.0']
   - beautifulsoup4[version='>=4.8.0,<4.9']


  Package               Version  Build               Channel                  Size
────────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────────

  attrs                  20.3.0  pyhd3deb0d_0        conda-forge/noarch     Cached
  beautifulsoup4          4.8.2  py36h9f0ad1d_1      conda-forge/osx-64     157 KB
  brotlipy                0.7.0  py36ha3de816_1001   conda-forge/osx-64     357 KB
  ca-certificates     2020.12.5  h033912b_0          conda-forge/osx-64     Cached
  cachetools              4.1.1  py_0                conda-forge/noarch     Cached
  certifi             2020.12.5  py36h79c6626_1      conda-forge/osx-64     143 KB
  cffi                   1.14.4  py36hfe0777f_1      conda-forge/osx-64     217 KB
  chardet                 4.0.0  py36h79c6626_1      conda-forge/osx-64     219 KB
  cryptography            3.3.1  py36hd858a44_1      conda-forge/osx-64     620 KB
  cycler                 0.10.0  py36_0              conda-forge/osx-64      14 KB
  dataclasses               0.7  py36_0              conda-forge/osx-64      28 KB
  diskcache               4.1.0  pyh9f0ad1d_0        conda-forge/noarch      37 KB
  freetype               2.10.4  h4cff582_1          conda-forge/osx-64     890 KB
  humanize                3.2.0  pyhd8ed1ab_0        conda-forge/noarch      38 KB
  icu                      68.1  h74dc148_0          conda-forge/osx-64     Cached
  idna                     2.10  pyh9f0ad1d_0        conda-forge/noarch     Cached
  importlib-metadata      3.4.0  py36h79c6626_0      conda-forge/osx-64      20 KB
  importlib_metadata      3.4.0  hd8ed1ab_0          conda-forge/noarch       3 KB
  iniconfig               1.1.1  pyh9f0ad1d_0        conda-forge/noarch       8 KB
  kiwisolver              1.3.1  py36h615c93b_1      conda-forge/osx-64      59 KB
  libblas                 3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libcblas                3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libcxx                 11.0.0  h4c3b8ed_1          conda-forge/osx-64     Cached
  libffi                    3.3  h046ec9c_2          conda-forge/osx-64     Cached
  libgfortran             5.0.0  9_3_0_h6c81a4c_16   conda-forge/osx-64      19 KB
  libgfortran5            9.3.0  h6c81a4c_16         conda-forge/osx-64       2 MB
  libiconv                 1.16  haf1e3a3_0          conda-forge/osx-64     Cached
  liblapack               3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libopenblas            0.3.12  openmp_h54245bb_1   conda-forge/osx-64     Cached
  libpng                 1.6.37  hb0a8c7a_2          conda-forge/osx-64     Cached
  libxml2                2.9.10  h802ca4f_3          conda-forge/osx-64       1 MB
  libxslt                1.1.33  h5739fc3_2          conda-forge/osx-64     488 KB
  llvm-openmp            11.0.1  h7c73e74_0          conda-forge/osx-64     268 KB
  lxml                    4.6.2  py36ha579a0c_1      conda-forge/osx-64       1 MB
  matplotlib              3.1.3  py36_0              conda-forge/osx-64       7 KB
  matplotlib-base         3.1.3  py36h11da6c2_0      conda-forge/osx-64       7 MB
  more-itertools          8.6.0  pyhd8ed1ab_0        conda-forge/noarch      37 KB
  ncurses                   6.2  h2e338ed_4          conda-forge/osx-64     Cached
  numpy                  1.19.5  py36h08dc641_1      conda-forge/osx-64       5 MB
  openssl                1.1.1i  h35c211d_0          conda-forge/osx-64     Cached
  packaging                20.8  pyhd3deb0d_0        conda-forge/noarch      34 KB
  pandas                 0.25.3  py36h4f17bb1_0      conda-forge/osx-64      10 MB
  pip                    20.3.3  pyhd8ed1ab_0        conda-forge/noarch       1 MB
  pluggy                 0.13.1  py36h79c6626_4      conda-forge/osx-64      29 KB
  py                     1.10.0  pyhd3deb0d_0        conda-forge/noarch      73 KB
  pycparser                2.20  pyh9f0ad1d_2        conda-forge/noarch     Cached
  pydantic                1.7.3  py36h7b4949f_0      conda-forge/osx-64       2 MB
  pyopenssl              20.0.1  pyhd8ed1ab_0        conda-forge/noarch      48 KB
  pyparsing               2.4.7  pyh9f0ad1d_0        conda-forge/noarch     Cached
  pysocks                 1.7.1  py36h79c6626_3      conda-forge/osx-64      27 KB
  pytest                  6.2.1  py36h79c6626_1      conda-forge/osx-64     429 KB
  pytest-datadir          1.3.1  py_0                conda-forge/noarch       7 KB
  pytest-mock             3.5.1  pyhd8ed1ab_0        conda-forge/noarch      19 KB
  python                 3.6.12  h6c3b2c9_0_cpython  conda-forge/osx-64      21 MB
  python-dateutil         2.8.1  py_0                conda-forge/noarch     Cached
  python_abi                3.6  1_cp36m             conda-forge/osx-64     Cached
  pytz                   2020.5  pyhd8ed1ab_0        conda-forge/noarch     244 KB
  readline                  8.0  h0678c8f_2          conda-forge/osx-64     Cached
  requests               2.25.1  pyhd3deb0d_0        conda-forge/noarch      51 KB
  requests-mock           1.7.0  py_0                conda-forge/noarch      20 KB
  setuptools             49.6.0  py36h79c6626_3      conda-forge/osx-64     916 KB
  six                    1.15.0  pyh9f0ad1d_0        conda-forge/noarch     Cached
  soupsieve               2.0.1  py36h9f0ad1d_0      conda-forge/osx-64      57 KB
  sqlite                 3.34.0  h17101e1_0          conda-forge/osx-64     Cached
  tk                     8.6.10  hb0a8c7a_1          conda-forge/osx-64     Cached
  toml                   0.10.2  pyhd8ed1ab_0        conda-forge/noarch      18 KB
  tornado                   6.1  py36h20b66c6_1      conda-forge/osx-64     643 KB
  typing_extensions     3.7.4.3  py_0                conda-forge/noarch     Cached
  ujson                   4.0.1  py36h22858aa_2      conda-forge/osx-64      47 KB
  urllib3                1.26.2  pyhd8ed1ab_0        conda-forge/noarch     Cached
  wheel                  0.36.2  pyhd3deb0d_0        conda-forge/noarch      31 KB
  xz                      5.2.5  haf1e3a3_1          conda-forge/osx-64     Cached
  zipp                    3.4.0  py_0                conda-forge/noarch     Cached
  zlib                   1.2.11  h7795811_1010       conda-forge/osx-64     Cached

  Summary:

  Install: 74 packages

  Total download: 56 MB

────────────────────────────────────────────────────────────────────────────────────

Now I don't really know any further.

[gabrielcnr]

@wolfv let me try here, but I had the same problem originally on Linux (RHEL 7) at work...

[gabrielcnr]

I can confirm that I'm seeing this both on Mac and on Linux. Just tried here and the same happened... Do you need more details about my environment? Please let me know and I can provide more info.

[gabrielcnr]

@wolfv - If add -c conda-forge at the end of the command to create the env, then it resolves pulling CPython... so, the hunch was good... :-)

[wolfv]

Yeah, IMHO conda-forge and main are becoming more and more incompatible. It's like using fedora and Ubuntu packages at the same time.

One should at least use strict channel priority, better yet completely remove Main on Linux and OS X.

The defaults channel never modified their python packages to be compatible with cpython/pypy

[SylvainCorlay]

I think that it is time communicating more about using mambaforge / miniforge, rather than miniconda.

[marcelotrevisani]

The defaults channel never modified their python packages to be compatible with cpython/pypy

That is right, but, it is because the main don't care about pypy, but the python on the main channel and the python with CPython on conda-forge are basically the same

So, it is not being incompatible, the fact that it is doing extensions to support other python implementations, that is a big difference