"cutover --run-v2v=true" runs into symbol lookup error: librpm_sequoia.so.1: undefined symbol: EVP_idea_cfb64, version OPENSSL_3.0.0

[preuss-b1-systems]

Let's start with the command I run and error I run into:

Command:

docker run -it --rm --privileged \
--network host \
-v /dev:/dev \
-v /opt/vmware-vix-disklib-distrib:/usr/lib64/vmware-vix-disklib:ro  \
--env-file <(env | grep OS_)   ghcr.io/vexxhost/migratekit:main   cutover   \
--vmware-endpoint <some endpoint>   \
--vmware-username <some username>   \
--vmware-password <some password>   \
--vmware-path <some path>   \
--flavor flavor id \
--network-mapping <network mapping> \
--availability-zone nova \
--run-v2v=true \
--debug

Error output:

libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
/usr/bin/supermin: symbol lookup error: /lib64/librpm_sequoia.so.1: undefined symbol: EVP_idea_cfb64, version OPENSSL_3.0.0
libguestfs: trace: v2v: launch = -1 (error)
virt-v2v-in-place: error: libguestfs error: /usr/bin/supermin exited with 
error status 127, see debug messages above

A few remarks:

1. the migrate command before showed no errors and worked flawlessly
2. the cutover command work properly if I run with --runv2v=false (but I need to inject virtio drivers)
3. note that openssl 3.0.0 does not include IDEA ciphers:

[fedora@migrations-vm-b1-fedora44 Docker]$ docker run -it --rm --privileged --network host -v /dev:/dev -v /opt/vmware-vix-disklib-distrib:/usr/lib64/vmware-vix-disklib:ro  --env-file <(env | grep OS_)   --entrypoint /usr/bin/openssl ghcr.io/vexxhost/migratekit:main  ciphers -v ALL | grep -i idea
[fedora@migrations-vm-b1-fedora44 Docker]$

I went on from here creating my own versionof the migratekit container with enabled legacy ciphers:
Dockerfile:

FROM ghcr.io/vexxhost/migratekit:main

USER root

# Copy custom OpenSSL configuration
COPY openssl-legacy.cnf /etc/ssl/openssl-legacy.cnf

ENV OPENSSL_CONF=/etc/ssl/openssl-legacy.cnf

CMD ["/bin/bash"]

openssl-legacy.cnf :

openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

Result:

[fedora@migrations-vm-b1-fedora44 Docker]$ docker run -it --rm --privileged --network host -v /dev:/dev -v /opt/vmware-vix-disklib-distrib:/usr/lib64/vmware-vix-disklib:ro  --env-file <(env | grep OS_)   --entrypoint /usr/bin/openssl migratekit-legacy:latest ciphers -v ALL | grep -i idea
IDEA-CBC-SHA                   SSLv3   Kx=RSA      Au=RSA   Enc=IDEA(128)              Mac=SHA1

Still no IDEA-CFB or IDEA-CFB-64 and even worse no EVP_idea_cfb64.

I'm not sure but I think this is a compatibility issue the binaries of librpm_sequoia and openssl. I guess that worked pre OpenSSL 3.0.0

I tried to build the image using Fedora40 as the base image, but golang 1.25 is only shipped in Fedora43 and up. (Of course I tried Fedora43 but that also already contains openssl 3.0.0.)

I tested this using Ubuntu24.04 and Fedora44 as docker hosts. (hoping the compatibility issue was somehow with libraries of the host system) - same result.

Step to reproduce:

1. Take a fresh Ubuntu24.04 Cloud image or a fresh Fedora44 cloud image.
2. apt/dnf update/upgrade
3. disable apparmor on ubuntu
4. install docker
5. install vddk
6. run docker container commands to migrate and cutover a vmware vm with run-v2v

Solution

I'm not really sure what the best path forward is: newer librpm-sequoia? newer supermin? older openssl?

[rhochmayr]

Hey there,

I had the same issue and fixed it by limiting the scope of LD_LIBRARY_PATH to nbdkit child only.

See:
rhochmayr@6545775
(The change to the dockerfile can be ignored.)

// nbdkit's VDDK plugin needs VMware's bundled libraries under
// /usr/lib64/vmware-vix-disklib/lib64 on its library search path.
// Previously this was done with os.Setenv(), which mutates migratekit's
// own process environment and is therefore inherited by every subsequent
// child — including virt-v2v-in-place and the supermin it spawns.
//
// VMware VDDK ships its own libcrypto.so.3 / libssl.so.3 under that
// directory. When they take precedence over Fedora's /lib64 versions,
// librpm_sequoia.so.1 (used by supermin) ends up linked against VDDK's
// libcrypto, which lacks the EVP_idea_cfb64 symbol that Fedora's
// rpm-sequoia requires, producing:
//
// /usr/bin/supermin: symbol lookup error: /lib64/librpm_sequoia.so.1:
// undefined symbol: EVP_idea_cfb64, version OPENSSL_3.0.0
//
// Using cmd.Env (instead of os.Setenv) confines LD_LIBRARY_PATH to the
// nbdkit invocation so downstream children see a clean environment.

Happy to start a fresh fork and re-apply changes so they can be merged back.

[preuss-b1-systems]

Thank You!

This worked for me too!

I can't really tell if limiting the scope of LD_LIBRARY_PATH to nbdkit may have some other side effect, elsewhere in the code, but if not this looks like a very clean solution. So please fork, reapply and pull request.