I've been working on a Docker image with an Open Watcom toolchain for CI/CD purposes. Shall we cooperate?

[volkertb]

Hi!

@jmalak pointed me to this project when I ran into an issue regarding the project that I'm working on. That project is debian-open-watcom, which builds and publishes Docker images containing an installed and ready-to-use Open Watcom v2 toolchain on top of a (slim) Debian base image, for use in CI/CD projects, such as GitHub Actions Workflows.

It's actually a variant of another project I made earlier named debian-djgpp, which is similar, but with a DJGPP toolchain instead.

There are definitely overlaps between your project and mine, and it looks like (in hindsight) I may have reinvented the wheel here and there.

On the other hand, I believe that offering the toolchain prepackaged in a container image may result in faster pipelines than having a GitHub Action download and install the toolchain in the runner on each build. Also downloaded Docker images can be cached in the runners, whereas the GitHub Action would always download and install the toolchain in every pipeline run. And also the images could be used in other CI/CD systems such as GitLab and others as well as GitHub Actions Workflows.

I'm sure there are pros and cons to both approaches, though.

But regarding the overlapping parts, perhaps we could cooperate a bit?

One question I do have w.r.t. your project: when version 2.0 is selected, exactly which build does the setup-watcom action install? New 2.0 builds get built and published upstream regularly.

Anyway, I hope we can be of use to one and other in some way. I'm curious to read about what you all think.

Regardless, thank you for your contributions to the developer community and have a Happy New Year! 🙂

[madebr]

Hello!

I've been thinking about this for the past month.
Just kidding, I forgot to answer. Sorry about that!

I think both projects (docker + "GitHub native" typescript) are useful:

• you can easily pull a docker container to your dev machine, and debug why things are (not) working
• the github script can be used on whatever os/docker container it supports. For example you can run it inside a FreeBSD/Fedora/alpine/Windows/... docker container or virtual machine in the github azure cloud.

We could implement caching using @actions/cache, but this is not done because

1. open-watcom/setup-watcom never became the limiting factor
2. most people use 2.0, which is unstable and the default. The latest development version should then be used and not some tagged or "old cached" version. Because jmalek is constantly pushing fixes to open-watcom, the cache would become outdated quickly. (this also answers your question)

The latest argument is of course invalid when you use a fixed tagged version, such as 1.9.
And we can of course cache 2.0 development versions by checking the hash of the master branch.
(That's what I do in libsdl-org/setup-sdl)

Because of the 2nd point, I think we want to keep using GitHub javascript. By switching to docker containers, we would suddenly lose Windows support and have to rebuild the docker container every time jmalek pushes a new commit to open-watcom master.
We would also have to create images for older open-watcom releases, and update it when the base container updates.

For other ci services, such as GitLab, that don't have a similar concept as actions, I think a docker container would be useful.

That being said, of course we're open to collaboration.

[volkertb]

Makes sense. Perhaps it would be useful to change this issue into a discussion, to continue our conversation on ideas for collaboration, and then create isssues for any concrete things that come up? Thanks.

[madebr]

I think the discussions page of open-watcom-v2 would be good place to do this, if only to advertise your repo/work.
Alternatively, feel free to drop a line on the open-watcom discord server.

[stuaxo]

I did a demake of some of the bits of setup-openwatcom to bash - it's only 1.9 on Linux, but enough for me to get going - in no way a replacement for any of the projects here

https://gist.github.com/stuaxo/c140314d7ea07b32865df354a67ef188

[jmalak]

It is interesting that all people try to run OW installer (in background) to install OW instead simple unzip content of installer to preffered directory (it is standard ZIP archive) and setup a few environment variables. What you need additionaly do is setup execute bit on files in binl subdirectory (it is done by installer, nothing more). Installation can be done by a few shell commands only. It is used by setup-watcom, it works for any OW version.

[stuaxo]

It didn't occur to me that it's a zip file (I didn't run file on it) - in that case unzipping makes more sense, still it was a fun(?) trying to work out how to get the installer to cooperate.

If I come back to this I'll make it use unzip as that's more portable (and sane).

[jmalak]

Of cause you don't know it, but it is best ask on Github or on Discord server to get info about OW internals and it will safe lot of your work. It is as self extracting ZIP archive. Executable installer program is add to ZIP archive (it contains full binary image). you can use any installer image they differ only in executable installer program but ZIP image is same. OW doesn't need any special setup for OS except a Environment variables.

[stuaxo]

Embarassingly: I found a version, I started (but not finished) where I do treat the installer as a zip file - it turns out I am pretty forgettful and easily distracted :)

I think it makes sense that this project is in a proper language like typescript: at the same time, my motivation for a translation to bash was that typescript + node is quite a heavy requirement for an installer (though totally fine in the world of gh-actions).

I hope you guys collaborate: it would be great if .debs were released by the OW project itself, if only on the github, if that was the case I probably wouldn't have started experiementing - though some of it is just for the fun of scripting.

(FreeDOS packages would be really useful for people doing retro computing bits too).

EDIT: It looks like FreeDOS has OW packages, I should look into that.

[jmalak]

The problem is that people from Linux tends to compile all from sources instead of usage compiled version. Also it is GNU world with all these specific thing which is not compatible with anything other except GNU tools. OW is multiplatform that cannot be connected to any OS specific formats or tools etc. it is reason why we support our installer which is multiplatform and independent on OS as much as possible for every supported hosts and don't create any host OS specific packages.
Now following hosts are supported DOS, 16/32/64-bits Windows, OS/2, 32/64 bits Linux. I think FreeDos will use OW DOS installer or some ZIP files. All these tends to do thing specific for theirs OS instead of OS neutral way, because they are concentrated only on single OS. The same issue with building system, we use our own, because existing build systems don't cover all our supported host OSes.

[volkertb]

@stuaxo I just found your workaround for marking the right Linux binaries as executable after unpacking the installer as a ZIP file. I stole it and used it in a new release of the debian-open-watcom container image. I hope you don't mind. 😉

Thanks for sharing that!