Find ways to leverage sectorlisp

[stevefan1999-personal]

https://github.com/jart/sectorlisp/blob/main/sectorlisp.S

This thing can fit in 512 bytes on the MBR

Here's the full article about it: https://justine.lol/sectorlisp2/

[stikonas]

The question though is not just whether it can fit to MBR but whether it is suitable for building more complex stuff.

[stevefan1999-personal]

The question though is not just whether it can fit to MBR but whether it is suitable for building more complex stuff.

You can! As part of the Lisp family of language, sectorlisp itself is nonetheless, the smallest metacircular evaluator (which implies homoiconicity, which means that the source code itself can also be treated as data as well, and I can use the program in the same language to generate code that it can run in the same language too, i.e. template expansion in C++ except I can actually copy and paste it).

That means we can first write stage0 in Lisp, then use sectorlisp as a bootstrapping mechanism, to implement an even more powerful Lisp runtime that can run our stage0.

This should form one of the smallest bootstrap because everything is within the Lisp closure, that you don't even need to have Forth or a subset of C in the follow up, so everything would just be fitting in tight and right for stage0 just by using Lisp alone.

For example, we can use sectorlisp to write a "kernel" and try to control some CPU resources first, say like find means to allocate stack space, or to implement malloc, then we can leverage the homoiconicity of Lisp to write a Lisp evaluator within Sectorlisp, and eventually run something like Scheme or Common Lisp, entirely in memory.

Caveat: the more sophisticated program itself, would be on some kind of ROM, however. But heck, if we can extend sectorlisp to do macro evaluation, that means we don't even need Scheme or CL, since lambda calculus itself is Turing complete, and now we have lambda calculus on the language level right at runtime. Being Turing complete means you can evaluated anything, except on codes exhibiting the (in)famous halting problem (well, that depends on your point of view whether undecidable and subsequently semi-decidable problem would be any meaningful).

But anyway, from that point on, we elevated from a primitive boot state into having an interpreter that is not just easier than implementing BASIC, but being even more powerful than BASIC from a fundamental level. You can do everything you wanted.

This would be a crazy, insanely challenging, but quite sound and valid idea I suppose.

[stikonas]

That means we can first write stage0 in Lisp, then use sectorlisp as a bootstrapping mechanism, to implement an even more powerful Lisp runtime that can run our stage0.

But how would you do this in practice? I'm not claiming it can't be done (since I didn't look into it with enough details). To bootstrap on baremetal we kind of need to have an ability to read/write and also jump into another code.

So let's say we are trying to implement something like hex0 in sectorlisp. It reads some data (say input for more powerful hex1 linker), writes output. But then we need to execute that newly built hex1. How can we do this in sectorlisp?

[stevefan1999-personal]

That means we can first write stage0 in Lisp, then use sectorlisp as a bootstrapping mechanism, to implement an even more powerful Lisp runtime that can run our stage0.

But how would you do this in practice? I'm not claiming it can't be done (since I didn't look into it with enough details). To bootstrap on baremetal we kind of need to have an ability to read/write and also jump into another code.

Interesting question, so what you mean is that we don't have the ability to do absolute & indirect jumps. But what if we just put everything in sectorlisp...entirely?

I mean, sure, you lost the ability to run arbitary host instructions, since everything is hidden behind the "virtual machine" of sectorlisp.

Yeah, you do lost a degree of freedom of code execution since everything is wrapped inside Lisp...but what if we can find ways to evaluate a sectorlisp program that emits programs that host instruction set accepts to a certain memory location, then we "eject" sectorlisp runtime and jump to that code instead?

Obviously it is kind of cheating, since we also leveraged the fact that most CPU follows more or less, a modified von neumann architecture design, that is, again, giving us the ability to treat code/string as data, which is related to the homoiconicity idea (except this time we treat code as data structure within the language itself) but I guess this is reasonable and not really a big hack.

Yet the problem is, after we jump out and escaped from the sectorlisp world...there is no coming back, practically speaking. So it means now you have to face the x86 wilderness. You can run anything you want just like before.

So let's say we are trying to implement something like hex0 in sectorlisp. It reads some data (say input for more powerful hex1 linker), writes output. But then we need to execute that newly built hex1. How can we do this in sectorlisp?

Ha, you got me. But the problem now becomes: why do you need hex0 in sectorlisp in the first place, given that sectorlisp is even more powerful than hex0 or hex1, so powerful to a point, that we can even jump straight into stage2? Or let's say: can our bootstrapping program simply be sectorlisp directly in ROM, just like BIOS or UEFI? And sectorlisp already implemented a "shell" that you can write Lisp code directly from keyboard input...So is hex0/hex1 even necessary?

Now I guess this would not be a very popular defense, so if you still insist on getting hex0 first...you can rewrite the same thing in Lisp no problem, since it is just a basic lexer that doesn't even need much advanced parsing techniques to begin with, you just need to parse hex numbers and write it to a designated part of memory, and execute that memory when ready. Then you will need the "eject" idea I mentioned aboved: after collecting all the hex strings, you process them in sectorlisp, write the equivalent stream of instructions into a pool of executable memory, then you jump into it.

But again, bare in mind, from that point on, sectorlisp lose control and it will just become an ordinary computer running arbitary host instructions.

Anyway, if you don't want sectorlisp as the very foundation, and want to type sectorlisp from hex0/hex1, then sure you will still need to keep hex0/hex1 before sectorlisp in the chain of execution. However, sectorlisp is already capable of replacing hex0/hex1 and can go straight into a mix between stage1 and stage2, that we are missing some high level features to control the resources of the machine.

[stevefan1999-personal]

@stikonas also take a look at this, should be able to explain everything: https://youtu.be/xD14SLU2u2k?si=AFxtDNVmgfb46Zik&t=554