how is babby formed

[natevw]

Would be nice at some point to have an official overview of how this works. I did find https://www.npmjs.org/package/lua-colony#readme to be helpful although maybe a bit has changed since then.

AFAICT, compilation is more of a syntax conversion — e.g. JavaScript uses braces, Lua uses "end"; plus a few tricks/conventions to pull off "various things" (?). But once you've got the code roughed into shape, the rest of the work is done by "monkeypatched" Lua builtins (primitive strings/numbers/etc and object/array-ish tables) with methods and operators to the point where they ± match JavaScript's behavior.

JavaScript code like ""+{abc:42,fn:function () {}}.abc more or less becomes:

((""))+(_obj({
  ["abc"]=(42),
  ["fn"]=(function (this)
--[[28]] if true then return (false); end
end)
}).abc)

… and the + operator ends up doing JavaScript-ish string concatentation due to a custom .__add metatable method on the Lua side (code in runtime's colony-init.lua). The .abc property lookup similarly goes through js_proto_get via a custom .__index method^Wmetamethod.

[tcr]

Agreed. js_pairs, js_obj, js_typeof, js_new should also be called out as expected variables that the source<->source converter expects, as well as the bit library to provide JS's bit operations.

That said, I've spent this week on https://github.com/tcr/colonyjit-compiler, which will replace this compiler with a straight bytecode compiler in C/C++. This should simplify a lot of expectations about how the compilation works.

However, should probably update the README starting with your post summary and completing it, so it's still useful as the js-to-lua project everyone wants.

[natevw]

Ah, interesting. Under ColonyJIT the JS will be handled with the same metamethods, therefore ± the same runtime code, right?

(I'm trying to finish UCS-2 strings and in the process have been figuring out why/how they end up being UTF-8 in the first place. The answer there, I think, is that Lua doesn't provide any "metamethod"-style hooks for primitive instantiation, and therefore we just have to kind of keep the values we end up with rather than e.g. tracking the actual string buffer as UTF16 in a metatable entry. Since Lua's string type is basically just an array of char-sized integers, colony has made the assumption that source code and therefore the string literals within it are UTF-8. I'm guessing it might still be possible to pretty much ignore the original "Lua string"ness and do everything on top of a memoized UTF-16 representation, but that's not the approach you've taken so far and I'm not sure it's worth changing directions at this point or not…)

[tcr]

Precisely, at first the compiler will be equivalent.

So you're right in that the compiler is basically just encoding UTF-8 strings as UTF-8 byte sequences. (UTF-8 is ideal for a number of low-memory reasons, I think we've had this discussion before—at least for now I'd be interested in pursuing this approach and we can fall back if it becomes unwieldly. Plus the amount of UTF-8 tooling/libraries out there exceeds UCS-2 by far.)

Metatables encode all the "visible" aspects of UTF-8ness from char integral indexing ("string"[5] == 'n') or "string".length == 6) so from the Lua layer my approach was to just have these methods iterate the string as UTF-8 and Do The Right Thing when it encounters an astral character. This is naturally unoptimized, but most of it's relegated to C functions so I'm not wholly concerned. Random indexing is pretty uncommon and we can probably memoize string length in the VM eventually.