NATIVECOLORS
-----------------------------------------------------------------------------------
<macton@gmail.com>
<macton@insomniacgames.com>

UPDATE 27 Jan 2013: I think I'm going to drop the host-side nexe support. (1) It's a
lot of stuff to juggle, (2) I really don't want to track down the issues getting it
to work on an arm host (not to mention msys on Windows), and (3) this project is
already *way* big enough. #protip Cut ruthlessly.

UPDATE 25 Jan 2013: Back to the action. Next step: Getting everything to work properly 
with ARM Chromebook as host.

This is an evolving project. It initially started out conceptually as just adding a
few standard interfaces to make porting some things easier. But quickly, a new 
end-game has revealed itself.

It's currently in a working state, but very far from the long term goals. Both are 
described below.

Input, questions and participation are welcomed.


LONG-TERM GOALS
---------------
Create a self-contained, portable user-space. 

  - Install a light posix-ish OS in-browser with no host required (aside from 
    web/socket server)
  - Boot into OS. Full filesystem mounted (with meta-data support for 
    user-permissions, etc.)
  - Run shell and typical unix commands (e.g. BusyBox)
  - Build new applications locally, in browser with gcc
  - Support package manager (e.g. apt-get)


CURRENT STATE
-------------
NativeColors is currently divided into three main support libraries called 
NativeYellow, NativeBlack and NativeBlue.
  - Only Pepper 19 is supported.
  - All libraries are functional, it's more a question of how much functionality.
  - The interfaces are evolving and definitely not stable.
  - Have a look at the test files for current working state and functionality.


NATIVEYELLOW
------------
nativecolors/host/src/libraries/sys-headers/
nativecolors/host/src/libraries/nativeyellow/

NativeYellow is the support library for running NaCl executables on the host 
platform. Consider it an extension to libc. Mostly it's a few small patches to 
standard functions to help with porting applications and libraries. Modifications 
to system headers (e.g. stdlib.h, pthread.h) are separated into sys-headers to 
simplify dependencies.

All host executables are run through sel_ldr and are limited to the functionality 
of that irt. However, the libraries and executables should be usable without i
modification on any platform that supports sel_ldr.

Status:
  - ioctl TIOCGWINSZ gets LINES and COLUMNS from the environment for tty display
  - pthread rw_lock supported
  - writev supported
  - helper functions added: e.g. strurlendcode, strurldecode

See tests: nativecolors/host/src/tests/


NATIVEBLACK
-----------
nativecolors/client/src/libraries/nativeblack/

NativeBlack is a C API partially generated from the NaCl C PPAPI for client 
(in-browser) applications.  The goals of the API are:

  - Provide only a thin wrapper on the PPAPI (no or minimal runtime cost)
  - Simplify the API without changing it singificantly. 
    - e.g. Assume only a single instance per module.
    - The names and purpose of the API functions should be coresspond directly to 
      the C PPAPI
  - Add basic utility functions
    - All additional functions or structures outside of the C PPAPI should be 
      marked "#nablack-utility"
    - The TWEAK macros give access to specific variables from the browser for 
      development.
  - Add basic conventions

Status:
  - Full Pepper 19 API
  - Simplified single-instance interface
  - All PPAPI interfaces are initialized on startup

You can use the NativeBlack lib alone if you want a slightly simplified interface 
to the NaCl PPAPI where most of the boilerplate code is taken care of. You just 
provide the entry functions.

See example: nativecolors/client/src/tests/macton-01-nativeblack-hello/


NATIVEBLUE
----------
nativecolors/client/src/libraries/sys-headers/
nativecolors/client/src/libraries/nativeblue/

NativeBlue is where things start to get interesting. This is a support library for 
client (in-browser) applications which builds on top of NativeBlack to provide a 
more posix-like interface. 

  - At the basic level, it's equivalent to NativeYellow for the client. It just 
    fills in some of the gaps in libc.
  - Standard fileio functions are supported (using File System API to write 
    in-client files.)
  - stdin/stderr/stdout TTY is managed and provided in-browser by hterm
  - PPAPI functions are placed on to a command queue and processed once per frame.
    The provided functions can be called from any thread at any time.

See example: nativecolors/client/src/tests/macton-05-nativeblue-fileio/

CLIENT PORTS
------------
  - hterm ported
    - Dependency on chrome.i18n patched out

  - pdcurses (libcurses.a) ported
    - All output commands functional
    - TTY output to hterm
    - #todo stdin support (kbm)
    - No 256 color palette support
    - see example src: nativecolors/client/src/ports/curses-firework
    - see example app: nativecolors/client/srv/www/curses-firework

  - cJSON (libcjson.a) ported
    - Fully functional. 
    - Used directly by nativeblue

  - libarchive (libarchive.a) ported
    - Support for ISO9960 writes removed
    - Support for mtree removed
    - Not fully tested. 
    - #todo add tests

  - zlib (libz.a) ported
    - Fully functional

HOST PORTS
----------
  - pdcurses (libcurses.a) ported
    - All output commands functional
    - TTY output to hterm
    - #todo stdin support (kbm)
    - No 256 color palette support
    - see example src: nativecolors/host/src/ports/curses-firework
    - see example nexe: nativecolors/host/bin/$(TARGET)/nacl-curses-firework

  - echo ported
    - see nexe: nativecolors/host/bin/$(TARGET)/nacl-echo

  - libarchive (libarchive.a) ported
    - Support for ISO9960 writes removed
    - Support for mtree removed
    - Not fully tested. 
    - #todo add tests

  - zlib (libz.a) ported
    - Fully functional
    - see example src: nativecolors/host/src/ports/nacl-gunzip
    - see example app: nativecolors/host/bin/$(TARGET)/nacl-gunzip


HOW-TO USE
----------
  - Make sure nacl_sdk is installed and working
  - export NATIVECOLORS_ROOT to this directory in your shell.


DIRECTORY STRUCTURE
-------------------
Loosely based on fhs http://www.pathname.com/fhs/pub/fhs-2.3.html

/client
  - Files intended for use on the client. But built on the host.

/client/include
  - Installed include files for client libraries and executables.

/client/lib
  - Installed libraries for client executables.

/client/src
  - Buildable source for client libraries and executables.

/client/src/packages
  - Read-only hosted external source.  
  - Do not change from original source. Any necessary patches should be applied
    in /client/src/ports.

/client/src/ports
  - Build scripts and patches to build packages. May be libraries or applications.

/client/src/libraries
  - Libraries (non-ports) for building client applications

/client/src/projects
  - Client applications (non-ports)

/client/src/tests
  - Test applications to run on client.

/client/srv
  - Files expected to be served from a server running on the environment hosting 
    these files

/client/srv/www
  - Files expected to be served from an http server. This directory is assumed to 
    be the root of that server.
  - Projects from /client/src/ports and /client/src/projects are installed in this 
    directory.

/host
  - Files intended for use on the host. But built on the host.

/host/include
  - Installed include files for host libraries and executables.

/host/lib
  - Installed libraries for host executables.

/host/bin
  - Pre-built binaries which can be run on the host.

/host/src
  - Buildable source for host libraries and executables.

/host/src/packages
  - Read-only hosted external source.  
  - Do not change from original source. Any necessary patches should be applied 
    in /host/src/ports.

/host/src/ports
  - Build scripts and patches to build packages. May be libraries or applications.

/host/src/libraries
  - Libraries (non-ports) for building host applications

/host/src/projects
  - Client applications (non-ports)

/host/src/tests
  - Test applications to run on host.


TODO
----
Potential roadmap:
  - userspace kernel (ioctl)
  - loop device driver
  - FUSE driver
  - FUSE file based on AncientFS/UnixFS 
    http://osxbook.com/blog/2008/12/16/more-user-space-file-system-goodies/
  - FUSE ext driver http://alperakcan.org/?open=projects&project=fuse-ext2
  - Socket API + server proxy?
  - BusyBox  http://www.busybox.net/
  - GCC

Some stuff I want to do in no particular order:

  - shell
  -- kb support for curses+nativeblue+hterm
  -- kb support for curses+nativeyellow
  -- Shells: tcsh? sh? bash? ssh? screen?
  -- launch process
  -- copy environment to sandbox file system
  - gcc.nexe
  - ddlt.nexe
  - Use Tundra build system
  - native-host server to provide file i/o via dropbox API. Use liboauth.
  -- Or maybe dropbox API to FileSystem API sync. 
     See: http://www.html5rocks.com/en/tutorials/file/filesystem/
  -- I wonder if a FUSE port and then a dropbox FUSE driver would be better?
  - native-host server to provide socket proxy for client builds. Use 
    libwebsockets.
  - Build some stuff from nacl_ports
  -- bzip2-1.0.6
  -- expat-2.0.1
  -- ffmpeg-0.5
  -- fftw-3.2.2
  -- fontconfig-2.7.3
  -- FreeImage-3.14.1
  -- freetype-2.1.10
  -- ImageMagick-6.5.4-10
  -- jpeg-6b
  -- jsoncpp-0.5.0
  -- libpng-1.2.40
  -- libtomcrypt-1.17
  -- libtommath-0.41
  -- lua-5.1.4
  -- openssl-1.0.0e
  -- tiff-3.9.1
  -- tinyxml
  -- x264-snapshot-20091023-2245
  - Separate lib/bin directories per target and configuration type
  - Include necessary native-host stuff from nacl_sdk
  - Note on naming. 
    e.g. see: http://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html
  - vim.nexe
  - Selectively redirect stdout to another host over socket
  -- Select new driver for /dev/stdout?
  - Auto-map function calls from javascript environment to nativeblue
  - POSIX IPC http://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltf/index.html
  - glut
  - remap gles functions to command queue