libk  Check-in [338044baaa]

; vim: ft=nasm
%define lin.call.exit 1
%define lin.call.fork 2
%define lin.call.read 3
%define lin.call.write 4
%define lin.call.open 5
%define lin.call.close 6

%define lin.call.chdir 12

%define lin.reg.n 6
%define lin.reg.0 eax
%define lin.reg.1 ebx
%define lin.reg.2 ecx
%define lin.reg.3 edx
%define lin.reg.4 esi
%define lin.reg.5 edi

%define lin.call int 0x80 ; sysenter is allegedly the
  ; politically correct option but it does not actually
  ; appear to work without a whole lot of extra bullshit

; todo: learn vdsos

; vim: ft=nasm
%define lin.call.exit 60
%define lin.call.fork 57
%define lin.call.read 0
%define lin.call.write 1
%define lin.call.open 2
%define lin.call.close 3

%define lin.call.chdir 80

%define lin.reg.n 6
%define lin.reg.0 rax
%define lin.reg.1 rdi
%define lin.reg.2 rsi
%define lin.reg.3 rdx
%define lin.reg.4 r10
%define lin.reg.6 r8
%define lin.reg.7 r9

%define lin.c.0 rdi
%define lin.c.1 rsi
%define lin.c.2 rdx
%define lin.c.3 rcx
%define lin.c.4 r8
%define lin.c.5 r9

%define lin.call syscall
; todo: learn vdsos

bits 32
global kio_posix_fd_write

%include "../arch/x86.lin.32.inc"
; vim: ft=nasm

kio_posix_fd_write:
	mov lin.reg.0, lin.call.write
	mov lin.reg.1, [esp + 4] ; holy god but this took the most
	mov lin.reg.2, [esp + 8] ; stupidly long time to fucking
	mov lin.reg.3, [esp + 12]; figure out
	lin.call
	ret

bits 64
global kio_posix_fd_write

%include "../arch/x86.lin.64.inc"
; vim: ft=nasm

kio_posix_fd_write:
	mov lin.reg.0, lin.call.write
	; mov lin.reg.1, lin.c.0 - nop
	; mov lin.reg.2, lin.c.1 - nop
	; mov lin.reg.3, lin.c.2 - nop
	lin.call
	ret

kio: posix

posix: ${OUT}/kio_posix_fd_write.${TARGET}.o

${OUT}/%.lin.32.o: %.lin.32.s
	yasm -felf32 $< -o $@

${OUT}/%.lin.64.o: %.lin.64.s
	yasm -felf64 $< -o $@

# kmem

**kmem** is a libk module that contains various functions for memory allocation and deallocation. it uses the **short** naming convention with the glyph `m`.

## module functions

**kmem** supplies two module-level functions, used to interact with the `kmptr` container type.

 * `kmfree(kmptr) → void` - free, downref, or ignore the pasted object as appropriate
 * `kmshred(kmptr) → void` - free, downref, or ignore the pasted object as appropriate. if deallocating, zero its contents
 * `kmstat(void*) → kmptr` - convenience function to wrap a pointer to a non-managed object in a `kmptr` struct, so it can be passed to functions that accept arbitrary objects. `kmptr p = kmstat(raw)` is equivalent to `kmptr p = { kmptr_kind_static, raw, NULL }`.
 * `kmtaint(&kmptr) → void` - "taints" a `kmptr` object by setting it to be shredded when freed. this may be desirable if the object pointed to contains privileged information.

## types

kmem defines the following types:
 
 * `struct kmptr` - abstract pointer object
	* `enum kmptr_kind`
 * `struct kmcell` - abstract memory cell
	* `enum kmcell_kind`
 * `struct kmref` - a reference-counted cell
 * `struct kmnode` - a node in an allocation tree
 * `struct kmpool` - a memory pool

`kmptr` and `kmcell` are both very similar. the difference is that a kmptr points to a region in memory and can be passed around freely. a `kmcell` is the actual in-memory representation of an allocation cell. a `kmcell` cannot be usefully instantiated; rather, it is downcast from an actual cell type (e.g. `kmnode n; kmcell* s = (kmcell*)(&n)`)

### kmptr

kmem functions can operate on both raw pointers and the `kmptr` struct type. `kmptr` is a generic struct that can contain any kind of pointer. this is useful if you wish to allocate different objects in different manners, but pass them on into a single interface.

memory pointed at by `kmptr` pointers can be freed either with the usual specialized function, or by passing the `kmptr` structure itself to the generic function `kmfree`, which will handle it appropriately, even if it's a pointer to a garbage-collected object or to a static region of memory.

a `kmptr` has the following layout:

 * `kmptr_kind kind` - codes the type of pointer
 * `kmshred shred` - an enum. if `kmshred_yes`, the value will be zeroed or otherwise made unreadable on free. if no, `kmfree` will consult `src` for shred policy if it is not NULL.
 * `void* ref` - the raw pointer enclosed by `cell`
 * `kmcell* cell` - a pointer to an object enclosure, typically either a memory pool or a referencing-counting object. NULL if not needed.
 
the convenience function `kmstat(void*) → kmptr` wraps a pointer to a static object in a `kmptr` struct.

#### kmptr_kind

`kmptr_kind` is an enum with one of the following values.

 * `kmptr_kind_none` - not a valid pointer
 * `kmptr_kind_static` - points to a static region of space. `kmptr` instances with this kind will be ignored by `kmfree`.
 * `kmptr_kind_heap` - a traditional heap pointer.
 * `kmptr_kind_pool` - points to a region stored in a memory pool.
 * `kmptr_kind_ref` - points to a reference-counted object.
 * `kmptr_kind_node` - points to a reference-counted object.

### kmcell

`kmcell` is a stub struct used to disambiguate between source types.a "source" is an object that can hold an allocated object, such as the heap, a memory pool, a fixed-length array on stack, or a fixed-length global array. all values produced by a kmem allocation function point to within a `kmcell`.

 * `kmptr_kind kind` - kind of cell
 * `size_t sz` - kind of cell (data plus all fields)
 * `kmshred shred` - shredding flag

### kmref

`kmref` is a struct that constitutes the in-memory representation of a reference-counted cell.

 * `kmcell_kind kind = ref` - kind of cell
 * `size_t sz` - size of cell (data plus all fields)
 * `kmshred shred` - shredding flag
 * `size_t refs` - number of active references 
 * `kmcell* src` - source, if any
 * `char data[]` - content of cell

### kmnode

`kmnode` is a struct that constitutes the in-memory representation of a tree node.

 * `kmcell_kind kind = node` - kind of cell
 * `size_t sz` - size of cell (data plus all fields)
 * `kmshred shred` - shredding flag
 * `kmnode* parent` - parent node
 * `kmnode* child` - first child node
 * `kmnode* lastchild` - last child node
 * `kmnode* prev` - previous sibling, NULL if first
 * `kmnode* next` - next sibling, NULL if last
 * `char data[]` - content of cell

### kmpool

 * `kmcell_kind kind = pool` - indicates the kind of source
 * `size_t sz` - size of cell (data plus all fields)
 * `kmshred shred` - shredding flag
 * `size_t cellsz` - size of individual pool cells
 * `kmpoolcell* top` - pointer to most recently allocated pool cell
 * `kmpoolcell* bottom` - pointer to most recently freed pool cell
 * `kmpoolcell data[]` - content of cell

### kmpoolcell

 * `kmpoolcell* last` - pointer to last element allocated before this one
 * `char data[]` - pool data

### kmshred

`kmshred` is an enum used to indicate whether an object should be "shredded" (written over) in memory when it's deleted. this is a useful means to ensure that privileged information is not accidentally left in memory after use. if the shredding mechanism is not useful, compile libk with the flag `KFmem_noshred` to exclude its functions and fields.

 * `kmshred_yes` - marks an object to shred on free
 * `kmshred_no` - marks an object not to shred on free

## naming convention

kmem function names are based on the **method** of allocation and the **action** being performed. methods are listed in the section below. kmem defines a number of standardized actions, though not every method uses every action. the character listed in brackets is suffixed to the name of the method to produce a function name: for instance, `kmheapa` will allocate memory on the heap, while `kmrefd` will decrement the reference count of its argument.

 * initialize [i]  - initializes a memory store on the heap
 * initialize fixed [if]  - initialize a memory store on the stack or in a fixed-size global
 * allocate [a]  - return a raw pointer to a new region of memory of the given size, ready to write, or NULL if not possible. contents of that region undefined. takes parameter (size_t sz).
 * allocate pointer object [ao]  - like *allocate*, but returns a `kmptr` instead of a raw `void*`.
 * zero [z]  - allocate a new region of memory and zero it before returning it for writing. 
 * zero pointer object [zo]  - like *zero*, but returns a `kmptr` instead of a raw `void*`.
 * free [f]  - free a section of memory, either decrementing a reference count or returning it to whatever pool it came from.
 * shred [s]  - destroy whatever was in the segment of memory, then return it to the pool it came from.
 * destroy [x]  - tears down a memory store
 * upref [u] - increments a reference counter

## methods

kmem currently supports the following methods of memory management, along with which methods are defined for it. (note that `a` implies `z` and `f` implies `s`). a method may be excluded from a libk binary by defining the flag `KFmem_no[name]`, e.g. `KFmem_noheap`

 * `heap` [af] - standard heap allocation
   * `kmheapa(size_t) → void*` - allocate
   * `kmheapz(size_t) → void*` - zero-allocate
   * `kmheapao(size_t) → kmptr` - allocate pointer object
   * `kmheapzo(size_t) → kmptr` - zero-allocate pointer object
   * `kmheapf(void*) → void` - free
   * `kmheaps(void*) → void` - shred
 * `ref` [afu] - reference-counted heap object
   * `kmrefa(kmcell*, size_t) → void*` - allocate
   * `kmrefz(kmcell*, size_t) → void*` - zero-allocate
   * `kmrefao(kmcell*, size_t) → void*` - allocate pointer object
   * `kmrefzo(kmcell*, size_t) → void*` - zero-allocate pointer object
   * `kmreff(void*) → void` - downref; free if last ref
   * `kmrefs(void*) → void` - downref and mark for shred on last ref
 * `pool` [ixaf] - memory pool
   * `kmpooli(kmcell*, size_t sz, size_t n) → kmpool*` - initialize a fixed memory pool (a pool of `n` cells of length `sz`)
   * `kmpoolx(kmpool*) → void` - tear down a memory pool
   * `kmpoola(kmpool*) → void*` - allocate from pool
   * `kmpoolz(kmpool*, size_t) → void*` - zero-allocate from pool
   * `kmpoolao(kmpool*, size_t) → void*` - allocate pointer object
   * `kmpoolzo(kmpool*, size_t) → void*` - zero-allocate pointer object
   * `kmpoolf(void*) → void` - downref; free if last ref
   * `kmpools(void*) → void` - downref and mark for shred on last ref
 * `tree` [af] - uses a node-child strategy. when a node is freed, all of its children are automatically freed as well.
   * `kmtreea(kmcell* src, void* parent, size_t) → void*` - create a tree node. if `parent` is NULL, the node will the top of a new tree. if src is null, allocate on-heap.
   * `kmtreez(kmcell* src, void* parent, size_t) → void*` - like `kmtreea` but zeroed 
   * `kmtreeao(kmcell* src, void* parent, size_t) → kmptr` - like `kmtreea` but returns a `kmptr` 
   * `kmtreezo(kmcell* src, void* parent, size_t) → kmptr` - like `kmtreez` but returns a `kmptr` 
   * `kmtreef(void*) → kmptr` - frees a node and all its children

# libk

libk is intended as a modernized replacement *(not* reimplementation) for libc.

## manifesto

normally, all C binaries (and binaries from other languages, depending on the platform) use a combination of libraries to get things done: POSIX libraries (interfaces common to UNIX-like operating systems) and libc, the C standard library. unlike POSIX, libc is part of the C language -- it's a standardized interface to various critical parts of the operating system, things like IO, system clock access, random number generation, and more.

it's also a piece of shit.

libc is ancient, and it shows. it contains decades worth of cruft, masses of different interfaces with completely different design, horrible hacks to get around the fundamental shifts in basic computer architecture that have occurred over the past half-century, and vendor-specific extensions that make porting code a nightmare. using it is painful, tedious, error-prone, and unsafe. for various reasons, there are many different implementations of libc, but all of them have that same broken, bloated interface in common. as far as i can tell, there are been no serious attempts to create an actual *alternative* to libc - a new system interface that takes into account the decades of painful lessons we programmers have learned since the heydays of UNIX.

hence, libk.

libk aims to offer a better, safer, and most importantly, less unpleasant foundation for modern code in C or any other language.

## goals

libk's goals are far-reaching, and suggestions are welcome. note however that libk is *not* intended to be a kitchen-sink library like libiberty. it's meant to do one thing, and to it well: to provide an easy- and pleasant-to-use foundation for modern open source projects. below is a list of some of the project's major goals.

 1. **IO.** libc's basic input/output mechanisms are dreadful, built at entirely the wrong level of abstraction. libk is intended to make many more primitives available to the user, and offer a sliding scale of abstraction so libk is suitable for a wide range of needs.
 2. **file manipulation.** libc's file manipulation primitives are a relic of a bygone age and in dire need of upgrading.
 3. **terminal manipulation.** libc has no provision for simple output formatting, a task that requires a combination of ANSI codes and in some cases pty manipulation with POSIX APIs, both of which are somewhat dark wizardry. this situation forces many innocent coders to drag in the entire unholy bulk of the aptly named library `ncurses`, much of whose code has been utterly obsolete for the last twenty years and whose API is one of the most singularly hateful ones in existence. libk therefore should offer a simple, straightforward way to do gracefully-degrading terminal sorcery.
 0. **tooling.** libk is intended as more than just a library. it's also intended to work with some basic tooling to automate tasks that current binary tooling is inadequate for -- for instance, embedding binary data into a program binary. (see module [kgraft](kgraft))
 0. **modularity.** libk is not part of the C specification and it isn't always going to be practical for developers to expect the entire library to be present on the end-user's computer. so libk is designed to be usable in many different ways -- as a traditional lirbary, as a static library, in full form or with only components needed by the developer, to be distributed either on its own or as part of an binary.
 0. **compatibility.** code that links against libk should be able to compile and run on any operating system. in the ideal case (Linux or FreeBSD) it will be able to do so without touching any other system libraries; for less ideal environments like Windows, libk will when necessary abstract over system libraries or libc itself.

## naming conventions

one of the most frustrating things about libc is its complete and total *lack* of a naming convention. in C, every function and global is injected into a single global namespace, including macros. this means that every libc header you include scatters words all over that namespace, potentially clobbering your function with a macro!

libk is designed to fix this (in hindsight) glaring error.

however, a common problem with libraries is the proliferation of inordinately long and hard-to-type function names such as `SuperWidget_Widget_Label_Font_Set()`. this may be tolerable in IDEs with robust auto-complete or when referencing a highly-specific, sparsely-used library; it is however completely intolerable in the case of a core library with heavily used functionality.

therefore, libk uses two slightly different naming conventions: the **short** convention, for core functions the user will call frequently, and the **full** convention, for less-commonly used functions. the inconvenience of remembering which is which will hopefully be outweighed by the keystrokes (and bytes) saved.

in the **full** convention, a function's name is prefixed with its module name followed by an underscore. thus, `kfile/open.c` will be invoked as `kfile_open()`.

in the **short** convention, the function name is prefixed by the letter `k` followed by the module's "glyph" -- a one- or two-letter sequence that represents the module, usually the first one or two characters. therefore, `kio/write.c` is invoked as `kiowrite`.

which naming convention a module uses should be specified at the top of its documentation. if it uses the short convention, its glyph should be specified as well

### atoms

libk uses the concept of "atoms" (small, regular strings of text) to standardize common references, such as operating systems or processor architectures.

#### operating systems

these atoms will be used to reference operating systems.

 * Linux: `lin`
 * Haiku: `haiku`
 * Android: `android`
 * FreeBSD: `fbsd`
 * NetBSD: `nbsd`
 * OpenBSD: `obsd`
 * Darwin/Mac OS X: `dar`
 * MS-DOS: `dos`
 * FreeDOS: `fdos`
 * Windows: `win`

#### file extensions

 * C function implementations: `*.c`
 * C module headers: `*.h`
 * ancillary C headers: `*.inc.h`
 * assembly code: `*.s`

#### arches

these atoms will be used to reference particular system architectures. these will mostly be used in the filenames of assembly code.

## macros

libk will not in any circumstance use macros to encode magic numbers, instead using typedef'd enums. all libk macros begin with the uppercase letter `K` -- e.g. `Kmacro`. macros that can be defined by the user to alter the behavior of the api should begin with `KF` if they are on/off flags, or `KV` otherwise. **macros should only be defined if the flag `KFclean` is *not* defined.**

## languages

libk uses only three languages: C (\*.c, \*.h), yasm (\*.s), and make (makefile).

other assemblers will probably be necessary for the more exotic targets, however.

## repository structure

libk uses a rigorously directory structure for code, and deviations from this structure will not be tolerated without extremely good reason.

all libk code is dispersed into modules: `kcore` for internals, `kio` for I/O, `kgraft` for binary packing, etc. each module has a folder in the root directory. (libk does not have submodules.) inside each module's directory should be a header with the same name as the module and a folder for each operating system (see **naming conventions** above).

each function should be kept in a separate file within its module's directory. when OS or architecture-specific code is needed, the file's name should be a list of one or more of the fields [arch, OS, bits, format] separated by a `.` -- for instance, the 32-bit x86 haiku version of a function called `write` defined in assembly would be named `write.haiku.x86.32.s`. however, if a function has an extraordinarily large number of versions, they may instead be stored in a folder with the same name as the function.

each module should have a header named the same thing as the module except without the `k` prefix. (e.g. the header for `kio` is `kio/io.h`) located in its folder. this is the header that the end-user will be importing, and should handle any user-defined flags to present the API the user has selected.

each module directory should contain a makefile that can build that module. see **makefiles** below. all makefiles should be named `makefile` (**not** `Makefile`).

each module should contain a markdown file. this file's name should be the name of the parent directory suffixed with `.md`; for instance, `kterm` should contain the file `kterm/kterm.md`. this file should document the module as thoroughly as possible 

the repository root and each module may also contain the directory `misc`. this directory may be used to store miscellaneous data such as ABI references, developer discussions, and roadmaps. if the `misc` directory is deleted, this must not affect the library or build system's function in any way - that is, nothing outside a `misc` folder may reference a `misc` folder or anything inside it, including documentation. the `misc` directory should be removed when its contents are no longer needed. in most cases, the repository wiki and forum should be used instead of the `misc` folder.

the folder `arch` in the root of the repository contains syscall tables and ABI implementations for various architectures.

## makefiles

libk uses `make` as its build system. makefiles should be handwritten. there will be one global makefile in the root of the repository, and one makefile for each module.

each rule should be prefixed with ${OUT}, to allow retargeting of the build-dir with the OUT environment variable. this is particularly important since the makefiles chain.

the rest is TBD.

## build process

libk has a number of targets. all files generated by a `make` invocation will be stored in the folder "out" at the root of the repository. this directory may be deleted entirely to clean the repository.

**defs** will create the directory `out/k/` and populate it with module header files. the `k/` directory shall be suitable to copy to `/usr/include` or similar. these header files will copied by building the `defs` target of each module's makefile.

**libk.so** will build the dynamically linked form of libk, according to the build variables set

**libk.a** will build the statically linked form of libk, according to the build variables set

**tool** will build the executables used for modules such as `kgraft`.

there is no **clean** target. to clean the repository, simply delete the directory `out/`.

## authors

so far, this is a one-woman show. contributions are welcome however.

 * lexi hale <lexi@hale.su>

## caveats

the main coder, lexi hale, is first and foremost a writer, not a coder. this is a side-project of hers and will remain so unless it picks up a significant amount of attention.

while PRs adding support for Windows, OS X, and other operating systems will be gratefully accepted, the maintainer is a Linux and FreeBSD developer, will not be writing such support infrastructure herself, and has limited ability to vet code for those platforms.

## license

libk is released under the terms of the [GNU AGPLv3](LICENSE). contributors do not relinquish ownership of the code they contribute, but agree to release it under the same terms as the overall project license.

the AGPL may seem like an inappropriately restrictive license for a project with such grandiose ambitions. it is an ideological choice. i selected it because libk is intended very specifically as a contribution to the *free software* community, a community that i hope will continue to grow at the expense of closed-source ecosystems. i have no existence in enabling people or corporations to profit from keeping secrets, especially not with my own free labor (or anyone else's, for that matter).

if you disagree with this philosophy, you are welcome to continue using libc.

## what does the k stand for?

nothing. it was chosen in reference to libc - the letter C was part of the original roman alphabet, while K was added later by analogy to the Greek kappa ‹κ›. in my native language, the older letter ‹c› can make a number of different sounds based on context, including [k] and [s], while ‹k› is fairly consistently used for the sound [k]. hopefully the analogy is obvious.

this project has nothing to do with KDE.

export OUT=$(PWD)/out
export TARGET=x86.lin.64

all: kio

$(OUT)/kio.o:
	cd kio && make kio