libk

libk is intended as a modernized replacement (not reimplementation) for libc.

manifesto

normally, all C binaries (and binaries from other languages, depending on the platform) use a combination of libraries to get things done: POSIX libraries (interfaces common to UNIX-like operating systems) and libc, the C standard library. unlike POSIX, libc is part of the C language -- it's a standardized interface to various critical parts of the operating system, things like IO, system clock access, random number generation, and more.

it's also a piece of shit.

libc is ancient, and it shows. it contains decades worth of cruft, masses of different interfaces with completely different design, horrible hacks to get around the fundamental shifts in basic computer architecture that have occurred over the past half-century, and vendor-specific extensions that make porting code a nightmare. using it is painful, tedious, error-prone, and unsafe. for various reasons, there are many different implementations of libc, but all of them have that same broken, bloated interface in common. as far as i can tell, there are been no serious attempts to create an actual alternative to libc - a new system interface that takes into account the decades of painful lessons we programmers have learned since the heydays of UNIX.

hence, libk.

libk aims to offer a better, safer, and most importantly, less unpleasant foundation for modern code in C or any other language. it also aims to be much smaller, simpler, and faster than glibc to build so that there's no arduous bootstrapping process for new architectures.

currently, the only dependency on libc in any form is arch/typesize.c, a small binary tool which uses libc IO routines to print type information it calculates; however, this could also be augmented to use POISX IO routines, or even potentially libk IO routines to remove any external dependency at all -- the work would be nontrivial, but fully feasible. further, the file it creates can also in extremis be created by hand. the final compiled libc binaries and headers do not depend on or reference libc in any way; typesize is only a makedepend.

goals

libk's goals are far-reaching, and suggestions are welcome. note however that libk is not intended to be a kitchen-sink library like libiberty. it's meant to do one thing, and to do it well: to provide an easy- and pleasant-to-use foundation for modern open source projects. below is a list of some of the project's major goals.

1. IO. libc's basic input/output mechanisms are dreadful, built at entirely the wrong level of abstraction. libk is intended to make many more primitives available to the user, and offer a sliding scale of abstraction so libk is suitable for a wide range of needs.
2. file manipulation. libc's file manipulation primitives are a relic of a bygone age and in dire need of upgrading.
3. terminal manipulation. libc has no provision for simple output formatting, a task that requires a combination of ANSI codes and in some cases pty manipulation with POSIX APIs, both of which are somewhat dark wizardry. this situation forces many innocent coders to drag in the entire unholy bulk of the aptly named library ncurses, much of whose code has been utterly obsolete for the last twenty years and whose API is one of the most singularly hateful ones in existence. libk therefore should offer a simple, straightforward way to do gracefully-degrading terminal sorcery.
4. memory management. the single memory management function malloc() provided by libc is absolutely pitiful. this is 2019. modern applications have much more exotic allocation needs, and a standard library should offer a range of allocators and management techniques, as well as abstract pointer objects so that pointers to objects of different allocation types (including static or stack allocation!) can be mixed freely and safely.
5. intrinsic reentrancy. because jesus christ, libc.
6. interprocess communication. libc offers no useful IPC abstractions over the paltry array of tools POSIX &co. give us to work with. we can do better.
7. tooling. libk is intended as more than just a library. it's also intended to work with some basic tooling to automate tasks that current binary tooling is inadequate for -- for instance, embedding binary data into a program binary. (see module kgraft)
8. modularity. libk is not part of the C specification and it isn't always going to be practical for developers to expect the entire library to be present on the end-user's computer. so libk is designed to be usable in many different ways -- as a traditional library, as a static library, in full form or with only components needed by the developer, to be distributed either on its own or as part of a binary.
9. compatibility. code that links against libk should be able to compile and run on any operating system. in the ideal case (Linux or FreeBSD) it will be able to do so without touching any other system libraries; for less ideal environments like Windows, libk will when necessary abstract over system libraries or libc itself.
10. sane error-handling. every time you type errno god murders a puppy.
11. ease of distribution. libk should enable to user to create completely static binaries, free of any local dependency and trivial to distribute.

dependencies

libk is designed to be as portable and dependency-free as possible. ideally, it will be possible to compile code against libk using nothing but libk itself.

compiling libk is also designed to be as easy as possible. it has only two external dependencies, the macro processor m4, needed for compile-time header generation, and optionally the commonmark compiler cmark, which is only needed if you wish to typeset the documentation in manpage, html, or pdf format. this process also requires groff, but groff is a standard part of most UNIX systems.

a different macro processor, gpp, was used in early versions of libk, however, it was so obscure and took so much overly fragile infrastructure to make it work that the cleaner syntax just wasn't worth it; i've since deleted the gpp infra and ported the macro files to m4.

naming conventions

one of the most frustrating things about libc is its complete and total lack of a naming convention. in C, every function and global is injected into a single global namespace, including macros. this means that every libc header you include scatters words all over that namespace, potentially clobbering your function with a macro!

libk is designed to fix this (in hindsight) glaring error.

however, a common problem with libraries is the proliferation of inordinately long and hard-to-type function names such as SuperWidget_Widget_Label_Font_Size_Set(). this may be tolerable in IDEs with robust auto-complete or when referencing a highly-specific, sparsely-used library; it is however completely intolerable in the case of a core library with heavily used functionality.

therefore, libk uses two slightly different naming conventions: the short convention, for core functions the user will call frequently, and the full convention, for less-commonly used functions. the inconvenience of remembering which is which will hopefully be outweighed by the keystrokes (and bytes) saved.

in the full convention, an identifier's name is prefixed with its module name followed by an underscore. thus, kgraft/list.c is invoked as kgraft_list().

in the short convention, identifiers are prefixed by the letter k followed by the module's "glyph" -- a one- or two-letter sequence that represents the module, usually the first one or two characters. therefore, kfile/open.c is invoked as kfopen.

which naming convention a module uses should be specified at the top of its documentation. if it uses the short convention, its glyph should be specified as well

in both naming conventions, the following rules apply:

1. the possible values of enumeration types are always preceded by the name of the enumeration type and an underscore. for instance, the enum ksalloc has a value named ksalloc_static. exception: an enum named <S>_kind, where <S> is a struct type, may simply use the prefix <S>_.
2. macros begin with the uppercase letter K -- e.g. Kmacro. macros that can be defined by the user to alter the behavior of the api should begin with KF if they are on/off flags, or KV otherwise.
3. capital letters are only used in macro prefixes.
4. low-level function names are prefixed with the API they call into. for example, the function that performs the POSIX syscall write is named kio_posix_fd_write. a wrapper around the Windows function CreateProcess() might be called kproc_win_createprocess.

atoms

libk uses the concept of "atoms" (small, regular strings of text) to standardize common references, such as operating systems or processor architectures.

operating systems

these atoms will be used to reference operating systems.

• Linux: lin
• Haiku: hai
• Android: and
• FreeBSD: fbsd
• NetBSD: nbsd
• OpenBSD: obsd
• Darwin/Mac OS X/iOS: dar
• MS-DOS: dos
• FreeDOS: fdos
• Windows: win
• Windows MinGW: mgw

file extensions

• C function implementations: *.c
• C module headers: *.h
• ancillary C headers: *.inc.h
• assembly code: *.s

arches

these atoms will be used to reference particular system architectures. these will mostly be used in the filenames of assembly code.

• Intel/AMD x86: x86
• ARM: arm (aarch64 is specified by os=arm bits=64)
• MIPS: mips
• Itanium: ia64 (no bits)
• PowerPC: ppc

localization

libk does not interface, respect, or wrap system locale APIs in any way. localization, for the most part, is the responsibility of the developer. this is necessary in order to prevent hidden state from accreting, which lets us make certain invariant guarantees about library behavior that can prevent highly confusing bugs or potentially even have security implications.

this is not to say that libk supports only one language, one calendar, and one culture. mechanisms will exist to produce localized output; however, they require the developer to explicitly pass localization flags and state. that is to say, they are stateless and opt-in only. the user changing an environment variable will never cause, e.g., decimal points to turn into commas unless the coder explicitly specified that behavior.

libk uses UTF8 exclusively. it has no concept of codepages or non-unicode charsets.

macros

libk will not in any circumstance use macros to encode magic numbers, instead using typedef'd enums. all libk macros begin with the uppercase letter K -- e.g. Kmacro. macros that can be defined by the user to alter the behavior of the api should begin with KF if they are on/off flags, or KV otherwise. macros should only be defined by the libk headers if the flag KFclean is not defined at the time of inclusion.

include guards take the form of the bare module name prefixed by KI. so to test if k/term.h has been included, you could write #ifdef KIterm.

languages

libk uses only five widely-used and standardized computer languages: C (*.c, *.h), yasm (*.s), awk (*.awk), commonmark (*.md), and bash (*.sh). further languages may not be introduced into the project without explicit advance approval of the maintainer herself.

other assemblers will probably be necessary for the more exotic targets, however.

repository structure

libk uses a strict directory structure for code, and deviations from this structure will not be tolerated without extremely good reason.

total segregation is maintained between source code, temporary files, and output objects. source is found in module directories (k*/). the destination for temporary files and output objects are retargetable via the environment variables gen= to=, but default to gen/ and out/, which are excluded from repo with fossil's ignore-glob settingapproval of the maintainer herself.

all libk code is dispersed into modules: kcore for internals, kio for I/O, kgraft for binary packing, etc. each module has a folder in the root directory. (libk does not have submodules.) inside each module's directory should be a header with the same name as the module (see naming conventions above).

each function should be kept in a separate file within its module's directory. the file's name should consist of the dot-separated fields [name, class, "c"] for C sources, or [name, class, arch, OS, bits, format, "s"] for assembly sources, where "name" is the name of the function without the module prefix and "class" is rt if the file is part of the libk runtime, or fn otherwise. this distinction is necessary because while the static library libk.a can include runtime objects, the shared library libk.so cannot. examples:

• a C file in the module kstr named kscomp would be named kstr/comp.fn.c
• a runtime assembly file called boot in the module kcore for x86-64 linux would be named kcore/boot.rt.x86.lin.64.s
• the 32-bit x86 haiku version of a function called kiowrite defined in assembly would be named kio/write.fn.x86.hai.32.s.

each module should have a header named the same thing as the module except without the k prefix. (e.g. the header for kio is kio/io.h) located in its folder. this is the header that the end-user will be importing, and should handle any user-defined flags to present the API the user has selected.

each module should contain a markdown file. this file's name should be the name of the parent directory suffixed with .md; for instance, kterm should contain the file kterm/kterm.md. this file should document the module as thoroughly as possible

each module may contain any number of files of the name *.exe.c. this files will be treated as tools by the build system and compiled as executables, rather than libraries. they should be compiled to $to/$module.$tool

the repository root and each module may also contain the directory misc. this directory may be used to store miscellaneous data such as ABI references, developer discussions, and roadmaps. if the misc directory is deleted, this must not affect the library or build system's function in any way - that is, nothing outside a misc folder may reference a misc folder or anything inside it, including documentation. the misc directory should be removed when its contents are no longer needed. in most cases, the repository wiki and forum should be used instead of the misc folder.

the folder arch in the root of the repository contains syscall tables and ABI implementations for various architectures.

build system

libk uses a very simple build system. the entire project is built with the build.sh script in the project root. build.sh need not be modified so long as all you're adding is functions, runtime stubs, headers, documentation files, or macro files in line with the standard naming convention. it will need to be modified to add or remove modules. it does not track dependencies, recompiling the entire project in one fell swoop. while in theory this could be slow and inefficient, in practice, it's not meaningfully slower on the average run than the old make-based system was.

the shell-based build system was chosen for a number of reason. firstly, libk originally used a GNU make-based build system, but this was unwieldy and unreliably - make simply does not have the necessary capabilities to build a project of this nature. the original build system relied heavily on recursion, which is impossible to use while preserving idempotency. multithreaded building was likewise impossible.

relying on make also had a number of subtler downsides. GNU make's capacities vary significantly across versions, and not all users will necessarily have access to the correct version in their repos. even a version as recent as make 4.1 couldn't build a project that build without issue in 4.2. given that libk is explicitly intended to be widely portable, this variance was alarming. make is also very indirect - it's a pain in the ass to tell it what to do sometimes, and it's well nigh on impossible to decipher the intricate way in which a sufficiently complex set of makefiles interlocks. this is both user- and developer-hostile. by contrast, a simple, linear build script with clearly delineated functions makes the build process much easier to understand, and therefore to tweak should a user need to modify some aspect of the process to compile libk on her system.

finally, make simply has a smaller install base than bash. with the build.sh build system, more people can successfully build libk with less effort.

any time you change build script, always specify precisely what changed in your commit log and what affects this might have on the surrounding code. be sure to comment any new code as thoroughly as you can. the goal is that anyone who is familiar with bash should be able to learn the build process simply by reading build.sh.

build process

to build libk, you must invoke build.sh with the proper parameters. at minimum you must set the following environment variables:

• os={atom} - an atom representing the operating system you are building libk for
• arch={atom} - an atom representing the processor architecture you are building libk for
• bits={atom} - if your processor has multiple variants with different word lengths (such as x86-32 vs. x86-64), specify the word length in this variable; otherwise, leave it unset.

further optional variables may be set to control the build process and determine what targets it produces.

• library=static {static|shared|both} - this variable controls whether the build process will produce libk.a, libk.so, or both.
• to=out {path} - an alternate path to store build artifacts in
• gen=gen {path} - an alternate path to store generated source in
• cc=<autodetect> {executable} - the compiler to compile C sources with
• m4=<autodetect> {executable} - the m4 binary to compile the macro sources with
• asm=<autodetect> {executable} - the assembler to assemble the assembly listings with. it must take Intel-syntax input and handle nasm-style macros. only yasm and nasm are likely to be viable.
• doc=yes {yes|no} - whether to typeset the documentation (very slow with all three formats set to "yes")
• doc_html=yes {yes|no} - enable or disable html output of the documentation
• doc_pdf=yes {yes|no} - enable or disable pdf output of the documentation
• doc_man=yes {yes|no} - enable or disable manpage output of the documentation
• verbose=quiet {silent|quiet|loud} - control level of verbosity. silent silences most output.

two other shell scripts complete the build system: * install.sh - installs compiled libraries, objects, documentation, and headers into the appropriate directories.

design principles

there are four overriding principles that guide the design of libk.

1. it should be easy to write code that uses it.
2. it should be easy to read code that uses it.
3. the simple, obvious way of using libk should produce the most optimal code.
4. code that uses libk should be idiomatic C.

for these reasons, the codebase follows a number of strict rules.

booleans are banned

there are a number of reasons for this.

the first is simply that the boolean type in C is a bit messy and libk headers are intended to import as few extra files as possible.

the second is that boolean-using code can be hard to read. consider a struct declaration of the form rule r = { 10, buf, true, false, true }: the meaning of this declaration is opaque unless you've memorized the structure's definition.

instead, libk uses enums liberally. so the above might be rewritten as e.g.:

rule r = { 10, buf,
	rule_kind_undialectical,
	rule_action_expropriate,
	rule_target_bourgeoisie
};

this makes code much more legible and has the added benefit of making the definitions easier to expand at a later date if new functionality is needed without breaking the API or ABI.

error handling

every module has a cond type (e.g. kscond for kstr or kconf_cond for kconf). this is an enumeration type that represents every possible error a module can return, and every cond type obeys a number of invariants (in addition to the usual namespacing rules:

1. every member of a cond type has an integer value that is unique among all modules' cond types. for instance, this means that kscond_ok has an integer value that is not equal to kmcond_ok.
2. every member of a cond type has a member *cond_ok which represents total success. this member's integer value is always an exact multiple of the "module offset", the number of condition values allocated to each module (currently 0x7F).
3. the kokay() function, defined in kcore, when called on a member of a cond type will return true if that member represents total success and false otherwise.
4. every cond type has a value *cond_fail, for instance kconf_cond_fail. if a condition is greater than or equal to its module's fail value, it represents a total failure. if it is lesser than the fail value, it represents either success, partial success, or some other condition that does not equate to total failure.
5. if a cond value is greater than its module's *cond_ok value but less than the *cond_fail value, it represents a condition that is neither total success nor total failure; for instance, a partial write, or an attempt to free memory that does not need to be freed.

the reason each error value is unique is that this allows us to map every single condition code unambiguously to an error message. a forthcoming kcore function will do exactly that and produce an error string that can be displayed to a user or for debugging purposes. another useful property is that each integer value has at most only one possible meaning in the context of error codes, allowing for centralized error handling - a kscond and a kmcond can both be turned into an int without loss of information, and e.g. kscond_fail != kmcond_fail != kconf_cond_fail.

this is particularly useful as C does not have exceptions, and thus the only viable error-handling mechanisms are early-exit and early-return; any cond value can be propagated up the function stack without losing its unique meaning.

the value of each condition code is determined at compile time.

authors

• lexi velartrill hale lexi@hale.su
• lachs0r
• glowpelt

caveats

the main coder, lexi hale, is first and foremost a writer, not a coder. this is a side-project of hers and will remain so unless it picks up a significant amount of attention.

while MRs adding support for Windows, OS X, and other operating systems will be gratefully accepted, the maintainer is a Linux and FreeBSD developer, will not be writing such support infrastructure herself, and has limited ability even to vet code for those platforms.

legal

libk is released under the terms of the EUPL. at present, contributors do not immediately relinquish ownership of the code they contribute, but agree to release it under the same terms as the overall project license. however, contributors agree to relinquish ownership of their code to the official libk development directorate if and when that organization is formally brought into existence by the project founders. the above notwithstanding, under no circumstances may contributors be deprived of credit for contributions which remain in the libk trunk, whether or not they are the legal owners of that code.

the EUPL may seem like an inappropriately restrictive license for a project with such grandiose ambitions. it is an ideological choice. we have selected it because libk is intended very specifically as a contribution to the free software community, a community that i hope will continue to grow at the expense of closed-source ecosystems. i have no interest in enabling people or corporations to profit from keeping secrets, especially not with my own free labor (or anyone else's, for that matter).

if you disagree with this philosophy, you are welcome to continue using libc.

what does the k stand for?

nothing. it was chosen in reference to libc - the letter C was part of the original roman alphabet, while K was added later by analogy to the Greek kappa ‹κ›. in my native language, the older letter ‹c› can make a number of different sounds based on context, including [k] and [s], while ‹k› is fairly consistently used for the sound [k]. and for orthographical reasons, [k] is often represented by the digraph ‹ck› - that is, a C followed by a K. hopefully the analogies are obvious.

this project has nothing to do with KDE.