Release notes
QSOE 0.4
Sound — on every board, on both kernels, and clean while the rest of the machine is busy.
0.4 is the audio release. QSOE plays sound over HDMI from the Unmatched's GK208, over DisplayPort on the K3, and over HDMI on the VisionFive 2, on QSOE/N and on QSOE/L alike, with one player and one engine under four drivers. A dropped frame is defined precisely in the source, and the counters report what happened rather than what was hoped. Sound is in this release because it is the instrument for everything after it: a late buffer is heard long before it would be noticed any other way.
The instrument earned its place before the release was out. Playback on QSOE/L tore while the test suite ran beside it, and nothing in the player or the driver was slow: the audio data, and the driver's wake-up every period, went through taskman — the one process every other process waits on. Both now bypass it, and the same song beside the same suite is clean. What sound found is described below, because it is the kind of thing this release exists to find.
Around it, 0.4 makes QSOE a system that is installed rather than unpacked: the base
system is a package, a package manager adds to it, a C compiler installed that way has
built and run its first program on QSOE's own filesystem, and update-os
installs new kernels from the running system. One boot archive boots both kernels; QSOE/L
boots by Multiboot3 on every board, the K3 included, with no elfloader. The C library is one
binary for both kernels, over a kernel library whose interface is written down. And the
shell runs jobs in the background, without a fork().
The component set
| Component | What it is | 0.3 | 0.4 |
|---|---|---|---|
nq |
QSOE/N — the Skimmer microkernel and its taskman | 0.30 | 0.36 |
lq |
QSOE/L — the seL4 taskman, its seam and its kernel patches | 0.26 | 0.32 |
libc |
the C library, libqsoe and crt0 |
0.19 | 0.25 |
quser |
the shared userspace: shell, drivers, servers, utilities | 0.18 | 0.24 |
qupkg |
the package manager (separate project, shipped in the base system) | — | 0.2 |
mr-bml |
the bootloader (separate project, shipped alongside) | 1.0 | 1.0 |
The umbrella records the tested set in
component.list; make prepare checks out exactly these. The
firmware the boards run is HFI BIOS 1.4.2.
On this page
Highlights
Fourteen things that make 0.4 a different system to listen to, and to keep, than 0.3.
Sound on every board, on both kernels. deva-hda drives an
HD Audio controller — the GK208's HDMI function on the Unmatched, or QEMU's —
deva-k3dp the K3's DisplayPort and deva-jh7110hdmi the
VisionFive 2's HDMI, and deva-pwmdac is written for the VisionFive 2's
3.5 mm jack. Each is its hardware over one engine, libaudio, behind a
/dev/snd/pcmC0D0p node:
frames arrive by write(), everything else is a devcontrol()
command. The drivers are the same binaries on QSOE/N and QSOE/L.
A dropped frame, defined. A frame the hardware played from a period the client had not filled — the definition is in the engine's header, and every counter is measured against it: at each period's end, less than a period waiting counts the whole period as dropped and one underrun. A period the interrupt thread slept through is judged on its own and counted as a late wake; the silent tail of a drain is not a drop. The engine is two threads and one ring with no lock — the dispatcher owns the write cursor, the interrupt thread the hardware's — and the interrupt thread runs above the disk's priority. It also learned to count honestly after NVIDIA's controller raised its interrupt a few frames before the period boundary and the bookkeeping fell one period behind.
qplay. WAV and Ogg Vorbis, every operand a track, so
qplay * plays a folder. When a device lacks a file's rate — the
VisionFive 2 takes only 48 kHz, most music is 44.1 — it converts through a
polyphase windowed-sinc filter with an exact ratio. With -v it prints what the
driver counted: frames written, played and dropped, interrupts, late wakes, and a histogram
of the interrupt thread's lateness.
taskman off the audio path. On QSOE/L a player's writes are larger than seL4's IPC buffer and went through taskman to be copied; and a driver's interrupt thread woke its own dispatcher, 47 times a second, with a pulse that went through taskman too. A spawn off the NVMe holds taskman for tens of milliseconds, and during it the song tore — 407 underruns in one song beside the suite on the VisionFive 2. Now a player attaches a window at setup and its writes go straight to the driver, and a process's pulses to itself go into a lock-free queue in the process and a signal, never a message. taskman's request rate during playback fell from about fifty a second to its background level, and the song beside the suite is clean on both kernels.
Windows given back while the server lives. A window — the pages
taskman maps once into both ends of a connection so that large messages need no third
party — used to keep its server side until the server exited, so a long-lived audio
driver leaked one per player. Each window record now carries a state that the server's
library and taskman move only by compare-and-swap: taskman takes the server's side back
only when it is idle, and never asks the server. Testing it by killing players one after
another found that the resource-server framework dropped a cancelled client's call
unanswered, holding its window for good; it answers EINTR now. Twenty-one kills
and twenty plays later, the driver held nothing it should not.
The base system is a package. The image is an installed
qsoe-base: /usr/bin, /usr/sbin, /usr/lib
and /usr/share are links into its directory in the package store, and init
selects the qsoe-base that matches the boot archive it booted. The test suite,
the network test tools, the GPIO and I2C drivers and the audio samples are packages beside it.
/usr is 256 MiB, and a Linux FUSE driver, built on the same filesystem
library as fs-qrv, reads and writes it on a development machine.
qupkg. A package manager in FreePascal, ported from the
package manager of Serge Vakulenko's Braam system. The repository index is signed with
Ed25519 by keys an anchor shipped with the system vouches for, and names every package by
its SHA-256; packages unpack once into an immutable store; what is installed is a
generation of links, switched in with one rename, so a power cut
leaves the old set or the new one and never half of each. A repository is any directory
reachable by path — including one on another station over QSP.
QSOE compiles C. qupkg install qjmcc brings a C compiler that
runs on QSOE, with its assembler, the linker and the C library's SDK as dependencies, and
qjmcc -o hello hello.c builds a program that runs — preprocessed,
compiled, assembled and linked on QSOE's own filesystem. It is the first step of the 1.0
self-hosting criterion, taken through the package system rather than around it.
One C library for both kernels. The QNX-style native API —
channels, messages, pulses, Sync*, threads — moved out of
libc.so into libqsoe.so, built once per kernel and loaded by each
taskman under the one name. libc.so above it is now a single binary for QSOE/N
and QSOE/L, and libqsoe.so calls nothing in it. What the two export is written
down, one line per symbol in abi/qsoe-abi0.exports; the build fails on a removed
or unlisted export, packages depend on qsoe-abi0, and a call one kernel cannot
make yet is a stub that announces itself rather than a missing name. There is no weak
linkage anywhere: a program that needs a name nothing defines is refused at spawn with the
name, instead of running into a call at address zero.
One boot archive; QSOE/L by Multiboot3, on the K3 too. The same modpkg.cpio
boots both kernels: it carries both taskmen and both libqsoe variants, and each
kernel takes its own. QSOE/L no longer nests the elfloader, seL4, taskman and the userland
in one image: seL4 is patched to boot from mr-bml's multiboot3 command, a stub
places taskman and hands it the archive as a module and a record of the boot, and the kernel
is a file beside the archive like Skimmer's. The seL4 changes are a patch series now, one
change per patch. taskman.elf went from 1.6 MB to 516 KB. And the K3,
QSOE/N only in 0.3, runs QSOE/L as well, on k3sel4, an overlay of seL4 16.0.0
for the SpacemiT SoC booted the same way — so every board runs both kernels.
QSOE installs its own kernel. update-os fetches a boot set
— the archive and each board's kernels — from a development station over QSP into
/usr/boot, all or nothing, keeping the previous set one step back. The boot
loader reads it straight from the system volume, found by its label. The first QSOE/L boot of
the K3 from files QSOE had deployed itself ran the whole suite in eleven seconds.
Jobs in the background, without fork(). cmd &
spawns the program itself, or a second shell for shell code; jobs,
fg, wait, kill %n and $! work. A
background job reads /dev/null, and a finished one is reported at the next
prompt.
A thread can be ended wherever it is. On Skimmer, ThreadDestroy
reaches a thread spinning in user mode on another hart: the request is a bit in the one word
another hart may write, an IPI knocks the target into the kernel, and the target ends itself
on its way back out. Every park is a cancellation point that can be broken — message
queues, nanosleep, the sync objects, InterruptWait,
ThreadJoin — so kill -9 of a runaway program means what it
says.
Hundreds of suite runs in one boot. Skimmer ran 270 passes of the suite on the K3 in one boot, after three races — each one in a few hundred process ends — were found and fixed, and every write to a thread's flags now panics, naming the line, if it comes from the wrong hart. QSOE/L ran 104, after two walls: a memory pool nobody counted, drained 72 KiB per process, and a fault flag at bit 16 that took every request on a connection past the 65,536th for a fault. A server is now told, by a disconnect pulse, of every connection a killed process left open.
The full change list
Everything that changed since 0.3, by area. Open the parts you want — or open them all and read it as one document.
Sound
- The contract (
<qsoe/audio.h>): nodes/dev/snd/pcmC<card>D<dev>p; frames bywrite();INFO,SETUP,START,STOP,DRAIN,STATUSandCLEAR_STATSbydevcontrol(). S16_LE; no resampling or mixing in a driver. A device says how large a ring it can hold. libaudio: the PCM engine every driver links — the lock-free ring, the period judgment, the dropped-frame accounting, and a period wake that is one pulse to the server's own channel.deva-hda: HD Audio over PCI, stereo; MSI only. On the Unmatched it maps only the GPU register pages it touches — the whole BAR was refused once the console had carved past it.deva-k3dp: the K3's DisplayPort through its I2S and audio DMA, fixed at 48 kHz. Its descriptor ring lives in an SRAM that is not cache-coherent and is written back explicitly — the difference between noise and music.deva-jh7110hdmianddeva-pwmdacon the VisionFive 2, over the DesignWare AXI DMA; the two share the SoC's one DMA interrupt, so one runs at a time, and HDMI is the one started at boot. Whether a link is HDMI or DVI is the firmware's decision, from the sink's EDID: the driver refuses a DVI link rather than change the picture.qplay: WAV and Ogg Vorbis (stb_vorbis; musl'slibm), tracks and folders,--tone, the rate converter with-Rto force or forbid it, real-time priority with-P, and the counters with-v.- Under QEMU
AUDIO=1 ./emu.shattaches an HD Audio controller and records what the guest plays into a WAV file on the host, whichwavcheck.pyjudges; the suite's[audio]group runs withsuite --audio.
Kernels and IPC
Skimmer
- Thread cancellation reaches a thread that is not listening (above):
td_mpflagsas the one cross-hart word,lwkt_unpark(), the victim ending itself in the trap tail;MsgSendvncandSyncMutexLockbreak only for termination, as their contract says.ThreadCancelwakes a parked receiver withEINTRat once. - Three races in thread teardown fixed: an exit that marked itself a
zombie before dropping its address space, two sweeps claiming one orphan, and a lost
read-modify-write on
td_flagsbetweenThreadJoinand the zombie's hart.td_flagsbelongs to its hart, and a write from another one panics with the file and line. hartwatch: each hart watches the next one's tick count, and a hart silent for two seconds is named once, with its counters, its current thread and its trace ring.- A thread's CPU time is kept, banked once per switch with interrupts
already off;
ps -Handsysinfoprint it. It found an audio gap on the Unmatched: a player using 16% of a hart where it needed 3%. - A killed or faulting process has a disconnect pulse sent for every connection it held; every pulse names its connection, as on QNX.
mremapmoves a mapping, and themmapcursor stops below the main stack, which a large enough mapping used to reach.
QSOE/L
- seL4 is patched from
patches/seL4/, one change per patch: among them the modern SBI extensions only, a cold reset through SBI SRST sorebootno longer powers the board off, and the Multiboot3 boot. - Self-pulses bypass taskman (above): a process connecting to its own ringed channel receives the channel's notification and its connection's id, and its pulses go into a per-channel lock-free queue with a sequence number per cell, since the sender may be an interrupt thread that preempted a receiver on its own hart.
- Pulses no longer wait inside taskman: each receiving process has a pulse page with a ring per channel, which taskman fills and the receiver reads itself. The fetch that a server once made, at whatever moment its pulse came, could meet a taskman blocked in a call to that same server.
- Windows: a server learns where a window is from a records page taskman fills before the client hears of it, and never asks taskman — the earlier hello and query could deadlock against a taskman that was itself the client. Windows are given back while the server lives (above).
libseam: the client side of IPC — connections, windows,MsgSend— is written once and compiled into bothlibqsoe.soand taskman, which opens the program it loads exactly as a client opens a file.- Every thread starts through a fence: RISC-V keeps no coherence between
data stores and instruction fetch, seL4 gives userspace no remote
fence.i, and a program placed on another hart than the one taskman wrote it from faulted on its first instruction, at random. Every thread now starts at a page offence.i; jr t0. - Condvars park, on one park queue with the futex contract; a parked thread can be destroyed without hanging its joiner.
- The master pool is counted: thread and channel creation draw from the
caller's memory, answered reply objects are reused, and
/sys/statscarries eight gauges of what is left. The fault flag moved from bit 16 to bit 30, guarded by asserts. - A refused spawn gives back everything it took, through one exit that frees in teardown order; the K3 ran twenty passes in one boot with taskman's slots flat.
/dev/urandom; a child no longer outlives its parent on paper;sysinfosays “not measured” for the CPU load seL4 does not account, andps -Hprints-for a thread's CPU time, not a zero.
Memory and the loader
libqsoe.sobeneathlibc.so(above). taskman bindslibqsoe.sofirst, thenlibc.soagainst it, then the program's other libraries, then the program.- A library is found on disk:
/libin the boot archive, then/usr/lib,/usr/local/liband the package farm's/usr/pkg/lib. A library read from disk is kept until reboot, in a slot of its own, so its text is shared like the archive's. - Images arrive 64 KiB a message, not 896 bytes: on QSOE/N into taskman's scratch area by the kernel's large-message copy, on QSOE/L through a window taskman keeps on each filesystem connection. On QSOE/L a relocation pass also maps each page once rather than once per write. Together: a 500 KB spawn on QEMU went from 404 to 166 ms, and a suite pass on the K3 from 19.9 to 11 s.
- A spawn carries its arguments by reference, up to
TM_ARG_MAX; taskman reads the strings straight onto the child's stack. The 1 KiB blob and its limits of 16 and 32 strings are gone. - A refused spawn leaves taskman as it was on QSOE/N too: a spawn that failed half-way through loading left pages mapped in taskman's own space, and the next one made Skimmer panic.
Processes, sessions and signals
posix_spawnattributes: process groups, session and ids applied before the child exists, and signal mask, ignored signals and scheduling in the child's own startup; a terminating signal is reported, sowaitpidtells Ctrl-C from an exit.- Ctrl-C reaches a command run by a script or
sh -c: only an interactive shell hands the terminal over now. The file manager's commands could not be interrupted before. - The loader's errno reaches the requester:
ENOEXECfor a file that is neither an image nor a#!script, which is the one distinction a shell needs to run a file as a script. - A refused spawn releases the descriptor holders it claimed, so a pipe's write end does not keep a phantom writer.
- taskman's log is a 64 KiB ring from the first line of the boot,
readable at
/sys/logand merged intosloginfo./proc/<pid>/infocounts a process's memory, threads, channels and connections.
Filesystems, paths and resource managers
- A link may lead out of a filesystem. A server that meets a symbolic link
whose target leaves its mount answers
ERELINKwith the path its walk reached, and the client's library resolves it from the root and repeats the verb — every path verb goes through one function now. Found by the package system's link farm, whose absolute targets cross from/varinto/usr. /varlives on the disk, as a link in the boot archive to/usr/var; so do the package system's repository and anchor files under/usr/conf.- Pathmgr links on request (root's): owned by their maker, removed only by
it, gone when it exits, followed in chains; QSP uses one to make a station's own name
under
/netlead back to/. A directory the tree implies —/dev/snd, made by/dev/snd/pcmC0D0p— resolves as one. lstatof the archive's own links returns the link; archive files report their entry's time rather than 1970.libqrvfs: the filesystem's format and operations in one library, underfs-qrvand the FUSE driver alike.- The resource-server framework answers a cancelled client's deferred call
with
EINTRinstead of dropping it, and tells a server when its client ends. - Ownership is checked on
chownandchmod: the framework applies the POSIX rules for every server built on it, so only root gives a file away, and only its owner or root changes its mode, its group or its times.
Packages and the system image
qupkg:update,search,info,install(by name or from a.qupfile),remove,autoremove,upgrade,list,files,verify,clean. A.qupis a newc cpio — the format taskman reads for the boot archive — compressed as one LZ4 frame. The farm linksbin,sbin,liband the manual pages; the defaultPATHends with them.- The image: an installed
qsoe-base, a 256 MiB/usrvolume labelledqsoe-usr, and the optional packagesqsoe-tests,qsoe-net-tools,qsoe-gpio-i2candqsoe-audio-samples. The disk keeps what the system wrote between builds, andfscheckruns before every boot. qsoe-c-sdk: the headers,crt0.o,libc.soandlibqsoe.soto link against; the compiler packages in the repository depend on it.update-os(above), and/usr/local/sbinon the default search path for the administrator's tools.
Drivers and hardware
- The audio drivers (see Sound).
pci-server: single-message MSI, and two BARs the firmware's layout got wrong; the DesignWare and PLDA MSI demultiplexers run at an interrupt thread's priority rather than an ordinary user's.devn-gemreceives on its interrupt. It polled every millisecond, which capped a QSP transaction at one per poll: a 57.9 MB copy from a development station to the Unmatched took 73 s on QSOE/L. On interrupts it takes 18 s there and 12 s on QSOE/N — four and six times faster. The DesignWare and GEM drivers' watchdogs act only on a frame left a whole period, so they no longer steal ordinary traffic.- The JH7110 libraries — its clocks, the AXI DMA — and GPIO and
I2C drivers for the VisionFive 2, shipped as a package with an
i2cprobetool. - The console: a serial terminal's size is asked for, by
resizeand bygettybefore each login; the file manager's cursor rests where input happens.
The C library
- The split (above):
libqsoe.so, its ABI file, the no-weak-symbol check on every image.open,read,write,nanosleep,_Exitandposix_spawnare kernel-neutral and live in the shared body. - stdio reads a buffer's worth per
read(), not one byte. pathmgr_symlink()andpathmgr_unlink();lstat()reports a link's own record; a path through a link to a directory lands in the directory.mremapthat moves,access()that asks the file,strtok_r, musl'slibm; a thread attribute's priority 0 means the ordinary user priority, as the header always said.
Shell and utilities
- Background jobs (above). A command that cannot be run is that command's failure, not the shell's.
- New:
tail(with-f),head,sha512sum,ls -d,resize,qplay,qupkg,update-os,workday; manual pages for several of them. qviewpages down with Space.- The shared suite grew from 502 checks to nearly eight hundred:
793 of 793 pass on QSOE/N under QEMU, 785 of 786 on QSOE/L — the one is the
[QSOE/N only]case of cancelling a blocked thread, reported rather than hidden. It ships as theqsoe-testspackage.
Boot, build and tooling
- One boot archive (
make modpkg) for both kernels;modpkg-l.cpiois gone. QSOE/L's build producessel4-<board>.elf, a Multiboot3 kernel; the elfloader rules are gone. lq/emu.shboots QEMU the way the boards boot: HFI firmware, mr-bml,multiboot3andmodule3, from a small disk laid bymklqboot.sh.- HFI BIOS 1.4.2: the VisionFive 2's reboot no longer wedges. The video firmware puts an HDMI link in HDMI mode only when the sink's EDID says it is one, and the audio drivers never change the picture.
- A FUSE driver mounts a QSOE volume on Linux, by device or by label.
Bugs worth naming
Found in this cycle, each with what it taught.
- The song that tore beside the suite, bisected by ear: clean beside CPU loops, three hundred spawns from RAM and a hundred reads of the suite's binary; torn beside spawns from the disk. taskman was reading an image, and a driver's own wake-up was waiting in its queue. A third party on a data path is a party to every stall the data path can meet.
- A cancelled call dropped unanswered held a window, a reply object and a slot per killed client; six kills exhausted an audio driver's address space. Found by killing the player in a loop — the simplest test there is.
- A program faulted on its first instruction, at random, on a board with many harts and never on QEMU: instruction fetch on the new hart still saw the frame's previous tenant.
- Every request on a new connection taken for a fault after 65,536 connections, because the fault flag lived where a connection id could reach it.
- A shell that could not interrupt what a script ran: the terminal was handed to a pid whose process group had no members.
- The receive park kept a stale message pointer, and once a park loop re-tested before every switch, a spurious wake handed one message to a server forever.
- QSP at 0.79 MB/s on a gigabit link was a driver polling once a millisecond, phase-locked into every transaction.
Open every section · the same ground is covered in the
per-component CHANGELOG.md files in the
source repositories.
Known gaps
Stated rather than implied away. These are the things 0.4 does not do.
- Timing on QSOE/L.
nanosleepis a spin onrdtimeand a yield, because taskman has no timer of its own to wake a sleeper;TimerSettimeis a stub, sopoll()ignores its timeout; and taskman's timers fire only when a request arrives. - No timed waits on either kernel:
pthread_cond_timedwait,TimerTimeoutfor a condvar, a boundedInterruptWait. - Cancelling a blocked thread on QSOE/L takes effect when the call returns,
not before, as does a signal caught by a handler; on Skimmer the thread is woken with
EINTR. A client REPLY-blocked on a server that exits parks for good there. - taskman serves from one thread on both kernels, so loading a program still delays every other request; the pool on QSOE/N is held to one thread until a lost wake between harts is found.
- QSP moves 896 bytes per read with one transaction in flight, does not use windows, and carries no credentials the far side acts on. The K3 on QSOE/L is the slowest station, for a reason not yet measured.
- No
dlopen(), and constructors inDT_INIT_ARRAYare not run. - A clock kept by the platform cannot be set (the K3), and the VisionFive 2 has no battery-backed clock QSOE recognizes.
- The VisionFive 2's jack and its HDMI cannot play at once; HDMI is the one started at boot.
qupkghas no command to switch back to an earlier generation, though every generation it made is still on the disk.- A home directory is persistent, its working directories are not:
~/projand~/binare in memory, refilled byworkday. fs-tmpfsdoes not check permissions, on the object or on the path to it.- A program that keeps a connection to its own channel must detach it before it exits: the descriptors closed at exit include it, and the close message sent to a channel nobody is receiving on waits for good.
Getting it
Pre-built images are published at github.com/qsoe-dev/dl; the front page lists what each file is and how to run it under QEMU or install it on a board. Under QEMU, name the variant — it selects the machine — and boot through the distribution's edk2:
# unpack the images (virtio.img is QSOE/L's root under QEMU) gunzip nvme.img.gz virtio.img.gz # QSOE/N; for QSOE/L say l, and pick the matching entry in mr-bml's menu NVME_IMG=./nvme.img VIRTIO_IMG=./virtio.img ./run-nvme.sh n --uefi
To build from source:
git clone https://gitlab.com/qsoe/os cd os make prepare # fetch the 0.4 component set (see component.list) make # build both variants: QSOE/N, then QSOE/L make nvme # the disk image, its system volume an installed qsoe-base make dist # optional: the QEMU disk images
The manuals — Design.pdf, UserGuide.pdf,
ProgrammingBook.pdf, AppPortingGuide.pdf,
Networking.pdf and LibcReference.pdf — are published
alongside each release at github.com/qsoe-dev/doc.
The Design manual has a new chapter on the window, the mechanism behind large messages on
QSOE/L, and one on storage and packages; the User Guide has chapters on sound and on
packages. Documentation is a release gate: no version ships before the manuals are brought
up to the tree.
Next is 0.5 — the real-time package: priorities, interrupt service threads, priority inheritance and bounded paths end to end, every step between an interrupt and the thread that services it accounted for — and a first honest worst-case latency figure. Sound is how it will be heard.
Back to QSOE Systems