QSOE Systems

Release notes

QSOE 0.5

Real time — a task that holds its deadline while the machine is abused on purpose, and a first worst-case latency figure, with its conditions.

November 2026 · succeeds 0.4 (October 2026) · Apache-2.0

0.5 is the real-time release. The roadmap said it was done when two sentences were true: a hard-real-time task holds its deadline while the rest of the system is deliberately abused, and a first honest worst-case latency figure exists. Neither sentence had a program behind it, so the cycle wrote them: test_rtload is the abuse, test_deadline the task and test_latency the figure. They stand outside the test suite, because the suite answers pass or fail and these answer with numbers a person has to read. On the K3 under QSOE/N, in October, a task with 400 µs of work every 2 ms ran 15,000 periods against CPU spinners, a storm of process creations and a disk written with an fsync behind every 32 KiB, missed none, and was released 7 µs late at worst. On the Unmatched under QSOE/L, on seL4 17.0.0, with a process started every second, an interrupt reaches its thread in 3.0 µs at the median and 17.9 µs at worst. Under a storm of process creations QSOE/L has no figure yet for this kernel, and its last one, on the K3, was bad; Known gaps says how bad.

The figure earned its place before it was published. That same measurement on stock seL4 had a worst case of 83 to 96 µs against 7 to 11 on the idle machine, and nothing in QSOE was slow: seL4 holds its kernel lock with interrupts off through up to 100 work units of a revoke or a delete, and creating or ending a process is hundreds of them. The kernel is now built to look for an interrupt every 10, and the suite takes the same 32 seconds. QSOE/N went through the same hunt: a walk of 4096 channel slots at every process end, a 16 MiB unmap at every spawn from disk, and a console line printed inside a system call with interrupts off for 3.7 ms. A spawn storm on the K3 now has its worst interrupt at 49 µs, down from 83. What is left, and what nobody can yet explain, is under Known gaps.

Around it: a thread that waits for a lower one now lends it its priority, on both kernels, and both run one set of priority bands. Every board has a timer of its own for QSOE/L to sleep on, so a sleep there no longer burns a hart. An audit of who may do what closed a number of real holes, among them raw disks any user could read. Programs may be large: qfpc now builds itself on QSOE/L. And sound, which 0.4 named as how this release would be heard, was checked by ear on the boards during the cycle, including a song streamed from a development station over the network, clean during a parallel compiler build on the K3.

The release was verified on the VisionFive 2. There the tagged build ran the suite twice on each kernel: QSOE/N 172 passed, 0 failed; QSOE/L 159 passed, 0 failed, with 8 known gaps and 4 skipped. The K3 and the Unmatched ran the build just before it, without the last fixes: 154 passed, 0 failed, 8 gaps and 4 skipped under QSOE/L on both.

The component set

ComponentWhat it is0.40.5
nq QSOE/N — the Skimmer microkernel and its taskman 0.360.39
lq QSOE/L — the seL4 taskman, its seam and its kernel patches 0.320.35
libc the C library, libqsoe and crt0 0.250.28
quser the shared userspace: shell, drivers, servers, utilities 0.240.27
qupkg the package manager (separate project, shipped in the base system) 0.20.4
k3sel4 seL4 for the SpacemiT K3 (separate project) —0.4
mr-bml the bootloader (separate project, shipped alongside) 1.01.0

The umbrella records the tested set in component.list; make prepare checks out exactly these. seL4 is now 17.0.0. The firmware the boards run is HFI BIOS 1.4.2.

On this page

Highlights

Thirteen things that make 0.5 a different system to depend on than 0.4.

A deadline held under abuse. test_rtload runs, in a process of its own, CPU spinners, a spawn storm, a file rewritten with an fsync behind every write, unpaced console output and a read over a station link. It proves each load is running before the measurement starts, because a figure measured against pressure that was not there is a figure about nothing. test_deadline is the periodic task: a counted amount of work, not a spin on the clock, on an absolute schedule. On the K3 under QSOE/N, with a 2 ms period and 400 µs of work: released 3 µs late at the median and 4 at worst when idle; 3 and 7 against spinners, spawns and disk, 15,000 periods, none missed, and adding the console flood changed nothing measurable. Under QSOE/L the same task, parking between releases, against one spinner, spawns and disk: 17 and 53 µs, 15,000 periods, none missed. Both are October's figures, QSOE/L's on seL4 16.0.0.

A first worst-case latency figure. test_latency arms a board timer to fire at an instant it knows, and the thread attached to the interrupt reads the CPU's counter as its first act on waking. The instant is measured, not assumed: the timer is first polled with its interrupt off to calibrate the prediction, and the spread is printed as the figure's uncertainty. The path is the whole of it: the interrupt controller, the kernel's dispatch, the scheduler, the return to user level. On the Unmatched under QSOE/L, seL4 17.0.0, 20,000 interrupts a run: idle, 2.9 to 3.0 µs at the median and 7.0 to 10.9 at worst over four runs; with one process started per second, 17.0 to 17.9 µs at worst, 13.9 at p99.9, none past 50. On the K3 under QSOE/N the worst under a spawn storm is 49 µs, none past 50. An emulator cannot measure either.

Ten work units, not a hundred. A revoke or a delete in seL4 runs with the kernel lock held and interrupts off, and looks for a pending interrupt every KernelMaxNumWorkUnitsPerPreemption units, about one capability each. Stock seL4 says 100, so with a process started per second the Unmatched saw a worst case of 83 to 96 µs and 8 or 9 samples past 50. Built with 10: 17.0 to 17.9 µs and none past 50. A preempted call is restarted with its work kept, taskman sees no difference, and the suite ran in 32.3 s against 32.2 s.

Priority inheritance, on both kernels. On Skimmer a thread now lends its priority through every kind of wait the kernel can see: a mutex and chains of them, a message while it is the server's work, a pulse, a busy channel, whose every serving thread is raised. A thread's own priority is kept apart from the one it runs at, and SchedGet reports both. A mutex inherits unless told otherwise, as on QNX, and PRIO_PROTECT's ceiling holds. seL4 lends no priority, so on QSOE/L the C library does the lending: a mutex raises its owner along a chain within the process, and a server adopts a client's priority as a request arrives, until its next receive or a reply. Against sixteen CPU hogs on QSOE/L, the suite's case acquires the mutex in 21 ms with inheritance and 522 ms without. What QSOE/L cannot do yet is listed under Known gaps.

One set of priorities, and a ceiling. <qsoe/prio.h> names the same numbers for both kernels: a new process at 10, servers at 16, interrupt threads at 21. An ordinary user's ceiling is 15, one below the servers, so no process without root can starve a server, a driver or taskman; root's is 254. Taskman waits at 16 and does each request's work at its client's priority, so a real-time task's spawn is served at its priority and a compiler's is not served above 16. Until this cycle QSOE/L's servers ran at the default, 10, where a CPU load at 12 starved every one of them.

A timer of its own on every board. seL4 keeps the architectural timer and offers user level none, so a sleep on QSOE/L was a spin that held its hart. On the Unmatched it was worse: devu-xhci's hotplug thread, sleeping that way, stalled the boot. Now devt-plat owns a spare timer block and holds each sleeper's reply: the goldfish RTC under QEMU, the soc-timer on the K3, the JH7110's timer on the VisionFive 2 — not in the firmware's device tree, so that driver is keyed on the SoC — and a PWM block on the Unmatched, which has no timer for user level: a PWM is a counter with comparators that raise interrupts, and that is all a wake-up needs. It takes pwm0, whose outputs light the LEDs, which go dark, and never a block a pwm-fan node drives. On the K3, 200 sleeps of 2 ms are 17 µs late at the median and 21 at worst.

seL4 17.0.0, which loses no IPI. seL4 kept one slot per core for the IPI sent to it, so a remote call could overwrite a reschedule in flight, and a thread made runnable on an idle core never ran: in QEMU, about one suite run in eight hung. A patch of ours held it through the cycle; 17.0.0 fixes it upstream, and QSOE/L now builds on that release with the patch gone. Reboot and power-off used seL4's debug calls, which a release kernel does not have, so a board asked to reboot simply stopped. seL4 now gives the root task an SBI capability, in release kernels too, limited to System Reset and the Base extension.

Who may do what. Raw disks were 0644: any user could read the partitions, and through them every file whatever its mode, /etc/shadow included. That was a security hole, and it is closed: they are 0600. So were others. Any process could register a name over /dev/console or a filesystem, be handed a copy of another's open file by naming a number, bind itself to a root pipeline, map any physical address on QSOE/N, map the network driver's live ring, or read a session's screen. fs-tmpfs checked no permission at all. Each of these now asks who is making the request, and memory is granted to one client or by a token rather than to everyone.

Nothing a program asks stops the kernel. On Skimmer one ecall with a bad number, a MAP_FIXED over a mapped page, or a pthread_detach() each stopped the machine. A call the kernel does not serve now ends the program with SIGSYS, announced by name; a map over a map is refused with nothing changed; ThreadDetach exists. And loops of crashing processes on the K3, thousands a minute for hours, found the races that QEMU never showed: a thread freed before its registers were saved, teardown and taskman's work on one process running at once on two harts, a notice to a server dropped from a full ring. Taskman now tears a process down, as QNX's process manager does, and a notice rides inside the object it is about, so none can be lost.

Large programs. Every fixed address a process has is defined once, for both kernels, in <qsoe/addrspace.h>, and a program has the first GiB to itself. On QSOE/L no program could be larger than about 2 MiB; it now takes images up to 64 MiB of program region, 8 MiB of it from the file. A program says how much stack it needs, by the linker, up to 8 MiB. Under mallocng sits an arena that takes memory in chunks, where every small mapping used to be a request to taskman and, on QSOE/L, 2 MiB: qfpc compiling itself asked for 2 MiB and was given 336. It now builds itself on QSOE/L.

USB beyond the keyboard. devu-xhci takes message-signaled interrupts on PCI controllers rather than polling them. It drives USB MIDI (/dev/midi0, tested with an Akai MPK261), CDC-ACM serial ports (/dev/acm0, acm1) and DirtyJTAG probes (/dev/jtag0), and lsusb asks a new /dev/usb what is on the bus and what claimed it. On a K3, a bitstream went through /dev/jtag0 into an Olimex GateMate board and the FPGA machine's own loader took a program on /dev/acm0.

The system says what it is. Both kernels keep a 16 KiB ring of everything they write to the console from the first line of the boot, and sloginfo now opens with it: one time line from the kernel's first word through taskman to USB. The boot archive carries its own package record, so qupkg files qsoe-boot lists the RAM disk and qupkg verify qsoe-boot re-hashes it. qupkg owner names the package a file came from. /dev nodes have device numbers of their own, and ttyname() stands on them.

A suite that reads, and a “0 failed” that means one thing. Each case is one line of short step names with its verdict in a column. On QSOE/L, a case of the [QSOE/N only] tail that fails is a GAP, counted apart and listed, so 0 failed means the same on both kernels: nothing new is wrong. Every priority rule has a case, with a control that must show the inversion. The tests left the image: they are the qsoe-tests package, and runtests fetches and runs them.

The full change list

Everything that changed since 0.4, by area. Open the parts you want — or open them all and read it as one document.

Real time

The figures

MeasurementConditionMedianWorst
Deadline, QSOE/N, K3: release lateness, 2 ms period, 400 µs of work idle 3 µs4 µs
the same, 15,000 periods, none missed spinners, 3,410 spawns, 22 MiB written with fsync 3 µs7 µs
Deadline, QSOE/L, K3: the same task, parking; 15,000 periods, none missed 1 spinner, spawns, disk 17 µs53 µs
Interrupt to its thread, QSOE/N, K3 idle, 100,000 interrupts 2.9 µs5.3 µs
the same spawn storm; none past 50 µs —49 µs
Interrupt to its thread, QSOE/L, Unmatched, seL4 17.0.0, 20,000 interrupts a run, ±1 µs idle, four runs 2.9–3.0 µs 7.0–10.9 µs
the same, stock seL4 (100 work units) one process started per second; 8 or 9 samples past 50 µs —83–96 µs
the same, as shipped (10 work units) one process started per second; p99.9 13.9 µs, none past 50 µs 3.0 µs17.0–17.9 µs

The deadline rows and the idle K3 row are from October, QSOE/L's on seL4 16.0.0. The K3 has not been re-measured on seL4 17.0.0 with 10 work units. On QSOE/L the interrupt is measured through devt-plat -P; see Known gaps for what that changes.

The programs

  • test_rtload: five loads, each aimed at a different path that could be unbounded. Every load is proven before the span starts, and one that was asked for and could not run makes the program exit non-zero.
  • test_deadline: the work is a counted number of units, fitted on the idle machine, since a task that spins until the clock says 400 µs does less work under load. The schedule is absolute, so a late release cannot push the rest back and hide what it cost; the sample arrays are touched first, so a first-touch fault is not recorded as the system being late. A missed deadline is in the exit status. At a 1 ms period with 200 µs of work on the K3 under QSOE/N, 5,000 periods missed none, 33 µs at worst.
  • test_latency: delays are pseudo-random, so samples cannot fall into step with a periodic load; a watchdog ends the run, with the block's registers, when an armed interrupt does not arrive; an interrupt earlier than predicted by more than the uncertainty is reported as the figures not holding.
  • devt-plat -P: on QSOE/L the timer server owns the interrupt, and stopping it would turn every sleep into a spin and starve the load. So the server, in this mode, arms for an instant the program names and answers with when its interrupt thread woke. Off by default; root only; 50 µs to 250 ms; one probe at a time. A sample counts only when the arming it belongs to is proven to have matched: an interrupt can arrive after the arming that raised it was replaced, and one sample in 40,000 once came out 53 µs early.
  • QEMU cannot say. Eight harts on an eight-core host miss 125 deadlines of 1,000: the guest's clock is the host's and its harts are host threads. It shows that the programs work, and nothing about the system.

Paths hunted out

  • seL4 preemption: 10 work units, not 100 (see Highlights). A build directory configured before this keeps its cache: delete its kernel.elf, or make distclean.
  • A process's end looked at every channel in the system: all 4096 slots, a cache miss each, 30 to 55 µs with interrupts off at every end on the K3. Each address space now lists its own channels. The teardown went from 55 to 25 µs (worst 88 to 36), and a spawn storm's worst interrupt from 83 to 49.
  • A spawn from disk unmapped a 16 MiB window first, because its releases did not give back exactly what its reads had mapped: 4096 slots and a TLB fence on every hart, 50 to 77 µs per spawn with interrupts off. A read now reports what it mapped, and one release gives exactly that back.
  • A spawn broadcasts its TLB and I-cache change once, not once per window, some ten times per spawn into M-mode on all harts. A transplant went from 28 to 9 µs on average, and a storm ran 10% more spawns.
  • Setting the clock printed a line inside the system call, with interrupts off for the 3.7 ms it took to send. It prints nothing now.
  • A crash storm no longer floods the console: the first 16 reports, then one in 1,000, with a count.
  • An interrupt thread's helpers: a thread created by an interrupt thread inherits its band, so devn-dwmac and devn-gem now give their watchdog and link helpers the server band. At 21, above taskman, one of them spinning on taskman's hart stopped the K3's boot under QSOE/L.
Priorities

Both kernels

  • <qsoe/prio.h>: a new process at QSOE_PRIO_USER (10; it was 16 on Skimmer), resource managers at QSOE_PRIO_SERVER (16), interrupt threads at QSOE_PRIO_IST (21). qsoe_prio_band() raises the caller to a band, and a refusal is announced.
  • Servers and drivers take their band as they start. A thread attached to an interrupt is in the interrupt band without asking; the audio interrupt thread runs one above the rest.
  • A thread starts at its creator's priority when its attributes name none, and a process's first thread at the priority its spawn asked for, rather than at the default and raised later: a child of a raised parent could starve before its first instruction.
  • The ceiling: ThreadCreate and SchedSet above it fail with EPERM, as does SchedSet of another process's thread unless the caller is root.
  • _msg_info.priority: a server is told the priority its sender runs at, or a pulse's own. On QSOE/L every message states it, and it is held to the client's ceiling, so no client can claim more than it could set itself.
  • QSOE_CHF_FIXED_PRIORITY makes a channel that lends its servers nothing; libc's own waits — poll(), pause(), the signal thread — use it, as their pulses are notifications rather than work.
  • Taskman serves each request at its client's priority, or at its base of 16, whichever is higher.
  • sysconf() says so: _POSIX_THREAD_PRIO_INHERIT and _POSIX_THREAD_PRIO_PROTECT are defined, applied in full on QSOE/N and within a process on QSOE/L.

Skimmer

  • A message lends only while it is the server's work: from the rendezvous until a reply, from whichever thread sends one, the sender leaving, or that thread's next MsgReceive. A request parked to be answered later stops lending, as on QNX. The send queue is taken highest priority first.
  • A pulse lends until the receiver's next MsgReceive, never more than its sender has. Pending pulses and waiting senders are taken highest priority first.
  • Mutexes: waiters are woken highest priority first; one ticket lock, the innermost in the kernel, guards the walk along a chain. A mutex knows its owner by tid, so a thread that ended holding one no longer made the next thread of its slot the owner; the waiters stay parked until a MutexRevive, as on QNX.
  • SchedSet on a boosted server used to stop the kernel on an assertion. It sets the own priority and keeps the loans.
  • ThreadCreate honors the priority in the thread's attributes, which it ignored.

QSOE/L

  • The lending is libc's own bookkeeping, under the table lock that guards the park queue, which already records who waits for what. A thread's four claims — its base, the mutexes it holds, a client it serves, a PROTECT ceiling — are kept apart and re-applied as each is withdrawn.
  • That lock runs at the process ceiling. In the first version of this code a thread lowered itself out of it while holding the lock, was starved by the spinners, and everything waiting on the lock waited too: 479 ms of a 500 ms measurement, the inversion the code existed to prevent.
  • A thread's maximum controlled priority is its process's ceiling, set again whenever the credentials change. It used to be the priority the thread was made at, too low for anyone to lend it more.
  • ps shows a thread's priority and its hart; every thread used to read as priority 0. A priority lent on top, which libc applies straight to the kernel, is not seen there.
Timers and sleeping
  • devt-plat: one driver per timer block, matched by what the running system says it has, and started first by init on QSOE/L only. It proves the hardware at start-up and stands down if no interrupt arrives. Nothing is armed more than 250 ms ahead, so a missed interrupt costs lateness and not eternity.
  • The seam computes an absolute deadline on CLOCK_MONOTONIC and asks the server, so a sleeper is reply-blocked rather than spinning. In QEMU, ten seconds of sleeping used to cost one full host core on top of idle, and now cost nothing.
  • A sleeper finds a restarted server. A failed send drops the connection and looks the server up again before spinning. After a restart in QEMU the new server answered 693 sleeps, where it had answered none.
  • The K3's soc-timer, everything read from the device tree. The block is gated at reset, and a gated block reads zeroes without faulting, so it is made to prove it answers. Register writes are verified, as the block sits across a clock domain. The gate is named by the node's clocks property, not its timer id: the wrong one reads back exactly as asked and enables a clock with no counter behind it.
  • The Unmatched's PWM: comparator 0 armed one-shot at the finest scale the delay fits in; its clock, timed against the time CSR at start-up, measures 129.99 MHz. QSOE/L now boots to a login there.
  • The JH7110's timer: eight 32-bit down counters, channel 0 serving /dev/timer; 24000499 Hz on the board. 200 sleeps of 2 ms are 20 µs late at the median, one of them 3.1 ms late.
  • QSOE_SUBSYS_TIMER: WAIT carries an absolute deadline; STAT reports the hardware and how many sleepers an interrupt did not wake, so a timer that stops being delivered says so in a number rather than a hang; PROBE is the latency test's surface.
Kernels

Skimmer

  • A page the kernel has just mapped is reachable from the next instruction. The map paths used a plain fence, and the K3's cores keep a walk that found no page table and use it until told otherwise: measured past a second and millions of retries. Each map path now ends with sfence.vma on its hart. QEMU caches no translations and never shows this.
  • A tid is a number in its own process, as QNX numbers threads: the first thread is 1, others the lowest free, at most 256 at once. A thread of another process cannot be named. A new thread has its tid before any of its code runs.
  • ThreadDetach works; numbered and served by nothing, it used to stop the machine.
  • Teardown: the last thread out only leaves its page table, marks the address space and sends PROC_END; taskman does its own work and then has the kernel tear the process down. The last thread cannot be preempted on its way out holding a sync bucket's lock.
  • A switch reads the outgoing thread's address space before publishing it as off its hart, and the incoming thread clears its predecessor's running flag only once that one is saved.
  • Notices: DISCONNECT, UNBLOCK and PROC_END each travel in a node inside the object they are about, queued ahead of every pulse. One process ending with 250 open /dev/nulls used to overflow the ring of 8, leaving taskman a record of a process that no longer existed.
  • 63 processes can live at once, up from 31.
  • Memory: MAP_PHYS reaches devices only, and only for root; stacks end in guard pages; PROT_NONE withholds access, where it used to be read access; a map over a mapped page is refused, not a panic.
  • A crash cannot loop: when the crash stub cannot run, the kernel ends the process and tells taskman.
  • A connection can carry an identity, delivered unforgeably in place of the sender's: the mechanism a relay needs to act for the user it relays for. Nothing uses it yet.
  • A condvar wait could lose a signal from another hart; boot page-table changes now reach every hart's TLB; ThreadCreate's round-robin skips the K3's auxiliary cores.
  • The kernel's copies of libc's ABI are gone: syscall numbers, structs, codes and flags come from one header each.

QSOE/L

  • seL4 17.0.0, with patches 0001 to 0008 unchanged and 0009 gone; make prepare refuses a clone of another release than the pin.
  • The SBI capability, the RISC-V twin of Arm's SMC capability, in initial slot 15; a badge narrows it to one call. A kernel without it, or firmware that refuses, now ends in a logged EIO, and the machine keeps running. k3sel4 carries the same capability.
  • A pulse wakes the thread that receives it. A pulse wakes only the thread its channel's Notification is bound to, and that was the channel's creator, so a worker already blocked in MsgReceive was never woken: 5 times in 5, once measured that way. MsgReceive now moves the binding to the calling thread.
  • Taskman says which harts the machine has. The system map carries them, by the same two gates as QSOE/N. The K3 reads harts 0xfffd usable, 0xff00 auxiliary, and a thread nobody placed stays off the A100s.
  • A runmask names a hart, translated through the loader's map; it used to be read as a seL4 core index, off by one on the K3 from hart 2 up.
  • The system map says where the PLIC is: sysinfo said PLIC @0x0, 0 source(s) on every PLIC board.
  • Addresses: the service block at 0x7FE00000; a stack region with nothing mapped beneath it, so an overflow faults; a GiB for anonymous mmap; taskman's root CNode at 216 slots; 128 frames of anonymous memory per process; a worker's stack ends in a guard page.
  • MAP_PHYS is root's alone: seL4 keeps RAM out of its reach, but any process could map any device's registers.
  • 32 fixed-priority channels per process; a refused window names its program; ThreadDestroy of a thread holding the table lock is EBUSY rather than a hang.
Taskman and processes
  • Taskman is never a client of another process, on either kernel. It used to flush every filesystem and read /etc/passwd, group and shadow itself, from inside its dispatch, so a server that did not answer held every request in the system. TM_REQ_SYNC now answers which servers to flush and libc sends to them; the credential files are read by the root process that asks for the reload and handed over. On QSOE/N, spawn's image reads, which drop the lock, are taskman's last sends.
  • QSOE/N's dispatch pool is back, three threads; it seemed to hang the K3 at boot, and the cause was a driver thread placed on an auxiliary core.
  • dup shares the open file description on QSOE/N, as POSIX says. A copy used to be a second open with its own offset, so cmd >f 2>&1 lost output.
  • 256 open-file contexts on QSOE/N, up from 32, and taskman hears when a process ends holding its files: a few dozen kills used to exhaust them.
  • A process's end always reaches taskman: PROC_END replaces PROC_FAULT, the fault included, and a record still held as alive is closed as killed and named on the console.
  • A program says how much stack it needs: ld -z stack-size=N, read from PT_GNU_STACK; nothing, or less, gets 64 KiB, and more than 8 MiB is refused with a line saying so.
  • Two segments may share a page, mapped once with both segments' rights, on both kernels. A segment that overlaps the one before it is refused.
  • A program has the first GiB to itself (0x10000 to 0x40000000). On QSOE/N the sysmap sat inside it, and the first 2 MiB program ran its data over it.
  • A failed claim fails the spawn with ENFILE, after taking back the claims already made; the child used to start holding a descriptor its server knew nothing about.
Who may do what
  • A raw disk is root's: the NVMe and virtio nodes are 0600, not 0644.
  • The path space is root's: registering a name, or pointing one at another server, is EPERM for anyone else.
  • A copy of an open goes to the process that claimed it. _IO_DUP cloned any connection's open for any caller who named a number. Now a client sends _IO_DUPPREP on the connection it means to copy, which only a holder can do, and the dup that follows is served only for that claim. It holds for taskman, for resource managers, for the pipe manager, and for the fleet relay, where a guessed number used to reach an open file on another machine.
  • A signal needs the right to send it: root, or the same real or saved user. kill() reports EPERM rather than rewriting every refusal as ESRCH.
  • A pipe is its maker's. Any process could bind itself to any pipe by a sequential id.
  • Memory grants: a region may be published to a client by its client_id, or carry a token of 64 random bits, for a grant that belongs to a role rather than a process.
  • A driver's network ring goes to the client that attaches it. Measured before the change: an unrelated process mapped 397 KB of the live ring.
  • The console's text plane goes to the program holding the screen, by token. A grant anyone could map let every process read a session's screen, a password echoed on it included, and write over it.
  • fs-tmpfs enforces its mode bits, write permission on the directory, and the sticky rule on /tmp.
  • /dev/sysact is root's (0600), and a chord is acted on only from a connection root opened.
  • The console and serial lines: injecting input needs an open root made; the foreground group and the window size belong to whoever holds the line.
  • PCI configuration writes and MSI-X calls are root's; reading the bus is unchanged. The log can be erased only by root. GPIO and I2C honor the open: a pin change or a bus write needs one that asked to write.
Drivers and hardware
  • devu-xhci takes MSI on PCI, the way devb-nvme and deva-hda do: an IMSIC interrupt on an AIA board, a pulse from pci-server where the host bridge owns the MSI block (the Unmatched, the VisionFive 2). A wake that finds the event ring empty reads USBSTS and drains once more. Polling is the last resort, and says so.
  • /dev/usb and lsusb: one line per device, or the hub tree with -t, and each device's interfaces with -v; what claimed each device is the column that matters. The node is 0444, and deliberately not a way to reach a device.
  • USB MIDI 1.0: whole messages as plain bytes, every cable merged; a read parks, and poll() works.
  • CDC-ACM: one node per port, paired from the union functional descriptor rather than by position, set to 115200 8N1 with DTR and RTS raised. A read is posted and left queued, since a console says nothing for minutes and then a great deal.
  • DirtyJTAG: /dev/jtag0 carries the probe's command stream; the driver knows no JTAG at all. One program holds it at a time. xhci_bulk() waits against a completion slot per device, so loading a bitstream does not fight the hotplug walk.
  • A USB class driver gives its buffer back: on QSOE/L each one was a whole 2 MiB frame, kept for the life of the process.
  • An NVMe interrupt can arrive before its completion. On the K3's A100 cores, 4,249 of 48,000 wakes found the queue empty, and the device sends no second interrupt, so the submitter blocked for good. A wake that finds nothing now reads the controller's status register first: PCIe does not let that read pass the device's earlier writes. devb-nvme also asks the system map for its interrupt thread's hart.
  • poll() works on the Unmatched's serial line: the console login's terminal-size query read until a byte came, and login: never appeared.
  • Eight consoles on the screen: devc-hficon serves /dev/con1 to con8, Alt-F1 to Alt-F8 bring one to the front, and sysinit starts a getty on every one. Each has its own input, foreground group and full-screen claim. Shift-PgUp and Shift-PgDn look back three screens.
  • The keyboard driver spells a modified function or navigation key the way an xterm does, and qsh and the commander read such sequences to their end.
  • Sound: checked by ear on the boards this cycle, as above.
  • grouvr and grouvr-envoy: from a QSOE partition, grouvr talks to the gRouvr hypervisor, and a request from another partition to reboot or power off this one goes to sysact, which flushes the filesystems first. On bare metal the envoy says so and exits.
Filesystems and the network
  • fs-qrv and fs-tmpfs keep an unlinked file while it is open, until the last close. The node used to be freed at once: on /tmp a reader saw a file of 4294967296493 bytes, and on the disk { rm a; echo second > b; echo first; } > a left “first” in b, for good. A volume halted with such a file open is not marked clean, so the next boot's fscheck frees it.
  • A rename over a name never leaves it missing: a power cut between the two steps used to lose both.
  • fscheck reads each inode block once: 8,327 reads became 355 on a 256 MB volume, 3.5 s became 0.19 s on the K3. Boot runs fscheck -c -p, which checks only when the volume says a check is due.
  • Files on another station: rm, mv, chmod, chown, touch, truncate, ln -s and readlink work under /net/<station>/. The QSP wire format gained UNLINK, RENAME, READLINK, SETATTR and SYMLINK. A station that is not answering says EHOSTDOWN; a rename across stations is EXDEV.
The C library
  • The umask is applied. Taskman kept it and nothing took it off a mode. Now open(O_CREAT) does. That changes what programs get: under the default 022 a new file is 0644, not 0666.
  • sinf, cosf, expf, logf, powf, sqrtf: musl 1.2.5's, each within one ulp of its double counterpart, rounded; a program calling one used to fail to link.
  • ttyname() and ttyname_r(), declared until now and defined nowhere, on <qsoe/devnum.h>: a number for every node QSOE serves, Linux's wherever Linux fixes one.
  • malloc locks. It locked only once libc.need_locks was set, and nothing here set it, though every process has a second thread from the start. Two threads allocating at once corrupted the heap. It now always locks.
  • malloc takes memory a chunk at a time: 2 MiB on QSOE/L, 256 KiB on QSOE/N.
  • Priorities: sched_curpriority in struct sched_param; the mutex attributes' protocol and ceiling; qsoe_proc_prio_ceiling(). A mutex's protocol is in the high half of sync_t.count, zero meaning inherit, so a statically initialized mutex inherits without anyone having said so.
  • A spawn that asks for no priority starts the child at the spawning thread's own, as POSIX and QNX have it.
  • <qsoe/addrspace.h>; a static assertion checks that no two regions overlap. The sysmap page moved to 0x7FE01000, so a program that reads it is rebuilt with these headers.
  • environ: a program using it saw an empty vector, as libc kept the environment under an alias the loader does not redirect. qjmcc could find neither its assembler nor its linker.
  • sigaction() on SIGABRT ended the process with a jump to address 0: a mask copied over a buffer sized for musl's raw syscall, 120 bytes over the return address.
  • Answers that say what they do: mmap of a file fails with ENOSYS and says so once, where QSOE/L handed back zeroed memory and QSOE/N refused without a word; a fault the kernel ends reads as the same signal as one the process ends; a refused system call reads as SIGSYS.
  • gettid() in <process.h>, as on QNX 8; qsoe_conn_set_cred(); <qsoe/usb.h>; <qsoe/klog.h>; <qsoe/skimmer_sysno.h> as the one place QSOE/N's call numbers are written, where four copies had already drifted into a stack overwrite.
  • qsoe_alloc_phys says how to give the memory back: munmap, and the header now says what each kernel does with it.
  • The new calls join qsoe-abi0 at minor 1.
Shell and utilities
  • New: printf, on (on -p <priority> command, for looking at a system something higher is starving), lsusb, runtests.
  • uptime prints the time, how long the system has been up and how busy its harts have been since.
  • ps -w N waits N seconds before its snapshot, so that started high before a load it looks at a system too busy to start a new ps.
  • ls -l shows the modification time, in POSIX's two forms.
  • qsh: ! history expansion, with word designators and modifiers; ~user; completion of a word with a ~ in it.
  • more and less are links to qview.
  • sloginfo opens with the kernel's boot, under the kernel's name.
  • getty no longer spins when its line refuses a read: a second getty on a line that has one printed its banner forever. Its banner ends with the line it serves.
  • emu.sh boots what is there. It builds nothing now; it used to re-stage /usr on every start and quietly replace what the last run wrote.
Tests
  • One line per case, a failed step in capitals with its sentence under the line, and a list of what failed after the results. suite -v prints one line per check, as before.
  • GAP, not FAIL (see Highlights).
  • Priority inheritance, checked: a server at its client's priority; a mutex owner at its waiter's, along a chain and against hogs, with a PRIO_NONE control that must show the inversion; when a message's loan ends; a busy channel's servers raised; pulses that lend; PRIO_PROTECT; a mutex left by an ended owner; the process ceiling. What QSOE/L passes is required of it; what it cannot do stays in the [QSOE/N only] tail, with the reason in each file's header.
  • [mutex-loan] measures the inversion it claims to. On QSOE/L every hog had stopped before the span began, and the control passed for the wrong reason. Hogs now start at their priority and say when they spin, and the span starts once they all do. A gate keeps the high thread from holding a hart against them, which failed the case on every other pass on the Unmatched; it then ran nineteen times in a row.
  • New groups and cases: [dev] (no two devices share a number, a raw disk refused to an ordinary user, the umask, ttyname()); [msgprio]; math, over subnormals and arguments past 109; [dup] shared and claim; [thread] tids; [smp] pulse-worker; [crash] sigabrt; [mem] recycle; [QSOE/N only] bad-syscall; stack overflows; a hundred spawns killed while holding open files.
  • The suite runs as anyone, skipping what a user's ceiling forbids, and Ctrl-C ends it.
  • Flakes removed: spawn cases no longer race the child they end; [signals] kill waits a time rather than a hundred yields; [smp] cases take their harts from the system map.
Packages and the boot archive
  • qsoe-boot: /qsoe-boot.pkg at the archive's root records every file with its digest and every link with its target. It describes the archive that booted; install, remove and upgrade refuse it, as it changes only with qsoe-base, at the next boot.
  • qupkg owner <path>... follows every link on the way and names each package that records the path or the file it leads to; a path nobody records exits with status 1.
  • Source packages: qupkg source checks a source archive as an install checks a package — signature, size, digest — and unpacks it into /usr/src/, running no script; qupkg build-dep installs what building it needs.
  • Files are 0644, executables 0755, whatever the umask. The store, the database and a source tree used to be anyone's to rewrite.
  • A commit cut off at its rename is finished, not lost: with active missing and active.new naming a generation, the next command completes it. The next install used to build from an empty set.
  • The tests are a package, in the repository /etc/repositories names; the image no longer carries them.
Bugs worth naming

Found in this cycle, each with what it taught.

  • A worst case of 96 µs with nothing in QSOE slow. The stall was inside the kernel's own idea of how often to look up. A latency figure with a load beside it finds what an idle one cannot: 11 µs idle said nothing about it.
  • One more server stopped the boot. Starting the timer server moved the NVMe interrupt thread from hart 7 to hart 8, the first A100, where an MSI can outrun its completion. A hart count nobody published looked exactly like a single-core machine.
  • A timer that could not be told anything. A gate chosen by the timer id read back exactly as asked, and the only thing that showed it was printing the register's value instead of a verdict.
  • “crit_exit underflow” after five hours of crash loops: a thread published as off its hart before its registers were stored, so the store went into a new thread's block, which later resumed inside another's exit.
  • A shell capturing command output stopped every spawn after it. A realloc that grew across 2 MiB took its frame from taskman's master pool, whose cap the process's exit could not reach, and the next spawn found its slot occupied. Twenty lines of C and one run reproduce it.
  • A flake of 1 in 20 that was a certainty. A pulse to a blocked worker was never delivered; it only looked rare because a worker arriving later found it without a wake.
  • A test measuring nothing. Sixteen hogs had finished before the span began, and the control passed. The same rule as test_rtload's: prove the pressure was there.
  • A hole hidden by a frame. The audit of /dev looked at the servers under /dev, and taskman is not one of them, so the larger half of the dup hole sat outside it.
  • A question asked wrongly. The first fix asked a dup “did you open this?”, which POSIX answers no for every inherited descriptor, and shell redirections began to fail. A claim on the connection is the right question.

Open every section · the same ground is covered in the per-component CHANGELOG.md files in the source repositories.

Known gaps

Stated rather than implied away. These are the things 0.5 does not do, and the things it cannot yet explain.

Causes not yet known

  • Process creation is about 4 ms slower while the timer server runs in its measuring mode: 13.9 to 17.9 ms, and the suite 12 s longer, on the same kernel, taskman and thread placement, with sleeps and interrupts unchanged. Cause unknown. So QSOE/L's interrupt figures come from a system that is not quite the shipped one.
  • A tail on QSOE/N. One run under a parallel compiler build on the K3 had about one interrupt in a thousand at 230 to 310 µs. It did not come back in the runs after it, and its cause is not known. A worst case is only as good as the longest run behind it.
  • About 20 µs without a preemption point remains on QSOE/L during process creation, at 10 work units. It has not been found.
  • One 2 ms sleep in 200 was 3.1 ms late on the VisionFive 2 under QSOE/L.
  • devt-plat started by hand on QSOE/N froze the Unmatched twice. QSOE/N does not need it, and init does not start it there. The same experiment has not been run on the K3.
  • One record on QSOE/N outlived its process after a night of crash loops, with no lost notice behind it. Taskman now closes any such record and names it, but why that one happened is not known.
  • Two log lines nobody has explained yet: aux: read -41 on QSOE/N, and a null capability #73 on QSOE/L.

Not measured

  • The K3 under QSOE/L. Its last figures are October's, on seL4 16.0.0 with 100 work units: 252 µs at worst under the full load and 9.4 ms under a spawn storm. They were not taken again on 17.0.0 with 10 work units. A spawn storm on QSOE/L has no figure for the shipped kernel on any board: the 17.9 µs on this page is under one process a second, not a storm.

Real time and priorities

  • The spread between the kernels is a trade, not a verdict: QSOE/L's periodic task parks between releases and gives its hart back to the system, which costs its wake some tens of microseconds.
  • QSOE/L's inheritance stops short of QSOE/N's. A server waits for its next message at its own priority, so it is raised only once it has run to take a higher client's request. A busy server is not raised by a higher client or pulse waiting behind it. seL4 tells a server nothing when its client leaves. Pulses come first in, first out. A mutex chain stops at the process boundary.
  • On QSOE/N a kernel copy cannot be preempted. A system call runs with interrupts off, so a message's copy holds its hart for as long as the copy takes. On the VisionFive 2, cyclictest at priority 200 saw 18 to 20 ms beside a 4 MB message, one stall each way; about 5 ms beside a console line from taskman, when it landed on the same hart; and about 2 ms in rounds of starting and killing a process. The same run beside a parallel compile, six jobs for 96 seconds, stayed at 16 µs. So the 49 µs above is a spawn storm's figure, not a bound on the system. Bounded copying and zeroing are the plan for 0.6.
  • On QSOE/N, page zeroing and one large reply per spawn set what is left of the spawn storm's worst case, near 50 µs.
  • QSOE/L still spins in places: below 50 µs, and whenever no timer can be reached, a sleep is a spin that yields only to its equals.
  • No timed waits on either kernel: pthread_cond_timedwait, TimerTimeout for a condvar, a bounded InterruptWait.
  • A signal cannot interrupt a blocked call. On QSOE/L a thread holding libc's table lock cannot be destroyed either: ThreadDestroy answers EBUSY.
  • pthread_mutex_getprioceiling and _setprioceiling are stubs that announce themselves.

QSOE/L

  • A client REPLY-blocked on a server that exits is never released.
  • A process's main thread cannot be joined.
  • Spawning a second process before init runs is not handled.
  • With several threads receiving on one channel, a pulse wakes the latest to come in, where QNX would wake the highest-priority waiter.
  • CPU load is not counted, and uptime says so.

Everywhere else

  • Across the net, a station acts as itself. The far station performs a request with its own credentials, which are root's; the ids the request carries are not consulted. QSOE/N has the mechanism a relay needs, and nothing uses it yet.
  • /dev/net/en0 is 0666: any user can send raw frames.
  • fstat() of a pipe end is ENOSYS, and of a closed descriptor on QSOE/N EINVAL rather than EBADF.
  • isatty() tests S_ISCHR alone: it is true for /dev/null, and sets no ENOTTY. Serial drivers report st_dev 4.
  • mmap of a file fails with ENOSYS.
  • Math: the rest of the float set, and the doubles asin, acos, sinh, cosh, tanh, log10, hypot, cbrt, fmin, fmax and fma. Subnormals on the K3 and on U74 cores may hit a slow path.
  • A USB device unplugged from a root port is never released: the scan walks hub ports only, so a class driver goes on offering a device that is not there. No class driver gives back a transfer ring.
  • qmk exits with status 2 on Ctrl-C rather than ending by the signal.

Getting it

Pre-built images are published at github.com/qsoe-dev/dl; the front page lists what each file is and how to run it under QEMU or install it on a board. Under QEMU, name the variant — it selects the machine — and boot through the distribution's edk2:

# unpack the images (virtio.img is QSOE/L's root under QEMU)
gunzip nvme.img.gz virtio.img.gz

# QSOE/N; for QSOE/L say l, and pick the matching entry in mr-bml's menu
NVME_IMG=./nvme.img VIRTIO_IMG=./virtio.img ./run-nvme.sh n --uefi

To build from source:

git clone https://gitlab.com/qsoe/os
cd os
make prepare   # fetch the 0.5 component set (see component.list)
make all       # build both variants: QSOE/N, then QSOE/L
make nvme      # the disk image, its system volume an installed qsoe-base
make dist      # optional: the QEMU disk images

The manuals — Design.pdf, UserGuide.pdf, ProgrammingBook.pdf, AppPortingGuide.pdf, Networking.pdf and LibcReference.pdf — are published alongside each release at github.com/qsoe-dev/doc. Documentation is a release gate: no version ships before the manuals are brought up to the tree.

Next is 0.6 — self-hosting: qjmcc on the target rebuilds QSOE from source on its own filesystem, and a build started on the board produces binaries that boot the board. Each board's disk becomes two volumes as well: a system volume that can always be recreated, and a data volume that holds what the machine has accumulated.

Back to QSOE Systems