Release notes
QSOE 0.5
Real time — a task that holds its deadline while the machine is abused on purpose, and a first worst-case latency figure, with its conditions.
0.5 is the real-time release. The roadmap said it was done when two sentences were true: a
hard-real-time task holds its deadline while the rest of the system is deliberately abused,
and a first honest worst-case latency figure exists. Neither sentence had a program behind
it, so the cycle wrote them: test_rtload is the abuse,
test_deadline the task and test_latency the figure. They stand
outside the test suite, because the suite answers pass or fail and these answer with numbers
a person has to read. On the K3 under QSOE/N, in October, a task with 400 µs of
work every 2 ms ran 15,000 periods against CPU spinners, a storm of process creations
and a disk written with an fsync behind every 32 KiB, missed none, and was
released 7 µs late at worst. On the Unmatched under QSOE/L, on seL4 17.0.0, with
a process started every second, an interrupt reaches its thread in 3.0 µs at the
median and 17.9 µs at worst. Under a storm of process creations QSOE/L has no
figure yet for this kernel, and its last one, on the K3, was bad; Known gaps says how bad.
The figure earned its place before it was published. That same measurement on stock seL4 had a worst case of 83 to 96 µs against 7 to 11 on the idle machine, and nothing in QSOE was slow: seL4 holds its kernel lock with interrupts off through up to 100 work units of a revoke or a delete, and creating or ending a process is hundreds of them. The kernel is now built to look for an interrupt every 10, and the suite takes the same 32 seconds. QSOE/N went through the same hunt: a walk of 4096 channel slots at every process end, a 16 MiB unmap at every spawn from disk, and a console line printed inside a system call with interrupts off for 3.7 ms. A spawn storm on the K3 now has its worst interrupt at 49 µs, down from 83. What is left, and what nobody can yet explain, is under Known gaps.
Around it: a thread that waits for a lower one now lends it its priority, on both kernels,
and both run one set of priority bands. Every board has a timer of its own for QSOE/L to
sleep on, so a sleep there no longer burns a hart. An audit of who may do what closed a
number of real holes, among them raw disks any user could read. Programs may be large:
qfpc now builds itself on QSOE/L. And sound, which 0.4 named as how this release
would be heard, was checked by ear on the boards during the cycle, including a song streamed
from a development station over the network, clean during a parallel compiler build on the
K3.
The release was verified on the VisionFive 2. There the tagged build ran the suite twice on each kernel: QSOE/N 172 passed, 0 failed; QSOE/L 159 passed, 0 failed, with 8 known gaps and 4 skipped. The K3 and the Unmatched ran the build just before it, without the last fixes: 154 passed, 0 failed, 8 gaps and 4 skipped under QSOE/L on both.
The component set
| Component | What it is | 0.4 | 0.5 |
|---|---|---|---|
nq |
QSOE/N — the Skimmer microkernel and its taskman | 0.36 | 0.39 |
lq |
QSOE/L — the seL4 taskman, its seam and its kernel patches | 0.32 | 0.35 |
libc |
the C library, libqsoe and crt0 |
0.25 | 0.28 |
quser |
the shared userspace: shell, drivers, servers, utilities | 0.24 | 0.27 |
qupkg |
the package manager (separate project, shipped in the base system) | 0.2 | 0.4 |
k3sel4 |
seL4 for the SpacemiT K3 (separate project) | — | 0.4 |
mr-bml |
the bootloader (separate project, shipped alongside) | 1.0 | 1.0 |
The umbrella records the tested set in
component.list; make prepare checks out exactly these. seL4 is now
17.0.0. The firmware the boards run is HFI BIOS 1.4.2.
On this page
Highlights
Thirteen things that make 0.5 a different system to depend on than 0.4.
A deadline held under abuse. test_rtload runs, in a process
of its own, CPU spinners, a spawn storm, a file rewritten with an fsync behind
every write, unpaced console output and a read over a station link. It proves each load is
running before the measurement starts, because a figure measured against pressure that was
not there is a figure about nothing. test_deadline is the periodic task: a
counted amount of work, not a spin on the clock, on an absolute schedule. On the K3 under
QSOE/N, with a 2 ms period and 400 µs of work: released 3 µs late
at the median and 4 at worst when idle; 3 and 7 against spinners, spawns and disk, 15,000
periods, none missed, and adding the console flood changed nothing measurable. Under QSOE/L the same task, parking between releases, against one spinner, spawns
and disk: 17 and 53 µs, 15,000 periods, none missed. Both are October's figures,
QSOE/L's on seL4 16.0.0.
A first worst-case latency figure. test_latency arms a board
timer to fire at an instant it knows, and the thread attached to the interrupt reads the
CPU's counter as its first act on waking. The instant is measured, not assumed: the timer is
first polled with its interrupt off to calibrate the prediction, and the spread is printed as
the figure's uncertainty. The path is the whole of it: the interrupt controller, the kernel's
dispatch, the scheduler, the return to user level. On the Unmatched under QSOE/L, seL4
17.0.0, 20,000 interrupts a run: idle, 2.9 to 3.0 µs at the median and 7.0 to
10.9 at worst over four runs; with one process started per second, 17.0 to
17.9 µs at worst, 13.9 at p99.9, none past 50. On the K3 under QSOE/N the worst
under a spawn storm is 49 µs, none past 50. An emulator cannot measure either.
Ten work units, not a hundred. A revoke or a delete in seL4 runs with the
kernel lock held and interrupts off, and looks for a pending interrupt every
KernelMaxNumWorkUnitsPerPreemption units, about one capability each. Stock
seL4 says 100, so with a process started per second the Unmatched saw a worst case of 83 to
96 µs and 8 or 9 samples past 50. Built with 10: 17.0 to 17.9 µs and
none past 50. A preempted call is restarted with its work kept, taskman sees no difference,
and the suite ran in 32.3 s against 32.2 s.
Priority inheritance, on both kernels. On Skimmer a thread now lends its
priority through every kind of wait the kernel can see: a mutex and chains of them, a
message while it is the server's work, a pulse, a busy channel, whose every serving thread
is raised. A thread's own priority is kept apart from the one it runs at, and
SchedGet reports both. A mutex inherits unless told otherwise, as on QNX, and
PRIO_PROTECT's ceiling holds. seL4 lends no priority, so on QSOE/L the C library
does the lending: a mutex raises its owner along a chain within the process, and a server
adopts a client's priority as a request arrives, until its next receive or a reply. Against
sixteen CPU hogs on QSOE/L, the suite's case acquires the mutex in 21 ms with inheritance
and 522 ms without. What QSOE/L cannot do yet is listed under Known gaps.
One set of priorities, and a ceiling. <qsoe/prio.h> names
the same numbers for both kernels: a new process at 10, servers at 16, interrupt threads at
21. An ordinary user's ceiling is 15, one below the servers, so no process without root can
starve a server, a driver or taskman; root's is 254. Taskman waits at 16 and does each request's
work at its client's priority, so a real-time task's spawn is served at its priority and a
compiler's is not served above 16. Until this cycle QSOE/L's servers ran at the default, 10,
where a CPU load at 12 starved every one of them.
A timer of its own on every board. seL4 keeps the architectural timer and
offers user level none, so a sleep on QSOE/L was a spin that held its hart. On the Unmatched
it was worse: devu-xhci's hotplug thread, sleeping that way, stalled the boot.
Now devt-plat owns a spare timer block and holds each sleeper's reply: the
goldfish RTC under QEMU, the soc-timer on the K3, the JH7110's timer on the
VisionFive 2 — not in the firmware's device tree, so that driver is keyed on the
SoC — and a PWM block on the Unmatched, which has no timer for user level: a PWM is a
counter with comparators that raise interrupts, and that is all a wake-up needs. It takes pwm0, whose outputs light the LEDs, which go dark, and never a block a
pwm-fan node drives. On the K3, 200 sleeps of 2 ms are 17 µs
late at the median and 21 at worst.
seL4 17.0.0, which loses no IPI. seL4 kept one slot per core for the IPI sent to it, so a remote call could overwrite a reschedule in flight, and a thread made runnable on an idle core never ran: in QEMU, about one suite run in eight hung. A patch of ours held it through the cycle; 17.0.0 fixes it upstream, and QSOE/L now builds on that release with the patch gone. Reboot and power-off used seL4's debug calls, which a release kernel does not have, so a board asked to reboot simply stopped. seL4 now gives the root task an SBI capability, in release kernels too, limited to System Reset and the Base extension.
Who may do what. Raw disks were 0644: any user could read the partitions,
and through them every file whatever its mode, /etc/shadow included. That was a
security hole, and it is closed: they are 0600. So were others. Any process could register a
name over /dev/console or a filesystem, be handed a copy of another's open file
by naming a number, bind itself to a root pipeline, map any physical address on QSOE/N, map
the network driver's live ring, or read a session's screen. fs-tmpfs checked no
permission at all. Each of these now asks who is making the request, and memory is granted to
one client or by a token rather than to everyone.
Nothing a program asks stops the kernel. On Skimmer one ecall
with a bad number, a MAP_FIXED over a mapped page, or a pthread_detach()
each stopped the machine. A call the kernel does not serve now ends the program with
SIGSYS, announced by name; a map over a map is refused with nothing changed;
ThreadDetach exists. And loops of crashing processes on the K3, thousands a
minute for hours, found the races that QEMU never showed: a thread freed before its registers
were saved, teardown and taskman's work on one process running at once on two harts, a
notice to a server dropped from a full ring. Taskman now tears a process down, as QNX's
process manager does, and a notice rides inside the object it is about, so none can be
lost.
Large programs. Every fixed address a process has is defined once, for both
kernels, in <qsoe/addrspace.h>, and a program has the first GiB to itself.
On QSOE/L no program could be larger than about 2 MiB; it now takes images up to
64 MiB of program region, 8 MiB of it from the file. A program says how much stack it needs, by the linker, up to
8 MiB. Under mallocng sits an arena that takes memory in chunks, where every small
mapping used to be a request to taskman and, on QSOE/L, 2 MiB: qfpc
compiling itself asked for 2 MiB and was given 336. It now builds itself on QSOE/L.
USB beyond the keyboard. devu-xhci takes message-signaled interrupts on PCI
controllers rather than polling them. It drives USB MIDI (/dev/midi0, tested with
an Akai MPK261), CDC-ACM serial ports (/dev/acm0, acm1) and DirtyJTAG
probes (/dev/jtag0), and lsusb asks a new /dev/usb what
is on the bus and what claimed it. On a K3, a bitstream went through /dev/jtag0
into an Olimex GateMate board and the FPGA machine's own loader took a program on
/dev/acm0.
The system says what it is. Both kernels keep a 16 KiB ring of everything
they write to the console from the first line of the boot, and sloginfo now opens
with it: one time line from the kernel's first word through taskman to USB. The boot archive
carries its own package record, so qupkg files qsoe-boot lists the RAM disk and
qupkg verify qsoe-boot re-hashes it. qupkg owner names the package a
file came from. /dev nodes have device numbers of their own, and
ttyname() stands on them.
A suite that reads, and a “0 failed” that means one thing. Each
case is one line of short step names with its verdict in a column. On QSOE/L, a case of the
[QSOE/N only] tail that fails is a GAP, counted apart and listed, so 0 failed
means the same on both kernels: nothing new is wrong. Every priority rule has a case, with a
control that must show the inversion. The tests left the image: they are the
qsoe-tests package, and runtests fetches and runs them.
The full change list
Everything that changed since 0.4, by area. Open the parts you want — or open them all and read it as one document.
Real time
The figures
| Measurement | Condition | Median | Worst |
|---|---|---|---|
| Deadline, QSOE/N, K3: release lateness, 2 ms period, 400 µs of work | idle | 3 µs | 4 µs |
| the same, 15,000 periods, none missed | spinners, 3,410 spawns, 22 MiB written with fsync |
3 µs | 7 µs |
| Deadline, QSOE/L, K3: the same task, parking; 15,000 periods, none missed | 1 spinner, spawns, disk | 17 µs | 53 µs |
| Interrupt to its thread, QSOE/N, K3 | idle, 100,000 interrupts | 2.9 µs | 5.3 µs |
| the same | spawn storm; none past 50 µs | — | 49 µs |
| Interrupt to its thread, QSOE/L, Unmatched, seL4 17.0.0, 20,000 interrupts a run, ±1 µs | idle, four runs | 2.9–3.0 µs | 7.0–10.9 µs |
| the same, stock seL4 (100 work units) | one process started per second; 8 or 9 samples past 50 µs | — | 83–96 µs |
| the same, as shipped (10 work units) | one process started per second; p99.9 13.9 µs, none past 50 µs | 3.0 µs | 17.0–17.9 µs |
The deadline rows and the idle K3 row are from October, QSOE/L's
on seL4 16.0.0. The K3 has not been re-measured on seL4 17.0.0 with 10 work units. On QSOE/L the interrupt is
measured through devt-plat -P; see Known gaps for what
that changes.
The programs
test_rtload: five loads, each aimed at a different path that could be unbounded. Every load is proven before the span starts, and one that was asked for and could not run makes the program exit non-zero.test_deadline: the work is a counted number of units, fitted on the idle machine, since a task that spins until the clock says 400 µs does less work under load. The schedule is absolute, so a late release cannot push the rest back and hide what it cost; the sample arrays are touched first, so a first-touch fault is not recorded as the system being late. A missed deadline is in the exit status. At a 1 ms period with 200 µs of work on the K3 under QSOE/N, 5,000 periods missed none, 33 µs at worst.test_latency: delays are pseudo-random, so samples cannot fall into step with a periodic load; a watchdog ends the run, with the block's registers, when an armed interrupt does not arrive; an interrupt earlier than predicted by more than the uncertainty is reported as the figures not holding.devt-plat -P: on QSOE/L the timer server owns the interrupt, and stopping it would turn every sleep into a spin and starve the load. So the server, in this mode, arms for an instant the program names and answers with when its interrupt thread woke. Off by default; root only; 50 µs to 250 ms; one probe at a time. A sample counts only when the arming it belongs to is proven to have matched: an interrupt can arrive after the arming that raised it was replaced, and one sample in 40,000 once came out 53 µs early.- QEMU cannot say. Eight harts on an eight-core host miss 125 deadlines of 1,000: the guest's clock is the host's and its harts are host threads. It shows that the programs work, and nothing about the system.
Paths hunted out
- seL4 preemption: 10 work units, not 100 (see Highlights). A build
directory configured before this keeps its cache: delete its
kernel.elf, ormake distclean. - A process's end looked at every channel in the system: all 4096 slots, a cache miss each, 30 to 55 µs with interrupts off at every end on the K3. Each address space now lists its own channels. The teardown went from 55 to 25 µs (worst 88 to 36), and a spawn storm's worst interrupt from 83 to 49.
- A spawn from disk unmapped a 16 MiB window first, because its releases did not give back exactly what its reads had mapped: 4096 slots and a TLB fence on every hart, 50 to 77 µs per spawn with interrupts off. A read now reports what it mapped, and one release gives exactly that back.
- A spawn broadcasts its TLB and I-cache change once, not once per window, some ten times per spawn into M-mode on all harts. A transplant went from 28 to 9 µs on average, and a storm ran 10% more spawns.
- Setting the clock printed a line inside the system call, with interrupts off for the 3.7 ms it took to send. It prints nothing now.
- A crash storm no longer floods the console: the first 16 reports, then one in 1,000, with a count.
- An interrupt thread's helpers: a thread created by an interrupt thread
inherits its band, so
devn-dwmacanddevn-gemnow give their watchdog and link helpers the server band. At 21, above taskman, one of them spinning on taskman's hart stopped the K3's boot under QSOE/L.
Priorities
Both kernels
<qsoe/prio.h>: a new process atQSOE_PRIO_USER(10; it was 16 on Skimmer), resource managers atQSOE_PRIO_SERVER(16), interrupt threads atQSOE_PRIO_IST(21).qsoe_prio_band()raises the caller to a band, and a refusal is announced.- Servers and drivers take their band as they start. A thread attached to an interrupt is in the interrupt band without asking; the audio interrupt thread runs one above the rest.
- A thread starts at its creator's priority when its attributes name none, and a process's first thread at the priority its spawn asked for, rather than at the default and raised later: a child of a raised parent could starve before its first instruction.
- The ceiling:
ThreadCreateandSchedSetabove it fail withEPERM, as doesSchedSetof another process's thread unless the caller is root. _msg_info.priority: a server is told the priority its sender runs at, or a pulse's own. On QSOE/L every message states it, and it is held to the client's ceiling, so no client can claim more than it could set itself.QSOE_CHF_FIXED_PRIORITYmakes a channel that lends its servers nothing; libc's own waits —poll(),pause(), the signal thread — use it, as their pulses are notifications rather than work.- Taskman serves each request at its client's priority, or at its base of 16, whichever is higher.
sysconf()says so:_POSIX_THREAD_PRIO_INHERITand_POSIX_THREAD_PRIO_PROTECTare defined, applied in full on QSOE/N and within a process on QSOE/L.
Skimmer
- A message lends only while it is the server's work: from the rendezvous
until a reply, from whichever thread sends one, the sender leaving, or that thread's next
MsgReceive. A request parked to be answered later stops lending, as on QNX. The send queue is taken highest priority first. - A pulse lends until the receiver's next
MsgReceive, never more than its sender has. Pending pulses and waiting senders are taken highest priority first. - Mutexes: waiters are woken highest priority first; one ticket lock, the
innermost in the kernel, guards the walk along a chain. A mutex knows its owner by tid,
so a thread that ended holding one no longer made the next thread of its slot the owner;
the waiters stay parked until a
MutexRevive, as on QNX. SchedSeton a boosted server used to stop the kernel on an assertion. It sets the own priority and keeps the loans.ThreadCreatehonors the priority in the thread's attributes, which it ignored.
QSOE/L
- The lending is libc's own bookkeeping, under the table lock that guards
the park queue, which already records who waits for what. A thread's four claims —
its base, the mutexes it holds, a client it serves, a
PROTECTceiling — are kept apart and re-applied as each is withdrawn. - That lock runs at the process ceiling. In the first version of this code a thread lowered itself out of it while holding the lock, was starved by the spinners, and everything waiting on the lock waited too: 479 ms of a 500 ms measurement, the inversion the code existed to prevent.
- A thread's maximum controlled priority is its process's ceiling, set again whenever the credentials change. It used to be the priority the thread was made at, too low for anyone to lend it more.
psshows a thread's priority and its hart; every thread used to read as priority 0. A priority lent on top, which libc applies straight to the kernel, is not seen there.
Timers and sleeping
devt-plat: one driver per timer block, matched by what the running system says it has, and started first by init on QSOE/L only. It proves the hardware at start-up and stands down if no interrupt arrives. Nothing is armed more than 250 ms ahead, so a missed interrupt costs lateness and not eternity.- The seam computes an absolute deadline on
CLOCK_MONOTONICand asks the server, so a sleeper is reply-blocked rather than spinning. In QEMU, ten seconds of sleeping used to cost one full host core on top of idle, and now cost nothing. - A sleeper finds a restarted server. A failed send drops the connection and looks the server up again before spinning. After a restart in QEMU the new server answered 693 sleeps, where it had answered none.
- The K3's soc-timer, everything read from the device tree. The block is
gated at reset, and a gated block reads zeroes without faulting, so it is made to prove it
answers. Register writes are verified, as the block sits across a clock domain. The gate
is named by the node's
clocksproperty, not its timer id: the wrong one reads back exactly as asked and enables a clock with no counter behind it. - The Unmatched's PWM: comparator 0 armed one-shot at the finest scale the delay fits in; its clock, timed against the time CSR at start-up, measures 129.99 MHz. QSOE/L now boots to a login there.
- The JH7110's timer: eight 32-bit down counters, channel 0 serving
/dev/timer; 24000499 Hz on the board. 200 sleeps of 2 ms are 20 µs late at the median, one of them 3.1 ms late. QSOE_SUBSYS_TIMER:WAITcarries an absolute deadline;STATreports the hardware and how many sleepers an interrupt did not wake, so a timer that stops being delivered says so in a number rather than a hang;PROBEis the latency test's surface.
Kernels
Skimmer
- A page the kernel has just mapped is reachable from the next instruction.
The map paths used a plain fence, and the K3's cores keep a walk that found no page table
and use it until told otherwise: measured past a second and millions of retries. Each map
path now ends with
sfence.vmaon its hart. QEMU caches no translations and never shows this. - A tid is a number in its own process, as QNX numbers threads: the first thread is 1, others the lowest free, at most 256 at once. A thread of another process cannot be named. A new thread has its tid before any of its code runs.
ThreadDetachworks; numbered and served by nothing, it used to stop the machine.- Teardown: the last thread out only leaves its page table, marks the
address space and sends
PROC_END; taskman does its own work and then has the kernel tear the process down. The last thread cannot be preempted on its way out holding a sync bucket's lock. - A switch reads the outgoing thread's address space before publishing it as off its hart, and the incoming thread clears its predecessor's running flag only once that one is saved.
- Notices:
DISCONNECT,UNBLOCKandPROC_ENDeach travel in a node inside the object they are about, queued ahead of every pulse. One process ending with 250 open/dev/nulls used to overflow the ring of 8, leaving taskman a record of a process that no longer existed. - 63 processes can live at once, up from 31.
- Memory:
MAP_PHYSreaches devices only, and only for root; stacks end in guard pages;PROT_NONEwithholds access, where it used to be read access; a map over a mapped page is refused, not a panic. - A crash cannot loop: when the crash stub cannot run, the kernel ends the process and tells taskman.
- A connection can carry an identity, delivered unforgeably in place of the sender's: the mechanism a relay needs to act for the user it relays for. Nothing uses it yet.
- A condvar wait could lose a signal from another hart; boot page-table
changes now reach every hart's TLB;
ThreadCreate's round-robin skips the K3's auxiliary cores. - The kernel's copies of libc's ABI are gone: syscall numbers, structs, codes and flags come from one header each.
QSOE/L
- seL4 17.0.0, with patches 0001 to 0008 unchanged and 0009 gone;
make preparerefuses a clone of another release than the pin. - The SBI capability, the RISC-V twin of Arm's SMC capability, in initial
slot 15; a badge narrows it to one call. A kernel without it, or firmware that refuses,
now ends in a logged
EIO, and the machine keeps running.k3sel4carries the same capability. - A pulse wakes the thread that receives it. A pulse wakes only the thread
its channel's Notification is bound to, and that was the channel's creator, so a worker
already blocked in
MsgReceivewas never woken: 5 times in 5, once measured that way.MsgReceivenow moves the binding to the calling thread. - Taskman says which harts the machine has. The system map carries them, by
the same two gates as QSOE/N. The K3 reads
harts 0xfffd usable, 0xff00 auxiliary, and a thread nobody placed stays off the A100s. - A runmask names a hart, translated through the loader's map; it used to be read as a seL4 core index, off by one on the K3 from hart 2 up.
- The system map says where the PLIC is:
sysinfosaidPLIC @0x0, 0 source(s)on every PLIC board. - Addresses: the service block at 0x7FE00000; a stack region with nothing
mapped beneath it, so an overflow faults; a GiB for anonymous
mmap; taskman's root CNode at 216 slots; 128 frames of anonymous memory per process; a worker's stack ends in a guard page. MAP_PHYSis root's alone: seL4 keeps RAM out of its reach, but any process could map any device's registers.- 32 fixed-priority channels per process; a refused window names its
program;
ThreadDestroyof a thread holding the table lock isEBUSYrather than a hang.
Taskman and processes
- Taskman is never a client of another process, on either kernel. It used to
flush every filesystem and read
/etc/passwd,groupandshadowitself, from inside its dispatch, so a server that did not answer held every request in the system.TM_REQ_SYNCnow answers which servers to flush and libc sends to them; the credential files are read by the root process that asks for the reload and handed over. On QSOE/N, spawn's image reads, which drop the lock, are taskman's last sends. - QSOE/N's dispatch pool is back, three threads; it seemed to hang the K3 at boot, and the cause was a driver thread placed on an auxiliary core.
dupshares the open file description on QSOE/N, as POSIX says. A copy used to be a second open with its own offset, socmd >f 2>&1lost output.- 256 open-file contexts on QSOE/N, up from 32, and taskman hears when a process ends holding its files: a few dozen kills used to exhaust them.
- A process's end always reaches taskman:
PROC_ENDreplacesPROC_FAULT, the fault included, and a record still held as alive is closed as killed and named on the console. - A program says how much stack it needs:
ld -z stack-size=N, read fromPT_GNU_STACK; nothing, or less, gets 64 KiB, and more than 8 MiB is refused with a line saying so. - Two segments may share a page, mapped once with both segments' rights, on both kernels. A segment that overlaps the one before it is refused.
- A program has the first GiB to itself (0x10000 to 0x40000000). On QSOE/N the sysmap sat inside it, and the first 2 MiB program ran its data over it.
- A failed claim fails the spawn with
ENFILE, after taking back the claims already made; the child used to start holding a descriptor its server knew nothing about.
Who may do what
- A raw disk is root's: the NVMe and virtio nodes are 0600, not 0644.
- The path space is root's: registering a name, or pointing one at another
server, is
EPERMfor anyone else. - A copy of an open goes to the process that claimed it.
_IO_DUPcloned any connection's open for any caller who named a number. Now a client sends_IO_DUPPREPon the connection it means to copy, which only a holder can do, and the dup that follows is served only for that claim. It holds for taskman, for resource managers, for the pipe manager, and for the fleet relay, where a guessed number used to reach an open file on another machine. - A signal needs the right to send it: root, or the same real or saved user.
kill()reportsEPERMrather than rewriting every refusal asESRCH. - A pipe is its maker's. Any process could bind itself to any pipe by a sequential id.
- Memory grants: a region may be published to a client by its
client_id, or carry a token of 64 random bits, for a grant that belongs to a role rather than a process. - A driver's network ring goes to the client that attaches it. Measured before the change: an unrelated process mapped 397 KB of the live ring.
- The console's text plane goes to the program holding the screen, by token. A grant anyone could map let every process read a session's screen, a password echoed on it included, and write over it.
fs-tmpfsenforces its mode bits, write permission on the directory, and the sticky rule on/tmp./dev/sysactis root's (0600), and a chord is acted on only from a connection root opened.- The console and serial lines: injecting input needs an open root made; the foreground group and the window size belong to whoever holds the line.
- PCI configuration writes and MSI-X calls are root's; reading the bus is unchanged. The log can be erased only by root. GPIO and I2C honor the open: a pin change or a bus write needs one that asked to write.
Drivers and hardware
devu-xhcitakes MSI on PCI, the waydevb-nvmeanddeva-hdado: an IMSIC interrupt on an AIA board, a pulse frompci-serverwhere the host bridge owns the MSI block (the Unmatched, the VisionFive 2). A wake that finds the event ring empty readsUSBSTSand drains once more. Polling is the last resort, and says so./dev/usbandlsusb: one line per device, or the hub tree with-t, and each device's interfaces with-v; what claimed each device is the column that matters. The node is 0444, and deliberately not a way to reach a device.- USB MIDI 1.0: whole messages as plain bytes, every cable merged; a read
parks, and
poll()works. - CDC-ACM: one node per port, paired from the union functional descriptor rather than by position, set to 115200 8N1 with DTR and RTS raised. A read is posted and left queued, since a console says nothing for minutes and then a great deal.
- DirtyJTAG:
/dev/jtag0carries the probe's command stream; the driver knows no JTAG at all. One program holds it at a time.xhci_bulk()waits against a completion slot per device, so loading a bitstream does not fight the hotplug walk. - A USB class driver gives its buffer back: on QSOE/L each one was a whole 2 MiB frame, kept for the life of the process.
- An NVMe interrupt can arrive before its completion. On the K3's A100 cores,
4,249 of 48,000 wakes found the queue empty, and the device sends no second interrupt, so
the submitter blocked for good. A wake that finds nothing now reads the controller's status
register first: PCIe does not let that read pass the device's earlier writes.
devb-nvmealso asks the system map for its interrupt thread's hart. poll()works on the Unmatched's serial line: the console login's terminal-size query read until a byte came, andlogin:never appeared.- Eight consoles on the screen:
devc-hficonserves/dev/con1tocon8, Alt-F1 to Alt-F8 bring one to the front, and sysinit starts a getty on every one. Each has its own input, foreground group and full-screen claim. Shift-PgUp and Shift-PgDn look back three screens. - The keyboard driver spells a modified function or navigation key the way an xterm does, and qsh and the commander read such sequences to their end.
- Sound: checked by ear on the boards this cycle, as above.
grouvrandgrouvr-envoy: from a QSOE partition,grouvrtalks to the gRouvr hypervisor, and a request from another partition to reboot or power off this one goes tosysact, which flushes the filesystems first. On bare metal the envoy says so and exits.
Filesystems and the network
fs-qrvandfs-tmpfskeep an unlinked file while it is open, until the last close. The node used to be freed at once: on/tmpa reader saw a file of 4294967296493 bytes, and on the disk{ rm a; echo second > b; echo first; } > aleft “first” inb, for good. A volume halted with such a file open is not marked clean, so the next boot'sfscheckfrees it.- A rename over a name never leaves it missing: a power cut between the two steps used to lose both.
fscheckreads each inode block once: 8,327 reads became 355 on a 256 MB volume, 3.5 s became 0.19 s on the K3. Boot runsfscheck -c -p, which checks only when the volume says a check is due.- Files on another station:
rm,mv,chmod,chown,touch,truncate,ln -sandreadlinkwork under/net/<station>/. The QSP wire format gainedUNLINK,RENAME,READLINK,SETATTRandSYMLINK. A station that is not answering saysEHOSTDOWN; a rename across stations isEXDEV.
The C library
- The umask is applied. Taskman kept it and nothing took it off a mode. Now
open(O_CREAT)does. That changes what programs get: under the default 022 a new file is 0644, not 0666. sinf,cosf,expf,logf,powf,sqrtf: musl 1.2.5's, each within one ulp of its double counterpart, rounded; a program calling one used to fail to link.ttyname()andttyname_r(), declared until now and defined nowhere, on<qsoe/devnum.h>: a number for every node QSOE serves, Linux's wherever Linux fixes one.- malloc locks. It locked only once
libc.need_lockswas set, and nothing here set it, though every process has a second thread from the start. Two threads allocating at once corrupted the heap. It now always locks. - malloc takes memory a chunk at a time: 2 MiB on QSOE/L, 256 KiB on QSOE/N.
- Priorities:
sched_curpriorityinstruct sched_param; the mutex attributes' protocol and ceiling;qsoe_proc_prio_ceiling(). A mutex's protocol is in the high half ofsync_t.count, zero meaning inherit, so a statically initialized mutex inherits without anyone having said so. - A spawn that asks for no priority starts the child at the spawning thread's own, as POSIX and QNX have it.
<qsoe/addrspace.h>; a static assertion checks that no two regions overlap. The sysmap page moved to 0x7FE01000, so a program that reads it is rebuilt with these headers.environ: a program using it saw an empty vector, as libc kept the environment under an alias the loader does not redirect.qjmcccould find neither its assembler nor its linker.sigaction()onSIGABRTended the process with a jump to address 0: a mask copied over a buffer sized for musl's raw syscall, 120 bytes over the return address.- Answers that say what they do:
mmapof a file fails withENOSYSand says so once, where QSOE/L handed back zeroed memory and QSOE/N refused without a word; a fault the kernel ends reads as the same signal as one the process ends; a refused system call reads asSIGSYS. gettid()in<process.h>, as on QNX 8;qsoe_conn_set_cred();<qsoe/usb.h>;<qsoe/klog.h>;<qsoe/skimmer_sysno.h>as the one place QSOE/N's call numbers are written, where four copies had already drifted into a stack overwrite.qsoe_alloc_physsays how to give the memory back:munmap, and the header now says what each kernel does with it.- The new calls join
qsoe-abi0at minor 1.
Shell and utilities
- New:
printf,on(on -p <priority> command, for looking at a system something higher is starving),lsusb,runtests. uptimeprints the time, how long the system has been up and how busy its harts have been since.ps -w Nwaits N seconds before its snapshot, so that started high before a load it looks at a system too busy to start a newps.ls -lshows the modification time, in POSIX's two forms.- qsh:
!history expansion, with word designators and modifiers;~user; completion of a word with a~in it. moreandlessare links toqview.sloginfoopens with the kernel's boot, under the kernel's name.gettyno longer spins when its line refuses a read: a second getty on a line that has one printed its banner forever. Its banner ends with the line it serves.emu.shboots what is there. It builds nothing now; it used to re-stage/usron every start and quietly replace what the last run wrote.
Tests
- One line per case, a failed step in capitals with its sentence under the
line, and a list of what failed after the results.
suite -vprints one line per check, as before. - GAP, not FAIL (see Highlights).
- Priority inheritance, checked: a server at its client's priority; a mutex
owner at its waiter's, along a chain and against hogs, with a
PRIO_NONEcontrol that must show the inversion; when a message's loan ends; a busy channel's servers raised; pulses that lend;PRIO_PROTECT; a mutex left by an ended owner; the process ceiling. What QSOE/L passes is required of it; what it cannot do stays in the[QSOE/N only]tail, with the reason in each file's header. [mutex-loan]measures the inversion it claims to. On QSOE/L every hog had stopped before the span began, and the control passed for the wrong reason. Hogs now start at their priority and say when they spin, and the span starts once they all do. A gate keeps the high thread from holding a hart against them, which failed the case on every other pass on the Unmatched; it then ran nineteen times in a row.- New groups and cases:
[dev](no two devices share a number, a raw disk refused to an ordinary user, the umask,ttyname());[msgprio];math, over subnormals and arguments past 109;[dup] sharedandclaim;[thread] tids;[smp] pulse-worker;[crash] sigabrt;[mem] recycle;[QSOE/N only] bad-syscall; stack overflows; a hundred spawns killed while holding open files. - The suite runs as anyone, skipping what a user's ceiling forbids, and Ctrl-C ends it.
- Flakes removed: spawn cases no longer race the child they end;
[signals] killwaits a time rather than a hundred yields;[smp]cases take their harts from the system map.
Packages and the boot archive
qsoe-boot:/qsoe-boot.pkgat the archive's root records every file with its digest and every link with its target. It describes the archive that booted;install,removeandupgraderefuse it, as it changes only withqsoe-base, at the next boot.qupkg owner <path>...follows every link on the way and names each package that records the path or the file it leads to; a path nobody records exits with status 1.- Source packages:
qupkg sourcechecks a source archive as an install checks a package — signature, size, digest — and unpacks it into/usr/src/, running no script;qupkg build-depinstalls what building it needs. - Files are 0644, executables 0755, whatever the umask. The store, the database and a source tree used to be anyone's to rewrite.
- A commit cut off at its rename is finished, not lost: with
activemissing andactive.newnaming a generation, the next command completes it. The next install used to build from an empty set. - The tests are a package, in the repository
/etc/repositoriesnames; the image no longer carries them.
Bugs worth naming
Found in this cycle, each with what it taught.
- A worst case of 96 µs with nothing in QSOE slow. The stall was inside the kernel's own idea of how often to look up. A latency figure with a load beside it finds what an idle one cannot: 11 µs idle said nothing about it.
- One more server stopped the boot. Starting the timer server moved the NVMe interrupt thread from hart 7 to hart 8, the first A100, where an MSI can outrun its completion. A hart count nobody published looked exactly like a single-core machine.
- A timer that could not be told anything. A gate chosen by the timer id read back exactly as asked, and the only thing that showed it was printing the register's value instead of a verdict.
- “crit_exit underflow” after five hours of crash loops: a thread published as off its hart before its registers were stored, so the store went into a new thread's block, which later resumed inside another's exit.
- A shell capturing command output stopped every spawn after it. A
reallocthat grew across 2 MiB took its frame from taskman's master pool, whose cap the process's exit could not reach, and the next spawn found its slot occupied. Twenty lines of C and one run reproduce it. - A flake of 1 in 20 that was a certainty. A pulse to a blocked worker was never delivered; it only looked rare because a worker arriving later found it without a wake.
- A test measuring nothing. Sixteen hogs had finished before the span began,
and the control passed. The same rule as
test_rtload's: prove the pressure was there. - A hole hidden by a frame. The audit of
/devlooked at the servers under/dev, and taskman is not one of them, so the larger half of the dup hole sat outside it. - A question asked wrongly. The first fix asked a dup “did you open this?”, which POSIX answers no for every inherited descriptor, and shell redirections began to fail. A claim on the connection is the right question.
Open every section · the same ground is covered in the
per-component CHANGELOG.md files in the
source repositories.
Known gaps
Stated rather than implied away. These are the things 0.5 does not do, and the things it cannot yet explain.
Causes not yet known
- Process creation is about 4 ms slower while the timer server runs in its measuring mode: 13.9 to 17.9 ms, and the suite 12 s longer, on the same kernel, taskman and thread placement, with sleeps and interrupts unchanged. Cause unknown. So QSOE/L's interrupt figures come from a system that is not quite the shipped one.
- A tail on QSOE/N. One run under a parallel compiler build on the K3 had about one interrupt in a thousand at 230 to 310 µs. It did not come back in the runs after it, and its cause is not known. A worst case is only as good as the longest run behind it.
- About 20 µs without a preemption point remains on QSOE/L during process creation, at 10 work units. It has not been found.
- One 2 ms sleep in 200 was 3.1 ms late on the VisionFive 2 under QSOE/L.
devt-platstarted by hand on QSOE/N froze the Unmatched twice. QSOE/N does not need it, and init does not start it there. The same experiment has not been run on the K3.- One record on QSOE/N outlived its process after a night of crash loops, with no lost notice behind it. Taskman now closes any such record and names it, but why that one happened is not known.
- Two log lines nobody has explained yet:
aux: read -41on QSOE/N, and a null capability #73 on QSOE/L.
Not measured
- The K3 under QSOE/L. Its last figures are October's, on seL4 16.0.0 with 100 work units: 252 µs at worst under the full load and 9.4 ms under a spawn storm. They were not taken again on 17.0.0 with 10 work units. A spawn storm on QSOE/L has no figure for the shipped kernel on any board: the 17.9 µs on this page is under one process a second, not a storm.
Real time and priorities
- The spread between the kernels is a trade, not a verdict: QSOE/L's periodic task parks between releases and gives its hart back to the system, which costs its wake some tens of microseconds.
- QSOE/L's inheritance stops short of QSOE/N's. A server waits for its next message at its own priority, so it is raised only once it has run to take a higher client's request. A busy server is not raised by a higher client or pulse waiting behind it. seL4 tells a server nothing when its client leaves. Pulses come first in, first out. A mutex chain stops at the process boundary.
- On QSOE/N a kernel copy cannot be preempted. A system call runs with
interrupts off, so a message's copy holds its hart for as long as the copy takes. On
the VisionFive 2,
cyclictestat priority 200 saw 18 to 20 ms beside a 4 MB message, one stall each way; about 5 ms beside a console line from taskman, when it landed on the same hart; and about 2 ms in rounds of starting and killing a process. The same run beside a parallel compile, six jobs for 96 seconds, stayed at 16 µs. So the 49 µs above is a spawn storm's figure, not a bound on the system. Bounded copying and zeroing are the plan for 0.6. - On QSOE/N, page zeroing and one large reply per spawn set what is left of the spawn storm's worst case, near 50 µs.
- QSOE/L still spins in places: below 50 µs, and whenever no timer can be reached, a sleep is a spin that yields only to its equals.
- No timed waits on either kernel:
pthread_cond_timedwait,TimerTimeoutfor a condvar, a boundedInterruptWait. - A signal cannot interrupt a blocked call. On QSOE/L a thread holding libc's
table lock cannot be destroyed either:
ThreadDestroyanswersEBUSY. pthread_mutex_getprioceilingand_setprioceilingare stubs that announce themselves.
QSOE/L
- A client REPLY-blocked on a server that exits is never released.
- A process's main thread cannot be joined.
- Spawning a second process before init runs is not handled.
- With several threads receiving on one channel, a pulse wakes the latest to come in, where QNX would wake the highest-priority waiter.
- CPU load is not counted, and
uptimesays so.
Everywhere else
- Across the net, a station acts as itself. The far station performs a request with its own credentials, which are root's; the ids the request carries are not consulted. QSOE/N has the mechanism a relay needs, and nothing uses it yet.
/dev/net/en0is 0666: any user can send raw frames.fstat()of a pipe end isENOSYS, and of a closed descriptor on QSOE/NEINVALrather thanEBADF.isatty()testsS_ISCHRalone: it is true for/dev/null, and sets noENOTTY. Serial drivers reportst_dev4.mmapof a file fails withENOSYS.- Math: the rest of the float set, and the doubles
asin,acos,sinh,cosh,tanh,log10,hypot,cbrt,fmin,fmaxandfma. Subnormals on the K3 and on U74 cores may hit a slow path. - A USB device unplugged from a root port is never released: the scan walks hub ports only, so a class driver goes on offering a device that is not there. No class driver gives back a transfer ring.
qmkexits with status 2 on Ctrl-C rather than ending by the signal.
Getting it
Pre-built images are published at github.com/qsoe-dev/dl; the front page lists what each file is and how to run it under QEMU or install it on a board. Under QEMU, name the variant — it selects the machine — and boot through the distribution's edk2:
# unpack the images (virtio.img is QSOE/L's root under QEMU) gunzip nvme.img.gz virtio.img.gz # QSOE/N; for QSOE/L say l, and pick the matching entry in mr-bml's menu NVME_IMG=./nvme.img VIRTIO_IMG=./virtio.img ./run-nvme.sh n --uefi
To build from source:
git clone https://gitlab.com/qsoe/os cd os make prepare # fetch the 0.5 component set (see component.list) make all # build both variants: QSOE/N, then QSOE/L make nvme # the disk image, its system volume an installed qsoe-base make dist # optional: the QEMU disk images
The manuals — Design.pdf, UserGuide.pdf,
ProgrammingBook.pdf, AppPortingGuide.pdf,
Networking.pdf and LibcReference.pdf — are published
alongside each release at github.com/qsoe-dev/doc.
Documentation is a release gate: no version ships before the manuals are brought up to the
tree.
Next is 0.6 — self-hosting: qjmcc on the target
rebuilds QSOE from source on its own filesystem, and a build started on the board produces
binaries that boot the board. Each board's disk becomes two volumes as well: a system volume
that can always be recreated, and a data volume that holds what the machine has
accumulated.