comp.os.psos

Long-form reference · 9,737 words · about a 42-minute read

Where comp.os.psos sat

The second level of a comp.* name is the department, and comp.os.* was the department for operating systems. Most of its rooms were named for systems with a general public — the comp.os.linux.* branch, comp.os.ms-windows.*, comp.os.os2.*, the DOS groups. A smaller shelf held the systems that ran inside equipment rather than on desks, and that is where comp.os.psos sits. Read the Internet Systems Consortium's newsgroups file today and the neighbours are all still listed: comp.os.vxworks, described as The VxWorks real-time operating system.; comp.os.qnx, Using and developing under the QNX operating system.; comp.os.lynx, Discussion of LynxOS and Lynx Real-Time Systems.; comp.os.os9, Microware's OS-9 real-time operating system. Beside them stand the two general rooms, comp.realtime — Issues related to real-time computing. — and comp.arch.embedded — Embedded computer systems topics.

Notice how narrow the shelf is. comp.arch.embedded is a field and comp.realtime is a discipline, but comp.os.psos, like comp.os.vxworks and comp.os.lynx beside it, is one commercial product, sold by one company — in this case a company in Sunnyvale, California — to engineers who had already committed a product line to it. Groups that narrow exist on Usenet in only two circumstances: either the subject has a very large amateur following, or it has a professional following large enough to make a room worth the trouble. This is the second kind. For the shape of the hierarchy it belongs to, see the comp.* hierarchy page.

The ISC newsgroups file gives each group one line: the name, a tab, and a single sentence. The line for this one is the same in the list issued in 2026 as it was in the control message that created the group in December 1998, and as it was in the list issued in 2002. It reads:

The realtime operating system pSOS+.

Two small things about that sentence are worth recording, because recording small things is what a directory page is for. Realtime is unhyphenated, and there is a full stop at the end. Neither Request for Discussion wrote it that way: both proposed The real-time operating system pSOS+, hyphenated and unpunctuated. The normalised form first appears in the Call for Votes of December 1998 and then in everything downstream of it — the result, the creating control message, and every newsgroups file since. Which hand made the change, and on whose instruction, the surviving documents do not say, and this page will not guess; what they show is where in the paperwork the change happened. One further honesty is owed: the sentence at the head of this page is not the ISC line but the directory's own paraphrase of it, and the two are recorded here side by side rather than quietly reconciled.

A Request for Discussion, May 1998

comp.os.* is part of the Big Eight, which means the group could not simply be willed into existence; it had to be proposed, discussed, voted on and then created by a control message. The whole of that paperwork survives in the Internet Systems Consortium's mirror of the news.announce.newgroups archive, in a single file of some seventy-six kilobytes holding six postings with their original headers: two Requests for Discussion, two Calls for Votes, and the result, which appears twice.

The first Request for Discussion was posted on 19 May 1998 by Liam Friel, writing from an Irish address, to the moderated group news.announce.newgroups with follow-ups directed to news.groups. It was cross-posted to comp.realtime, comp.os.misc and comp.arch.embedded — which is to say, to the three places where the traffic the new group wanted was already happening.

The rationale is short and unusually concrete about its evidence:

The pSOS operating system from Integrated Systems Inc is a real-time operating system, suitable for embedded systems. pSOS is very widely used in industry. A quick scan of the "help wanted" announcements in the various newsgroups shows several hundred current requests for pSOS developers.

That is a proposal arguing its case from the job adverts, which is as good a proxy for the size of a professional community as anyone had in 1998. It goes on to note that pSOS discussion was appearing in comp.realtime and comp.arch.embedded, and occasionally in the groups belonging to rival systems — comp.os.vxworks is named — with no room of its own anywhere.

The charter that follows is a plain statement of a technical group's business. It opens by declaring the group open to discussion of all technical aspects of the Integrated Systems real-time operating system and related products such as compilers, debuggers and integrated development environments, and then enumerates seven purposes:

• technical issues related to the pSOS operating system and its components;
• technical issues related to developing application software for pSOS;
• technical issues related to developing device driver software for pSOS;
• technical issues related to porting pSOS to new hardware;
• technical issues related to development tools for pSOS, and the pRISM+ development environment;
• issues related to older versions of development tools and environments;
• general questions and requests for information on pSOS and pSOS-related issues.

The sixth of those is the tell. A charter clause reserving space for questions about obsolete tool versions is written by somebody who knows that his readers are maintaining products shipped three years ago with a compiler two releases out of date, and who does not want them told to upgrade and go away.

Three things were excluded: advertising of products or services, recruitment announcements, and binary attachments, with authors told to put the material on an FTP or web site and post a pointer instead.

A second Request for Discussion followed on 30 October 1998, five months later, revised in small ways that are legible as the results of the intervening discussion. A second proponent, Evandro Menezes, had joined. The binary ban had softened from attachments of any kind to large attachments, with a line added that PGP signatures in posts were acceptable. And a new paragraph had appeared in the rationale:

There is a pSOS mailing list maintained by ISI for discussion of pSOS issues. This has suffered in the past from system downtime at ISI. A newsgroup would be a more convenient and reliable forum for the exchange of pSOS information.

That is the argument for Usenet compressed into three sentences, and it is not the argument one might expect. The complaint is not that the vendor was censoring its mailing list, or charging for it, or reading it. The complaint is that the vendor's server kept falling over. A newsgroup was distributed infrastructure that no single company had to keep running — and the distribution list at the foot of that second Request for Discussion names, alongside the five newsgroups, the vendor's own pSOS mailing list, subscription address at isi.com included. The proposal to replace the list was circulated on the list.

The vote of December 1998

The first Call for Votes went out on 4 December 1998, taken by Dave Cornejo under the banner of the Usenet Volunteer Votetakers — the standing pool of neutrals who ran Big-Eight ballots — with votes to be mailed to an address at his own domain and the poll closing at 23:59:59 UTC on 25 December. A last Call for Votes, superseding the first, was posted on 14 December. The general machinery of Requests for Discussion, Calls for Votes and waiting periods is set out on the soc.* hierarchy page; what follows is how it ran here.

Voting was by electronic mail and by hand. A ballot had to contain exactly one of two sentences — I vote YES on comp.os.psos or I vote NO on comp.os.psos — and, if the voter's mail software did not carry a real name, a separate line beginning Voter name:. The rules were strict and slightly weary in tone: one vote per person and one account per voter; votes mailed directly from the voter, with anonymous, forwarded and proxy votes invalid; ballots submitted through web forms treated as anonymous and therefore void; votes from non-existent addresses invalid; counting automated, so a malformed ballot simply would not register. Voters were warned that a signature line was not a name and told to chase an acknowledgement if none arrived within three days.

The Call for Votes also states, in a single sentence, the theory of the whole exercise: the purpose of a Usenet vote is to determine the genuine interest of persons who would read the proposed newsgroup. Soliciting votes from the disinterested, it adds, defeats that purpose — which is why distributing pre-marked copies of a ballot counted as fraud.

The result was posted on 26 December 1998 under the subject line RESULT: comp.os.psos passes 289:13. There were 289 votes in favour and 13 against, 302 valid votes in all, with one abstention and eight invalid ballots. Both standing thresholds were cleared without argument: yes votes had to be at least two-thirds of the valid total, and had to exceed no votes by at least a hundred. Here they were ninety-six per cent of the total and ahead by 276.

A five-day period followed in which irregularities could be alleged. Nothing in the archive records an allegation, and nothing stopped the creation: on 31 December 1998 at 15:30 UTC a PGP-signed newgroup control message went out from the group-admin address at isc.org, over the name David C. Lawrence and carrying the news.announce.newgroups moderation approval, telling the world's news servers the group existed. It was sent again six hours later, then on 1 January, on 7 January and finally on 31 January 1999 — the routine belt and braces of a network on which not every server heard the first announcement, and on which every administrator decided individually whether to honour it.

The control archive holds those five messages and nothing else. There is no rmgroup: comp.os.psos was never removed by control message, and the ISC's current active file still lists it with the flag y, meaning an unmoderated group open for posting. A name in a namespace file is not evidence of traffic, but it is evidence that nobody ever formally took the room away.

Who voted, and what the roll shows

Big-Eight rules required the vote-taker to publish the entire roll — every address, every name, every vote — so that the count could be checked by anyone who cared to. For a group with no surviving traffic archive of its own, that roll is by some distance the most informative document that exists about who this group was for.

One employer dominates it. Of the 289 ballots in the yes column, fifty-nine carry a Philips address, spread across some seventeen distinct Philips subdomains: Eindhoven addresses and the Natlab research laboratory in the Netherlands, and business units at Southampton, Caen, Hamburg, Bangalore and in Australia among others. A further twenty-eight came from the vendor itself, at isi.com. Seventeen came from Data Respons, the Norwegian embedded-systems company founded in 1986. Twelve came from Iskratel, the telecommunications equipment maker at Kranj in Slovenia. The rest are scattered across semiconductor companies, instrument makers, defence and avionics suppliers, telecommunications equipment vendors and a long tail of one-address consultancies. By country of top-level domain the roll runs 164 .com, 35 Norwegian, 12 Slovenian, 11 .net, 9 Swedish, 8 German, and single figures for a dozen more.

What the roll is not is academic. Seven addresses in the entire yes column end in .edu, and one more belongs to a Hong Kong university: eight academic addresses in 289. The contrast with a comparable vote of the period is sharp and can be counted. The ballot that created comp.sys.arm four years earlier, and whose result was posted on 1 December 1994, passed 349 to 19; of the 349 yes votes, 203 came from United Kingdom addresses, and 107 of those 203 were .ac.uk — university ones — with a further sixteen .edu addresses elsewhere in the roll. Rather less than half of that ballot was academic, but rather less than half is still a world away from eight in 289. comp.os.psos was voted into existence almost entirely by people at work, on company time, using company mail, because the product was on their desks.

The thirteen no votes are recorded with names and addresses like everything else, and the record does not say why any of them voted as they did; the ballot had no space for a reason, and the discussion that might have explained them ran in news.groups, which the news.announce.newgroups archive does not carry. There was one abstention, which the rules recorded but did not count.

The eight invalid ballots are a small museum of 1998 electronic mail, and they are worth listing exactly because the roll lists the reason beside each one. Six were rejected for containing no vote statement at all — the sender had replied to the Call for Votes without the required sentence — and every one of those six came from a Philips address, which says something about how the ballot was passed around inside one company. One was rejected as an invalid address: the roll prints it as an at-sign and a domain with nothing before it. And one arrived as an X.400 address, complete with personal-name, organisational-unit, private-management-domain and administrative-management-domain fields and a country code of FI, routed out of a Finnish broadcaster's gateway; it was ruled ineligible. Usenet's voting rules assumed Internet mail, and in 1998 that assumption still excluded people.

What a real-time operating system is

The phrase misleads almost everybody on first contact, so it is worth being exact. A real-time operating system is not a fast operating system. What it sells is not speed but determinism: a bounded, knowable worst case. A real-time computation is one whose correctness depends not only on producing the right answer but on producing it within a defined time of the event that demanded it. A supercomputer grinding through a climate model is not doing real-time work however quickly it finishes. An anti-lock braking controller is, and once its timing is met, further speed buys it nothing at all.

Deadlines are classified by what happens when one is missed. A hard deadline is one whose breach is a total system failure — engine management, a cardiac pacemaker, a flight control surface. A firm deadline is one where the result is worthless once late, but where occasional lateness can be absorbed by some other mechanism: an assembly line that ejects the part and carries on. A soft deadline is one where lateness merely degrades quality — a video frame arriving late, an audio buffer under-running. The distinction matters commercially as well as intellectually, because a vendor selling into hard-deadline markets has to be able to state numbers and defend them, and a vendor selling into soft ones does not.

The scheduling model that made hard deadlines tractable on small hardware is preemptive priority scheduling, and it is deliberately unfair. Every task is given a fixed priority; the highest-priority runnable task runs; the instant a higher-priority task becomes runnable — because an interrupt woke it, or a message arrived, or a timer expired — the kernel takes the processor away from whatever is running and gives it to the newcomer. A time-sharing system does the opposite, sharing the processor out to keep everyone tolerably served. A real-time kernel starves the low-priority work without apology, because starving it is the point: it is how the urgent work is guaranteed its processor.

Fixed priorities can be assigned analytically rather than by intuition. Rate-monotonic scheduling, from Liu and Layland's 1973 paper in the Journal of the ACM, assigns higher priorities to tasks that run more often and yields a schedulability test based on processor utilisation; earliest-deadline-first schedules dynamically by whichever deadline is nearest and, ignoring switching overhead, copes with any load up to full utilisation. Neither is common in general-purpose systems, because both require the designer to supply something a general-purpose system never knows: a worst-case bound on how long each piece of work takes.

The number engineers actually argued about was interrupt latency: the interval between a device asserting an interrupt line and the first instruction of the handler executing, plus, for work that must be done in a task rather than a handler, the further interval before the woken task is scheduled. Latency is not zero because kernels must sometimes make themselves uninterruptible — while manipulating a ready queue, for instance — and every such critical section adds to the worst case. Hence the whole design discipline of a real-time kernel: keep the sections in which interrupts are disabled as short as they can be made, and publish the worst one. Integrated Systems published, for pSOS+, a context switch of under ten microseconds. Figures like that were quoted for a particular processor at a particular clock rate, and they were marketing as much as measurement; the useful thing about them was that a competitor could try to beat them.

A kernel of this generation was correspondingly small, and it was small by omitting things. It shipped as a library that the application linked against, so that the finished program was one binary with the operating system inside it. There was typically no separation between user and kernel modes, no process isolation, and no demand paging — paging in particular being intolerable, since a page fault turns a bounded operation into an unbounded one. Everything shared one address space, which meant a wild pointer in a low-priority task could quietly corrupt a high-priority one. That trade, determinism and size in exchange for protection, is the defining bargain of the classical embedded kernel, and the two decades that followed were largely spent unwinding it: memory protection units, partitioned architectures, and certification regimes that require the partitions to be proved.

Priority inversion, and four resets on Mars

The classic way for a correctly written priority-scheduled system to fail is priority inversion, and it deserves setting out slowly because every part of it looks reasonable in isolation.

Take three tasks — call them high, medium and low — and one resource guarded by a mutual-exclusion lock, which the high and low tasks both need. The low task takes the lock. The high task then wants it, cannot have it, and blocks; so far this is ordinary, and a well-written low task will release the lock promptly. But now the medium task, which does not want the resource at all, becomes runnable. Being higher in priority than the low task, it preempts it. The low task therefore cannot finish and cannot release the lock. The high task remains blocked — not by the medium task, which never touched the resource, and not really by the low task, which is not running either, but by the interaction of the two. The priority ordering has been inverted: the highest-priority task in the system is waiting on the lowest, for as long as the middle of the system cares to run.

There are three standard answers. Priority inheritance temporarily promotes the task holding the lock to the priority of the highest-priority task waiting for it, so that the medium task can no longer preempt it; the promotion lapses when the lock is released. The priority ceiling protocol gives the lock itself a priority — the highest of any task that will ever take it — and raises whoever holds it to that ceiling immediately. The blunt third answer, common in the simplest embedded systems, is to disable interrupts around the critical section, which collapses the whole scheduling problem into two priorities and makes inversion impossible at the cost of latency for everything else. Each answer costs something at run time, which is why kernels of the 1990s shipped inheritance as an option to be enabled rather than as the default.

The canonical demonstration of what that default costs is the Mars Pathfinder lander, which touched down in Ares Vallis on 4 July 1997. It is worth stating plainly that Pathfinder did not run pSOS: its computer was a radiation-hardened IBM RAD6000 running VxWorks, the product of the company that would buy pSOS's owner less than three years later. The incident belongs on this page because it is the reference case that every RTOS discussion of the following decade — this newsgroup's included — had in its background.

The mechanism was documented on 15 December 1997 by Glenn Reeves, who led the Pathfinder flight software team, in an account written to correct the versions then circulating. The spacecraft's 1553 data bus ran on a repeating eight-hertz cycle driven by two tasks: a bus scheduler, the highest-priority task in the system apart from one internal VxWorks task, and a data distribution task, the third highest, behind a task controlling entry and landing. A meteorology task, very low priority, received its data through an inter-process communication mechanism built on pipes. That mechanism used the select facility, and select protects its list of file descriptors with a mutual-exclusion semaphore. The meteorology task was preempted while in the act of releasing that semaphore. Medium-priority tasks ran. The data distribution task then tried to send it new data, blocked on the same semaphore, and did not finish its cycle. The bus scheduler woke, found that the distribution task had missed what was a hard deadline in the design, declared the error — and the spacecraft's response to that error was to reset the computer.

It happened four times: on 5, 10, 11 and 14 July 1997. No collected data was lost, but each reset terminated the ground-commanded activities in progress and cost the remainder of that day. The team reproduced the fault in a laboratory duplicate in under eighteen hours, using a trace and logging facility that had been left in the flight software deliberately — Reeves is emphatic that it was not left in by luck, and cites the team's philosophy of testing what you fly and flying what you test. The fix, uploaded on 21 July, was to set a global configuration variable so that the select library created its semaphores with priority inheritance enabled. Wind River had left that option out of the default for optimum performance, and Reeves notes that the variable itself was undocumented: you would know about it only from the source.

What makes the episode canonical is not that anyone was careless. The kernel behaved as designed, the option existed, and the fault had been seen before launch and triaged as a glitch that appeared only under load nobody had anticipated — pre-launch testing had used best-case data rates, and the surface returned better than best case. It is the standing demonstration that in a real-time system the interactions between correct components are where the failures live, and it is the reason that modern real-time documentation discusses inheritance protocols in its opening chapters rather than in an appendix.

Software Components Group, 1982

pSOS was created in about 1982 by Alfred Chao and developed by his own company, Software Components Group, a California corporation. The name is generally expanded as Portable Software On Silicon. It was written in Motorola 68000 assembly language and optimised hard from the beginning, and through the 1980s it became a default choice for embedded designs built on that processor family.

The 68000 was the right horse. Motorola's own engineers had defined a standard bus for 68000 systems in 1979 — VERSAbus, whose 41-page description was published that November, and out of which the Eurocard-format VMEbus grew — and the board industry that followed made the 68000 the standard 32-bit part for rack-mounted embedded design for a decade. A kernel hand-written in its assembly language could be very small and very quick, and smallness and quickness were the two things a board designer could measure.

pSOS was modular earlier than most, which is the characteristic that survived every later rewrite. Debugging that understood the operating system's own objects, device drivers that plugged in rather than being compiled in, a TCP/IP stack, language libraries and disk subsystems were all available as separable pieces; source-level debugging and multiprocessing support came later.

The company's most durable contribution, though, was not a product. The RTEMS project — the open-source real-time executive begun in the late 1980s for the United States Army Missile Command — recorded the standards lineage in its C User's Guide in three sentences that are worth having exactly. The Real Time Executive Interface Definition, RTEID, was developed by Motorola with technical input from Software Components Group. RTEID was adopted by the VMEbus International Trade Association as a baseline draft for its own proposed standard multiprocessor real-time executive interface, the Open Real-Time Kernel Interface Definition or ORKID. And the two groups then worked with the IEEE's P1003.4 committee to see that the functionality of their proposals was adopted as the real-time extensions to POSIX. The application programming interface of a small Californian company's 68000 kernel is therefore an ancestor of an IEEE standard, and RTEMS still ships a Classic API descended from RTEID today.

Integrated Systems, 1991 to 1999

Integrated Systems Inc. was founded in California in 1980 or 1981 by Narendra K. Gupta, known as Naren Gupta, and its own filings are more precise than the round numbers usually quoted for him: he was president from the company's formation in 1980 until May 1994, chief executive from 1987 until May 1994, secretary from September 1989, and chairman of the board from March 1993 until the company was sold. Summit Partners invested in 1987; the company listed on Nasdaq in 1990 under the ticker INTS. Its founding product was not an operating system at all but MATRIXx, a control-system design and simulation environment introduced in 1983 — which is why the company that ended up owning pSOS came to the embedded market from the modelling end rather than the kernel end.

The purchase of Software Components Group is recorded in the exhibit list of ISI's own annual reports: an Agreement and Plan of Reorganization among Integrated Systems, Software Components Group, Inc. and Alfred Chao dated 9 August 1991, an Agreement of Merger dated 20 August 1991, and a report to the Securities and Exchange Commission filed on 3 September 1991. The price has been reported as about twenty million dollars.

What ISI then did with it defined the product for the rest of its life. The kernel was renamed pSOS+ and most of it was rewritten in C, which is what allowed it off the 68000 and onto everything else. A multiprocessing variant, pSOS+m, was built on the same interface. The optional components were expanded and given the same naming convention. And a development environment was wrapped around the whole thing, arriving in its final form as pRISM+ — the name that appears twice in the newsgroup's charter, because by 1998 you did not buy a kernel, you bought an environment.

ISI bought steadily through the decade. It acquired FlexOS — the modular real-time multitasking system descended from Digital Research's Concurrent DOS line — from Novell in July 1994 for a reported three million dollars, and offered it alongside pSOSystem, the one aimed at deeply embedded markets and the other at point of sale. It took TakeFive Software GmbH under a stock exchange agreement dated 31 October 1995, bringing in the SNiFF+ source-comprehension environment. Diab Data, a compiler house that had been in the business since 1986, joined the same year as an independent operating subsidiary and supplied the optimising C and C++ cross-compilers. And for the automotive market ISI built pOSEK, described in its own filings as a customisable, standards-based real-time operating system with a smaller footprint than the proprietary kernels the industry had been using.

The scale claims are the vendor's own and should be read as such. The company's product pages in 1998 claimed over 4,500 design wins. Its annual report filed in May 1999 claimed more than thirty-eight million installations worldwide. A Wind River page for pSOSystem 3 still on the web in October 2005 put it at more than forty million customer devices. Nobody audited any of these numbers, and the interesting thing about them is not their precision but their shape: a product whose installed base was counted in devices shipped, not in seats sold, and which therefore kept growing after the last release.

What pSOSystem actually contained

The vocabulary is worth separating, because the charter uses all of it. pSOS+ was the kernel. pSOSystem was the licensed product: the kernel plus the optional components plus the board support. pRISM+ was the development environment on the host. A reader who has only ever met the name pSOS has met the middle third of a product family.

The vendor's own description of the kernel is admirably terse. pSOS+, its 1998 product page said, is small at seventeen to forty kilobytes, fast at under ten microseconds for a context switch, fully pre-emptible, re-entrant and deterministic. Its view of an application had exactly three kinds of thing in it: tasks, input-output device drivers, and interrupt service routines. Its services were task management, semaphores, events, timers, fixed and variable length queues, and asynchronous signals. Scheduling was priority-based, with time-based and pre-emptive scheduling specified per task and changeable while the system ran. Objects could be created, altered and destroyed at run time, and tasks could be loaded into a running target and added to or removed from the schedule. The company's annual report, describing the same family a year later, put the floor lower still: as little as sixteen kilobytes of storage.

One design decision deserves emphasis because it is the one that made the porting business possible: every hardware-specific element — processor initialisation, device drivers — was kept outside the kernel. The vendor's stated pay-off was an unchanging kernel image, with interrupt and exception handling faster, more deterministic and under the developer's control rather than the operating system's. It is also the reason a board support package could be a separable, purchasable, arguable-about thing.

pSOS+m, the multiprocessor implementation, shared the single-processor interface so that an application could be spread across nodes without being rewritten. It was fully heterogeneous, allowing different processor families in one application, and agnostic about how the nodes were joined — a memory bus, serial or parallel links, or something proprietary. It ran a master-slave architecture, tolerated faults, and supported hot-swapping slaves, which tells you what it was sold for: telecommunications shelves where cards are pulled and replaced while the system runs.

At the other end of the range sat pSOSelect, a modular cut-down kernel for the Motorola 683xx microcontroller family including the MC68360 QUICC, keeping the pSOS+ interface but configurable down to under 1.8 kilobytes of ROM and 320 bytes of RAM. The marketing phrase was choose what you use, and the number worth holding on to is the 320 bytes.

Around the kernel sat the optional components, all sharing the vendor's lower-case-p convention:

• pNA+ — the embedded TCP/IP stack, organised in four layers: a socket layer following 4.3 BSD syntax and semantics, a transport layer of TCP and UDP, an IP layer doing routing and fragmentation, and a hardware-dependent network interface layer supplied separately. ARP, ICMP and IGMP sat alongside. Its two selling points were a zero-copy buffer scheme and support for unnumbered point-to-point links, the latter to save subnet addresses in devices with many links.
• pHILE+ — the file system manager, handling four formats: ISI's own hierarchical Unix-style format tuned for real-time access, MS-DOS floppies and hard disks, ISO 9660 CD-ROM, and NFS as client and server.
• pREPC+ — a fully re-entrant ANSI-compliant C library of more than eighty-five commonly used functions, with a C++ class library beside it to bind C++ code to the kernel's services.
• pRPC+ — Sun-compatible remote procedure call and external data representation services, so that an embedded device and a Unix host could call each other's procedures.
• pROBE+ — the debug and target agent, described in its own section below.
• The I/O supervisor — a device-independent driver interface around which the kernel defined a standard set of six I/O services a driver might support, intended to keep application code portable across peripherals and, in the vendor's phrase, protected from I/O device obsolescence.
• SCSI support, board support packages, a graphics component and DISI — also on the 1998 component list.

The networking package went further than a stack. Its Internet applications library offered re-entrant implementations of FTP and TFTP in client and server configurations, Telnet as client and server, pSHELL for local and remote system access, and BOOTP as client and server. In 2026 that list reads as unremarkable. In the middle 1990s the selling point was that a device with a few megabytes of memory and no screen could speak exactly the same protocols as the workstation being used to configure it — which is what turned a great deal of equipment into something an engineer could reach from a desk instead of a ladder.

Targets, hosts and the cross-development bench

Nothing about this subject makes sense without the physical arrangement it assumed. The engineer did not sit at the machine being programmed. The machine being programmed was a board in a rack, or a card in a telephone exchange, or a module inside an instrument, and it very often had no keyboard, no display and one serial port. Development happened on a host workstation and the result travelled to the target across a wire.

ISI's filings and product literature record both ends of that arrangement precisely. The host side: pRISM+ ran on Windows NT and Windows 95 and on Solaris and Hewlett-Packard Unix workstations; earlier literature for the small kernel lists IBM PC compatibles, Sun SPARCstations, the HP 9000 Series 700 and the IBM RS/6000. The compilers were cross-compilers, generating code for a processor the host did not have. The debugger ran on the host and spoke to an agent on the target over serial line or Ethernet.

The target side, as stated in the annual report filed in May 1999, is a portrait of embedded computing before consolidation. pRISM+ supported the Motorola 68xxx family, PowerPC, MIPS, ARM, Intel x86 and the Mitsubishi M32R; pSOSystem was supported in addition on the Intel i960, Hitachi SH, ColdFire, Fujitsu SPARClite, NEC V8xx and ST-20. The 1998 product page adds StrongARM to the list. Six or seven mutually incompatible instruction sets, none of them dominant, all of them shipping in volume — and a real-time vendor's competitive position consisting largely in the length of that list and the quality of the ports.

The presence of ARM and StrongARM there is where this page's subject and another's genuinely touch. The architecture itself, its licensing model and its spread through embedded products are the business of the comp.sys.arm page and are not retold here; what belongs here is the commercial fact that by 1998 an ARM port was something a real-time vendor had to have, and that a kernel written in 68000 assembly could never have acquired one. ISI's rewrite of pSOS in C during the early 1990s is the decision that made the rest of that list possible.

Board support packages, and the business of porting

A board support package is the layer between a kernel that knows nothing about hardware and a board that is nothing but hardware. It initialises the processor and the memory controller, configures chip selects and the bus, sets up the timer that will drive the system clock tick, brings up a serial console and an Ethernet controller, and hands the linker a memory map. It is the code that runs before there is anything recognisable as a running system, and it is the code that fails first.

ISI's own definition is exact: each package provides a software template of skeleton device driver code and the low-level system functions a particular piece of hardware needs, with the driver code specific to individual peripheral devices rather than to the board those devices sit on. A number of packages were supplied in source at no additional cost, which was not generosity. A customer's board was never quite the reference board, and a binary package that could not be edited would have been useless to the people buying it.

The catalogue ISI published in 1998 reads as an inventory of what embedded systems were actually built on at the end of the century:

• PowerPC — Motorola ADS8xx and MBX8xx, the MVME1600, MVME2300, MVME2600 and MVME3600, the IBM 403GA evaluation board, the Force PowerCore 604e, the EST SBC821 and SBC860, and a simulator.
• 68k — Motorola MVME162 in three variants, plus the MVME167, MVME172lx and MVME177; the FADS302, FADSENA302, EVM332 and EMV340 development boards; EST's SBC340, SBC341 and SBC360; and the QUICC.
• ColdFire — the SBC5204 and SBC5206.
• MIPS — IDT's 79S460 and 79S465, the Densan DVE 3900 and DVE 4100, LSI's MR4001 and 100, the Baja and the RTE 4100.
• ARM — ARM Ltd's own PID7 board and its Armulator instruction-set simulator, and Digital's EBSA-110.
• x86 — a standard PC, the Intel 386ex, the National ns486sxf.
• i960 and Hitachi SH — the CVME964 and the EV960 evaluation boards in five variants, and Densan's DVE7604 and DVE7708.

Porting is the activity that list implies. If your board was not on it, you wrote the package; if it was, you found out how your board differed from the one the package was written for. Either way the work was specific and unglamorous — getting the memory controller correct before a single line of C could execute, persuading the cache to behave, discovering that the reference design's oscillator was not the one your purchasing department had bought — and it generated precisely the class of question a newsgroup answers better than a manual. Somebody else has done this. Nobody has written it down.

Debugging a machine with no screen

The third recurring genre of an embedded group follows from the second. When the system under test cannot print, cannot stop, and in the early stages cannot even initialise its serial port, how do you find out what it is doing?

pSOSystem's answer was pROBE+, described by its vendor as a kernel-aware component functioning both as a cross-development target agent and as a standalone target debugger; in that second role it was used to bring up new hardware and to develop applications, and it could also perform system profiling to help analyse and optimise performance. As an agent it ran in two modes. System-level debugging stopped all activity at a breakpoint, which is what you need when the thing you are debugging is a device driver or a board. Task-level debugging stopped individual tasks while the rest of the system kept running, which is what you need when the system is a telephone exchange and stopping it is not an option. Both modes used the same user and driver interfaces.

Host-side debuggers connected to that agent over serial or Ethernet: ISI's own SearchLight, CAD-UL's Organon and Software Development Systems' SingleStep are the three named in the pRISM+ literature. What made them worth the money was kernel-object awareness — the ability to look at a queue, a semaphore or a memory region as the operating system's own object rather than as a region of memory that had to be decoded by hand.

Below the agent there was hardware. For genuine bring-up, before any of your code can be trusted, the tools were an in-circuit emulator — a box whose probe replaces the processor in its socket and pretends to be it, while recording everything — or a background debug monitor, the small debug port Motorola built into the 683xx and ColdFire parts, later generalised into JTAG. ISI's literature for the small kernel names emulator and background debug monitor support explicitly for chip-level development, and these are the instruments you reach for when the failure happens before there is a console to print to.

It is worth noticing what all this implies about the tone of a group like this one. The alternative to asking a question was attaching a logic analyser, so the questions tended to be precise, the answers long, and the register unsentimental.

October 1999 to February 2000: the acquisition

The group was not quite ten months old when the agreement to sell its subject to its principal competitor was signed, and thirteen months old when the sale completed.

The transaction is documented in Wind River's own filings, which is the version worth using. On 21 October 1999 Wind River Systems, Integrated Systems and a wholly owned Wind River subsidiary named University Acquisition Corp. entered into an Agreement and Plan of Merger and Reorganization. The mechanism was the standard reverse triangular merger: the subsidiary merged into Integrated Systems, Integrated Systems survived, and the survivor became a wholly owned subsidiary of Wind River.

The terms, as reported to the Securities and Exchange Commission: each outstanding Integrated Systems share was exchanged for 0.92 of a Wind River share; 22,488,916 Wind River shares were issued in aggregate; outstanding Integrated Systems options were converted into Wind River options. The merger was intended to qualify as a tax-free reorganisation and was accounted for as a pooling of interests.

It completed on 15 February 2000. Wind River's press release of that day states the price in the only terms a stock-for-stock deal can be stated in:

Based on Wind River Systems' closing price on February 15, 2000 of $41.50, the stock-for-stock merger is valued at approximately $930 million.

The two companies began operating as one organisation the following day, with, in the release's own accounting, more than five hundred software engineers, two hundred and fifty services engineers, and a sales force that had doubled. The chief executives of both companies are quoted — Tom St. Dennis for Wind River, Chuck Boesenberg for Integrated Systems — warmly and at no point about products. Neither VxWorks nor pSOS is named anywhere in the announcement.

For readers of a newsgroup named after one of those products, the situation needed no interpretation. The buyer owned the competing kernel. The trade press did the arithmetic on the day the deal was announced: EDN's news item of 22 October 1999 was headed Wind River to buy No. 2 RTOS player.

What the acquisition meant for users

The published accounts of what followed are consistent with each other and with the filings. After early statements that pSOS support would continue, development was stopped. Wind River announced a convergence: a version of VxWorks that would support pSOS system calls, no further releases of pSOS, and a recommendation that existing customers move across.

The clearest dating available comes from the annual reports. pSOSystem is listed as one of Wind River's operating system product families in the reports filed on 1 May 2001 and 30 April 2002 — in the second of those it sits in a list with OSEKWorks, VSPWorks and BSD/OS. The word does not appear at all in the report filed on 30 April 2003. Somewhere in that twelve-month gap the product stopped being something the company described to its shareholders.

It did not stop being something the company described to its customers. A Wind River page for pSOSystem 3 was still on the web in October 2005, written entirely in the past tense — at the time of its release, it says, pSOSystem 3 served as a modular, high-performance, memory-protected real-time operating system — with the supported hosts narrowed to Windows NT 4 and 2000 and Solaris, and the supported targets narrowed to parts of the PowerPC and MIPS families. That is what the long tail of a discontinued product looks like from the vendor's side: a page that exists so that the people still running it can find a datasheet. In Japan the company was still selling the exit as a scheduled product of its own. Its training catalogue, as it stood in October 2002, listed a one-day course called pSOS to VxWorks Migration, running from ten in the morning to five in the afternoon, lecture and practical, sixteen places, thirty-six thousand yen: Part I on moving from pSOS to VxWorks — kernel calls, the I/O subsystem, the file system, networking — and Part II on the Tornado development environment.

A competitor moved within five weeks of the merger closing. On 20 March 2000 Express Logic of San Diego announced an Evacuation Kit for pSOS users: a compatibility layer defining basic pSOS services, with some limitations, on top of its own ThreadX kernel, aimed, in the release's words, at pSOS customers who cannot migrate their application to VxWorks due to royalties, cost, size, or performance reasons. The release is marketing and reads like it — it offers pSOS+ customers an alternative to having a competitor's technology forced upon them — but it is also evidence, from an interested party writing at the time, of what the migration question looked like to the customers being migrated. Other vendors sold comparable API-compatibility and porting products in the years that followed, and the open-source RTEMS continued to offer a Classic API descended from the same RTEID lineage that Software Components Group had helped define twenty years earlier.

Underneath the corporate history there is a general point about embedded software that this group illustrates better than most, and it is worth stating without drama. An operating system with a modest installed base of long-lived products does not die when its vendor is acquired. It stops being sold, stops being ported to new silicon, and stops being patched. It does not stop running. A telecommunications line card, a laser printer controller, a piece of process instrumentation, a medical device designed around pSOS in the middle 1990s was very likely still in service ten years later, and the software inside it did not change because two companies in California had merged. The installed base of a discontinued embedded operating system very probably peaks some years after its final release.

What ends, and ends fairly abruptly, is not the software but the ability to get help with it. The vendor's engineers move to the surviving product. The mailing list is retired. The people who know why the memory controller has to be initialised in that particular order take other jobs. That is precisely the service a technical newsgroup supplies, and it is why traffic in a group like this one tends to outlive the product's commercial life for a while — carried by the maintainers rather than the developers — and then to stop, not because the equipment has been switched off but because the last person who was still asking has retired or given up.

The market around it

Integrated Systems named its own competitors in its annual reports, which is a more reliable list than any retrospective — and the list moved. The report for the year to February 1996 named, for the pSOSystem business, Mentor Graphics through its acquisition of Microtec Research; Microware Systems Corporation; and Wind River Systems. Three years later, in the report filed in May 1999, the same sentence named Wind River Systems, Microsoft through its introduction of Windows CE, and Sun Microsystems with its acquisition of Chorus. The RTOS houses had been joined at the top of the threat list by two general-purpose software companies, which is as compact a statement as one could want of what was happening to the embedded market in the late 1990s. Set out with their origins, the field this group's readers were choosing from looked like this.

• VxWorks, from Wind River Systems of Alameda, California, introduced in 1987 — the eventual buyer, and the system pSOS users were ultimately pointed at.
• VRTX, the Versatile Real-Time Executive, released in September 1981 by Hunter & Ready, a company founded in 1980 by James Ready and Colin Hunter. The firm became Ready Systems, merged with Microtec Research in 1993, went public in 1994 and was acquired by Mentor Graphics in 1995; VRTX was later superseded by Nucleus.
• QNX, first released in 1982 by Quantum Software Systems, founded on 30 March 1980 by Gordon Bell and Dan Dodge, who had met the idea in a real-time operating systems course at the University of Waterloo. A microkernel design, and the company later renamed itself after the product.
• LynxOS, first written in 1986 and sold by Lynx Real-Time Systems: a Unix-like real-time system whose distinguishing pitch was POSIX conformance, sold heavily into avionics and defence.
• OS-9, from Microware Systems Corporation, dating from 1979 and the Motorola 6809 before being ported to the 68000 in 1983 and beyond.
• RTEMS, begun in the late 1980s for the United States Army Missile Command and released as open source, carrying the RTEID-derived Classic API.
• ThreadX, from Express Logic of San Diego — the migration target described above, and the one of these whose commercial pitch was explicitly royalty-free licensing.

Each of the commercial kernels on that list had its own application programming interface, its own board support catalogue, its own tool chain and its own licensing model, and a company adopting one was committing a product line to it for the ten or fifteen years that industrial equipment stays in production. That is the whole explanation for the length and heat of an RTOS-selection argument, and it is why the argument was conducted by engineers who had read the datasheets rather than by enthusiasts. This page takes no position in it; the comparison above is offered as a map of the field, not as a ranking.

The standards that shaped the field

Four bodies of standardisation ran across this market, and all four turn up in the products this group discussed.

The RTEID and ORKID line. Described above: Motorola's Real Time Executive Interface Definition, written with technical input from pSOS's original authors, adopted by the VMEbus International Trade Association as a baseline draft for ORKID, and steered by both groups towards the IEEE's real-time work. It is the reason several kernels of the era look alike from the application's side.

POSIX real-time. IEEE Std 1003.1b-1993 defined the real-time extensions — priority scheduling, real-time signals, clocks and timers, semaphores, message passing, shared memory, asynchronous and synchronous input-output, and memory locking — giving a portable spelling for the facilities every real-time kernel had previously invented for itself. Later profiles cut the standard down to sizes an embedded system could actually implement. POSIX conformance became a selling point in its own right, most conspicuously for LynxOS, and it was regularly cited in comparisons against proprietary interfaces, the pSOS+ one among them.

Automotive: OSEK. OSEK — Offene Systeme und deren Schnittstellen für die Elektronik in Kraftfahrzeugen, open systems and their interfaces for motor-vehicle electronics — was founded in 1993 by a consortium of German motor manufacturers and suppliers: BMW, Robert Bosch, DaimlerChrysler, Opel, Siemens and Volkswagen, together with the University of Karlsruhe. In 1994 the French manufacturers Renault and PSA Peugeot Citroën joined, bringing a parallel effort called VDX, Vehicle Distributed eXecutive, and the joint body produced specifications for an operating system, a communications stack and network management for electronic control units. Integrated Systems built pOSEK to it. OSEK's specifications are the ancestors of AUTOSAR, and the reason a modern car's software has a family resemblance to a 1990s embedded kernel.

Safety certification. DO-178B, produced jointly by RTCA committee SC-167 and EUROCAE working group WG-12 and published on 1 December 1992, appearing in Europe as ED-12B, was the de facto standard for airborne software until DO-178C replaced it in 2012. ARINC 653 defined an avionics application executive interface with space and time partitioning, so that applications of different software assurance levels could share one processor in an integrated modular avionics architecture without contaminating each other. Between them these turned the claim our kernel is deterministic from a datasheet line into something an auditor could ask to see evidence for, and they are the reason the classical single-address-space kernel eventually had to grow memory protection.

What the group carried

A caution before the list. This directory holds the group's creation record, not its articles. What can honestly be said about content is what the charter authorised, what the surrounding technical world made inevitable, and what the proposal's own authors said they expected. No thread is described here, because none is in the record this page is built from.

On that basis, the recurring genres of a group of this kind were:

• Board support package questions. Which package is closest to my board; what does the reference package assume; why does the timer tick at the wrong rate.
• Porting. Bringing the kernel up on hardware nobody at the vendor had seen, which the charter names explicitly as its fourth purpose.
• Debugging. Target agents, emulators, background debug monitors, and the interpretation of a system that has stopped without saying why.
• Component questions. The behaviour of the TCP/IP stack under load, the choice between file system formats, the reentrancy of a library function, the interaction of one component with another.
• Tools and versions. The charter's sixth purpose reserves space for older tool and environment versions, which is a maintenance community writing its own admission ticket.
• Selection debates. Comparisons against VxWorks, QNX, VRTX, LynxOS and the rest before February 2000, and comparisons of migration routes after it.

And the charter records what was not wanted: no advertising of products or services, no recruitment announcements, and no binary attachments — softened in the second Request for Discussion to large binaries, with PGP signatures explicitly allowed. The recruitment exclusion has a quiet irony to it, since the evidence the proposal offered for the group's necessity was the volume of pSOS job advertisements elsewhere on the network.

What the record does not show

The honest inventory of gaps, since a reference page that does not keep one is not a reference page.

• No readership figure exists. There is no subscriber count, no traffic estimate and no readership survey for comp.os.psos. The 289 people who voted for it are the only enumerated population this page can offer, and a vote is not a readership.
• No FAQ has been found. Many technical groups maintained a periodically posted answer sheet; if this one did, it has not survived where such things are usually kept.
• The discussion is missing. The news.announce.newgroups archive holds the moderated proposal documents only. The debate that ran in news.groups between the two Requests for Discussion, and produced the revisions visible between them, is not in it — which is also why this page cannot say who normalised the newsgroups line.
• The end date is unknown. The group was never removed by control message and is still carried in the ISC namespace files with the flag for an open unmoderated group. When the last article was posted is not something this page can establish, and it will not guess.
• The vendor numbers are unaudited. Design wins, installations and device counts are all the manufacturer's own claims, restated here as claims.
• One attractive story is unconfirmed. Later reference works record that NXP Semiconductors — Philips Semiconductors, spun out in 2006 — took pSOS for its TriMedia media processors from Wind River and went on supporting it there. It would make a tidy ending, given that Philips staff cast the largest single block of votes that created this group. This page has not been able to confirm it against a primary or contemporaneous source, and therefore does not assert it.

Scope and limits

news2mail.com carried Usenet newsgroups to electronic mail between 2000 and 2004, and this directory is the surviving map of what it carried. That period overlaps the group's life almost exactly at the wrong end: the gateway opened in the year pSOS was discontinued. This page is therefore a description of a newsgroup and of the world it was created to serve, assembled from the administrative record and from the primary documents of the companies involved. It is not a history of the traffic, and it is not a manual.

A word about the address, since it is unusual and is not a mistake. This page lives at /Comp/Os/Psos.html, with capital letters, because that is exactly how it was linked when the directory was built, and the web server that answers for it distinguishes upper case from lower. The capitals were kept deliberately. The whole point of rebuilding a preserved directory is that the addresses in it go on resolving; tidying them into lower case would break every link that already points here, including the ones in other people's bibliographies.

The sources used are, for the group itself, the Internet Systems Consortium's mirrors of the news.announce.newgroups and control archives and its current and 2002 namespace files; for the companies and the transaction, the annual reports and current reports that Integrated Systems and Wind River Systems filed with the United States Securities and Exchange Commission, and the press releases attached to them; for the products, the vendors' own contemporaneous web pages as preserved in the Internet Archive; for the standards lineage, the RTEMS project's own C User's Guide; for the Mars Pathfinder incident, the account written by the leader of its flight software team; and for the general technical and standards material, ordinary encyclopaedic and standards-body references. Where those sources are silent, this page says so rather than filling the gap. The rest of the directory is indexed on the all-groups list.