The news.* hierarchy — Usenet about itself

Long-form reference · 3,728 words · about a 16-minute read

The one hierarchy about itself

The Big Eight are the seven hierarchies produced by the renaming of 1987 and humanities.*, added in 1995. Of the eight, news.* is the only one whose subject is the network carrying it: the arguments about Usenet were conducted on Usenet, in groups created by the procedure they were arguing about.

The newsgroups file the Internet Systems Consortium distributes, from which a server can build its own group table, carries twenty-six groups under news.*, each with the short description a newsreader shows beside the name.

• news.admin.announce — Announcements for news administrators. (Moderated)
• news.admin.censorship — Censorship issues in news administration.
• news.admin.hierarchies — Network news hierarchies.
• news.admin.misc — General topics of network news administration.
• news.admin.moderation — Technical and social issues of newsgroup moderation. (Moderated)
• news.admin.net-abuse.email — Discussion of abuse of email systems.
• news.admin.net-abuse.misc — Network facility abuse, including spamming.
• news.admin.net-abuse.usenet — Discussion of abuse of the Usenet system.
• news.admin.peering — Article flow, server peering requests, & offers.
• news.admin.technical — Technical aspects of maintaining network news. (Moderated)
• news.announce.important — General announcements of interest to all. (Moderated)
• news.announce.newgroups — Calls for newgroups & announcements of same. (Moderated)
• news.announce.newusers — Explanatory postings for new users. (Moderated)
• news.answers — Repository for periodic USENET articles. (Moderated)
• news.groups — Discussions and lists of newsgroups.
• news.groups.proposals — Development of Big 8 proposals. (Moderated)
• news.groups.questions — Where can I find talk about topic X?
• news.lists.filters — Notices for automated news filtering systems.
• news.lists.misc — News-related statistics and lists. (Moderated)
• news.misc — Discussions of USENET itself.
• news.newusers.questions — Q & A for new users of Usenet. (Moderated)
• news.software.b — Discussion about Usenet B-news-compatible software.
• news.software.misc — Usenet-related software other than readers and servers.
• news.software.nn — Discussion about the “nn” Usenet news reader package.
• news.software.nntp — The Network News Transfer Protocol.
• news.software.readers — Discussing software for reading network news (Usenet).

Four families do most of the work: news.admin.* for running a server, news.announce.* for the moderated record, news.groups and its descendants for deciding what should exist, and news.software.* for the programs. news.lists.misc carries statistics and lists; it was created by control message in December 1996, in the same run that removed news.lists, the group to which Brian Reid's monthly Usenet readership report had gone, sampling volunteer news readers and extrapolating across the network.

The renaming that produced the original seven hierarchies is told on the soc.* hub, and the deliberately looser creation mechanism of alt.* on its own page. The group index lists what the gateway itself carried.

news.groups and news.announce.newgroups

The procedure rested on a division between two groups. news.groups was unmoderated and was where the argument happened; news.announce.newgroups was moderated and was where the record was kept, so that only what its moderation team approved counted as an official step. By the account of the Big-8 Management Board, which later inherited it, the announcement group was created in August 1989. The procedural documents abbreviate it to n.a.n and open with the rule that gives it its standing:

Only postings to news.announce.newgroups authorized by the n.a.n moderation team are considered official in this process. All time limits and deadlines will be based on the Date headers of those posts.

For most of the 1990s the moderator was David C. Lawrence, who signed himself tale and whose approval address followed him from rpi.edu to uunet.uu.net and then to isc.org. An administrivia posting of his from March 1991 shows how much of the process was one person's routine: he undertook to post on Sunday and Wednesday nights, with the summaries going out only in the Sunday batch; he had begun crossposting everything to news.groups so the process could be followed in one place; and he noted that calls for votes would carry a Followup-To of poster, to discourage voting by posting.

Lawrence stepped back in October 2002. The team's announcement of 13 October records that the PGP key used to issue control messages in the Big Eight had passed to Russ Allbery, that Todd M. McComb and a volunteer signing as piranha were joining him in overseeing the process, and that the creation guidelines Allbery had until then maintained unofficially would become the official ones.

That archive is why so much of this is checkable. The Internet Systems Consortium keeps the newgroup, rmgroup and checkgroups control messages seen on Usenet since early 1991, with about a month around August 1992 missing by its own account; the collection began at rpi.edu, moved to uu.net in the autumn of 1992 and later to isc.org. A second archive holds the news.announce.newgroups postings themselves.

How a group was created: RFD, CFV, votetaker, result

The procedure was written down and revised repeatedly. The version standing while this gateway ran is dated February 2003; it warns proponents that the business typically takes three months and must be followed precisely, and points beginners at a group-mentors address for help.

1. Request for Discussion. A proposal began when the moderation team posted an RFD carrying a rationale, a charter for every group to be created or changed, and moderator information for any group proposed as moderated. It had to be crossposted to news.groups with Followup-To set there.
2. What could be proposed. Only four kinds of change: create a group, remove one by subsuming it into an existing group, change a group's moderation status, or rename one. Each change in a combined proposal was voted on separately. Name components had to be lowercase ASCII letters, digits, plus signs or dashes, each with at least one letter and no more than twenty characters.
3. Discussion. At least twenty-one days, all of it appearing in news.groups. A further RFD was required if the name, the moderation status or the charter changed materially, or if sixty days passed. A proposal substantially similar to a failed one could not return for six months.
4. Call for Votes. The poll was not run by the proponent, who submitted a Proponent Questionnaire to the Usenet Volunteer Votetakers; a member of the UVV posted a CFV built from it, between ten and sixty days after the latest RFD.
5. Voting. Twenty-one days, with a second identical CFV near the middle. Votes went by mail to the votetaker, one per person, and were acknowledged by mail; a bounced acknowledgement voided the vote. Anonymous, forwarded and proxy votes were invalid, web forms counting as proxies, and the CFV was not to be redistributed by anyone but the votetaker.
6. Result. The votetaker tallied and posted the outcome, listing the name, a form of the address and the vote of everyone who had voted. A change passed only with at least a hundred more YES than NO votes and at least twice as many YES as NO.
7. Objection and implementation. Five days were left for objections in news.groups, after which what had passed was implemented by control messages issued by the team.

The details show what the process was defending against: a ballot circulated to a sympathetic audience is not the same instrument as one published in a single announcement group, and publishing every voter meant a suspicious tally could be examined by anyone.

The control message

A vote decided nothing by itself. Groups existed when a server acted on a control message, and servers were free not to. Three types governed the group list: newgroup, naming a group to create and optionally marking it moderated; rmgroup, naming one to remove; and checkgroups, carrying a hierarchy's whole list so a server could reconcile its own. The guidelines are precise about timing: messages went out at half past ten in the morning, United States Eastern time, Monday to Thursday, at least a hundred and twenty hours after the result.

Because a forged control message could delete a group everywhere, the messages came to be signed. The team used one key for control messages and, from June 2003, a separate and more modern key to sign every approved article in the group itself, using the PGPMoose protocol and the implementation in Andrew Gierth's News::Article Perl module, with the signature carried in an X-Auth header. Administrators who had been judging approvals by the Path header were told to verify the signature instead.

The control-message archive applies its own sanity rules, which amount to a definition of a legal group name: at least two components, drawn from lowercase letters, digits, plus signs, underscores, full stops and dashes; the sequences all and ctl forbidden as components; a first component beginning with a letter and not equal to control, to or example; and a whole name no longer than eighty octets.

news.admin.* and the running of a network with no owner

The administrative groups were where the people who operated servers talked to each other, and they were reorganised by the same procedure they applied to everyone else. The second news.admin reorganisation vote closed at the end of 3 April 1995 with 587 valid votes, taken by Ian Kluft for a proposal from Jonathan Grobe, and the result went out the next day. It created news.admin.hierarchies (401 to 89), news.admin.net-abuse.announce (391 to 182) and news.admin.net-abuse.misc (442 to 91), and removed news.config, news.future, news.newsites and news.software.notes. The result posting itself observes that the subject had been controversial.

Later that year Grobe proposed news.admin.censorship, which passed 213 to 30 on 13 October 1995 with Michael Handler as votetaker. Its RFD is a contemporary statement of what the administrative groups were arguing about: it estimates that about a third of the traffic in news.admin.misc concerned censorship in some form, and lists as examples government action such as the Exon bill, administrators removing individuals for alleged net abuse, article cancellations, and the activities of the Church of Scientology. The charter deliberately declines to define censorship, leaving that to the posters.

The net-abuse groups were split apart in 1996. Tim Skirvin posted the reorganisation RFD on 9 July 1996; Jani Patokallio took the vote, and the result of 9 November 1996 recorded 486 valid votes and six changes, all passing: news.admin.announce, renamed from news.admin.net-abuse.announce (443 to 32), news.admin.net-abuse.email (451 to 28), news.admin.net-abuse.policy (441 to 36), news.admin.net-abuse.bulletins (448 to 34), news.admin.net-abuse.sightings (434 to 40) and news.admin.net-abuse.usenet (447 to 32). The rationale was traffic. Three of the six are in the current list — news.admin.announce, news.admin.net-abuse.email and news.admin.net-abuse.usenet — and the policy, bulletins and sightings groups are not.

What none of these groups had was authority. A vote in news.groups could not compel a server to carry a group, and a consensus in news.admin.misc could not compel anyone to run software they disliked. Every mechanism here was advisory.

Spam, cancels and the Breidbart index

Two postings of 1994 changed what the administrative groups discussed. The first widely recognised Usenet spam is dated 18 January 1994, a religious tract headed as a global alert, posted by a system administrator at Andrews University. The one that is remembered, and is usually and wrongly called the first, was an advertisement for legal services headed as a green card lottery notice, posted on 12 April 1994 by the Arizona lawyers Laurence Canter and Martha Siegel.

Out of the reaction came a definition that deliberately avoided judging content: Usenet convention defined spamming as excessive multiple posting, the repeated posting of a message or of substantially similar messages, rather than as advertising.

The measure that gave the definition teeth was the Breidbart Index, devised by Seth Breidbart. For each copy of a substantively identical article, take the square root of the number of groups it was crossposted to and add the roots together: an article to nine groups and another to sixteen give three plus four, an index of seven. The commonly cited threshold was an index of twenty within forty-five days across all hierarchies, with individual hierarchies free to set stricter limits by their own votes.

Enforcement, where it happened, was by third-party cancel messages issued by volunteers running cancelbots. One of the earliest such uses was by the microbiology professor Richard Depew, to remove anonymous postings from science newsgroups; the best known early one came in June 1994, when Arnt Gulbrandsen cancelled the second wave of the green card postings within minutes of the first article. Conventions grew up so administrators could choose which cancels to accept: a pseudo-site name in the Path header identified the criterion, and a monitored contact address went in an X-Canceled-By header.

None of it was uncontested: contemporary accounts record a long argument among news administrators over whether cancelling anything set a precedent tending towards censorship, or whether it was the ordinary use of a tool the protocol already provided. The record also holds a demonstration of what automation could do when it went wrong. ARMM, for Automated Retroactive Minimal Moderation, was a cancelbot written by the same Depew — Dick Depew of Munroe Falls, Ohio — meant to cancel posts from anonymous-posting sites; its recogniser triggered on its own generated control messages, and on the night of 30 March 1993 it broke loose and spammed news.admin.policy with a recursive explosion of over two hundred messages.

Alternatives were proposed that left the decision with the reader or the server rather than the canceller. The NoCeM protocol let authenticated third parties issue notices that an administrator could use to delete articles from the spool and a reader could use as a third-party killfile, instead of a cancel that acted wherever it was honoured; a group for its policy questions, news.admin.nocem, was proposed by Rosalind Hengeveld and passed 144 to 42 in February 1998 with Jim Davis as votetaker. At the other end sat the Usenet Death Penalty, under which the articles of an entire provider were ignored or cancelled until it dealt with what came out of it. The case usually cited ran against UUNET from 1 August 1997 and was called off on 6 August as the volume of spam from it fell.

news.answers, the periodic postings and the MIT archive

The other great administrative invention of the hierarchy was the periodic posting. A formal RFD for a group in which to collect frequently asked question lists was posted on 10 April 1991 by Glenn R. Stone of Georgia Tech, and is filed in the news.announce.newgroups archive under news.answers. It puts the name, the posting method and the charter openly to the readership, and proposes an informal preliminary vote on the name alone before any call for votes.

The group was created by control message on 23 September 1991, and within thirteen months was a victim of its own success. It was moderated by Jonathan I. Kamens at MIT, and his RFD of October 1992 sets out four problems in order: sites declining to carry the group because of its size, readers no longer able to follow it day to day as the charter had envisaged, complaints about unrelated periodic postings sitting side by side, and the growing burden on a single moderator. The remedy was a set of per-hierarchy answers groups. The result of 6 January 1993 passed all six: comp.answers 258 to 27, misc.answers 246 to 28, rec.answers 241 to 31, sci.answers 251 to 27, soc.answers 240 to 29 and talk.answers 232 to 35.

What made the system an archive rather than a genre was a set of headers. A periodic posting carried an Archive-name giving its path in the archive, a Last-modified date and often a revision number, and a Posting-frequency; an Expires header kept it visible until the next issue, and a Supersedes header replaced the previous one. Ballots in the 1992 vote went to a mail server at pit-manager.mit.edu. What became of the postings themselves, in the FAQ archive at MIT that reference works still cite by its old file transfer paths, is told on the comp.* hub.

The self-reference runs deep. The Big Eight creation guidelines were themselves a news.answers periodic posting: the February 2003 revision carries an Archive-name of usenet/creating-newsgroups/big-eight, a posting frequency of monthly, a Supersedes header pointing at the previous month's copy and an Expires header set five weeks ahead.

Inducting new arrivals

news.announce.newusers was the moderated group carrying the explanatory articles a newcomer was expected to read, reposted on a cycle. By the account in the RFD that Jon Bell filed in September 2007 to propose removing it, it dates from the middle 1980s, and in 2001 its previous moderator, Mark Moraes, passed control to the moderation board of news.newusers.questions. Its later history is a small study in how a volunteer institution ends: the last approved postings appeared in May 2006, after a server crash or an operating-system upgrade stopped the software that had been posting them automatically.

Bell had also been a proponent of the change that made news.newusers.questions moderated, and that proposal is the clearest surviving example of the full procedure at work. The first RFD went out on 6 December 1996, a second on 24 February 1997 and a third on 17 March. The call for votes was posted on 17 April 1997 by David Bostwick of the Usenet Volunteer Votetakers, with the second copy on 28 April; voting closed at the end of 8 May; and the result, posted on 19 May 1997, recorded 357 YES and 25 NO out of 382 valid votes, with two abstentions and one invalid vote, followed by the customary five-day window for objections.

The rationale says a good deal about Usenet in 1997. The proponents wrote that traffic had risen to around a thousand articles a day, of which fewer than a fifth were questions and answers; the rest were test postings, requests for contacts and general chatter. They rejected conventional hand moderation on two grounds, one practical and one closer to a statement of manners: the volume made it impossible, and they did not think it appropriate to welcome new users by rejecting their first articles. They proposed instead a robomoderator that would append to each accepted article a paragraph explaining what the group was for.

The software groups

The last family under news.* is about the programs. news.software.readers covered the client end, news.software.nntp the transport protocol and the servers that spoke it, news.software.b the transport software descended from B News, and news.software.nn a single newsreader popular enough to be given a group of its own; the gateway carried that one, and this directory keeps a page for it. Programming and operating systems belonged in the comp.* hierarchy.

The names aged badly, and there was an attempt to fix them. On 1 November 1995 Jonathan Grobe posted an RFD to rename news.software.b to news.software.c-news and news.software.nntp to news.software.inn, and to add news.software.future and news.software.misc. His rationale is a snapshot of the software landscape: news.software.nntp had become the main discussion place for INN — InterNetNews, released by Rich Salz in 1991 and presented at the Summer 1992 USENIX conference — while C News was discussed in news.software.b, neither name made this obvious, and both groups ended up covering INN so that readers followed both. The canonical list today still carries news.software.b and news.software.nntp under their original names, alongside news.software.misc, and neither of the proposed replacements.

Two software groups went by decision rather than neglect. news.software.notes, for the Notesfiles system that predated and competed with Usenet's own software, was among the groups removed in the 1995 administrative reorganisation. news.software.anu-news, for the VMS news package of that name, was created in July 1989 and removed by the Big-8 Management Board on 29 March 2007, under a procedure that had by then replaced the vote: the last call for comments drew no comments at all, and the RFD recorded that the group had carried no on-topic article since a two-message thread in November 2005.

The end of voting

The mechanism did not so much fail as run out of voters. On 13 September 2005 Todd McComb, writing for the news.announce.newgroups team, announced that, because of more stringent vote validation and lower turnout, a proposal would need fifty more valid YES votes than NO rather than a hundred, the two-thirds margin remaining in force. The announcement acknowledges that the hundred-vote rule had stood for nearly twenty years and had come to symbolise Usenet newsgroup creation.

In December 2005 Bill Aten, writing as coordinator of the Usenet Volunteer Votetakers, announced that the body would step aside for a new team; he described himself as its lone remaining member after more than ten years, with eighty-nine calls for votes taken by his own records. The Big 8 Management Board was formed in 2005 out of former moderators of news.announce.newgroups, and from late 2006 the closing RFD in a proposal was headed as a last call for comments in place of a call for votes. On 30 September 2006 Russ Allbery, who had held the control-message key since 2002 and had first arrived in news.groups as a college freshman in 1993, posted a formal notice that he was stepping down as moderator, thanking the group for thirteen years.

The transition was carried out by the new procedure on itself. An RFD to create news.groups.proposals, a moderated group for the announcement, discussion and development of Big Eight proposals, was posted by Dave Sill on 9 October 2006; a second followed on 23 October; the third, on 8 November, was headed as a last call for comments and opened five days for final comment. On 14 November the board posted a result stating that it had decided by consensus to create the group, which was created on 18 November 2006. Its charter routes official discussion of active proposals into it and leaves general argument about creation policy and about the history of the Big Eight where it had always been, in news.groups.

What survives is the paper trail: the requests for discussion, the calls for votes, the tallies with every voter listed, the policy announcements and the control messages, all archived and public. How to reach the surviving groups is set out on the access page, with the background on what Usenet is.