Usenet newsgroup

Last updated

A Usenet newsgroup is a repository usually within the Usenet system, for messages posted from users in different locations using the Internet. They are discussion groups and are not devoted to publishing news. Newsgroups are technically distinct from, but functionally similar to, discussion forums on the World Wide Web. Newsreader software is used to read the content of newsgroups. Before the adoption of the World Wide Web, Usenet newsgroups were among the most popular Internet services.

Contents

Communication is facilitated by the Network News Transfer Protocol (NNTP) which allows connection to Usenet servers and data transfer over the internet. Similar to another early (yet still used) protocol SMTP which is used for email messages, NNTP allows both server-server and client-server communication. This means that newsgroups can be replicated from server to server which gives the Usenet network the ability to maintain a level of robust data persistence as a result of built-in data redundancy. However, most users will access using only the client-server commands of NNTP and in almost all cases will use a GUI for browsing as opposed to command line based client-server communication specified in the NNTP protocol. [1]

Types

Newsgroups generally come in either of two types, binary or text. There is no technical difference between the two, but the naming differentiation allows users and servers with limited facilities to minimize network bandwidth usage. Generally, Usenet conventions and rules are enacted with the primary intention of minimizing the overall amount of network traffic and resource usage. Typically, the newsgroup is focused on a particular topic of interest. A message sent for publication on a newsgroup is called a "post". Some newsgroups allow posts on a wide variety of themes, regarding anything a member chooses to discuss as on-topic, while others keep more strictly to their particular subject, frowning on off-topic posts. The news admin (the administrator of a news server) decides how long posts are kept on their server before being expired (deleted), which is called retention. Different servers will have different retention times for the same newsgroup; some may keep posts for as little as one or two weeks, others may hold them for many years.

Back when the early community was the pioneering computer society, the common habit seen with many posts was a notice at the end that disclosed whether the author had (or was free of) a personal interest (financial, political or otherwise) in making the post. This is rarer now, and the posts must be read more skeptically, as with other media. Privacy and phishing issues have also risen in importance.

Usenet newsgroups posters and operators usually do not make money from their occupations on the platform.

The number of newsgroups grew from more than 100 as of 1983 [2] to more than 110,000, but only 20,000 or so of those are active. [3] Newsgroups vary in popularity; some newsgroups receive fewer than a dozen posts per year while the most popular can get several thousand in under an hour.

Binary

October 2020 screenshot showing 60 PB of usenet group data Usenet-total-storage.jpg
October 2020 screenshot showing 60 PB of usenet group data

While newsgroups were not created with the intention of distributing files such as pictures, sound and video, they have proven to be quite effective for this. As of 2022, some remain popular as an alternative to BitTorrent to share and download files. [5]

Because newsgroups are widely distributed, a file uploaded once will be spread to many other servers and can then be downloaded by an unlimited number of users. More useful is that users download from a local news server, rather than from a more distant machine with perhaps limited connectivity, as may be the case with peer-to-peer technology. In fact, this is another benefit of newsgroups: it is usually not expected that users share. If every user makes uploads then the servers would be flooded; thus it is acceptable and often encouraged for users to just leech.

There were originally a number of obstacles to the transfer of binary files over Usenet. Usenet was originally designed with the transmission of text in mind, and so the encoding of posts caused losses in binary data where the data was not part of the protocol's character set. Consequently, for a long while, it was impossible to send binary data as such. As workarounds, codecs such as Uuencode and later Base64 and yEnc were developed which encoded the binary data from the files to be transmitted (e.g. sound or video files) to text characters which would survive transmission over Usenet. At the receiver's end, the data needed to be decoded by the user's news client.

Additionally, there was a limit on the size of individual posts so that large files could not be sent as single posts. To get around this, Newsreaders were developed which were able to split long files into several posts. Intelligent newsreaders at the other end could then automatically group such split files into single files, allowing the user to easily retrieve the file. These advances have meant that Usenet is used to send and receive many terabytes of files per day.

There are two main issues that pose problems for transmitting large files over newsgroups. The first is completion rates and the other is retention rates. The business of premium news servers is generated primarily on their ability to offer superior completion and retention rates, as well as their ability to offer very fast connections to users. Completion rates are significant when users wish to download large files that are split into pieces; if any one piece is missing, it is impossible to successfully download and reassemble the desired file. To work around the problem, a redundancy scheme known as Parchive (PAR) is commonly used.

Many major news servers have a retention time of more than seven years. [6] A number of websites exist to keep an index of files posted to binary newsgroups.

Partly because of such long retention times, as well as growing uploading and downloading speeds, Usenet is also used by individuals to store backup data in a practice called Usenet backup, or uBackup. [7] While commercial providers offer easier-to-use online backup services, storing data on Usenet is free of charge (although access to Usenet itself may not be). A user must manually select, prepare and upload the data. Because anyone can download the backup files, the data is typically encrypted. After the files are uploaded, the uploader has no control over them; they are automatically distributed to all Usenet providers that subscribe to the newsgroup they are uploaded to, so there will be copies of them spread all around the world.

Moderated newsgroups

Most Newsgroups are not moderated. A moderated newsgroup has one or more individuals who must approve posts before they are published. A separate address is used to submit posts and the moderators then propagate those they approve of. The first moderated newsgroups appeared in 1984 under mod.* according to RFC 2235, "Hobbes' Internet Timeline".

Distribution

Transmission within and at the bounds of the network uses the Network News Transfer Protocol (NNTP) (Internet standard RFC 3977 of 2006, updating RFC 977 of 1986).

Newsgroup servers are hosted by various organizations and institutions. Most Internet service providers host their own news servers, or rent access to one, for their subscribers. There are also a number of companies who sell access to premium news servers.

Every host of a news server maintains agreements with other nearby news servers to synchronize regularly. In this way news servers form a redundant network. When a user posts to one news server, the post is stored locally. That server then shares posts with the servers that are connected to it for those newsgroups they both carry. Those servers do likewise, propagating the posts through the network. For newsgroups that are not widely carried, sometimes a carrier group is used for crossposting to aid distribution. This is typically only useful for groups that have been removed or newer alt.* groups. Crossposts between hierarchies, outside of the Big 8 and alt.* hierarchies, are prone to failure.

Hierarchies

Newsgroups are often arranged into hierarchies, theoretically making it simpler to find related groups. The term top-level hierarchy refers to the hierarchy defined by the prefix before the first dot.

The most commonly known hierarchies are the Usenet hierarchies. So for instance newsgroup rec.arts.sf.starwars.games would be in the rec.* top-level Usenet hierarchy, where the asterisk (*) is defined as a wildcard character. There were seven original major hierarchies of Usenet newsgroups, known as the "Big 7":

These were all created in the Great Renaming of 1986–1987, before which all of these newsgroups were in the net.* hierarchy. At that time there was a great controversy over what newsgroups should be allowed. Among those that the Usenet cabal (who effectively ran the Big 7 at the time) did not allow were those concerning recipes, recreational drug use, and sex.

This situation resulted in the creation of an alt.* (short for "alternative") Usenet hierarchy, under which these groups would be allowed. Over time, the laxness of rules on newsgroup creation in alt.* compared to the Big 7 meant that many new topics could, given time, gain enough popularity to get a Big 7 newsgroup. There was a rapid growth of alt.* as a result, and the trend continues to this day. Because of the anarchistic nature with which the groups sprang up, some jokingly referred to ALT standing for "Anarchists, Lunatics and Terrorists" (a backronym).

In 1995, humanities.* was created for the discussion of the humanities (e.g. literature, philosophy), and the Big 7 became the Big 8.

The alt.* hierarchy has discussion of all kinds of topics, and many hierarchies for discussion specific to a particular geographical area or in a language other than English.

Before a new Big 8 newsgroup can be created, an RFD (Request For Discussion) must be posted into the newsgroup news.announce.newgroups, which is then discussed in news.groups.proposals. Once the proposal has been formalized with a name, description, charter, the Big-8 Management Board will vote on whether to create the group. If the proposal is approved by the Big-8 Management Board, the group is created. Groups are removed in a similar manner.

Creating a new group in the alt.* hierarchy is not subject to the same rules; anybody can create a newsgroup, and anybody can remove it, but most news administrators will ignore these requests unless a local user requests the group by name.

Further hierarchies

There are a number of newsgroup hierarchies outside of the Big 8 (and alt.*) that can be found on many news servers. These include non-English language groups, groups managed by companies or organizations about their products, geographic/local hierarchies, and even non-internet network boards routed into NNTP. Examples include (alphabetically):

Additionally, there is the free.* hierarchy, which can be considered "more alt than alt.*". There are many local sub-hierarchies within this hierarchy, usually for specific countries or cultures (such as free.it.* for Italy).

See also

Related Research Articles

rn is a Usenet newsreader written by Larry Wall and originally released in 1984. It was one of the first newsreaders to take full advantage of a character-addressable CRT terminal. Previous newsreaders, such as readnews, were mostly line-oriented and designed for use on the printing terminals which were common on the early Unix minicomputers where the Usenet software and network originated. Later variants of the original rn program included rrn, trn, and strn.

The Great Renaming was a restructuring of Usenet newsgroups that took place in 1987. B News maintainer and UUNET founder Rick Adams is generally considered to be the initiator of the Renaming.

<span class="mw-page-title-main">News server</span> Type of server software

A news server is a collection of software used to handle Usenet articles. It may also refer to a computer itself which is primarily or solely used for handling Usenet. Access to Usenet is only available through news server providers.

The Network News Transfer Protocol (NNTP) is an application protocol used for transporting Usenet news articles (netnews) between news servers, and for reading/posting articles by the end user client applications. Brian Kantor of the University of California, San Diego, and Phil Lapsley of the University of California, Berkeley, wrote RFC 977, the specification for the Network News Transfer Protocol, in March 1986. Other contributors included Stan O. Barber from the Baylor College of Medicine and Erik Fair of Apple Computer.

<i>alt.*</i> hierarchy Subclass of Usenet newsgroups

The alt.* hierarchy is a major class of newsgroups in Usenet, containing all newsgroups whose name begins with "alt.", organized hierarchically. The alt.* hierarchy is not confined to newsgroups of any specific subject or type, although in practice more formally organized groups tend not to occur in alt.*. The alt.* hierarchy was created by John Gilmore and Brian Reid.

alt.sex is a Usenet newsgroup – a discussion group within the Usenet network – relating to human sexual activity. It was popular in the 1990s. An October 1993 survey by Brian Reid reported an estimated worldwide readership for the alt.sex newsgroup of 3.3 million, that being 8% of the total Usenet readership, with 67% of all Usenet "nodes" carrying the group. At that time, alt.sex had an estimated traffic of 2,300 messages per month.

<span class="mw-page-title-main">Google Groups</span> Service from Google that provides discussion groups

Google Groups is a service from Google that provides discussion groups for people sharing common interests. Until February 2024, the Groups service also provided a gateway to Usenet newsgroups, both reading and posting to them, via a shared user interface. In addition to accessing Google groups, registered users can also set up mailing list archives for e-mail lists that are hosted elsewhere.

yEnc is a binary-to-text encoding scheme for transferring binary files in messages on Usenet or via e-mail. It reduces the overhead over previous US-ASCII-based encoding methods by using an 8-bit encoding method. yEnc's overhead is often as little as 1–2%, compared to 33–40% overhead for 6-bit encoding methods like uuencode and Base64. yEnc was initially developed by Jürgen Helbing, and its first release was early 2001. By 2003 yEnc became the de facto standard encoding system for binary files on Usenet. The name yEncode is a wordplay on "Why encode?", since the idea is to only encode characters if it is absolutely required to adhere to the message format standard.

Forté Agent was an email and Usenet news client used on the Windows operating system. Agent was conceived, designed and developed by Mark Sidell and the team at Forté Internet Software in 1994 to address the need for an online/offline newsreader which capitalized on the emerging Windows GUI framework. By 1995, Agent had expanded to become a full-featured email client and remains a widely used application for integrating news and email communication on Windows. Agent supports POP email but not IMAP.

tin (newsreader)

tin is an open-source, text-based, threaded newsreader, used to read and post messages on Usenet, the worldwide distributed discussion system.

<span class="mw-page-title-main">Newsreader (Usenet)</span> Application that reads articles on Usenet distributed throughout newsgroups

A newsreader is a software application that reads articles on Usenet distributed throughout newsgroups. Newsreaders act as clients which connect to a news server, via the Network News Transfer Protocol (NNTP), to download articles and post new articles. In addition to text-based articles, Usenet is also used to distribute binary files, generally in dedicated "binaries" newsgroups.

Usenet II was a proposed alternative to the classic Usenet hierarchy, started in 1998. Unlike the original Usenet, it was peered only between "sound sites" and employed a system of rules to keep out spam.

This is a comparison of Usenet newsreaders.

Control messages are a special kind of Usenet post that are used to control news servers. They differ from ordinary posts by a header field named Control. The body of the field contains control name and arguments.

yProxy is a Network News Transfer Protocol (NNTP) proxy server for the Windows operating system. yProxy's main function is to convert yEnc-encoded attachments to UUE-encoded attachments on the fly. The main purpose of this is to add functionality to NNTP newsreaders that do not have native support for yEnc. The inventor of yEnc recommends yProxy for use by Windows users whose newsreaders do not support yEnc decoding.

<span class="mw-page-title-main">Usenet</span> Worldwide computer-based distributed discussion system

Usenet, USENET, or, "in full", User's Network, is a worldwide distributed discussion system available on computers. It was developed from the general-purpose Unix-to-Unix Copy (UUCP) dial-up network architecture. Tom Truscott and Jim Ellis conceived the idea in 1979, and it was established in 1980. Users read and post messages to one or more topic categories, known as newsgroups. Usenet resembles a bulletin board system (BBS) in many respects and is the precursor to the Internet forums that have become widely used. Discussions are threaded, as with web forums and BBSes, though posts are stored on the server sequentially.

XOVER is a Network News Transfer Protocol (NNTP) command used to return information from a news server's overview (NOV) database..

Easynews, Inc is a HW Media Usenet/newsgroup reseller. Founded in 1994, Easynews service is available to individual users through a subscription model and as an outsourced service to internet service providers. Easynews offers Usenet access both through traditional Network News Transfer Protocol (NNTP) servers as well as a web interface using a standard web browser.

Spotnet is a protocol on top of Usenet, providing a decentralized alternative to usenet indexing websites, and the NZB format in general. Spotnet allows users to create and browse private 'newsservers', or decentralized repositories of files and information. Members share spots with one another, similar to the seeding process in torrent sharing. Spotnet experienced tremendous growth since the closure of FTD in 2011.

References

  1. ↑ Feather, CDW (October 2006). Network News Transfer Protocol (NNTP). IETF. doi: 10.17487/RFC3977 . RFC 3977 . Retrieved 3 June 2019.
  2. ↑ Emerson, Sandra L. (October 1983). "Usenet / A Bulletin Board for Unix Users". BYTE. pp. 219–236. Retrieved 31 January 2015.
  3. ↑ "List of Active Newsgroups". www.tech-insider.org. Retrieved 2024-12-20.
  4. ↑ "Usenet storage is more than 60 petabytes (60000 terabytes)". binsearch.info. Archived from the original on 2020-05-21. Retrieved October 20, 2020.
  5. ↑ Gregersen, Erik; Hosch, William L. (2022-02-17). "newsgroup". Encyclopedia Britannica . Retrieved 2023-04-28.
  6. ↑ "Retention Increase to 2600 Days at NewsDemon". Newsdemon.com. 28 September 2015. Retrieved April 10, 2016.
  7. ↑ "usenet backup (uBackup)". Wikihow.com. Retrieved February 14, 2012.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.