Ftp Web Hosting - J2Ee Hosting Sun Java Developing Blog

Listing New Articles When a newsreader first connects to a new server and the user chooses a newsgroup to browse, the newsreader will want to retrieve a list of new articles, those posted or received since the last login by the user. The newnews command is used for this purpose. Three mandatory arguments must be supplied: the name of the group or groups to query, the start date, and the start time from which to list. The date and time are each specified as six- digit numbers, with the most significant information first; yymmdd and hhmmss, respectively: newnews junk 990101 000000 230 New news follows <7g2o5r$aa$6@news.vbrew.com> <7g5bhm$8f$2@news.vbrew.com> <7g5bk5$8f$3@news.vbrew.com> . Selecting a Group on Which to Operate When the user selects a newsgroup to browse, the newsreader may tell the news server that the group was selected. This simplifies the interaction between newsreader and news server; it removes the need to constantly send the name of the newsgroup with each command. The group command simply takes the name of the selected group as an argument. Many following commands use the group selected as the default, unless another news- group is specified explicitly: group junk 211313 junk The group command returns a message indicating the number of active messages, the low-water mark, the high- water mark, and the name of the group, respectively. Note that while the number of active messages and the high-water mark are the same in our example, this is not often the case; in an active news server, some articles may have expired or been deleted, lowering the number of active messages but leaving the high-water mark untouched. Listing Articles in a Group To address newsgroup articles, the newsreader must know which article numbers represent active articles. The listgroup command offers a list of the active article numbers in the current group, or an explicit group if the group name is supplied: listgroup junk 211 Article list follows 1 2 3 . Retrieving an Article Header Only The user must have some information about an article before she can know whether she wishes to read it. We mentioned earlier that some commands allow the article header and body to be transferred separately. The head command is used to request that the server transmit just the header of the specified article to the newsreader. If the user doesn’t want to read this article, we haven’t wasted time and network bandwidth transferring a potentially large article body unnecessarily. Articles may be referenced using either their number (from the listgroup command) or their message identifier: head 2 221 2 <7g5bhm$8f$2@news.vbrew.com> head Path: news.vbrew.com!not-for-mail From: terry@richard.geek.org.au Newsgroups: junk Subject: test message number 2 Date: 27 Apr 1999 21:51:50 GMT

xpath MessageID Report problems to . NNTP reader mode has a lot of commands. Many of these are designed to make the life of a newsreader easier. We mentioned earlier that there are commands that instruct the server to send the head and the body of articles separately. There are also commands that list the available groups and articles, and others that allow posting, an alternate means of sending news articles to the server. Listing Available Groups The list command lists a number of different types of information; notably the groups supported by the server: list newsgroups 215 Descriptions in form “group description”. control News server internal group junk News server internal group local.general General local stuff local.test Local test group . Listing Active Groups list active shows each supported group and provides information about them. The two numbers in each line of the output are the high-water mark and the low-water mark — that is, the highest numbered article and lowest numbered article in each group. The newsreader is able to form an idea of the number of articles in the group from these. We’ll talk a little more about these numbers in a moment. The last field in the output displays flags that control whether posting is allowed to the group, whether the group is moderated, and whether articles posted are actually stored or just passed on. These flags are described in detail in Chapter 23. An example looks like this: list active 215 Newsgroups in form “group high low flags”. control 0000000000 0000000001 y junk 0000000003 0000000001 y alt.test 0000000000 0000000001 y . Posting an Article We mentioned there was a difference between pushing an article and posting an article. When you are pushing an article, there is an implicit assumption that the article already exists, that it has a message identifier that has been uniquely assigned to it by the server to which it was originally posted, and that it has a complete set of headers. When posting an article, you are creating the article for the first time and the only headers you supply are those that are meaningful to you, such as the Subject and the Newgroups to which you are posting the article. The news server you post the article on will add all the other headers for you and create a message ID that it will use when pushing the article onto other servers. All of this means that posting an article is even easier than pushing one. An example posting looks like this: post 340 Ok From: terry@richard.geek.org.au Subject: test message number 1 Newsgroups: junk Body: This is a test message, please feel free to ignore it. . 240 Article posted We’ve generated two more messages like this one to give our following examples some realism.

From: terry@gw.vk2ktj.ampr.org Subject: test message sent with ihave Newsgroups: junk Distribution: world Path: gw.vk2ktj.ampr.org Date: 26 April 1999 Message-ID: <123456@gw.vk2ktj.ampr.org> Body: This is a test message sent using the NNTP IHAVE command. . 235 All NNTP commands are case insensitive, so you may enter them in either upper- or lowercase. The IHAVE command takes one mandatory argument, it being the Message ID of the article that is being pushed. Every news article is assigned a unique message ID when it is created. The IHAVE command provides a way of the NNTP server to say which articles it has when it wants to push articles to another server. The sending server will issue an IHAVE command for each article it wishes to push. If the command response code generated by the receiving NNTP server is in the “3xx” range, the sending NNTP server will transmit the complete article, including it’s full header, terminating the article with a period on a line by itself. If the response code was in the “4xx” range, the receiving server has chosen not to accept this article, possibly because it already has it, or because of some problem, such as running out of disk space. When the article has been transmitted, the receiving serve issues another response code indicating whether the article transmission was successful. Changing to NNRP Reader Mode Newsreaders use their own set of commands when talking to a news server. To activate these commands, the news server has to be operating in reader mode. Most news servers default to reader mode, unless the IP address of the connecting host is listed as a news-forwarding peer. In any case, NNTP provides a command to explicitly switch into reader mode: mode reader 200 news.vbrew.com InterNetNews NNRP server INN 1.7.2 08-Dec-1997 ready/ (posting ok). help 100 Legal commands authinfo user Name|pass Password|generic
article [MessageID|Number] body [MessageID|Number] date group newsgroup head [MessageID|Number] help ihave last list [active|active.times|newsgroups|distributions|distrib.pats|/ overview.fmt|subscriptions] listgroup newsgroup mode reader newgroups yymmdd hhmmss [”GMT”] [] newnews newsgroups yymmddhhmmss [”GMT”] [] next post slave stat [MessageID|Number] xgtitle [group_pattern] xhdr header [range|MessageID] xover [range] xpat header range|MessageID pat [morepat…]

is not really an option. We will therefore discuss only the NNTP daemon contained in the nntpd package and leave out the client programs. If you wish to run a large news site, you should look at the InterNet News package, or INN, that was written by Rich Salz. It provides both NNTP and UUCP-based news transport. News transport is definitely better than nntpd. We discuss INN in detail in Chapter 23, Internet News. The NNTP Protocol We’ve mentioned two NNTP commands that are key to how news articles are pushed or pulled between servers. Now we’ll look at these in the context of an actual NNTP session to show you how simple the protocol is. For the purposes of our illustration, we’ll use a simple telnet client to connect to an INN-based news server at the Virtual Brewery called news.vbrew.com. The server is running a minimal configuration to keep the examples short. We’ll look at how to complete the configuration of this server in Chapter 23. In our testing we’ll be very careful to generate articles in the junk newsgroup only, to avoid disturbing anyone else. Connecting to the News Server Connecting to the news server is a simple as opening a TCP connection to its NNTP port. When you are connected, you will be greeted with a welcome banner. One of the first commands you might try is help. The response you get generally depends upon whether the server believes you are a remote NNTP server or a news- reader, as there are different command sets required. You can change your operating mode using the mode command; we’ll look at that in a moment: $ telnet news.vbrew.com nntp Trying 172.16.1.1… Connected to localhost. Escape character is ‘^]’. 200 news.vbrew.com InterNetNews server INN 1.7.2 08-Dec-1997 ready help 100 Legal commands authinfo help ihave check takethis list mode xmode quit head stat xbatch xpath xreplic For more information, contact “usenet” at this machine. . The responses to NNTP commands always end with a period (.) on a line by itself. The numbers you see in the output listing are response codes and are used by the server to indicate success or failure of a command. The response codes are described in RFC-977; we’ll talk about the most important ones as we proceed. Pushing a News Article onto a Server We mentioned the IHAVE command when we talked about pushing news articles onto a news server. Let’s now have a look at how the IHAVE command actually works: ihave <123456@gw.vk2ktj.ampr.org>

Chapter 22 - NNTP and the nntpd Daemon Network News Transfer Protocol (NNTP) provides for a vastly different approach to news exchange from C News and other news servers without native NNTP support. Rather than rely on a batch technology like UUCP to transfer news articles between machines, it allows articles to be exchanged via an interactive network connection. NNTP is not a particular software package, but an Internet standard described in RFC-977. It is based on a stream-oriented connection, usually over TCP, between a client anywhere in the network and a server on a host that keeps Netnews on disk storage. The stream connection allows the client and server to interactively negotiate article transfer with nearly no turnaround delay, thus keeping the number of duplicate articles low. Together with the Internet’s high-transfer rates, this adds up to a news transport that surpasses the original UUCP networks by far. While some years ago it was not uncommon for an article to take two weeks or more before it arrived in the last corner of Usenet; it is now often less than two days. On the Internet itself, it is even within the range of minutes. Various commands allow clients to retrieve, send, and post articles. The difference between sending and posting is that the latter may involve articles with incomplete header information; it generally means that the user has just written the article.132 Article retrieval may be used by news transfer clients as well as newsreaders. This makes NNTP an excellent tool for providing news access to many clients on a local network without going through the contortions that are necessary when using NFS. NNTP also provides for an active and a passive way to transfer news, colloquially called “pushing” and “pulling.” Pushing is basically the same as the ihave/sendme protocol used by C News (described in Chapter 21, C News). The client offers an article to the server through the IHAVE msgid command, and the server returns a response code that indicates whether it already has the article or if it wants it. If the server wants the article, the client sends the article, terminated by a single dot on a separate line. Pushing news has the single disadvantage that it places a heavy load on the server system, since the system has to search its history database for every single article. The opposite technique is pulling news, in which the client requests a list of all (available) articles from a group that have arrived after a specified date. This query is performed by the NEWNEWS command. From the returned list of message IDs, the client selects those articles it does not yet have, using the ARTICLE command for each of them in turn. Pulling news needs tight control by the server over which groups and distributions it allows a client to request. For example, it has to make sure that no confidential material from newsgroups local to the site is sent to unauthorized clients. There are also a number of convenience commands for newsreaders that permit them to retrieve the article header and body separately, or even single header lines from a range of articles. This lets you keep all news on a central host, with all users on the (presumably local) network using NNTP-based client programs for reading and posting. This is an alternative to exporting the news directories via NFS, as described in Chapter 21. An overall problem of NNTP is that it allows a knowledgeable person to insert articles into the news stream with false sender specification. This is called news faking or spoofing.133 An extension to NNTP allows you to require user authentication for certain commands, providing some measure of protection against people abusing your news server in this way. There are a number of NNTP packages. One of the more widely known is the NNTP daemon, also known as the reference implementation. Originally, it was written by Stan Barber and Phil Lapsley to illustrate the details of RFC-977. As with much of the good software available today, you may find it prepackaged for your Linux distribution, or you can obtain the source and compile it yourself. If you choose to compile it yourself, you will need to be quite familiar with your distribution to ensure you configure all of the file paths correctly. The nntpd package has a server, two clients for pulling and pushing news, and an inews replacement. They live in a B News environment, but with a little tweaking, they will be happy with C News, too. However, if you plan to use NNTP for more than offering newsreaders access to your news server, the reference implementation 132 When posting an article over NNTP, the server always adds at least one header field, NNTP-Posting-Host:. The field contains the client’s hostname. 133 The same problem exists with the Simple Mail Transfer Protocol (SMTP), although most mail transport agents now provide mechanisms to prevent spoofing.

addgroup This script adds a group to your site locally. The proper invocation is: addgroup groupname y|n|m|=realgroup The second argument has the same meaning as the flag in the active file, meaning that anyone may post to the group (y), that no one may post (n), that it is moderated (m), or that it is an alias for another group (=realgroup). You might also want to use addgroup when the first articles in a newly created group arrive earlier than the newgroup control message that is intended to create it. delgroup This script allows you to delete a group locally. Invoke it as: delgroup groupname You still have to delete the articles that remain in the newsgroup’s spool directory. Alternatively, you might leave it to the natural course of events (i.e., expiration) to make them go away. addmissing This script adds missing articles to the history file. Run it when there are articles that seem to hang around forever. newsboot This script should be run at system boot time. It removes any lock files left over when news processes were killed at shutdown, and closes and executes any batches left over from NNTP connections that were terminated when shutting down the system. newsrunning This script resides in /usr/lib/news/input and may be used to disable unbatching of incoming news, for instance during work hours. You may turn off unbatching by invoking: /usr/lib/news/input/newsrunning off It is turned on by using on instead of off.

string and the output of uuname, respectively. C News is very laconic about version messages; it returns a simple, unadorned C. Again, you should never issue such a message unless you have made sure that it cannot leave your (regional) network. Replies to sendsys messages can quickly bring down a UUCP network.131 C News in an NFS Environment A simple way to distribute news within a local network is to keep all news on a central host and export the relevant directories via NFS so that newsreaders may scan the articles directly. The overhead involved in retrieving and threading articles is significantly lower than NNTP. NNTP, on the other hand, wins in a heterogeneous network where equipment varies widely among hosts, or where users don’t have equivalent accounts on the server machine. When you use NFS, articles posted on a local host have to be forwarded to the central machine because accessing adminstrative files might otherwise expose the system to race conditions that leave the files inconsistent. Also, you might want to protect your news spool area by exporting it read-only, which also requires forwarding to the central machine. C News handles this central machine configuration transparently to the user. When you post an article, your newsreader usually invokes inews to inject the article into the news system. This command runs a number of checks on the article, completes the header, and checks the file server in /etc/news. If this file exists and contains a hostname different from the local host’s name, inews is invoked on that server host via rsh. Since the inews script uses a number of binary commands and support files from C News, you have to either have C News installed locally or mount the news software from the server. For the rsh invocation to work properly, each user who posts news must have an equivalent account on the server system, i.e., one to which she can log in without being asked for a password. Make sure that the hostname given in server literally matches the output of the hostname command on the server machine, or else C News will loop forever in an attempt to deliver the article. We discuss NFS is detail in Chapter 14, The Network File System. Maintenance Tools and Tasks Despite the complexity of C News, a news administrator’s life can be fairly easy; C News provides you with a wide variety of maintenance tools. Some of these are intended to be run regularly from cron, like newsdaily. Using these scripts greatly reduces daily care and feeding requirements of your C News installation. Unless stated otherwise, these commands are located in /usr/lib/news/maint. (Note that you must become user news before invoking these commands. Running them as a superuser may render critical newsfiles inaccessible to C News.): newsdaily The name already says it: run this once a day. It is an important script that helps you keep log files small, retaining copies of each from the last three runs. It also tries to sense anomalies, like stale batches in the incoming and outgoing directories, postings to unknown or moderated newsgroups, etc. Resulting error messages are mailed to the newsmaster. newswatch This script should be run regularly to look for anomalies in the news system, once an hour or so. It is intended to detect problems that will have an immediate effect on the operability of your news system, in which case it mails a trouble report to the newsmaster. Things checked include stale lock files that don’t get removed, unattended input batches, and disk space shortage. I wouldn’t try this on the Internet, either.

If C News finds mismatches between the checkgroups list and the active file, it produces a list of commands that would bring your site up to date and mails it to the news administrator. The output typically looks like this: From news Sun Jan 30 16:18:11 1994 Date: Sun, 30 Jan 94 16:18 MET From: news (News Subsystem) To: usenet Subject: Problems with your active file The following newsgroups are not valid and should be removed. alt.ascii-art bionet.molbio.gene-org comp.windows.x.intrisics de.answers You can do this by executing the commands: /usr/lib/news/maint/delgroup alt.ascii-art /usr/lib/news/maint/delgroup bionet.molbio.gene-org /usr/lib/news/maint/delgroup comp.windows.x.intrisics /usr/lib/news/maint/delgroup de.answers The following newsgroups were missing. comp.binaries.cbm comp.databases.rdb comp.os.geos comp.os.qnx comp.unix.user-friendly misc.legal.moderated news.newsites soc.culture.scientists talk.politics.crypto talk.politics.tibet When you receive a message like this from your news system, don’t believe it automatically. Depending on who sent the checkgroups message, it may lack a few groups or even entire hierarchies; you should be careful about removing any groups. If you find groups are listed as missing that you want to carry at your site, you have to add them using the addgroup script. Save the list of missing groups to a file and feed it to the following little script: #!/bin/sh # WHOIAM=`whoami` if [ “$WHOIAM” != “news” ] then echo “You must run $0 as user ‘news’” >&2 exit 1 fi # cd /usr/lib/news while read group; do if grep -si “^$group[[:space:]].*moderated” newsgroup; then mod=m else mod=y fi /usr/lib/news/maint/addgroup $group $mod done sendsys, version, and senduuname Finally, there are three messages that can be used to find out about the network’s topology. These are sendsys, version, and senduuname. They cause C News to return the sys file to the sender, as well as a software version

This is a log of all error messages created by C News. These messages do not include logs of articles junked due to being sent to an invalid wrong group or other user errors. This file is mailed to the newsmaster (usenet by default) automatically by newsdaily if it is not found empty. errlog is cleared by newsdaily. errlog.o keeps old copies and companions. batchlog This file logs all runs of sendbatches. It is usually of scant interest. It is also attended by news- daily. watchtime This is an empty file created each time newswatch runs. Control Messages The Usenet news protocol knows a special category of articles that evoke certain responses or actions by the news system. These are called control messages. They are recognized by the presence of a Control: field in the article header, which contains the name of the control operation to be performed. There are several types of them, all of which are handled by shell scripts located in /usr/lib/news/ctl. Most of these messages perform their action automatically at the time the article is processed by C News without notifying the newsmaster. By default, only checkgroups messages will be handed to the newsmaster, but you may change this by editing the scripts. The cancel Message The most widely known message is cancel, with which a user can cancel an article sent earlier. This effectively removes the article from the spool directories, if it exists. The cancel message is forwarded to all sites that receive news from the groups affected, regardless of whether the article has been seen already. This takes into account the possibility that the original article has been delayed over the cancellation message. Some news systems allow users to cancel other people’s messages; this is, of course, a definite no-no. newgroup and rmgroup Two messages dealing with creation or removal of newsgroups are the newgroup and rmgroup messages. News- groups below the “usual” hierarchies may be created only after a discussion and voting has been held among Usenet readers. The rules applying to the alt hierarchy allow for something close to anarchy. For more information, see the regular postings in news.announce.newusers and news.announce.newgroups. Never send a new- group or rmgroup message yourself unless you definitely know that you are allowed to. The checkgroups Message checkgroups messages are sent by news administrators to make all sites within a network synchronize their active files with the realities of Usenet. For example, commercial Internet Service Providers might send out such a message to their customers’ sites. Once a month, the “official” checkgroups message for the major hierarchies is posted to comp.announce.newgroups by its moderator. However, it is posted as an ordinary article, not as a control message. To perform the checkgroups operation, save this article to a file, say /tmp/check, remove everything up to the beginning of the control message itself, and feed it to the checkgroups script using the following command: # su news -c “/usr/lib/news/ctl/checkgroups” < /tmp/check This will update your newsgroups file from the new list of groups, adding the groups listed in localgroups. The old newsgroups file will be moved to newsgroups.bac. Note that posting the message locally rarely works, because inews, the command that accepts and posts articles from users, refuses to accept that large an article.

C News does not expire by scanning the newsgroup’s directory, but simply checks the history file if the article is due for expiration.129 If your history file somehow gets out of sync, articles may be around on your disk forever because C News has literally forgotten them.130 You can repair this by using the addmissing script in /usr/lib/news/maint, which will add missing articles to the history file or mkhistory, which rebuilds the entire file from scratch. Don’t forget to become user news before invoking it, or else you will wind up with a history file unreadable by C News. Miscellaneous Files There are a number of files that control the behavior of C News, but are not essential. All of them reside in /etc/news. We describe them briefly here: newsgroups This is a companion file of active that contains a list of each newsgroup name along with a one-line description of its main topic. This file is automatically updated when C News receives a checknews control message. localgroups If you have a lot of local groups, you can keep C News from complaining about them each time you receive a checkgroups message by putting their names and descriptions in this file, just as they would appear in newsgroups. mailpaths This file contains the moderator’s address for each moderated group. Each line contains the group name followed by the moderator’s email address (offset by a tab). Two special entries are provided as defaults: backbone and internet. Both provide, in bang-path notation, the path to the nearest backbone site and the site that understands RFC-822 style addresses (user@host). The default entries are: internet backbone You do not have to change the internet entry if you have eximor sendmail installed; they understand RFC822 addressing. The backbone entry is used whenever a user posts to a moderated group whose moderator is not listed explicitly. If the newsgroup’s name is alt.sewer and the backbone entry contains path!%s, C News will mail the article to path!alt-sewer, hoping that the backbone machine is able to forward the article. To find out which path to use, ask the news-admin at the site that feeds you. As a last resort, you can also use uunet.uu.net!%s. distributions This file is not really a C News file, but is used by some newsreaders and nntpd. It contains the list of distributions recognized by your site and a description of their (intended) effects. For example, Virtual Brewery has the following file: world everywhere in the world local Only local to this site nl Netherlands only mugnet MUGNET only fr France only de Germany only brewery Virtual Brewery only log This file contains a log of all C News activities. It is culled regularly by running newsdaily; copies of the old log files are kept in log.o, log.oo, etc. errlog 129 The article’s date of arrival is kept in the middle field of the history line and given in seconds since January 1, 1970. 130 I don’t know why this happens, but it does from time to time.