FidoNet in Africa (Part 2)

THE EVOLUTION OF A NATIONAL NETWORK ________________________________________________________________________ ________________________________________________________________________ (Part 2)

%--------------------------------------------------------------------- Figure 3 -- RSCS - FidoNet - uucp gateway

%---------------------------------------------------------------------

After the two systems had been running in parallel for some time, it became apparent that there was a trade off between their capabilities:

uucp allowed flexibility in addressing by supporting genuine RFC822 format messages, but suffered from an inefficient file transfer protocol.

FidoNet, on the other hand, used the extremely efficient zmodem transfer protocol, but placed awkward restrictions on the format of incoming email addresses.

Despite some initial disagreement, it was recognised that uucp had more advantages than disadvantages, so several techniques were used to try and improve the throughput of the uucp ``g'' protocol. These were batching -- collecting many mail messages into a single mail packet, and software compression of the resulting packet before transmission. These techniques worked well in a low volume environment, but as time passed and more traffic was routed through the uucp link, batches grew larger until compressed batches many megabytes in size were being routinely handled. On a noisy and sometimes unreliable dialup link, these large packets caused severe financial problems because uucp is incapable of restarting a transmission at the point of failure. This particular problem is handled very well by the zmodem protocol used by FidoNet.

To overcome the financial implications of failed transfers, a thresholding algorithm was implemented, that limited the size of batches to 100k bytes. Finally uucp transfer was working as efficiently as could be expected given the shortcomings of long distance international dialup. At no stage did throughput exceed 450 characters per second for uucp, while FidoNet regularly attained 900 characters per second, leading to much soul searching over telephone accounts.

5. TCP/IP

---------

The move to uucp was a breakthrough, and gave a taste of the capabilities of real networking. It gave, for the first time, reliable access to Usenet news and to other network resources. With this access came an explosion of knowledge -- it was possible to communicate in public forums, and to obtain public domain shareware over the network.

One of the unexpected benefits of Usenet news was the discovery of the public domain PCRoute package, which allowed a standard PC to be configured to act as an IP router. Even more interesting was that PCRoute supported SLIP.

This package turned out to be the key that opened up a number of possibilities that had been closed by the sheer cost of commercially available IP routers.

After some experimentation, the first TCP/IP link between two academic sites in South Africa occurred in August 1990, when the University of Cape Town and Rhodes University linked their networks together using PCRoute and 19.2kb serial lines over the existing UNINET-ZA high speed trunks. This experiment was so successful that the University of Natal in Durban soon linked up as well, driven largely by dissatisfaction with the reliability and shortcomings of the Kermit link.

The nature of UNINET-ZA changed -- the old, makeshift methods were de- emphasized, and expertise in TCP/IP was developed.

Initially, growth of the TCP/IP component of UNINET-ZA was slow. After the original three sites were connected, there was a long period when nothing much seemed to happen. However, internally at these sites, there was tremendous background activity.

Public domain software such as nntp, sendmail, smail3, and versions of telnet for IBM PC's was discovered, obtained, configured and tested. Familiarity with domain name servers grew, as well as familiarity with the type of issues that could be expected in future as more sites linked up. The problems of configuring a stand alone internet, isolated from the real root name servers, were solved. Local area networking became cost effective, and the use of TCP/IP within the universities grew rapidly.

%--------------------------------------------------------------------- Figure 4 -- RSCS - FidoNet - uucp - TCP/IP gateway.

%---------------------------------------------------------------------

At this stage, UNINET-ZA was still running RSCS, FidoNet, uucp and all the Kermit links in addition to TCP/IP. DECNet gateways at the University of Cape Town and the University of Natal in Durban, together with the increasing use of PCRoute, meant that more and more sites were accessible. Critical mass had been achieved, and the benefits of wide area networking to the wider community were becoming self evident.

Administratively, though, there were complications with the large numbers of sites wishing to get connected and it became apparent that a decision would have to be made as to which protocol should be officially supported as the addressing structures in each one were fundamentally incompatible.

The gateway at Rhodes had by now evolved into something that would transfer mail between multiple protocols. It had grown unwieldy, and was increasingly difficult to maintain. The partial support for RFC822 and total lack of support for RFC821 in the locally developed software caused increasing problems as real domain style addresses began to appear.

There was only one sensible decision to be taken, and that was to switch entirely to the domain naming system in the context of TCP/IP. This was the death knell for the protocols that could not adapt.

6. Internet

-----------

By July 1991 several things had happened in the development of UNINET- ZA. The mail and Usenet news volumes had grown to such an extent that the cost of the monthly telephone account was well in excess of the cost of a dedicated line, and the decision was made to set up a permanent link.

Most important, the time was right politically -- the Comprehensive Anti-Apartheid Act had just been repealed, and some of the restrictions on contact with South Africa had fallen away. It was possible to do business with NSFNet, and arrange for access to the Internet at last.

Negotiations about dedicated lines began in earnest, and two options presented themselves:

a 9600 baud analogue line using public domain ka9q software for the routers.

a 56kb digital line using Cisco routers.

The second option was temporarily out of the question for monetary reasons, and it was decided to proceed with the cheaper option, on the premise that, while far from ideal, it would provide a valuable starting point and give experience in managing a national network.

For the sake of continuity, the connection was to be made to RainNet, a regional network in Portland, Oregon, who were connected in turn to Alternet.

The physical process of connecting two remote TCP/IP networks was well understood, and was not complex. What did give some cause for concern was that these two networks had their own separate root name servers -- a dummy one on the South African side, and real ones in the U.S. For the separate networks to link up correctly, the dummy root servers would have to be eliminated.

To minimize disruption on the South African side, and to avoid disruption entirely on the Internet, the cutover was carried out in several independent but sequential steps:

The original uucp mail gateways were linked, simply replacing the uucp transport with an smtp transport. Routing files forced delivery of mail to the gateway machines on either end of the link.

Router blocks between RainNet and Alternet were removed, allowing FTP and telnet access via IP number to a restricted number of sites.

Router blocks between Alternet and NSFNet were removed, allowing access by IP number to the entire Internet.

A copy of the real root domain information was copied to the system running the dummy root domain, allowing peer to peer outgoing SMTP.

After a stabilisation period, and considerable double checking, the .ZA zone was integrated into the root servers and peer to peer incoming SMTP became possible.

Planned, but overlooked for an extended period due to a series of internal misunderstandings, was the registration of the in-addr.arpa domains for UNINET-ZA sites.

This process was carried out over several weeks, starting in November 1991, and ran smoothly with the notable exception of traffic volumes, which grew exponentially as users became aware that access to mail based archive servers was no longer being artificially blocked at a central mail gateway. Fortunately the summer holidays interfered with this discovery, allowing the cutover to be completed.

At the start of the new academic year, in mid February 1992, traffic volumes again increased dramatically, highlighting several problem areas:

static default routes to [0.0.0.0] had been configured, through ignorance, into PCRouters all over the country.

many name server configurations were unhealthy.

the RIP implementation used in PCRoute and ka9q made it easy to set up router loops that could bring UNINET-ZA to its knees.

errors on the 9600 baud line caused the modems and routers to hang.

Tracking down these problems and rectifying them, in conjunction with increasing user expectations, has proved a frustrating and thankless task which is still in progress.

7. Present Status

------------------

Response time over the international link has always been poor. Under quiescent conditions, the end to end ping time is 960 msec -- made up of 640 msec satellite delay, and 320 msec due to the 9600 baud link. As soon as there is traffic on the line, ping times degenerate, with 30 second ping times being considered ``good''. Things can get even worse, with ping times in excess of 100 seconds being common. It is surprising that the TCP/IP protocols work at all under such circumstances, as the original design could not have forseen their use when ping times are measured in minutes.

After a promising start, the link has become a victim of its own success, and, for all practical purposes is unusable for extended periods.

Plans are underway to:

replace the ka9q routers with Cisco routers, improving the reliability of the connection.

replace the digital multiplexers and PCRouters in use on the internal trunks with Cisco routers interfacing directly to point to point 64kb digital circuits.

upgrading the international link to 56 kb, increasing the bandwidth and thus improving response times to the point where telnet and FTP become viable.

So near, yet so far... and so frustrating!

8. Lessons and Conclusions

--------------------------

The implementation of UNINET-ZA has been an exercise in boot-strapping from a position of ignorance and isolation to a situation where it can be compared to that of many regional networks in the U.S.A.

Much has been learned over the last five years, and these lessons can best be summarized into a series of unrelated and sometimes self-evident points. While some are offered tongue in cheek, others are serious:

Where there is the will, there is a way.

Simple beginnings should not be overlooked -- they can be used to prove that more complex tasks are possible.

Once you have access to information, it becomes easy to discover what you should have done in the first place.

You get what you pay for -- but very often this is good enough.

Wide area networking makes local area networking worthwhile.

The least expected (and thus the least understanding) users become totally reliant on the full time availability of the network.

The identical silly questions have to be answered for each new generation of users and network administrators.

Parkinson's Law is directly applicable to network bandwidth -- no matter how much bandwidth is available, it will be used up.

Mail based archive servers and effective utilisation of bandwidth are mutually exclusive concepts.

Wide area networking is a full time job, and needs support infrastructure.

Networking breaks down barriers between people and cultures.

The text books were right all along -- it is possible to make theory work in the real world.

And last, but not least, is the realization that networking, while sometimes aggravating, can also be lots of fun.

=====================================================================

10. A more cheerful postscript

-------------------------------

In early June of 1992, commercial Cisco routers were installed, replacing the ka9q routers used on the international link. There was an immediate, and dramatic improvement in reliability and response times. The causes of the bad response times using the ka9q routers was never discovered, but interestingly, measurements of TCP/IP traffic on the United Kingdom <-> U.S. link have shown that badly configured Domain Name Servers can use up large quantities of bandwidth. This ties in with our experiences, but is not a complete explanation. A passing comment, that ka9q was originally developed for amateur packet radio use where complete loss of signal for extended periods can be experienced, and that it thus buffers rather than drops packets, may be another part of the explanation.

Cisco routers have also been installed on most of the major internal UNINET-ZA trunks, which are now running at 64kb. This has improved response times to the point where access to specialised computer systems at remote campuses is very workable.

Finally, a contract has been signed to provide a 56kb link to the NSFNet. This will be installed in late 1992.

=======================================================================

-- F.F. Jacot Guillarmod-Computing Centre-Rhodes University- Grahamstown Internet: ccfj@hippo.ru.ac.za Telephone: +27 461 22023 xt 284