SUNET Internet2 Land Speed Record: 69.073 Pbmps

From San Jose, CA, USA to Luleå, Sweden

Swedish University Network

Börje Josefsson 2004-04-14

Background:

SUNET is the organization for the national higher research and education network (NREN) of Sweden. SUNET operates the GigaSunet network, which is built with 10 Gbit/sec DWDM connections in a redundant infrastructure, connecting PoPs in 22 cities, nationwide, and using redundant 2,5 Gbit/sec connections as access towards the universities. It is used by researchers, teachers, students, and administrative personnel on 32 universities and colleges nationwide. In addition to this, some central government museums and external organizations are also connected to the network.

Internet Land Speed Record:

On April 14, 2004, SUNET transferred around 840 Gigabytes of data in less than 30 minutes, using a single TCP stream between one host at the Luleå University of Technology (LTU) in Sweden (close to the Arctic circle), and one host connected to a Sprint PoP in San Jose, CA, USA. The network path used is the GigaSunet backbone - shared with other users of the Swedish universites, and the SprintLink core network, used by all the customers of Sprintlink.

The transfer was done with the ttcp program, available for many different platforms. We have chosen to use NetBSD for our tests, due to the scalability of the TCP code. ttcp is usually included in Unix and Linux systems. Windows users can download a Win32 version from pcusa.

Network setup:

Network path:

The path spans across two continents, Europe and the US, as shown in this picture:

traceroute to 130.242.94.246, 64 hops max, 40 byte packets
1 sl-bbh-sj-12-0-0.sprintlink.net (198.67.139.33) 0.236 ms
2 sl-bb21-sj-4-0.sprintlink.net (198.67.139.245) 0.424 ms
3 sl-bb20-sj-14-0.sprintlink.net (144.232.3.157) 0.575 ms
4 sl-bb20-stk-12-0.sprintlink.net (144.232.20.98) 2.366 ms
5 sl-bb21-stk-14-0.sprintlink.net (144.232.4.233) 2.369 ms
6 sl-bb22-kc-2-0.sprintlink.net (144.232.20.168) 42.294 ms
7 sl-bb21-kc-6-0.sprintlink.net (144.232.2.133) 42.728 ms
8 sl-bb26-fw-13-0.sprintlink.net (144.232.8.63) 54.263 ms
9 sl-bb27-fw-15-0.sprintlink.net (144.232.11.86) 54.265 ms
10 sl-bb20-pen-13-0.sprintlink.net (144.232.8.64) 104.294 ms
11 sl-bb21-pen-14-0.sprintlink.net (144.232.16.34) 104.264 ms
12 sl-bb23-rly-0-0.sprintlink.net (144.232.20.32) 106.555 ms
13 sl-bb27-rly-10-0.sprintlink.net (144.232.14.142) 106.525 ms
14 sl-bb26-rly-13-0.sprintlink.net (144.232.14.181) 106.836 ms
15 sl-bb25-chi-3-0.sprintlink.net (144.232.20.89) 126.406 ms
16 sl-bb24-chi-14-0.sprintlink.net (144.232.26.82) 126.544 ms
17 sl-bb25-nyc-5-0.sprintlink.net (144.232.9.157) 149.763 ms
18 sl-bb20-msq-2-0.sprintlink.net (144.232.20.74) 151.193 ms
19 sl-bb20-tuk-10-0.sprintlink.net (144.232.20.173) 153.001 ms
20 sl-bb21-tuk-15-0.sprintlink.net (144.232.20.133) 152.933 ms
21 sl-bb21-lon-14-0.sprintlink.net (144.232.19.70) 221.801 ms
22 sl-bb23-lon-14-0.sprintlink.net (213.206.128.54) 221.866 ms
23 sl-bb22-lon-15-0.sprintlink.net (213.206.128.160) 237.432 ms
24 sl-bb20-bru-14-0.sprintlink.net (213.206.129.42) 226.842 ms
25 sl-bb21-bru-15-0.sprintlink.net (80.66.128.42) 226.808 ms
26 sl-bb20-ams-14-0.sprintlink.net (213.206.129.45) 401.930 ms
27 sl-bb21-ams-15-0.sprintlink.net (217.149.32.34) 230.289 ms
28 sl-bb20-ham-14-0.sprintlink.net (213.206.129.50) 236.246 ms
29 sl-bb21-ham-15-0.sprintlink.net (217.147.96.46) 236.282 ms
30 sl-bb21-cop-13-0.sprintlink.net (213.206.129.57) 241.014 ms
31 sl-bb20-cop-15-0.sprintlink.net (80.77.64.33) 241.052 ms
32 sl-bb21-olo-13-0.sprintlink.net (213.206.129.72) 346.130 ms
33 sl-bb20-olo-15-0.sprintlink.net (80.77.104.32) 248.705 ms
34 sl-bb20-sto-14-0.sprintlink.net (213.206.129.30) 289.444 ms
35 sl-tst1-sto-0-0.sprintlink.net (213.206.131.10) 256.332 ms
36 stockholm1.POS14.sunet.se (130.242.94.221) 256.336 ms
37 vasteras1-POS4.sunet.se (130.242.82.10) 346.368 ms
38 gavle1-POS4.sunet.se (130.242.81.49) 260.723 ms
39 lulea1-pos0.sunet.se (130.242.81.42) 318.898 ms
40 dino.dc.ltu.se (130.242.94.246) 276.138 ms

All routers in the path are Cisco high-end routers. Note that this is a path shared with other users of the GigaSunet and Sprint networks! The following graph shows one of the links in the GigaSunet core during the day (several transmissions were done) - showing the record-traffic shared with the normal usage.

Results:

According to the Internet2 LSR contest rule #5A, IPv4 TCP single stream, we acheived the following results, using a publically available snapshot of the upcoming version of the 2.0 version of the NetBSD operating system, and using a MTU of 4470 bytes:

838860800000 bytes in 1588 real seconds = 4226 Mbit/sec

The complete output from ttcp during the transmission as seen from the transmitter and the receiver. The test run lasted 1588 seconds (26 minutes, 28 seconds).

A tcpdump output is available for the first 20MB of a transmission, both in raw tcpdump format and as readable tcpdump output.

Internet2 Land speed record submission:

According to contest rule #7, the distance should be calculated as the terrestrial distance between the cities where we do router hops. Referring to the Great Circle Mapper, the distance is 16,343 km (10,157 miles). We have then used the airport of the city in question as it's location.

Record submitted for the IPv4 single stream class is 69.073 Petabit-meters/second (which is a 12% increase of the previous record).

Compared to the previous record, we can note that we acheived this, using less powerful end hosts, with 150% longer distance, and we used only about half the MTU size (which generates heavier CPU-load on the end-hosts). Most notable is perhaps that our result was acheived on the normal GigaSunet and Sprintlink production infrastructures, shared by millions of other users of those networks.

End system hardware and configuration:

The end hosts are off-the-shelf Dell 2650 servers, each with only a single Intel Xeon 2.0 GHz processor, 512 Mbyte of RAM and using the Intel PRO/10GbE LR network adapters.

NetBSD operating system configuration (apart from default configuration, available in the snapshots as of April 2004):

Kernel compile-time parameters:

Sysctl parameters:

Ifconfig settings:

ifconfig dge0 10.0.0.1/30 ip4csum tcp4csum udp4csum link0 link1 mtu 4470 up

Observation:
We noted that it is the PC hardware (excluding the Intel PRO/10GbE network adapter) that is the limiting factor in our setup. The operating system, the network adapter, as well as the network itself, including the routers, are capable of handling more traffic than this, but the PCI-X bus and the memory bandwith in the end hosts are currently the bottlenecks.

Summary, according to Internet2 standards:

14 April 2004

Partners:

Swedish University Network

Special thanks to:

Thanks to:

Also special thanks to Sprint for providing bandwidth, and for facilities and housing for the host in San Jose!

Contact information:

Hans Wallberg CEO, SUNET Hans.Wallberg@sunet.se
Börje Josefsson CTO, SUNET. LSR test coordniator bj@sunet.se
Peter Löthberg Sprintlink LSR coordinator roll@sprint.net
Anders Magnusson LSR technical test manager. ragge@ltu.se