Tales from the crypt, a first look into the new SUNET

av Fredrik "Hugge" Korsbäck den 29 Okt 2015

Now its time to kick some life into this blog again. All tenders on equipment is done and we at SUNET has a pretty good idea on how the network will look like and what components the network will be built upon, finally we get to take on some gloves and working boots and get to work. Since the new network will be quite far from any whitepaper or schoolbook I have ever read on how you build a classical carrier network there is a lot of proof-of-concept and labs that needs to happen with the vendors to actually make sure that the the things we have bought will actually work and live up to our expectations from the tenders.

To make a very long story short SUNET will build a national carrier-network spanning through the whole of Sweden on leased darkfibre. The optical network is built and owned by SUNET and the unorthodox thing that we will do is that we we will not put any transponders in the optical line-system.

Transponder you say? Whats that?

When you build a network today, you commonly use Ethernet as the protocol of choice for communication between devices, the benefits are many, it’s cheap and low complex and it’s been around since the dawn of time. However when building long-distance networks we need a little bit more help of the link-layer protocol. When the dinosaurs roamed the earth we used SONET/SDH as the protocol of choice, it was de-facto at big telcos and we all got used to all its oddities and quirks, but it was very helpful when transporting data over long dinstances. OTN or G.709 as its called in ITU-T is the successor of SONET/SDH but has better support and better understanding on how WDM (wavelength division multiplexing) works and has a lot of features for optical networking built into the protocol. FEC (forward error correction) is one of the most important parts of the OTN-protocol and is one of the major factors on why an OTN-framed signal can reach much further then its Ethernet brethren’s.

OTN is usually implemented as a wrapper around your Ethernetframes and this is where the Transponder comes into the picture. The transponder accepts a regular Ethernet-signal on one side and then wraps it into an OTN-frame and sets an appropriate wavelength that fits into a DWDM-topology and shoots it out on the other side of the card. In SUNETs case we will build the network with fully coherent receivers as well which does compensation for both CD (Chromatic Dispersion) and PMD (Polarization Mode Dispersion) electronically in the Interface. This means that the network we are building does not have need for DCM (Dispersion Compensation Modules), which is the classical way of controlling CD. A DCM is essentially just a big spool of fibre with a special type of characteristic that can compensate the dispersion; this adds a lot of length to the fibrespan, which then adds attenuation.

Analog CFP2
1x100G Coherent.

As I said before the traditional way of using Transponders is to put them into a shelf in your optical system, usually a DWDM-system of some kind. So from the endpoint (a router or a switch perhaps) you use a regular Ethernet interface with regular fibre or cooper interface, plugs it into the Transponder which then wraps the Ethernet-frames inside OTN-framing. To build a point-to-point link we then need to use 6 physical interfaces and atleast 4 different line-cards. In the new SUNET we will use brand new technology that enables us to put these transponder-functionalities right into the router instead. This means that the outgoing interface from the router will put the payload into OTN-frames directly and have optical capabilities to set the correct wavelength and these optics will as I said before also be Coherent to compensate onboard for CD and PMD. This was done by a few company’s in the 10Gbps era but now that the only logical choice for a carrier is to build with 100Gbps we are quite early in the adoption of this, so early that the only cards available on the market still has the “BETA” stickers on them…

DCM module
The current lab-node

In the SUNET-datacentre on Tulegatan 11 a test platform has been put together to really test all the equipment that has been brought in. The coherent-cards especially needs to tested out very carefully since these are fresh from the factory and still considered to be in the acceptance phase, SUNET is among the first ones in the world to try this kind of product out. Engineers from Juniper have been on site during the whole test procedure to be able to fix problems we run into as quickly as possible since there isn’t much room in this project for failures. Our little lab consists currently of two MX480-routers (what we will use as customer placed equipment) These two are connected together through the Coherent 100Gbps interfaces which we have put about 90km of fibres inbetween to simulate what a real network-stretch could look like in the real network. In the routers we also have multiple 100Gbps Ethernet cards and 10x10Gbps cards and these we have connected a packet-cannon (EXFO FTB-500 1x100G) that make sure that the routers is constantly being feed with 100Gbps of traffic coming in on the client-ports.

Almost 90km of fibre in spools and DCM's
The router interfaces

The network will run for a few days with full traffic saturation on all ports and we assume that not a single error or dropped packet is noticed anywhere since this is just to do acceptance of the routing-platform. If we start to get faults this early in the Proof-Of-Concept it would be disastrous. In Week 45 we will get a complete test-system of the optical network with ROADMs and amplifiers as well and it will be tested all through Week 45 and then moved onto a real fibrespan between Västerås and Stockholm sometime in week 46 or 47. It will be especially interesting to see if we are able to run a Coherent 100G DWDM signal in the same fibre as a regular 100G ER4 and multiple 10Gbps DWDM ER signals and not get any crosstalk or non-linear behavior.

We will also try to break the network as much as we can. Drop spools of fibre on the floor, maybe slam a few rackdoors on cabling and inject poor wavelengths into the system to see what happens. Maybe my dog can come and eat abit on the fibre aswell.

Network Map

It will also be interesting to see which SUNET that first can reach these kind of numbers.

jtac@mx41-re0> show interfaces et-2/0/0 | match rate
Input rate : 95470380480 bps (23491727 pps)
Output rate : 95470364128 bps (23491723 pps)



Fredrik "Hugge" Korsbäck

Network architect and chaosmonkey for AS1653 and AS2603. Fluent in BGP hugge@nordu.net