Voodoo in SunetC
Sunet got the opportunity to test Junipers new DWDM card for the QFX switching platform. It was on beta testing run in Europe and we got to steal it for a week before it was going back to the US. There might be some dark magic going on as the internal Juniper project is called Voodoo. The Voodoo card can run different line speeds of 100G, 150G and 200G on the same optical spectrum with different type of modulation/encoding using SD-FEC (Soft Decision Forward Error Correction) with 25% overhead to achieve better reach. Compared to the 100G DWDM cards that Sunet has deployed today in SunetC that are are running QPSK modulation with differential encoding and SD-FEC with 15% overhead. For Sunet it was very interesting to see how far we could push these cards on the SunetC network at those new speeds (modulations). The card is a 6x200G card so succesfully deploying this gives you 1.2Tbit/s of DWDM interconnect between the switches.
We were a bit surprised when Juniper dropped of a huge road-case at the office.
It took four guys to lift it into the datacenter, as our office-DC has two steps and no ramp.
The SunetC optical network consists of DWDM boxes from Adva. With ROADM at drop sites, EDFA and Raman for amplification on the spans. It’s optimized for high OSNR by putting Ramans on all fibers with more than 16dB loss. The system is colorless, so there are no passive or active filters, only passive splitters and combiners. That means that all the optical waves that comes into a site from a degree goes directly to the receiver on the DWDM card. The receiver needs to tune to its specific wave.
A line degree at a drop site looks like this:
From Juniper we got one QFX10008 and two Voodoo cards. As we only had one QFX we needed to connect it to two sites to be able to ”snake” an optical wave around the network as a wave can only pass the same Optical node once. That problem was solved by a 10km darkfiber to the other Sunet pop in Stockholm. Unfortunately we did not have any extra fiber pair between those sites but we did manage to find an optical splitter so we could move over two production links to share the same fiber during the test. To cope with the extra loss of that dark fiber we had to put two EDFA at the end where the Juniper switch was located, one as pre amp and the other as a post amp.
The QFX can do layer-3 but I guess it is mostly a switch as all configured IP-interfaces got the same mac-address. Try to ping another side and that side has the same mac-address. ARP doesn’t like that.
The default configuration of the QFX is to run per port spanning tree. If you connect all 6 ports back to back, five ports are going to go in to block. All solvable, but you tend to forget those things when you normally work with real routers.
We had the Voodoo channels going into the Adva system as alien/external – waves/channels (different terminology is used by different vendors but they all mean the same thing). Each wave was 50GHz wide, we would like to have had the capability to run them at 75GHz to have less impact from all the cascading filters in ROADMs. But Adva is a bit late on that (software) feature so we cant do it yet on our current software.
Since we have OTDR data on all the fibers in the network we know the exact length of each span between each amplifier. Together with the output from the ADVA NMS inventory report for each external channel and a small python script we can combine that data and get the total fiber length for each wave and also the number of amplifier and ROADM-nodes each channel passes.
The Juniper have three different line encodings to play with and six interfaces in each card so we configured the optical network for three paths and ran two channels on each path.
One new thing compared to the old DWDM cards we used is that Juniper separated the optical interface from the Ethernet interface. All the optical part was configured under ot-x/y/z and that interface could have the speed of 100, 150 or 200Gbps. That was then mapped to one, two or three et-x/y/z interfaces. For example, when running at 150Gbps (8QAM) you needed to have two ot- interface going the same path and those together was mapped to three et- interfaces. A bit confusing at first but it makes sense once you have typed the appropriate commands a few times. Also, it helps if you actually read the manual before starting to configure and set up different combinations of encoding and modulation to see what is actually supported or not.
Line side encoding QPSK gives a bit rate of 100Gbps and you could configure it with the default SD-FEC at 25% overhead or the legacy of 15% as well as differential or non-differential encoding. We went for the best with 25% and non-differential encoding. We started on a 300km path through the SunetC network and tested different paths until we ended up with a path of ~4000km. The Adva NMS takes up to 10min to setup a new wave with all the steps of tearing down the old and creating a new with a lot of point and click so it was a bit of tedious work, but compared to configuring it manually in every ROADM its fairly simple.
The Length are in km, the Loss is the combined loss of the fiber from the OTDR measurement in dB. The Gain is the sum of the configured gain of the EDFA and Ramans amps on the path in dB. Raman Gain is the sum of all the Raman amps and that number is part of the Total Gain. The numbers show how many ROADMs, EDFAs and Ramans the span passes. Last column shows the total number of fiber spans on the path, where a span is the fiber between two amps.
This is the actual path through our network with its correct stretches, this might not look like much but Sweden is a long country, if we lay out 4000km on another map it gives a better perspective of scale.
Optical read out from the switch…
The Chromatic dispersion is not negative it’s just that Juniper think it’s a signed INT so it wraps around and becomes negative…. this problem existed in MX aswell before we reported it, now we did it again.
The SNR is the electrical Signal to Noise Ratio and not the optical. The optical is harder to figure out but you can get a good estimate by looking at the Spectrum analyzer graphs below.
We got help from Juniper with collecting the data from the QFX (custom event-scripts) along with some excel magic.
As you can see on the FEC values there are still some margin to the FEC limit on 3.4e-2. In one direction on one of the links we were cutting it a lite bit to close, as you can see the Uncorrected codewords counting up. We didn’t have the time to dig down and find the issue for that, might be as simple as a dirty patch, we did scope all connectors but it’s not always enough.
We tried running QPSK over 4600km but the Voodoo card was not able to sync up, so we are pretty sure we hit maximum feasible distance
This line encoding gives a transfer rate of 150Gbps. As expected was the reach shorter but still impressive at 2338km.
This setup had worse FEC than QPSK but was running totally error free. Indicating that the QPSK with errors where due to something else
This is for us the most interesting line encoding where we get a bit rate of 200Gbps in a 50GHz channel window. The distance is even shorter than 8QAM but still long to reach most places in Sweden from Stockholm.
This was cutting it very close to what the FEC could handle. At least one of the two links where running error free. We had 16QAM running at 1700km on both links error free with a FEC of 2,4E-2 but it is more fun to find the limit.
We connected an EXFO OSA to the splitter that combined all the QFX interfaces and this is the graph of the waves going in to the QFX.
The first three channels are production traffic. The dip in the beginning around 1528nm is caused by the OSC channel filter at 1528nm.
The six channels you see below in the zoomed in graph are
- Channel 1+2: QPSK ~4000km @ 2x 100G
- Channel 3+4: 8QAM ~2300km @ 2x 150G
- Channel 5+6: 16QAM ~1800km @ 2x 200G
We pulled out the fiber to the second port in the switch, this is much faster than turning the laser off and on as the interface is very slow to turn on the laser and tune to its wavelength. The DWDM system have all the amps and ROADM filters open so the thing that we see in the graph are the background noise from all the amps on that channel. The difference between the background noise and the signal power becomes the OSNR (Optical Signal to Noise Ratio).
One interesting effect is that the other channel that goes the same path gets an extra amplification when the (other) channel is removed. This can probably be explained in great detail by some physicists but in short, the Raman light boost the light it hits and with one wave instead of two there is a higher probability that the light from the first wave gets hit by the Raman light and gets an extra boost. Makes sense right? Maybe Juniper is onto something here when calling the productline Voodoo…..
Channel 3 and 4 are shorter (2300km) than 1 and 2 (4000km) and therefore a lower background noise and a better OSNR is observed. As a result, we can run a better or more advanced encoding.
Test against the current card Sunet run, Cordoba
The Juniper 100G DWDM MIC (Codename Cordoba) for the MPC3E MX card should work with the Voodoo card if you set it to SDFEC 15% overhead (25% is default) and QPSK modulation. It didn’t work at the first try and we punched the switch hard for being naughty. The Voodoo card didn’t even send out light and any channels configured for 8QAM went down, I hope that this is just a beta software issue, we punched it again to make it behave. Turns out that you need to change the encoding to “differential” since the standard is non-differential. After that it came up without any problems and the 8QAM channels came up again. Here is a low level command that confirms that we now have Cordoba compatible mode as well.
Why AC400? What does it mean? Its the name of the DSP that Juniper buys to realize their Coherent portfolio and its a Commercial Off The Shelf product from Acacia. This chipset can be found in many other DWDM-equipments aswell as these type of DCI-style switchcards but also in more traditional transponders. Acacia is essentially the optical equivalent of what Broadcom is in the switching-world.
We take these cross-compability issues quite seriously and would really like to prefer vendors to stop playing games. This was a important test for us to make sure that it actually worked and it did after we did some….voodoo on it. In theory any vendors box that has this same chipset should work. And even better is if the vendor also implement staircase-fec which is a generic line-standard that anyone can use and it is being used. Read more about that on Kristian Larssons blog from how Terastream is testing 100G DWDM lineside interop. CLICK ME
More nice graphs for the Voodoo links.
These tests are always fun to make. It proves to us that we did our due diligence when we built our optical network from the bottom-up. The network is optimized for highest possible OSNR and it pays off now when we add signals with advanced modulation, we are able to actually run these stuff up to the maximum spec more or less and whenever we need to add 200G signals we know we are already in a good position to add these without to much hassle.
With TeliaCarrier press-release about running TIP Voyager opto-whitebox from Stockholm to Hamburg on a 1089km stretch between Stockholm and Hamburg on the Acacia 16QAM modulation format (voyager and this card uses the same DSP) we just have one reaction up here in the 1900km club.
// Dennis, Magnus and hugge from SUNET test-labs