Sites and fibers being delivered.

av Fredrik "Hugge" Korsbäck den 3 Mar 2016

Hi again.

The last few weeks the SUNET network rollout-team has been very busy driving around Sweden, we have received delivery-confirmations on a lot of sites and fibers from our suppliers and we want to be first on site to inspect and perform OTDR-measurements on delivered fibers. There is a prerequisite from the tender that every fiber delivered to us should come with a complete bi-directional OTDR-measurement before being accepted in production. We have these already since a year back, and to follow-up on this we did a few unannounced sample-test of both OTDR-measurements and how the actual sites looks and the results was not living up to our high demands. This made us take the decision of sending teams around and go and visit every site and inspect every fiber that is built for us before we accept them as finalized  and delivered from our vendor. With 100+ sites and multiple fibers going out from every site, it’s quite a hefty job. Luckily the sites and optical paths is delivered in batches so we can tick them off one by one.

First sites and optical paths to be delivered, alot of driving for two days.
First sites and optical paths to be delivered, alot of driving for two days.

Oh and also, this inspection and measurements needs to be carried out before our installation-team gets on-site, which is on a rolling-schedule from week 8 and onward to install the optical and routed backbone of the new SUNET. Doing measurements with equipment installed and light on the fibre take twice the amount of time since both ends needs to be disconneted to be able to perform measurements. So we need to be ahead not only because of this reason, but we need to be ahead so we run into any problems before our sub-contractor do since we do not want them standing still on a tightly planned network rollout due to something as simple as the lock to the site not working properly.

Toolbox prepared
Driving conditions = excellent.
One of our nodes south of Gävle, first stop of the journey.

Most of our fiberplant is being delivered through aerial fiber hanging in the powerlines in the Swedish grid, this means that sub-stations and powerplants is natural breakingpoints where a amplifier or a drop-site could be placed.

In Gävle we will place routers and drop wavelengths to HIG (Högskolan i Gävle) so the core-site consists of two racks, movable on wheels so we can get behind the racks.
Fusing and power-feeds in working condition and equipped as specified.
ODF in Gävle, APC connectors is used everywhere to minimize reflection.

Fibers are quite sensitive to physical damage. The cable itself (if its a regular patch-cable) can withstand quite the beating (as shown in one of our videos before). But the connectors is very fragile, dropping a cable on the floor and with a unlucky hit on the connector it can be ruined forever. This is not uncommon and below is the result of a OTDR where we suspect one of the connectors being bad, the measurement dies just after a few meters from leaving the OTDR-tool.

OTDR measurement on one fiber of the pair going northbound, distance is as expected and measurement looks good overall.
OTDR measurement on the other fiber in the same pair as the previous picture, here we can clearly see the fiber is cut somewhere very close to the site since the dampening is huge after the first connector

When we do, or when someone else do any type of work, maintenance or installation of our fibers it’s mandatory to put a microscope on the fiber, EVERYTIME someone removes a cable from anything in the optical fabric, doesn’t matter if you just disconnect it to change TX and RX, scope and clean if necessary is mandatory. On most paths we will have RAMAN amplification and if there is any type of dirt when putting the connector together will most likely result in a permanent damage, an etching in the glass. Since the light is very strong close to a RAMAN-amplifier.

Below is a microscopic-view inside of the RX-side of a amplifier (remember that RAMAN pumps light in the RX-direction). Here someone has plugged in a fiber and managed to get some dirt inside or it was like this when coming from the factory, we dont know. This resulted in a permanent etch in the amplifier which caused a 10dB loss.. Amplifier accepted as DOA by vendor ADVA.

Uncleanable etch on a amplifier.
Uncleanable etch on a amplifier.

For the first day i managed to squeeze in all 7 sites between Stockholm and Sundsvall.

All setup at the 5th site of the day, just south of Sundsvall
Outside of a amplifier-hut, here the 4G-antenna can be seen where our OutOfBand-network comes in.
Big thanks to whoever visited this site before me....
Every site has battery-backup, expiry date and functionality of UPS and batteries is tested at every visit.
Coresite in Sundsvall
In rack power distribution in Sundsvall, where there will be a core router and DWDM ROADM nodes
Fire-suppression in Sundsvall.
Sundsvall is located deep in a bunker, indoor base-station to get cell-reception.

Day 2 i headed out from Sundsvall and headed inwards the country and north, passing two power-stations north of Östersund to finish off in Östersund and take a flight back home.

Day2 started 05:30 to make sure i didn't miss the flight in Östersund
Fact: Reindeers does not care about hitting the horn, nor do they care if you poke them with the car. 15min delay 🙁
All of our sites is DC-powered and for that, we need rectifiers.
All measurements takes about 30 minutes per site, and is mostly about waiting, luckily i had 4G reception and could stream X-Files.

All in all the inspections and sites was better than what we have seen before in the more southern part of Sweden. There was a few access-problems (the road to the site in Kilforsen was completely oversnowed with 1.2m of snow for example), some broken patches and some minor issues here and there in terms of power and fusing but nothing that will delay us heavily.

Reports is being sent to the vendor (Tele2) constantly to make sure they have the time to fix our issues before we come with a truck full of equipment to every site and expect to be able to turn up links. We will continue to do these kind of measurements and inspections on a rolling schedule throughout March and April and if someone is interested from our community to tag along when we are in the vicinity, feel free to contact us.

Fredrik "Hugge" Korsbäck

Network architect and chaosmonkey for AS1653 and AS2603. Fluent in BGP hugge@nordu.net