Dark Fiber Testing and Debugging Process

Latest revision as of 19:55, 12 November 2012

Dark Fiber is where ACD provides physical fiber cable connectivity between two or more sites, without providing the electronics necessary to light the cable. ACD generally does not operate or maintain the equipment for the customer, although ACD usually is needed to assist the customer to determine if the connection issue is actually a fiber issue (ACD's responsiblity), or customer equipment (Customers responsibility). ACD should be proactive in facilitating resolution and testing. ACD has the necessary equipment to effectively test fiber strands, and usually customers do not due to the expense of the equipment.

[edit] Point of Demarcation and Responsibility

ACD's network elements:

1. Fiber Cable that is outside of the building, conduit, pole attachments and entrance facilities into the building.
   1. In some instances where ACD is utilitizing the customer provided entrance facilities (i.e. conduit that on the customer grounds, or inside the customers building) the customer has provided us with entrance facilities, and is having ACD use them.
2. Fiber Cable coming into the building; and is prior to the fiber termination box.
3. The fiber termination box, and the connectors directly attached to the fiber termination box.

Customer's network elements:

1. Patch cables from the fiber termination box to their equipment.
2. Network Switches/Routers
3. Optics; typically these are pluggable modules, such as GBICs, SFPs, Xenpaks and other industry names for optical Modules.

[edit] Problem Reporting by Customer

When the customer contacts ACD we need to quickly determine the issue the customer is reporting.

1. Complete Outage: Where there is no connectivity between sites.
   1. Is the outage affecting all fibers between a site or only specific fibers?
   2. ACD needs to know which fibers are having the issues.
2. Transmission Errors: Transmission occurring, but CRC errors on customers interfaces are accumulating
   1. Is this occurring on all fibers between a site or only specific fibers?
   2. Are the errors on both on the TX and RX sides of the interfaces on both sides?
3. Intermittent Outage: Transmission is occurring, but is stopping occassionally.
   1. What time of day?
   2. Any pattern to the intermittency?
4. Low Light Level: The light emitted from the customer's laser is low on the RX end?
   1. What is the DB attenuation of the fiber strand that is having a problem?
   2. What is the loss ratio from the Launch Laser to the Terminating location?
   3. Is the low light level occurring on all fiber strands?

Additional Questions

The more information we have the quicker we can isolate the issue and get the connection up and running. These other question that help ACD reduce the time of its own debugging process are:

1. Did the problem report start when the Customer was performing any maintenance?
2. Did the problem report start when ACD was performing any maintenance that the customer is aware of (i.e. during a posted maintenance window relevant to the customers services)?
3. What testing has the customer done on their equipment?
4. What testing has the customer done on the fiber strands provided by ACD?
   1. Has the customer put a light meter on the RX end of the strand that is having a problem?
   2. Has the customer run an OTDR test?
   3. Has the customer swapped any equipment?

[edit] Possible Reasons for Customer Reported Problems

ACD is committed to resolving all customer issues, and assisting to resolve issues that are not related to ACD caused issues. We want the customer's connections with ACD to be resolved promptly. ACD will deploy personnel proactively to examine customer equipment and connections to isolate where the error is, even if we initially suspect that the issue may not be on ACD's provided dark fiber.

Often times it takes both parties to correctly isolate where the problem is, particularly on issues that are on the customers equipment. Usually the customer does not have the equipment in inventory to do dark fiber testing. Therefore, it is necessary that ACD be proactive in deploying employees with test sets that can determine if there is an issue on ACD fiber or Customer Equipment.

1. Customer Reported Complete Outage: A complete outage is where there is no transmission between the sites, and is the most common sort of outage.
   1. Outage that is due to problems with ACD's network elements:
      1. Storm/Weather damage to the cable
      2. Tree Trimming damage to our cable. Pole owners do not typically notify us of tree trimming activity.
      3. Utility or Contractor damage to the cable.
         1. Sometimes other utility contractors accidentally damage our cable when performing work on another cable or doing underground work. ACD typically does not know about this work beforehand.
      4. ACD caused a outage during a splicing. ACD mis-spliced the fibers on a cable.
         1. An ACD maintenance notice should have been posted on the same cable the customer is using.
         2. ACD employee mis-identified the fiber strands on the cable, and that is the cause of the error.
      5. Outage at the fiber panel.
         1. Usually due to too much pulling of fiber jumper cables on the fiber panel, or mishandling by ACD or Customer. Reseating or replacing jumpers often will fix this problem.
         2. Some fiber panels are used to service multiple customers, and ACD or the end-user customer has inappropriately moved or damaged jumpers.
         3. ACD will typically not have a maintenance notice posted on customer facing access to the fiber panel.
   2. Outage that is due to problems with Customer's network elements ACD is responsible for proactively participating and resolving the issue to make sure the customer is back up and running as soon as possible.
      1. Router/Switch/SONET Equipment Issue.
         1. Confirm with the customer that
      2. Equipment failed or not powered on.
      3. GBIC/SFP/Xenpak is defective.
         1. We find that this the most common reason for outage. Lasers generally degrade (signal weakens) over time an this should be one of the first items to be looked at. Lasers tend to be high failure rate objects.
         2. ACD often does not have the same inventory of GBIC/SFP/Xenpak as the customer uses, however if we do, provide them to the customer, for eventually replacement of billing to get them back up and running. Many of the products are unique, and therefore swapping out lasers on both ends will often be necessary.
      4. Mis-configuration of equipment.
         1. Usually a maintenance notice will be posted by the customer.
      5. Jumper cables damaged.
2. Customer Reported Transmission Errors
   1. Outage that is due to problems with ACD Network Elements = Include all issues on Customer Reported Complete Outage
   2. Outage that is due to problems with Customer's network elements ACD is responsible for proactively participating and resolving the issue to make sure the customer is back up and running as soon as possible.
      1. Transmission errors are usually found to be on customer equipment.
         1. If light levels are low on the transmitting customer lasers, the errors can be on ACD's fiber if the transmitting lasers are at the edge or margin of the distance that they are supposed to be able to go.
         2. Generally ACD recommends that the customer upgrade fiber optic lasers to stronger levels if they are near the margin of DB loss of recievers.
      2. Jumpers not being placed fully within the jacks is a common reason why we see transmission errors.
         1. This often occurs when jumpers are yanked or pulled excessively and are pulled or stretched from the Fiber panel.
         2. Fiber Panel SC-SC or LC-LC modules have occasionally been at fault
3. Customer Reported Intermittent Outage
   1. Outage that is due to problems with ACD Network Elements = Include all issues on Customer Reported Complete Outage & Customer Reported Transmission Errors
      1. Cable is streached somewhere, possibly in the splice case, possibly at the fiber panel.
      2. Need to determine if the issue is weather related.
   2. Outage that is due to problems with Customer Network Elements = Include all issues on Customer Reported Complete Outage & Customer Reported Transmission Errors
4. Customer Reported Low Light Levels:
   1. Outage that is due to problems with ACD Network Elements = Include all previous sections.
      1. Special care is needed to determine the source customer laser strength (on bothend), and calculate the DB loss of the link based upon the source strength of the connection, compared to the outside plant readings.
      2. Special care should reviewed on the fiber panel modules, and inside the fiber panel to ensure that the fiber strands are loose and not streached.
   2. Outage that is due to problems with Customer Network Elements = Include all previous sections.
      1. Pay special attention to the strength of any attenuators that the customer has placed on the network.
      2. Pay special attention to the comparative strength of pair of lasers that are used on the network. Often laser strength degrades over time on GBIC/SFP/Xenpak and other optics, and the degradation may be at different rates, and one end of the link may not have enough laser stregth to push through the natural loss on the cable.

[edit] Impact of ACD testing

In order to test dark fiber strands that are having issues, the customer equipment needs to be disconnected. There is no way to test fibers without disconnecting equipment, therefore any request should be with the understanding that it will be service affecting, at least on the fiber strands that the customer requests testing.

If there are spare adjacent fibers that are on the same cable, we can test these for continuity of the overall fiber cable. If continuity is good on the spare fibers, this can mean that the overall cable is not fully cut. If the spare strands are functional, equipment can be reloacted to the spare cable.

The customer should advise ACD:

1. Invasive Testing: That they authorize or not authorize ACD to perform immediate intrusive testing.
   1. The fibers they authorize testing on means an existing connection will be down during the testing, if it is not already down.
   2. ACD can test adjacent spare fiber to determine overall cable integrity, the customer should advise if they authorize intrusive testing on spare fibers.
2. Maintenance Window Testing: Customer may request that ACD can perform testing during a maintenance window.
   1. ACD's ability to initiate repairs will not start in most circumstances until testing with an OTDR can be performed.

[edit] Equipment and Documentation Required for Testing

1. Light Meter: A light meter is a devices that is able to read the light strength from the emitting laser. The light meter tells whether the laser light being emitted is strong or weak, and within the range of the light sensor to read the light. As light travels down a fiber cable, the strength of light is slowly degraded.
   1. ACD typically sets the light meter to 1550 wavelengths, however the meter should always be set to the wavelegth of the transmitting laser. Usually this is 1310 or 1550 for most equipment nowadays.
2. OTDR Meter: A Optical Time Domain Reflectometer (OTDR) is an expensive device that sends a series of signals down a fiber strand to determine a number of aspects about the fiber cable. A OTDR device can tell where a) A bad splice exists b) Potentially where there is damage that is decreasing the strength of the light signal, c) determine the length of the fiber strand. This is most relevant if the length is less than the actual length of a cable, this means that there is a break in the cable somewhere.
   1. ACD sets OTDR testing equipment to 1550 nanometers wavelengths. Variation in the the wavelength selection will substantially adjust the data results.
3. OTDR Documentation: OTDR documentation are the documents that are created at the time of delivery of the fiber service to the customer. This documentation lists the characteristics of each individual fiber strand, just after it was built. By comparing the historical OTDR readings to the current OTDR reading it can be determined if there is now a difference in the cable. Only by knowing the difference between OTDR documentation at the time of delivery and current OTDR test results, will the problem likely be isolated.
4. As-Built Drawings: As built drawings are documents that describe the construction performed by ACD. These drawings detail the overall route of the cable, including poles that the cable is attached to, handholes, splice cases, fiber termination panels, and whether the fiber is above ground or below ground.

[edit] Step by Step Troubleshooting Process

1. Troubleshooting process
   1. If the customer allows invasive testing:
      1. Go to SITE A, place a light meter on the live RX fiber (the fiber with the light), record the reading, the reading should be between -4DB margin and -22DB. While driving to the sites, if possible, the As-built route should be driven to check for any obvious outside plant damage.
         1. If outside this range may signify that the light is either too hot or too cold for the sensor on the customers equipment.
            1. Equipment can operate outside of this range, so if it is outside this range, it does not mean that it is the primary issue, just that this is something to review and potentially adjust.
         2. In general it is best to have the DB RX range be between -8 to -16 or so.
      2. At SITE A, The light meter should be placed on the the live TX fiber (from the customers local equipment). The reading here should confirm that the customer's laser at SITE A is sending light.
      3. The light strength should be adequate to transmit the light minus the expected DB loss of the fiber at the customer's site.
      4. At SITE A, Place OTDR meter on the fiber panel and shoot all fibers to the customer SITE B.
         1. Record all information, and compare it to the base line documentation we have when the fiber was constructed.
         2. All data should be the same, if there is substantial variation, then the fiber cable is at fault.
            1. If there are spare strands, and the existing OTDR readings are fine on the spare strands, but not fine on the existing operational strands, relocate the customer to spare strands temporarily to get them operational. Both ends will need to be visited to ensure to relocate the customer.
      5. If no issues are found on the light meter or the OTDR, proceed to SITE B, and perform the same testing performed aT SITE A
         1. While en-route between sites, if possible it would be preferable that the route taken to drive is the same route as As-builts to check for any obvious outside plant damage.
         2. Light meter testing should be performed on the RX fiber and the TX fibers.
         3. The RX level at SITE B should equal the output of the laser at SITE A minus the DB loss on the OTDR report. If you go back to SITE A the same data should be the same or similar, and within -2 to -22 db range, and preferably within the -6 to -16 range with the loss of fibers.
            1. Pay special attention to any attennuators that may be on the customers equipment, as the attenuators substantially adjust the strength of the light. attenuators are occassionaly the root issue, particularily if there is issue with CRC errors
      6. If no issues are found on the fiber cable, all the same testing should be performed with and through the customer's patch cords and attenuators. Testing should be performed with the light meter and the OTDR to detect any issues related to loss that could occur through the fiber patch cords, attennuators and/or the fiber panels.
   2. If the customer does not allow invasive testing:
      1. If there are spare fibers, then we can still go to the site and test on the spare, unconnected fiber ports. OTDR readings should be taken at both SITE A and SITE B and compared to the base line construction OTDR readings.
         1. If the results show clean OTDR shots, the outage is either due to the individual strands that we provide are bad (i.e. the particular strands that the customer is live on), or it is the customers equipment, patch cords or fiber panels. If possible test from the customer fiber panel (if it exists). If the OTDR results are not clean, then there is likely damage to the overall fiber cable, and Outside Plant Repairs should be started.
         2. If the results show clean at both sites, the route should be driven, inspecting for any obvious damage.