All of our equipment except for our core router in Dallas is currently offline. We are currently investigating why this is; it may be a failed PDU or failed switch. We will continue to work to bring everything back up ASAP.
Update @ 9:16pm CDT: We submitted a ticket with the facility prior to posting the original event. We continue to wait for them to call us back. We will quickly know more about the culprit and be able to proceed in restoring services after they call us.
Update @ 10:16pm CDT: The facility called us and we have been working with them. The problem is a switch failure. We are having them connect up a spare switch that we have on-site, but swapping it in will take time, due to having to unmount/remount and move all of the cables. We and they will continue to devote all our attention to this and get everything up again as quickly as we can.
Update @ 12:16am CDT on 5/22: We have been waiting for the facility to call us back for the last two hours -- they indicated that their plan was to assess the work and then call back. We are attempting to contact them again to find out what is going on.
Update @ 12:46am CDT on 5/22: The facility technician says that he took an unauthorized lunch break in the middle of our work, which was off-the-charts unacceptable and we will be escalating. He has resumed work in moving the switch. We are trying to impress upon him the importance of getting the location back up as quickly as possible, and then doing whatever bookkeeping is necessary afterward.
Update @ 1:42am CDT on 5/22: Work continues on moving the switch. Apparently physical considerations in the cabinet have forced the technician to make lots of extra changes. We are nearing the end of the work.
Update @ 3:24am CDT on 5/22: The location is back online, though not at full capacity. We will be overnighting some replacement hardware to Dallas on Wednesday to get it back up to full strength, and scheduling a maintenance event for its installation.