The service problems began two weeks ago on Wednesday, November 22nd. There was a main water line break in Kingsport which flooded the underground channels where Sprint had installed their copper wiring. This resulted in, as we understand it, every single telco circuit devoted to data transfer being shorted out. There were physically no existing telco circuits to transfer our services to. We were forced to wait until repairs could be made. This affected some of our out-going connections to the Internet itself, but more importantly, it affected our inter-connectivity to all of our dial-up POP locations where customers dial into. Customers could dial into the equipment, but the equipment had no connectivity to our servers or the Internet. Restoring these circuits took some time, naturally, as Sprint worked through the night to effect repairs. As the individual circuits were brought back on line, we here at MountaiNet also worked through the night to restore connections to all of our equipment. Unfortunately, some of the main inter-connecting circuits could not be brought back up to full working order, likely due to engineering changes within the Sprint network in an effort to restore connectivity.
Finally, rather than continuing to expend resources attempting to repair these existing circuits, a decision was made to change the entire backbone connection for all of Northeast Tennessee to an entirely new set of circuits. Normally a move of this magnitude would take months as all new telco circuits have to be ordered and installed by the local telephone companies. But MountaiNet has been working for the last several months to implement a new high-speed DS-3 backbone (capable of 45Mb roughly equivalent to thirty T-1 circuits) so as to improve Internet access speeds to all users. Recently connectivity to that new DS-3 circuit had been enabled into the Johnson City Central Office of Sprint. Sprint was gracious enough to work with MountaiNet to rush installation of the local connecting loops to this new circuit out to all of our equipment locations. What would have normally taken months was accomplished in days, at least in terms of creating new telco circuits out to all our equipment locations. This enabled us to begin the arduous task of moving all users and equipment over to the new network. Without going too deeply into all the technical details involved in this move, the quick explanation of the steps involved is as follows: A) ALL MountaiNet equipment had to receive new Internet addresses manually (as in driving out to each POP location and configuring all the devices on site) B) each piece of equipment then had to be manually reconfigured to serve out these new addresses to dial-up customers C) all servers had to be reconfigured for new addresses D) all the inter-connecting routing information had to be changed to use the new addresses E) Domain Name Service listings (conversion from the text listings such as www.whatever.com into real Internet addresses normally transparent to users) had to be manually changed so users could actually browse on the Internet once connected...and, finally, F) Internic, the governing body of the Internet, had to be made aware of all these changes. Once completed, all the changes had to be updated and synchronized throughout the Internet, itself. This is done somewhat automatically, as Internet routers periodically update each other of any routing changes so data packet flow is always optimized for best throughput. This router interaction adds additional data traffic across the network. This additional data traffic will temporarily affect network speed until all changes have been updated throughout the entire system.
Even with this simple explanation of all the steps involved, we here at MountaiNet hope all of our users can grasp and appreciate the sheer magnitude of the physical work involved in making all these changes. We had planned to implement this new high-speed backbone after the first of the year anyway to improve service, but in a much more controlled method. Our plans were for a slow, methodical migration of all users over a period of weeks, not days, with much testing involved at each step. Instead MountaiNet was forced to begin implementation these changes virtually overnight and continue working, non-stop, until all equipment and routing issues were resolved. Naturally, much of the available manpower here at MountaiNet was devoted to correcting the circuit problems that existed. With much of our network down, the volume of phone calls here to our office was staggering. We know a good number of the actual calls made here have not been answered due to the overwhelming number of calls. Now that basic connectivity has been restored, if you are experiencing on-going problems other than slow network speed, which we know is still not at full capacity yet, please do not hesitate to contact our technical support staff at 1-(888) 583-7339 or email us at support@mounet.com.
We want everyone to know that we have now established basic connectivity to all equipment and the basic services have been restored to all customers. Please understand the process is not complete. We have only established the very basic inter-connectivity to all equipment. And part of our users are now connecting to our new DS-3 network and some are still on our old network. Patching these two networks together, as we have been forced to do, does adversely affect services somewhat. And, as said before, only the very basic inter-connectivity has been established in some areas. We will continue to work with Sprint to implement more inter-connecting circuits so as to bring all users back up to full speed. This will be done as quickly as humanly possible. In summary, we are back in service...not at full capacity, but we are now working again.
There are still some problems and bandwidth issues to address, but we will continue to work toward establishing full bandwidth to all users as quickly as humanly possible. We can only hope our valued customers can understand that we were not directly responsible for these outages. No MoutaiNet network equipment was ever down during this outage and we fell we have done all that was possible to rectify the problems and restore Internet service. Again, we here at MountaiNet can only offer our sincere apologies as we try to explain the reasons for these outages. The one positive note to all of this is that connectivity to the DS-3 backbone has now been established here locally. This means we can now move forward working to improve our services as we originally intended. While some customers have access to the DS-3 backbone now; once upgrades are completed, all MountaiNet customers will have full access to the largest Internet access pipeline available here in the local area. We certainly hope you all enjoy the upcoming holiday season and join with us to ring in a New Year of high speed Internet access.
Sincerely;
Your friends here at MountaiNet