April 1 was yet another in a series of painful and embarrassing outages we have experienced since deploying the 3Com (was US Robotics) "state of the art" soft-switch, designed to improve performance and reliability of our dial-up network. While our experience has been anything but improved and certainly not reliable, we do still believe that in the long run these problems will be fixed and the results will be worth the wait. Here is why:

1. Our goal, when the network is operating properly, is to achieve connect speeds which are increased approximately 25% or more.
2. Modem Time to Answer (when trunks are not blocked or busy) is virtually instantaneous. (The modems answer before the caller hears a ring.)
3. The full network is capable of V.90 (and ultimately V.92) yielding higher speeds. We can now more readily continue such upgrades, which in some cases would not have been previously possible.
4. We have re-built the network in such a way as to make it scalable and sustainable. As most everyone is probably aware, the scene has changed since the demise of the technology stocks. Companies like SoVerNet remain competitive only if they continue to change and evolve as well. We believe the changes we have made are essential to survive in the ISP business.

In brief, here is what has happened.

• Mid-2001: we installed a new telephone/data switch designed to handle all of the dial-up Internet calls throughout the state. It was turned on and tested through the fall. It looked good.
  
  
• December–January: we moved out of our remote locations throughout the state and consolidated our dial-up network equipment to this new equipment. This involved moving hundreds of phone lines and all of the equipment related to it. Verizon indicated that this was one of the largest such projects in Vermont's telephone history. It took longer and had more problems than anyone anticipated.
  
  
• February–Present: we have been dealing with the fallout on two fronts:
  1. the new switch, when faced with the tremendous load of our more than 20,000 customers experienced new problems not encountered during testing.
  2. the changes in the layout of the network, namely where the phone lines go, revealed new problems requiring more changes and repairs.

Now, having committed to the technology, there was no going back. We are still in the midst of a coordinated effort between our customers telephone companies, the manufacturer of the new switch and SoVerNet's own technical team to work through and resolve the connection and performance problems we continue to face.

It has been a huge project. Much bigger and more difficult than expected. While it may not yet be apparent to you, there has been progress. A lot of progress. The time and commitment by every company and every person in this project has been huge. I must take a moment to thank the staff of Verizon and 3Com for going the extra mile. I also want to recognize the dedication and commitment of the SoVerNet teams, both technical and customer facing, for their tireless work and steadfast commitment; fixing the problems and explaining it to customers.

To our customers, know that we will see this through. We are here for the long haul. We are aware that the outages have caused you serious inconveniences. Know that we have not and we will not reduce our commitment to provide high quality service in a way that will exceed your expectations.

--te

==============
Tony Elliott - CEO

April 11
STATE OF THE NETWORK UPDATE

Along with this regular maintenance announcement, here is a status report on work we are doing to resolve the problems which some of you have experienced when calling our dial-up network. 3Com/Comworks has set up a
special task force to work with us on resolving the dialup problems. They are flying in equipment late this week which is intended to address some of the major (busy signal) issues that have been plaguing us. On a related note, our own engineers have created methods of pro-actively seeking blocked ports on the switch, in order to release them automatically. The effect is that it makes the issues much less noticeable.

If you are so inclined, please continue to alert us to problems by emailing customer-service@sover.net as this continues to be useful in our diagnosis.

Thank you for all of the supportive email you have sent to us. We remain committed to resolving all of these issues completely and quickly.

top

April 15
STATE OF THE NETWORK UPDATE

Along with this emergency maintenance announcement, here is a status report on work we are doing to resolve the problems which some of you have experienced when using our dialup network.

The special task force that 3Com/CommWorks has set up to work with us on resolving the dialup problems continues to meet and review progress at least three times a week. We are currently tracking more than a dozen open issues, and believe we have resolved three or four major ones.

Last week a software upgrade to the modem component of our dialup system seemed to introduce a problem where sessions were disconnected after five to fifteen minutes. By Friday we backed this upgrade out of the system and this particular problem seems to have been alleviated. 3Com/CommWorks is aware of this and working to fix it.

The replacement equipment for the signaling gateway component of our dialup system arrived today (Monday) and we are working with 3Com/CommWorks engineering on site to install and cut over to the new equipment in the very early hours of tomorrow morning (hence this announcement). We are--cautiously--optimistic that some of the most vexing difficulties we've endured will be fixed by this upgrade.

top

April 16
NETWORK UPDATE

The replacement equipment for the signaling gateway component of our dialup system was installed and put in service very early this morning (2:00am to 3:30am). So far, this component seems to be performing well, although there were some minor glitches affecting the Brattleboro area that required manual intervention before service was completely restored. We continue to work closely with 3Com/CommWorks, Verizon and others to permanently resolve the problems that we are currently working around.

top

April 23
STATE OF THE NETWORK UPDATE

We have gotten some feedback that indicates some of you are worried that we have under-provisioned our dialup network. Please be assured that this is not the case (if only it were that simple). In fact our dialup network is currently capable of supporting nearly 5,000 simultaneous calls and the most we've ever had is just over 3,000.

Now for the not-so good news.

Several separate service-affecting incidents have occurred that are the result of a common underlying cause, dubbed "circuit state mismatch," where our switch and Verizon's switch disagree about the state of a particular line. This affects service because all the calls from an area are sent via the mismatched circuit until manual intervention (sometimes by both SoVerNet and Verizon) gets the switches synchronized again. There were service disruptions of this type affecting the Brattleboro and Burlington areas early last week (Monday evening). Customers attempting to call from an area being impacted by a circuit state mismatch will experience a (regular) busy signal or "ring-no-answer." Please don't hesitate to report this type of problem to us. We're getting better (and faster) at detecting and correcting this, but our techniques are admittedly imperfect and your reports may help get your service restored faster.

We are also aware of the additional difficulties some of you are having establishing a dialup connection even when the phone is answered and the modems engage in "negotiation." We are working with the vendor to isolate and characterize this problem so we can ultimately solve it, but your feedback is extremely valuable here, since this problem is impacting an ill-defined sub-set of our customers and we are still desperately seeking a common denominator.

Despite our optimistic report about the replacement equipment our vendor provided, we sustained a major dialup outage in the middle of the week (Wednesday evening), which took most of the next day to recover from. We have located the cause of this particular problem and do not expect it to resurface.

Most recently we have been plagued with spontaneous reboots of various "Access Router Cards" (ARCs), which typically affect a calling area at a time. A customer impacted by this problem will find their session unceremoniously disconnected, although redialing will usually get them back online in short order. 3Com has offered us an (untested) upgrade that may address this problem. We are debating the merits of continuing beta-test their code versus running scheduled reboots of our own to work around the problem. We continue to work closely with 3Com/CommWorks, Verizon and others to permanently resolve the problems that we are currently aware of and
any new ones that may crop up.

The cause of an email problem you might have experienced over the past two days has been isolated and should be fixed.

top