A History of iPass' Core Security Infrastructure—Part II

Continuing the story...

Operations worked closely with Engineering to add some additional logic to the new Transaction Server application which enabled us to achieve not only a high performance and stable application, but also a much deeper level of monitoring of application health.

While this work was underway in Engineering, we were in the midst of designing a network architecture for each Transaction Centre which helped achieve our primary design goal of "Always Available."

We prototyped a relatively new device on the market at that time, which performed load balancing at the application layer—this would be an essential element to our design.

We introduced redundancy into each device that sits in between the public Internet and the Transaction Server so that every Server always has at least two possible communication channels. This includes redundant border routers, redundant firewalls and a fully redundant switching fabric.

Leveraging the rule of three, we deployed at least three physical Transaction Servers at each site. Several sites which were relatively high volume were deployed with six Transaction Servers (rule of three x 2).

We also wanted to bring in multiple circuits, from diverse providers, to each Transaction Centre. In order to do so, however, we needed to acquire our own "address space"—block of IP numbers on the public Internet—as well as our own "autonomous system number." (ASN for short—this is used by the routers which comprise the backbone of the Internet in order to identify each other and in selecting path via which to route traffic).

After some negotiation with the American Registry for Internet Numbers (ARIN), iPass managed to register itself as an official presence on the Internet.

Meanwhile, in the System Engineering group, we were in the process of identifying affordable hardware which could be remotely administered and had acceptable performance and redundancy characteristics.

Several of the major hardware manufacturers offer a special line of equipment which they certify as being "NEBS Compliant." NEBS stands for New Equipment Building Standards and is a standard which was issued by Bellcore, I believe, back in the day. It specifies the requirements for any equipment to be deployed into a telco carrier facility. These systems have to be able to sustain a certain amount of seismic activity, ambient temperature fluctuations, moisture, etc.

For reliable storage, we also selected directly attached, hardware accelerated RAID systems (RAID = Redundant Array of Independent/Inexpensive Disks, or other things depending on who you ask).

The particular RAID configuration we used is called "RAID 1+0" which means we create mirrors for every disk drive for redundancy, then stripe data across the mirrored sets of disks for performance.

The philosophy here was to use multiple, active, inexpensive servers to achieve "Always Available" rather than investing in expensive, big iron systems like a mainframe. This approach allowed us to keep our capital costs in check, while providing a cloud of systems which were all being actively utilized but capable of taking over the load from each other automatically.

In essence, we became fault absorbent.

However, building out the Transaction Centres in this way was not enough. We also knew we had to improve the logic of the endpoints—by this, I mean the NetServers which we deploy at every one of our partner supplier locations.

Operations and Architecture developed a proprietary and sophisticated algorithm which allows the NetServer to dynamically change the Transaction Centre it is communicating with in real time, should it detect a momentary failure of some sort.

The NetServer can move back and forth as needed among the different Transaction Centres until it finds one which is available, and then it will continue to utilize the available Centre. Eventually, the NetServer will automatically reconfigure itself to use its primary Centre when it automatically detects the failure has cleared.

This algorithm allows us to bring up or bring down entire Transaction Centres, with no impact to service. Hence, we can absorb entire site outages, though they are extremely rare occurances.

In the end, the core of our service has essentially been 100% available for the last six or seven years now and continues to run smoothly.

While this two-part Perspective piece is an extremely brief and high level snapshot of some of the highlights that went into building our authentication fabric, I hope that you have found it to be insightful into the inner workings of iPass.

It took some years and many talented people, but I believe the result is outstanding and a key differentiator for our company in this competitive industry. In the last seven or so years since we completed the implementation of the new Transaction Centres, I personally have never been paged. We've had a few transient faults, caused by human error; excepting these, we successfully reached our primary design goal of being "Always Available."

And that is a piece of how iPass came to be.