A different look at Stock Exchange technologies …

   

There were a number of good presentations at my Middle Eastern conference, but the one that stood out was from the Amman Stock Exchange (ASE) in Jordan.  Their CIO, Mohammad Hisham Al Khatib,
presented their approach to technology and business continuity.  He's
been voted CIO of the year in the region by the leading trade
publication ACN, and delivered a quiet but quite astounding
presentation on over-capacity planning.

What do I mean by over-capacity?

Well,
their policy is to build systems that can handle five times the volume
of their daily averages.  Therefore, they process an average 20,000
transactions a day using a system built for 100,000 at peak (it can
actually handle 200,000 but that's another story). 

Established
in 1999, ASE now supports 1,500 users across the country, trading
US$17.4 billion in 2007 for a total market capitalisation of US$41
billion.   The users are spread out around Jordan, and there is no
formal trading floor. 

They use web services for their ticker info, as can be seen from their website,
and through these web services they get over over 15 million hits per
hour from internet users during trading hours.  This is load balanced
over 13 RISC machines using a 42 megabyte internet connection over
fibre optics.

All the web services were developed using Open Source software, and they've had no system failures since 2005. 

The
bit that intrigued me during the presentation, was the bit about their
Business Continuity Planning (BCP) and Disaster Recovery (DR), as they
built the BCP to be fully redundant.  Fully redundant networking
infrastructure with fully redundant servers and a fully redundant
Storage Area Network (SAN) environment.

The BCP is built with one
priority - high availability - and it runs over two disaster recovery
sites, one hot and one cold, with all attention focused upon fast
business recovery.  This is why they run synchronous mirroring from the
live system to the hot standby system over an 8 gigabyte fibre optic
link in real-time.   

It is also why they run a total number of
40 Unix and 20 Linux Servers, as well as blades and Microsoft servers.
This seems like overkill for a 20,000 transaction, 1,500 user exchange,
but their #1 priority is high availability so the fact that most of the
systems are never used is of no concern.  The fact is that they want
the reassurance that if they ever are needed then they must be there,
ready and able. 

The last major disaster, for example, was in
2004 when the main system went down.  The DR site located 2.5
kilometres away from the production site, came live within eight
minutes.

The second priority by the way is latency, like most,
and their view is that anything that takes more than 500 milliseconds
to process is of no value.  In other words, 501 milliseconds is
out-of-date and unacceptable.  Like most, they also find peak spikes in
their processing at opening and closing trading hours, so that's why
they focus upon high capacity with high availability and low latency.

I
was impressed with the discussions of their continuity plans, but then
I like the sounds of megamips processing with unlimited budgets.  Is
this best-of-breed therefore, I wonder, or am I just thinking this
looks good?