Designing for Performance on Wall Street

On Your Marks…

Since the start of this decade, the US financial stock markets have experienced massive industry changes resulting from regulatory, competitive and innovative forces. These changes have led market participants to engage in an all out arms race towards extremely low-latency, high-throughput performance computing in an attempt to stay competitive.

Unfortunately, arms races like this one are sometimes driven by reaction than by vision. Not knowing, for example, how switched fabric based technologies can specifically address your performance troubles should prevent anyone in the purchasing of InfiniBand based technologies. Thankfully organizations such as STAC Research have begun to address a much neglected need for accurate performance measurements across this industry’s technology platforms.

What’s Driving All This?

Electronic Trading

Since the start of the 21st century, the adoption of electronic trading among hedge funds, broker dealers, and investment banks has skyrocketed. In fact as of 2007, electronic trading on the New York Stock Exchange (NYSE) makes up 60 to 70 percent of the daily volume. Technological innovation and economies of scale have led to the widespread digitization the stock trader’s profession resulting in highly advanced trading strategies. This form of trading relies on algorithms that seek profitability by scanning exchanges and other electronic execution venues with storms of buy/sell/cancel/replace orders at near wire speed. Similarly, this form of trading thrives on discovering the best prices before competitors, forcing many of the algo trading engines to circumvent the traditional sources for market data (i.e. Bloomberg, Reuters) and instead connect directly to to the source of the bid or ask quotes.


The Securities and Exchange Commission has designed many new regulations towards protecting investors in this new electronic marketplace. Regulation NMS (or RegNMS), in particular, forces all market participants a chance at the best price for any individual security, on any of the available electronic marketplaces during market hours. This regulation has resulted in a smart order routing strategies where traders and the algorithms implementing their strategies fulfill regulatory requirements and at the same time maximize their returns. Similar regulation has been introduced for the European markets in the form of The Markets in Financial Instruments Directive, or MiFID.

Many other types of regulation have been introduced with the goal of improving transparency in the financial markets. These regulations mandate record retention policies that weigh heavily on the storage capabilities of complying firms. For example, a firm wishing to store each level-1 market data quote disseminated from all the US ECNs, exchanges, and OTC markets is now facing a 600 million quote per day reality (based on January 2008 volume). Assuming off-the-shelf database products and straightforward indexing schemes to store this data means having to allocate 60 GB of storage daily. The point here is that regulation is impacting the industry resulting in changes that expose limits in network bandwidth as well as data storage.


Technological innovations in hardware, software, and networking technologies have enabled creative new opportunities for discovering liquidity and making money on wall street. Market participants, of all sizes, are maximizing their technology investments and utilizing the high-performance, high bandwidth solutions to stay competitive. Smaller sized hedge funds, for example, can now adopt relatively inexpensive off the shelf solutions towards generating startling returns that fill their investment banking brethren with envy.

High Performance Elements Uncovered

In this 3 part series, I’ll present the elements of high-performance computing in today’s US stock markets, and how these element are specifically designed to address the performance problems that emerged from the restructuring of this age-old industry. First, I’ll cover the problem of exploding message rates and the bandwidth problem, followed by the new low-latency reality in the need for speed. Finally, I’ll show how all this high-performance messaging is leading to the storage dilemma.

Tagged , ,
%d bloggers like this: