Designing for Performance on Wall Street – The Bandwidth Problem

Exploding Message Rates

Previously, we introduced the industry changes driving the technology performance problems on Wall Street. The first of these problems is growing bandwidth utilization resulting from the exploding message rates in market data feeds. Take for example the industry options feed, OPRA. Since 2001, the peak message rate for OPRA rose from 7000 messages per second (mps) to well over 300,000 mps in 2007. 2008 projections are racing towards the 700,000 mps mark. With message sizes averaging 120 bytes, network infrastructure, consuming OPRA, is pushed to support peak bit rates of 672 million bits per second (Mbps).

Footprint Minimizing Approaches

As expected, these message rates have forced network engineers, system administrators, and software engineers to come up with innovative solutions. The result is that Wall Street today relies heavily on multicast routing as a bandwidth minimizing technology and is looking to FAST, Conflation, Colocation and other approaches towards minimizing bandwidth utilization.

Multicast Routing

Multicast is a routing paradigm supported by modern routers that minimizes bandwidth utilization by allowing one or more systems to publish a single stream of packets to one or more receivers. Multicast-enabled routers achieve their efficiency by ensuring there is only one packet copy on the network branches they control if an only if there are subscribers on that branch. Additionally, in cases where there are subscribers, all non-subscribing nodes can reject the multicast packets at the Network Interface Controller (NIC) level, avoiding costly CPU time and benefiting the need for speed. For example, if a network branch has two multicast-group subscribers, the network’s router(s) will route a single copy of the stream’s packets to both subscribers.

For multicast routing to work, two things must exist, an addressing scheme that permits communicating to multiple receivers as well as a subscription mechanism, permitting multiple subscribers to joint a multicast-group.

FAST Compression

FIX Adapted for STreaming, or FAST, is a highly-efficient way of compressing market data quote and trade messages that focuses on maximizing the compression ratio and minimizing the compression latency. Early proof-of-concepts resulted in compression ratios around 80%, shrinking some trade messages from their original 241 bytes to a compressed size of 29 bytes. FAST compression has the added benefit of compressing faster when data volumes rise.


Conflation refers to a pub/sub mechanism which allows for throttled and/or finer event-based subscriptions to real time market data streams. Typically, subscribing to market data directly from the source of the data (i.e. exchanges), versus the vendors (i.e. bloomberg), means losing the sophisticated subscription/filtering mechanisms in favor of lightening fast data access (although this may all change with NYSE’s aquisition of Wombat Software). The result is having to process significantly more information in order to permit algos and other electronic trading applications to quickly discover best prices before their competitors. Vendors of direct market access distribution platforms have introduced conflation as a method to minimize the amount of data processed by electronic trading applications, while maintaining the minimal latency characteristics of market data directly from its source. For example, an electronic algorithm looking to purchase any number of shares in a stock, may be overwhelmed if that stock’s price changes 700 times a second. Instead, with conflation, the algo can subscribe to the available latest price and receive a single message during that second. Furthermore, an algo looking to purchase IBM stock only after it crosses the $100.00 mark between 3pm and 4pm may also suppress all quote messages for IBM during that period, and instead receive a single message if the event of interest occurs.


Colocation, also known as proximity hosting, refers to the practice whereby firms physically place the systems running their algorithms in the same geographic location as the systems that power the stock exchanges. This practice ensures minimal latency in processing buy/sell orders from the algo engines but has the additional benefit that the escalating market data volumes feeding these algorithms will be contained within the same network infrastructure. Firms that choose to colocate their algos circumvent the need to purchase and deploy expensive WAN infrastructure that can accommodate the insatiable bandwidth requirements of market data feeds. In other words, colocating customers will be rewarded with the lowest latency while at the same time ridding themselves of the network infrastructure supporting market data volumes.

API Efficiency

I mentioned how electronic trading and specifically how electronic algorithms are generating bursts of buy/sell orders that are executed as quickly as they are canceled. One approach for minimizing the bandwidth utilization of this activity was to introduce batching operations, such as bulk cancels, in the transactional interface (i.e. FIX Protocol) between broker and electronic marketplace. Another approach was to create generic messages, such as pegged orders, that can “follow” changing market conditions without the need to reissue new orders. Yet another approach was to suppress acknowledgment messages between electronic marketplace and algo, resulting in less bandwidth utilization. What all these approaches have in common is they all require modification to the transactional API between the algo and the electronic marketplace.

Feed Partitioning

OPRA remains the largest volume market data feed for the US capital markets with 2008 projections racing towards the 900,000 messages per second figure. In 2006 OPRA’s distributor, SIAC, shifted from distributing the feed over 12 separate network lines, to a total of 24 lines in an attempt to minimize the per line bandwidth requirements. This move has removed the per line bandwidth pressures many subscribers were facing, despite retaining the ridiculous bandwidth required to process the entire feed.

The bandwidth problem facing firms in the capital markets is only one part of the problem. Next we’ll show how their need for speed is presenting significant computing challenges as well.

Tagged , ,
%d bloggers like this: