Measuring performance (Mosaic OA + CJC Private Cloud)
Case Study on mosaicOA - IT Operational Analytics.
MosaicOA and the CJC private cloud - a case study
The Holy Grail
For financial data professionals
Operational metrics are emerging as the Holy Grail of financial data professionals. For capital markets institutions active in multiple asset classes and across financial centres, execution venues and regulatory regimes, maintaining the data infrastructure to support the increasingly complex trading business is an ongoing headache.
With so many moving parts, real time data and transaction messaging systems are prone to system overloads, bottlenecks and other potential pitfalls that can disrupt data flows and – worse still – bring trading to a halt.
Keeping tabs on an IT infrastructure as critical and complex as this is a major challenge. Existing solutions can be expensive and often not fit for purpose. Furthermore, it’s tough to get management buy-in for the investment required to ensure the smooth-running of trading data systems.
"The benefits of a holistic view of market data and trade messaging system performance are varied, ranging from more proactive load management and advanced system planning, to the avoidance of expensive and potentially damaging downtime"
– Paul Tomblin, Chief Technology Officer, CJC
Indeed, the concept of IT operations analytics (ITOA) has gained traction among financial institutions in the past two years as they strive to gain control over their critical processes. ITOA is the process of applying big data analytics to large volumes of data to produce business insights.
A major North American investment bank recently went live with CJC’s cloud-based ITOA platform. The bank was struggling to manage its market data infrastructure and wanted to collect performance metrics to identify potential bottlenecks so they could be dealt with before they caused operational problems.
Pain points and drivers
Monitoring metrics at the investment bank for its market data operations had become a top internal challenge when it set out to put in place a mechanism that would allow them to capture performance indicators, store the metrics for a period of several months or potentially years and then perform trend analysis and visualisation in order to fix problems before they happened.
The firm wanted to be able to assess the health of its data distribution infrastructure, by monitoring a range of commoditised indicators, like CPU rates and industry-specific indicators such as instrument update rates, middleware backbone throughput, server/client messaging and stale records.
The investment bank already had a real-time monitoring platform in place, but it was only able to conduct historical or future trend analysis on a small and limited set of its data. In fact, the market data infrastructure was generating more than a million metric points per second, but the bank could store and assess less than 0.5% for a short-term period. This small amount of data had to indicate the performance of data systems from Solace, Thomson Reuters Enterprise Platform (TREP), Bloomberg Platform, Wombat and Exegy.
The cost of running projects to capture and store these data sets was proving prohibitively expensive in large part due to the internal infrastructure costs associated with such a system. Furthermore, it would have been a major challenge for the bank’s internal IT resource to add any level of visualisation to enable users to more easily identify and solve potential issues.
"The cost of running projects to catch and collect data sets internally can prove prohibitively expensive. Cloud services represent a significant cost advantage."
– Paul Gow, Chief Executive Officer, CJC
In the Spring of 2015, the bank asked CJC to present a plan for capturing and vizualising those metrics, allowing it to analyze the performance of its market data infrastructure in real time. By July, CJC had delivered a blueprint, after which they kicked off an eight month design and implementation programme.
Specifically, the bank wanted to expand the number and type of performance metrics they could analyze in order to monitor resource and load requirements. The bank was also concerned about average CPU utilisation throughout its infrastructure. Servers with different roles had different results for disc utilisation, cache size and bandwidth and each of these measurements could be in flux. As well as keeping the market data system in place, the availability of reliable, consistent IT performance data could also be used to help justify the project to the bank’s procurement teams, as well as support ongoing resource requests going forward.
To put these considerations in place, the bank spent 18 months with CJC and came up with a solution for market data metrics that is now being considered across other areas of the bank, including equities, fixed income and foreign exchange.
"CJC did all the heavy lifting externally, so I didn't have to check that the right service level agreements were in place and that the data worked."
– Director of IT and Market Data, Tier 1 Global Investment Bank
CJC worked with the bank to design a system from scratch, but using existing tools from the marketplace. A key design element was the bank’s desire to host the platform externally in a private cloud, rather than building it on-premises and maintaining expensive servers for data capture and storage.
Since the maintenance cost per server hosted onsite starts at around $20,000 a year, the bank realised that storing terabytes or petabytes of operational data on its own internal infrastructure could become very costly. The existing monitoring facility at the bank was based on MySQL, which had disk space constraints, requiring users to be selective about what they stored for trading and analytics and putting pressure on what could be stored for analytics in the future.
Given that the data was system monitoring metrics, the bank did not consider it sensitive or valuable to anyone except the bank itself, making it suitable for a cheaper cloud implementation.
To prepare for a remotely hosted solution, the bank took steps to ensure the new monitoring environment would work on the CJC private cloud platform. It collected the performance metrics and fed them to CJC’s private cloud environment for analysis and visualisation, with the output fed back to the bank via secure, HTML-based front ends.
The project team initially had to address internal concerns about using the cloud, both from a security standpoint and from the need to ensure mobility of application. The firm wanted to make sure its cloud-based functions would be portable between cloud providers so that it could easily choose between them to take advantage of commercially compelling situations as they arose.
After completing the quality assurance phase, the bank had to put what was developed through its procurement process. As well as the usual considerations, the project had to fit with the firm’s broader 10-year vision to shift certain functions and processes to the cloud, whether private or public. Under this plan, the bank would retain those processes that required low latency responses within its own co-located environments, and migrate less response-intensive functions to the cloud.
Over time, this will significantly reduce its requirement for its own data centres.
The performance monitoring project provided an ideal vanguard initiative to address security, encryption and privacy issues more broadly. CJC’s private cloud met this criterion. But pushing functions and systems outside into the cloud is not without obstacles. Challenges include the need to migrate legacy applications that aren’t written to cope with the cloud, as well as information security, firewalls and privacy.
"The performance monitoring project provided an ideal vanguard initiative to address security, encryption and privacy issues more broadly. CJC’s private cloud fitted this criteria. "
– Steve Moreton, Senior Technical Director, CJC
As of Q3 2017 the client has 'true' capacity management in New York, Toronto and London with a road map including Tokyo, Hong Kong and Sydney. There are now 2.5 million metrics being stored in the CJC private cloud.
Requests for additional resource often face internal resistance, but making the case with system performance metrics helps. By predicting system outages, proactively taking action and demonstrating understanding of their causes, the bank is able to highlight value to potential users and management.
Potential data outages at the investment bank are now being averted, allowing the market data group to provide better services for its internal teams and end users. Other units in the bank, such as equities, fixed-income and foreign exchange, will have the same metrics to consider.
Although the system doesn’t directly enhance alpha generation in the bank, by preventing outages it adds demonstrable value. Explaining this value to users and managers was key to gaining support for the project and to expanding it to support other areas of the business.
Is your firm actively discussing requirements for monitoring metrics ?
We specialize in cloud technology, as well as consultancy and managed services that optimize data and trading technology operations, processes and commercial control for global capital markets firms. We help our clients:
- Transform their data platforms, both technically and commercially, to control costs
- Access innovative technology solutions
- Understand and comply with a raft of regulatory obligations
- Improve operational processes and control
With a global reach and offices in London, New York, Hong Kong and Singapore, we'd be delighted to learn of the challenges you face around data and how we may be able to help.
For further information: