Background

Recently we have been working with a major financial service company to upgrade their technology systems from Oracle to an Amazon Web Services (AWS) cloud-based system.  One of our major goals in moving to cloud is to increase development teams’ flexibility and to reduce technology costs relative to our client’s legacy environment. The client is obligated to monitor its regulatory and other trading exchanges. The client is challenged by steadily increasing market volumes, evolving exchanges, and changing regulatory rules. The existing legacy regulatory platform is no longer effectively having the flexibility supporting regulatory complexity.
 

Goal

CTI was selected to assist client in analysis and migrating from the legacy Regulatory platform to cloud base by using AWS and other Big Data technologies such as Hadoop, MapReduce, Spark, Storm, Hive, Java, and Apache Hbase. By migrating to a new regulatory platform the client will able to handle future market growth while containing escalating costs and increasing monitor regulatory operational efficiency.

​

Large amounts of data are being analyzed each day. Key business cases include:

1. Index and volatility manipulation

2. Front running option before stock

3. Order misrepresentation and mis-execution

4. Broker and Firm activity monitor

5. Excessive order and quoting

6. Book Trade Through

 

Technologies Utilized 

• EMR

• EC2

• S3

• Hadoop

• HIVE

 

Solution​

Client’s program to implement market regulation applications using Big Data and cloud technologies is containing escalating platform costs and improving operational efficiency. This frees resources to focus on improving systems to provide greater flexibility and effectiveness for regulatory analysts in several key ways.

 

Client’s Hadoop-based platform allows the use of less expensive, industry standard hardware. The Hadoop-based architecture lends itself to deployment on a public cloud such as Amazon Web Services, allowing client to benefit from the operational economies of scale of such a vender, as well as providing the elasticity to avoid over-provisioning to handle peak loads.

 

The legacy environment was also challenged with capacity limitations, causing client to spend much effort and analysis determining where and when available capacity could be used for various types of analytics. By removing the capacity limitations, the Hadoop/cloud combination virtually eliminates this effort.

 

Cloudera Manager automatically manages extremely complex operational tasks within Hadoop, allowing client to focus resources and expertise on areas of strategic regulatory importance rather than fundamental IT tasks. This saves costs while increasing client’s ability to accomplish core objectives.

 

Meanwhile, the use of HBase to access the order lifecycle graph database has reduced response times for certain complex queries by orders of magnitude. One complex query in particular that took ninety minutes to run was reduced to ten seconds. This allows analysts to rapidly iterate and quickly converge on answers that would have been prohibitive in the prior system.

 

Impact

One of the consequences from the shift to cloud is our client’s infrastructure, which is dynamic rather than fixed. This has already proven to be very beneficial for application re-runs as well as testing flexibility. We have recently achieved re-runs in a couple of weeks that would have normally taken several months in the legacy environment.