5 Big Data Trends to Watch in the Financial Industry

Financial institutions are constantly seeking new ways to maintain a competitive advantage and increase efficiency. These days, many institutions are turning to technology as competition intensifies and the regulatory environment becomes increasingly uncertain. In order to stay afloat in the industry, these institutions are incorporating big data into their business strategy.

Big data and analytics have seen great innovation and investment in the past few years. The ecosystem has given birth to new tools and approaches that make it possible to grow while keeping costs low.

But what does that ecosystem really look like? “Big data” has become one of those buzz words tossed around the industry with little context around what it actually means. What is big data and how is it affecting our industry?


What is Big Data?

Big data describes the volume of both structured and unstructured data that businesses capture on a regular basis and the methodologies used to manage and analyze it. The concept was popularized in 2001 by Doug Laney, an industry analyst who broke down its definition into the three V’s: volume, velocity, and variety.

  • Volume refers to the ever-increasing amount of data collected. The scale of big data is what earned it its name and is partly what makes it so important to know how to manage it.
  • Velocity refers to the accelerating speed at which data is created and processed. In particular, big data has made it possible to process data in real-time.
  • Variety refers to the different forms and sources of data. In fact, 90% of data created is “unstructured,” meaning it is not easily captured, searched, or analyzed.1

In 2013, Mark van Rijmenam published an article that introduced four more V’s to further define the complexity of big data:

  • Veracity refers to the reliability of data. While data is powerful, having inaccurate data is worthless. Programs, models, and analyses are only as dependable as the data on which they are built.
  • Variability refers to the uncertain nature of data whose meaning is constantly changing. As data becomes more and more complex, the context in which the data originated largely influences the meaning of the data itself.
  • Visualization refers to the need to present data in an accessible and meaningful way. The reporting functionality of big data is equally as important as the processing and analysis.
  • Value refers to the positive business outcomes created by a a data solution. Businesses should focus on getting maximum value from their big data strategy. Ultimately, the ability to become more efficient, proactive, and predictive can lead to a substantial competitive advantage.2


5 Financial Big Data Trends to Watch


1. Using Big Data to Strengthen Models

With the ever increasing scale of data, banks in particular will be able to use data to strengthen their models. Incorporating newer types of data allows for more informed risk decisions that better ensure regulatory compliance, particularly as new regulations are passed.3 Banks are reaching out to technology and consulting firms for help with their big data efforts, as many do not have the necessary talent already on staff.4 For portfolio managers, the inclusion of new data points and unstructured data also enhances models, resulting in better trades and subsequently higher ROI.5 The challenge for managers is finding appropriate new data points to use as “trading signals.”6


2. Updated Data Processing and Storage

Across the financial services industry, data storage is changing, as many companies shift from enterprise data warehouses (EDW) to logical data warehouses (LDW), which allow for predictive analytics. The EDW has a hard time piecing together data from different sources, but its replacement, the more versatile LDW, can make sense of data from several different places and measure its own performance.7

Data warehousing in general has relied on the ETL (extraction, transformation, and loading) tools, in which data was manipulated and structured on a nightly basis. However, ETL tools do not provide analytics in a timely manner, and as the warehouse type changes, it also makes sense to change the approach for putting data into the warehouse. Therefore, ETL is flipping into ELT (extraction, loading, transformation), where data is staged real time and is not structured until a request is received from the user.8


3. Increased Interest in Analysts with Experience in Python and R

As the amount of data to be analyzed grows, tools to make this process easier, such as the open source languages Python and R are becoming increasingly popular. Because of their community APIs, developers can do more with these tools and have greater freedom to build what they need.9 This is especially true of R, which has CRAN (R packages developed by users). Python allows for collaboration with other users and is considered “general use,” meaning that it can be picked up quickly. Both have data visualization capabilities, although R is slightly superior in that regard. These tools allow for a faster and more automated preparation of analysis.10


4. Machine Learning Improves Efficiency in Fraud and Risk Management

Older analytics tools are becoming more and more inadequate given the amount of big data to be processed. Many hope that with increased use of machine learning, predictive analytics capabilities will improve. Obviously, models become more accurate as they are fed more data—and more accurate models will prevent companies from making as many poor risk choices. For the financial services industry, machine learning is especially helpful in gleaning insights from the data and reducing the occurrence of fraud.11


5. Blockchain Technology

There’s much speculation about the place of blockchain in the financial services industry. For many, the appeal lies in its ability to decentralize databases by linking separate transaction information together through a line of computer code, effectively doing away with a central governing body. It also offers a more secure and efficient way to share data.12 Bitcoin is the most well-known use of blockchain, but many other uses have been identified. In the January 2016 Finextra white paper, Banking on Blockchain, Michael King, Chairman of Credits (a blockchain platform provider) predicts that it could ease the process of anti-money laundering regulation—authorities could simply reference a bank’s blockchain.13 Although blockchain is fairly new and still requires a lot of development, it has great potential.

In the not-distant future, “big data” will simply become “data,” as the adoption of these trends and technologies becomes increasingly universal. For now, big data remains a hot topic that we expect an increasing number of industry leaders to investigate and embrace as they seek to maintain their competitive advantage.


[1] http://dataconomy.com/2014/05/seven-vs-big-data/

[2] https://datafloq.com/read/3vs-sufficient-describe-big-data/166

[3] http://www.mckinsey.com/business-functions/risk/our-insights/the-future-of-bank-risk-management

[4] https://thefinancialbrand.com/63538/banking-data-flywheel-analytics-ai/

[5] http://www.ravenpack.com/media/filer_public/04/e6/04e6caf3-db57-47b2-864a-9deaed1484aa/portfolio_managers_turn_to_big_data_for_more_alpha___929_media.pdf

[6] https://www.ft.com/content/f62ee814-f510-11e5-803c-d27c7117d132

[7] https://tdwi.org/Articles/2015/10/20/Defining-the-Logical-Data-Warehouse.aspx

[8] https://www.lynda.com/Hadoop-tutorials/Comparing-big-data-ELT-traditional-ETL/385663/424483-4.html#tab

[9] https://www.ngdata.com/big-data-technology-trends-in-banking/

[10] http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/getting-big-impact-from-big-data

[11] http://www.sas.com/en_us/insights/analytics/machine-learning.html

[12] http://knowledge.wharton.upenn.edu/article/blockchain-technology-will-disrupt-financial-services-firms/

[13] https://www.ingwb.com/media/1609652/banking-on-blockchain.pdf