Up: 2:3-4

A big data approach to analyzing market volatility

Kesheng Wu; E. Wes Bethel; Ming Gu; David Leinweber; Oliver Rübel

Algorithmic Finance (2013), 2:3-4, 241-267
DOI: 10.3233/AF-13030

Published: Abstract, PDF.
Archived: SSRN.


Understanding the microstructure of the financial market requires the processing of a vast amount of data related to individual trades, and sometimes even multiple levels of quotes. This requires computing resources that are not easily available to financial academics and regulators. Fortunately, data-intensive scientific research has developed a series of tools and techniques for working with a large amount of data. In this work, we demonstrate that these techniques are effective for market data analysis by computing an early warning indicator called Volume-synchronized Probability of Informed trading (VPIN) on a massive set of futures trading records. The test data contains five and a half year's worth of trading data for about 100 most liquid futures contracts, includes about 3 billion trades, and takes 140GB as text files. By using (1) a more efficient file format for storing the trading records, (2) more effective data structures and algorithms, and (3) parallelizing the computations, we are able to explore 16,000 different parameter combinations for computing VPIN in less than 20 hours on a 32-core IBM DataPlex machine. On average, computing VPIN of one futures contract over 5.5 years takes around 1.5 seconds on one core, which demonstrates that a modest computer is sufficient to monitor a vast number of trading activities in real-time – an ability that could be valuable to regulators.

By examining a large number of parameter combinations, we are also able to identify the parameter settings that improves the prediction accuracy from 80% to 93%.

Managing Editor

Philip Maymin

University of Bridgeport

Deputy Managing Editor

Jayaram Muthuswamy

Kent State University

Advisory Board

Kenneth J. Arrow

Stanford University

Herman Chernoff

Harvard University

David S. Johnson

AT&T Labs Research

Leonid Levin

Boston University

Myron Scholes

Stanford University

Michael Sipser

Massachusetts Institute of Technology

Richard Thaler

University of Chicago

Stephen Wolfram

Wolfram Research

Editorial Board

Associate Editors

Peter Bossaerts

California Institute of Technology

Emanuel Derman

Columbia University

Ming-Yang Kao

Northwestern University

Pete Kyle

University of Maryland

David Leinweber

Lawrence Berkeley National Laboratory

Richard J. Lipton

Georgia Tech

Avi Silberschatz

Yale University

Robert Webb

University of Virginia

Affiliate Editors

Giovanni Barone-Adesi

University of Lugano

Bruce Lehmann

University of California, San Diego

Unique Features of the Journal

Open access
Online articles are freely available to all.
No submission fees
There is no cost to submit articles for review. There will also be no publication or author fee for at least the first two volumes.
Authors retain copyright
Authors may repost their versions of the papers on preprint archives, or anywhere else, at any time.
Enhanced content
Enhanced, interactive, computable content will accompany papers whenever possible. Possibilities include code, datasets, videos, and live calculations.
Algorithmic Finance is the first journal in the Financial Economics Network of SSRN to allow comments.
The journal is published by IOS Press. In addition, the journal maintains an archive on SSRN.com.
While the journal does reserve the right to change these features at any time without notice, the intent will always be to provide the world's most freely and quickly available research on algorithmic finance.
Online ISSN: 2157-6203
Print ISSN: 2158-5571