https://quant-traders.com

Introduction to Python for Quantitative Trading

Traditionally, there have been two general ways of analyzing market data:

  • fundamental analysis – focused on underlying fundamental data
  • technical analysis – focused on charts and price movements

In recent years, computer science and mathematics revolutionized trading, it has become dominated by computers helping to analyze vast amounts of available data.  Algorithms are responsible for making trading decisions faster than any human being could. Machine learning and data mining techniques are growing in popularity, all that falls under one broad category called ‘quantitative trading’ or ‘algorithmic trading’.

Below, I intend to provide you with basic tools for handling and analyzing market data with aim of generating profit from buying and selling financial instruments.

Python Programming Language

Currently, among the hottest programming languages for finance, you’ll find R and Python, alongside languages such as C++, C# and Java. I think Python or R is the right choice for many traders today. In this post, I assume you’re more or less starting from scratch or with very basic knowledge of Python, which by the way is one of the more approachable languages.

It is good to get the feeling of general Python programming before moving on with application to trading, there is a number of books and tutorials most available free or almost free:

  1. A Byte of Python by Swaroop C H
  2. Python for Everybody – Prof.Charles Severance
  3. Python Programming  by Wikibooks
  4. Think Python: How would you Think Like a Computer Scientist by Allen Downey
  5. Dive Into Python 3 by Mark Pilgrim

All blow examples of code are for Python 3.5 with Anaconda distribution available there – www.continuum.io

The whole point of trading is to predict with certain probability what will be market behavior in future and take advantage of that. Very often it can be as simple as ‘go long’ while expecting market prices to go up or ‘go short’ while expecting market prices to drop.

Defining our ‘view’ on market or expectations about future price changes usually takes some kind of market data analysis, to do it we need data first.

Data Import

There are many ways to import data to python, one of most common is using pandas-datareader package (starting with Pandas 0.19 on), it allows to import data from multiple sources like Yahoo Finance, Google Finance, Quandl, World Bank or OECD.

The easiest way to install it is:

pip install pandas-datareader

or

conda install -c anaconda pandas-datareader

When pandas-datareader is installed getting historical data takes only a few inputs, start and end date of a period we require the data for, ticker symbol and source of data.

Something like this:

import pandas_datareader.data as web
import datetime
start = datetime.datetime(2000, 1, 1)
end = datetime.datetime(2017, 1, 1)
data = web.DataReader('AAPL', 'yahoo', start, end)

Let’s check what we have:

print (data.tail())

Yahoo Finance gives back:

  • Date – quotation date
  • Open – open price
  • High – highest price for the day
  • Low – lowest price of the day
  • Close – close price
  • AdjClose – close rice with adjustments eg.stock split or dividend.
  • Volume – trade volume for the day

Let’s check Google Finance, we need as above Apple for a period of 01/01/2000 to  01/01/2017.

import pandas_datareader.data as web
import datetime
start = datetime.datetime(2000, 1, 1)
end = datetime.datetime(2017, 1, 1)
data = web.DataReader("AAPL", 'google', start, end)
print (data.tail())

We get an exact same set of data, except AdjClose value, so stock splits or dividends are not included.

Another very valuable source of financial/economic data can be Quandl.com.

import pandas_datareader.data as web
symbol = 'WIKI/AAPL' # or 'AAPL.US'
data = web.DataReader(symbol, 'quandl', "2000-01-01", "2017-01-01")
print(data.tail())

Again just like Yahoo Finance, Quandl delivers AdjClose information. There is plenty of other sources accessible via pandas-datareader, for more detailed information and examples pelase go there:

https://pandas-datareader.readthedocs.io/en/latest/

Pandas-datareader is very useful and offers plenty of options, although not the only solution, you can also use libraries like Quandl.

Running this line of code installs the package:

pip install quandl

Getting data is very similar to Pandas-datareader:

import quandl 
data = quandl.get("WIKI/AAPL", start_date="2000-10-01", end_date="2017-01-01")
print(data.tail())

After downloading the data it is always useful to save a local copy for further work, as generating online query every time data is required may be very time consuming for larger data sets eg. 100 tickers.

import pandas_datareader.data as web
import datetime
start = datetime.datetime(2000, 1, 1)
end = datetime.datetime(2017, 1, 1)
data = web.DataReader('AAPL', 'yahoo', start, end)
data.to_csv('data.csv')

After securing an own copy of the data, we can quickly read it in using pandas:

 

import pandas as pd
data = pd.read_csv('data.csv', index_col='Date', parse_dates=True)
print(data.tail())

Working with Data

Having all data saved, we can start looking at them in more detail, for purpose of this intro, we will use Adjusted Close values only. Let’s select that from the whole dataset:

AdjClose=data['Adj Close']
print(AdjClose.tail())

Next thing useful for trading would be to try and plot it, we can do it using pandas:

AdjClose.plot()

As we are about to use quantitative methods, let’s see some statistics about the Adjusted Close price values:

AdjClose.describe()

For a variety of reasons that are out of scope for this text, it is better to work with daily returns rather than nominal prices of financial instruments. There is a very simple way to returns using pandas:

Rets = AdjClose.pct_change(1)

We can also plot returns:

Rets.plot()

Sometimes instead of simple returns, we may like to use log returns, it is easy to do it using Numpy:

import numpy as np
LogRets = np.log(AdjClose.pct_change()+1)

Lets plot log returns:

LogRets.plot()

In next part of this introduction, we will move on to code first trading strategy.

Was the above useful? Please share with others on social media.

If you want to look for more information on Python or Trading, check online courses available at udemy.com.

Recommended reading list:

 

Pairs Trading: Quantitative Methods and Analysis

The first in-depth analysis of pairs trading
Pairs trading is a market-neutral strategy in its most simple form. The strategy involves being long (or bullish) one asset and short (or bearish) another. If properly performed, the investor will gain if the market rises or falls. Pairs Trading reveals the secrets of this rigorous quantitative analysis program to provide individuals and investment houses with the tools they need to successfully implement and profit from this proven trading methodology. Pairs Trading contains specific and tested formulas for identifying and investing in pairs, and answers important questions such as what ratio should be used to construct the pairs properly.
Ganapathy Vidyamurthy (Stamford, CT) is currently a quantitative software analyst and developer at a major New York City hedge fund.
Machine Trading: Deploying Computer Algorithms to Conquer the Markets (Wiley Trading)

Dive into algo trading with step-by-step tutorials and expert insight
Machine Trading is a practical guide to building your algorithmic trading business. Written by a recognized trader with major institution expertise, this book provides step-by-step instruction on quantitative trading and the latest technologies available even outside the Wall Street sphere. You'll discover the latest platforms that are becoming increasingly easy to use, gain access to new markets, and learn new quantitative strategies that are applicable to stocks, options, futures, currencies, and even bitcoins. The companion website provides downloadable software codes, and you'll learn to design your own proprietary tools using MATLAB. The author's experiences provide deep insight into both the business and human side of systematic trading and money management, and his evolution from proprietary trader to fund manager contains valuable lessons for investors at any level.

Algorithmic trading is booming, and the theories, tools, technologies, and the markets themselves are evolving at a rapid pace. This book gets you up to speed, and walks you through the process of developing your own proprietary trading operation using the latest tools.

Utilize the newer, easier algorithmic trading platforms
Access markets previously unavailable to systematic traders
Adopt new strategies for a variety of instruments
Gain expert perspective into the human side of trading
The strength of algorithmic trading is its versatility. It can be used in any strategy, including market-making, inter-market spreading, arbitrage, or pure speculation; decision-making and implementation can be augmented at any stage, or may operate completely automatically. Traders looking to step up their strategy need look no further than Machine Trading for clear instruction and expert solutions.
Applied Quantitative Methods for Trading and Investment

This much-needed book, from a selection of top international experts, fills a gap by providing a manual of applied quantitative financial analysis. It focuses on advanced empirical methods for modelling financial markets in the context of practical financial applications.
Data, software and techniques specifically aligned to trading and investment will enable the reader to implement and interpret quantitative methodologies covering various models.

The unusually wide-ranging methodologies include not only the 'traditional' financial econometrics but also technical analysis systems and many nonparametric tools from the fields of data mining and artificial intelligence. However, for those readers wishing to skip the more theoretical developments, the practical application of even the most advanced techniques is made as accessible as possible.

The book will be read by quantitative analysts and traders, fund managers, risk managers; graduate students in finance and MBA courses.
Quantitative Technical Analysis: An integrated approach to trading system development and trading management

This book, the fifth by Dr. Howard Bandy, discusses an integrated approach to trading system development and trading management.

It begins with a discussion and quantification of the several aspects of risk.
1. The trader's personal tolerance for risk.
2. The risk inherent in the price fluctuations of the issue to be traded.
3. The risk added by the trading system rules.
4. The trade-by-trade risk experienced during trading.

An original objective function, called "CAR25," based on risk-normalized profit potential is developed and explained. CAR25 is as near a universal objective function as I have found.

The importance of recognizing the non-stationary characteristics of financial data, and techniques for handling it, are discussed.

There is a general discussion of trading system development, including design, testing, backtesting, optimization, and walk forward analysis. That is followed by two parallel development paths -- one using traditional trading system development platform and the second machine learning.

Recognizing the importance of position sizing in managing trading, an original technique based on empirical Bayesian analysis, called "dynamic position sizing" and quantified in a metric called "safe-f," is introduced. Computer code implementing dynamic position sizing is included in the book.

56 fully disclosed, ready-to-run, and downloadable programs are included.
Finding Alphas: A Quantitative Approach to Building Trading Strategies

Design more successful trading systems with this practical guide to identifying alphas
Finding Alphas seeks to teach you how to do one thing and do it well: design alphas. Written by experienced practitioners from WorldQuant, including its founder and CEO Igor Tulchinsky, this book provides detailed insight into the alchemic art of generating trading signals, and gives you access to the tools you need to practice and explore. Equally applicable across regions, this practical guide provides you with methods for uncovering the hidden signals in your data. A collection of essays provides diverse viewpoints to show the similarities, as well as unique approaches, to alpha design, covering a wide variety of topics, ranging from abstract theory to concrete technical aspects. You'll learn the dos and don'ts of information research, fundamental analysis, statistical arbitrage, alpha diversity, and more, and then delve into more advanced areas and more complex designs. The companion website, www.worldquantchallenge.com, features alpha examples with formulas and explanations. Further, this book also provides practical guidance for using WorldQuant's online simulation tool WebSim® to get hands-on practice in alpha design.

Alpha is an algorithm which trades financial securities. This book shows you the ins and outs of alpha design, with key insight from experienced practitioners.

Learn the seven habits of highly effective quants
Understand the key technical aspects of alpha design
Use WebSim® to experiment and create more successful alphas
Finding Alphas is the detailed, informative guide you need to start designing robust, successful alphas.
Inside the Black Box: A Simple Guide to Quantitative and High Frequency Trading

New edition of book that demystifies quant and algo trading
In this updated edition of his bestselling book, Rishi K Narang offers in a straightforward, nontechnical style—supplemented by real-world examples and informative anecdotes—a reliable resource takes you on a detailed tour through the black box. He skillfully sheds light upon the work that quants do, lifting the veil of mystery around quantitative trading and allowing anyone interested in doing so to understand quants and their strategies. This new edition includes information on High Frequency Trading.

Offers an update on the bestselling book for explaining in non-mathematical terms what quant and algo trading are and how they work
Provides key information for investors to evaluate the best hedge fund investments
Explains how quant strategies fit into a portfolio, why they are valuable, and how to evaluate a quant manager
This new edition of Inside the Black Box explains quant investing without the jargon and goes a long way toward educating investment professionals.
Automated Trading with R: Quantitative Research and Platform Development

Learn to trade algorithmically with your existing brokerage, from data management, to strategy optimization, to order execution, using free and publicly available data. Connect to your brokerage’s API, and the source code is plug-and-play.

Automated Trading with R explains automated trading, starting with its mathematics and moving to its computation and execution. You will gain a unique insight into the mechanics and computational considerations taken in building a back-tester, strategy optimizer, and fully functional trading platform.

The platform built in this book can serve as a complete replacement for commercially available platforms used by retail traders and small funds. Software components are strictly decoupled and easily scalable, providing opportunity to substitute any data source, trading algorithm, or brokerage. This book will:

Provide a flexible alternative to common strategy automation frameworks, like Tradestation, Metatrader, and CQG, to small funds and retail traders
Offer an understanding of the internal mechanisms of an automated trading system
Standardize discussion and notation of real-world strategy optimization problems
What You Will Learn

Understand machine-learning criteria for statistical validity in the context of time-series
Optimize strategies, generate real-time trading decisions, and minimize computation time while programming an automated strategy in R and using its package library
Best simulate strategy performance in its specific use case to derive accurate performance estimates
Understand critical real-world variables pertaining to portfolio management and performance assessment, including latency, drawdowns, varying trade size, portfolio growth, and penalization of unused capital
Who This Book Is For

Traders/practitioners at the retail or small fund level with at least an undergraduate background in finance or computer science; graduate level finance or data science students
Quantitative Trading with R: Understanding Mathematical and Computational Tools from a Quant's Perspective

Quantitative Finance with R offers a winning strategy for devising expertly-crafted and workable trading models using the R open source programming language, providing readers with a step-by-step approach to understanding complex quantitative finance problems and building functional computer code.
Quantitative Momentum: A Practitioner's Guide to Building a Momentum-Based Stock Selection System (Wiley Finance)

The individual investor's comprehensive guide to momentum investing
Quantitative Momentum brings momentum investing out of Wall Street and into the hands of individual investors. In his last book, Quantitative Value, author Wes Gray brought systematic value strategy from the hedge funds to the masses; in this book, he does the same for momentum investing, the system that has been shown to beat the market and regularly enriches the coffers of Wall Street's most sophisticated investors. First, you'll learn what momentum investing is not: it's not 'growth' investing, nor is it an esoteric academic concept. You may have seen it used for asset allocation, but this book details the ways in which momentum stands on its own as a stock selection strategy, and gives you the expert insight you need to make it work for you. You'll dig into its behavioral psychology roots, and discover the key tactics that are bringing both institutional and individual investors flocking into the momentum fold.

Systematic investment strategies always seem to look good on paper, but many fall down in practice. Momentum investing is one of the few systematic strategies with legs, withstanding the test of time and the rigor of academic investigation. This book provides invaluable guidance on constructing your own momentum strategy from the ground up.

Learn what momentum is and is not
Discover how momentum can beat the market
Take momentum beyond asset allocation into stock selection
Access the tools that ease DIY implementation
The large Wall Street hedge funds tend to portray themselves as the sophisticated elite, but momentum investing allows you to 'borrow' one of their top strategies to enrich your own portfolio. Quantitative Momentum is the individual investor's guide to boosting market success with a robust momentum strategy.
Quantitative Trading: Algorithms, Analytics, Data, Models, Optimization

The first part of this book discusses institutions and mechanisms of algorithmic trading, market microstructure, high-frequency data and stylized facts, time and event aggregation, order book dynamics, trading strategies and algorithms, transaction costs, market impact and execution strategies, risk analysis, and management. The second part covers market impact models, network models, multi-asset trading, machine learning techniques, and nonlinear filtering. The third part discusses electronic market making, liquidity, systemic risk, recent developments and debates on the subject.
https://quant-traders.com

Intro to Algorithmic Trading.

Introduction to Algorithmic Trading

Financial institutions have been rapidly increasing the usage of digital technology since the 1970s. Competitive market enforced having the latest technology in many operations that used to be manual in the past. Looks like one of those would be trading on the markets, where good old ‘gut-feeling’ is becoming less and less useful and leaves a place to clusters of CPUs or GPUs, ‘Big Data’ and so-called ‘Algorithmic Trading’.

What is Algorithmic Trading?

Algorithmic trading means the use of electronic platforms for entering trading orders with an algorithm which executes pre-programmed trading instructions whose variables may include timing, price or quantity of the order. Trades are executed with marginal or without any human intervention, algorithms require investors to first specify their goals in terms of mathematical variables.

The overall trend observed during the past few decades has been towards quantitative ans algorithmic approach in investing and there’s seems to be no reversal from that on the horizon. Algorithmic trading plays a major role in the financial markets and there is good chance it will play even bigger in the future.

Depending on investors needs, customized algorithms range from simple to highly sophisticated, computers implement trades following the exactly prescribed instructions. Algorithmic trading has become possible due to fully electronic infrastructure in stock trading systems. The term black box trading is used to describe very advanced systems operating without or with minimal human intervention.

Pros and Cons of Algorithmic Trading

Algorithmic trading has several advantages compared to trading carried out by a human trader. Computer systems have a much shorter reaction time and reach a higher level of reliability. The decisions made by a computer system rely on the underlying strategy with a specified set of rules, which lead to a reproducibility of these decisions. Consequently, back-testing and improving the strategy by varying the rules possible. Algorithmic trading also ensures objectivity in trading decisions and is not exposed to subjective influences. When trading many different securities at the same time, one computer system may substitute many human traders. Observation and the trading of securities of a large variety become possible for companies without employing hundreds of traders.

Nevertheless, the automated trading requires monitoring, altogether it results in a better performance of the investment strategy as well as in lower trading costs.

It is challenging to automate the whole process from investment decision to execution, there is always a trade-off between system stability and robustness and complexity. The less complex the system the more solid operation, on the other hand, lack of complexity may lead to lower profitability.

Algorithmic trading has been often related to increasing volatility on the markets, according to some algorithms may produce snowball effects and so-called ‘flash crashes’.  Another aspect in a discussion about algorithmic trading is its effect on liquidity. These trading strategies may improve liquidity if they act as market makers, but on the other hand, if they create extra imbalance the effect may just the opposite. There are scientific findings that indicate, algorithmic trading improves liquidity and enhances the informativeness of quotes.  It helps reducing transaction costs and risk, improves entry speed, increases trade control and reduces bid/ask spread.

Automated Trading System

The concept of an automated trading system refers to an idea of computer trading program that automatically submits trades to a given exchange. Depending on a trading frequency the speed of data may have a significant influence whether the trading system is profitable or not. An automated trading system consists of many layers all of which must be taken into consideration. First of all, trading system must have a reliable market data feed. This data is being fed to algorithms to find if there are currently such market conditions, which would have been likely profitable trading opportunities according to back-test (Alpha Model). If the back-test support market conditions then the risk management should be taken into account: does this trade fit to our current portfolio’s risk profile and what kind of effect it has on portfolio’s total market exposure (Risk Model). Assuming the potential trade ticks two above boxes, it can be executed on the market. Before executing an order, a strategy should also tell when the trade should be exited if the market turns against the trade but also when to close the profitable trade (Transaction Cost Model).

The result of a strategy will be the returns over time of those trades and the performance measures. Just to scratch the surface, there are two main kinds of strategies to perform in quantitative trading, mean-reversion and momentum. If the price of an asset is relative low or high respect a reference, for example, the average historical price of a share, and it is expected the come back to that reference, the strategies exploiting this situation are called mean-reversion strategies. For example, it is possible to make a strategy with two stocks highly correlated in time. If the prices are moving in opposite direction (making the difference between them greater) buying one and selling the other with the same amount of money expecting a reverse in that movement. Momentum strategies follow a trend in a certain direction.

Pitfalls to avoid


In the strategy research process, there are numerous ways to fall in pitfalls, leading in well-performed strategies with historical data but useless with real-time prices.

The main issues may be:

  • Taking into consideration stocks splits and dividends
  • Missing data and abnormally differences between periods
  • Survivorship bias, problem testing only companies that are ‘alive’
  • Use data that it has not been generated during the period analyzed
  • Overfitting, use parameters to fit the historical data
  • Overfitting, use parameters to fit the historical data

To begin with, the data sources could contain errors. The dividends are discounted in the stock price, the past data is usually adjusted to not have big drops in prices when, actually, the money is in the account of the investor. Also, the prices have to be adjusted when the company decides to divide a share into multiples shares, the same way the prices have to do. The reliability of the data has associated a data cost to pay to the provider and must to discount to the final result of the strategy. An important associated cost and typical pitfall to fall not including it in the strategy result is the transaction cost. This is the price paid to the broker to send the orders to the market. For this reason, the strategies with fewer trades and great returns in each trade are good to avoid this cost. The pitfall called survivorship bias is when that is not including in the strategy research the companies that went into bankruptcy. Another important pitfall is overfitting, strategy performs well only with historical data used for tuning, but underperforms in live conditions. To avoid this bias, the strategy has to be tested with additional data or “out-of-the-sample” data before going life. Last but not least, look-ahead bias, that is using data that were not available in the test period, for example, a financial ratio that is brought at the end of the year.

Strategy performance

Traders use a variety of ratios to measure and compare the performance of different strategies. This process is present in the whole cycle of testing and tuning to ensure the performance will be good at a time when strategy executes real trades.

Most often the media or investment fund reports show the returns to tell how an investment had performed, it says little of the characteristics and risk of the investment.

Things like long periods of negative returns or high returns followed by big drops are not always expected by investors, to take that into consideration few measures have been developed:

Sharpe Ratio – it measures the excess return of the portfolio (above the risk-free rate) in terms of excess portfolio’s volatility. The ratio measures just the alpha component of the total return. Important to note that the Sharpe Ratio does not distinguish between positive and negative returns at the volatility.

Sortino Ratio – it is a modification of the Sharpe Ratio where only the downside volatility penalizes the ratio.

Drawdown measures like ‘max drawdown’ that is the maximum percentage decrease in capital encountered in the period or ‘max drawdown period’ that is maximum period in which strategy was losing money

Risk Management

Understanding the concept of risk is crucial for creating a good risk management strategy. Risk can be defined as the uncertainty of outcomes. In the financial literature, a risk is the likelihood of losses resulting from unexpected events related to movements in the market. Extreme events (like crashes) may have a low probability of occurring but also may cause a big loss. These low probability events are more intractable because they are usually hard to anticipate. Algorithmic trading is exposed to different types of risk which could be generally categorized into two main types: market risk (related to market behavior) and operational risk (eg. infrastructure failure).

Summary

As financial markets become more competitive, financial institutions and private investors have started to turn to automated trading, to gain competitive advantage. With the ability to communicate electronically with exchanges and other electronic trading venues, it is now possible to construct automated trading systems to analyze the changing market data and place orders when certain criteria are met. These systems can be customized to execute almost any trading strategy.  Different people might utilize algorithmic trading systems to achieve different objectives although most of them contain one of the basic layers:

  • The trading signal generation
  • Risk management
  • Systems that automate the trade execution

Was the above useful? Please share with others on social media.

If you want to look for more information on Python or Trading, check online courses available at udemy.com.

Recommended reading list:

 

Pairs Trading: Quantitative Methods and Analysis

The first in-depth analysis of pairs trading
Pairs trading is a market-neutral strategy in its most simple form. The strategy involves being long (or bullish) one asset and short (or bearish) another. If properly performed, the investor will gain if the market rises or falls. Pairs Trading reveals the secrets of this rigorous quantitative analysis program to provide individuals and investment houses with the tools they need to successfully implement and profit from this proven trading methodology. Pairs Trading contains specific and tested formulas for identifying and investing in pairs, and answers important questions such as what ratio should be used to construct the pairs properly.
Ganapathy Vidyamurthy (Stamford, CT) is currently a quantitative software analyst and developer at a major New York City hedge fund.
Machine Trading: Deploying Computer Algorithms to Conquer the Markets (Wiley Trading)

Dive into algo trading with step-by-step tutorials and expert insight
Machine Trading is a practical guide to building your algorithmic trading business. Written by a recognized trader with major institution expertise, this book provides step-by-step instruction on quantitative trading and the latest technologies available even outside the Wall Street sphere. You'll discover the latest platforms that are becoming increasingly easy to use, gain access to new markets, and learn new quantitative strategies that are applicable to stocks, options, futures, currencies, and even bitcoins. The companion website provides downloadable software codes, and you'll learn to design your own proprietary tools using MATLAB. The author's experiences provide deep insight into both the business and human side of systematic trading and money management, and his evolution from proprietary trader to fund manager contains valuable lessons for investors at any level.

Algorithmic trading is booming, and the theories, tools, technologies, and the markets themselves are evolving at a rapid pace. This book gets you up to speed, and walks you through the process of developing your own proprietary trading operation using the latest tools.

Utilize the newer, easier algorithmic trading platforms
Access markets previously unavailable to systematic traders
Adopt new strategies for a variety of instruments
Gain expert perspective into the human side of trading
The strength of algorithmic trading is its versatility. It can be used in any strategy, including market-making, inter-market spreading, arbitrage, or pure speculation; decision-making and implementation can be augmented at any stage, or may operate completely automatically. Traders looking to step up their strategy need look no further than Machine Trading for clear instruction and expert solutions.
Applied Quantitative Methods for Trading and Investment

This much-needed book, from a selection of top international experts, fills a gap by providing a manual of applied quantitative financial analysis. It focuses on advanced empirical methods for modelling financial markets in the context of practical financial applications.
Data, software and techniques specifically aligned to trading and investment will enable the reader to implement and interpret quantitative methodologies covering various models.

The unusually wide-ranging methodologies include not only the 'traditional' financial econometrics but also technical analysis systems and many nonparametric tools from the fields of data mining and artificial intelligence. However, for those readers wishing to skip the more theoretical developments, the practical application of even the most advanced techniques is made as accessible as possible.

The book will be read by quantitative analysts and traders, fund managers, risk managers; graduate students in finance and MBA courses.
Quantitative Technical Analysis: An integrated approach to trading system development and trading management

This book, the fifth by Dr. Howard Bandy, discusses an integrated approach to trading system development and trading management.

It begins with a discussion and quantification of the several aspects of risk.
1. The trader's personal tolerance for risk.
2. The risk inherent in the price fluctuations of the issue to be traded.
3. The risk added by the trading system rules.
4. The trade-by-trade risk experienced during trading.

An original objective function, called "CAR25," based on risk-normalized profit potential is developed and explained. CAR25 is as near a universal objective function as I have found.

The importance of recognizing the non-stationary characteristics of financial data, and techniques for handling it, are discussed.

There is a general discussion of trading system development, including design, testing, backtesting, optimization, and walk forward analysis. That is followed by two parallel development paths -- one using traditional trading system development platform and the second machine learning.

Recognizing the importance of position sizing in managing trading, an original technique based on empirical Bayesian analysis, called "dynamic position sizing" and quantified in a metric called "safe-f," is introduced. Computer code implementing dynamic position sizing is included in the book.

56 fully disclosed, ready-to-run, and downloadable programs are included.
Finding Alphas: A Quantitative Approach to Building Trading Strategies

Design more successful trading systems with this practical guide to identifying alphas
Finding Alphas seeks to teach you how to do one thing and do it well: design alphas. Written by experienced practitioners from WorldQuant, including its founder and CEO Igor Tulchinsky, this book provides detailed insight into the alchemic art of generating trading signals, and gives you access to the tools you need to practice and explore. Equally applicable across regions, this practical guide provides you with methods for uncovering the hidden signals in your data. A collection of essays provides diverse viewpoints to show the similarities, as well as unique approaches, to alpha design, covering a wide variety of topics, ranging from abstract theory to concrete technical aspects. You'll learn the dos and don'ts of information research, fundamental analysis, statistical arbitrage, alpha diversity, and more, and then delve into more advanced areas and more complex designs. The companion website, www.worldquantchallenge.com, features alpha examples with formulas and explanations. Further, this book also provides practical guidance for using WorldQuant's online simulation tool WebSim® to get hands-on practice in alpha design.

Alpha is an algorithm which trades financial securities. This book shows you the ins and outs of alpha design, with key insight from experienced practitioners.

Learn the seven habits of highly effective quants
Understand the key technical aspects of alpha design
Use WebSim® to experiment and create more successful alphas
Finding Alphas is the detailed, informative guide you need to start designing robust, successful alphas.
Inside the Black Box: A Simple Guide to Quantitative and High Frequency Trading

New edition of book that demystifies quant and algo trading
In this updated edition of his bestselling book, Rishi K Narang offers in a straightforward, nontechnical style—supplemented by real-world examples and informative anecdotes—a reliable resource takes you on a detailed tour through the black box. He skillfully sheds light upon the work that quants do, lifting the veil of mystery around quantitative trading and allowing anyone interested in doing so to understand quants and their strategies. This new edition includes information on High Frequency Trading.

Offers an update on the bestselling book for explaining in non-mathematical terms what quant and algo trading are and how they work
Provides key information for investors to evaluate the best hedge fund investments
Explains how quant strategies fit into a portfolio, why they are valuable, and how to evaluate a quant manager
This new edition of Inside the Black Box explains quant investing without the jargon and goes a long way toward educating investment professionals.
Automated Trading with R: Quantitative Research and Platform Development

Learn to trade algorithmically with your existing brokerage, from data management, to strategy optimization, to order execution, using free and publicly available data. Connect to your brokerage’s API, and the source code is plug-and-play.

Automated Trading with R explains automated trading, starting with its mathematics and moving to its computation and execution. You will gain a unique insight into the mechanics and computational considerations taken in building a back-tester, strategy optimizer, and fully functional trading platform.

The platform built in this book can serve as a complete replacement for commercially available platforms used by retail traders and small funds. Software components are strictly decoupled and easily scalable, providing opportunity to substitute any data source, trading algorithm, or brokerage. This book will:

Provide a flexible alternative to common strategy automation frameworks, like Tradestation, Metatrader, and CQG, to small funds and retail traders
Offer an understanding of the internal mechanisms of an automated trading system
Standardize discussion and notation of real-world strategy optimization problems
What You Will Learn

Understand machine-learning criteria for statistical validity in the context of time-series
Optimize strategies, generate real-time trading decisions, and minimize computation time while programming an automated strategy in R and using its package library
Best simulate strategy performance in its specific use case to derive accurate performance estimates
Understand critical real-world variables pertaining to portfolio management and performance assessment, including latency, drawdowns, varying trade size, portfolio growth, and penalization of unused capital
Who This Book Is For

Traders/practitioners at the retail or small fund level with at least an undergraduate background in finance or computer science; graduate level finance or data science students
Quantitative Trading with R: Understanding Mathematical and Computational Tools from a Quant's Perspective

Quantitative Finance with R offers a winning strategy for devising expertly-crafted and workable trading models using the R open source programming language, providing readers with a step-by-step approach to understanding complex quantitative finance problems and building functional computer code.
Quantitative Momentum: A Practitioner's Guide to Building a Momentum-Based Stock Selection System (Wiley Finance)

The individual investor's comprehensive guide to momentum investing
Quantitative Momentum brings momentum investing out of Wall Street and into the hands of individual investors. In his last book, Quantitative Value, author Wes Gray brought systematic value strategy from the hedge funds to the masses; in this book, he does the same for momentum investing, the system that has been shown to beat the market and regularly enriches the coffers of Wall Street's most sophisticated investors. First, you'll learn what momentum investing is not: it's not 'growth' investing, nor is it an esoteric academic concept. You may have seen it used for asset allocation, but this book details the ways in which momentum stands on its own as a stock selection strategy, and gives you the expert insight you need to make it work for you. You'll dig into its behavioral psychology roots, and discover the key tactics that are bringing both institutional and individual investors flocking into the momentum fold.

Systematic investment strategies always seem to look good on paper, but many fall down in practice. Momentum investing is one of the few systematic strategies with legs, withstanding the test of time and the rigor of academic investigation. This book provides invaluable guidance on constructing your own momentum strategy from the ground up.

Learn what momentum is and is not
Discover how momentum can beat the market
Take momentum beyond asset allocation into stock selection
Access the tools that ease DIY implementation
The large Wall Street hedge funds tend to portray themselves as the sophisticated elite, but momentum investing allows you to 'borrow' one of their top strategies to enrich your own portfolio. Quantitative Momentum is the individual investor's guide to boosting market success with a robust momentum strategy.
Quantitative Trading: Algorithms, Analytics, Data, Models, Optimization

The first part of this book discusses institutions and mechanisms of algorithmic trading, market microstructure, high-frequency data and stylized facts, time and event aggregation, order book dynamics, trading strategies and algorithms, transaction costs, market impact and execution strategies, risk analysis, and management. The second part covers market impact models, network models, multi-asset trading, machine learning techniques, and nonlinear filtering. The third part discusses electronic market making, liquidity, systemic risk, recent developments and debates on the subject.
https://quant-traders.com

Example code for Numerai – deep learning.

In a previous post on Numerai, I have described very basic code to get into a world of machine learning competitions. This one will be a continuation, so if you haven’t read it I recommend to do it- here. In this post, we will add little more complexity to the whole process. We will split out 20% of training data as validation set so we can train different models and compare performance. And we will dive into deep neural nets as predicting model.

Ok, let’s do some machine learning…

Let’s start with importing what will be required, this step is similar to what we have done in the first model. Apart from Pandas, we import “StandardScaler” to preprocess data before feeding them into neural net. We will use “train_test_split” to split out 20% of data as a test set. “roc_auc_score” is a useful metric to check and compare performance of the model, we will also need neural net itself – that will be classifier from ‘scikit-neuralnetwork’ (sknn).

Imports first:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sknn.mlp import Classifier, Layer

As we have all required imports, we can load the data from csv(remember to update the system path to downloaded files):

train = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_training_data.csv")
test = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_tournament_data.csv")
sub = pd.read_csv("/home/m/Numerai/numerai_datasets/example_predictions.csv")

Some basic data manipulation required:

sub["t_id"]=test["t_id"]
test.drop("t_id", axis=1,inplace=True)
labels=train["target"]
train.drop("target", axis=1,inplace=True)
train=train.values
labels=labels.values

In next four lines, we will do what is called standardization. The result of standardization (or Z-score normalization) is that the features will be rescaled so that they’ll have the properties of a standard normal distribution with μ=0 and σ=1.

scaler = StandardScaler()
scaler.fit(train)
train = scaler.transform(train)
test = scaler.transform(test)

Next line of code will split original downloaded train set to train and test set, basically we set aside 20% of original train data to make sure we can check the out of the sample performance – to avoid overfitting.

X_train, X_test, y_train, y_test = train_test_split(train,labels, test_size=0.2, random_state=35)

Having all data preprocessed we are ready to define model, set number of layers in neural network, and a number of neurons in each layer. Below few lines of code to do it:

nn = Classifier(
layers=[
Layer("Tanh", units=50),
Layer("Tanh", units=200),
Layer("Tanh", units=200),
Layer("Tanh", units=50),
Layer("Softmax")],
learning_rule='adadelta',
learning_rate=0.01,
n_iter=5,
verbose=1,
loss_type='mcc')

“units=50” – states a number of neurons in each layer, number of neurons in first layer is determined by a number of features in data we will feed in.

“Tahn” – this is kind of activation function, you can use other as well eg. rectifier, expLin, sigmoid, or convolution. In last layer the activation function is Softmax – that’s usual output layer function for classification tasks. In our network we have five layers with a different number of neurons, there are no strict rules about number of neurons and layers so it is more art than science, you just need to try different versions and check what works best.

In our network we have five layers with a different number of neurons, there are no strict rules about a number of neurons and layers so it is more art than science, you just need to try different versions and check what works best. After layers we set learning rule to ‘adadelta’ again more choice available: sgd, momentum, nesterov, adagrad or rmsprop just try and check what works best.

“learning_rule=’adadelta'” – sets learning algorithm to ‘adadelta’, more choice available: sgd, momentum, nesterov, adagrad or rmsprop just try and check what works best, you can mix them for different layers.

“learning_rate=0.01” – learning rate, often as rule of thumb you start with ‘default’ value of 0.01, but other values can be used, mostly anything from 0.001 to 0.1.

“n_iter=5” – number of iterations ‘epochs’, the higher the number the longer process of learning will take, 5 is as example only, one need to look at error after each epoch, at some point it will stop dropping, I have seen anything from 50 to 5000 so feel free to play with it.

“verbose=1” – this parameter will let us see progress on screen.

“loss_type=’mcc’ ” – loss function, ‘mcc’ typical for classification tasks.

As the model is set, we can feed data and train it, depending on how powerful your pc is it can take from seconds to days. It is recommended to use GPU computing for neural networks training.

nn.fit(X_train, y_train)

Below line validates the model against 20% of data we have set aside before.

print('Overall AUC:', roc_auc_score(y_test, nn.predict_proba(X_test)[:,1]))

Using above code we can play around with different settings and neural networks architectures, checking the performance. After finding the best settings, they can be applied for prediction to be uploaded to Numerai, just run last three lines(just remember to update system path to save the file):

y_pred = nn.predict_proba(test)
sub["probability"]=y_pred[:,1]
sub.to_csv("/home/m/Numerai/numerai_datasets/Prediction.csv", index=False)

I hope above text was useful and you can now start playing around with deep learning for trading predictions for Numerai. If you have any comments or questions please feel free to contact me.

Full code below:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sknn.mlp import Classifier, Layer

train = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_training_data.csv")
test = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_tournament_data.csv")
sub = pd.read_csv("/home/m/Numerai/numerai_datasets/example_predictions.csv")

sub["t_id"]=test["t_id"]
test.drop("t_id", axis=1,inplace=True)

labels=train["target"]
train.drop("target", axis=1,inplace=True)

train=train.values
labels=labels.values

scaler = StandardScaler()
scaler.fit(train)
train = scaler.transform(train)
test = scaler.transform(test)

X_train, X_test, y_train, y_test = train_test_split(train,labels, test_size=0.2, random_state=35)

nn = Classifier(
layers=[
Layer("Tanh", units=50),
Layer("Tanh", units=200),
Layer("Tanh", units=200),
Layer("Tanh", units=50),
Layer("Softmax")],
learning_rule='adadelta',
learning_rate=0.01,
n_iter=5,
verbose=1,
loss_type='mcc')

nn.fit(X_train, y_train)

print('Overall AUC:', roc_auc_score(y_test, nn.predict_proba(X_test)[:,1]))

y_pred = nn.predict_proba(test)
sub["probability"]=y_pred[:,1]
sub.to_csv("/home/m/Numerai/numerai_datasets/Prediction.csv", index=False)

Was the above useful? Please share with others on social media.

If you want to look for more information on Python or Trading, check online courses available at udemy.com.

Recommended reading list:

 

Pairs Trading: Quantitative Methods and Analysis

The first in-depth analysis of pairs trading
Pairs trading is a market-neutral strategy in its most simple form. The strategy involves being long (or bullish) one asset and short (or bearish) another. If properly performed, the investor will gain if the market rises or falls. Pairs Trading reveals the secrets of this rigorous quantitative analysis program to provide individuals and investment houses with the tools they need to successfully implement and profit from this proven trading methodology. Pairs Trading contains specific and tested formulas for identifying and investing in pairs, and answers important questions such as what ratio should be used to construct the pairs properly.
Ganapathy Vidyamurthy (Stamford, CT) is currently a quantitative software analyst and developer at a major New York City hedge fund.
Machine Trading: Deploying Computer Algorithms to Conquer the Markets (Wiley Trading)

Dive into algo trading with step-by-step tutorials and expert insight
Machine Trading is a practical guide to building your algorithmic trading business. Written by a recognized trader with major institution expertise, this book provides step-by-step instruction on quantitative trading and the latest technologies available even outside the Wall Street sphere. You'll discover the latest platforms that are becoming increasingly easy to use, gain access to new markets, and learn new quantitative strategies that are applicable to stocks, options, futures, currencies, and even bitcoins. The companion website provides downloadable software codes, and you'll learn to design your own proprietary tools using MATLAB. The author's experiences provide deep insight into both the business and human side of systematic trading and money management, and his evolution from proprietary trader to fund manager contains valuable lessons for investors at any level.

Algorithmic trading is booming, and the theories, tools, technologies, and the markets themselves are evolving at a rapid pace. This book gets you up to speed, and walks you through the process of developing your own proprietary trading operation using the latest tools.

Utilize the newer, easier algorithmic trading platforms
Access markets previously unavailable to systematic traders
Adopt new strategies for a variety of instruments
Gain expert perspective into the human side of trading
The strength of algorithmic trading is its versatility. It can be used in any strategy, including market-making, inter-market spreading, arbitrage, or pure speculation; decision-making and implementation can be augmented at any stage, or may operate completely automatically. Traders looking to step up their strategy need look no further than Machine Trading for clear instruction and expert solutions.
Applied Quantitative Methods for Trading and Investment

This much-needed book, from a selection of top international experts, fills a gap by providing a manual of applied quantitative financial analysis. It focuses on advanced empirical methods for modelling financial markets in the context of practical financial applications.
Data, software and techniques specifically aligned to trading and investment will enable the reader to implement and interpret quantitative methodologies covering various models.

The unusually wide-ranging methodologies include not only the 'traditional' financial econometrics but also technical analysis systems and many nonparametric tools from the fields of data mining and artificial intelligence. However, for those readers wishing to skip the more theoretical developments, the practical application of even the most advanced techniques is made as accessible as possible.

The book will be read by quantitative analysts and traders, fund managers, risk managers; graduate students in finance and MBA courses.
Quantitative Technical Analysis: An integrated approach to trading system development and trading management

This book, the fifth by Dr. Howard Bandy, discusses an integrated approach to trading system development and trading management.

It begins with a discussion and quantification of the several aspects of risk.
1. The trader's personal tolerance for risk.
2. The risk inherent in the price fluctuations of the issue to be traded.
3. The risk added by the trading system rules.
4. The trade-by-trade risk experienced during trading.

An original objective function, called "CAR25," based on risk-normalized profit potential is developed and explained. CAR25 is as near a universal objective function as I have found.

The importance of recognizing the non-stationary characteristics of financial data, and techniques for handling it, are discussed.

There is a general discussion of trading system development, including design, testing, backtesting, optimization, and walk forward analysis. That is followed by two parallel development paths -- one using traditional trading system development platform and the second machine learning.

Recognizing the importance of position sizing in managing trading, an original technique based on empirical Bayesian analysis, called "dynamic position sizing" and quantified in a metric called "safe-f," is introduced. Computer code implementing dynamic position sizing is included in the book.

56 fully disclosed, ready-to-run, and downloadable programs are included.
Finding Alphas: A Quantitative Approach to Building Trading Strategies

Design more successful trading systems with this practical guide to identifying alphas
Finding Alphas seeks to teach you how to do one thing and do it well: design alphas. Written by experienced practitioners from WorldQuant, including its founder and CEO Igor Tulchinsky, this book provides detailed insight into the alchemic art of generating trading signals, and gives you access to the tools you need to practice and explore. Equally applicable across regions, this practical guide provides you with methods for uncovering the hidden signals in your data. A collection of essays provides diverse viewpoints to show the similarities, as well as unique approaches, to alpha design, covering a wide variety of topics, ranging from abstract theory to concrete technical aspects. You'll learn the dos and don'ts of information research, fundamental analysis, statistical arbitrage, alpha diversity, and more, and then delve into more advanced areas and more complex designs. The companion website, www.worldquantchallenge.com, features alpha examples with formulas and explanations. Further, this book also provides practical guidance for using WorldQuant's online simulation tool WebSim® to get hands-on practice in alpha design.

Alpha is an algorithm which trades financial securities. This book shows you the ins and outs of alpha design, with key insight from experienced practitioners.

Learn the seven habits of highly effective quants
Understand the key technical aspects of alpha design
Use WebSim® to experiment and create more successful alphas
Finding Alphas is the detailed, informative guide you need to start designing robust, successful alphas.
Inside the Black Box: A Simple Guide to Quantitative and High Frequency Trading

New edition of book that demystifies quant and algo trading
In this updated edition of his bestselling book, Rishi K Narang offers in a straightforward, nontechnical style—supplemented by real-world examples and informative anecdotes—a reliable resource takes you on a detailed tour through the black box. He skillfully sheds light upon the work that quants do, lifting the veil of mystery around quantitative trading and allowing anyone interested in doing so to understand quants and their strategies. This new edition includes information on High Frequency Trading.

Offers an update on the bestselling book for explaining in non-mathematical terms what quant and algo trading are and how they work
Provides key information for investors to evaluate the best hedge fund investments
Explains how quant strategies fit into a portfolio, why they are valuable, and how to evaluate a quant manager
This new edition of Inside the Black Box explains quant investing without the jargon and goes a long way toward educating investment professionals.
Automated Trading with R: Quantitative Research and Platform Development

Learn to trade algorithmically with your existing brokerage, from data management, to strategy optimization, to order execution, using free and publicly available data. Connect to your brokerage’s API, and the source code is plug-and-play.

Automated Trading with R explains automated trading, starting with its mathematics and moving to its computation and execution. You will gain a unique insight into the mechanics and computational considerations taken in building a back-tester, strategy optimizer, and fully functional trading platform.

The platform built in this book can serve as a complete replacement for commercially available platforms used by retail traders and small funds. Software components are strictly decoupled and easily scalable, providing opportunity to substitute any data source, trading algorithm, or brokerage. This book will:

Provide a flexible alternative to common strategy automation frameworks, like Tradestation, Metatrader, and CQG, to small funds and retail traders
Offer an understanding of the internal mechanisms of an automated trading system
Standardize discussion and notation of real-world strategy optimization problems
What You Will Learn

Understand machine-learning criteria for statistical validity in the context of time-series
Optimize strategies, generate real-time trading decisions, and minimize computation time while programming an automated strategy in R and using its package library
Best simulate strategy performance in its specific use case to derive accurate performance estimates
Understand critical real-world variables pertaining to portfolio management and performance assessment, including latency, drawdowns, varying trade size, portfolio growth, and penalization of unused capital
Who This Book Is For

Traders/practitioners at the retail or small fund level with at least an undergraduate background in finance or computer science; graduate level finance or data science students
Quantitative Trading with R: Understanding Mathematical and Computational Tools from a Quant's Perspective

Quantitative Finance with R offers a winning strategy for devising expertly-crafted and workable trading models using the R open source programming language, providing readers with a step-by-step approach to understanding complex quantitative finance problems and building functional computer code.
Quantitative Momentum: A Practitioner's Guide to Building a Momentum-Based Stock Selection System (Wiley Finance)

The individual investor's comprehensive guide to momentum investing
Quantitative Momentum brings momentum investing out of Wall Street and into the hands of individual investors. In his last book, Quantitative Value, author Wes Gray brought systematic value strategy from the hedge funds to the masses; in this book, he does the same for momentum investing, the system that has been shown to beat the market and regularly enriches the coffers of Wall Street's most sophisticated investors. First, you'll learn what momentum investing is not: it's not 'growth' investing, nor is it an esoteric academic concept. You may have seen it used for asset allocation, but this book details the ways in which momentum stands on its own as a stock selection strategy, and gives you the expert insight you need to make it work for you. You'll dig into its behavioral psychology roots, and discover the key tactics that are bringing both institutional and individual investors flocking into the momentum fold.

Systematic investment strategies always seem to look good on paper, but many fall down in practice. Momentum investing is one of the few systematic strategies with legs, withstanding the test of time and the rigor of academic investigation. This book provides invaluable guidance on constructing your own momentum strategy from the ground up.

Learn what momentum is and is not
Discover how momentum can beat the market
Take momentum beyond asset allocation into stock selection
Access the tools that ease DIY implementation
The large Wall Street hedge funds tend to portray themselves as the sophisticated elite, but momentum investing allows you to 'borrow' one of their top strategies to enrich your own portfolio. Quantitative Momentum is the individual investor's guide to boosting market success with a robust momentum strategy.
Quantitative Trading: Algorithms, Analytics, Data, Models, Optimization

The first part of this book discusses institutions and mechanisms of algorithmic trading, market microstructure, high-frequency data and stylized facts, time and event aggregation, order book dynamics, trading strategies and algorithms, transaction costs, market impact and execution strategies, risk analysis, and management. The second part covers market impact models, network models, multi-asset trading, machine learning techniques, and nonlinear filtering. The third part discusses electronic market making, liquidity, systemic risk, recent developments and debates on the subject.
https://quant-traders.com

Numerai tournament. Introductory code in Python.

In this post, I want to share, how simple it is to start competing in machine learning tournaments – Numerai. I will go step by step, line by line explaining what is doing what and why it is required.

Numerai is a global artificial intelligence competition to predict the behavior. Numerai is a little bit similar to Kaggle but with clean datasets, so we can pass over long data cleansing process.  You just download the data, build a model, and upload your predictions, that’s it. To extract most of the data you would initially do some feature engineering, but for simplicity of this intro, we will pass this bit over.  One more thing we will pass on is splitting out validation set, the main aim of this exercise is to fit ‘machine learning’ model to training dataset. Later using fitted model, generate a prediction.  All together it shouldn’t take more than 14 simple lines of python code, you can run them as one piece or run part by part in interactive mode.

Let’s go, let’s do some machine learning…

A first thing to do is to go to numer.ai, click on ‘Download Training Data’  and download datasets, after unzipping the archive, you will have few files in there, we are interested mainly in three of them. It is worth noting what is a path to the folder as we will need it later.

I assume you have installed python and required libraries, if not there is plenty of online tutorials on how to do it, I recommend installing Anaconda distribution. It it time to open whatever IDE you use, and start coding, first few lines will be just importing what we will use later, that is Pandas and ScikitLearn.

import pandas as pd 
from sklearn.ensemble import GradientBoostingClassifier

Pandas is used to import data from csv files and do some basic data manipulations, GradientBoostingClassifier as part of ScikitLearn will be the model we will use to fit and do predict. As we have required libraries imported let’s use them… in next three lines, we will import data from csv to memory.  We will use ‘read_csv’  method from pandas, all you need to do is amend the full path to each file, wherever you have extracted numerai_datasets.zip.

train = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_training_data.csv")
test  = pd.read_csv("/home/m/Numerai/numerai_datasets/numerai_tournament_data.csv")   
sub  = pd.read_csv("/home/m/Numerai/numerai_datasets/example_predictions.csv")

What above code does it creates three data frames and imports the csv files we have we have previously extracted from downloaded numerai_datasets.zip.

‘train’ –  this dataset contains all required data to train our model, so it has both ‘features’ and ‘labels’, so you can say it has both questions and answers that our model will ‘learn’

‘test’ – this one contains features but does not contain ‘labels’, you can say it contains questions and our model will deliver answers.

‘sub’ – it is just template for uploading our prediction

Let’s move on,  in next line will copy all unique row id’s from ‘test’ to ‘sub’ to make sure each predicted value will be assigned to a right set of features, let’s say we put question number next to our answer so whoever checks the test would now.

sub["t_id"]=test["t_id"]

As we have copied the ids to ‘sub’, we don’t need them anymore in ‘test’ (all rows will stay in same order), so we can get rid of them.

test.drop("t_id", axis=1,inplace=True)

In next two lines, we will separate ‘labels’ or target values from train dataset.

labels=train["target"]

train.drop("target", axis=1,inplace=True)

As we have prepared ‘train’ dataset, we can get our model to learn from it. First, we select model we want to use, it will be Gradient BoostingClassifier from ScikitLearn – no specific reason for using this one, you can use whatever you like eg. random forest, linear regression…

grd = GradientBoostingClassifier()

As we have a model defined, let’s have it learn from ‘train’ data.

grd.fit(train,labels)

Ok, now our model is well trained and ready to make predictions, as the task is called ‘classification’ we will predict what is a probability of each set of features belongs to one of two classes ‘0’ or ‘1’.

y_pred = grd.predict_proba(test)

We have a long list of predicted probabilities called ‘y_pred’, let’s attach it to ‘id’ we had separated previously.

sub["probability"]=y_pred[:,1]

And save it in csv format, to get uploaded.

sub.to_csv("/home/m/Numerai/numerai_datasets/SimplePrediction.csv", index=False)

The last thing to do is go back to numer.ai website and click on ‘Upload Predictions’… Good luck.

This was very simplistic and introductory example to start playing with numer.ai competitions and machine learning. I will try and come back with gradually more complicated versions, if you have any questions, suggestions or comments please go to ‘About’ section and contact me directly.

The full code below:

import pandas as pd 
from sklearn.ensemble import GradientBoostingClassifier 
train = pd.read_csv("C:/Users/Downloads/numerai_datasets/numerai_training_data.csv") 
test = pd.read_csv("C:/Users/Downloads/numerai_datasets/numerai_tournament_data.csv") 
sub = pd.read_csv("C:/Users/Downloads/numerai_datasets/example_predictions.csv") 
sub["t_id"]=test["t_id"] 
test.drop("t_id", axis=1,inplace=True) 
labels=train["target"] 
train.drop("target", axis=1,inplace=True)
grd = GradientBoostingClassifier() 
grd.fit(train,labels) 
y_pred = grd.predict_proba(test) 
sub["probability"]=y_pred[:,1] 
sub.to_csv("C:/Users/Downloads/numerai_datasets/SimplePrediction.csv", index=False)

Was the above useful? Please share with others on social media.

If you want to look for more information on Python or Trading, check online courses available at udemy.com.

Recommended reading list:

 

Pairs Trading: Quantitative Methods and Analysis

The first in-depth analysis of pairs trading
Pairs trading is a market-neutral strategy in its most simple form. The strategy involves being long (or bullish) one asset and short (or bearish) another. If properly performed, the investor will gain if the market rises or falls. Pairs Trading reveals the secrets of this rigorous quantitative analysis program to provide individuals and investment houses with the tools they need to successfully implement and profit from this proven trading methodology. Pairs Trading contains specific and tested formulas for identifying and investing in pairs, and answers important questions such as what ratio should be used to construct the pairs properly.
Ganapathy Vidyamurthy (Stamford, CT) is currently a quantitative software analyst and developer at a major New York City hedge fund.
Machine Trading: Deploying Computer Algorithms to Conquer the Markets (Wiley Trading)

Dive into algo trading with step-by-step tutorials and expert insight
Machine Trading is a practical guide to building your algorithmic trading business. Written by a recognized trader with major institution expertise, this book provides step-by-step instruction on quantitative trading and the latest technologies available even outside the Wall Street sphere. You'll discover the latest platforms that are becoming increasingly easy to use, gain access to new markets, and learn new quantitative strategies that are applicable to stocks, options, futures, currencies, and even bitcoins. The companion website provides downloadable software codes, and you'll learn to design your own proprietary tools using MATLAB. The author's experiences provide deep insight into both the business and human side of systematic trading and money management, and his evolution from proprietary trader to fund manager contains valuable lessons for investors at any level.

Algorithmic trading is booming, and the theories, tools, technologies, and the markets themselves are evolving at a rapid pace. This book gets you up to speed, and walks you through the process of developing your own proprietary trading operation using the latest tools.

Utilize the newer, easier algorithmic trading platforms
Access markets previously unavailable to systematic traders
Adopt new strategies for a variety of instruments
Gain expert perspective into the human side of trading
The strength of algorithmic trading is its versatility. It can be used in any strategy, including market-making, inter-market spreading, arbitrage, or pure speculation; decision-making and implementation can be augmented at any stage, or may operate completely automatically. Traders looking to step up their strategy need look no further than Machine Trading for clear instruction and expert solutions.
Applied Quantitative Methods for Trading and Investment

This much-needed book, from a selection of top international experts, fills a gap by providing a manual of applied quantitative financial analysis. It focuses on advanced empirical methods for modelling financial markets in the context of practical financial applications.
Data, software and techniques specifically aligned to trading and investment will enable the reader to implement and interpret quantitative methodologies covering various models.

The unusually wide-ranging methodologies include not only the 'traditional' financial econometrics but also technical analysis systems and many nonparametric tools from the fields of data mining and artificial intelligence. However, for those readers wishing to skip the more theoretical developments, the practical application of even the most advanced techniques is made as accessible as possible.

The book will be read by quantitative analysts and traders, fund managers, risk managers; graduate students in finance and MBA courses.
Quantitative Technical Analysis: An integrated approach to trading system development and trading management

This book, the fifth by Dr. Howard Bandy, discusses an integrated approach to trading system development and trading management.

It begins with a discussion and quantification of the several aspects of risk.
1. The trader's personal tolerance for risk.
2. The risk inherent in the price fluctuations of the issue to be traded.
3. The risk added by the trading system rules.
4. The trade-by-trade risk experienced during trading.

An original objective function, called "CAR25," based on risk-normalized profit potential is developed and explained. CAR25 is as near a universal objective function as I have found.

The importance of recognizing the non-stationary characteristics of financial data, and techniques for handling it, are discussed.

There is a general discussion of trading system development, including design, testing, backtesting, optimization, and walk forward analysis. That is followed by two parallel development paths -- one using traditional trading system development platform and the second machine learning.

Recognizing the importance of position sizing in managing trading, an original technique based on empirical Bayesian analysis, called "dynamic position sizing" and quantified in a metric called "safe-f," is introduced. Computer code implementing dynamic position sizing is included in the book.

56 fully disclosed, ready-to-run, and downloadable programs are included.
Finding Alphas: A Quantitative Approach to Building Trading Strategies

Design more successful trading systems with this practical guide to identifying alphas
Finding Alphas seeks to teach you how to do one thing and do it well: design alphas. Written by experienced practitioners from WorldQuant, including its founder and CEO Igor Tulchinsky, this book provides detailed insight into the alchemic art of generating trading signals, and gives you access to the tools you need to practice and explore. Equally applicable across regions, this practical guide provides you with methods for uncovering the hidden signals in your data. A collection of essays provides diverse viewpoints to show the similarities, as well as unique approaches, to alpha design, covering a wide variety of topics, ranging from abstract theory to concrete technical aspects. You'll learn the dos and don'ts of information research, fundamental analysis, statistical arbitrage, alpha diversity, and more, and then delve into more advanced areas and more complex designs. The companion website, www.worldquantchallenge.com, features alpha examples with formulas and explanations. Further, this book also provides practical guidance for using WorldQuant's online simulation tool WebSim® to get hands-on practice in alpha design.

Alpha is an algorithm which trades financial securities. This book shows you the ins and outs of alpha design, with key insight from experienced practitioners.

Learn the seven habits of highly effective quants
Understand the key technical aspects of alpha design
Use WebSim® to experiment and create more successful alphas
Finding Alphas is the detailed, informative guide you need to start designing robust, successful alphas.
Inside the Black Box: A Simple Guide to Quantitative and High Frequency Trading

New edition of book that demystifies quant and algo trading
In this updated edition of his bestselling book, Rishi K Narang offers in a straightforward, nontechnical style—supplemented by real-world examples and informative anecdotes—a reliable resource takes you on a detailed tour through the black box. He skillfully sheds light upon the work that quants do, lifting the veil of mystery around quantitative trading and allowing anyone interested in doing so to understand quants and their strategies. This new edition includes information on High Frequency Trading.

Offers an update on the bestselling book for explaining in non-mathematical terms what quant and algo trading are and how they work
Provides key information for investors to evaluate the best hedge fund investments
Explains how quant strategies fit into a portfolio, why they are valuable, and how to evaluate a quant manager
This new edition of Inside the Black Box explains quant investing without the jargon and goes a long way toward educating investment professionals.
Automated Trading with R: Quantitative Research and Platform Development

Learn to trade algorithmically with your existing brokerage, from data management, to strategy optimization, to order execution, using free and publicly available data. Connect to your brokerage’s API, and the source code is plug-and-play.

Automated Trading with R explains automated trading, starting with its mathematics and moving to its computation and execution. You will gain a unique insight into the mechanics and computational considerations taken in building a back-tester, strategy optimizer, and fully functional trading platform.

The platform built in this book can serve as a complete replacement for commercially available platforms used by retail traders and small funds. Software components are strictly decoupled and easily scalable, providing opportunity to substitute any data source, trading algorithm, or brokerage. This book will:

Provide a flexible alternative to common strategy automation frameworks, like Tradestation, Metatrader, and CQG, to small funds and retail traders
Offer an understanding of the internal mechanisms of an automated trading system
Standardize discussion and notation of real-world strategy optimization problems
What You Will Learn

Understand machine-learning criteria for statistical validity in the context of time-series
Optimize strategies, generate real-time trading decisions, and minimize computation time while programming an automated strategy in R and using its package library
Best simulate strategy performance in its specific use case to derive accurate performance estimates
Understand critical real-world variables pertaining to portfolio management and performance assessment, including latency, drawdowns, varying trade size, portfolio growth, and penalization of unused capital
Who This Book Is For

Traders/practitioners at the retail or small fund level with at least an undergraduate background in finance or computer science; graduate level finance or data science students
Quantitative Trading with R: Understanding Mathematical and Computational Tools from a Quant's Perspective

Quantitative Finance with R offers a winning strategy for devising expertly-crafted and workable trading models using the R open source programming language, providing readers with a step-by-step approach to understanding complex quantitative finance problems and building functional computer code.
Quantitative Momentum: A Practitioner's Guide to Building a Momentum-Based Stock Selection System (Wiley Finance)

The individual investor's comprehensive guide to momentum investing
Quantitative Momentum brings momentum investing out of Wall Street and into the hands of individual investors. In his last book, Quantitative Value, author Wes Gray brought systematic value strategy from the hedge funds to the masses; in this book, he does the same for momentum investing, the system that has been shown to beat the market and regularly enriches the coffers of Wall Street's most sophisticated investors. First, you'll learn what momentum investing is not: it's not 'growth' investing, nor is it an esoteric academic concept. You may have seen it used for asset allocation, but this book details the ways in which momentum stands on its own as a stock selection strategy, and gives you the expert insight you need to make it work for you. You'll dig into its behavioral psychology roots, and discover the key tactics that are bringing both institutional and individual investors flocking into the momentum fold.

Systematic investment strategies always seem to look good on paper, but many fall down in practice. Momentum investing is one of the few systematic strategies with legs, withstanding the test of time and the rigor of academic investigation. This book provides invaluable guidance on constructing your own momentum strategy from the ground up.

Learn what momentum is and is not
Discover how momentum can beat the market
Take momentum beyond asset allocation into stock selection
Access the tools that ease DIY implementation
The large Wall Street hedge funds tend to portray themselves as the sophisticated elite, but momentum investing allows you to 'borrow' one of their top strategies to enrich your own portfolio. Quantitative Momentum is the individual investor's guide to boosting market success with a robust momentum strategy.
Quantitative Trading: Algorithms, Analytics, Data, Models, Optimization

The first part of this book discusses institutions and mechanisms of algorithmic trading, market microstructure, high-frequency data and stylized facts, time and event aggregation, order book dynamics, trading strategies and algorithms, transaction costs, market impact and execution strategies, risk analysis, and management. The second part covers market impact models, network models, multi-asset trading, machine learning techniques, and nonlinear filtering. The third part discusses electronic market making, liquidity, systemic risk, recent developments and debates on the subject.