In this article Frank Smietana, one of QuantStart's expert guest contributors describes the Python open-source backtesting software landscape, and provides advice on which backtesting framework is suitable for your own project needs.
Backtesting is arguably the most critical part of the Systematic Trading Strategy (STS) production process, sitting between strategy development and deployment (live trading). If a strategy is flawed, rigorous backtesting will hopefully expose this, preventing a loss-making strategy from being deployed.
A number of related capabilities overlap with backtesting, including trade simulation and live trading. Backtesting uses historic data to quantify STS performance. Trading simulators take backtesting a step further by visualizing the triggering of trades and price performance on a bar-by-bar basis. Simulated/live trading deploys a tested STS in real time: signaling trades, generating orders, routing orders to brokers, then maintaining positions as orders are executed.
Most frameworks go beyond backtesting to include some live trading capabilities. This is convenient if you want to deploy from your backtesting framework, which also works with your preferred broker and data sources. Quantopian/Zipline goes a step further, providing a fully integrated development, backtesting, and deployment solution.
The Python community is well served, with at least six open source backtesting frameworks available. They are however, in various stages of development and documentation. If you enjoy working on a team building an open source backtesting framework, check out their Github repos.
Before evaluating backtesting frameworks, it’s worth defining the requirements of your STS.
What asset class(es) are you trading? While most of the frameworks support US Equities data via YahooFinance, if a strategy incorporates derivatives, ETFs, or EM securities, the data needs to be importable or provided by the framework. Asset class coverages goes beyond data. Can the framework handle finite length futures & options and generate roll-over trades automatically? What about illiquid markets, how realistic an assumption must be made when executing large orders?
What data frequency and detail is your STS built on? A trading system requiring every tick or bid/ask has a very different set of data management issues than a 5 minute or hourly interval. Hedge funds & HFT shops have invested significantly in building robust, scalable backtesting frameworks to handle that data volume and frequency. Some platforms provide a rich and deep set of data for various asset classes like S&P stocks, at one minute resolution.
What order type(s) does your STS require? At a minimum, limit, stops and OCO should be supported by the framework.
Level of support & documentation required. The early stage frameworks have scant documentation, few have support other than community boards.
The Components of a Backtesting Framework
Data and STS acquisition: The acquisition components consume the STS script/definition file and provide the requisite data for testing. If the framework requires any STS to be recoded before backtesting, then the framework should support canned functions for the most popular technical indicators to speed STS testing. Users determine how long of a historical period to backtest based on what the framework provides, or what they are capable of importing.
Performance testing applies the STS logic to the requested historic data window and calculates a broad range of risk & performance metrics, including max drawdown, Sharpe & Sortino ratios. Most all of the frameworks support a decent number of visualization capabilities, including equity curves and deciled-statistics.
Optimization tends to require the lion’s share of computing resources in the STS process. If your STS require optimization, then focus on a framework that supports scalable distributed/parallel processing.
In the context of strategies developed using technical indicators, system developers attempt to find an optimal set of parameters for each indicator. Most simply, optimization might find that a 6 and 10 day moving average crossover STS accumulated more profit over the historic test data than any other combination of time periods between 1 and 20. Already with this trivial example, 20 * 20 = 400 parameter combinations must be calculated & ranked.
In a portfolio context, optimization seeks to find the optimal weighting of every asset in the portfolio, including shorted and leveraged instruments. On a periodic basis, the portfolio is rebalanced, resulting in the purchase and sale of portfolio holdings as required to align with the optimized weights.
Position sizing is an additional use of optimization, helping system developers simulate and analyze the impact of leverage and dynamic position sizing on STS and portfolio performance.
Six Backtesting Frameworks for Python
Standard capabilities of open source Python backtesting platforms seem to include:
- Event driven
- Very flexible, unrestrictive licensing
- Decent collection of pre-defined technical indicators
- Standard performance metric calculation/visualization/reporting capabilities
PyAlgoTrade is a muture, fully documented backtesting framework along with paper- and live-trading capabilities. Data support includes Yahoo! Finance, Google Finance, NinjaTrader and any type of CSV-based time-series such as Quandl. Supported order types include Market, Limit, Stop and StopLimit.
PyAlgoTrade supports Bitcoin trading via Bitstamp, and real-time Twitter event handling.
bt - Backtesting for Python
bt “aims to foster the creation of easily testable, re-usable and flexible blocks of strategy logic to facilitate the rapid development of complex trading strategies”.
The framework is particularly suited to testing portfolio-based STS, with algos for asset weighting and portfolio rebalancing. Modifying a strategy to run over different time frequencies or alternate asset weights involves a minimal code tweak. bt is built atop ffn - a financial function library for Python.
This platform is exceptionally well documented, with an accompanying blog and an active on-line community for posting questions and feature requests. Backtrader supports a number of data formats, including CSV files, Pandas DataFrames, blaze iterators and real time data feeds from three brokers. These data feeds can be accessed simultaneously, and can even represent different timeframes. Supported brokers include Oanda for FX trading and multi-asset class trading via Interactive Brokers and Visual Chart.
pysystemtrade developer Rob Carver has a great post discussing why he set out to create yet another Python backtesting framework and the arguments for and against framework development. The backtesting framework for pysystemtrade is discussed in Rob’s book, "Systematic Trading".
pysystemtrade lists a number of roadmap capabilities, including a full-featured back tester that includes optimisation and calibration techniques, and fully automated futures trading with Interactive Brokers. Open source contributors are welcome.
Zipline is an algorithmic trading simulator with paper and live trading capabilities. Accessible via the browser-based IPython Notebook interface, Zipline provides an easy to use alternative to command line tools. Supported and developed by Quantopian, Zipline can be used as a standalone backtesting framework or as part of a complete Quantopian/Zipline STS development, testing and deployment environment. Zipline provides 10 years of minute-resolution historical US stock data and a number of data import options.
QSTrader is a backtesting framework with live trading capabilities. QuantStart Founder Michael Halls-Moore launched QSTrader with the intent of building a platform robust and scalable enough to service the needs of institutional quant hedge funds as well as retail quant traders. QSTrader currently supports OHLCV "bar" resolution data on various time scales, but does allow for tick data to be used.
Both backtesting and live trading are completely event-driven, streamlining the transition of strategies from research to testing and finally live trading. Core strategy/portfolio code is often identical across both deployments.
The main benefit of QSTrader is in its modularity, allowing extensive customisation of code for those who have specific risk or portfolio management requirements.
Embracing the Backtest
It is human nature to focus on the reward of developing a (hopefully profitable) STS, then rush to deploy a funded account (because we are hopeful), without spending sufficient time and resources thoroughly backtesting the strategy. But backtesting is not just a gatekeeper to prevent us from deploying flawed strategies and losing trading capital, it also provides a number of diagnostics that can inform the STS development process. For example, testing an identical STS over two different time frames, understanding a strategy’s max drawdown in the context of asset correlations, and creating smarter portfolios by backtesting asset allocations across multiple geographies.
In future posts, we'll cover backtesting frameworks for non-Python environments, and the use of various sampling techniques like bootstrapping and jackknife for backtesting predictive trading models.