In the previous article in the Advanced Trading Infrastructure series I discussed and presented both the code and initial unit tests for the Position class that stores positional information about a trade. In this article we will consider the Portfolio class, used to store a list of Position classes, as well as a cash balance.
In the last month I've made a lot of progress on QSTrader, the open-source backtesting and live-trading engine that is the culmination of these articles. In fact, I've actually finalised an entire end-to-end "first draft" of the code, which makes use of a simplistic (but highly unprofitable!) test strategy, to ensure the code works as it should. However, I still wish to write these articles sequentially by explaining each module and how it works.
I'm hopeful that by doing so it will make it much easier for many of you to contribute to the project by adding various new components, such as risk handlers or portfolio sizers that others in the QuantStart community can make use of.
At this stage there is little-to-no documentation beyond these articles and a big part of making QSTrader a viable backtesting library is to have it extremely well documented. Once the code is further along I will start to produce some in-depth documentation and tutorials that should help you get backtesting very quickly, irrespective of your choice of operating system or trading frequency.
To reiterate, the project can always be found at https://www.github.com/mhallsmoore/qstrader under a liberal open-source MIT license.
Component Design Reminder
In the previous article we talked briefly about the components that make up QSTrader. I have now extended this list to include the "full" set of components necessary for a backtest.
Many of these modules will be familiar to users of QSForex and my previous event-driven backtester used in Successful Algorithmic Trading. The primary difference here is that each of these classes is unit tested and will be much more feature-rich than those in previous versions.
The current design is as follows:
- Position - The
Positionclass encapsulates all data associated with an open position in an asset. That is, it tracks the realised and unrealised profit and loss (PnL) by averaging the multiple "legs" of the transaction, inclusive of transaction costs. - Portfolio - The
Portfolioclass that encapsulates a list ofPositions, as well as a cash balance, equity and PnL. - PositionSizer - The
PositionSizerclass provides thePortfolioHandler(see below) with guidance on how to size positions once a strategy signal is received. For instance, thePositionSizercould incorporate a Kelly Criterion approach. - RiskManager - The
RiskManageris used by thePortfolioHandlerto verify, modify or veto any suggested trades that pass through from thePositionSizer, based on the current composition of the portfolio and external risk considerations (such as correlation to indices or volatility). - PortfolioHandler - The
PortfolioHandlerclass is responsible for the management of the currentPortfolio, interacting with theRiskManagerandPositionSizeras well as submitting orders to be executed by anExecutionHandler. - Event - The
Eventclass and its inherited subclass are used to pass around event messages to each component of the system. They are always sent to a Python event queue to be read by these components. Event subclasses includeTickEvent,OrderEvent,SignalEventandFillEvent. - Strategy - The
Strategyclass handles the logic of generating trading signals based on the pricing information. It sends these signals to thePortfolioHandler. - ExecutionHandler - The
ExecutionHandlerreads inOrderEvents and producesFillEvents, based either on a simulated fill scenario or the actual fill information from a brokerage, such as Interactive Brokers. - PriceHandler - This class is designed to be subclassed to allow connection to multiple data sources such as CSV, HDF5, RDBMS (MySQL, SQLServer, PostgreSQL), MongoDB or a brokerage live-streaming API, for instance.
- Backtest - The
Backtestclass ties together all of the previous components to produce a simulated backtest. It is "swapped out" with a live trading engine class (to be developed), along with aPriceHandlerandExecutionHandler, once live trading is to be carried out.
What's missing from this list so far? Perhaps the most important missing piece is any mechanism for calculating trade strategy statistics and viewing the results. This includes performance metrics like Sharpe Ratio and Maximum Drawdown, as well as an equity curve, returns profile and drawdown curve.
Rather than strongly-coupling the results to the PortfolioHandler class, as in the previous QSForex and the event-driven backtester codes, we are going to generate a Results or Statistics class that will calculate and store the necessary performance metrics based on the results of a backtest. We can then use these classes to produce further "client" utilities, such as a web interface or GUI tool, to view the results of a backtest.
In addition, there is no mention of robustness, logging or monitoring within the above list. These are crucial components in a production-ready backtesting and live trading engine and will be added as the project develops. These components will likely make use of some form of server/cloud infrastructure, such as Amazon Web Services (or other cloud vendor).
Let's now turn our attention to the Portfolio class. In later articles we will consider the PortfolioHandler and how it interacts with the PositionSizer and RiskManager.
Portfolio
I want to emphasise again at this stage that the Portfolio class found in QSTrader is very different from that used in QSForex or the event-driven backtester. I have split the previous designs of the portfolio into two classes now, one called Portfolio and the other called PortfolioHandler.
What is the reason for this change? Primarily, I wanted to create a lean Portfolio class that did very little except store the current cash value and a list of Position objects. The only method that is called publicly (to borrow a C++ term!) is transact_position, which simply tells the Portfolio to update its position in a particular equity. It handles all of the necessary profit and loss (PnL) calculations, leading to both realised and unrealised PnL.
This means that the PortfolioHandler class can concentrate on other tasks, such as interacting with the RiskManager and PositionSizer classes, leaving all of the necessary financial calculations to the Portfolio. It also makes it more straightforward to test each class individually, as one is heavy on financial calculation, while the other is used more for interacting with other components.
I'll output the code listings for both position.py and position_test.py in full and then run through how each of them works.
Note that any of these listings are subject to change, since I will be continually making changes to this project. Eventually I hope others will collaborate by providing Pull Requests to the codebase.
portfolio.py
from decimal import Decimal
from qstrader.position.position import Position
class Portfolio(object):
def __init__(self, price_handler, cash):
"""
On creation, the Portfolio object contains no
positions and all values are "reset" to the initial
cash, with no PnL - realised or unrealised.
"""
self.price_handler = price_handler
self.init_cash = cash
self.cur_cash = cash
self.positions = {}
self._reset_values()
def _reset_values(self):
"""
This is called after every position addition or
modification. It allows the calculations to be
carried out "from scratch" in order to minimise
errors.
All cash is reset to the initial values and the
PnL is set to zero.
"""
self.cur_cash = self.init_cash
self.equity = self.cur_cash
self.unrealised_pnl = Decimal('0.00')
self.realised_pnl = Decimal('0.00')
def _update_portfolio(self):
"""
Updates the Portfolio total values (cash, equity,
unrealised PnL, realised PnL, cost basis etc.) based
on all of the current ticker values.
This method is called after every Position modification.
"""
for ticker in self.positions:
pt = self.positions[ticker]
self.unrealised_pnl += pt.unrealised_pnl
self.realised_pnl += pt.realised_pnl
self.cur_cash -= pt.cost_basis
pnl_diff = pt.realised_pnl - pt.unrealised_pnl
self.cur_cash += pnl_diff
self.equity += (
pt.market_value - pt.cost_basis + pnl_diff
)
def _add_position(
self, action, ticker,
quantity, price, commission
):
"""
Adds a new Position object to the Portfolio. This
requires getting the best bid/ask price from the
price handler in order to calculate a reasonable
"market value".
Once the Position is added, the Portfolio values
are updated.
"""
self._reset_values()
if ticker not in self.positions:
bid, ask = self.price_handler.get_best_bid_ask(ticker)
position = Position(
action, ticker, quantity,
price, commission, bid, ask
)
self.positions[ticker] = position
self._update_portfolio()
else:
print(
"Ticker %s is already in the positions list. " \
"Could not add a new position." % ticker
)
def _modify_position(
self, action, ticker,
quantity, price, commission
):
"""
Modifies a current Position object to the Portfolio.
This requires getting the best bid/ask price from the
price handler in order to calculate a reasonable
"market value".
Once the Position is modified, the Portfolio values
are updated.
"""
self._reset_values()
if ticker in self.positions:
self.positions[ticker].transact_shares(
action, quantity, price, commission
)
bid, ask = self.price_handler.get_best_bid_ask(ticker)
self.positions[ticker].update_market_value(bid, ask)
self._update_portfolio()
else:
print(
"Ticker %s not in the current position list. " \
"Could not modify a current position." % ticker
)
def transact_position(
self, action, ticker,
quantity, price, commission
):
"""
Handles any new position or modification to
a current position, by calling the respective
_add_position and _modify_position methods.
Hence, this single method will be called by the
PortfolioHandler to update the Portfolio itself.
"""
if ticker not in self.positions:
self._add_position(
action, ticker, quantity,
price, commission
)
else:
self._modify_position(
action, ticker, quantity,
price, commission
)
As with the position.py listing in the previous article we make extensive use of the Python decimal module. As I've mentioned before this is an absolute necessity in financial calculations as otherwise you will receive rounding errors due to the mathematics of floating point operations.
In the initialisation method of the Portfolio class we take a PriceHandler parameter as well as an initial cash balance (which is a Decimal datatype, not a floating point value). This is all we need to create a Portfolio instance.
In the method itself we create an initial cash and current cash value. We then create a dictionary of positions and finally call the _reset_values method, that resets all cash calculations and sets all PnL values to zero:
class Portfolio(object):
def __init__(self, price_handler, cash):
"""
On creation, the Portfolio object contains no
positions and all values are "reset" to the initial
cash, with no PnL - realised or unrealised.
"""
self.price_handler = price_handler
self.init_cash = cash
self.cur_cash = cash
self.positions = {}
self._reset_values()
As mentioned above, _reset_values is called upon initialisation, but it is also called upon every position modification. This may seem unwieldy, but it heavily reduces errors in the calculation process. It simply resets the current cash and equity values to the initial cash value and then zeroes the PnL values:
# portfolio.py
def _reset_values(self):
"""
This is called after every position addition or
modification. It allows the calculations to be
carried out "from scratch" in order to minimise
errors.
All cash is reset to the initial values and the
PnL is set to zero.
"""
self.cur_cash = self.init_cash
self.equity = self.cur_cash
self.unrealised_pnl = Decimal('0.00')
self.realised_pnl = Decimal('0.00')
The next method is _update_portfolio. This method is also called after every position modification (i.e. transaction). For every ticker in the Portfolio, the unrealised and realised PnL of the whole portfolio are increased by each positions PnL, while the current available cash is reduced by the positions cost basis. Finally, the difference in realised and unrealised PnL is applied to the current cash and the total portfolio equity is adjusted:
# portfolio.py
def _update_portfolio(self):
"""
Updates the Portfolio total values (cash, equity,
unrealised PnL, realised PnL, cost basis etc.) based
on all of the current ticker values.
This method is called after every Position modification.
"""
for ticker in self.positions:
pt = self.positions[ticker]
self.unrealised_pnl += pt.unrealised_pnl
self.realised_pnl += pt.realised_pnl
self.cur_cash -= pt.cost_basis
pnl_diff = pt.realised_pnl - pt.unrealised_pnl
self.cur_cash += pnl_diff
self.equity += (
pt.market_value - pt.cost_basis + pnl_diff
)
While this may seem a little complex, I have carried these calculations out primarily so that they reflect how portfolios are adjusted in major brokerages, particular Interactive Brokers. It means that the backtesting engine should produce values close to that of live trading, under the assumption of slippage and transaction costs.
The next two methods are _add_position and _modify_position. Originally, I had these two methods as the "publicly" callable methods for creating new positions and then subsequently modifying them. I later felt that it wasn't necessary for the user to keep track of whether to add or modify a position, and so I introduced a wrapper method, called transact_position that now correctly utilises the necessary method depending upon the existence of a ticker in the positions dictionary.
_add_position takes an action (buy or sell), a ticker symbol, a quantity of shares, the fill price and the cost of commission, as parameters. Firstly we reset the entire portfolio values and then get the best bid and ask price of the ticker from the price handler object. Then we create the new Position, utilising these bid and ask prices to get an up to date "market value". Finally we add the Position instance to the positions dictionary, using the ticker symbol as a key*.
Notice that we call _update_portfolio to update all market values at this stage. The method also handles the case where the position already exists, printing some information to the console. In the future we will replace all instances of console output such as this with more robust logging mechanisms.
*This will have design implications later when we come to handle renaming of ticker symbols, multiple share classes and other corporate actions. However, for simplicity at this stage we will make use of the ticker symbol as it is unique for our purposes.
# portfolio.py
def _add_position(
self, action, ticker,
quantity, price, commission
):
"""
Adds a new Position object to the Portfolio. This
requires getting the best bid/ask price from the
price handler in order to calculate a reasonable
"market value".
Once the Position is added, the Portfolio values
are updated.
"""
self._reset_values()
if ticker not in self.positions:
bid, ask = self.price_handler.get_best_bid_ask(ticker)
position = Position(
action, ticker, quantity,
price, commission, bid, ask
)
self.positions[ticker] = position
self._update_portfolio()
else:
print(
"Ticker %s is already in the positions list. " \
"Could not add a new position." % ticker
)
_modify_position is similar to add position except that we call transact_shares of the Position class instead of creating a new position:
# portfolio.py
def _modify_position(
self, action, ticker,
quantity, price, commission
):
"""
Modifies a current Position object to the Portfolio.
This requires getting the best bid/ask price from the
price handler in order to calculate a reasonable
"market value".
Once the Position is modified, the Portfolio values
are updated.
"""
self._reset_values()
if ticker in self.positions:
self.positions[ticker].transact_shares(
action, quantity, price, commission
)
bid, ask = self.price_handler.get_best_bid_ask(ticker)
self.positions[ticker].update_market_value(bid, ask)
self._update_portfolio()
else:
print(
"Ticker %s not in the current position list. " \
"Could not modify a current position." % ticker
)
The method that is actually externally called is transact_position. It encompasses both creation and modification to a Position object. It simply chooses the correct method out of _add_position and _modify_position when making a new share transaction:
# portfolio.py
def transact_position(
self, action, ticker,
quantity, price, commission
):
"""
Handles any new position or modification to
a current position, by calling the respective
_add_position and _modify_position methods.
Hence, this single method will be called by the
PortfolioHandler to update the Portfolio itself.
"""
if ticker not in self.positions:
self._add_position(
action, ticker, quantity,
price, commission
)
else:
self._modify_position(
action, ticker, quantity,
price, commission
)
That concludes the Portfolio class. It provides a robust self-contained mechanism for grouping Position classes with a cash balance.
For completeness you can find the full code for the Portfolio class on Github at portfolio.py.
portfolio_test.py
As with position_test.py, I've created portfolio_test.py, which includes a basic sanity check unit test for multiple transactions of AMZN and GOOG shares. There is certainly more work to be done here to check larger, more diverse portfolios, but this at least ensures that the system is calculating values as it should.
As with the tests for the Position class these have been checked against the values produced by Interactive Brokers using the demo account of Trader Workstation. As before, I do fully anticipate finding new edge cases, and possibly bugs, but hopefully the current sanity check and calculation test should provide confidence in the Portfolio results.
The full listing of position_test.py is as follows:
from decimal import Decimal
import unittest
from qstrader.portfolio.portfolio import Portfolio
class PriceHandlerMock(object):
def __init__(self):
pass
def get_best_bid_ask(self, ticker):
prices = {
"GOOG": (Decimal("705.46"), Decimal("705.46")),
"AMZN": (Decimal("564.14"), Decimal("565.14")),
}
return prices[ticker]
class TestAmazonGooglePortfolio(unittest.TestCase):
"""
Test a portfolio consisting of Amazon and
Google/Alphabet with various orders to create
round-trips for both.
These orders were carried out in the Interactive Brokers
demo account and checked for cash, equity and PnL
equality.
"""
def setUp(self):
"""
Set up the Portfolio object that will store the
collection of Position objects, supplying it with
$500,000.00 USD in initial cash.
"""
ph = PriceHandlerMock()
cash = Decimal("500000.00")
self.portfolio = Portfolio(ph, cash)
def test_calculate_round_trip(self):
"""
Purchase/sell multiple lots of AMZN and GOOG
at various prices/commissions to check the
arithmetic and cost handling.
"""
# Buy 300 of AMZN over two transactions
self.portfolio.transact_position(
"BOT", "AMZN", 100,
Decimal("566.56"), Decimal("1.00")
)
self.portfolio.transact_position(
"BOT", "AMZN", 200,
Decimal("566.395"), Decimal("1.00")
)
# Buy 200 GOOG over one transaction
self.portfolio.transact_position(
"BOT", "GOOG", 200,
Decimal("707.50"), Decimal("1.00")
)
# Add to the AMZN position by 100 shares
self.portfolio.transact_position(
"SLD", "AMZN", 100,
Decimal("565.83"), Decimal("1.00")
)
# Add to the GOOG position by 200 shares
self.portfolio.transact_position(
"BOT", "GOOG", 200,
Decimal("705.545"), Decimal("1.00")
)
# Sell 200 of the AMZN shares
self.portfolio.transact_position(
"SLD", "AMZN", 200,
Decimal("565.59"), Decimal("1.00")
)
# Multiple transactions bundled into one (in IB)
# Sell 300 GOOG from the portfolio
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.92"), Decimal("1.00")
)
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.90"), Decimal("0.00")
)
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.92"), Decimal("0.50")
)
# Finally, sell the remaining GOOG 100 shares
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.78"), Decimal("1.00")
)
# The figures below are derived from Interactive Brokers
# demo account using the above trades with prices provided
# by their demo feed.
self.assertEqual(self.portfolio.cur_cash, Decimal("499100.50"))
self.assertEqual(self.portfolio.equity, Decimal("499100.50"))
self.assertEqual(self.portfolio.unrealised_pnl, Decimal("0.00"))
self.assertEqual(self.portfolio.realised_pnl, Decimal("-899.50"))
if __name__ == "__main__":
unittest.main()
The first task is to carry out the correct imports. We import the unittest module as well as the Portfolio object itself:
from decimal import Decimal
import unittest
from qstrader.portfolio.portfolio import Portfolio
In order to create a functioning Portfolio class we need a PriceHandler class to provide bid and ask values for each ticker. However, we have not coded up any price handler objects yet - so what are we to do?
As it turns out, this is a common pattern in unit testing. To overcome this difficulty, we can create a mock object. Essentially, a mock object is a class that simulates the behaviour of its real counterpart, thus allowing functionality to be tested on other classes that make use of it. Hence we need to create a PriceHandlerMock class that provides the same interface as a PriceHandler, but ultimately just returns preset values, rather than carrying out any "real" price calculations.
The PriceHandlerMock object has an empty initialisation method, but exposes the get_best_bid_ask method that is found on the real PriceHandler. It simply returns preset bid/ask values for GOOG and AMZN shares that we will be transacting in the further unit tests below:
class PriceHandlerMock(object):
def __init__(self):
pass
def get_best_bid_ask(self, ticker):
prices = {
"GOOG": (Decimal("705.46"), Decimal("705.46")),
"AMZN": (Decimal("564.14"), Decimal("565.14")),
}
return prices[ticker]
The actual unit tests consist of creating a new, rather verbosely named, class called TestAmazonGooglePortfolio. As with all unit tests in Python it is derived from the unittest.TestCase class.
In the setUp method we set the price handler mock object, the initial cash and create the Portfolio:
class TestAmazonGooglePortfolio(unittest.TestCase):
"""
Test a portfolio consisting of Amazon and
Google/Alphabet with various orders to create
round-trips for both.
These orders were carried out in the Interactive Brokers
demo account and checked for cash, equity and PnL
equality.
"""
def setUp(self):
"""
Set up the Portfolio object that will store the
collection of Position objects, supplying it with
$500,000.00 USD in initial cash.
"""
ph = PriceHandlerMock()
cash = Decimal("500000.00")
self.portfolio = Portfolio(ph, cash)
The only unit test method we create is called test_calculate_round_trip. Its goal is to calculate full round-trip trades of AMZN and GOOG, making sure that the financial calculations of the Position and Portfolio classes are correct. "Correct" in this instance means that they match the values calculated by Interactive Brokers when I carried out this situation in Trader Workstation. I've hardcoded these values into the unit test.
The first part of the method carries out multiple transactions in both GOOG and AMZN at various prices and commission costs. I took these prices directly from those calculated by Interactive Brokers (IB) when I carried out these actual trades in the demo account. "BOT" is IB terminology for buying a share, while "SLD" is terminology for selling a share.
Once the full set of transactions have been completed, the positions are both netted out to zero in quantity. They will have no unrealised PnL, but will have a finite realised PnL, as well as modifications to current cash and the total equity value:
# portfolio_test.py
def test_calculate_round_trip(self):
"""
Purchase/sell multiple lots of AMZN and GOOG
at various prices/commissions to check the
arithmetic and cost handling.
"""
# Buy 300 of AMZN over two transactions
self.portfolio.transact_position(
"BOT", "AMZN", 100,
Decimal("566.56"), Decimal("1.00")
)
self.portfolio.transact_position(
"BOT", "AMZN", 200,
Decimal("566.395"), Decimal("1.00")
)
# Buy 200 GOOG over one transaction
self.portfolio.transact_position(
"BOT", "GOOG", 200,
Decimal("707.50"), Decimal("1.00")
)
# Add to the AMZN position by 100 shares
self.portfolio.transact_position(
"SLD", "AMZN", 100,
Decimal("565.83"), Decimal("1.00")
)
# Add to the GOOG position by 200 shares
self.portfolio.transact_position(
"BOT", "GOOG", 200,
Decimal("705.545"), Decimal("1.00")
)
# Sell 200 of the AMZN shares
self.portfolio.transact_position(
"SLD", "AMZN", 200,
Decimal("565.59"), Decimal("1.00")
)
# Multiple transactions bundled into one (in IB)
# Sell 300 GOOG from the portfolio
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.92"), Decimal("1.00")
)
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.90"), Decimal("0.00")
)
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.92"), Decimal("0.50")
)
# Finally, sell the remaining GOOG 100 shares
self.portfolio.transact_position(
"SLD", "GOOG", 100,
Decimal("704.78"), Decimal("1.00")
)
# The figures below are derived from Interactive Brokers
# demo account using the above trades with prices provided
# by their demo feed.
self.assertEqual(self.portfolio.cur_cash, Decimal("499100.50"))
self.assertEqual(self.portfolio.equity, Decimal("499100.50"))
self.assertEqual(self.portfolio.unrealised_pnl, Decimal("0.00"))
self.assertEqual(self.portfolio.realised_pnl, Decimal("-899.50"))
Clearly there is scope for producing far more unit tests of this situation, especially when more exotic positions are used, such as those with forex, futures or options. However, at this stage we are simply supporting equities and ETFs, which means more straightforward position handling.
The full listing can be found on Github at portfolio_test.py.
Next Steps
Now that we've discussed both the Position and Portfolio classes we need to consider the PortfolioHandler. This is the class that interacts with the PositionSizer and RiskManager to produce orders and receive fills that ultimately determine our equity portfolio (and thus profitability!).
Since I am much further ahead with the actual software development of QSTrader than I am with the articles explaining how it works, I'll be presenting some more advanced trading strategies using the software soon, rather than waiting until all of the articles have been completed.
