In this article we will be discussing how to obtain intraday historic US equities data from the DTN IQFeed market data vendor. It is possible to obtain the data through a socket connection to the local IQLink server that is provided when an account is created. In this article we will make use of a streaming socket connection with Python to buffer this data and create CSV files of intraday market data for US equities.
Note: I have no affiliation with DTN IQFeed beyond being a customer. I am simply writing this article to help those who have an IQFeed account (or are considering getting one) to download data without the need for GUI software.
I will assume that you already have an account with IQFeed. If not then it is possible to obtain a two-week free trial upon sign-up. After signing up to the required exchanges and level of data granularity you will be asked to download the IQLink launcher. This tool only natively works on Windows but can be executed on a Mac or a Linux machine under WINE with a bit of work!
Firing up the IQLink launcher brings up the connection dialog:
DTN IQFeed IQLink Launcher
Clicking 'Start IQLink' will launch the server. You'll be prompted for your username and password. Once the server is running it is necessary to create a stream socket connection to a local port (9100 is the default). You can then send messages across this socket and receive the data in a buffered fashion.
The first task is to create the
iqfeed.py file and import both the system and socket libraries:
# iqfeed.py import sys import socket
The buffering of the data is handled by the
read_historical_data_socket function. It requires a socket object and the number of bytes to buffer per read. The function simply appends the latest batch of buffered data to a string and returns it once the "!ENDMSG!" string is found within (i.e. the buffer has reached the end of the data!):
# iqfeed.py def read_historical_data_socket(sock, recv_buffer=4096): """ Read the information from the socket, in a buffered fashion, receiving only 4096 bytes at a time. Parameters: sock - The socket object recv_buffer - Amount in bytes to receive per read """ buffer = "" data = "" while True: data = sock.recv(recv_buffer) buffer += data # Check if the end message string arrives if "!ENDMSG!" in buffer: break # Remove the end message string buffer = buffer[:-12] return buffer
The socket must connect to the local machine on port 9100. For this example we are going to download four stock symbols: SPY, AAPL, GOOG and AMZN from the start of 2014 to the present.
IQFeed accepts messages in the following format:
CMD,SYM,[options]\n. Notice the newline character. This must be added otherwise the message will not work. The provided options are [bars in seconds],[beginning date: CCYYMMDD HHmmSS],[ending date: CCYYMMDD HHmmSS],[empty],[beginning time filter: HHmmSS],[ending time filter: HHmmSS],[old or new: 0 or 1],[empty],[queue data points per second].
We will be using the following example message:
"HIT,GOOG,60,20140101 075000,,,093000,160000,1\n". This says to return historical data (
HIT) for the GOOG ticker symbol, at a rate of once per 60 seconds (i.e. minutely bars), from 07:50:00 on the 1st January 2014 with no end (i.e. to "yesterday"). The data is filtered to only be present from 09:30:00 until 16:00:00, which is the NYSE market time opening hours, using new data.
The first task is to define the host, port and symbols for download. Each of the four symbols is looped over and the message for historical data is constructed. Then a socket is opened.
AF_INET specifies that a tuple of
(host, port) should be expected when connecting.
SOCK_STREAM states that the socket should be a stream socket.
Once the socket is opened the message is sent and the historical data is buffered, after which the socket is closed. All of the line endings are removed and the data is written to a file called "sym.csv" in the same directory as the Python code, where "sym" is the ticker symbol:
# iqfeed.py if __name__ == "__main__": # Define server host, port and symbols to download host = "127.0.0.1" # Localhost port = 9100 # Historical data socket port syms = ["SPY", "AAPL", "GOOG", "AMZN"] # Download each symbol to disk for sym in syms: print "Downloading symbol: %s..." % sym # Construct the message needed by IQFeed to retrieve data message = "HIT,%s,60,20140101 075000,,,093000,160000,1\n" % sym # Open a streaming socket to the IQFeed server locally sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((host, port)) # Send the historical data request # message and buffer the data sock.sendall(message) data = read_historical_data_socket(sock) sock.close # Remove all the endlines and line-ending # comma delimiter from each record data = "".join(data.split("\r")) data = data.replace(",\n","\n")[:-1] # Write the data stream to disk f = open("%s.csv" % sym, "w") f.write(data) f.close()
The format of the data is given by:
[YYYY-MM-DD HH:mm:SS],[OPEN],[LOW],[HIGH],[CLOSE],[VOLUME],[OPEN INTEREST]
A typical set of lines within a file would be as follows:
2012-01-03 09:31:00,30.6400,30.5000,30.6400,30.5100,6128,6128 2012-01-03 09:32:00,30.5600,30.4900,30.4900,30.5600,6528,400 2012-01-03 09:33:00,30.5000,30.5000,30.5000,30.5000,6672,144 2012-01-03 09:34:00,30.3800,30.3400,30.3400,30.3500,8423,1751 2012-01-03 09:35:00,30.5300,30.5300,30.5300,30.5300,8623,200 2012-01-03 09:36:00,30.6400,30.5500,30.5500,30.6400,9423,800 2012-01-03 09:37:00,30.6500,30.6500,30.6500,30.6500,10329,906 2012-01-03 09:38:00,30.6900,30.6600,30.6900,30.6600,12329,2000 2012-01-03 09:39:00,30.7200,30.6400,30.6500,30.7200,13729,1400 2012-01-03 09:40:00,30.7500,30.6900,30.7200,30.7500,17029,3300
The data goes back quite a long way in terms of years of availability. However it can take some time to download once you start considering large quantities of symbols going back five years or more. If you wish to download data types other than equities then take a look at the IQFeed symbol guide. Obviously you will need to be subscribed to the appropriate exchange for the symbol download to work.
Be aware that the intraday data provided by IQFeed is non-adjusted. That is to say it does not account for corporate actions such as dividends or stock splits. You will need to carry out your own corporate action handling.comments powered by Disqus
You'll get instant access to a free 10-part email course packed with hints and tips to help you get started in quantitative trading!
Every week I'll send you a wrap of all activity on QuantStart so you'll never miss a post again.
Real, actionable quant trading tips with no nonsense.