In this article we will be discussing how to obtain intraday historic US equities data from the DTN IQFeed market data vendor. It is possible to obtain the data through a socket connection to the local IQLink server that is provided when an account is created. In this article we will make use of a streaming socket connection with Python to buffer this data and create CSV files of intraday market data for US equities.

Note: I have no affiliation with DTN IQFeed beyond being a customer. I am simply writing this article to help those who have an IQFeed account (or are considering getting one) to download data without the need for GUI software.

Python Socket Connection to IQFeed

I will assume that you already have an account with IQFeed. If not then it is possible to obtain a two-week free trial upon sign-up. After signing up to the required exchanges and level of data granularity you will be asked to download the IQLink launcher. This tool only natively works on Windows but can be executed on a Mac or a Linux machine under WINE with a bit of work!

Firing up the IQLink launcher brings up the connection dialog:

DTN IQFeed IQLink Launcher

Clicking 'Start IQLink' will launch the server. You'll be prompted for your username and password. Once the server is running it is necessary to create a stream socket connection to a local port (9100 is the default). You can then send messages across this socket and receive the data in a buffered fashion.

The first task is to create the iqfeed.py file and import both the system and socket libraries:

# iqfeed.py

import sys
import socket

The buffering of the data is handled by the read_historical_data_socket function. It requires a socket object and the number of bytes to buffer per read. The function simply appends the latest batch of buffered data to a string and returns it once the "!ENDMSG!" string is found within (i.e. the buffer has reached the end of the data!):

# iqfeed.py

def read_historical_data_socket(sock, recv_buffer=4096):
    """
    Read the information from the socket, in a buffered
    fashion, receiving only 4096 bytes at a time.

    Parameters:
    sock - The socket object
    recv_buffer - Amount in bytes to receive per read
    """
    buffer = ""
    data = ""
    while True:
        data = sock.recv(recv_buffer)
        buffer += data

        # Check if the end message string arrives
        if "!ENDMSG!" in buffer:
            break
   
    # Remove the end message string
    buffer = buffer[:-12]
    return buffer

The socket must connect to the local machine on port 9100. For this example we are going to download four stock symbols: SPY, AAPL, GOOG and AMZN from the start of 2014 to the present.

IQFeed accepts messages in the following format: CMD,SYM,[options]\n. Notice the newline character. This must be added otherwise the message will not work. The provided options are [bars in seconds],[beginning date: CCYYMMDD HHmmSS],[ending date: CCYYMMDD HHmmSS],[empty],[beginning time filter: HHmmSS],[ending time filter: HHmmSS],[old or new: 0 or 1],[empty],[queue data points per second].

We will be using the following example message: "HIT,GOOG,60,20140101 075000,,,093000,160000,1\n". This says to return historical data (HIT) for the GOOG ticker symbol, at a rate of once per 60 seconds (i.e. minutely bars), from 07:50:00 on the 1st January 2014 with no end (i.e. to "yesterday"). The data is filtered to only be present from 09:30:00 until 16:00:00, which is the NYSE market time opening hours, using new data.

The first task is to define the host, port and symbols for download. Each of the four symbols is looped over and the message for historical data is constructed. Then a socket is opened. AF_INET specifies that a tuple of (host, port) should be expected when connecting. SOCK_STREAM states that the socket should be a stream socket.

Once the socket is opened the message is sent and the historical data is buffered, after which the socket is closed. All of the line endings are removed and the data is written to a file called "sym.csv" in the same directory as the Python code, where "sym" is the ticker symbol:

# iqfeed.py

if __name__ == "__main__":
    # Define server host, port and symbols to download
    host = "127.0.0.1"  # Localhost
    port = 9100  # Historical data socket port
    syms = ["SPY", "AAPL", "GOOG", "AMZN"]

    # Download each symbol to disk
    for sym in syms:
        print "Downloading symbol: %s..." % sym

        # Construct the message needed by IQFeed to retrieve data
        message = "HIT,%s,60,20140101 075000,,,093000,160000,1\n" % sym

        # Open a streaming socket to the IQFeed server locally
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.connect((host, port))

        # Send the historical data request
        # message and buffer the data
        sock.sendall(message)
        data = read_historical_data_socket(sock)
        sock.close

        # Remove all the endlines and line-ending
        # comma delimiter from each record
        data = "".join(data.split("\r"))
        data = data.replace(",\n","\n")[:-1]

        # Write the data stream to disk
        f = open("%s.csv" % sym, "w")
        f.write(data)
        f.close()

The format of the data is given by:

[YYYY-MM-DD HH:mm:SS],[OPEN],[LOW],[HIGH],[CLOSE],[VOLUME],[OPEN INTEREST]

A typical set of lines within a file would be as follows:

2012-01-03 09:31:00,30.6400,30.5000,30.6400,30.5100,6128,6128
2012-01-03 09:32:00,30.5600,30.4900,30.4900,30.5600,6528,400
2012-01-03 09:33:00,30.5000,30.5000,30.5000,30.5000,6672,144
2012-01-03 09:34:00,30.3800,30.3400,30.3400,30.3500,8423,1751
2012-01-03 09:35:00,30.5300,30.5300,30.5300,30.5300,8623,200
2012-01-03 09:36:00,30.6400,30.5500,30.5500,30.6400,9423,800
2012-01-03 09:37:00,30.6500,30.6500,30.6500,30.6500,10329,906
2012-01-03 09:38:00,30.6900,30.6600,30.6900,30.6600,12329,2000
2012-01-03 09:39:00,30.7200,30.6400,30.6500,30.7200,13729,1400
2012-01-03 09:40:00,30.7500,30.6900,30.7200,30.7500,17029,3300

The data goes back quite a long way in terms of years of availability. However it can take some time to download once you start considering large quantities of symbols going back five years or more. If you wish to download data types other than equities then take a look at the IQFeed symbol guide. Obviously you will need to be subscribed to the appropriate exchange for the symbol download to work.

Be aware that the intraday data provided by IQFeed is non-adjusted. That is to say it does not account for corporate actions such as dividends or stock splits. You will need to carry out your own corporate action handling.