Openfeed Developer Guide

Introduction

Openfeed is a new real-time market data protocol and feed from Barchart.

Architecture

Multicast UDP Connectivity

Multicast UDP is the most efficient way to subscribe to a large number of products at a time. Consider using the UDP feed if you need simultaneous access to many markets, or if you need the lowest latency.

Channels and Feeds

Market data over multicast UDP is segmented across a number of channels. A channel is a logical grouping of instruments, typically corresponding to a group of products from a particular exchange. A channel is identified by a numeric channelId in the range of 1-255

Each channel is composed of a set of three multicast groups, called feeds.

Channel ID Product Incremental Feed Snapshot Feed Definition Feed
1 CME 239.255.201.1:10001 239.255.202.1:20001 239.255.203.1:30001
2 CBOT 239.255.201.2:10002 239.255.202.2:20002 239.255.203.2:30002
3 NYMEX 239.255.201.3:10003 239.255.202.3:20003 239.255.203.3:30003
4 COMEX 239.255.201.4:10004 239.255.202.4:20004 239.255.203.4:30004
5 MGEX 239.255.201.5:10005 239.255.202.5:20005 239.255.203.5:30005
6 CME Europe 239.255.201.6:10006 239.255.202.6:20006 239.255.203.6:30006
7 BM&F 239.255.201.7:10007 239.255.202.7:20007 239.255.203.7:30007
8 ICE UK 239.255.201.8:10008 239.255.202.8:20008 239.255.203.8:30008
9 BATS 239.255.201.9:10009 239.255.202.9:20009 239.255.203.9:30009
10 NYSE Index 239.255.201.10:10010 239.255.202.10:20010 239.255.203.10:30010
11 FTSE Index 239.255.201.11:10011 239.255.202.11:20011 239.255.203.11:30011
12 S&P / DJ Index 239.255.201.12:10012 239.255.202.12:20012 239.255.203.12:30012
13 FOREX 239.255.201.13:10013 239.255.202.13:20013 239.255.203.13:30013
14 TSX 239.255.201.14:10014 239.255.202.14:20014 239.255.203.14:30014
15 239.255.201.15:10015 239.255.202.15:20015 239.255.203.15:30015
16 239.255.201.16:10016 239.255.202.16:20016 239.255.203.16:30016
17 239.255.201.17:10017 239.255.202.17:20017 239.255.203.17:30017
18 ERIS 239.255.201.18:10018 239.255.202.18:20018 239.255.203.18:30018
19 NASDAQ Mutual Funds 239.255.201.19:10019 239.255.202.19:20019 239.255.203.19:30019
20 FINRA OTC 239.255.201.20:10020 239.255.202.20:20020 239.255.203.20:30020
21 LME 239.255.201.21:10021 239.255.202.21:20021 239.255.203.21:30021
22 CFE 239.255.201.22:10022 239.255.202.22:20022 239.255.203.22:30022
23 Euronext 239.255.201.23:10023 239.255.202.23:20023 239.255.203.23:30023
24 ASX 239.255.201.24:10024 239.255.202.24:20024 239.255.203.24:30024
25 Montreal 239.255.201.25:10025 239.255.202.25:20025 239.255.203.25:30025
26 EUREX 239.255.201.26:10026 239.255.202.26:20026 239.255.203.26:30026
27 NASDAQ Index 239.255.201.27:10027 239.255.202.27:20027 239.255.203.27:30027
28 CBOE Index 239.255.201.28:10028 239.255.202.28:20028 239.255.203.28:30028
29 239.255.201.29:10029 239.255.202.29:20029 239.255.203.29:30029
30 239.255.201.30:10030 239.255.202.30:20030 239.255.203.30:30030
31 239.255.201.31:10031 239.255.202.31:20031 239.255.203.31:30031
32 239.255.201.32:10032 239.255.202.32:20032 239.255.203.32:30032
33 Hong Kong Futures 239.255.201.33:10033 239.255.202.33:20033 239.255.203.33:30033
34 239.255.201.34:10034 239.255.202.34:20034 239.255.203.34:30034
35 ICE 239.255.201.35:10035 239.255.202.35:20035 239.255.203.35:30035

Incremental Feed

The incremental feed is a sequence of messages that reflect updates to the current state of the channel.

An Openfeed message on the incremental feed can contain either an instrument definition or a market update. The translator will send all current Instrument Definitions upon Sunday system startup. If a new instrument is created during the week, the translator will send the instrument definition prior to any market updates.

The packet sequence number will increment by 1 with every packet. A gap in sequence numbers indicates packet loss, and the client will need perform the channel recovery procedure in order to guarantee the correctness of the books and session statistics.

Snapshot Feed

The snapshot feed is used to recover the state of the channel in case the client suffers an outage, or joins the channel late.

The snapshot feed operates in a loop over all the instruments on the channel. The loop is configured per-channel to generate market snapshots at a certain rate. Once all snapshots have been sent, the sequence number resets to 1 and the loop restarts. The total number of markets available on a channel at the time the snapshot was generated is recorded in the totalCount field.

When a snapshot is generated, the syncSequence field will be populated with the last packet sequence number sent on the incremental channel. This is useful for synchronizing between the snapshot and incremental feeds when performing the channel recovery procedure.

Definition Feed

The definition feed is used to recover the instrument definitions on the channel. If the client misses the instrument definition broadcast on the incremental feed at Sunday start up, the definitions can be recovered from the definition feed. Additionally, if the client experiences packet loss or a client failure, the definition feed can be used to recover any newly created instrument definitions that may have been missed.

Like the snapshot feed, the definition feed operates in a loop over all the instruments on the channel. The totalCount field is populated with the total number instruments available at the time packet was sent. Additional instruments added to the channel during the instrument loop will included in the loop before restarts. next loop.

The syncSequence field is populated with the most recent packet sequence number sent on the incremental feed.

Packet Format

A single UDP packet contains one binary header and zero or more openfeed messages.

Binary Header

Each UDP packet payload starts with a binary header. The header fields are big-endian, unsigned integers.

alt text

  • The sequence number starts at 1 and increments by 1 with with every packet (including heartbeats). Each feed (incremental, snapshot, definition) on a channel has its own distinct sequence numbers.

  • The reset number will be changed to an arbitrary value every time that the channel has been reset. A channel will normally reset once a week on Sunday. The reset will also occur if there is a system failure or upgrade in the middle of the week. A channel reset means that the sequence number has been set to 1, and that all books and session statistics on the channel should be cleared.

  • The channel id identifies which multicast channel the packet was sent on.

  • The message count is the number of Openfeed messages in this packet, following the header.

  • Sixteen bytes are reserved for internal or future use.

Message Packing

Each Openfeed message is prepended with its length in bytes, using the RawVarint32 encoding from Protocol Buffers. This format is compatible with the Protocol Buffers parseDelimitedFrom() method, making it easy for client code to parse the list of messages.

Channel Recovery

The channel recovery procedure can be used to synchronize the client's state with the ticker plant, so that the all books and session statistics are accurate and the incremental feed can be applied. The channel recovery procedure should be performed when:

  • The client starts up after the channel has begun broadcasting; i.e. the first incremental sequence number seen is greater than one.

  • The client experiences packet loss on the incremental feed; i.e. an incremental sequence number is greater than the previous sequence number plus one

  • The client experiences a system outage that causes channel state to be lost.

Market Sequence Number

In addition to the packet sequence numbers, the Openfeed tickerplant assigns sequence numbers to individual markets. The market sequence number is useful for performing channel recovery because it allows the client to identify which individual markets have been impacted by a minor outage (such as packet loss). Market sequence numbers start at one and increment by one with every Market Update.

Recovery Procedure

  1. Record the earliest continuous incremental packet sequence number, or ECIPSN. If a gap in incremental sequence numbers occurs, restart the procedure and reset the ECIPSN. If an old or duplicate packet is received, drop it (check the reset number to see if the channel has been reset.)

  2. When applying an incremental market update message, compare the market sequence number to the previous market sequence number for that market. If no previous update has been processed, treat it as zero. If the new market sequence number is equal to one plus the previous number, then this market has not been affected by the outage and the update can be applied normally. Otherwise, place the update in a queue for this market. Treat any instrument definitions on the incremental feed like normal.

  3. Begin processing the definition feed once the syncSequence is greater than or equal to the the ECIPSN - 1. Count the number of unique instrument definitions that have been recovered. Once the totalCount field equals the number of recorded definitions AND the syncSequence is greater than or equal to the ECIPSN minus one, then the definition recovery is complete. If the syncSequence is less than the ECIPSN minus one, then it is still possible that the missing incremental packet(s) defined a new instrument.

  4. Begin processing the snapshot feed once the syncSequence is greater than or equal to th ECIPSN - 1. If the syncSequence is less than the ECIPSN minus one, then the snapshot is stale and should be discarded. If the marketSequence field of the the market snapshot message is equal to the last market update that was applied, then this market was not affected by thee outage and does not need a snapshot. If the snapshot's marketSequence field is greater than the last applied market update, apply it to the current market state. Discard any queued market update messages that have sequence numbers less than or equal to the snapshot's marketSequence. Apply the rest of the queued updates, so long as the sequences numbers keep incrementing by one. If there is gap in the queued updates, then another snapshot will be needed to restore the market state. Continue processing the snapshot loop until all markets have been restored.

  5. Once all market snapshots and instruments have been recovered, the recovery procedure is complete.

TCP Connectivity

Message Specification

The Openfeed wire protocol is based on Google's protocol buffers. Protocol buffers is a data serialization format that makes it easy to interoperate between any platform or language.

The protocol is specified in the protocol buffer language, which describes the structure of the messages and fields. These .proto files can be run through a protocol buffer compiler, which will generate efficient code in your programming language of choice for decoding and encoding the messages.

The Openfeed protocol is specified in two files:

  • openfeed.proto defines the real time market data messages

  • instrument.proto defines the instrument definition messages

Book Management

penfeed may disseminate orderbook information in two different formats: market-by-price and market-by-order. The instrument definition will specify which style of book should be used.

Market by Price

With market-by-price, order book information is consolidated and dissemenated at the price level. Price levels begin at index 1 and continue until book_depth.

Market by Order

Market by order provides full order-by-order view of the market. Availability of this feature is limited by exchange functionality.