Openfeed Developer Guide
Openfeed is a new real-time market data protocol and feed from Barchart.
Multicast UDP Connectivity
Multicast UDP is the most efficient way to subscribe to a large number of products at a time. Consider using the UDP feed if you need simultaneous access to many markets, or if you need the lowest latency.
Channels and Feeds
Market data over multicast UDP is segmented across a number of channels. A channel is a logical grouping of instruments, typically corresponding to a group of products from a particular exchange. A channel is identified by a numeric channelId in the range of 1-255
Each channel is composed of a set of three multicast groups, called feeds.
|Channel ID||Product||Incremental Feed||Snapshot Feed||Definition Feed|
|12||S&P / DJ Index||220.127.116.11:10012||18.104.22.168:20012||22.214.171.124:30012|
|19||NASDAQ Mutual Funds||126.96.36.199:10019||188.8.131.52:20019||184.108.40.206:30019|
|33||Hong Kong Futures||220.127.116.11:10033||18.104.22.168:20033||22.214.171.124:30033|
The incremental feed is a sequence of messages that reflect updates to the current state of the channel.
An Openfeed message on the incremental feed can contain either an instrument definition or a market update. The translator will send all current Instrument Definitions upon Sunday system startup. If a new instrument is created during the week, the translator will send the instrument definition prior to any market updates.
The packet sequence number will increment by 1 with every packet. A gap in sequence numbers indicates packet loss, and the client will need perform the channel recovery procedure in order to guarantee the correctness of the books and session statistics.
The snapshot feed is used to recover the state of the channel in case the client suffers an outage, or joins the channel late.
The snapshot feed operates in a loop over all the instruments on the channel. The loop is configured per-channel to generate market snapshots at a certain rate. Once all snapshots have been sent, the sequence number resets to 1 and the loop restarts. The total number of markets available on a channel at the time the snapshot was generated is recorded in the totalCount field.
When a snapshot is generated, the syncSequence field will be populated with the last packet sequence number sent on the incremental channel. This is useful for synchronizing between the snapshot and incremental feeds when performing the channel recovery procedure.
The definition feed is used to recover the instrument definitions on the channel. If the client misses the instrument definition broadcast on the incremental feed at Sunday start up, the definitions can be recovered from the definition feed. Additionally, if the client experiences packet loss or a client failure, the definition feed can be used to recover any newly created instrument definitions that may have been missed.
Like the snapshot feed, the definition feed operates in a loop over all the instruments on the channel. The totalCount field is populated with the total number instruments available at the time packet was sent. Additional instruments added to the channel during the instrument loop will included in the loop before restarts. next loop.
The syncSequence field is populated with the most recent packet sequence number sent on the incremental feed.
A single UDP packet contains one binary header and zero or more openfeed messages.
Each UDP packet payload starts with a binary header. The header fields are big-endian, unsigned integers.
|Channel type||1||byte||0 - inc, 1 - snapshot, 2 - definitions.|
|Packet length||2||short||Total packet length, including header.|
|Sequence Number||8||long||Openfeed sequence.|
|Channel Reset||1||byte||Channel reset indicator.|
|Channel Id||2||short||Openfeed Channel Id|
|Message count||2||short||number of messages in the packet.|
The sequence number starts at 1 and increments by 1 with with every packet (including heartbeats). Each feed (incremental, snapshot, definition) on a channel has its own distinct sequence numbers.
The reset number will be changed to an arbitrary value every time that the channel has been reset. A channel will normally reset once a week on Sunday. The reset will also occur if there is a system failure or upgrade in the middle of the week. A channel reset means that the sequence number has been set to 1, and that all books and session statistics on the channel should be cleared.
The channel id identifies which multicast channel the packet was sent on.
The message count is the number of Openfeed messages in this packet, following the header.
Each Openfeed message is prepended with the following header. The header fields are big-endian, unsigned integers.
|Length||2||short||Message Length, not including header.|
|Message Type||1||byte||Message type. Trade/ BBO, etc.|
|Exchange Id||2||short||Exchange Id. This is a new enumerated value.|
|Market Id||8||long||Openfeed market id.|
|Market sequence Number||8||long||Instrument (market) sequence|
|Message Data||variable||protobuf||Openfeed protocol buffer encoded message|
The channel recovery procedure can be used to synchronize the client's state with the ticker plant, so that the all books and session statistics are accurate and the incremental feed can be applied. The channel recovery procedure should be performed when:
The client starts up after the channel has begun broadcasting; i.e. the first incremental sequence number seen is greater than one.
The client experiences packet loss on the incremental feed; i.e. an incremental sequence number is greater than the previous sequence number plus one
The client experiences a system outage that causes channel state to be lost.
Market Sequence Number
In addition to the packet sequence numbers, the Openfeed tickerplant assigns sequence numbers to individual markets. The market sequence number is useful for performing channel recovery because it allows the client to identify which individual markets have been impacted by a minor outage (such as packet loss). Market sequence numbers start at one and increment by one with every Market Update.
Record the earliest continuous incremental packet sequence number, or ECIPSN. If a gap in incremental sequence numbers occurs, restart the procedure and reset the ECIPSN. If an old or duplicate packet is received, drop it (check the reset number to see if the channel has been reset.)
When applying an incremental market update message, compare the market sequence number to the previous market sequence number for that market. If no previous update has been processed, treat it as zero. If the new market sequence number is equal to one plus the previous number, then this market has not been affected by the outage and the update can be applied normally. Otherwise, place the update in a queue for this market. Treat any instrument definitions on the incremental feed like normal.
Begin processing the definition feed once the syncSequence is greater than or equal to the the ECIPSN - 1. Count the number of unique instrument definitions that have been recovered. Once the totalCount field equals the number of recorded definitions AND the syncSequence is greater than or equal to the ECIPSN minus one, then the definition recovery is complete. If the syncSequence is less than the ECIPSN minus one, then it is still possible that the missing incremental packet(s) defined a new instrument.
Begin processing the snapshot feed once the syncSequence is greater than or equal to th ECIPSN - 1. If the syncSequence is less than the ECIPSN minus one, then the snapshot is stale and should be discarded. If the marketSequence field of the the market snapshot message is equal to the last market update that was applied, then this market was not affected by thee outage and does not need a snapshot. If the snapshot's marketSequence field is greater than the last applied market update, apply it to the current market state. Discard any queued market update messages that have sequence numbers less than or equal to the snapshot's marketSequence. Apply the rest of the queued updates, so long as the sequences numbers keep incrementing by one. If there is gap in the queued updates, then another snapshot will be needed to restore the market state. Continue processing the snapshot loop until all markets have been restored.
Once all market snapshots and instruments have been recovered, the recovery procedure is complete.
The Openfeed wire protocol is based on Google's protocol buffers. Protocol buffers is a data serialization format that makes it easy to interoperate between any platform or language.
The protocol is specified in the protocol buffer language, which describes the structure of the messages and fields. These .proto files can be run through a protocol buffer compiler, which will generate efficient code in your programming language of choice for decoding and encoding the messages.
The Openfeed protocol is specified in two files:
openfeed.proto defines the real time market data messages
instrument.proto defines the instrument definition messages
penfeed may disseminate orderbook information in two different formats: market-by-price and market-by-order. The instrument definition will specify which style of book should be used.
Market by Price
With market-by-price, order book information is consolidated and dissemenated at the price level. Price levels begin at index 1 and continue until book_depth.
Market by Order
Market by order provides full order-by-order view of the market. Availability of this feature is limited by exchange functionality.