We run an application which subscribes to depth data for hundreds of stocks. Recently, DTN has inconsistently been telling us that certain symbols are not found.
The problem started last week, but was fine by the time trading hours began. Today the problem persisted into market hours.
Our code worked for a year with no delay between the requests to watch the stocks. We added a delay today which helped a little.
The symbols we get the "not found" message for are not consistent and retrying a symbol will work about half of the time.

We run on Linux through WINE where DTN's depth application works consistently.
Here's an example for TIGR where it worked on the fifth try. Before the '|' character is our internal timestamp.

2015-12-28 13:53:14.553014|n,TIGR.PK
2015-12-28 13:53:28.262000|n,TIGR.PK
2015-12-28 13:53:32.867000|n,TIGR.PK
2015-12-28 13:53:34.322000|n,TIGR.PK
2015-12-28 13:53:35.177002|Z,TIGR.PK,PUMA,0.17,0.195,5000,5000,08:10:07.004,2015-12-28,52,08:10:07.004,T,T,T,
2015-12-28 13:53:35.177003|Z,TIGR.PK,CSTI,0.0001,0.23,10000,2500,08:30:24.045,2015-12-28,52,08:30:24.045,T,T,T,
2015-12-28 13:53:35.177004|Z,TIGR.PK,NITE,0.17,0.18,150895,5000,13:52:01.054,2015-12-28,52,13:52:01.054,T,T,T,
2015-12-28 13:53:35.177005|Z,TIGR.PK,BTIG,0.0001,0.49,10000,2500,07:35:03.056,2015-12-28,52,07:35:03.056,T,T,T,
2015-12-28 13:53:35.177006|Z,TIGR.PK,MAXM,0.01,200.,10000,1,07:35:24.001,2015-12-28,52,07:35:24.001,T,T,T,
2015-12-28 13:53:35.177007|Z,TIGR.PK,ETRF,0.,0.,0,0,99:99:99.000,2015-12-28,52,99:99:99.000,F,F,T,
2015-12-28 13:53:35.177008|Z,TIGR.PK,CANT,0.1,0.28,5000,2500,08:30:23.012,2015-12-28,52,08:30:23.012,T,T,T,
2015-12-28 13:53:35.177009|Z,TIGR.PK,CDEL,0.107,0.2,5000,20000,13:52:01.045,2015-12-28,52,13:52:01.045,T,T,T,
2015-12-28 13:53:35.177010|Z,TIGR.PK,ATDF,0.0001,0.55,10000,2000,10:08:09.040,2015-12-28,52,10:08:09.040,T,T,T,
2015-12-28 13:59:32.507000|2,TIGR.PK,NITE,0.17,0.18,141895,5000,13:59:32.420,2015-12-28,52,13:59:32.420,T,T,T,
2015-12-28 13:59:32.612001|2,TIGR.PK,CDEL,0.107,0.2,5000,20000,13:59:32.519,2015-12-28,52,13:59:32.519,T,T,T,
2015-12-28 13:59:32.612002|2,TIGR.PK,CDEL,0.1151,0.2,5000,20000,13:59:32.519,2015-12-28,52,13:59:32.519,T,T,T,
2015-12-28 14:11:06.603000|2,TIGR.PK,NITE,0.17,0.18,106895,5000,14:11:06.519,2015-12-28,52,14:11:06.519,T,T,T,
2015-12-28 14:11:06.737000|2,TIGR.PK,CDEL,0.1151,0.2,5000,20000,14:11:06.639,2015-12-28,52,14:11:06.639,T,T,T,
2015-12-28 14:11:06.737001|2,TIGR.PK,CDEL,0.1198,0.2,5000,20000,14:11:06.639,2015-12-28,52,14:11:06.639,T,T,T,
Edited by eporter on Dec 28, 2015 at 02:12 PM

Thanks for the additional detail, I will check with the server team and see if anything appears out of line that would help explain the inconsistencies and get back with you.

Tim

Retrying symbols seems to work with about 50% probability. We've seen multiple instances where a stock finally worked on the 4th attempt. Perhaps one backend server is misconfigured? It seems like there is some source of nondeterminism.

We only subscribe to penny stocks, and they have worked for months without any issues until last week. We didn't change anything on our side.

We continue to have problems today, and once again retrying seems to work around the issue. We're watching 1250 symbols, and below is a histogram of the number of symbols that succeeded after a given number of reattempts.

302 #1
126 #2
38 #3
17 #4
7 #5
2 #6
1 #8

Remarkably, one succeeded only on the 8th attempt to subscribe it. I'm not looking to cast blame, but I can't think of anything on our end that could cause this behavior. There seems to be about 50% chance of each attempt failing, and all attempts seem to be independent. It's like we have to keep retrying until we get routed to the right server.

Just an update on this, I believe I now understand what is happening here and I will get back with more detail once I get things confirmed.

Tim

I made a little progress on this by altering my code to wait for DTN to send me a 'Z' message or an 'n' message before watching another symbol. Previously my code would watch new symbols without waiting for a response to the prior requests. This worked just fine from late 2014 until Dec 28 2015.

Unfortunately, it still fails 2-3% of the time, again on random symbols. Maybe something changed in the server code when the 5.2 protocol was launched?

This is the result of changes in how the exchange sends out messages to our level 2 client and how we handle the occurrence when no data exists for a given symbol. We are looking at better methods of handling this, but there is still more we are planning on doing. I will check to see if a ETA is available yet, but I know this is something we would like to see go out as soon as possible.

Tim

Eric,

We are going to try to have this out Monday morning for you. What you can expect is that in the event of no current data being available, we will return a market maker of NODATA with zeroed fields to indicate that no positions on this symbol are currently held. Data will flow through the normal data stream when new data arrives and the NODATA market maker will drop off upon the receipt of new data. If all the data should clear off of the stack the NODATA market maker will return to hold the symbol open and valid.

Also, these rules only apply to symbols that have had recent L2 data activity. If a symbol has not had any recent L2 activity it will continue to respond with the current behavior. So expect Monday to take a little time to get settled in as the first bids or asks arrive for each symbol.

I will confirm every thing is up as expected Monday morning.

Tim

Tim,

We haven't seen any changes to the behavior of L2 data this week. We still get the 'n' messages at the same rate and I haven't seen the NODATA market maker yet.

I'll check with the server team today for an update on where they are with this. I will send you an email shortly to have you try something a little different as well.

Tim