Hello,

I am writing a software that tries to "glue" tick data for a particular symbol
each day at a certain hour.

I request a tick by tick data with some overlap with the old data which
I have already processed the day before and then, after the overlap has
finished, I start to record the ticks.

I use as a "glue" marker a certain number of trades.

The thing that I see is that from one day to the next there are some
very small differences in the number of trades, in the order of milliseconds,
or, different number of ticks.

Should I do a "lenient" match or this is a bug in my software because
the tick stream is fixed?

Thanks

Never mind, I have found the bug, just an off by one. Unfortunately I am not able to delete my post...

Sorry to write again, but I see now something strange.

The bug has been corrected and I am now testing.

I make some tick requests with some minutes distance.

This is the request

HTD,@ESU13,1,,,,1,HTD$3

I simply want to have one day of tick data from oldest to newest.

But I have seen something strange.

This is the part which is different

HTD$3,2013-07-23 04:30:57.146,1692.75,3,63549,1692.50,1692.75,2517378,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63550,1692.50,1692.75,2517482,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63551,1692.50,1692.75,2517483,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,2,63553,1692.50,1692.75,2517484,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,2,63555,1692.50,1692.75,2517485,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63556,1692.50,1692.75,2517486,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63557,1692.50,1692.75,2517487,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63558,1692.50,1692.75,2517488,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63559,1692.50,1692.75,2517489,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63560,1692.50,1692.75,2517490,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63561,1692.50,1692.75,2517491,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63562,1692.50,1692.75,2517492,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,1,63563,1692.50,1692.75,2517493,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,25,63588,1692.50,1692.75,2517494,C,43,01,
HTD$3,2013-07-23 04:31:27.661,1692.75,11,63599,1692.50,1692.75,2517495,C,43,01,
HTD$3,2013-07-23 04:31:27.662,1692.75,4,63603,1692.50,1692.75,2517516,C,43,01,
HTD$3,2013-07-23 04:31:27.662,1692.75,1,63604,1692.50,1692.75,2517517,C,43,01,
HTD$3,2013-07-23 04:31:53.076,1692.75,1,63605,1692.50,1692.75,2517758,C,43,01,


You can see that there are 14 trades with millisecond 661 and two trades with
millisecond 662

BUT, in a former run, I had 14 trades with millisecond 661 and THREE trades
(not 2) with millisecond 662.


The other data seem coherent. There is only this mismatch.

Is it a bug in my software or could it be that the stream of trades is not
100% reproducible?

Thank you

Hello, looking at our current history on all server farms, I am seeing the same as what you have posted.

Additionally, I don't believe that data has been modified/corrected in any way on our servers because the total volume field matches up and the tickIDs are increasing. Usually when a correction is done, at least one of these 2 fields will show some evidence of the correction happening.

As a result, I think we need to start with the assumption that it is a bug in your software. Do you still have the old data that you were comparing to? Can you post it (or email it to dev support)? There might be something there that will point to the issue. Can you tell me how the old data was acquired? Was it also using a tick history request similar to the one you posted above? Or was it gathered from streaming data?

Hello,

I am doing the request as HTD, with a day of overlap, for example if the last
tick stored is yesterday, I will make a request of 2 days.

Then of course there will be trades which will be before the "glue" point, and
I simply wait for the sentinel tick which is the first which must match.

I have altered my software putting a "lenient" flag in which it accepts also a partial
match, I will put this flag to false and if the problem arises again I will post here
the data which my software thinks as incoherent.

Thank you

Hello,

yesterday there were no problems, also with the lenient flag turned off.

But now I have found the "same" strange thing, I receive one tick less.

This is my application log:

[738933] Catching up, I have 92 ticks saved to check
[882392] Sending request HTD,@ESU13,1,,,,1,HTD$2 to the socket
[393913] HISTORY: [1249] I am alive @ 24-07-13 07:44:18+0200 line: 1688.50,2,23735,1688.25,1688.50,4719587,C,43,01,
[381033] Overlap fail @ 15 for tick Tick [t Wed Jul 24 10:43:44 CEST 2013, tRAW 1374655424329 p 169225] it had to be Tick [t Wed Jul 24 10:43:32 CEST 2013, tRAW 1374655412514 p 169225]
java.lang.IllegalStateException
[738281] The client has not been able to process HTD$2,2013-07-24 04:43:50.022,1692.50,2,59750,1692.25,1692.50,4911504,C,43,01, I quit!
[418935] ----- ioexce java.net.SocketException: socket closed

Looking inside I see that I expect TWO ticks at the time 1374655412514 (java time)
which is 4.43.32.514 New York Time. But apparently you give me only one,
then the next tick is at 4.43.44 but I am still waiting the tick @ 4.43.32.514

It seems "strange" because the inconsistency is at the same time as yesterday,
more or less. So it seems strange that it is a bug on my side, because all yesterday
went fine (and I made a lot of tests)

In any case I can do a lenient comparison... just to let you know.

Thank you.

Hello, like with yesterday's example, I'm not seeing any evidence that this was anything on our end. I have verified that the same data exists on all server farms in history (only a single trade at that timestamp for @ESU13) and there have been no corrections to the data. I even checked the replay of data coming into the IQFeed servers and only a single trade was recorded at that timestamp.

I'm not 100% sure it would help in this case but I would probably recommend changing your code to use the HTT request from IQFeed (instead of the HTD) request so that you can request data by timeframe (you would then have a maximum of 1s overlap between your two requests instead of 1 day).

Hello,

I will follow your suggestion of using HTT instead of HTD, I will end up requesting less data.

Regarding the overlap I will try to be lenient, maybe it is some bug which I did not found yet.

Regards.