Running out of Memory when looking at 400 equities level 1 or 76 futures contracts level 2. I am using the Python library which works great. I am having trouble determining what this means. It looks like my data feed stays up but this notification is concerning.

I'm a user like yourself, but I've seen a few posts in the past dealing with memory issue problems.

One thing: do you happen to know if your app makes a separate connection to IQConnect.exe for each symbol? Especially using python, I could see this adding up to a lot of CPU and memory overhead. Even though this strategy can be more convenient (because each program only has to deal with data for a single symbol), it can be a lot more memory and CPU efficient to open multiple symbol watches (say, 50 at a time) on a single connection to IQConnect and then use logic in your applicatoin to send each line to the right place based on the ticker symbol at the beginning (such as to a separate data file, to a data stream queue like Kafka, or to some other logic handler to calculate moving averages, etc).

For example, you could use a python dict where the key is the IQF symbol name and the value is an open file handle for that symbol (if you don't find it in the dict, you create a new file and then add it there) -- then check the start of each line to look up the right file to write it to. This is more complex because of the logic I mentioned but also because you'd need to do something like read a config file per process with the list of symbols you want to watch per process. If you got fancy, you could have a single file with all symbols, one per line, and then start the program by adding command line params that tell it which line to start at in the symbols file and how many of them to read for that particular process.

Anyway -- enough advice based on guessing. If you're already reading multiple symbols per process, then it must be something else.

---

-Mark D. Valor
Edited by mkvalor on Feb 23, 2024 at 02:01 AM
Edited by mkvalor on Feb 23, 2024 at 02:03 AM

It is always advisable to keep the number of connection to a minimum, and keep them open rather than frequently closing/reopening. IQFeed will allow you to collect to make multiple connections to the same socket from the same machine, and this has its uses. (Making 2/4 connections, and dividing the workload among them, can help overall speed in handling a large number of symbols.) But having a very large number of connections, or frequently disconnecting and reconnecting, are counterproductive and may contribute to this kind of situation happening.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

I'm wondering if keeping it on all the time will drain the battery?

It sounds like your system is encountering memory issues when processing a large amount of data for 400 equities at level 1 or 76 futures contracts at level 2. This notification indicates that your system's memory resources may be insufficient to handle the volume of data you're trying to process. It would be advisable to review your system's memory allocation and consider optimizing your code to handle large datasets more efficiently.