The Problem

The customer reported a problem when running the CORE HL7 Sender where the Data Folder contained several hundred thousand HL7 data files. When Running Locally they could see the Sender failing with error code - RUNTIME ERROR: Exception [10388450] - An item with the same key has already been added.

We identified in our source code where the order was being produced, but in our examination of the code we could not figure out HOW the error could possibly occur. So more investigation was needed.

The Description

In our customer service call with the customer in our discussions we determined that the only thing "special" about these HL7 data files was that they had extracted them all from a ZIP file into the Data Folder. Knowing this, it became clear to us what was happening. It was a bug in our "Folder Polling" object which has existed predominantly unchanged for a very long time with no reported errors. The actual cause of the error was that when the data files were extracted into the Data Folder the Last Modified date of all (or at least enough of the files) was set to the same value.

Our CORE Folder Poller

Our Folder Polling object is really very simple. It is designed to examine a folder and build an internal indexed list of all files matching a certain file spec (Example: *.hl7). Because of our requirement to always process HL7 data files in a "First In First Out" order the sorting index of this internal linked list consisted of: The Last Modified date of the file (down to the millisecond) formatted into a string PLUS a rolling integer value which would cycle between 1 and 9999 before resetting back to 1. Thus, even if files were detected with the same Last Modified date, the internal index would always be unique allowing up to 9999 unique values within ONE millisecond. Sounds pretty bullet-proof right! Well, not so much, it turns out.

Scenario to Recreate the Error

Prior to version 2.3.1 IF you have a ZIP file containing MORE than 9999 files AND you extract them to the Data Folder AND your machine is speedy enough to do this quickly enough so that all (or at least more than 9999) files have the EXACT same Last Modified date (down to the millisecond) it would break our folder polling object.

How We Fixed It

The code change was extremely simple. We just extended the rolling integer value to extend from a maximum of 4 digits (9999) to a maximum of 7 digits (9999999). In our estimation that is good enough. Theoretically, you can still recreate the error with a folder with more than 10 million HL7 data files with the EXACT same Last Modified date, but our response for customers that might encounter that problem is to say "Don't do that!". If you have massive numbers of HL7 data files to transmit we recommend that you move them into your Data Folder in manageable numbers (say 50 or 100 thousand at a time) and let them process through.