Delayed Fetch Forward Failures ****

  • Recently we have been encountering this error 'delayed fetch forward' during clinical workflow. It blocks 'process after' function in ImportConverter, leading blocking in our clinical workflow.

    We define ExportConverter and ImportConverter in dicom.ini as follows. Between 7 and 18 many MR scans are made. They all need to be processed by an executable after 3 seconds when it arrives to this server and backed up to PAK_BACKUP server after 18:00.

    Code
    ForwardCollectDelay = 120
    MaximumExportRetries = 0
    MaximumDelayedFetchForwardRetries = 0
    ExportConverters = 1
    ExportConverter0 = between "6", "20"; defer; forward series to Server03
    ImportConverter0 = process series after 3 by A.exe "%V0010,0020"

    In the DelayedFetchForwardFailures*** file we observe the following two unexpected situations:

    1. many scans, made a day early, are still queuing to be forwarded to their destination; They should be forwarded last night after 18:00;
    2. 'process series after 3' is still in the queue 'after 3'; they should be triggered 3 seconds after it arrives the server.

    We suspect that the queuing function in DGATE, due to the ExportConverter and ImportConverter we introduced, is overflowed.

    In dgate.cpp the maximum queue size is defined to 128.

    Code
    int QueueSize = 128; // number of entries in export/copy queue (2k per entry in memory)

    Could you guide us to solve the problem? Thanks in advance.

  • Hi,


    which version are you using?


    Before 1.5.0, after 3 is 3s after the reception of the first image, recently it is 3s after reception of the last image. This could mean 3s is too short for older versions, the same data could be forwarded multiple times. I would change the "after 3" to something like "after 60" at least.


    Marcel

  • We currently use version 1.4.19d1.

    We have updated to this version last year because it is able to count the time after reception of the last image instead of the first image.

    We have tested the 'after 3'. It did start counting after reception of the last image.

    The code change is logged

    Quote

    20180301 mvh into_queue_unique resets time stamp for every save; silent destroy2 method

    20190205 mvh Made it 1.4.19d1

  • I get this bug report from my clinical colleague. I am trying to figure out why and help to find a solution. I do not have access to the clinical work station therefore I am not allowed to changed any setting for testing, unfortunately. My colleague needs to know the reason and then can apply the fix.

    What is the difference between delayed forward (forward series to) and normal forward (forward to)?

    When i use 'between defer forward series to' and 'process series after by', they end up into the same queue. If 'between defer' is on hold, all the elements after it will be blocked. But it still does not explain why after 18:00 in our case the scan is not forwarded to the destination AE.

    For 'between defer forward to' it uses a different queue than 'process series after by'. In this case 'process series after by' will always be executed even when 'between defer forward to' is on hold.

    Do I understand correctly?

    Where can I read more about this queue?

  • Each export converter has its own queue. Operations that work on a single image use this queue. Operations that work on e.g. a series level are delayed until the whole series arrived. But there is only a single queue for all 'delayed' operations. So 'forward series' and 'process series' will be in the same queue. The defer command will just stop this queue from being processed.


    If anywhere this information is in the manual which I meticulously keep synchronised with the code. But this is free software, I rely on your help to solve these kind of issues.


    I will myself have to readup on 'between' I must have written that code 10-15 years sgo....


    Marcel

  • Thanks for your reply. I think the Delayed Fetch Forward error is related to the following error message in the serverstatus.log

    Code
    xx-xx-2020 xx:xx:xx [LOCALSERVER] ***preretrieve/forward xxx to: association lost

    I can not figure out why it gets this error. I suspect the destination server somehow rejects the coming scan.

    My colleague found the following code which might play a role.

    Code
    20100122        mvh     Added experimental C-CANCEL support for all C-MOVES

    The Link.Poll() function could indicate whether there is still data coming from the receiver after start the transfer. Why is there always the "cancel request"?

    I am not still learning the code.

  • Hi,


    the only possible message from that side is a cancel. But the problem can't be this because the message does not show "Recieved cancel request".


    Basically "***preretrieve/forward xxx to: association lost" means Conquest has hung up during a move request to the PACS.


    Can you see any more messages if you enable debug loggin?


    Marcel

  • Do you know why Conquest has hung up? I could not figure it out.

    We also notice in the log file for every 300s (TCPICTimeout), Conquest re-tries.

    Code
    ...
    19-08-2020 18:03:21 [Server01] C-Move Destination: "Server02 "
    19-08-2020 18:08:21 [Server01] ***preretrieve/forward xxx to: association lost
    ...
    19-08-2020 18:10:02 [Server01] C-Move Destination: "Server02 "
    19-08-2020 18:14:57 [Server01] Number of Images to send: 251
  • The summary of this issue:

    In current setting for our Dicom server Server01

    Code
    CacheVirtualData = 1
    OverlapVirtualGet = 0
    VirtualServerFor0 = Server02,FIXKODAK,CACHESTUDIES
    ForwardCollectDelay = 120
    MaximumExportRetries = 0
    MaximumDelayedFetchForwardRetries = 0
    ExportConverters = 1
    ExportConverter0 = ifmatch "%u", "Server03"; stop; between "6", "20"; defer; forward series to Server03

    When the daily ExportConverter0 is started, Server01 will do a query on itself to collect the list of scans it needs to forward. Since we enable the ‘virtual’ server, Server01 will query Server02 on IMAGE level. This query on Server02 might take much more time than the query on itself.

    Code
    dgate.cpp
    VirtualQueryCached(DDOPtr, "IMAGE", N, &BDDO);

    As the amount of scans for a patient on Server02 grows, the query time increases.

    When it reaches the TCPICTimeout, the ExportConverter0 failes. This is the moment when the "***preretrieve/forward xxx to: association lost" is logged.

    Then this forward task is listed into the delayed queue and tried over and over again.

    Solution:

    Remove the virtual server from Server01.

  • d.geng

    Changed the title of the thread from “Delayed Fetch Forward Failures 5678” to “Delayed Fetch Forward Failures ****”.
  • Thanks for this very detailed analysis!


    I had not expected a query to go wrong. You see the importance to share configuration files when reporting a complex failure.


    Conquest can selectively disable virtual queries and moves (through a private DICOM tag 0x9999,0x0802 in the query data), but not easily through a forward statement. I have added this to the bug list to add in a future release.


    regards


    Marcel

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!