My Conquest Setup :: Dicom Routing/Failover/Load Balance

  • Hey Conquest Users and Admins. I've been working on our setup for many many months and have finally got it to do everything we need it to do. So, I thought I'd share my configs in case anyone else is interested in doing the same type of thing.


    Prior to Conquest... We have two PACS servers, PACS01 192.168.1.11 and PACS02 192.168.1.12 for redundancy and failover purposes. Lets say ALL modalities are currently sending DICOM images to PACS01's IP/AE Title and then there's a problem with PACS01 or we need to do an update. To prevent downtime, we have to come in and physically touch every modality device and swap it over to PACS02 192.168.1.12 which can take a LOT of time. Then we can do our repairs or upgrades PACS01, THEN we can do it all over again. Swap all the modalities back over to PACS01, make the repairs or upgrades to PACS02 No bueno.


    With Conquest... Conquest is set up as our DICOM router and sits in front of our PACS servers. All Modalities send their DICOM images to Conquest instead of the two PACS server. Then in the senario above where we need to failover to PACS02 to resolve an issue on PACS01, we just edit the configs to point Conquest to PACS02 instead of having to touch every modality device in our facility. This is MUCH simpler and faster.


    My setup includes (2) two Ubuntu 12.04 LTS linux servers running Conquest 1.4.17c for DICOM routing, keepalived for network failover, and IPVS (via keepalived) for round robin network load balancing.


    The configs are almost exactly the same on both Conquest servers with minor differences for Conquest01/conquest02.


    Our Network Setup...
    Conquest01 192.168.1.21
    Conquest02 192.168.1.22
    Conquest VIP 192.168.1.23 <--- This is where all modalities will be sending DICOM images
    PACS01 192.168.1.11
    PACS02 192.168.1.12


    Our /conquest/dicom.ini is the same on both Conquest01 and Conquest02 servers. The only thing I believe we edited beyond the defaults is the last line ImportConverter0.


    Code
    # This file contains configuration information for the DICOM server# Example Linux version using SqLite# Copy this file to dicom.ini to use it[sscscp]MicroPACS = sscscpEdition = Personal# Network configuration: server name and TCP/IP port#MyACRNema = CONQUESTSRV1TCPPort = 5678# Reference to other files: known dicom servers; database layout; sopsACRNemaMap = acrnema.mapkFactorFile = dicom.sqlSOPClassList = dgatesop.lst# Host for postgres or mysql only, name, username and password for databaseSQLHost = localhostSQLServer = ./data/dbase/conquest.db3Username = dontcarePassword = dontcarePostGres = 0MySQL = 0SQLite = 1UseEscapeStringConstants = 0DoubleBackSlashToDB = 0#IndexDBF = 1#PackDBF = 0#LongQueryDBF = 1000# Configure databaseTruncateFieldNames = 10MaxFieldLength = 254MaxFileNameLength = 255FixPhilips = 0FixKodak = 0UIDPrefix = 99999.99999EnableReadAheadThread = 1PatientQuerySortOrder =StudyQuerySortOrder =SeriesQuerySortOrder =ImageQuerySortOrder =EnableComputedFields = 1TCPIPTimeOut = 300FailHoldOff = 60RetryDelay = 100QueueSize = 128WorkListMode = 0WorkListReturnsISO_IR_100 = 1DebugLevel = 5Prefetcher = 0LRUSort =AllowTruncate =DecompressNon16BitsJpeg = 1UseBuiltInJPEG = 1IgnoreOutOfMemoryErrors = 0PadAEWithZeros = 0FileNameSyntax = 3# Configuration of compression for incoming images and archivalDroppedFileCompression = unIncomingCompression = unArchiveCompression = as# Names of the database tablesPatientTableName = DICOMPatientsStudyTableName = DICOMStudiesSeriesTableName = DICOMSeriesImageTableName = DICOMImagesDMarkTableName = DICOMAccessUpdatesRegisteredMOPDeviceTable = RegisteredMOPIDsUIDToMOPIDTable = UIDToMOPIDUIDToCDRIDTable = UIDToCDRID# Banner and host for debug informationPACSName = CONQUESTSRV1OperatorConsole = 127.0.0.1# Configuration of disk(s) to store imagesMAGDeviceThreshhold = 0MAGDevices = 1MAGDevice0 = ./data/ImportConverter0 = forward to PACS01 org %u; destroy; # This forwards DICOM images to PACS01 with the AE Title of the originating sending modality, then destroys Conquest's copy of the files#ImportConverter0 = forward to PACS02 org %u; destroy; # Uncomment this line to make PACS02 the destination for DICOM images


    Our /conquest/acrnema.map is the same on both Conquest01 and Conquest02 servers.


    Code
    CONQUESTSRV1 127.0.0.1 5678 asPACS01 192.168.1.11 12000 UNPACS02 192.168.1.12 12000 UN


    For Failover with Keepalived and roud robin loadbalancing with IPVS you'll need to first install Keepalived and then update this script on both servers, there's a small difference between Conquest01's version and Conquest02's version of this config file.


    (for Ubuntu)

    Code
    sudo apt-get updatesudo apt-get install keepalived


    Conquest01's /etc/keepalived/keepalived.conf


    Code
    # Virtual IP for Image Destinationvrrp_instance VI_1 { # initial state state MASTER interface eth0 # arbitary unique number 0..255 # used to differentiate multiple instances of vrrpd virtual_router_id 1 # for electing MASTER, highest priority wins. # to be MASTER, make 50 more than other machines. priority 100 virtual_ipaddress { 192.168.1.23/24 } # Invoked to master transition notify_master "/etc/keepalived/bypass_ipvs.sh del 192.168.1.23" # Invoked to backup(slave) transition notify_backup "/etc/keepalived/bypass_ipvs.sh add 192.168.1.23" # Invoked to fault transition notify_fault "/etc/keepalived/bypass_ipvs.sh add 192.168.1.23"}# Virtual IP & Physical Conquest Serversvirtual_server 192.168.1.23 5678 { delay_loop 10 lb_algo wlc # Round Robin lb_kind DR # Direct Routing# persistence_timeout 50 nat_mask 255.255.255.0 protocol TCP# Physical MASTER (Conquest01) real_server 192.168.1.21 5678 { weight 1 TCP_CHECK { connect_timeout 1 connect_port 5678 } }# Physical BACKUP (Conquest02) real_server 192.168.1.22 5678 { weight 1 TCP_CHECK { connect_timeout 1 connect_port 5678 } }}



    Conquest02's /etc/keepalived/keepalived.conf
    The only difference here are these two lines.
    Instead of state MASTER, change it to state BACKUP
    Instead of priority 100, change it to priority 50


    Next, and Finally, add this script to both Conquest servers and don't forget to make it executable.


    Code
    chmod 755 /etc/keepalived/bypass_ipvs.sh



    BOOM! DONE! FINISHED!

  • There has been a problem with our setup which we've been unable to determine problem.


    Symptom :: I migrated all of our CT, dental, remote outside studies, and MRI modalities to send inages to Conquest, It works flawlessly for about 3-4 days. THEN.... we start getting calls that random images within a series aren't showing up in PACS, as the problem progresses, nothing at all will send to PACS.


    Attempted fix :: First, I tried restarting the Conquest dgate, which did not help. Then I rebooted our Conquest servers and updated our /conquest/dicom.ini and /etc/keepalived/keepalived.conf config files, making the following changes.


    /conquest/dicom.ini for both of our Conquest servers (Conquest01 & Conquest02
    RetryDelay = 30 <--- modified this line
    RetryForwardFailed = 1 <--- added this line


    /etc/keepalived/keepalived.conf
    persistence_timeout 10 <--- uncommented this line and modified it to 10
    nat_mask 255.255.255.0 <--- removed this line since we're not using NAT


    After these changes, we began sending all images back to our Conquest VIP 192.168.1.23, once again everything worked flawless for about 3-4 days.


    Symptom persists :: after about 3-4 days, our modalities are unable to send anything to Conquest, we get timeouts, unable to dicom ping, unable to send, no explanation why.


    I will continue to update this post until I find a solution. Stay tuned.

  • Hi,


    if you restart conquest it should work again if the culprit is conquest. I therefore suspect the issue may be in the database server which stays up. Did you try to restart that. What database are you using? It may be useful to try another one (SQLite is not bad at all).


    Marcel

  • Hey Marcel, I'm using the default SQLite. Would this be the proper command to restart the database?


    Code
    dgate -v -r


    What do you suggest for restarting the database? I believe I tried restarting both of our conquest servers as well.... But I'll be sure to try that again if/when our modalities are unable to send to Conquest. Thanks for your help on this. Once we get this system running reliably, you have no idea how much time/manpower it will save us during failovers.


    --
    Tim

  • Hi,


    importconverters can be used without database, but not exportconverters. Also the delayed forward statements (e.g. forward series) cannot be used as they rely on the database.


    Direct 'forward to XX channel *' should work fine. Remember to set ForwardAssociationLevel to e.g. SERIES.


    Marcel

  • Hi,


    have you looked into memory consumption of the different processes. And you are using 1.4.17c? There have been fixes in the forwarder and the option "channel *" has been added to improve performance if many clients are sending at once.


    Marcel

  • Hi Marcel, we are using 1.4.17c.


    I've updated the Conquest config file with the following changes


    Edits for no database
    SQLHost =
    SQLServer =
    Username =
    Password =
    PostGres = 0
    MySQL = 0
    SQLite = 0


    Although I've confirmed our Conquest forwards on a series level by default, I've added this line per your suggestion
    ForwardAssociationLevel = SERIES


    And my last question is regarding the channel option. I'm not sure if I've got the proper number/position of simicolons ";"


    Is this the correct format for our forward statement with the channel option added for automatic channels?
    ImportConverter0 = forward to VMLTA01_ARCHIVE org %u channel *; destroy;


    Thanks for your help!

  • It's been a week and no problems. The only change I made was disabling keepalived and sending our DICOM images to just one of the Conquest servers. This morning, I implemented the changes discussed previously, regarding running with no database and using the channel option. After implementing these two changes and restarting the Conquest dgate service, I received this message in the log file


    Mon Feb 17 09:22:51 2014 *** Not enough rights to write in MAG0
    Mon Feb 17 09:22:51 2014 Started zip and cleanup thread
    Mon Feb 17 09:22:51 2014 DGATE (1.4.17c, build Fri Nov 22 15:38:03 2013, bits 64) is running as threaded server
    Mon Feb 17 09:22:51 2014 Database type: NULL driver (black hole)


    I'm not sure what the MAG0 permission error is referrencing... Any thoughts?


    If the system runs fine like this for another week, then I'll re-enable keepalived for load balancing for another week after that and see if we run into the previous problems where Conquest was failing to send after 3-4 days, in which case I may create a script to restart keepalived every 2 days. I'll keep you guys posted as to what will ultimately be our most stable setup for performance/reliability.

  • Hi, I keep getting this error in Conquest. Originally, I thought it was a problem caused by the interaction with Keepalived, however, I've shutdown Keepalived and am running Conquest on its own and still get the error.



    After we receive this error, often a technician will call us and let us know one series from their study didn't store to PACS. We don't usually have this problem when sending studies directly to PACS. I'm wondering if perhaps Conquest is making too many simultaneous connections to our PACS or if Conquest just doesn't retry enough times...


    I've got these two options enabled in Conquest regarding retries.


    Quote

    RetryDelay = 30
    RetryForwardFailed = 1


    (1) Is there anything else I can do to resolve this problem or have Conquest retry more often, etc?
    (2) Also, is there any way to get more information as to why Conquest failed to send or connect to PACS and what in particular it was trying to send in that moment?


    Thanks for your help!


    --Tim

  • I also noticed we're getting several association aborts on our PACS, from sends from Conquest


    One error we get on our PACS is


    Unrecognized PDU


    Our PACS PDU is set at 128k and below


    Grep'ing for PDU in serverstatus.log on Conquest shows Conquest is sometimes receiving from modalities at 256k, I don't know for sure, however I'm assuming if Conquest receives at 256k, it's also sending to our PACS at 256k. How can we know for sure if this is the case?


    Most importantly, our PACS archive PDU length is set to not exceed certain values, so is there any way we can constrain the PDU length on Conquest (receiving from modalities/sending to PACS)? Thanks again!


    --Tim

  • Hi,


    the PDU size in conquest is set to 32 k, I do not think this is the problem.


    Import converters do not store any data so unfortunately they cannot retry.


    Conquest will start as many associations to forward as it gets incoming. Is the PACS sometimes down for maintenance? Otherwise use ExportConverters which do retry but need a database.


    You can add e.g. nop %u %m %i %o; as statement to you converter. This will log calling AE, modality, patientID and SOPInstanceUID.


    Marcel

  • I could really use some assistance please. I'm getting tons and tons of these messages on both of my conquest servers.


    Code
    root@vlconquest01:/conquest/logs# less 20140304/PacsTrouble.log-20140304Mon Mar 3 10:54:20 2014 *** ImportConverter0.4: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:20 2014 *** ImportConverter0.4: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:20 2014 *** ImportConverter0.4: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:20 2014 *** ImportConverter0.4: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:20 2014 *** ImportConverter0.3: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:21 2014 *** ImportConverter0.5: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:21 2014 *** ImportConverter0.5: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:21 2014 *** ImportConverter0.5: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:21 2014 *** ImportConverter0.5: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:21 2014 *** ImportConverter0.5: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:28 2014 *** ImportConverter0.7: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:28 2014 *** ImportConverter0.7: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:28 2014 *** ImportConverter0.7: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:29 2014 *** ImportConverter0.8: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 10:54:29 2014 *** ImportConverter0.8: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 12:32:35 2014 *** ImportConverter0.6: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 12:38:43 2014 *** ImportConverter0.18: Forward failed to connect to host PACS_ARCHIVEMon Mar 3 12:56:00 2014 *** ImportConverter0.6: Forward failed to connect to host PACS_ARCHIVE


    I found that some of these ImportConverter0.X (where 'X' is a number) have more instances than others.


    Code
    root@vlconquest01:/conquest/logs/20140304# awk '{print $7}' PacsTrouble.log-20140304 | sort | uniq -c 1 connection 2 ImportConverter0.0: 48 ImportConverter0.1: 1 ImportConverter0.18: 192 ImportConverter0.2: 1 ImportConverter0.3: 96 ImportConverter0.4: 257 ImportConverter0.5: 701 ImportConverter0.6: 105 ImportConverter0.7: 23 ImportConverter0.8:



    Code
    root@vlconquest02:/conquest/logs/20140304# awk '{print $7}' PacsTrouble.log-20140304 | sort | uniq -c
    5 connection
    1 ImportConverter0.1:
    1 ImportConverter0.11:
    1 ImportConverter0.14:
    450 ImportConverter0.15:
    1 ImportConverter0.2:


    On conquest01, ImportConverter0.6 has the most hits (701) and
    on conquest02, ImportConverter0.15 has the most hits (450).


    I don't quite know what to think of this, but we've got hundreds of instances where we are unable to send images to our PACS via conquest.


    Do you have any idea what could be the problem and how to fix this? Thanks for your help.


    --Tim

  • Tim,


    did you set ForwardAssociationLevel to e.g., SERIES or STUDY?


    This will limit the number of associations made to one per series or study sent. It may be that your PACS does not like that many associations, by default conquest will create one for each image.


    Marcelம

  • Hi Marcell,


    I set ForwardAssociationLevel = SERIES


    If it's set to SERIES, would there be an association per each SERIES or IMAGE?


    Our current ImportConverter is


    ImportConverter0 = forward to PACS_ARCHIVE org %u channel *; destroy;


    To add more logging, would it look like this?


    ImportConverter0 = forward to PACS_ARCHIVE org %u channel *; destroy; nop %u %m %i %o;


    I noticed in the manual, it has these additional settings


    ImportConverter1, ImportConverter2, ImportConverters =1, etc...


    I guess I don't understand when/why you would set these additional attributes, I hope this isn't a stupid question. But what are these for? ImportConverter1, ImportConverter2, ImportConverters =1



    If we use ExportConverters, would we still need the ImportConverter0 option? or would ExportConver0 take its place?

  • Hi,


    It would be one assocation per SERIES.


    I would use:


    ImportConverter0 = forward to PACS_ARCHIVE org %u channel *; nop %u %m %i %o; destroy;


    It calls ImportConverter0 .. ImportConverter99. ImportConverters is the count, it is not needed.


    ExportConverter0 run after storage, ExportConverters is needed.


    Marcel

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!