Conquest 1.5.0b with MS SQL issue

  • Hi all,


    I recently changed my PACS Server and I decided to use a stronger and faster DB (I used integrated SQLite before) so I configured MS SQL Server (2019) on Windows Server 2016 and the latest Conquest PACS 1.5.0 b.

    I quickly faced some big issues.

    Basically, PACS is OK at first but process dgate64.exe start to use more CPU (not so much ,about 5% instead of 0.5% max usually) and more RAM. At some point, memory grows to reach hundreds of Mb or even till 4 Gb and Conquest becomes unusable or simply crash. It got worse and yesterday I had to restart Conquest service about 4 or 5 times.

    Looks like a memory leak but after a few days digging logs I couldn't find any explanations or serious leads.


    Since yesterday evening, I put an older version of dgate64.exe (1.4.19 d2) and it seems to be stable now, no more RAM eating or abnormal CPU use.



    Question 1: I would like to know if some of you are using latest 1.5.0b with MS SQL and if you have some issues too, or if you set specific settings.


    I saw there are big additions in 1.5.0, especially multi threaded moves and forwards, don't know if this is related.


    Question 2: Other issue but I guess that is not a new one (because before I set the new server, I had one instance per modality, and I decided to move all to a single instance).


    Queries give no results if I check more than 1 modality as a filter (eg. DX + MR : 0 results ; no filter 25 results).


    I tried the procedure described in the following thread, without success: no change it works for Horos and Radiant but is not functionnal for MicroDicom, I can't figure out why this is different.

    OSIRIX/HOROS problem with multiple Modalities search


    Any help or feedback will be appreciated, clinicians are going to hate me for this change, maybe it is too late :)


    Thanks

  • Hi,


    Add 1) No obvious changes that could explain the leak. Different compiler for 64 bits though, with potentially different ODBC libraries. Is the leak there qwith SQlie?


    Add 2) DX + MR checks for studies with DX + MR, not studies with DX and with MR. The fix just takes that query entry out. Can you log what it asks for?


    Marcel

  • Hi Marcel,


    Thanks for your answer.


    1) I didn't check with any other DB as during testing phase everything was OK with MS SQL. The leak occures in production environnement, maybe linked with storing of some modalities as the process seems to grow when receiving images and not free all the RAM after. Now the server is in production and I cannot make those tests without disturbing clinical activity.


    Today I have the opportunity to see a live crash, and the 1.4.19d2 dgate version crashed too. I think this is link to our new scanner which generate huge series. Clinicians sent a 3 Gb study on the PACS, and then tried to retrieve it on several clients simultaneously. dgate uses more CPU and RAM than usual and it grows till about 4 Gb of RAM. It becomes so slow that it is unusable and I had to restart it manually. I've never seen this issue before, when I used one instance per modality, the other difference with my previous server is the MS SQL.

    As the pacs becomes instable, pacstrouble.log gets filled with sql connections errors. This happens with 1.5.0b version (see log below), but NOT with 1.4.19. No errors in log but the server crashes though.


    I believe this is partly due to the use of MS SQL. I am disappointed as indexing is so fast comparing to SQLite I used before, and it seems that I have to go back to SQLite.

    With a new testing instance of 1.5.0b using SQLite, I put the 3Gb scanner exam and downloaded it on my PC in 45s, dgate process took only a few Mb of RAM.

    MS SQL is obviously involved in this issue and I don't know why. I am certainly not the only one to use MS SQL but I can't find any other similar issue in the forum.



    2)

    Looks like Microdicom searches at study level, while Radiant search at image level, that might explain why Microdicom cannot retrieves results for multiple modalities search.

    Reading the forum I understand that modality is defined at series or image level.

    Not sure if there's a way to force an image level search in Microdicom though.

  • Hi,


    1) The ODBC code has not changed in ages, and was stable years ago. It may be in DLLs povided by the system.


    2) You see how conquest tries to (partially) support multiple modality matching.

    (DICOMStudies.StudyModal = 'DX\MR' or DICOMStudies.StudyModal LIKE 'DX\MR\%' or DICOMStudies.StudyModal LIKE '%\DX\MR\%' or DICOMStudies.StudyModal LIKE '%\DX\MR')


    They still have to be in the same order though. Interesting that microdicom sends lower case modalities which is wrong for sure. Study modality is defined on the study level so that is not the problem.


    I guess the query should be closer like:


    (DICOMStudies.StudyModal LIKE '%DX%' or DICOMStudies.StudyModal LIKE '%MR%' )

    I am not likely to change that code soon though, it is quite complex.


    Marcel

  • Hi Marcel,


    Thanks, I guess I have to either change DB type or / and configure one instance for each modality as it used to be.


    That way it would solve both of these issues, or at least make it a lot easier to manage.

    Put the things all together was a false good idea.

  • Hi Marcel,

    It's a brand new server, with 16 GB RAM. Most of the time, it uses about 40%, considering that MS SQL takes 4-5 GB itself.

    The process does not always crash, but as it grows it becomes so slow that it is unusable and I am forced to restart dgate.

    Resources of the server are barely used.

    Besides, when it does operate correctly, I noticed yesterday that the process took only a few MB of memory when transfering big CT exams, like 40-50 MB memory used to retrieve a 1 GB study.

    For now, I moved CT exams on a new instance using SQLite (seems OK for now), and might do the same thing for MRI for stability purpose if necessary as it seems to be linked with big studies.


    The main change with this new server is the use of MS SQL. Coincidence?

  • Using 1.5.0c (b + few hotfixes by Marcel) I can confirm that dgate64.exe uses about 200Mb while idle and can EASILY use up to 4Gb of ram (or even more) while 'working' but I've >never< seen it crash. Also the memory usage goes down after a while so it doesn't look like a memory leak.

    Back when I had 1.5.0b it was running 24/7 literally for months with zero issues (under moderate load - 150ish MR/CT per day)


    The difference is I'm using PostgresSQL. Maybe it's a db specific issue?

  • Using 1.5.0c (b + few hotfixes by Marcel) I can confirm that dgate64.exe uses about 200Mb while idle and can EASILY use up to 4Gb of ram (or even more) while 'working' but I've >never< seen it crash. Also the memory usage goes down after a while so it doesn't look like a memory leak.

    Back when I had 1.5.0b it was running 24/7 literally for months with zero issues (under moderate load - 150ish MR/CT per day)


    The difference is I'm using PostgresSQL. Maybe it's a db specific issue?

    DB specific issue is exactly what I think as I never faced such issue before, but again, I used to have one instance for each modality and I decided to put all together on one instance so the duty is a lot heavier.

    Till last week, CT exams are one a specific instance using SQLite DB. No issue at this moment but activity is a bit lighter than usual I guess.


    I will monitor and let you know.


    What db did you use before?


    Marcel


    I had 7 instances using SQLite on my previous server. On my new server, 1 instance using MS SQL at first (but it crashed a lot) and at this moment I still have MS SQL instance for all but the CT, and one instance for CT using SQlite.

  • Hi,


    Most people run with MySQL and postgres. I most likely use outdated ODBC connector libraries (I tend to rarely change compiler distributions), although that should not cause crashes. Is there information about the crashes?


    Marcel

  • Hi,


    I had some issues again last month, something like a memory leak, and dgate process that becomes so slow so I have to restart it.

    I found some "serious" lead as I was making some testing with an Orthanc server : empty queries make dgate process grows and even after a dicom client reports "no results" the process continues to grow for a while.

    If several clients do the same thing (especially with this awful settings "auto refresh" results in Horos / Osririx search) it could lead to potential instability.

    I have many clients, and a lot of them used by students who can mess with results "auto refresh" or empty queries.

    The issue does not occur making an empty query in the internal search module of Conquest GUI (memory usage does not vary in this case).


    So, I added this line in the lua section of dicom.ini :

    Code
    QueryConverter0 = if (Data.PatientID=="") and (Data.StudyDate=="") and (Data.PatientName=="") then script('destroy') end

    For now (after 2 weeks), I didn't have to restart dgate process and I feel that it is more stable and I faced no slow-down since this moment.


    I hope this will solve the issue for good.

  • I've had no issue on sqlite but database is way smaller as there are only CT.

    An empty query does not seem to use additional memory.


    I just tested a "big" query on MS SQL pacs (10 years), and Microdicom took about 1mn to show 0 results (instead of thousands).

    The issue is still there because since this moment, idle dgate process uses 0.5% CPU (usually, idle is between 0 to 0.1%), MS SQL process continue to use more CPU than usual, and also lsass.exe which always consume CPU each time the problem occures.


    In fact I don't understand why I get 0 results, neither why resources continue to be used after the query is apparently over.

    I am pretty sure the issue is related to big queries, but only MS SQL DB seems to be affected.

    Maybe my SQL Server needs some tuning, but I've already tried several tweaks with no results.


    Is there a way to force stop queries, let's say if results are > 100 ie?


    As I write I sent a new big query : dgate64 is now 0.9% CPU, lsass 2.6% and SQL 3.5% which is almost twice bigger, that looks like the previous query is stuck in background...

    Not sure I am clear, sorry.

  • Hi,


    can you try this:


    C:\dicom150beta>dgate64 --display_status:

    DICOM server 'CONQUESTSRV2' (version 1.5.0c, port 5678, bits 64) was started on

    Wed Apr 14 11:01:52 2021

    Old JPEG decoder=0, JPEGLIB jpeg codec=1, LIBJASPER jpeg2000 codec=1

    Run time (s) total 18, query 0, load 0, save 0, compress 0, process 0, gpps 0

    Associations=36; Threads=78 (1 open); Images sent=0, recieved=0, forwarded=0

    Images printed=0, in color=0

    Activity: Echo:0, Find:36, Move:0, Unknown:0, gpps:2402

    Images (de)compressed: NKI 0, JPEG 0, JPEG2000 0, RLE 0, Planes removed 0, Palet

    tes removed 0, Downsize 0

    Space on MAG0 : 12197 MByte

    Space on MAG1 : 12197 MByte

    Database type: ODBC connection


    How many open threads does yours show?


    Marcel

  • Marcel,


    I've run the diag 3 times : before query, during (big) query, after query showed 0 results in client.

    A lot more threads than yours, don't know if that matters.


    Code: Before Query
    C:\Windows\system32>C:\PACS\dgate64 --display_status:
    DICOM server 'CONQUEST' (version 1.5.0-alpha-test1, port 104, bits 64) was started on Mon Apr 12 09:10:39 2021
    Old JPEG decoder=0, JPEGLIB jpeg codec=1, LIBJASPER jpeg2000 codec=1
    Run time (s) total 15022, query 0, load 51, save 8, compress 6, process 0, gpps 9
    Associations=29462; Threads=28169 (1 open); Images sent=11910, recieved=2663, forwarded=0
    Images printed=0, in color=0
    Activity: Echo:4, Find:26410, Move:381, Unknown:0, gpps:5396794
    Images (de)compressed: NKI 0, JPEG 115, JPEG2000 0, RLE 0, Planes removed 0, Palettes removed 0, Downsize 0
    Space on MAG0 : 6818394 MByte
    Database type: ODBC connection
    Code: During query
    C:\Windows\system32>C:\PACS\dgate64 --display_status:
    DICOM server 'CONQUEST' (version 1.5.0-alpha-test1, port 104, bits 64) was started on Mon Apr 12 09:10:39 2021
    Old JPEG decoder=0, JPEGLIB jpeg codec=1, LIBJASPER jpeg2000 codec=1
    Run time (s) total 15027, query 0, load 51, save 8, compress 6, process 0, gpps 10
    Associations=29503; Threads=28211 (2 open); Images sent=11910, recieved=2672, forwarded=0
    Images printed=0, in color=0
    Activity: Echo:5, Find:26440, Move:381, Unknown:0, gpps:5409117
    Images (de)compressed: NKI 0, JPEG 115, JPEG2000 0, RLE 0, Planes removed 0, Palettes removed 0, Downsize 0
    Space on MAG0 : 6818376 MByte
    Database type: ODBC connection
    Code: 2mn after 0 results search
    C:\Windows\system32>C:\PACS\dgate64 --display_status:
    DICOM server 'CONQUEST' (version 1.5.0-alpha-test1, port 104, bits 64) was started on Mon Apr 12 09:10:39 2021
    Old JPEG decoder=0, JPEGLIB jpeg codec=1, LIBJASPER jpeg2000 codec=1
    Run time (s) total 15038, query 0, load 51, save 8, compress 6, process 0, gpps 10
    Associations=29578; Threads=28291 (2 open); Images sent=11910, recieved=2686, forwarded=0
    Images printed=0, in color=0
    Activity: Echo:5, Find:26501, Move:381, Unknown:0, gpps:5472920
    Images (de)compressed: NKI 0, JPEG 115, JPEG2000 0, RLE 0, Planes removed 0, Palettes removed 0, Downsize 0
    Space on MAG0 : 6818355 MByte
    Database type: ODBC connection

    You will notice that version is 1.5.0 alpha, I tried this version after similar issue on 1.5.0b, I guess I forgot to revert to a "stable" version.

    I just updated and am back to 1.5.0b now.


    Code: After update to 1.5.0b
    C:\Windows\system32>C:\PACS\dgate64 --display_status:
    DICOM server 'CONQUEST' (version 1.5.0b, port 104, bits 64) was started on Wed Apr 14 12:56:43 2021
    Old JPEG decoder=0, JPEGLIB jpeg codec=1, LIBJASPER jpeg2000 codec=1
    Run time (s) total 29, query 0, load 0, save 0, compress 0, process 0, gpps 0
    Associations=375; Threads=393 (1 open); Images sent=3, recieved=9, forwarded=0
    Images printed=0, in color=0
    Activity: Echo:0, Find:364, Move:2, Unknown:0, gpps:44400
    Images (de)compressed: NKI 0, JPEG 0, JPEG2000 0, RLE 0, Planes removed 0, Palettes removed 0, Downsize 0
    Space on MAG0 : 6818232 MByte
    Database type: ODBC connection

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!