Organise pacs's instances by modality and move data between

  • Dear Marcel,

    i would be grateful for Your advice/help to resolve my (i hope simple) strategy plan make happen ;) I'll try to put as simple status as i can. Right now i'm starting up new physical server (storage over 30TB compared to old 8TB). I've already set up 6 conquest instances to have them eventually connected with different modality machines (earlier it was one conquest per all) and 1 conquest gateway for client's workstations. Everything is based on Oracle Linux with mysql (as i was writing with You earlier on forum). Old server has to be up and running until i have everything online on new one, bad info is i have so little time (just 95GB space left).

    I've already copied whole data (over 7.5TB) of xr,dx,ct,mr exams from old server to new storage. I'm using temporary dgate instance where i've reinitialised db from these 7.5TB exams so i could now divide that chaotic magazine info earlier prepared, smaller conquest instances (by machine/modality). I don't know yet how to use lua (didn't have to), maybe it would make me happy in my case :) but becase of notime@work i've found in documentation that i can make temporary router instance where in dicom.ini, apart of rest of default parameters i can configure null db driver and sth i tried like below:


    ForwardAssociationLevel = SERIES (btw would be STUDY faster?)

    ExportConverters = 3

    ImportConverter0 = ifequal “%m”, “CT”; { forward to DCP2 channel *; destroy; }

    ImportConverter1 = ifequal “%m”, “XR”; { forward to DCP3 channel *; destroy; }

    ImportConverter2 = ifequal “%m”, “DX”; { forward to DCP3 channel *; destroy; }


    When i run from temp dgate instance (where i have 7.5TB db reinitialised) command: dgate --movestudies:local,router,20100101-20200618 for few minutes system is busy but then shell is free again to write, only in top i can see that mysql is still using cpu/mem but no exam transfers occures to DCP2/DCP3 instances. Do i miss something? Or is there any simplier solution i didn't get from documentation?


    ps1.
    When i make the movestudies command, but directly to last target conquest instance, everything is ok but i transfer all modality exams of course, so compiled instances are working fine in my opionion.


    ps2.

    By the way, 7.5TB is nearly 14mln images (NKI compressed), the mysql db is about 13GB. During moving to target smaller conquest instances, i'd like to start using jpeg2000 (i have new intel scalable 2nd gen).

    There is strange thing to me: on both arcnema i have n4 transfer, in target dicom.ini i have file syntax 4 and jk in dropped and incoming compression, but jk/dcm file on target is sometimes twice bigger than n4/v2 file on source, is sth wrong with my compilation/configuration or just NKI is today much better, source is ~512KB, target is 1.2MB and is seems to me as uncompressed dcm despite jk defined.

    OK, now i see i'm using first 1.5v and there was update fix for nki/jpeg compression on github, going to recompile.


    ps.3
    is compression nki or jpeg2000 multithreading or there is somekind of thread ahead during moving the studies between dgate instances?


    ps.4

    for my curiosity, did You Marcel saw someone using mysql table's compressed solution to handle conquest db?


    Much edited but i'm during tests that is why :) I appreciate Your kind advices.

  • Hi,


    your forward rules look good, apart from the used quotes which should be ". DCP2 and DCP3 are in acrnema.map? You do not have to set ExportConverters = 3. You can run the server as ./dgate -v to see the output that helps testing.


    ps.1, 2: please update indeed; compression was disabled by accident. I put the socket send fix for data corruption in GitHub as well last night. Have not seen it fail or help, so cannot hurt even though untested for the actual problem.


    ps. 3: nki compression is very simple and blindingly fast but not standard. jpeg2000 is very slow. I personally like jepgls better. Neither are multithreading apart when receiving/sending multiple instances at once but they do run in parallel with network transfer.


    ps. 4: no, have not seen that. Recently I had relatively poor performance with mysql; may be due to new default settings


    Good luck, Marcel

  • Just run a quick test on my raspberri pi. This works:


    nano dicom.ini


    Code
    MAGDevice0 = /home/pi/Desktop/conquest150a/Conquest-DICOM-Server/data/
    ;converters in connquest scripting language must be before [lua] section
    ImportConverter0 = ifequal "%m","CT"; {forward to 192.168.1.71:5679; destroy;}
    [lua]
    association = package.path=package.path..';'..Global.basedir..'lua/?.lua'


    sudo systemctl stop conquest

    ./dgate -v



    Move Originator AE := CONQUESTSRV2

    Importconverter0.0 executes: ifequal "CT","CT"

    ImportConverter0.1: forwarded object to 192.168.1.71:5679

    Importconverter0.2: destroyed received image

  • Hello Marcel, 2 days off with family but again fighting :) Yes, I have everything in acrnema. OK I've recompiled and it works but still with some problem i can't get by.

    First of all thank You very much for hint on jpeg-ls, almost three times faster than j2k with nearly same ratio i see on tests, but i think i will use n4 right now because of no much time left for more tests.


    Edited one more time to meet my actual configuration (names)


    The major problem right now is:

    Eventually i'd like to make command "dgate --movestudies:OCTOPUS,ROUTER,20120828-20200612" so router will move exams from OCTOPUS (temporary instance with old storage) to DCP1/DCP2/DCP3 according to modality rules.

    Above command ends with error (PacsTrouble) on OCTOPUS like:

    Sun Jun 21 16:50:25 2020 ***preretrieve/forward xxx to: association lost

    and on shell exit i see: 1 Move failed from OCTOPUS to ROUTER

    Command ends (going back to shall) but mysql is utilizing cpu/memory (i presume because of processing 13GB db) so i have to manually restart db server.

    When i try to narrow date range to 1 day for testing, it's working fine. I don't know what may be the cause of such behaviour.


    ps.

    Heh i've edited this post few times because of more problems earlier which i didn't get in first place You mensioned me about quotes " heheheh, i've deleted it so only one above i'd like to resolve :)

  • whatever I try with movestudies i get sth like below, hmm


    ./dgate -v --movestudies:OCTOPUS,ROUTER,20120101-20200630


    │(QualifyOn) (mapped) IP:172.16.1.111, PORT:1117

    │MyStudyRootRetrieveGeneric :: SearchOn

    │Query On Image

    │Issue Query on Columns: DICOMImages.SOPClassUI, DICOMImages.SOPInstanc, DICOMSeries.SeriesInst, DICOMStudies.StudyDate, DICOMS

    │tudies.StudyInsta,DICOMImages.ObjectFile,DICOMImages.DeviceName

    │Values: DICOMStudies.StudyDate >= '20120101' and DICOMStudies.StudyDate <= '20200630' and DICOMSeries.StudyInsta = DICOMStudie

    │s.StudyInsta and DICOMImages.SeriesInst = DICOMSeries.SeriesInst

    │Tables: DICOMImages, DICOMSeries, DICOMStudies

    │Protocol error 0 in PDU:Read

    │***preretrieve/forward xxx to: association lost

    │Arena 0:

    │system bytes = 1196032

    │in use bytes = 1044176

    │Arena 1:

    │system bytes = 1368064

    │in use bytes = 1366768

    │Arena 2:

    │system bytes = 450560

    │in use bytes = 421120

    │Total (incl. mmap):

    │system bytes = 3149824

    │in use bytes = 2967232

    │max mmap regions = 2

    │max mmap bytes = 446464

  • I thought it's working fine but unfortunately it's still bad. With monthly spliting i got about 240k images transfered and it stopped. I tried in different split ways, even split batch movestudies into weekly loop and deleted chanel * from routing rule with changing assoclevet to study, everything stops in different time, generally stops on source (sending) side with errors like below:


    Mon Jun 22 12:16:09 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:28:39 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:31:27 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:35:08 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:40:13 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:41:26 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:50:08 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:50:19 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 12:57:32 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 13:15:38 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 13:37:22 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 13:43:01 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 14:13:50 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 14:37:38 2020 ***preretrieve/forward xxx to: remote DICOM error

    Mon Jun 22 15:10:58 2020 ***Failed to Listen () - bind error

    Mon Jun 22 15:11:08 2020 ***Failed to Listen () - bind error

    Mon Jun 22 15:11:18 2020 ***Failed to Listen () - bind error

    Mon Jun 22 15:11:28 2020 ***Failed to Listen () - bind error

    Mon Jun 22 15:11:39 2020 ***Failed to Listen () - bind error


    Then on router log i can see:


    Mon Jun 22 12:50:19 2020 *** Out of disk space

    Mon Jun 22 12:57:32 2020 ***Error getting free store for device NULL

    Mon Jun 22 12:57:32 2020 *** Out of disk space

    Mon Jun 22 13:15:38 2020 ***Error getting free store for device NULL

    Mon Jun 22 13:15:38 2020 *** Out of disk space

    Mon Jun 22 13:37:22 2020 ***Error getting free store for device NULL

    Mon Jun 22 13:37:22 2020 *** Out of disk space

    Mon Jun 22 13:43:01 2020 ***Error getting free store for device NULL

    Mon Jun 22 13:43:01 2020 *** Out of disk space

    Mon Jun 22 14:13:50 2020 ***Error getting free store for device NULL

    Mon Jun 22 14:13:50 2020 *** Out of disk space

    Mon Jun 22 14:37:38 2020 ***Error getting free store for device NULL

    Mon Jun 22 14:37:38 2020 *** Out of disk space

    Mon Jun 22 15:10:46 2020 *** connection terminated

    Mon Jun 22 15:15:45 2020 *** ImportConverter1.1: Forward failed to send DICOM image to DCP2


    Finally on target instance log we get:


    Mon Jun 22 15:10:47 2020

    ***Client Error: command ffff failed **

    Mon Jun 22 15:10:47 2020 ***Connection Terminated


    Right now i'm out of new ideas. I've noticed You changed 1.5.0 to (a) and made some updates on github. Could You ogive me a clue please.

  • Dear Marcel,


    i have some more strange behaviour. Please have a look below, the log of sending host (OCTOPUS) to the virtual router, after command dgate --movestudies from 1 day only:
    /dgate -v --movestudies:OCTOPUS,ROUTER,20171130-20171130


    Sending file : /PACS_SYSTEM/OCTOPUS/data/DICOM_0/87111108436/1.2.840.113704.7.1.1.1236.1512058566.155_80136_000002_1512058920165f.v2

    Image Loaded from Read Ahead Thread, returning TRUE

    ReadAheadThread: readahead > 5854

    MyStudyRootRetrieveGeneric :: RetrieveOn

    RetrieveOn: givenout < 5847

    Sending file : /PACS_SYSTEM/OCTOPUS/data/DICOM_0/57061212794/1.3.46.670589.11.0.0.11.4.2.0.10530.5.5108.2017113010415431007_0000_000001_15120372840495.v2

    Image Loaded from Read Ahead Thread, returning TRUE

    ReadAheadThread: readahead > 5855

    Retrieve: remote connection dropped after 5847 images, 542 not sent

    ***preretrieve/forward xxx to: remote DICOM error

    C-Move (StudyRoot)

    UPACS THREAD 169: ENDED AT: Mon Jul 6 15:04:08 2020

    UPACS THREAD 169: TOTAL RUNNING TIME: 417 SECONDS


    The connection dropps regardless of association lever studies or image set in router. In Router dicom.ini i have only this:

    [sscscp]

    MicroPACS = sscscp

    # Network configuration: server name and TCP/IP port#

    MyACRNema = ROUTER

    TCPPort = 1117

    # Host for postgres or mysql only, name, username and password for database

    # SQLHost = localhost

    # SQLServer =

    # Username =

    # Password =

    # PostGres = 0

    # MySQL = 1

    # SQLite = 0

    # UseEscapeStringConstants = 0

    # DoubleBackSlashToDB = 1

    # Configure server

    ImportExportDragAndDrop = 1

    ZipTime = 05:

    UIDPrefix = 99999.99999

    EnableComputedFields = 1

    FileNameSyntax = 4

    # Configuration of compression for incoming images and archival

    DroppedFileCompression = un

    IncomingCompression = un

    ArchiveCompression = as

    # For debug information

    PACSName = ROUTER

    OperatorConsole = 127.0.0.1

    DebugLevel = 0

    # Configuration of disk(s) to store images

    # MAGDeviceFullThreshold = 1000

    # MAGDevices = 1

    # MAGDevice0 = /PACS_SYSTEM/ROUTER/data/

    #-----------------------------------------------------------

    # MOJE TESTY ROUTINGU DO ROZDZIALU NA MODALNOSCI

    ForwardAssociationLevel = SERIES

    ImportConverter0 = ifequal "%m", "MRI"; { forward to DCP1; destroy; }

    ImportConverter1 = ifequal "%m", "CT"; { forward to DCP2; destroy; }

    ImportConverter2 = ifequal "%m", "XR"; { forward to DCP3; destroy; }

    ImportConverter3 = ifequal "%m", "DX"; { forward to DCP3; destroy; }

    #-----------------------------------------------------------

    [lua]

    association = package.path=package.path..';'..Global.basedir..'lua/?.lua'


    Maybe I should suspect sth with mysql database behaviour because i noticed few times that in phpmyadmin i got lost connection during query process while no operations occured.

  • Ok, i got by with phpmyadmin connection lost so i know mysqld works fine. So there is problem with:
    Retrieve: remote connection dropped after xxx images, xxx not sent


    Any clue what to check? :)

  • Hi,


    It could be that mysql runs out of ports.I have seen this on windows and fixed it by allowing more ports and shorten the reconnect timout.


    Registry.WriteInteger('MaxUserPort', 65534);

    Registry.WriteInteger('TcpTimedWaitDelay', 30);


    Marcel

  • I'm using linux mysql 8 version and i think the only parameters i could change are like on official site and there is no such option like in windows maybe.
    https://dev.mysql.com/doc/refm…n-variable-reference.html


    Hmm, when i start command with different day, the logs are like below, example 1:

    | Locating file:MAG0 77072016952/1.3.46.670589.26.702420.2.20171211.

    │100831.285762_0001_000001_1512982401000c.v2

    │Locating file:MAG0 48111104341/1.3.46.670589.26.702420.2.20171211.

    │95851.285758_0001_000001_15129817990008.v2

    │Locating file:MAG0 48111104341/1.3.46.670589.26.702420.2.20171211.

    │95945.285759_0002_000001_15129818700009.v2

    │ReadAheadThread: readahead > 0000

    │RetrieveOn: givenout < 0000

    │Sending file : /PACS_SYSTEM/OCTOPUS/data/DICOM_0/31070703720/1.2.3

    │92.200036.9125.3.144177281241575.64861121429.236201_1001_001001_15

    │129865460553.v2

    │Image Loaded from Read Ahead Thread, returning TRUE

    │ReadAheadThread: readahead > 0001

    │ReadAheadThread: readahead > 0002

    │ReadAheadThread: readahead > 0003

    │ReadAheadThread: readahead > 0004

    │ReadAheadThread: readahead > 0005

    │ReadAheadThread: readahead > 0006

    │[recompress]: recompressed with mode = un (strip=1)

    │ReadAheadThread: readahead > 0007

    │ReadAheadThread: readahead > 0008

    │Retrieve: remote connection dropped after 0 images, 1392 not sent

    │***preretrieve/forward xxx to: remote DICOM error

    │C-Move (StudyRoot)

    │UPACS THREAD 21: ENDED AT: Mon Jul 6 18:53:59 2020

    │UPACS THREAD 21: TOTAL RUNNING TIME: 5 SECONDS


    the ROUTER instance gives only in Trouble:

    Mon Jul 6 18:54:11 2020 ***Error getting free store for device NULL

    Mon Jul 6 18:54:11 2020 *** Out of disk space


    But in the User.log ROUTER says:

    Mon Jul 6 18:53:51 2020 "C-Store","OCTOPUS "

    Mon Jul 6 18:53:59 2020 "C-Store","OCTOPUS "

    Mon Jul 6 18:54:11 2020 "C-Store","OCTOPUS "


    sometimes it ends like this:
    Retrieve: remote connection dropped after 8508 images, 1529 not sent


    mysqld.log are fine (no errors), is it possible in host i want to send from (OCTOPUS) i have strangely written data?.

  • Hi,


    The one with zero images is interesting. I assume it is reproducible?


    For testing, can you temporarily make the router do a 'destroy' only. This allows checking if the router hangs up or the downstream client.


    Marcel

  • Sorry for delay,

    After destroy only configuration, the situation is nearly similar. Take a look:


    1. Sending OCTOPUS instance shows in PacsServerstatus:

    ...
    MyStudyRootRetrieveGeneric :: RetrieveOn

    RetrieveOn: givenout < 8508

    Sending file : /PACS_SYSTEM/OCTOPUS/data/DICOM_0/62080704245/1.3.46.670589.11.0.0.11.4.2.0.10530.5.3732.2017112911311171000_0000_000000_15119518577b75.v2

    Image Loaded from Read Ahead Thread, returning TRUE

    ReadAheadThread: readahead > 8516

    Retrieve: remote connection dropped after 8508 images, 1529 not sent

    ***preretrieve/forward xxx to: remote DICOM error

    C-Move (StudyRoot)

    UPACS THREAD 3: ENDED AT: Sun Jul 12 12:34:31 2020

    UPACS THREAD 3: TOTAL RUNNING TIME: 367 SECONDS


    2. And the Router (destroy) instance shows in PacsTrouble:

    Sun Jul 12 12:34:31 2020 ***Error getting free store for device NULL

    Sun Jul 12 12:34:31 2020 *** Out of disk space


    Strange, after so many images retrieved with DB NULL DRIVER. BTW the situation is the same for that day i'm testin (exacly after 8508 images) when i put in ROUTER dicom.ini MagDvice enabled or not.

  • Dear Marcel, just noticed my fault entry in router's dicom.ini

    Firstly i had commented MagDivice lines and partially transfers were working, eventually ending with the 'no space left' errors. Even with enabling magdevice lines there were same errors. I realized there were space chars before configuration entry in each lines of magdev. Without spaces it's working...

  • Ok,


    If I understand correcly Conquest hang up because it reported an out of disk error! Not sure why that would happen only after thousands of images. Can you share your faulty dicom.ini lines?


    Marcel

  • Uh, while ago i've checked and surpisingly it failed again with transfers. Now, it stopped because of no free storage left where the ROUTER instance is installed. Router data dir had lot of exams despite i have destroy formula after forwarding. I checked that there are much different modality names in studies of these exams. I suppose right now i understand all of this strange situation. Pitty it was so long story :). Unfortunately during few years, old Conquest saved studies data with different 'modality name' (because of different DI machines). When we had tests of exact day to movestudies, and i had hidden space before MAGDevice entry in dicom.ini line (attached below), there was an error of no space left when ROUTER finally get from OCTOPUS exam with unknown modality to compare in rule.


    Now i've written more converter rules, like below and hope that it will be enough.

    ImportConverter0 = ifequal "%m", "MRI"; { forward to DCP1; destroy; }

    ImportConverter1 = ifequal "%m", "MR"; { forward to DCP1; destroy; }

    ImportConverter2 = ifequal "%m", "MR/PR"; { forward to DCP1; destroy; }

    ImportConverter3 = ifequal "%m", "PR/MR"; { forward to DCP1; destroy; }

    ImportConverter4 = ifequal "%m", "PR"; { forward to DCP1; destroy; }

    ImportConverter5 = ifequal "%m", "CT"; { forward to DCP2; destroy; }

    ImportConverter6 = ifequal "%m", "SC/CT"; { forward to DCP2; destroy; }

    ImportConverter7 = ifequal "%m", "SC"; { forward to DCP2; destroy; }

    ImportConverter8 = ifequal "%m", "TK"; { forward to DCP2; destroy; }

    ImportConverter9 = ifequal "%m", "XR"; { forward to DCP3; destroy; }

    ImportConverter10 = ifequal "%m", "CR"; { forward to DCP3; destroy; }

    ImportConverter11 = ifequal "%m", "DX"; { forward to DCP3; destroy; }

    ImportConverter12 = ifequal "%m", "MG"; { forward to DCP3; destroy; }

    # We have no mammo but some of xray are saved with such modality name (strange)


    Maybe it could be good to add some information in general documentation that ROUTER functionality saves exams (despite destroy rule) to it's MAGDevice when the incoming data with e.g. modality name or any other compared variable is missed.


    I think i should learn LUA now.

  • Hi,


    %m is the Modality (0008,0060) that should be single valued. It 0008,0061 is ModalitiesInStudy that is multiple valued. Since the router ImportConverters runs serially you can end with some logging and a destroy e.g.


    ImportConverter13 = nop Unrecognised Modality %m; destroy;


    The space in MagDevice is stripped, I think that cannot be the issue.


    regards


    Marcel

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!