Getting anonymized dicoms without changing original

    Hi all!


    I'm building the data extraction pipeline from medical institution. Worth noting that they use conquest server 1.4.15 in there, which I can't change. The data is stored in .v2 format and must be anonymized. Sadly, my experiments show that dgate64.exe "--anonymize" does the thing kind of in-place, deleting the original file and storing the new one with the new patid on the same server, but I mustn't affect their original data in any way. Amounts of the data are pretty large, so I need something automated and fast.

    So I can see three paths:

    1) Setting up yet another Conquest server for anonymization purposes. Then, I will query their original server, grab necessary v2 files, copy them and pass it to anonymization server. Bad thing is I'm afraid that another server will interfere with original one, and I really appreciate if you guys guide me on how to avoid that.

    2) Convert v2 files to dcm using the original server and then anonymize it using the downstream third-party software which I kinda can craft. Bad thing is I didnt get dgate64.exe '--convert_to_dicom:file,size,comp,f" command work. I don't know what 'comp' and 'f' stand for, and can't find any documentation since 1.4.15 version is pretty old.

    3) Convert v2 files to images using dgate64.exe '--convert_to_jpg'. It does the thing but the output has low quality and I lose all the dicom metadata which is not necessary for my purposes but I prefer to have it.


    Could you please guide me on what to choose and how to solve my problem?


    Best regards,

    Niko

    Hi Niko,


    1) You can safely copy data to another server. In this case I would suggest to use Lua scripting (dicommove) in the new server to control the transfer.

    or you can batch dgate --movepatient on the old server or the new server. You can anonymise on reception.


    2) comp is compression, f is frame that does need to be passed. But this works file by file and sends output to the console. Not my choice.


    3) Not good at all.


    Question, when you anonymise, are there any rules for the new Patient ID? This would complicate matters a bit.


    Marcel

    Thanks for the reply, Marcel!


    So I guess I'll take the first approach. I was just worried that setting another server on the same machine with the same defaults might somehow impair working of the old server. Could you please tell me how to do that dicommove thing?


    No, I don't have any rules on the new Patient ID, it can be whatever.

    Hi,


    I would install a new server with its own datastorage and sqlite as local database - i.e. all defaults. The only thing that interferes is the server port, which should be set to any other value than in the older server (e.g. 105 or 5679). And have enough disk space, you are going to double the size of each v2 file if NKI compressed, so you need 2x the existing storage extra.


    Then add to dicom.ini on the NEW server:


    ImportConverter0 = anonymize_script.cq


    And add e.g. to acrnema.map on the OLD server:


    NEWSERVER 127.0.0.1 5679 un


    and reload the list. Similarly in the NEW server:


    NEWSERVER 127.0.0.1 5679 un

    OLDSERVER 127.0.0.1 5678 un


    Then you can e.g. batch on the NEW server


    dgate64 --movestudies:OLDSERVER,NEWSERVER,20000101-20000131


    to copy all studies from January 2000, etc.


    Marcel

    Hey Marcel!


    Thanks to your guidance, I almost finished the task.

    I set up 1.5.0 Conquest setver for anonymization purposes since 1.4.15 distribution kit institution didn't contain anonymize_script.cq file. Everything works just fine, but some files are not getting transferred with --movestudies approach. I have found this in the PacsTrouble.log:


    15:42:06 *** lua run error [string "local first=true; local ae=[[ANON2]];local ..."]:1: attempt to call a nil value in 'local first=true; local ae=[[ANON2]];local level=[[STUDY]];local q={QueryRetrieveLevel=[[STUDY]],PatientID=[[]],PatientName=[[]],StudyDate=[[]],};local q2=DicomObject:new(); for k,v in pairs(q) do q2[k]=v end;local r2=dicomquery(ae, level, q2);local s=tempfile("txt") f=io.open(s, "wb");if r2==nil then f:write("no connection with "..ae.."\n") returnfile=s f:close() return end; local r = loadstring("return "..r2:Serialize())();r[1].QueryRetrieveLevel=nil; r[1].TransferSyntaxUID=nil; local keys={} for k,v in pairs(r[1]) do keys[#keys+1]=k end; table.sort(keys, function(a, b) return string.sub(a, 1, 7)<string.sub(b, 1, 7) end); if first then for k,v in ipairs(keys) do f:write(v.." ") end f:write("\n") end if first then f:write("---------------------------------------------------------------------------------------------------------------------------------------------------------------\n") end for k,v in ipairs(r) do for k2,v2 in ipairs(keys) do f:write("["..v[v2].."] ") end f:write("\n") end returnfile=s f:close();'


    With incoming folder and drag-and-drop I don't lose any files. What could go wrong?

    And if I want to move the images of particular body part and modality rather than date, what's the best approach? I'm thinking of small script which queries the storage server and calls --movestudy on the result seriesUIDs. But is there any batched approach? Maybe, there is a way to integrate the query with --movestudies command?


    Best regards,

    Niko

    Hi,


    the lua script should be in 1.5.0 not 1.4.15; that will not be compatible. You anonymise on reception.


    For more control download and install zerobrane studio for lua scripting, perform what is in zerobranestudio/install.lua and then look at samples in the lua folder.


    Marcel

    Hi! Sorry for abscence, I'm back at it again. For some weird reasons, the very same files are transmitted on another Windows machine with no problems, sorry for bothering you. But I'll keep my eyes on that.


    The guys from institution are saying that they're really short on the disk space to maintain yet another server. But anyways the data needs to be transferred through the web to our office. Is there a way to script scp file transfer just after they was added (and anonymized)?


    Best regards,


    Niko

    Hi!

    With the following command

    dgate64.exe "--submit:837269774.1490884100,,,,{remote location},{password}"


    I'm getting this kind of errors


    [ANONSRV] *** Importconverter-1.0 script not found: submit.cq

    [ANONSRV] *** Failed uploading archive to submit


    I didn't seem to find this file anywhere in the conquest installation folder. Do I need to create it? Do I need to add something about ExportConverters in the dicom.ini? I'm going to make this sumbitting thing a converter but at the moment I tried just command line version.


    Best regards,

    Niko

    Hi Niko,


    you have to put putty's pscp.exe in the conquest folder and run it once to cache the keys for the remote system.


    dgate64.exe "--submit:837269774.1490884100,,,,{user@remote.location},{password},anonymize_script.cq"


    or put curl.exe there to submit to a web page, e.g.:


    dgate64.exe '--submit2:837269774.1490884100,,,,curl -k -T %s https://www.server.com/folder,,call anonymize_script.cq; set PatientID to "newid"'


    Marcel

    I've done this, I checked that pscp is working with the same remote location, but no luck yet with


    "--submit:837269774.1490884100,,,,{user@remote.location},{password},anonymize_script.cq"


    The server locates the files, applies anonymization script but in the end is


    ANONSRV] *** Failed uploading archive to submit

    Ah, important note: pscp isn't working without explicit specifying to use port 22. By default it tries port 0 which isn't right. Maybe I need to specify this somehow int the submit command?


    dgate64.exe -v "--submit2:837269774.1490884100,,,,pscp.exe -v -P 22 -pw {password} %s {remote location},,call anonymize_script.cq" didn't succeed as well


    Best regards,

    Niko

    Hi Niko,


    Can you first check that your pscp command line works?


    This is the relevant code for submit:


    sprintf(temp, "pscp -pw %s -batch %s %s", ident, archfile, hostsubmit);

    if (system(temp)!=0)

    OperatorConsole.printf("*** Failed uploading archive to submit\n");


    and for submit2:


    sprintf(temp, ident, archfile, hostsubmit);

    if (system(temp)!=0)

    OperatorConsole.printf("*** Failed uploading archive to submit\n");


    They are equivalent, but submit2 gives more control. Maybe first use like an echo command (windows) or cat (linux) in submit2: to write out the actual command being generated to see if that is correct.


    dgate64.exe -v "--submit2:837269774.1490884100,,,,echo pscp.exe -v -P 22 -pw {password} %s {remote location} > file.txt,call anonymize_script.cq"

    regards,


    Marcel

    Pscp command line is working fine, I have added the keys to the cache and successfully transferred several files.


    With


    dgate64.exe -v "--submit2:837269774.1490884100,,,,echo pscp.exe -v -P 22 -pw {password} %s {remote location} > file.txt,call anonymize_script.cq"


    I'm getting "This file does not have an app associated with it for performing this action. Please install an app or, if one is already installed, create an association in the Default Apps Settings page." popup error window for anonymize_script.cq. In the server console there are multiple "*** Importconverter-1.0 script not found: submit.cq" messages (one for each image I guess). file.txt is not created. I'm sorry.


    Best regards,

    Niko

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login