Detecting corrupted files in large archive

  • 1. Wow, that's cool. Can I only use this in a database regeneration script or will the import work too?

    2. Do I need to replace only this file before compiling?

    I think that the damaged data should be put in a separate folder. But I will be satisfied with their removal

    Thank you very much I don’t even know how to thank you.

  • If you can explain what is available with the UIDs. This is due to the storage method by the server or due to the setting FileNameSyntax. OR the is just corrupted.
    And if the data can be partially restored, how?

  • Ok,


    Let's test first. I would start a new test server.


    Next to data, I would create a folder 'rejected'. Then add the following lines to the end of dicom.ini:


    [lua]

    RejectedImageConverter0 = dofile('lua/association.lua'); local a=string.gsub(Filename, 'incoming', '../rejected'); script('mkdir'..a:match("(.*/)")); filecopy(Filename, a)


    Then put some test data in incoming. Once you do this all files that do not load should be moved to the rejected folder and all files deleted in incoming.


    Next I will need to create an option such that jpeg errors/warning lead to rejection as well - I will do that later.


    Give a try, there may be other crashes or hangs. Once you find these please share those files with me. The script is slow because failed loads on incoming a retried 3 times with a 5s delay.


    Marcel

    Marcel van Herk is developer of the Conquest DICOM server together with Lambert Zijp.

    Edited 3 times, last by marcelvanherk: Edited after testing script ().

  • Oh,


    in dgate.cpp, change:


    for (int i=0; i<30; i++)


    into:


    for (int i=0; i<0; i++)


    To speed it up a lot.


    I will also add scripting to regen to allow scripts to detect and handle broken files.


    This will be the syntax:


    Code
    StopOnJPEGWarnings = 1
    [lua]
    RegenConverter0 = if not Data:Compress('un') then os.rename(Filename, Filename .. '.JpegError'); destroy() end; if not Data.PatientID then os.rename(Filename, Filename .. '.UIDerror'); destroy() end
    RegenFailConverter0 = os.rename(Filename, Filename .. '.FailedToLoad')

    Marcel

  • The associated changes in the program (several files) are now on github.


    Code
    StopOnJPEGWarnings = 1
    [lua]
    RegenConverter0 = if not Data:Compress('un') then os.rename(Filename, Filename .. '.Jpeg.Err'); destroy() end; if not Data.PatientID then os.rename(Filename, Filename .. '.UID.Err'); destroy() end
    RegenFailConverter0 = os.rename(Filename, Filename .. '.Load.Err')


    When you add the above lines to dicom.ini, incorrect files will be detected during regen, and labelled by the script with an extra extension, e.g.:



    Of course you can do anything you like in the two scripts. Hope this helps!


    Marcel

    Marcel van Herk is developer of the Conquest DICOM server together with Lambert Zijp.

    Edited once, last by marcelvanherk: Gave renames an .Err extension so that subsequent regens no longer see the files (regen tests the last 4 characters only). ().

  • Now I transfer the data(6.8TB) to the backup disk and then run the script.(it will take some time)

    please specify, have all the files been changed on the github?

    trnsyn.cxx dgate.cpp

    Can I just download the archive and compile file dgate ?

  • Yes,


    All files are on github - you need a full download and recompile. Both mechanisms are supported, but I would use the regen only as it is much fsater.


    By the way, the changes have been tested for your problem, but may break other server functionality. Lets see what othere report,


    Merry Christmas!


    Marcel

  • Merry Christmas!

    I will try to test with incoming folder and get this.


    ***Length exceeds remaining file size: 0003b57c


    ***[AddImageFile] /dicom_data/vc_cherkasy/incoming/0118_BUGERYA_I_O/CT_45337/3/1.2.840.113704.1.111.4728.1602768057.6742.dcm -FAILED: Error on Load

    ***Length exceeds remaining file size: 0003b57c


    ***[AddImageFile] /dicom_data/vc_cherkasy/incoming/0118_BUGERYA_I_O/CT_45337/3/1.2.840.113704.1.111.4728.1602768057.6742.dcm -FAILED: Error on Load

    my dicom.ini

    with and without string

    Code
    association = package.path=package.path..';'..Global.basedir..'lua/?.lua'


    maybe my fail?
    I think lua scripts dont working. I reinstall packages

    "apt-get install --reinstall lua5.1 lua5.1-dev liblua5.1-0 lua-socket"

    But its not helped.

    Next step test regen.

  • Hi,


    it retries 30 times (you can change the loop in dgate.cpp to shorten that). Thats why incoming is not a good idea. Many users upload from the internet into incoming and if that is slow the load will fail on partial files. Therefore it retries so much.


    I would try the regen option, use the lines in #27. To test I used;


    dgate --regendir:MAG0,CT_xxxxx


    Marcel

  • marcelvanherk

    Changed the title of the thread from “(Linux-Debian9)Fail in exporting from incoming” to “Detecting corrupted files in large archive”.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!