Detecting corrupted files in large archive

  • Somebody give answer. Whats going on


    Thu Dec 17 01:48:06 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:11 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:11 2020 ***Length = 7274617


    Thu Dec 17 01:48:11 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:16 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:16 2020 ***Length = 7274617


    Thu Dec 17 01:48:16 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:21 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:21 2020 ***Length = 7274617


    Thu Dec 17 01:48:21 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:26 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:26 2020 ***Length = 7274617


    Thu Dec 17 01:48:26 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:31 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:31 2020 ***Length = 7274617


    Thu Dec 17 01:48:31 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:36 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:36 2020 ***Length = 7274617


    Thu Dec 17 01:48:36 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:41 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 0000001f)


    Thu Dec 17 01:48:41 2020 ***Length = 7274617


    Thu Dec 17 01:48:41 2020 ***[AddImageFile] /data_disks/dicom_exchange/incoming/Kononovych_V_G_246/CT_75823/4/1.2.840.113704.1.111.140.1606900580.5444.dcm -FAILED: Error on Load

    Thu Dec 17 01:48:42 2020 ***Dynamic_Parse encountered an invalid element length during load of DCM file (in 00530054)


    Thu Dec 17 01:48:42 2020 ***Length = 10551429

  • Thu Dec 17 01:55:17 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:22 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:27 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:32 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:37 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:42 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:47 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:52 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

    Thu Dec 17 01:55:57 2020 ***[AddImageFile] /data_disks/dicom_vitacom/vc_cherkasy/incoming/20201/12/04/1151_PUZYR_N_PL/CT_46516/9/1.2.840.113704.1.111.3388.1607070156.13778.dcm -FAILED: file does not contain correct UIDs

  • ***[AddImageFile] /dicom_exch/incoming/BABYUK_M_M_1125/CT_46486/2/1.2.840.113704.1.111.1432.1606983429.4170.dcm -FAILED: Error on Load


    And how ignore?


    I have a problem. During the recovery process by destroying the database and moving them to a new server with 150b, it turned out that not all data was in order.

    I put all the data in the incoming folder
    Tell me where to add something so as not to delete the damaged files manually.


    (Mariadb database)

  • Now i have a problem, conquest not import from the incoming folder
    I have 100% load on process


    but nothing happens

    add line WatchFolder
    nothing happens
    but studies fine.
    Conquest does not want to import.

  • I use this script maybe i need
    drop him from memory or somewhere
    RejectedImageConverter0 = x=DicomObject:new(); m=':'; y=Data.PatientID; x='deletepatient'; a=x..m..y; print(a); servercommand(a); retry()

    disabling not effect.

  • Okay, i find problems with disk storage. Now IN Log

    Sun Dec 20 13:37:01 2020 ***A fatal error occurred (out of memory) - closing server

    Sun Dec 20 13:37:14 2020 *** Not enough rights to write in MAG0

    Sun Dec 20 13:43:37 2020 ***VR:ReAlloc out of memory allocating -946867618 bytes


    Sun Dec 20 13:43:37 2020 ***A fatal error occurred (out of memory) - closing server

    Sun Dec 20 13:43:53 2020 *** Not enough rights to write in MAG0

    Sun Dec 20 13:49:46 2020 ***VR:ReAlloc out of memory allocating -428730994 bytes


    Sun Dec 20 13:49:46 2020 ***A fatal error occurred (out of memory) - closing server

    Sun Dec 20 13:50:00 2020 *** Not enough rights to write in MAG0

    Sun Dec 20 13:51:28 2020 ***VR:ReAlloc out of memory allocating -394025210 bytes


    Sun Dec 20 13:51:29 2020 ***A fatal error occurred (out of memory) - closing server

    Sun Dec 20 13:51:44 2020 *** Not enough rights to write in MAG0

    Sun Dec 20 13:54:08 2020 ***VR:ReAlloc out of memory allocating -345509096 bytes



    When i fix rights and restart server does not see images in incoming folder.

  • Now when I fix Raid controller.
    I have two problems:
    1.Conquest does not see images in incoming folder
    2. I have corrupted/damaged dcm and don't know how to delete them not manually/

    okay I am writing here for those who encounter the same problem.
    if in the raid array one disk is not working properly, then the data can be selectively damaged.

    and you get an error like this:

    ***[AddImageFile] /dicom_olddata/incoming/11155__ABDOMEN_CE_BUTURLIA_M_B/CT_11156/12/1.2.392.200036.9116.2.6.1.48.1211418729.1540873727.13 -FAILED: Error on Load

    ***[AddImageFile] /dicom_olddata/incoming/11155__ABDOMEN_CE_BUTURLIA_M_B/CT_11156/12/1.2.392.200036.9116.2.6.1.48.1211418729.1540873727.13 -FAILED: Error on Load


    the script may cause the Conquest to stop pulling studies from the incoming folder and at the same time in the PacsTrouble.log files there may be no information why.

    remove your script restart server with command and start server with

    ./dgate --quit:

    Unfortunately, I did not find a single script or algorithm with which it was possible to ignore/delete/move broken files. PLEASE HELP.

  • and you know what, I move the files manually, and manually delete the broken ones and everything goes.

    The only thing is not clear why in the Incoming folder it selectively stops the process, the rights are configured.

    Moving directories and restarting the process helps. The truth is not clear why

  • Gragoro

    Changed the title of the thread from “Fail in exporting from incoming” to “(Linux-Debian9)Fail in exporting from incoming”.
  • Hi,


    I understand you have a large amount of images of which some are corrupted. Are you using a backup of the original drive?


    I think a script that tries to read each image, and moves corrupt images out of the folder may be the test way to go.


    Marcel

  • Hi,


    the data corruption causes Conquest to hang. I create a version of dgate that does not crash, by adding extra checks to attached trnsyn.zip.


    This allows me to load the data you shared without crashes. About 1/3 files are corrupt. There remains corruption in the jpeg data that does load such as this image. What do you want to do with these?


    Marcel


  • Some statistics on the failures:


    128 images could not load


    96 images miss UIDs (likely because of truncation)


    1407 images loaded

    of the loaded images, 137 images have jpeg errors. Most jpeg errors (101) are huffman code errors where part of the image may be recovered.


    All in all about 1/4 images has a problem.


    Marcel

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!