Cannot compute total size

retsil · Oct 14th 2008

I have hit the limit of my NAS device (2Tb) and I'm looking to archive data to a secondary NAS device.
I added the new device as MAG1 and ran the following

dgate -as1000,0
dgate -v -amMAG0,MAG1

The dgate starts printing the size of each dicom file to the console (Note I have added x in some places to protect patient identity).

Quote

K:\nmus_data\dicom\0x1x0x7\1.2.124.285.1.657524.816007.2x0x0x0x13x0x4\1.2.528.1.
1001.100.3.1673.1781.20021204110.200x0x0x2x5x5x5x2\1.2.528.1.1001.100.4.1673.178
1.20021204110.200x0x0x2x5x5x5x3.dcm = 225 KB

and then after a few minutes I got the following error

Quote

***Unable to query for image records to compute total size
MoveDataToDevice: *** could not compute KB used for patient: x3x9x8x AF\

DGATE (1.4.14, build Tue Sep 02 22:39:12 2008) is running as threaded server

Code

# This file contains configuration information for the DICOM server
# Do not edit unless you know what you are doing


[sscscp]
MicroPACS                = sscscp
Edition                  = Personal


# Network configuration: server name and TCP/IP port#
MyACRNema                = FINALARCHIVE
TCPPort                  = 4006


# Reference to other files: known dicom servers; database layout; sops
ACRNemaMap               = acrnema.map
kFactorFile              = dicom.sql
SOPClassList             = dgatesop.lst


# Host, database, username and password for MySql database
SQLHost                  = 10.8.116.41
SQLServer                = conquest
Username                 = conquest
Password                 = conquest
MySql                    = 1
DoubleBackSlashToDB      = 1


# Configure database
TruncateFieldNames       = 10
MaxFieldLength           = 254
MaxFileNameLength        = 255
FixPhilips               = 0
FixKodak                 = 0
KeepAlive                = 60
LargeFileSizeKB          = 1024
ZipTime                  = 05:
UIDPrefix                = 1.2.826.0.1.3680043.2.135.733168.61150031
EnableReadAheadThread    = 1
PatientQuerySortOrder    = 
StudyQuerySortOrder      = 
SeriesQuerySortOrder     = 
ImageQuerySortOrder      = 
EnableComputedFields     = 0
IndexDBF                 = 1
PackDBF                  = 0
LongQueryDBF             = 1000
TCPIPTimeOut             = 300
FailHoldOff              = 60
RetryDelay               = 100
QueueSize                = 128
WorkListMode             = 0
WorkListReturnsISO_IR_100 = 1
DebugLevel               = 0
Prefetcher               = 0
LRUSort                  = 
AllowTruncate            = 
DecompressNon16BitsJpeg  = 1
UseBuiltInDecompressor   = 1
IgnoreOutOfMemoryErrors  = 0
PadAEWithZeros           = 0
FileNameSyntax           = 8


# Configuration of compression for incoming images and archival
DroppedFileCompression   = un
IncomingCompression      = un
ArchiveCompression       = as


# Names of the database tables
PatientTableName         = DICOMPatients
StudyTableName           = DICOMStudies
SeriesTableName          = DICOMSeries
ImageTableName           = DICOMImages
DMarkTableName           = DICOMAccessUpdates
RegisteredMOPDeviceTable = RegisteredMOPIDs
UIDToMOPIDTable          = UIDToMOPID
UIDToCDRIDTable          = UIDToCDRID


# Banner and host for debug information
PACSName                 = FINALARCHIVE
OperatorConsole          = 127.0.0.1


# Configure email of error messages
MailHost                 = 
MailSignon               = 
MailFromName             = 
MailRcptName1            = 
MailCollectTime          = 1
MailWaitTime             = 10


# Configuration of disk(s) to store images
MAGDeviceThreshhold      = 0
MAGDevices               = 2
MAGDevice0               = K:\nmus_data\dicom\
MAGDevice1               = J:\nmus_data\dicom\
NightlyCleanThreshhold   = 0

Display More

marcelvanherk · Oct 14th 2008

Hi,

this could be a mysql specific issue. You can try to enable debug logging (debug log ON, arrows 5 times up) in the (windows) GUI and use:

dgate --selectlruforarchival:100000,0

This will start the selection task (for 100000 kb = 100 MB). The log will show all queries that are executed and could tell us what goes wrong.

To undo and try again first run:

dgate --restoremagflags:

To finally move the data you should use:

dgate --movedatatodevice:MAG1,MAG0.Archiving

Marcel

retsil · Oct 17th 2008

Thanks,

The archive appears to partially work now. However some records seem to have problems. I'm guessing that it is an escape caharcter problem with mysql.

Quote

mysql Ver 14.7 Distrib 4.1.20, for redhat-linux-gnu (i386) using readline 4.3

Code

17/10/2008 1:03:33 PM [FINALARCHIVE] Query Tables: DICOMImages
17/10/2008 1:03:33 PM [FINALARCHIVE] Columns     : DICOMImages.DeviceName, DICOMImages.ObjectFile
17/10/2008 1:03:33 PM [FINALARCHIVE] Where       : DICOMImages.ImagePat = '1x2x9x0 SPO' and DICOMImages.DeviceName = '0'
17/10/2008 1:03:33 PM [FINALARCHIVE] Order       : (null)
17/10/2008 1:03:33 PM [FINALARCHIVE] Query Tables: DICOMImages
17/10/2008 1:03:33 PM [FINALARCHIVE] Columns     : DICOMImages.DeviceName, DICOMImages.ObjectFile
17/10/2008 1:03:33 PM [FINALARCHIVE] Where       : DICOMImages.ImagePat = '1x3x7x1 SPO\' and DICOMImages.DeviceName = '0'
17/10/2008 1:03:33 PM [FINALARCHIVE] Order       : (null)
17/10/2008 1:03:33 PM [FINALARCHIVE] ***Unable to query for image records to compute total size
17/10/2008 1:03:33 PM [FINALARCHIVE] Query Tables: DICOMImages
17/10/2008 1:03:33 PM [FINALARCHIVE] Columns     : DICOMImages.DeviceName, DICOMImages.ObjectFile
17/10/2008 1:03:33 PM [FINALARCHIVE] Where       : DICOMImages.ImagePat = '0x6 4x x4 MG' and DICOMImages.DeviceName = '0'
17/10/2008 1:03:33 PM [FINALARCHIVE] Order       : (null)

Display More

marcelvanherk · Oct 17th 2008

Hi,

what is the actual patient ID of

'1x3x7x1 SPO\'

Marcel

retsil · Oct 21st 2008

The DICOM files were originally added to conquest using "dgate.exe --addimagefile"
It appears that the Patient ID had the initials of the operator included.

Quote

[FINALARCHIVE] Query Tables: DICOMImages
17/10/2008 1:03:20 PM [FINALARCHIVE] Columns : DICOMImages.DeviceName, DICOMImages.ObjectFile
17/10/2008 1:03:20 PM [FINALARCHIVE] Where : DICOMImages.ImagePat = '2440681 AF\' and DICOMImages.DeviceName = '0'
17/10/2008 1:03:20 PM [FINALARCHIVE] Order : (null)
17/10/2008 1:03:20 PM [FINALARCHIVE] ***Unable to query for image records to compute total size

Display More

I used dcmdump on the file and got

Quote

(0010,0020) LO [2440681 AF\] # 12, 2 PatientID

It also appears that conquest can't find these patients when using DICOM remote query, however the entries do exist when I access via mysqladmin.

It appears to only be affecting about 600 images so I'll just move them out and re-initialise (0.04% of all images)

retsil · Oct 21st 2008

Thanks for that,
I managed to start moving image data to the archive device

Quote

dgate --restoremagflags:
dgate --selectlruforarchival:100000,MAG0
dgate --movedatatodevice:MAG1,MAG0.Archiving

Note that I used --selectlruforarchival:100000,MAG0 because --selectlruforarchival:100000,0 doesn't work and repsonds with the answer Nothing to do.

urry · Jan 18th 2009

Hi,

I've recently arranged data archiving with Conquest v1.4.14 and use a batch with the command-lines proposed by retsil. Thanks, it works.
Additionally, in batch, I've encapsulated dgate commands by time logging echo lines to track process performance:

echo Start: %date% %time%
dgate --restoremagflags:
echo End: %date% %time%

echo Start: %date% %time%
dgate --selectlruforarchival:100000,MAG0
echo End: %date% %time%

echo Start: %date% %time%
dgate --movedatatodevice:MAG1,MAG0.Archiving
echo End: %date% %time%

With my relatevely big database (MSSQL containing >200 000studies, >6mln images) for every dgate line in the code above it executes exactly 5min no matter how many data you select to move (I tried to move separately 10MB, 100MB, 10GB).

marcelvanherk · Jan 18th 2009

Hi,

5 min seems on long side. Do you have database maintenance running on your MSSQL server?

Marcel

urry · Jan 26th 2009

Hi, Marcel
To say the true, we haven't done MSSQL maintenance for 2.5years. Now we've scheduled weekly maintenance and after first one the archiving commands run twice faster.

However, another issue came up. Archiving works properly when we archive MAG0 data to MAG1 (or any arbitrary MAG?). But could you please comment, is it possible to move data from let's say MAG2 to MAG3?

The reason is: previously we archived uncompressed onto MAG2 but now we want to move all MAG2 data onto another NAS and simultaneously compress it with j1.
I tried:
dgate --restoremagflags:
dgate --selectlruforarchival:1000000,MAG2
dgate --movedatatodevice:MAG3,MAG2.Archiving
Serverlog says: nothing to move.

Obviously, archiving is only possible from MAG0, isn't it?

marcelvanherk · Jan 26th 2009

Hi,

hm, can you show some more detailed logs? The commands look allright to me and should work. There is no difference between different MAGs I believe.

Marcel

urry · Jun 9th 2009

It all works but must to note that in our case if we move 100GB of image data, then we run first two steps:
dgate --restoremagflags:
dgate --selectlruforarchival:100000000,MAG2

then it starts to compress images (which are initially un-compressed) to j1. This compression process takes over 20h.
Then it's over we start 3rd process:
dgate --movedatatodevice:MAG3,MAG2.Archiving

Moving takes a few hours.

Marcel, please advise, can we could move 100GB to MAG3 without changing images compression and compress only then data is moved onto MAG3? This matters if in our case the MAG3 is a newer and faster NAS.

marcelvanherk · Jun 9th 2009

Hi,

Close the server, change archivecompression in dicom.ini to "as" which means as-is, and then start the server again. This will stop the compression step.

Marcel

marcelvanherk · Jun 9th 2009

Hi,

Close the server, change archivecompression in dicom.ini to "as" which means as-is, and then start the server again. This will stop the compression step.

Marcel

Cannot compute total size

Participate now!

Share