Anonymizing Pixel data

joelspangler · Jun 2nd 2014

I am starting to write up a lua script to block out pixel data (such as patient names on ultrasounds and CT dosage reports) as part of a research project that requires anonymized data. I have successfully gotten single frame images blocked out, but when multi-frame images are sent to Conquest, I get an error message "dgate64.exe has stopped working" message (details below). Conquest logs do not contain any information as to why it crashed.

Here's the lua script that I am running - this is in the VERY early stages. I will eventually research how to create a function with LUA so it's a little less repetitive. I will also eventually need to figure out how to tell the script to change multiple frames on multi-frame files (is there a way to get a count of frames?)... First things first is to figure out why it is crashing when trying to handle multi-frame images. The section of code it's choking on would be the code under the "-- US GE Logiq E9 normal images" comment

Code

-- Script to anonymize pixels when certain conditions are met--uncomment below print statements (and last line of this document)for debugging--print('------ Anon Pixel script running --------')--print ('Manuf', Data.Manufacturer)--print ('SeriesNum', Data.SeriesNumber)--print ('Modality', Data.Modality)--print ('SeriesDesc', Data.SeriesDescription)--print ('ManufacturerModelNam', Data.ManufacturerModelName)--print ('imagenumber', Data.ImageNumber)--print ('imagedesc', Data.ImageDescription)--print ('InstanceNumber', Data.InstanceNumber)-- Toshiba CT dose pageif Data.Manufacturer == "TOSHIBA" and Data.SeriesNumber == "1000" and Data.Modality == "CT"  then	print('------ condition for Toshiba dose page met------')	for r=0, 300 do		for c=0, 511 do setpixel(c, r, 0, 0) end		endend-- CT Series named "screen save"if Data.SeriesDescription == "Screen save" then	print('------ condition for screen save page met------')	for r=0, 63 do		for c=300, 511 do setpixel(c, r, 0, 0) end		endend-- US GE Logiq E9 Cover Pageif Data.ManufacturerModelName == "LOGIQE9" and Data.InstanceNumber == "0" then	print('------ condition for LogicE9 cover page met------')	for r=16, 45 do		for c=175, 350 do setpixel(c, r, 0, 0) end    for c=510, 685 do setpixel(c, r, 0, 0) end    for c=830, 950 do setpixel(c, r, 0, 0) end  end  for r=65, 90 do		for c=175, 350 do setpixel(c, r, 0, 0) end    for c=510, 685 do setpixel(c, r, 0, 0) end    for c=830, 950 do setpixel(c, r, 0, 0) end  end  for r=110, 140 do		for c=510, 685 do setpixel(c, r, 0, 0) end  endend-- US GE Logiq E9 normal imagesif Data.ManufacturerModelName == "LOGIQE9" and Data.InstanceNumber ~= "0" then	print('------ condition for GE LogiqE9 normal images met------')	for r=0, 65 do		for c=338, 670 do setpixel(c, r, 0, 0) end  endend--print('------ Anon Pixel script stopping --------')

And below is the details on the dgate64.exe crash as reported by windows.

Code

problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	dgate64.exe
  Application Version:	0.0.0.0
  Application Timestamp:	531c782f
  Fault Module Name:	dgate64.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	531c782f
  Exception Code:	c0000005
  Exception Offset:	000000000004e075
  OS Version:	6.1.7601.2.1.0.274.10
  Locale ID:	1033
  Additional Information 1:	8631
  Additional Information 2:	8631c8499cbbccba896fd9837e18128a
  Additional Information 3:	04e6
  Additional Information 4:	04e63b1969cbb1e767d00c4b31d53233


Read our privacy statement online:
  http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409


If the online privacy statement is not available, please read our privacy statement offline:
  C:\Windows\system32\en-US\erofflps.txt

Display More

joelspangler · Jun 2nd 2014

I also noticed that color images remain unchanged after running through the script. Any thoughts on why that might be?

marcelvanherk · Jun 3rd 2014

Hi Joel,

I have tested on multiframe data (this is 1.4.17d?), but not on color data.To speed things up you can best use getrow and setrow or even getimage and setimage. Number of frames comes from Data.NumberOfFrames

Marcel

joelspangler · Jun 3rd 2014

I am on 1.4.17d, and haven't found a multi-frame image that hasn't caused a crash yet. Could it be the compression of the image causing the problems? Should I try to make the program change the file to "un" first before trying to modify it. My current code also is setting frame 0 - setpixel(c, r, 0, 0) which works fine for single frame black and white files... do I need to start with frame 1 on multi-frame files?

Also, I'm having trouble identifying some series with "burned in" information. There is useful information in private tags, but I'm unable to figure out how to address private tags within LUA. I tired to do the following as a test. print ('tagtest', Data.00731003) --- That resulted in an error: D:\dicomserver-SQL\lua\BlockPixel1.lua:13: ')' expected near '.00731003'. Is there a way to address a specific "unnamed" or "private" tag?

I tried to figure out the setrow command - the syntax appears to be SetRow(y, frame, table). I tried setrow(1, 1, 0) - which causes dgate to crash. For "table" - would I have to create a table with the correct number of pixels to run this command?

Lastly, I figured out how to create a function - the code below does almost exactly what the one I posted yesterday does, but this one would be a bit easier for others to add to if/when it goes into more of a production mode.

Code

-- Script to anonymize pixels when certain conditions are met
--uncomment below print statements (and last line of this document)for debugging
print('------ Anon Pixel script running --------')
--print ('Manuf', Data.Manufacturer)
--print ('SeriesNum', Data.SeriesNumber)
--print ('Modality', Data.Modality)
--print ('SeriesDesc', Data.SeriesDescription)
--print ('ManufacturerModelNam', Data.ManufacturerModelName)
--print ('imagenumber', Data.ImageNumber)
--print ('imagedesc', Data.ImageDescription)
--print ('InstanceNumber', Data.InstanceNumber)
print ('numberofframes', Data.NumberOfFrames)


--Create blockpixel function to prevent repetitive code
function blockpixel(RowStart, RowEnd, ColumnStart, ColumnEnd)
  for r=RowStart, RowEnd do
    for c=ColumnStart, ColumnEnd do setpixel(c, r, 0, 0) end
  end
end
--End of Blockpixel function


-- Toshiba CT dose page #1
if Data.Manufacturer == "TOSHIBA" and Data.SeriesNumber == "1000" and Data.Modality == "CT"  and Data.InstanceNumber == "1" then
	print('------ condition for Toshiba dose page met------')
	blockpixel (1, 300, 0, 511)


end
-- Toshiba CT dose page #2 and above
if Data.Manufacturer == "TOSHIBA" and Data.SeriesNumber == "1000" and Data.Modality == "CT"  and Data.InstanceNumber ~= "1" then
	print('------ condition for Toshiba dose page met------')
	blockpixel (1, 60, 0, 511)
end
-- CT Series named "screen save"
if Data.SeriesDescription == "Screen save" then
	print('------ condition for screen save page met------')
	blockpixel (0, 63, 300, 511)
end
-- US GE Logiq E9 image 0
if Data.ManufacturerModelName == "LOGIQE9" and Data.InstanceNumber == "0" then
	print('------ condition for LogicE9 cover page met------')
	blockpixel (16, 45, 175, 350)
  blockpixel (16, 45, 510, 685)
  blockpixel (16, 45, 830, 950)
  blockpixel (65, 90, 175, 350)
  blockpixel (65, 90, 510, 685)
  blockpixel (65, 90, 830, 950)
  blockpixel (110, 140, 510, 685)
end
-- US GE Logiq E9 normal images
if Data.ManufacturerModelName == "LOGIQE9" and Data.InstanceNumber ~= "0" then
	print('------ condition for GE LogiqE9 normal images met------')
  blockpixel (0, 65, 338, 670)
end
-- US Phiips iU22 Cover page
if Data.ManufacturerModelName == "iU22" and (Data.InstanceNumber == "10001" or Data.InstanceNumber == "1") then
	print('------ condition for Philips iU22 Cover page met------')
  blockpixel (35, 65, 165, 600)
  blockpixel (35, 35, 730, 865)
  blockpixel (75, 107, 165, 600)
  blockpixel (75, 107, 665, 710)
  blockpixel (75, 107, 840, 870)
  blockpixel (115, 145, 165, 600)
  blockpixel (160, 185, 165, 355)
  blockpixel (390, 420, 255, 940)
end
-- US Phiips iU22 normal images
if Data.ManufacturerModelName == "iU22" and Data.InstanceNumber ~= "10001" then
  if Data.InstanceNumber ~= "1" then
    print('------ condition for Philips iU22 normal images met------')
    blockpixel (1, 30, 140, 525)
    blockpixel (40, 70, 140, 350)
  end
end
print('------ Anon Pixel script stopping --------')

Display More

marcelvanherk · Jun 3rd 2014

Yes,

Indeed you need to be at UN for pixel access (known issue).

The table is one you read with getrow. It lists all pixels in the row and you can change part of it and rewrite.

for row=100,200 do
a = getrow(row, frame)
sum = 0
for i=1,100 sum = sum + a[i] end -- average
for i=1,100 a[i]=sum/100 end -- replace
setrow(row, frame, a)
end

Any VR access: Data["0020,000d"].

Marcel

joelspangler · Jun 4th 2014

What is the best way to get the file to "un" before processing? Would I need to set storage on my archive to "un", and run my LUA as an exportconverer (rather than an inportconverter), or is there another way to decompress without requiring my archive to be set to "un"

marcelvanherk · Jun 4th 2014

Hi Joel,

you can call script('compression UN') or Data:Script('compression UN') from lua to decompress the data.

Marcel

joelspangler · Jun 12th 2014

I can now block areas in CINE loop objects, and am tweaking the script as needed to block out as much PHI as possible. My major limitation to completely anonymizing data at this point is color frame ultrasound images. As far as I can tell, dgate is trying to do something with the color images, but it fails to actually change the pixels.

Here's my current script which now includes a blockcine funtion. I've been focusing on GE Logic E9 ultrasounds. These are a bit annoying because PHI is "burned in" at different locations for 3 different kinds of images (Cover pages, Regular single frame images, and CINE loop images). Coming up with ways to identify these different types of images was fairly difficult, but this current revision has been successful in every images set from an E9 that I've thrown at it.

Code

-- Script to anonymize pixels when certain conditions are met
--uncomment below print statements (and last line of this document)for debugging
print('------ Anon Pixel script running --------')
--print ('Manuf', Data.Manufacturer)
--print ('SeriesNum', Data.SeriesNumber)
--print ('Modality', Data.Modality)
--print ('SeriesDesc', Data.SeriesDescription)
--print ('ManufacturerModelNam', Data.ManufacturerModelName)
--print ('imagenumber', Data.ImageNumber)
--print ('imagedesc', Data.ImageDescription)
--print ('InstanceNumber', Data.InstanceNumber)
print ('numberofframes', Data.NumberOfFrames)
-- print ('PACSAETITLE', Data["0073,1003"])


Data:Script('compression UN') 


--Create blockpixel function to black out a "box" on single frame images
function blockpixel(RowStart, RowEnd, ColumnStart, ColumnEnd)
  for r=RowStart, RowEnd do
    for c=ColumnStart, ColumnEnd do setpixel(c, r, 0, 0) end
  end
end
--End of Blockpixel function
--create blockcine function to black out a "box" on multi-frame dicom objects
function blockcine(RowStart, RowEnd, ColumnStart, ColumnEnd)
  for f= 0, Data.NumberOfFrames do
    for r=RowStart, RowEnd do
      for c=ColumnStart, ColumnEnd do setpixel(c, r, f, 0) end
    end
  end
end
--End of blockcine function
-- Toshiba CT dose page #1
if Data.Manufacturer == "TOSHIBA" and Data.SeriesNumber == "1000" and Data.Modality == "CT"  and Data.InstanceNumber == "1" then
	print('------ condition for Toshiba dose page met------')
	blockpixel (1, 300, 0, 511)


end
-- Toshiba CT dose page #2 and above
if Data.Manufacturer == "TOSHIBA" and Data.SeriesNumber == "1000" and Data.Modality == "CT"  and Data.InstanceNumber ~= "1" then
	print('------ condition for Toshiba dose page met------')
	blockpixel (1, 60, 0, 511)
end
-- CT Series named "screen save"
if Data.SeriesDescription == "Screen save" then
	print('------ condition for screen save page met------')
	blockpixel (0, 63, 300, 511)
end
-- US GE Logiq E9 image 0
if Data.ManufacturerModelName == "LOGIQE9" then
  if string.find(Data.ImageType, "GEMSSINGLEFRAME") then
    -- US GE Logiq E9 normal images
    print('------ condition for GE LogiqE9 normal single frame image met------')
    blockpixel (0, 65, 338, 670)
  else
    if Data.NumberOfFrames ~= nil then
      -- GE LogiqE9 cine images
      print('------ condition for GE LogiqE9 cine images met------')
      blockcine (0, 30, 170, 540)
    else
      if Data.InstanceNumber <= "1" then
        print('------ condition for LogicE9 cover page met------')
        blockpixel (16, 45, 175, 350)
        blockpixel (16, 45, 510, 685)
        blockpixel (16, 45, 830, 950)
        blockpixel (65, 90, 175, 350)
        blockpixel (65, 90, 510, 685)
        blockpixel (65, 90, 830, 950)
        blockpixel (110, 140, 510, 685)
      else
        print('------ condition for GE LogiqE9 normal single frame image met------')
        blockpixel (0, 65, 338, 670)        
      end
    end
  end
end
--US Phiips iU22 Cover page
if Data.ManufacturerModelName == "iU22" and (Data.InstanceNumber == "10001" or Data.InstanceNumber == "1") then
	print('------ condition for Philips iU22 Cover page met------')
  blockpixel (35, 65, 165, 600)
  blockpixel (35, 35, 730, 865)
  blockpixel (75, 107, 165, 600)
  blockpixel (75, 107, 665, 710)
  blockpixel (75, 107, 840, 870)
  blockpixel (115, 145, 165, 600)
  blockpixel (160, 185, 165, 355)
  blockpixel (390, 420, 255, 940)
end
-- US Phiips iU22 normal images
if Data.ManufacturerModelName == "iU22" and Data.InstanceNumber ~= "10001" then
  if Data.InstanceNumber ~= "1" then
    print('------ condition for Philips iU22 normal images met------')
    blockpixel (1, 30, 140, 525)
    blockpixel (40, 70, 140, 350)
  end
end
print('------ Anon Pixel script stopping --------')

Display More

marcelvanherk · Jun 12th 2014

Hi,

looks nice, I will look into color images. In any case, a getrow/setrow loop with clearing pixels in each row will be quicker than repeated calls to setpixel.

Marcel

marcelvanherk · Jun 12th 2014

Try setpixel(row, col, frame, 0, 0, 0) should work for color and b/w

Marcel

joelspangler · Jun 12th 2014

setpixel(row, col, frame, 0, 0, 0) works on color images, and does not seem to adversely affect b/w images... although I think it's actually setpixel (col, row, frame, 0, 0, 0). I'll play with a different model US perhaps one that uses color cine loops to see if I can find anything it doesn't work on. I'll post updated scripts as I find time to research where PHI is "burned in" for different models of machines.

As far as speed, it seems conquest is keeping up fine, even if I queue multiple exams back to back from my PACS. Since I rarely blank a whole row, I don't know that the getrow/setrow would be very useful at this point.

marcelvanherk · Jun 12th 2014

Great!

I timed a few microseconds per setpixel. A nice trick would be to replace the block you are masking by its mean pixel value, it would look nicer than just setting it to black.

Thanks for the script.

Marcel

joelspangler · Jun 12th 2014

A lot of the work is discovering what pixels need to be covered up - that work won't be lost. If the functions that I built are improved upon (such as your "neat trick" mentioned above") I wouldn't be upset about that at all. I don't know if building a function directly into dgate.exe to anonymise a "block" or range of pixels would be appropriate/faster than doing it in LUA.

Anonymizing Pixel data

Participate now!

Share