CT Scan Project Outline and Progress Log

Jun 21-25

• Email Charles Keller and ask him how many data sets he will give us and how much memory they will take up.
• Allocate space on the server. Figure out about how much memory we will need.
• By June 25, make two separate hard coded html pages for each of the two data sets we currently have. They are located at software2.sci.utah.edu.

• Using Perl, make a web based interface which allows the user to enter information on a data set into a form, automatically creating the html/xml page.
• In order for Toly to gain full comprehension of the functionality of the search engine, instead of tweaking what we already have with the sci image gallery, we will redo the project altogether. This will maximize Toly's learning and productivity if he becomes a permanent employee after the end of summer.
• Put the size of the file under the download button.
• Allocate space on the server for the datasets that we currently have. Clean up server from any files we don't need.
• Make a Perl program that reads a vff header and writes a relevant nrrd header.

• The main objective of this week is to model a search according to the existing SCI image search. Unlike in the sci image gallery, packets of data sets will not be used. Make sure this search engine works for the two candy scans before we add more.
• Decide on a private/public scheme. Should we use a user name and password to create an account? Who should we allow to view private images? How do we keep those people who can view certain private images from viewing all other private images?
• Get all the data sets from whoever has them and enter their information onto the web site. Each ct scan will have its own page because of the amount of information we will include about each.
• We will have to obtain several 2D thumbnail images of different angles of each 3D scan. These thumbnails is what the users will see when searching for the scan that they need.

• Write a program that looks into a given directory and automatically writes nrrd headers for all vff files in that directory.
• Add a capability to the internal web site that adds datasets to the xml database.
• Make the dataset info form able to update the xml database.
• Add about 10 datasets to the xml database and create thumbnails for them.
• Make sure the basic search engine works all right.
• Get information about manufacturer for scanner. Also ask Keller what information we should display on the web site that's in the VHH header.

• Get the upload capability to work.
• Set up a meeting with Noelle and Ross and decide whether we should add or take anything out of the web site.
• Write a program that writes vff header info to separate file so that we can see what goes into it without having to open up the huge .vff file.

• Add vff header info to the form and dataset output page.
• Fix bugs in upload.
• Upon upload check whether the file being uploaded is zipped or not and skip unzip phase if it's not.
• Discovered bugs: 1)When both fields are filled out in Upload Dataset, the 'cancel' button uploads the dataset. 2)Have script check whether uploading file already exists before the file is actually uploaded by matching the dataset ID with the folder name.
• Learn about Apache and .htaccess. Password protect the internal CT Datasets page.

• Read Gordon's papers regarding NRRD.
• Practice volume rendering with miter.
• Fix the bugs that we discovered last week.
• Get Keller Lab people to view our web site.
• Use regular expressions so that upon deletion of a datset, only letters, numbers, dashes, and underscores are allowed to be in the datasetId.

• Make a button that restores VFF header info back to its default values, which are stored in the VFF file. VFF header info will still be editable in the xml database but that should not change the original .vff file.
• After achieving several insights into how the dataset uploads, we have finally figured out why large datasets would not upload all the way. All we needed to do is free up some space in one partition of the file server so that we may temporarily copy the dataset there before copying it into the correct directory. Now we want to make it possible to change the directory where the file gets temporarily copied to.
• Make imgages for all the datasets and upload all the datasets that we have.
• Large datasets time out after they finish uploading and after the file is unzipped so we need to try uploading a huge dataset to see if it will time out before it finishes uploading.
• Instead of displaying the vff file when the user clicks on download, have the link bring up a download box.
• Give the user the option to sort the datasets that are displayed on the page by title, dataset id, or scan date independently of the search.

• Make it so that there is only 10 datasets per page and if there is more, make a function that displays a horizontal list of numbered pages. One problem that we need to fix is that the order of datasets (by name, date, or id) resets to by name when we go from page to page.
• Give Keller Lab computer access to web site and get feedback from Charles Keller.
• Learn to use Simian and Miter. Meet with Tommy and Mark. Install Simaian on my computer and get the appropriate driver for it.
• Learn about crontab. This may be our alternative to forking our upload and unzip processes. Also read up on forking, create a simple program that demonstrates how forking works, and then incorporate it into our CT datasets web site.
• cron job plan: when the dataset finishes uploading, the script will add an empty xml entry for that dataset and give it the status "ready for unzip". The web site will then display the empty "coming soon" dataset with default thumbnails and that gives the user the option to edit the dataset info before the dataset is even unzipped. Aferward, we will have a cron job run a separate script that unzips the dataset, creates the directory, makes thumbnails, and adds info to the xml database. In order to find the right zipped files to unzip, this perl script will go into the directory that contains newly uploaded datasets, takes their id, and goes into the xml database to check the status of that dataset. If the status is set to "unzipping" the script ignores that dataset and continues to look for zipped datasets that have the status "ready to unzip". That way we can avoid unzipping the same file twice if the amount of time it takes to unzip the file is longer than the interval at which we check for new files to unzip. After the script unzips a dataset and adds it to the database, it will delete the zipped file from the directory. The xml database will automatically be updated and the user can view the complete dataset info from the web site.

• Meet with Lance at the Keller Lab and discuss dataset upload procedures.
• Fix errors in the processing script.
• Implement a crontab.

• Install teem on dante.
• Write a script that automatically creates projections of newly uploaded datasets. We will then use the color image for our thumbnails.
• Find out how to make three different MIPs (var, sum, max) and combine them to create a color image.
• Add labels to thumbnail images and display only the images that are maximum projections. We want to show all three axis and label them in the format "x-axis".

• Have the script distinguish the vff file header from the binary part by looking for the ^L, which we determined to be a ctl-L, not an ASCII char.
• Switch to taint mode and untaint all the variables that we use to get information from the browser. Read up on untainting in the Programming Perl book.
• Even though I downsized the nrrd file of the datasets they were still too large for the 64 MB graphics card on my computer. I may have to use a windows machine in the lab that is shared by several people. Otherwise I would have to downsize the datasets to a poor resolution.
• Start the powerpoint presentation for next week.
• Install SCIRun

• Refine the program so that it is unbreakable.
• For security, turn on untaint mode and create regular expressions that will render user input innocuous.
• Meet with Ross, Dav, Greg, and Erik to discuss the progress of the project and what to do next.

• Write first draft of documentation for the project. Erik will create a diagram explaining the project.
• Display all 12 thumbnails on each dataset page.

• Add info to, revise, and improve the documentation. Copy the documentation to html and add it to the entry page.
• Make a thumbnail for newly uploaded datasets so that upon upload, the script won't display a broken image.
• Create the CT datasets archive entry page with necessary links and a short intro. Combine some thumbnail images to put on the main page.
• Start on the technical report.
• Find out about citation.

• By default do not display the vff header info in the datasets display page but allow the user to click on a link that reloads the page with all the vff info.
• Have Erik proofread the documentation. Add info to it and make refinements.
• Add code that calculates the physical size of the dataset, stores it in the xml database, and displays it in the dataset display page.

• Have Erik proofread the documentation. Add info to it and make refinements.
• Edit existing sci software login script to allow sci customers to access the datasets archive.
• Make the download button a submit type button in a form so that it sends the user id, time, date, and dataset file name to the get_file.pl script before downloading the dataset.
• In addition to the "list ct datasets" button, add a "back" button that uses javascript to go back in history.
• Get login to work. It will use the same password file as the software archive.

• Add info to, refine and create a second draft of the documentation. Organize all the information into sections, put anchor tags in every section, and create links to then throughout the documentation so that it is esily navigable.

• Create a user login to keep track of downloading. Users have to be registered to download datasets.
• Technical report and citations.
• Write a program that displays the ct scans. It allows the user to change angles of view.
• It lets the user take out some of the layers, thereby making the insides of the object visible.
• The user can look at different densities of the object that was scanned. For example, if you choose to view the bones of a mouse, you can change the permeability of the waves so that they are reflected only off of the bones or you can view what is just beneath the skin by setting the wavelength of the scanner to be bigger.
• Allow people to download this software and provide them with a user's manual.

Notes