Parsing literature searches

Rob MacLeod

Here I describe a program (a combination of shell and awk scripts actually) called parseref that converts the output of a literature search using PubMed into something BibTEX can understand. To run this requires support for the Bourne shell and the awk program, included automatically for Unix and the Mac OSX but available as a utility for Windoze. In fact, you will need the GNU awk program, called gawk on many systems, for some of the functions in the program.

1 The code and installation

Right click (control-click on the Mac) on the links below to download each of the files:

To work properly, both the shell and awk files must be in the same directory as the files you wish to convert, or they must at least be available in the PATH (or whatever it is called in your operating system). My preference is to place both the shell script and the awk file into my /bin directory, which is almost always in the PATH. Then I make a symbolic link pointing parseref to, so that execution requires only typing parseref.

2 Using parseref

At present, works only for searches of the PubMed version of Medline. If there is demand for other search engines, please let me know and I can whip something up. CPX was an old search engine that seems to have disappeared so if you find a new link to it, also please let me know. Sadly, there is a limit to the search engines freely available even for scientists.

Here are the steps required to do the parsing.

2.1 The Search

  1. Go to PubMed to find the reference(s) you want to retrieve for your BibTEX database then carry out the search and identify the references to be converted.
  2. In PubMed, look for the menu item that usually says ``SUMMARY''-it is next to the ``Display'' button. From this menu, select the option ``MEDLINE'' and then hit ``Display''. The result will be a detailed listing of all the elements of the reference citations, not meant for human reading, but great for computer programs.
  3. Highlight the part of this listing that includes the reference(s) you want to keep and then copy and paste it into a standard text editor window (e.g., emacs).
  4. Repeat these steps until you are finished saving all the references you want to keep; then save the file (I like using an extension of .pubmed so I know what the file contains.

2.2 The Parsing

  1. Run the program parseref on the file you have saved as follows:
      [-l] [-i ABC] infilename
    where, the -l option controls the formatting of authors' names--the default is Initials Last-name, e.g., ``R.S. MacLeod'' but for the -l option, we use Last-name, Initials, e.g., ``MacLeod, R.S.''
    -i ABC is how to set the initials that begin the key of each converted entry. Our local convention is each key begin with the initials of the owner of the BibTEX file. If you do not use the -i ABC option, all keys will begin with ``RSM'', my initials.

  2. If you forget the arguments or syntax, just type parseref by itself and you will be some help.

  3. The result of this operation should be a file with .bib extension with the same base filename as your input file. This should be in BibTEX format and so will fit into your existing file.

You can also run the awk script directly as follows if you have problems with the Unix shell script (include entire command on the same line):

    gawk -f parsepubmed.awk [initials=ABC ] [authortype=lastnamefirst] 
      infilename.pubmed >> outfilename.bib
where infilename.pubmed and outfilename.bib are the input and output files, respectively.

Note that the output file may also contain some string variables that we use a lot to ensure consistent listings for journal names. The current set of such strings is as follows:

  @String{j-BME = "IEEE Trans Biomed Eng"}
  @String{j-CR = "Circ Res"}
  @String{j-C = "Circulation"}
  @String{j-AJP = "Am J Physiol"}
  @String{j-ABE = "Ann Biomed Eng"}
  @String{j-JE = "J Electrocardiol"}

These will appear at the end of the output file and you can either use them or replace the journal variables with whatever strings you prefer.

About this document ...

Parsing literature searches

This document was generated using the LaTeX2HTML translator Version 2002-2-1 (1.71)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -split 3 -no_white -link 3 -no_navigation -no_math -html_version 3.2,math -show_section_numbers -local_icons parseref

The translation was initiated by Rob Macleod on 2009-10-10

Rob Macleod 2009-10-10