To Annotate for the Adverse Drug Event Detection for Pharmacovigilance project you will need to access Electronic Health Records (EHR) data. Since the records are from patients and contain personally identifiable information, you must complete CITI training and fill out user agreements. 

Marking or annotating the records requires a clear understanding of what needs to be marked and what relationships need to be made. The Annotation Guidelines is a living document and necessary resource to lead you through the process and provide useful examples.

  • Annotation Guidelines :
    •  This work is licensed under a Creative Commons Attribution 4.0 license.

      Recommended Citation:

          Belknap, S., Freund, E., Frid, N., Granillo, E., Keating, H., Li, Z., Prasad, R., Ramesh, B., Wang, V., Yu, H. (Oct. 29, 2015).The Annotation Guideline Manual: Extracting Adverse Drug Event Information from Clinical Narratives in electrical Medical Records [PDF file]. Ver. 2.1. Retrieved from

  • Most recent Guidelines changes
  • Annotation of PHI (also in Guidelines)
  • Annotation Work flow
  • Standards: Naranjo, CTCAE, MedDRA (not publicly available)

You will also use an annotation tool. Below are hyperlinks to videos demonstrating the use of the Knowtator Annotation Tool from the user's perspective. 'Knowtator is a general-purpose text annotation tool that is integrated with the Protégé knowledge representation system. Knowtator facilitates the manual creation of training and evaluation corpora for a variety of biomedical language processing tasks.'

Click the links to veiw very short tutorials. These are listed in the order they are best viewed.

To review the Videos on Mac: Download and install Windows Media Player for Mac. To view the WMV file, just launch the media player, select "File," "Open," choose the WMV file and click "OK" to load the movie. Click the "Play" button on the bottom of the screen and the video file begins to play.

Video 7:  can only viewed with Quick time player. It has low resolution and we will replace it soon.


Related Resources


Patient access to physicians' notes has the potential to improve understanding, medication adherence and healthcare outcomes. However, EHR notes contain medical jargon that may be difficult for patients to comprehend. To improve patients’ EHR note comprehension, we have developed NoteAid, a biomedical natural language processing system which translates medical jargon into consumer-oriented lay language.

We are working to improve the medical lay language terminology database for NoteAid. Medical synonym annotation is a part of such process. This annotation task is to judge whether the candidate terms found by an automatic system is a synonym, related term, or irrelevant term for a target medical term. The Medical Synonym Annotation Guideline provides rules and definitions related to this annotation task.


The Federal Government has devised rules for documents written for the lay people. Lay translation of medical passages made by NoteAid also follows these rules. The Excerpted Medical Plain-Language Guidelines adapted from the Federal Plain Language Guidelines shows you the key points in medical plain language writings.



Note: All guideline and manuals published on this page are licensed under a Creative Commons Attribution 4.0 license.Creative Commons License


The ADE pharmacovigiliance projects draws from publicly available standards and resources:

 CTCAE (Common Terminology Criteria for Adverse Events) is NCI's standardized classification of adverse effects of drugs used in cancer therapy. The general guideline is:

1 - Mild
2 - Moderate
3 - Severe
4 - Life-threatening
5 - Death

 MedDRA (Medical Dictionary for Regulatory Activities) is a medical terminology used by regulatory authorities and it is the adverse event classification dictionary endorsed by the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH). We are using the current version (17) in the annotation tool. To access from:

Naranjo Scoring is used to for estimation of the probability that a drug caused an adverse clinical event.