de.fu_berlin.ties.extract
Class EvaluatedExtractionContainer

java.lang.Object
  extended by de.fu_berlin.ties.extract.ExtractionContainer
      extended by de.fu_berlin.ties.extract.EvaluatedExtractionContainer
All Implemented Interfaces:
RestorableContainer, StorableContainer

public class EvaluatedExtractionContainer
extends ExtractionContainer

An extraction container that evaluates containers of predicted extractions against containers of true extractions (answer keys) and merges their contents, setting the evaluation states accordingly. Extractions in both containers are marked as EvalStatus.CORRECT (true positives), extractions found only in the container of predictions are marked as EvalStatus.SPURIOUS (false positives), while those occurring only in the answer keys are EvalStatus.MISSING (false negatives).

Two match modes are supported: in match-all mode, all extractions are evaluated. In match-best mode, only the recognition with the highest probability of each type is evaluated (in each batch), while the others are ignored and not counted; it is thus suffienct to match (or fail to match) a single answer key -- the other answer keys are marked as EvalStatus.ALTERNATIVEs and not counted.

Version:
$Revision: 1.13 $, $Date: 2006/10/21 16:04:13 $, $Author: siefkes $
Author:
Christian Siefkes

Field Summary
static String CONFIG_MATCH_ALL
          Configuration key: whether to use match-all or match-best as match mode.
static String CONFIG_MATCH_POSITION
          Configuration key for isMatchingPosition().
 
Constructor Summary
EvaluatedExtractionContainer(TargetStructure targetStruct, boolean matchAll, boolean matchPosition)
          Creates a new instance.
EvaluatedExtractionContainer(TargetStructure targetStruct, TiesConfiguration config)
          Creates a new instance.
 
Method Summary
 void add(Extraction extraction)
          Adds an extraction to this container.
protected  void addAllAndClear(Collection extractionColl, EvalStatus status, String source)
          Helper method that adds a bunch of extractions, setting their evaluation status and source as specified.
 void evaluateBatch(ExtractionContainer predicted, ExtractionContainer expected, String source)
          Evaluates a container of predicted extractions against a container of true extractions (answer keys) and adds them to this instance.
 boolean isMatchingAll()
          Returns the match mode.
 boolean isMatchingPosition()
          If true, the positions of extraction and answer keys must match; otherwise only their contents must match (string compare).
 String toString()
          Returns a string representation of this object.
 MultiFMetricsView viewMetrics()
          Returns a read-only view of the counts of true positives (correct extractions), false negatives (missing extractions) and false positives (spurious extractions) stored in this container of each type as well as for all types; and the metrics that can be calculated from these values.
 
Methods inherited from class de.fu_berlin.ties.extract.ExtractionContainer
addAll, filterOverlapping, getTargetStructure, iterator, iterator, last, lastN, lastN, remove, removeAll, removeLast, restoreEntries, size, storeEntries, unsetPositions
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

CONFIG_MATCH_ALL

public static final String CONFIG_MATCH_ALL
Configuration key: whether to use match-all or match-best as match mode.

See Also:
Constant Field Values

CONFIG_MATCH_POSITION

public static final String CONFIG_MATCH_POSITION
Configuration key for isMatchingPosition().

See Also:
Constant Field Values
Constructor Detail

EvaluatedExtractionContainer

public EvaluatedExtractionContainer(TargetStructure targetStruct,
                                    TiesConfiguration config)
Creates a new instance.

Parameters:
targetStruct - the target structure specifying the classes to recognize
config - used to configure this instance

EvaluatedExtractionContainer

public EvaluatedExtractionContainer(TargetStructure targetStruct,
                                    boolean matchAll,
                                    boolean matchPosition)
Creates a new instance.

Parameters:
targetStruct - the target structure specifying the classes to recognize
matchAll - sets the match mode: true for match-all mode, false for match-best mode
matchPosition - if true, the positions of extraction and answer keys must match; otherwise only their contents must match (string compare)
Method Detail

add

public void add(Extraction extraction)
         throws IllegalArgumentException
Adds an extraction to this container. Only evaluated extractions can be added to this container, i.e., the evaluation states EvalStatus.UNKNOWN and EvalStatus.TRUTH are not allowed. The evaluation states EvalStatus.IGNORED and EvalStatus.ALTERNATIVE are only allowed in match-best mode; in match-all mode (isMatchingAll() == true) they don't make sense and are thus rejected.

Overrides:
add in class ExtractionContainer
Parameters:
extraction - the extraction to add
Throws:
IllegalArgumentException - if the extraction's evaluation status is invalid (cf. above); or if the class of the extraction is not in the set of classes accepted by this container

addAllAndClear

protected void addAllAndClear(Collection extractionColl,
                              EvalStatus status,
                              String source)
Helper method that adds a bunch of extractions, setting their evaluation status and source as specified.

Parameters:
extractionColl - a collection of Extractions to add; the collection will be cleared by this method, i.e., it will be empty after the method returned
status - the status of all extractions in the collection will be set to this value prior to adding them
source - the source to set for all extractions in the collection; ignored if null

evaluateBatch

public void evaluateBatch(ExtractionContainer predicted,
                          ExtractionContainer expected,
                          String source)
                   throws IllegalArgumentException
Evaluates a container of predicted extractions against a container of true extractions (answer keys) and adds them to this instance. In match-best mode, only the extraction with the highest pR resp. probability of each type in this batch is evaluated.

Warning: If positions are ignored for matching (see constructor), the two given containers are modified by unsetting the positions of all extractions and removing duplicates (extractions with same type + text but different positions).

Parameters:
predicted - the container of predicted extractions
expected - the container of expected extractions (answer keys)
source - the source to add to add extractions of this batch; ignored if null
Throws:
IllegalArgumentException - if the target structures of the two containers differ from the target structure of this one

isMatchingAll

public boolean isMatchingAll()
Returns the match mode.

Returns:
true for match-all mode, false for match-best mode

isMatchingPosition

public boolean isMatchingPosition()
If true, the positions of extraction and answer keys must match; otherwise only their contents must match (string compare).

Returns:
the value of the attribute

toString

public String toString()
Returns a string representation of this object.

Overrides:
toString in class ExtractionContainer
Returns:
a textual representation

viewMetrics

public MultiFMetricsView viewMetrics()
Returns a read-only view of the counts of true positives (correct extractions), false negatives (missing extractions) and false positives (spurious extractions) stored in this container of each type as well as for all types; and the metrics that can be calculated from these values. This is not a snapshot but will change whenever the container is changed.

Returns:
a view of the counts and evaluation metrics


Copyright © 2003-2007 Christian Siefkes. All Rights Reserved.