mindmeld.models.evaluation module

This module contains base classes for models defined in the models subpackage.

class mindmeld.models.evaluation.EntityModelEvaluation(config, results)

Bases: mindmeld.models.evaluation.SequenceModelEvaluation

Generates some statistics specific to entity recognition

get_stats()

Prints model evaluation stats in a table to stdout

print_stats()

Prints model evaluation stats to stdout

class mindmeld.models.evaluation.EvaluatedExample

Bases: mindmeld.models.evaluation.EvaluatedExample

Represents the evaluation of a single example

example

The example being evaluated

expected

The expected label for the example

predicted

The predicted label for the example

proba

dict -- Maps labels to their predicted probabilities

label_type

str -- One of CLASS_LABEL_TYPE or ENTITIES_LABEL_TYPE

is_correct

class mindmeld.models.evaluation.ModelEvaluation(config, results)

Bases: mindmeld.models.evaluation.ModelEvaluation

Represents the evaluation of a model at a specific configuration using a collection of examples and labels.

config

ModelConfig -- The model config used during evaluation.

results

list of EvaluatedExample -- A list of the evaluated examples.

correct_results()

Returns:Collection of the examples which were correct Return type:iterable

get_accuracy()

The accuracy represents the share of examples whose predicted labels exactly matched their expected labels.

Returns:The accuracy of the model. Return type:float

get_stats()

Returns a structured stats object for evaluation.

Returns:Structured dict containing evaluation statistics. Contains precision, recall, f scores, support, etc. Return type:dict

incorrect_results()

Returns:Collection of the examples which were incorrect Return type:iterable

print_stats()

Prints a useful stats table for evaluation.

Returns:Structured dict containing evaluation statistics. Contains precision, recall, f scores, support, etc. Return type:dict

raw_results()

Exposes raw vectors of expected and predicted for data scientists to use for any additional evaluation metrics or to generate graphs of their choice.

Returns:tuple containing:

• NamedTuple: RawResults named tuple containing
• expected: vector of predicted classes (numeric value)
• predicted: vector of gold classes (numeric value)
• text_labels: a list of all the text label values, the index of the text label in
• this array is the numeric label

Return type:(tuple)

class mindmeld.models.evaluation.RawResults(predicted, expected, text_labels, predicted_flat=None, expected_flat=None)

Bases: object

Represents the raw results of a set of evaluated examples. Useful for generating stats and graphs.

predicted

list -- A list of predictions. For sequences this is a list of lists, and for standard classifieris this is a 1d array. All classes are in their numeric representations for ease of use with evaluation libraries and graphing.

expected

list -- Same as predicted but contains the true or gold values.

text_labels

list -- A list of all the text label values, the index of the text label in this array is the numeric label

predicted_flat

list -- (Optional): For sequence models this is a flattened list of all predicted tags (1d array)

expected_flat

list -- (Optional): For sequence models this is a flattened list of all gold tags

class mindmeld.models.evaluation.SequenceModelEvaluation(config, results)

Bases: mindmeld.models.evaluation.ModelEvaluation

get_stats()

Prints model evaluation stats in a table to stdout

print_stats()

Prints model evaluation stats to stdout

raw_results()

Returns the raw results of the model evaluation

class mindmeld.models.evaluation.StandardModelEvaluation(config, results)

Bases: mindmeld.models.evaluation.ModelEvaluation

get_stats()

Prints model evaluation stats in a table to stdout

print_stats()

Prints model evaluation stats to stdout

raw_results()

Returns the raw results of the model evaluation