mindmeld.active_learning.results_manager module

This module saves the results from the Active Learning Pipeline.

class mindmeld.active_learning.results_manager.ResultsManager(output_folder: str)

Bases: object

Handles the initialization of generated folder and its contents. Keeps record of experiment results.

create_experiment_folder(active_learning_params: Dict, tuning_level: List, classifier_tuning_strategies: List, tagger_tuning_strategies: List)

Creates the active learning experiment folder. :param active_learning_params: Dictionary representation of the params to store. :type active_learning_params: Dict :param tuning_level: The hierarchy levels to tune ("domain" or "intent" and/or "entity") :type tuning_level: list :param classifier_tuning_strategies: List of strategies to use for classifier tuning :type classifier_tuning_strategies: List[str] :param tagger_tuning_strategies: List of strategies to use for tagger tuning :type tagger_tuning_strategies: List[str]

create_folder(unformatted_path)

Creates a folder given an unformatted path. :param unformatted_path: Unformatted path to JSON file. :type unformatted_path: str

dump_json(unformatted_path: str, data: Dict)

Dump data to a JSON file. :param unformatted_path: Unformatted path to JSON file. :type unformatted_path: str :param data: Data to dump. :type data: Dict

format_path(unformatted_path)

Parameters:unformatted_path (str) -- Unformatted path to JSON file. Returns:Path formatted with the experiment folder. Return type:formatted_path (str)

load_json(unformatted_path: str)

Load JSON data from file. If the JSON file doesn't exist, an empty json file is created. :param unformatted_path: Unformatted path to JSON file. :type unformatted_path: str

Returns:Loaded JSON data. Return type:json_data (Dict)

static queries_to_dict(queries: List) → List

Convert a list of ProcessedQueries into a list dictionaries. :param queries: List of ProcessedQuery objects :type queries: List

Returns:

List of queries represented as a dict with the keys

"unannotated_text", "annotated_text", "domain", and "intent".

Return type:query_dicts (List)

set_experiment_folder_name(tuning_level, classifier_tuning_strategies, tagger_tuning_strategies) → str

Parameters:

• tuning_level (list) -- The hierarchy levels to tune ("domain" or "intent" and/or "entity")
• classifier_tuning_strategies (List[str]) -- List of strategies to use for classifier tuning
• tagger_tuning_strategies (List[str]) -- List of strategies to use for tagger tuning

Returns:

Creates the name of the current experiment folder

based on the current timestamp.

Return type:

experiment_folder_name (str)

update_accuracies_json(tuning_type: mindmeld.constants.TuningType, strategy: str, epoch: int, iteration: int, eval_stats)

Update accuracies.json with iteration metrics

update_json(unformatted_path: str, strategy: str, epoch: int, iteration: int, data)

Helper method to update json files. :param unformatted_path: Unformatted path to JSON file. :type unformatted_path: str :param strategy: Current training strategy. :type strategy: str :param epoch: Current epoch. :type epoch: int :param iteration: Current iteration. :type iteration: int :param data: Data to store for current strategy, epoch, and iteration. :type data: Dict or List

update_selected_queries_json(tuning_type: mindmeld.constants.TuningType, strategy: str, epoch: int, iteration: int, queries)

Update accuracies.json with iteration metrics

write_log_selected_queries_json(strategy: str, queries, tuning_type)

Update accuracies.json with iteration metrics

experiment_folder

**Returns* -- experiment_folder (str)* -- Path to the Active Learning experiment folder.