mindmeld.text_preparation.normalizers module

This module contains Normalizers.

class mindmeld.text_preparation.normalizers.ASCIIFold

Bases: mindmeld.text_preparation.normalizers.Normalizer

An ASCII Folding Normalizer.

fold_char_to_ascii(char)

Return the ASCII character corresponding to the folding token.

Parameters:char -- ASCII folding token Returns:a ASCII character Return type:char

fold_str_to_ascii(text)

Return the ASCII character corresponding to the folding token string.

Parameters:str -- ASCII folding token string Returns:a ASCII character Return type:char

static load_ascii_folding_table()

Load mapping of ascii code points to ascii characters.

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.Lowercase

Bases: mindmeld.text_preparation.normalizers.Normalizer

Lowercase Normalizer Class.

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.NFC

Bases: mindmeld.text_preparation.normalizers.Normalizer

Unicode NFC Normalizer Class. (Canonical Decomposition, followed by Canonical Composition)

For more details: https://unicode.org/reports/tr15/#Norm_Forms

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.NFD

Bases: mindmeld.text_preparation.normalizers.Normalizer

Unicode NFD Normalizer Class. (Canonical Decomposition)

For more details: https://unicode.org/reports/tr15/#Norm_Forms

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.NFKC

Bases: mindmeld.text_preparation.normalizers.Normalizer

Unicode NFKC Normalizer Class. (Compatibility Decomposition, followed by Canonical Composition)

For more details: https://unicode.org/reports/tr15/#Norm_Forms

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.NFKD

Bases: mindmeld.text_preparation.normalizers.Normalizer

Unicode NFKD Normalizer Class. (Compatibility Decomposition)

For more details: https://unicode.org/reports/tr15/#Norm_Forms

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.NoOpNormalizer

Bases: mindmeld.text_preparation.normalizers.Normalizer

A No-Ops Normalizer.

normalize(text)

Parameters:text (str) -- Input text. Returns:Returns the original text. Return type:normalized_text (str)

class mindmeld.text_preparation.normalizers.Normalizer

Bases: abc.ABC

Abstract Normalizer Base Class.

normalize(text)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

tojson()

Method defined to obtain recursive JSON representation of a TextPreparationPipeline.

Parameters:None. -- Returns:JSON representation of Preprocessor (dict) .

class mindmeld.text_preparation.normalizers.NormalizerFactory

Bases: object

Normalizer Factory Class

static get_normalizer(normalizer: str)

A static method to get a Normalizer

Parameters:normalizer (str) -- Name of the desired Normalizer class Returns:Normalizer Class Return type:(Normalizer)

class mindmeld.text_preparation.normalizers.RegexNormalizerRule(pattern: str, replacement: str)

Bases: mindmeld.text_preparation.normalizers.Normalizer

normalize(s)

Parameters:text (str) -- Input text. Returns:Normalized Text. Return type:normalized_text (str)

tojson()

Method defined to obtain recursive JSON representation of a TextPreparationPipeline.

Parameters:None. -- Returns:JSON representation of Preprocessor (dict) .

class mindmeld.text_preparation.normalizers.RegexNormalizerRuleFactory

Bases: object

static get_default_regex_normalizer_rule(regex_normalizer: str)

Creates a RegexNormalizerRule object based on the given rule and the current EXCEPTION_CHARS.

Parameters:regex_normalizer (str) -- Name of the desired RegexNormalizerRule Returns:Default Regex Normalizer Rule Return type:(RegexNormalizerRule)

static get_regex_normalizers(regex_norm_rules)

A static method to get a RegexNormalizerRule from regex_norm_rules.

Parameters:regex_norm_rules (List[Dict], optional) --

Regex normalization rules represented as dictionaries. The example rule below removes any text in parentheses. {

"pattern": "(.+?)", "replacement": ""

}

Returns:

List of RegexNormalizerRule ojects

created from the regex_norm_rules_provided.

Return type:regex_normalizer_rules (List[RegexNormalizerRule])

EXCEPTION_CHARS = "\\@\\[\\]'"