FastTextIndexer¶
-
class
ffp.subwords.hash_indexers.FastTextIndexer(n_buckets=2000000, min_n=3, max_n=6)¶ FastTextIndexer
FastTextIndexer is a hash-based subword indexer. It hashes n-grams with (a slightly) FNV-1a variant and maps the hash to a predetermined bucket space.
N-grams can be indexed directly through the __call__ method or all n-grams in a string can be indexed in bulk through the subword_indices method.
-
max_n¶ ‘uint32_t’
- Type
max_n
-
min_n¶ ‘uint32_t’
- Type
min_n
-
n_buckets¶ ‘uint64_t’
- Type
n_buckets
-
subword_indices(self, unicode word, uint64_t offset=0, bool bracket=True, bool with_ngrams=False)¶ Get the subword indices for a word.
- Parameters
word (str) – The string to extract n-grams from
offset (int) – The offset to add to the index, e.g. the length of the word-vocabulary.
bracket (bool) – Toggles bracketing the input string with < and >
with_ngrams (bool) – Toggles returning tuples of (ngram, idx)
- Returns
indices – List of n-gram indices, optionally as (str, int) tuples.
- Return type
- Raises
TypeError – If word is None.
-