NdArray¶

class ffp.storage.ndarray.NdArray(array: numpy.ndarray)[source]¶

Bases: numpy.ndarray, ffp.io.Chunk, ffp.storage.storage.Storage

Array storage.

Essentially a numpy matrix, either in-memory or memory-mapped.

Examples

>>> matrix = np.float32(np.random.rand(10, 50))
>>> ndarray_storage = NdArray(matrix)
>>> np.allclose(matrix, ndarray_storage)
True
>>> ndarray_storage.shape
(10, 50)

static __new__(cls, array: numpy.ndarray)[source]¶

Construct a new NdArray storage.

Parameters: array (numpy.ndarray) – The storage buffer.
Raises: TypeError – If the array is not a 2-dimensional float32 array.

property shape¶

The storage shape

Returns: (rows, cols) – Tuple with storage dimensions
Return type: Tuple[int, int]

classmethod load(file: BinaryIO, mmap=False) → ffp.storage.ndarray.NdArray [source]¶

Load Storage from the given finalfusion file.

Parameters

file (BinaryIO) – File at the beginning of a finalfusion storage
mmap (bool) – Toggles memory mapping the buffer.

Returns

storage – The storage from the file.

Return type

Storage

static read_chunk(file: BinaryIO) → ffp.storage.ndarray.NdArray [source]¶

Read the Chunk and return it.

The file must be positioned before the contents of the Chunk but after its header.

Parameters: file (BinaryIO) – a finalfusion file containing the given Chunk
Returns: chunk – The chunk read from the file.
Return type: Chunk

static mmap_storage(file: BinaryIO) → ffp.storage.ndarray.NdArray [source]¶

Memory map the storage.

Parallel method to ffp.io.Chunk.read_chunk(). Instead of storing the Storage in-memory, it memory maps the embeddings.

Parameters: file (BinaryIO) – File at the beginning of a finalfusion storage
Returns: storage – The memory mapped storage.
Return type: Storage

static chunk_identifier()[source]¶

Get the ChunkIdentifier for this Chunk.

Returns: chunk_identifier
Return type: ChunkIdentifier

write_chunk(file: BinaryIO)[source]¶

Write the Chunk to a file.

Parameters: file (BinaryIO) – Output file for the Chunk

ffp.storage.ndarray.load_ndarray(file: Union[str, bytes, int, os.PathLike], mmap: bool = False) → ffp.storage.ndarray.NdArray [source]¶

Load an array chunk from the given file.

Parameters

file (str, bytes, int, PathLike) – Finalfusion file with a ndarray chunk.
mmap (bool) – Toggles memory mapping the array buffer as read only.

Returns

storage – The NdArray storage from the file.

Return type

NdArray

Raises

ValueError – If the file did not contain an NdArray chunk.