NdArray¶
-
class
ffp.storage.ndarray.
NdArray
(array: numpy.ndarray)[source]¶ Bases:
numpy.ndarray
,ffp.io.Chunk
,ffp.storage.storage.Storage
Array storage.
Essentially a numpy matrix, either in-memory or memory-mapped.
Examples
>>> matrix = np.float32(np.random.rand(10, 50)) >>> ndarray_storage = NdArray(matrix) >>> np.allclose(matrix, ndarray_storage) True >>> ndarray_storage.shape (10, 50)
-
static
__new__
(cls, array: numpy.ndarray)[source]¶ Construct a new NdArray storage.
- Parameters
array (numpy.ndarray) – The storage buffer.
- Raises
TypeError – If the array is not a 2-dimensional float32 array.
-
property
shape
¶ The storage shape
-
classmethod
load
(file: BinaryIO, mmap=False) → ffp.storage.ndarray.NdArray[source]¶ Load Storage from the given finalfusion file.
- Parameters
file (BinaryIO) – File at the beginning of a finalfusion storage
mmap (bool) – Toggles memory mapping the buffer.
- Returns
storage – The storage from the file.
- Return type
-
static
read_chunk
(file: BinaryIO) → ffp.storage.ndarray.NdArray[source]¶ Read the Chunk and return it.
The file must be positioned before the contents of the
Chunk
but after its header.- Parameters
file (BinaryIO) – a finalfusion file containing the given Chunk
- Returns
chunk – The chunk read from the file.
- Return type
-
static
mmap_storage
(file: BinaryIO) → ffp.storage.ndarray.NdArray[source]¶ Memory map the storage.
Parallel method to
ffp.io.Chunk.read_chunk()
. Instead of storing theStorage
in-memory, it memory maps the embeddings.- Parameters
file (BinaryIO) – File at the beginning of a finalfusion storage
- Returns
storage – The memory mapped storage.
- Return type
-
static
-
ffp.storage.ndarray.
load_ndarray
(file: Union[str, bytes, int, os.PathLike], mmap: bool = False) → ffp.storage.ndarray.NdArray[source]¶ Load an array chunk from the given file.
- Parameters
file (str, bytes, int, PathLike) – Finalfusion file with a ndarray chunk.
mmap (bool) – Toggles memory mapping the array buffer as read only.
- Returns
storage – The NdArray storage from the file.
- Return type
- Raises
ValueError – If the file did not contain an NdArray chunk.