Welcome to Mofka’s documentation!#
Mofka is a streaming system for high-performance computing platforms and applications. It is loosely inspired by cloud streaming systems like Kafka, but is implemented using Mochi, a set of tools and methodologies for building highly-efficient HPC data services. As such, it benefits from high-speed HPC networks with the Mercury RPC and RDMA library and a high level of on-node concurrency using Argobots.
Mofka provides a C++ and a Python interface. One of its particularities is that it splits events into two parts: a data part, referencing potentially large, raw data, which Mofka will try its best not to copy more than necessary (e.g., by relying on RDMA to transfer it directly from a client application’s memory to a storage device on servers) and a metadata part, which consists of structured information about the data (usually expressed in JSON). Doing so allows Mofka to store each part independently, batch metadata together, and allow an event to reference (a subset of) the data of another event. This interface is also often more adapted to HPC applications, which manipulate large datasets and their metadata.