Welcome to Mofka’s documentation!

Mofka is a streaming system for high-performance computing platforms and applications. It is loosely inspired by cloud streaming systems like Kafka, but is implemented using Mochi, a set of tools and methodologies for building highly-efficient HPC data services. As such, it benefits from high-speed HPC networks with the Mercury RPC and RDMA library and a high level of on-node concurrency using Argobots.

Mofka follows the Diaspora Streaming API, a C++ and Python API for using and developing streaming systems for HPC. Client applications (producers and consumers) using Mofka should refer to its documentation. The present documentation only covers the configuration of the Mofka engine and the use of the Mofka driver for the Diaspora Stream API.

Contents:

• Installing Mofka
  • Installing in a Spack environment
• Getting started
  • Deploying Mofka
  • Interacting with Mofka with diaspora-ctl
  • Creating a topic and a partition
  • Shutting down Mofka
• Advanced deployments
  • Making the master database persistent
  • Running multiple processes
  • Creating persistent partitions
  • More configuration
  • Setting up multithreading
  • Improving I/O performance
  • Full configuration
• Default partition manager — architecture
  • Overview
  • On-disk layout
  • Write path — producer to disk
  • Read path — feeding a consumer
  • Read path — random access (getData)
  • Caches and buffer pools
  • Tuning notes

Indices and tables

• Index

• Module Index

• Search Page