Small Kafka: Nisshi + SQLite on a free t3.micro (AWS Free Tier)
Achieving 7000 msg/s (~6.7MB/s) with just 27MB of memory on a free t3.micro EC2 in AWS
Writing on software development
All of my long-form thoughts on software development, collected in chronological order.
Nisshi SQLite: 48MB/sec, 114ms max latency with 20MB RSS
Using flame graphs to remove a hot path, stop copying data and switching to a fast CRC32
Tuning Nisshi: 600,000 record/s with 13MB of RAM
Writing 585 MB/s with just 13MB of memory usage.
SQLite storage for Nisshi
Using SQLite as a storage engine with Nisshi, a Kafka compatible streaming platform, producing and consuming Protobuf messages using generated test data.
Route, Layer and Process Kafka Messages with Nisshi Services
Learn how Nisshi uses the Service and Layer traits to route, layer and process Apache Kafka messages in a modular and composable way.
Effortlessly Convert Kafka Messages to Apache Parquet with Nisshi: A Step-by-Step Guide
Learn how to use Nisshi to validate and automatically convert Kafka messages into Apache Parquet format, simplifying data processing. In this tutorial, we demonstrate using a Protocol Buffer schema to transform taxi ride data into Parquet files, with support for Apache Avro and JSON schemas as well. Discover how Nisshi integrates with Apache Kafka, supports schema validation, and allows easy configuration with storage engines like S3 and PostgreSQL. Learn how to use tools like DuckDB to query Parquet files, making your data pipeline seamless and efficient.
Smoke Testing with the Bash Automated Testing System 🦇
Nisshi, a Kafka-compatible broker using S3 or PostgreSQL, undergoes automated smoke testing with GitHub workflows and BATS.
Using Nisshi with Tigris on Fly
In this article we deploy Nisshi on Fly using Tigris Data S3 compatible storage.
Apache Kafka protocol with serde, quote, syn and proc_macro2
Using serde, quote, syn and proc_macro2 to implement the Kafka protocol in Rust.