Small Kafka: Nisshi + SQLite on a free t3.micro (AWS Free Tier)
Achieving 7000 msg/s (~6.7MB/s) with just 27MB of memory on a free t3.micro EC2 in AWS
I’m Peter, the founder of nisshi.io, we develop, license and support an Apache Kafka® compatible broker with SQLite, PostgreSQL and S3 storage engines with Apache Parquet, Apache Iceberg and Delta Lake.
Nisshi SQLite: 48MB/sec, 114ms max latency with 20MB RSS
Using flame graphs to remove a hot path, stop copying data and switching to a fast CRC32
Tuning Nisshi: 600,000 record/s with 13MB of RAM
Writing 585 MB/s with just 13MB of memory usage.
SQLite storage for Nisshi
Using SQLite as a storage engine with Nisshi, a Kafka compatible streaming platform, producing and consuming Protobuf messages using generated test data.