59 engineers on Hacker News describe using Apache Kafka first-hand: 11 endorse it, 5 regret it, 7 are mixed.
“I did a lot of work with Kafka but what we constantly ran into was managing expectations--costs were higher”
mixed · production · lisbbb · 2025-10-29 · source ↗ · wrong label?“We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log”
endorse · production · munksbeer · 2025-08-24 · source ↗ · wrong label?“It proved its quality in practice and with performance tests.”
endorse · production · dlandiak · 2023-12-18 · source ↗ · wrong label?“still used a 6 broker Kafka cluster to keep metadata ... Kafka's compute and storage model doesn't scale too well for the extreme use cases”
mixed · production · kabochya · 2023-08-08 · source ↗ · wrong label?“Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db”
regret · production · victor106 · 2023-03-14 · source ↗ · wrong label?“on Kafka, even though I have used it a bunch of times, I usually have no clue; just restart and pray.”
regret · production · tluyben2 · 2021-12-18 · source ↗ · wrong label?“Amazon managed kafka has been rock solid since we switched to it. When we ran Kafka ourselves though we definitely watched it melt down”
mixed · production · EamonnMR · 2021-11-24 · source ↗ · wrong label?“etcd/kafka/redis (if you can consider those a database) inside docker/crio inside k8s. Works really well.”
endorse · production · dilyevsky · 2021-07-28 · source ↗ · wrong label?“For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues”
regret · production · jpgvm · 2021-05-17 · source ↗ · wrong label?“The 'superpower' Kafka gives you is the ability to turn back time if something goes wrong and the ability to orchestrate really big pipeline.”
endorse · production · EamonnMR · 2020-12-08 · source ↗ · wrong label?“Our cluster of > 15 Kafka nodes and 3 node zk ensemble has had no major issues in over 4 years in production.”
endorse · production · 4 yr · neeleshs · 2018-12-30 · source ↗ · wrong label?“I moved on to Kafka later and didn't regret it a minute.”
endorse · production · endymi0n · 2018-12-27 · source ↗ · wrong label?“I love Kafka, but I can't think of a more painful software to maintain in production.”
mixed · production · root_axis · 2018-12-27 · source ↗ · wrong label?“Kafka as consensus. At the time you had to use an infinite kafka queue since pruning it would throw fabric nodes into a panic”
regret · personal · DIVx0 · 2018-08-06 · source ↗ · wrong label?“in years of running Kafka it was never Kafka's fault.”
endorse · production · Xorlev · 2017-11-02 · source ↗ · wrong label?“we run a relatively small kafka cluster on GCP and it's pretty much hassle-free.”
endorse · production · jurre · 2017-11-01 · source ↗ · wrong label?“We started deploying our Kafka cluster as a set of N EC2 instances but we started running into a bunch of issues... Now that multiple teams are using Kafka”
mixed · production · TheHydroImpulse · 2017-05-31 · source ↗ · wrong label?“Currently, Clojure + Kafka + Apache Storm is the scaleability trifecta.”
endorse · production · unit91 · 2017-03-31 · source ↗ · wrong label?“We run Kafka (and I believe Cassandra) on top of Centos/ZFS in AWS and haven't had any issues so far.”
endorse · production · vlahmot · 2016-12-29 · source ↗ · wrong label?“I am using Protocol Buffers (and evaluating Cap'n Proto) for network serialization and Kafka for buffering/queuing, and overall the library support is quite good.”
endorse · production · olauzon · 2016-12-21 · source ↗ · wrong label?“We're using it at my work but I'm not really in love with it and will probably be trading it out for AMQP”
regret · production · fusiongyro · 2016-09-19 · source ↗ · wrong label?“We attempted to use Kafka as part of a job management system... a long-running task could block an entire partition's worth of work”
mixed · production · Jasper_ · 2016-01-02 · source ↗ · wrong label?“We run a large Kafka cluster on AWS, and it is one of our highest-maintenance services.”
mixed · production · mikebabineau · 2013-11-14 · source ↗ · wrong label?“I also use Kafka on other tech stacks but that's another level completely and use case.”
neutral · production · sgt · 2025-12-10 · source ↗ · wrong label?“consuming from and producing to Kafka among other things”
neutral · production · hnurl · 2025-07-18 · source ↗ · wrong label?“I used to work in a big Kafka shop that you've almost certainly heard of, and they taught me how to deal with the latter.”
neutral · production · nrr · 2024-12-30 · source ↗ · wrong label?“stack traces wrapping Erlang stack traces in a Kafka consumer”
neutral · production · red_hare · 2024-11-19 · source ↗ · wrong label?“Your Kafka messages will only be picked up by your processes.”
neutral · production · pards · 2024-10-06 · source ↗ · wrong label?“We replaced an older kafka system that did lots of transformations to the data making it impossible to source the origin of a field”
neutral · production · fennecbutt · 2024-06-22 · source ↗ · wrong label?“decoupling services by using tools like kafka and debezium”
neutral · production · 10 yr · tkiolp4 · 2023-12-03 · source ↗ · wrong label?“We run these on Kafka or rest api.”
neutral · production · orochimaaru · 2023-10-30 · source ↗ · wrong label?“Eventually, I switched to Kafka and haven't revisited the original solution since.”
neutral · personal · mianos · 2023-09-25 · source ↗ · wrong label?“we aren't doing the "Kafka all the things" model... simply logs go in, logs go out, nothing else”
neutral · production · Spivak · 2023-09-21 · source ↗ · wrong label?“our biggest care and feeding task was adding new event types to our kafka-to-redshift ETL thing”
neutral · production · EamonnMR · 2023-08-07 · source ↗ · wrong label?“We run Kafka with kubernetes, but its so locked down/customized we might as well run it on bare metal.”
neutral · production · javier2 · 2022-07-23 · source ↗ · wrong label?“Cloudflare does not use a managed Kafka service, we run it ourselves.”
neutral · production · mattboyle · 2022-07-20 · source ↗ · wrong label?“the ingestion (happening via a lot of Kafka consumers) throughput was high enough that SQLite started to crumble”
neutral · production · manish_gill · 2022-04-25 · source ↗ · wrong label?“We used kafka mirrormaker 1, with 2 ways sync”
neutral · production · pavelevst · 2022-04-07 · source ↗ · wrong label?“Alfred sits on the socket and dumps message into Kafka. Deli sits on the Kafka queue, stamps messages”
neutral · production · thisrobot · 2021-12-22 · source ↗ · wrong label?“We replaced Kafka with a PostgreSQL table a couple of years ago.”
neutral · production · 2 yr · preetamjinka · 2021-12-18 · source ↗ · wrong label?“E.g. data in Kafka: https://amplitude.engineering/reducing-kafka-costs-with-z-ta...”
neutral · production · paladin314159 · 2021-11-09 · source ↗ · wrong label?“we routinely will spin up dependencies that will run a local kafka and zookeeper for them to run”
neutral · production · sethammons · 2021-09-24 · source ↗ · wrong label?“openzfs/ubuntu in prod for kafka clusters”
neutral · production · anotherhue · 2021-07-29 · source ↗ · wrong label?“I have been using AMQP/Rabbit/Kafka previously (in that order)”
neutral · production · turtlequeue · 2021-05-18 · source ↗ · wrong label?“publish it to Kafka to be consumed by literally hundreds of other services.”
neutral · production · kingnothing · 2020-12-10 · source ↗ · wrong label?“standardised hooks to extract these schemas definitions from each HTTP endpoint or Kafka consumer.”
neutral · production · hcarvalhoalves · 2020-07-23 · source ↗ · wrong label?“we replaced Kafka with Change Streams( little over 7k messages per second)”
neutral · production · truth_seeker · 2020-06-15 · source ↗ · wrong label?“We still have Kafka for our application logs and performance monitoring”
neutral · production · lflux · 2020-03-27 · source ↗ · wrong label?“Models report a finished state over Kafka after getting persisted to buckets in Google cloud”
neutral · production · NegatioN · 2019-11-12 · source ↗ · wrong label?“We run three nodes right now on m5.xlarge instances. At $.42/hr for the managed kafka... I think we'll keep it self hosted for now”
neutral · production · chosenken · 2018-11-29 · source ↗ · wrong label?“I hadn't heard of Pulsar before but just checked it out as we use Kafka.”
neutral · production · aphistic · 2017-12-11 · source ↗ · wrong label?“we're now testing/benchmarking Kafka in our staging environment.”
neutral · personal · caffeineninja · 2017-05-31 · source ↗ · wrong label?“our Data Science team uses Kafka for these cases, but I'm not familiar with the details”
neutral · production · jud_white · 2017-05-31 · source ↗ · wrong label?“We use Kafka as our messaging layer (you can either install on your own Kafka or use our internal)”
neutral · production · capkutay · 2016-12-21 · source ↗ · wrong label?“We run many different types of data services, mysql/hadoop/hbase/kafka/redis/zookeeper/elasticsearch/etc”
neutral · production · paulfurtado · 2016-11-04 · source ↗ · wrong label?“For development environments we are running MSSQL, Elasticsearch, Mongo, Kafka, Zookeeper”
neutral · production · paultyng · 2016-11-01 · source ↗ · wrong label?“We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers”
neutral · production · jgrahamc · 2016-09-09 · source ↗ · wrong label?“We switched because of we was need to use latest Java libs for Kafka.”
neutral · production · magicdream · 2015-07-23 · source ↗ · wrong label?“kafka --> storm --> druid architecture that we rely on”
neutral · production · medriscoll · 2014-04-04 · source ↗ · wrong label?