26 engineers on Hacker News describe using Apache Spark first-hand: 3 endorse it, 2 regret it, 2 are mixed.
“we run 20k Spark jobs a day in the cloud, and it is incredibly wasteful to start whole Spark instance to touch ten megabyte csv file”
mixed · production · glogla · 2022-12-28 · source ↗ · wrong label?“Then we moved to Spark and hit JVM GC slowdowns on the amount of allocated memory.”
regret · production · marcinzm · 2022-08-02 · source ↗ · wrong label?“I've used Spark in prod. Worked great and was the highest QPS service in the company I worked at.”
endorse · production · gravypod · 2019-06-13 · source ↗ · wrong label?“We run 30 node, autoscaling clusters which stay close to 100% for most of the time.”
endorse · production · 4 yr · threeseed · 2018-07-10 · source ↗ · wrong label?“We run a Spark cluster for distributed processing. Spark is awesome - makes it dead easy to parallelize work.”
endorse · production · cldellow · 2017-09-12 · source ↗ · wrong label?“Last time I tinkered with Spark (this was over a year ago) it was so buggy, next to useless, but perhaps things have changed.”
regret · personal · gtrubetskoy · 2014-10-10 · source ↗ · wrong label?“Spark is very memory sensitive compared to hadoop, so using a columnar store and only pulling out the data that you absolutely need goes a very long way.”
mixed · production · subprotocol · 2014-05-30 · source ↗ · wrong label?“the code was written by idiots in "Spark/Scala" (their favorite buzzword”
neutral · production · lisbbb · 2025-10-29 · source ↗ · wrong label?“We have clickhouse, spark, trino/dremio, starrocks, druid, apache drill, and apache datafusion.”
neutral · production · Epa095 · 2024-12-22 · source ↗ · wrong label?“we run spark jobs and ray jobs on distributed clusters”
neutral · production · bluecoconut · 2023-08-04 · source ↗ · wrong label?“Ganglia was okay for monitoring Spark jobs running in a Yarn cluster”
neutral · production · EdwardDiego · 2023-06-27 · source ↗ · wrong label?“Wrote and debugged a lot of R and Python, fair bit of Spark too.”
neutral · production · FridgeSeal · 2022-12-22 · source ↗ · wrong label?“at least I can run 3 different spark pipelines and 4 Python applications without needing to dig into the specifics”
neutral · production · FridgeSeal · 2022-08-01 · source ↗ · wrong label?“a few years of data engineering jobs around Spark and ML”
neutral · production · 3 yr · 62951413 · 2021-12-25 · source ↗ · wrong label?“At work we are batch-processing a decent amount of data every night using Spark.”
neutral · production · InjectionVE · 2021-08-21 · source ↗ · wrong label?“Companies where I've seen it were typically targeting Spark as the execution environment”
neutral · production · legerdemain · 2020-12-24 · source ↗ · wrong label?“Back a few years ago (2017 I think) another dev and I took a Spark based system that was taking 3+ days to do aggregate processing over HBase-stored data.”
neutral · production · nl · 2020-09-21 · source ↗ · wrong label?“usually the solution looks something like Spark, all to do something that would have been sufficiently performant”
neutral · production · throwaway894345 · 2020-08-13 · source ↗ · wrong label?“Spark is used to fetch data for the most part.”
neutral · production · NegatioN · 2019-11-12 · source ↗ · wrong label?“We use HBase as our primary store and index via map/reduce or Spark Batch.”
neutral · production · KennyCason · 2018-12-01 · source ↗ · wrong label?“the majority of the comparisons/benchmarks we've run have been against MemSQL+Spark and Cassandra+Spark”
neutral · production · plamb · 2017-10-09 · source ↗ · wrong label?“We run all of our Spark jobs in job-scoped Cloud Dataproc clusters.”
neutral · production · natekupp · 2017-06-06 · source ↗ · wrong label?“I picked this book once I started working on the implementation of my project into production - we use Apache Spark (Scala) at work.”
neutral · production · gghyslain · 2016-12-22 · source ↗ · wrong label?“We use Apache Spark with a mixture of different algorithms to analyze over 1.5b flight prices”
neutral · production · dribel · 2015-07-08 · source ↗ · wrong label?“data analysis (Apache Spark, using flambo (https://github.com/yieldbot/flambo).”
neutral · production · loevborg · 2014-11-04 · source ↗ · wrong label?