Data processing

Links

• Bigslice - System for fast, large-scale, serverless data processing using Go.

• Reflow - Language and runtime for distributed, incremental data processing in the cloud.

• Self-managing serverless computing with Bigmachine (2019)

• Bigslice: a cluster computing system for Go (2019)

• When your data doesn’t fit in memory: the basic techniques (2019) (HN)

• Differential Dataflow - Implementation of differential dataflow using timely dataflow on Rust. (Book) (HN)

• The Log: What every software engineer should know about real-time data's unifying abstraction (2013)

• Luna - Data processing and visualization environment built on a principle that people need an immediate connection to what they are building.

• Guide To The Data Lake — Modern Batch Data Warehousing (2020)

• Plumbing At Scale (2020) - Event Sourcing and Stream Processing Pipelines at Grab.

• Differential Dataflow! But at what COST? (2017) (HN)

• Timely Dataflow and Total Order (2020)

• Nuclio - High-Performance Serverless event and data processing platform.

• Apache Spark - Unified analytics engine for large-scale data processing. (PySpark) (PySpark Style Guide) (Article)

• Spark: The Definitive Guide Book (2018) (Code)

• Batch - Event replay platform. Version control for data passing through your messaging systems. (HN)

• A log/event processing pipeline you can't have (2019) (HN)

• mm-ADT - Multi-Model Abstract Data Type. Distributed virtual machine capable of integrating a diverse collection of data processing technologies. (Code)

• Data Preprocessing in Machine Learning (2020)

• lakeFS - Open source layer that delivers resilience and manageability to object-storage based data lakes. (Web)

• Baker - High performance, composable and extendable data-processing pipeline for the big data era.

• Cylon - Fast, scalable distributed memory data parallel library for processing structured data. (Web)

• cuGraph - GPU Graph Analytics.

• Opaque - Secure Apache Spark SQL.

• Apache Beam - Unified programming model for Batch and Streaming. (Web)

• Stitch - Simple, extensible ETL built for data teams.

• Databricks - Unified Data Analytics. (GitHub) (CLI)

• AugMix - Simple Data Processing Method to Improve Robustness and Uncertainty.

• Snapflow - Framework for building end-to-end functional data pipelines from modular components.

• Workflow Description Language (WDL) - Way to specify data processing workflows with a human-readable and writeable syntax.

• Cloudfuse - Open source serverless data solutions. Future of data pipelines. (GitHub)

• Create your own data stream for Kafka with Python and Faker (2021)