Awesome Big Data
A curated list of awesome big data frameworks, resources and other awesomeness. Inspired byawesome-php, awesome-python, awesome-ruby, hadoopecosystemtable & big-data.
Your contributions are always welcome!
- Apache Hadoop – framework for distributed processing. Integrates MapReduce (parallel processing)， YARN (job scheduling) and HDFS (distributed file system).
- AddThis Hydra – distributed data processing and storage system originally developed at AddThis.
- AMPLab SIMR – run Spark on Hadoop MapReduce v1.
- Apache Crunch – a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce.
- Apache DataFu – collection of user-defined functions for Hadoop and Pig developed by LinkedIn.
- Apache Flink – high-performance runtime, and automatic program optimization.
- Apache Gora – framework for in-memory data model and persistence.
- Apache Hama – BSP (Bulk Synchronous Parallel) computing framework.
- Apache MapReduce – programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
- Apache Pig – high level language to express data analysis programs for Hadoop.
- Apache S4 – framework for stream processing, implementation of S4.
- Apache Spark – framework for in-memory cluster computing.
- Apache Spark Streaming – framework for stream processing, part of Spark.
- Apache Storm – framework for stream processing by Twitter also on YARN.
- Apache Tez – application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN.
- Apache Twill – abstraction over YARN that reduces the complexity of developing distributed applications.
- Cascalog – data processing and querying library.
- Cheetah – High Performance, Custom Data Warehouse on Top of MapReduce.
- Concurrent Cascading – framework for data management/analytics on Hadoop.
- Damballa Parkour – MapReduce library for Clojure.
- Datasalt Pangool – alternative MapReduce paradigm.
- DataTorrent StrAM – real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
- Facebook Corona – Hadoop enhancement which removes single point of failure.
- Facebook Peregrine – Map Reduce framework.
- Facebook Scuba – distributed in-memory datastore.
- Google Dataflow – create data pipelines to help them?ingest, transform and analyze data.
- Google MapReduce – map reduce framework.
- Google MillWheel – fault tolerant stream processing framework.
- JAQL – declarative programming language for working with structured, semi-structured and unstructured data.
- Kite – is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
- Metamarkers Druid – framework for real-time analysis of large datasets.
- Netflix PigPen – map-reduce for Clojure whiche compiles to Apache Pig.
- Nokia Disco – MapReduce framework developed by Nokia.
- Pinterest Pinlater – asynchronous job execution system.
- Pydoop – Python MapReduce and HDFS API for Hadoop.
- Stratosphere – general purpose cluster computing framework.
- Streamdrill – usefull for counting activities of event streams over different time windows and finding the most active one.
- Twitter Scalding – Scala library for Map Reduce jobs, built on Cascading.
- Twitter Summingbird – Streaming MapReduce with Scalding and Storm, by Twitter.
- Twitter TSAR – TimeSeries AggregatoR by Twitter.
Document Data Model
- Actian Versant – commercial object-oriented database management systems .
- Crate Data – is an open source massively scalable data store. It requires zero administration.
- Facebook Apollo – Facebook’s Paxos-like NoSQL database.
- jumboDB – document oriented datastore over Hadoop.
- LinkedIn Espresso – horizontally scalable document-oriented NoSQL data store.
- MarkLogic – Schema-agnostic Enterprise NoSQL database technology.
- MongoDB – Document-oriented database system.
- RavenDB – A transactional, open-source Document Database.
- RethinkDB – document database that supports queries like table joins and group by.
Key Map Data Model
Note: There is some term confusion in the industry， and two different things are called “Columnar Databases”. Some, listed here, are distributed, persistent databases built around the “key-map” data model: all data has a (possibly composite) key， with which a map of key-value pairs is associated. In some systems, multiple such value maps can be associated with a key， and these maps are referred to as “column families” (with value map keys being referred to as “columns”).
Another group of technologies that can also be called “columnar databases” is distinguished by how it stores data, on disk or in memory — rather than storing data the traditional way， where all column values for a given key are stored next to each other, “row by row”， these systems store all columnvalues next to each other. So more work is needed to get all columns for a given key， but less work is needed to get all values for a given column.
The former group is referred to as “key map data model” here. The line between these and the Key-value Data Model stores is fairly blurry.
The latter, being more about the storage format than about the data model, is listed under Columnar Databases.
You can read more about this distinction on Prof. Daniel Abadi’s blog: Distinguishing two major types of Column Stores.
- Apache Accumulo – distribuited key/value store, built on Hadoop.
- Apache Cassandra – column-oriented distribuited datastore, inspired by BigTable.
- Apache HBase – column-oriented distribuited datastore, inspired by BigTable.
- Facebook HydraBase – evolution of HBase made by Facebook.
- Google BigTable – column-oriented distributed datastore.
- Google Cloud Datastore – is a fully managed, schemaless database for storing non-relational data over BigTable.
- Hypertable – column-oriented distribuited datastore, inspired by BigTable.
- InfiniDB – is accessed through a MySQL interface and use massive parallel processing to parallelize queries.
- OhmData C5 – improved version of HBase.
- Tephra – Transactions for HBase.
- Twitter Manhattan – real-time, multi-tenant distributed database for Twitter scale.
Key-value Data Model
- Aerospike – NoSQL flash-optimized, in-memory. Open source and “Server code in ‘C’ (not Java or Erlang