Open-Source Data and Machine Learning Software by the Numbers
Ian Hellström | 16 March 2020 | 3 min read
Let’s take a closer look at the more popular open-source tools for data engineering, machine learning, and container orchestration.
While popularity is by no means an indication of quality, it does show towards which technologies the industry converges. That can inform product management, especially with regard to integrations.
To gauge popularity, I shall look at open-source software available on GitHub, and focus on the number of stars on each repository as well as Google searches in the category ‘software’ over the past five years.
Why Google? It has been the world’s leading search engine for at least ten years straight, so it does capture trends fairly accurately.
The chart below shows the number of stars on GitHub over time for various data processing frameworks. It has been produced with Star History.
It’s obvious that Apache Spark leads the pack on GitHub as well as Google.
Scio is Spotify’s open-source Scala API for Apache Beam, so not a data processing framework. In fact, Apache Beam is itself an SDK, for which you choose different runners (e.g. Spark, Flink, or Google Cloud Dataflow).
Spotify’s Luigi has been losing ground to Airflow, which was created by Airbnb, and Argo is trending in Google web searches.
Chronos is a fault-tolerant alternative to cron, which is perhaps not a workflow scheduler in the same sense as Airflow and Luigi. Please note that Pinball is no longer maintained.
TensorFlow rules the machine learning libraries:
When you look on Google Trends, you see that TensorFlow is almost twice as often sought as the next framework: PyTorch. The latter is catching up though.
For databases of all kinds, DB-Engines is the standard popularity meter. What is interesting to note is that certain trends visible in GitHub are not transparent in DB-Engines’ ranking.
PostgreSQL is the open-source leader of relational OLTP databases on GitHub:
Please observe that some projects are mirrored to GitHub (e.g. PostgreSQL and SQLite), so stars and other statistics may not be perfectly representative of the project’s main activity elsewhere.
SQL: OLAP and SQL Engines
Presto leads in the category of data warehouses and SQL engines:
Some entries are (distributed) SQL query engines and not relational databases per se. These include Presto, Hive, Drill, and Impala. Hive, for example, is a SQL engine on top of Hadoop.
When you compare OLTP databases and OLAP engines together, Presto and ClickHouse are ahead of PostgreSQL on GitHub, even though DB-Engines lists these in 26th and 39th position, respectively. Below I list the combined RDBMSs and query engines based on their GitHub stars and contributors (as of this writing). The rank shown is from DB-Engines where available.
The lack of contributors for SQLite is likely due to the fact that the GitHub repository is a mirror.
Google agrees with DB-Engines, which indicates that the community on GitHub is perhaps not entirely representative of popularity in general.
Both DB-Engines and GitHub agree that Redis is the most popular key-value store:
Aerospike does not have nearly as many stars as DB-Engine’s ranking (6th) suggests.
In the category of document-oriented databases, GitHub has Elasticsearch, RethinkDB, and MongoDB as the top 3, whereas DB-Engines sees MongoDB as the current leader:
Note that Elasticsearch is a search engine for documents, which is why DB-Engines has it listed as the most popular search engine rather than a document store.
Wide Column Stores
Cassandra is the queen of wide column stores:
ScyllaDB has been catching up to Cassandra on GitHub although it’s in a stable 8th position according to DB-Engines.
For graph databases, Neo4j is still ahead, although ArangoDB, a multi-model NoSQL database, is gaining in popularity:
Undoubtedly, Kubernetes is captain of the container orchestrators.
Mesos is not a container but rather generic workload orchestrator. Marathon is the direct competition of Kubernetes, but typically runs on Mesos, which is why I have included both.
As can be expected, web searches on Google back up Kubernetes’ supremacy.