Tl;dr: This blog post describes how we developed an efficient and reliable Python ecosystem using Pants, an open source build system, and addresses Coinbase manages Python applications at scale.
by Coinbase Computing Platform team
Python is one of the most used programming languages for data scientists, machine learning practitioners and blockchain researchers at Coinbase. Over the past few years, we have witnessed the growth of Python applications designed to solve many challenging problems in the cryptocurrency world, such as Airflow data pipelines, blockchain analytics tools, machine learning applications, etc. Wait. According to our internal data, the number of Python applications has nearly doubled since Q3 2022. According to our internal data, there are approximately 1,500 data processing pipelines and services developed in Python today. At the time of this writing, the total number of builds is about 500 per week. We expect wider adoption as more Python-centric frameworks (like Ray, Modin, DASK, etc.) are adopted into our data ecosystem.
Pants are designed to use real-world Python for first-class use cases, including handling 3rd party dependencies. In fact, parts of Pants itself are written in Python (the rest in Rust).
The pants have a mild learning curve. It has much less friction than other tools. One-click installation experience of tools, simple configuration files, and moderate maintenance costs.
Python is the most powerful tool for machine learning and data science applications One of the popular programming languages. However, prior to the Python-first build system Pants, our internal investment in the Python ecosystem was low compared to Golang and Ruby – Coinbase’s primary choice for writing services and web applications.
Lack of simplified release process: Code changes Often complex cross-repository updates and releases are required. There is no automated workflow to perform integration and staging testing of related changes. The lack of coherent observability and reliability results in huge engineering overhead.
Building a company-wide monorepo requires an elite team. We don’t have enough staff to replicate the monorepos success story on Facebook, Twitter and Google.
Figure 1. Dependency diagram for a Machine Learning Platform (MLP) project. Down Diagram showing Coinbase’s repository architecture, where the green blocks represent the new Python ecosystem we’re building. Inter-repository operability is achieved through the service layer, including code artifacts and schema registries. Figure 2. Coinbase’s repository architecture
#Third party dependencies# Third-party dependencies ├── 3rdparty │ ├── Dependency 1│ │ ├── build│ │ ├── requirements.txt│ │ └── resolve1.lock # lockfile│ ││ └── Dependency 2│ │ ├── Build│ │ ├── required. text │ │ └── resolve2.lock…│# Shared library├── lib│# Top level project folder ├── project1 # Project name │ ├── src │ │ └── python│ │ ├── data block │ │ │ ├── Build│ │ │ ├─ ─ owner │ │ │ ├── gateway. py│ │ │ …│ │ └── Notebook │ │ ├── build│ │ ├ ── Owner │ │ ├── etl_job.py │ │ …│ └── test│ └── p ython │ ├── data block │ │ ├── BUILD│ │ ├── gateway_test.py│ │ …│ └── notebook │ ├── build│ ├── etl_job_test.py │ …├── Item 2 …│# Dockerfile├── dockerfiles│# Tools for lint, formatting, etc. ├── Tools│# Buildkite CI Workflow ├ ── .buildkite
│ ├── pipeline .yml
│ └── Hook
│
#Pants Library
├── Pants
├──pants.toml
└── pants. ci.toml
- Figure 3. Pynest repository structure
- Below is a list of the main elements of the repository and their explanations .
1. Third Party
Third party dependencies are placed in this file clip. Pants will parse the requirements.txt file and automatically generate a “python_requirement” target for each dependency. Pants’ multiple lock files feature supports multiple versions of the same dependency. This feature makes it possible for projects to have conflicts in direct or transitive dependencies. Pants generates lock files to pin every dependency and ensure reproducible builds. More explanation of Pants Multilock is in the Dependency Management section.
2. library
3. Project Folder
4. Code Owner File
Code Owners files ( OWNERS ) are added to folders to define the person or team responsible for the code in the folder tree. The CI workflow call script compiles all OWNERS files into CODEOWNERS files under “.github/”. Code owner approval rules require all pull requests to be approved by at least one group of code owners before being merged.
5. Tools
The Tools folder contains configuration files for code quality tools such as flake8, black, isort, mypy, etc. These files are referenced by Pants to configure the linter.
6. Buildkite Workflow
Coinbase uses Buildkite as a CI platform. Buildkite workflows and hook definitions are defined in this folder. A CI workflow defines something like
steps like Check if the dependency lock file needs to be updated.
to generate a code coverage report. 7.Dockerfiles
Dockerfiles are defined in this folder. The docker image is built by a CI workflow and deployed by Codeflow – Coinbase’s on-premise platform.
8. Pants This folder contains Pants scripts and configuration files (pants.toml, pants.ci.toml). This article describes how we built PyNest using the Pants build system. In our next blog post, we will explain dependency management and CI/CD.
