Lyft engineering decomposed a monolithic structure into a collection a microservices back at the end of 2018. Modular development environments based on Docker containers eventually moved towards the cloud. Recent articles have described how they were moved to the cloud. The development tooling industry struggled to keep paceAs time passed, the number and variety of microservices exploded. The engineer’s machine had to be re-invented in order to create development environments.
The original goal was to create a Docker container orchestration environment that engineers could use to test. It would make use of multi-tenant environments to its advantage in production, making it cheaper and quicker to scale than the existing solution.
Lyft’s local development environment was called Devbox – short for “development environment in a box” – and consisted of some tooling managing a local virtual machine and its configuration, including database seeding, package and image download, and installation. Developers only needed to issue one command in order to create an environment that could receive requests.
The need to share these environments with others eventually emerged. Onebox was born from Devbox’s move to the cloud. Onebox was essentially a Devbox environment, running on an EC2 instance. Engineers preferred Onebox to Devbox because it had more capacity and could download images faster.
(source: https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-1-a2f5d9a77813)
As microservices increased in number, it became more difficult and more time-consuming to create and launch Onebox instances.
Each service may have interaction trees that are many levels deep. This could make it difficult to manage the environment instance. Debugging could be frustrating because the observation tools couldn’t keep up with all the running environments. Engineers also had to consider the whole system rather than focusing on one component. This increased their cognitive load.
According to Lyft engineering, an engineer’s code changing process can be divided into an “inner dev loop” and an “outer dev loop”. The first loop involves only changing code and running tests. This should take you only a few moments to give feedback. The second, which involves continuous integration and code review, can take longer.
Onebox environments were slow to set-up and start, with visible instability. Therefore, engineers relied on the outer dev loop’s CI test to validate each code change iteration.
These frustrations and growing pains were eliminated by shifting the focus to Restoring the development environment to the engineer’s laptopWhile simultaneously rebuilding the inner development loop.
After realizing that running code within containers is not an abstraction, it was decided to run service code natively on MacOS in an isolated environment without containers or virtual machines.
Lyft’s backend services are written primarily in Python or Go. Frontend services are written mainly in Node.
- Python services can be isolated by using immutable virtual environment. Every time the requirements.txt file is modified, a new virtual environment will be created.
- Go services use the Go modules toolchain in order to automatically download and link all dependencies whenever the commands are issued. go runOr go testAre executed.
- To create the right environment for each Node Service based on its metadata, a wrapper has been built around nodeenv.
Locally running specialised services like datastores are possible, and often use containers. Datastores are loaded with new data at startup using scripts maintained the team responsible for the service.
Locally starting a service takes several steps. It is tedious and risky to manually execute it.
TiltLyft uses this to manage the service’s lifecycle and environment. Each service comes with a Tiltfile that outlines the steps required to make it run locally. Running services also reload themselves when an engineer changes their code in an IDE. This further reduces the inner development loop.
You must interact with the service in addition to operating it. Lyft has many transport formats that can be used to request services, such as gRPC and protobuf/HTTP.
Engineers use Lyft’s tool to compose and send local service requests. They also take advantage of autocomplete features that are possible through the tool’s integration to the service’s IDL.
The net result was that local service users can send requests directly into their service’s API rather than testing via the mobile app talking with public APIs. This increases the developer’s knowledge of their API and decreases the amount of time needed to fix an error.
Other professionals agree that local environments for development are a good thing. Sotheby’s Vice President of Engineering is an example. James TurnbullThey were described as follows: Engineers can be more efficient and less expensive.Cloud-based environments are more reliable.