Pre-production environments are used by most companies when deploying web sites and software. This allows them to test new changes before they are made available to users. This can have many benefits, including adding layers of defense to find bugs and problems, but it can also lead to increased cost and complexity. Techniques such as Continuous DeliveryEncourage teams to ensure that software is always available. This will allow for a shift away from testing environments and complex branching structures, and a move towards simpler setups. SqueakyA company that helps businesses to understand how their visitors use their web site or app without invading people’s privacy, i.e. – has taken a different approach. why they don’t use a staging environment. They believe this allows them to ship faster and reduce the number of production issues.
A blog post describes their approach. Lewis MonteithSqueaky details the problems they encountered with staging environments.
Pre-live environments cannot be compared to production. A cloud-native, scalable app will often need more resources in order to handle load. However the cost of building the exact same setup for pre-live makes it prohibitive. This can lead to configuration drift and a scaled-down architecture for pre-live environments, which can make it difficult to test in these environments.
There’s always a queue, which makes releases larger and reduces ownership: If multiple developers or teams wish to release code simultaneously, pre-live environments can prove to be a bottleneck. Waiting for the environment can cause delays and compromises in testing, especially when tests fail and everyone must wait for them to be fixed. Branch divergence is a problem for developers when there are many changes that need to be merged.
To reduce the queue, this then leads to releases being bundled together which means it’s more likely that bugs are introduced, and for it to be difficult to track down exactly which change and whose change caused the problem, as issues are isolated and developers may not realise changes have gone to production.
Accountability is replaced by process Pre-live environments are usually managed by ops-focused groups. Thus, deploying to pre-live environments can result in an implicit handover of responsibility between developers and ops teams.
Squeaky’s alternative approach aims to resolve or avoid these issues – with four key tenets to make this work.
Only merge code is ready for launch: This approach is supported by appropriate tests and validation of the changes in development.
Flat branching strategy All branches are forked starting at the main branch. Any changes made are never merged back there. Smoke testing happens locally on a developer’s computer.
Feature flagging for high-risk changes: Squeaky might ship significant changes behind the feature flag if they are concerned about performance or user reactions to a change. They have the ability of doing this on a per user basis.
Hands-on deployments: monitoring, logging and alarms are used comprehensively to ensure there’s no issues, and Squeaky also use blue/green deployments to deploy changes to a subset of users until they are sure everything is OK.
Squeaky’s dropping of a staging environment in favour of many continuous delivery principles has changed the mindset towards shipping software. It is important to increase confidence that changes are ready for production by removing the buffer for them being made. This has resulted in lower costs and complexity as well as a faster development cycle.