One day, pretty early in the morning, I got a message with a screenshot from the CTO saying:

Any thoughts on how we keep these cleaned up?

Feature Branch deployments in ECS

A few months earlier, we were discussing the need to deploy feature branch environments as a solution to avoid our delivery process getting stuck, as we were deploying more than the QA team could handle. We created a feature branch deployment pipeline, and it was working great, as every important feature was tested in isolation from the QA environment (no merges until approved by QA). Essentially, we stopped having complaints about our queue to production getting stuck due to defective code or incomplete acceptance criteria.

“This is devs’ responsibility; we could set a reminder in the channel,” I said in response. But soon, I understood that devs were not aware of how the feature environments work or the costs associated with them. Even though the pipelines for feature environments had one additional step for teardown, devs were not executing it after QA validation.

After a few minutes, I came back with a proposal to execute a cleanup every day, and during our daily meeting, I let everyone know that daily at 8:00 PM CST, all the feature environments will be flushed automatically. This process guarantees that only the required feature environments are running while maintaining the costs at a minimum. Supposing an environment is deployed at 9:00 AM CST and it runs for 11 hours until 8:00 PM CST, this setup costs less than $0.50.

Having the feature environments in the way I set them up before the cleanup was fairly cheap (around $1 daily) to maintain, but as we were not cleaning the environments properly, this was increasing our monthly billing by $300 every 10 environments. With the cleanup set up, we would need to deploy over 600 feature branches monthly (we were not even close to deploying a tenth of it) in order to reach the $300.

So, by implementing this automated daily cleanup at 8:00 PM CST, we’ve not only ensured that our feature branch environments remain lean and cost-effective, costing less than $0.50 per typical 11-hour deployment, but we’ve also proactively addressed a potential $300 monthly overspend for every ten neglected environments. This approach strikes a balance between providing developers with the isolated testing environments they need and maintaining responsible resource management, allowing us significant headroom for future feature development without unnecessary expense.