When we implemented the Trunk-based deployment flow, our pipeline to prod sped up a lot. This was really cool, but at the same time, we were producing more code than the QA team could handle, preventing us from deploying specific features to production if there was something already in the queue. We kinda solved this by creating feature branch deployments, but the problem was devs merging to QA instead of deploying the testing envs, causing the issue again.

One of the first infrastructure implementations I made was migrating the main app from 3 EC2 servers to ECS Fargate. This was the outcome of a conversation I had with the CTO when I proposed helping with some DevOps tasks.

There were several reasons that pushed me to commit to this. First, when I started working for the company, I had access to AWS and figured out that our servers were oversized and we were paying too much for them. The main server CPU usage was under 5%; this one was in charge of running a bunch of cron jobs, and the other 2 servers were under 2% on CPU usage. RAM usage was similar.

Starting to use IaC templates was an improvement in our operation, as when we wanted to deploy a new environment, we simply copied a similar template and made some small changes to adapt it to the new desired infrastructure. Soon, we found out an issue with this pattern: we were creating application load balancers for every new service deployed.

There were not so many, something around 9 services, maybe, but the issue is we also implemented lower environments for testing purposes. In our case, Development and QA, besides Production, and they are an exact replica, obviously with smaller capacity. But when we talk about the ALBs, they have a base cost of around $19 for having them running and a variable cost depending on usage, called LCU cost. In the following table, we can see the total costs associated:

Before I started making changes to the infrastructure, the rule was simply to go one step up in server size. Every time there was a spike and it was confirmed that the database was the root cause, the server size was increased, and at some point we had a single 8xlarge instance for our main database.

In the past, there was no logging system implemented, and the actions were based only on resource usage. We started getting better insight after our Datadog implementation in AWS. We set alerts for resource usage, and we also enabled logs from the database, including the slow query logs.

You might be thinking, who deploys a database in a public subnet? Even worse, it was deployed in the AWS default public subnet with publicly accessible enabled. This is what happens when services are deployed just because, without thinking about the implementation, but only about business requirements.

At first, it was a bit complex to simply change the publicly accessible setting to disabled as there were a few external data analytics tools that were pulling information from the database every few hours. The first time I tried to do so, I ended up in a call with the CTO and the Data Analytics team asking me why I disabled the public access. Simple answer - Security.