Case study
Optimizing CI/CD and Infrastructure at LiveOak Bank
Problem
Inefficient environment setup and slow, monolithic CI/CD pipelines delayed delivery and increased costs.
Solution
Transitioned infrastructure from AWS CloudFormation to Terraform, centralized secrets management, and redesigned GitLab pipelines for modularity and trunk-based development.
Result
Reduced environment setup time by 30%, cut pipeline execution costs by 50%, and accelerated delivery speed by 50%.
In 2023, LiveOak Bank was grappling with the growing pains of a rapidly expanding development organization. The core DevOps challenges stemmed from two areas: infrastructure and CI/CD execution.
On the infrastructure side, AWS CloudFormation was the prevailing standard, but its use was fragmented. Different teams maintained their own templates, leading to inconsistent environments and duplicated effort. Security and compliance were becoming concerns due to secrets being stored in varied, sometimes insecure ways. The setup process for a new environment could take days, requiring manual tweaks and cross-team coordination — a significant bottleneck in an industry where speed and compliance are equally critical.
The CI/CD pipelines presented a parallel challenge. GitLab pipelines had evolved organically over time into large, monolithic scripts. Every new feature deployment ran through the same heavy process, whether it required all steps or not. This created waste in runner minutes, slowed down releases, and made troubleshooting difficult. The rigidity also discouraged experimentation — developers hesitated to adjust pipeline logic for fear of breaking existing jobs.
Strategic Actions
As Vice President of Software Engineering (Senior Development Operations Engineer), I approached the problem with a dual modernization initiative:
-
Infrastructure Modernization
-
Led the migration from AWS CloudFormation to Terraform, enabling a modular and reusable infrastructure-as-code approach.
-
Introduced centralized secrets management using secure vaulting methods, reducing the risk of credential sprawl and improving audit readiness.
-
Standardized Terraform modules across teams, creating a consistent foundation for all environments.
-
-
Pipeline Redesign
-
Overhauled GitLab CI/CD configurations to be modular and aligned with trunk-based development.
-
Created reusable pipeline components, allowing teams to include only the stages they needed for a particular deployment.
-
Implemented automated validation steps early in the pipeline to catch errors faster, reducing re-runs.
-
-
Developer Experience Improvements
-
Documented the new process in a developer-friendly format, making onboarding smoother.
-
Provided team training on Terraform, secrets management, and the updated pipelines.
-
Impact
The results were tangible and measurable:
-
30% Faster environment setup, allowing project teams to move from concept to build significantly quicker.
-
50% Fewer GitLab runner minutes, translating directly into reduced operational costs.
-
50% Faster delivery cycles, enabling features and fixes to reach production in half the time.
-
Improved security posture through consistent, centralized secrets handling.
-
Enhanced morale among developers due to reduced friction and greater autonomy in deployments.
This initiative not only solved the immediate inefficiencies but established a scalable foundation for LiveOak Bank’s future development growth. The improvements continue to pay dividends, both in speed-to-market and in developer satisfaction.