TL;DR

A security audit revealed that a former employee still had repository access due to manual offboarding gaps. Integrating with our corporate IDAM required an expensive SaaS tier. We migrated to self-hosted GitLab, automated user lifecycle management, cut costs dramatically, and now own our DevOps destiny.

Prologue

“Why the hell does the guy who left the organization two days back still have access to the company repositories?” boomed our Managing Director.

Our DevOps team sprang into action. We had depended on HR to deliver the deboarding list, but somewhere, a communication gap broke the chain—and during a critical security audit, we were left painfully exposed.

That was unacceptable.

The recommended fix was to integrate GitLab with the company’s Identity and Access Management (IDAM) using OAuth, automating user onboarding and offboarding. But this feature was locked behind a costly enterprise SaaS tier—far beyond our planned budget.

The path forward was clear: Self-Hosted GitLab—on our own terms.

Why We Looked Beyond SaaS

When we first adopted GitLab SaaS, it felt like a dream:

  • Plug-and-play DevOps
  • Quick onboarding
  • No maintenance hassles

But as our scale and maturity grew, so did our concerns:

  • We needed tight user access controls integrated with IDAM
  • We needed data residency compliance
  • We needed predictable budgeting
  • And we wanted freedom from vendor lock-in

SaaS was no longer a match for our vision—not without a premium price tag.

Exploring Deployment Strategies

Once we decided to self-host, we kicked off a thorough Proof of Concept across three options:

  1. Kubernetes-based deployment
  2. Docker-based deployment
  3. Manual installation on a dedicated Ubuntu VM

During our PoCs, we experimented with Kubernetes-based setups that separated PostgreSQL, Redis, and object storage for ultimate scalability. We even tested Dockerized installations to simplify container orchestration.

On paper, these looked powerful—but they also came with higher operational overhead, more moving parts, and cost implications for our current scale.

In the end, the manual installation with GitLab’s omnibus package on a dedicated Ubuntu VM made the most sense. It followed our KISS (Keep It Simple and Stupid) philosophy, gave us faster time to value, and reduced the skills ramp-up for the team.

Our Migration Process

Migrating a living DevOps platform is never “copy-paste.”

We started by studying GitLab’s built-in backup and restore documentation—but quickly realized critical elements like runner tokens, environment variables, and OAuth mappings wouldn’t carry over cleanly.

We dived into GitLab’s API documentation, determined to leave no piece of our DevOps soul behind. We scripted custom solutions, one line of code at a time, to:

  • Backup and export all repositories
  • Export issues, merge requests, milestones
  • Recreate project-level metadata
  • Validate user roles and group memberships

Every day brought a new challenge:

  • Permissions mismatches
  • 429 rate-limiting errors
  • Missing secrets
  • Broken pipeline histories

We refused to be beaten. We kept iterating, testing, refining. Our scripts grew robust enough to support repeated dry runs until our confidence was bulletproof.

Here’s a glimpse of the kind of code we built to automate the migration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/bin/bash
cat $1 | while read -r record; do
  date
  export GROUPID=$(echo $record | awk -F ',' '{print $1}')
  export KEY=$(echo $record | awk -F ',' '{print $3}')
  export VALUE=$(echo $record | awk -F ',' '{print $4}')
  export MASKED=$(echo $record | awk -F ',' '{print $5}')
  export PROTECTED=$(echo $record | awk -F ',' '{print $6}')

  echo "GROUPID: $GROUPID"
  echo "KEY: $KEY"
  echo "MASKED: $MASKED"
  echo "PROTECTED: $PROTECTED"

  curl --insecure --request POST \
       --header "PRIVATE-TOKEN: $DESTINATION_PRIVATE_TOKEN" \
       --data "key=$KEY&value=$VALUE&protected=$PROTECTED&masked=$MASKED" \
       "$DESTINATION_GITLAB_HOST/api/v4/groups/$GROUPID/variables"

  echo "Import process initiated."
  date
  sleep 10
done

Our Architecture

After thorough PoCs, we finalized the omnibus GitLab package on a dedicated Ubuntu VM, which included:

  • PostgreSQL
  • Redis
  • Gitaly
  • Bundled NGINX

This gave us a proven, stable foundation, matching our KISS philosophy and reducing maintenance complexity.

For runners, we initially deployed dedicated hosts. It worked well in low-concurrency scenarios, but when multiple parallel pipelines started firing off, performance nosedived. Adding more runner hosts triggered serious cost alarms.

We tuned, adapted, and kept things running—but how we finally mastered those runners is another story entirely.

The key win: No more dependency on HR emails. No more manual deboarding delays.

Challenges We Faced

  • Secrets and environment variables didn’t port reliably—we had to manually validate them
  • OAuth user mapping broke on the first attempt, requiring creative debugging
  • Runner tokens failed on cutover—re-registration was needed
  • Some pipelines crashed on their first post-migration build

We documented every pitfall in a team knowledge base, which later became our best shield for ongoing maintenance.

Benefits Realized

  • Costs reduced dramatically—no SaaS premium license
  • Seamless, secure user offboarding through IDAM integration
  • In-house control of upgrades, pipelines, and security
  • Predictable budgeting
  • Freedom from vendor lock-in

Sure, self-hosting needs more love and care—but we own our DevOps destiny now.

Lessons Learned

  1. Validate user roles and permissions—again, and again, and again
  2. Invest in a staging environment—it’s priceless
  3. Automate, but document religiously
  4. Plan for ongoing upgrades and patching
  5. Work closely with HR, InfoSec, and finance—alignment is everything

Final Thoughts

A security scare pushed us to reimagine our entire DevOps stack. In the end, we built a solution that is cheaper, safer, and far more resilient.

We didn’t just migrate a tool—we migrated an entire way of thinking about security, ownership, and engineering excellence.

If your organization struggles with costly SaaS subscriptions, compliance headaches, or clunky offboarding, consider self-hosting. The journey is challenging but the rewards are worth it.

Editorial Note

This article was originally published on Medium and has been revised for uk4.in.

Original publication date: 2025-07-02

Original link: https://www.linkedin.com/pulse/breaking-free-why-team-mavericks-devops-den-moved-from-uttam-jaiswal/