The Problem I Was Trying to Solve

When developers push code changes, someone should review that code for security issues. Things like SQL injection, hardcoded passwords, or broken authentication logic. These are documented under something called the OWASP Top 10 — the most common and dangerous security risks in web applications.

In real companies, security engineers do this manually Or they pay thousands of dollars a month for tools that do it automatically.

I couldn't afford those tools. So I built something instead.

What I Built

A self-hosted AI agent that watches every Pull Request, reads the code changes, and asks an AI model — "does this look dangerous?"

The result shows up directly on the PR as a comment. If the risk score is high enough, it also fires a Slack alert.

Here is the full flow:

Developer opens a Pull Request → GitHub Actions triggers the workflow → build_and_test job runs first → ai_security_gate job starts → Fetch changed file list from git → Filter out irrelevant files → Grab each file's diff → Wrap in XML tags → Send to NVIDIA NIM API → Parse the JSON response → Post comment on PR → risk_score ≥ 6 → Send Slack alert

The Python scripts live on the VM at /opt/security-agent/ — not inside the repo. And the ai_security_gate job only runs on pull requests, not on every push to main.

Both of those decisions were intentional. I'll explain why.

The Two Jobs in the Workflow

The GitHub Actions workflow has two jobs that run in sequence.

jobs:
  build_and_test:
    runs-on: self-hosted
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Install dependencies
        run: npm install
      - name: Run tests
        run: npm test

  ai_security_gate:
    runs-on: self-hosted
    needs: build_and_test
    if: github.event_name == 'pull_request'

build_and_test runs first. If tests fail, the security scan never even starts. No point scanning broken code.

ai_security_gate only fires on pull requests. Push directly to main? No scan. Open a PR? Scan runs automatically.

The needs: build_and_test line enforces that order. Clean and simple.

Why the Scripts Live on the VM

Most GitHub Actions tutorials put everything inside the repo. I did it differently — the Python scripts live at /opt/security-agent/ on the VM itself.

The workflow just calls them:

- name: Run Gatekeeper Script
  env:
    PR_NUMBER: ${{ github.event.pull_request.number }}
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    python /opt/security-agent/scan_diff.py

Why? A few reasons.

The scripts contain configuration and logic I don't want exposed in the repo. The .env file with API keys lives at /opt/security-agent/.env — completely separate from the codebase. Anyone who clones the repo gets the workflow file, not the scanner internals.

I can update the scanner without touching the repo at all. SSH into the VM, edit the script, done. Next PR scan picks up the changes automatically.

How It Finds the Changed Files

The scanner starts by asking git — what actually changed in this PR?

def get_pr_diff():
    diff_output = subprocess.check_output(
        ["git", "diff", "origin/main...HEAD", "--name-only"],
        text=True,
        timeout=30
    )
    files = [f for f in diff_output.strip().split('\n') if f]
    return files

origin/main...HEAD means "everything that changed between main and the tip of this PR branch." The --name-only flag gives back just the file names, not the actual diff yet.

Then it filters out anything that doesn't need scanning:

EXCLUDED_EXTENSIONS = {
    '.md', '.txt', '.json', '.lock', '.yaml', '.yml',
    '.png', '.jpg', '.jpeg', '.svg', '.pdf', '.env', '.pem'
}

MAX_FILE_SIZE_BYTES = 50 * 1024  # 50KB limit

Notice .env and .pem are in that exclusion list. Those files should never contain diff-worthy changes in a PR anyway. If they do, that's a different problem.

The 50KB limit per file is a practical guard. Massive auto-generated files would eat your token budget fast. Skip them.

The XML Trick

After filtering, each file's actual diff gets fetched and wrapped in XML tags:

sanitized_payload.append(
    f"<file name=\"{file_path}\">\n{file_diff}\n</file>"
)

So what the AI model actually receives looks like this:

<file name="src/auth/login.py">
diff --git a/src/auth/login.py b/src/auth/login.py
-    query = "SELECT * FROM users WHERE email = %s"
+    query = f"SELECT * FROM users WHERE email = '{email}'"
</file>

Why XML and not just plain text? The tags create clear boundaries. The model knows everything inside <file> tags is code to analyze, not instructions to follow. It makes the output significantly more consistent.

One thing I want to be upfront about — XML tags do not prevent prompt injection. If someone puts something malicious in a code comment inside their PR, the model will still see it. The XML helps with structure and clarity, not security. The real protection is validating the output on your side and never using the AI score as an automatic merge block.

Talking to the AI Model

The scanner uses the OpenAI SDK pointed at NVIDIA's API endpoint instead of OpenAI's:

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=nvidia_key,
    timeout=90.0
)

Same SDK, different model, different URL. That's it.

The system prompt tells the model exactly what it is and exactly how to respond:

system_instruction = (
    "You are an automated corporate security gateway. Analyze the following code diffs wrapped in XML elements. "
    "Flag OWASP findings, logical vulnerabilities, or accidental credential exposures. "
    "You must output your findings strictly as valid raw JSON matching this scheme:\n"
    "{\n  \"risk_score\": <0-10 integer>,\n  \"findings\": [\"Description of issue 1\", \"Description of issue 2\"]\n}"
)

And the API call:

completion = client.chat.completions.create(
    model="nvidia/nemotron-3-ultra-550b-a55b",
    messages=[
        {"role": "system", "content": system_instruction},
        {"role": "user", "content": f"Review these PR diff changes:\n\n{diff_data}"}
    ],
    temperature=0.2,
    top_p=0.95,
    max_tokens=4000,
    extra_body={"chat_template_kwargs": {"thinking": False}},
    stream=False
)

Three settings that matter here:

temperature: 0.2 — Keeps the model focused and deterministic. High temperature means creative output. Creative output means broken JSON.

thinking: False — NVIDIA's models have a chain-of-thought reasoning mode. Great for complex reasoning tasks. Terrible when you need clean structured JSON output. Disable it.

timeout: 90.0 — The API can be slow under load. Without a timeout your script hangs forever and the PR just sits there.

Parsing the Response

The model is supposed to return clean JSON every time. It mostly does. But "mostly" isn't good enough when your pipeline depends on it.

response_content = completion.choices[0].message.content.strip()

if response_content.startswith("```json"):
    response_content = response_content.split("```json")[1].split("```")[0].strip()
elif response_content.startswith("```"):
    response_content = response_content.split("```")[1].split("```")[0].strip()

results = json.loads(response_content)

Even with temperature: 0.2 and explicit instructions to return raw JSON, the model sometimes wraps the output in markdown code fences. This strips them out before parsing.

If json.loads still fails after that, the whole thing crashes with an exception and sys.exit(1). The pipeline fails visibly. That's intentional — a silent failure that marks a vulnerable PR as clean would be worse than a noisy crash.

What Gets Posted Back

Once the findings are parsed, they go back to the PR as a comment:

comment_body = (
    f"### 🛡️ AI Security Gatekeeper Audit\n\n"
    f"**Risk Score:** `{risk_score}/10`\n\n"
    f"#### Findings:\n{bullet_findings if bullet_findings else '- No major vulnerability signatures detected.'}"
)

And if the risk score hits 6 or above, Slack gets a message:

if risk_score >= 6:
    send_slack_alert(risk_score, findings, safe_pr, safe_repo)

The Slack message uses color coding. Findings get truncated at 1000 characters so the Slack notification stays readable.

The Cost Breakdown

GitHub Advanced Security costs \(49 per seat per month. For a 10-person team that's \)490. Enterprise SAST tools start at $1,000+.

This setup costs less than a Netflix subscription.

What It Actually Catches

It's good at finding things that are visible in a diff:

• SQL injection when someone switches from parameterized queries to f-strings

• Hardcoded API keys or passwords accidentally committed

• Path traversal — user input going directly into file paths

• XSS sinks — unescaped user input going into HTML

It struggles with things that need full codebase context:

• Business logic bugs that only make sense when you understand the whole application

• Vulnerabilities where the dangerous sink is in a completely different file

• Race conditions

This is a first-pass filter. It surfaces the obvious stuff fast. It does not replace a real security review

Lessons From Building This

Consistent JSON output is harder than it sounds. Low temperature, explicit schema in the prompt, disabling chain-of-thought, and stripping markdown fences after — you need all of it. Any one piece alone isn't enough.

Keeping scripts off the repo was the right call. Separation between the workflow definition and the scanner logic feels clean. The repo stays simple. The scanner stays flexible.

A failing pipeline beats a silent pass. When JSON parsing fails, the script exits with code 1. The whole job fails red. That's uncomfortable but it's honest. A crashed scan is visible. A silently-passed vulnerable PR is not.

What's Next

A few things I want to add when I get time:

• Move the risk threshold from hardcoded 6 to an environment variable so it's configurable without touching code

• Suggested fix in the PR comment — not auto-applied, just shown so the developer knows what to change

• Better handling of massive PRs instead of just hitting the token limit

Try It Yourself

Everything is open source. The repo has the workflow file and the structure. You bring your own VM, your own NVIDIA NIM API key, and drop the scripts in /opt/security-agent/.

Repo: github.com/munnavuyyuru/github-actions-ai-gatekeeper

If something is wrong or could be done better, I genuinely want to know. Drop it in the comments.

Built by

Venkata Bhargav Vuyyuru

I thought deploying Docker containers to Kubernetes would be straightforward.

After all, my applications were already running perfectly with Docker.

I assumed Kubernetes would simply orchestrate the containers I had already built.

Instead, I ran into ImagePullBackOff errors, broken networking, ConfigMap issues, and a MySQL database that refused to start because of a missing CSI driver.

This article isn't a Kubernetes tutorial. It's the story of how I built, broke, debugged, and finally deployed two applications—from a local kind cluster to Amazon EKS—and the lessons I learned along the way.

Note: All the source code, Dockerfiles, Kubernetes manifests, architecture diagrams, and deployment guides are available in my GitHub repository:
https://github.com/munnavuyyuru/k8s-deployments

What You'll Learn

By the end of this article, you'll understand:

• How I containerized multiple applications using Docker

• Why I used both Docker Hub and Amazon ECR

• How I deployed a 2-tier application on a local kind cluster

• How I deployed a 3-tier application on Amazon EKS

• How I debugged real Kubernetes issues like localhost networking, ConfigMap scoping, and PersistentVolume provisioning failures

• The lessons I learned that every beginner should know before working with Kubernetes

Project Overview

To understand Kubernetes properly, I decided to build two different projects instead of deploying a simple "Hello World" application.

The first project was intentionally simple so I could learn the fundamentals of Kubernetes networking and deployments.

The second project introduced persistent storage, namespaces, secrets, ingress, and cloud infrastructure, making it much closer to a real production application.

2-Tier Architecture Diagram

Frontend → Nginx Reverse Proxy → Backend

3-Tier Architecture Diagram

Ingress → Frontend → API → MySQL

Stage 1: Containerizing the Applications with Docker

Before writing a single Kubernetes manifest, I wanted to make sure every application worked perfectly inside Docker.

This became one of the most valuable habits I developed during the project.

If an application doesn't work correctly inside a Docker container, Kubernetes won't magically fix it.

For the 2-tier application, I created:

• A Docker image for the Node.js backend

• A separate Docker image for the Nginx frontend

• A multi-stage Docker build for the frontend to reduce the final image size

For the 3-tier application, I created three separate Docker images:

• Flask Frontend

• Flask API

• MySQL Database

Since MySQL stores application data, I deployed it later using persistent storage rather than treating it like a stateless container.

Before touching Kubernetes, I tested every image locally using docker build and docker run. This helped me isolate Docker-related problems before introducing Kubernetes into the workflow.

Stage 2: Publishing Images to a Container Registry

Once the Docker images were working locally, the next step was making them available for Kubernetes.

For my first project, Docker Hub was sufficient.

Since the cluster was relatively simple, pushing images to Docker Hub and configuring imagePullSecrets allowed Kubernetes to authenticate and pull private images successfully.

For the second project running on Amazon EKS, I wanted a more production-oriented workflow.

Instead of Docker Hub, I used Amazon Elastic Container Registry (ECR).

Using ECR simplified image management inside AWS and integrated naturally with the EKS cluster.

Stage 3: Deploying the 2-Tier Application on kind

With the images available, it was finally time to deploy my first Kubernetes application.

I chose kind (Kubernetes in Docker) because it provides a lightweight local Kubernetes cluster without requiring cloud resources.

The deployment included:

• A dedicated namespace

• Deployments for both frontend and backend

• Services for internal and external communication

• A ConfigMap containing the Nginx reverse proxy configuration

One of the most important design decisions was exposing only the frontend.

The backend remained private using a ClusterIP Service, while the frontend was exposed externally through MetalLB, which provided LoadBalancer functionality inside the kind cluster.

At this point, I finally had a fully working application running on Kubernetes.

Applying Kubernetes manifests.

Verifying Deployments, Pods, Services, and ConfigMaps.

Stage 4: Deploying the 3-Tier Application on Amazon EKS

After successfully deploying the local application, I wanted to experience Kubernetes in a real cloud environment.

I created an Amazon EKS cluster and deployed a complete three-tier architecture.

Compared to the first project, this deployment introduced several additional Kubernetes concepts.

The application consisted of:

• Separate namespaces for frontend, backend, and database

• ConfigMaps for application configuration

• Secrets for database credentials

• Deployments for the stateless frontend and API

• A StatefulSet for MySQL

• Persistent storage backed by Amazon EBS

• An Ingress resource for external access

Unlike the previous deployment, the database required persistent storage because losing data whenever a pod restarted would not be acceptable.

Using a StatefulSet ensured that MySQL maintained a stable identity while PersistentVolumes preserved the database across pod restarts.

Separate namespaces for better resource organization

MySQL running successfully inside a StatefulSet

Backend API pods running successfully

The Biggest Challenges I Faced

This was the most educational part of the project.

Every issue forced me to understand Kubernetes more deeply instead of simply following documentation.

Challenge 1: The "localhost" Networking Trap

Problem

The frontend loaded successfully, but every API request failed.

Root Cause

My frontend JavaScript was calling:

http://localhost:8001

Inside the browser, localhost refers to the user's own machine, not the Kubernetes backend running inside the cluster.

Investigation

I checked:

• Browser Network Tab

• Kubernetes Services

• Pod Logs

Everything looked healthy except the frontend was pointing to the wrong location.

Solution

Instead of calling localhost directly, I changed the API endpoint to a relative path:

/api

Then I configured Nginx as a reverse proxy to forward every /api/* request to the backend Kubernetes Service.

This completely solved the networking issue while keeping the backend hidden from the public.

Challenge 2: Amazon EBS CSI Driver Failure

Problem

The MySQL pod remained stuck in the Pending state.

Investigation

Running:

• kubectl get pvc

• kubectl describe pvc

• kubectl get events

revealed that Kubernetes couldn't provision a PersistentVolume.

Root Cause

The Amazon EBS CSI Driver had failed to install because the required IAM permissions were missing.

Solutions :

• Removed the broken add-on

• Installed the EKS Pod Identity Agent

• Reinstalled the EBS CSI Driver

• Associated the correct IAM role

Once the driver became healthy, Kubernetes automatically provisioned the PersistentVolume, and the MySQL pod started successfully.

CSI Driver Fix Screenshot

Challenge 3: ConfigMap Not Found

Problem

The backend API pod failed with: CreateContainerConfigError

Root Cause

The pod expected a ConfigMap named mysql-config.

However, the ConfigMap existed inside the database namespace, while the backend application was running inside the backend namespace.

ConfigMaps are namespace-scoped, so Kubernetes couldn't access it.

Solution

I created another ConfigMap inside the backend namespace.

After applying the new ConfigMap, the application started immediately.

This taught me one of Kubernetes' most important rules:

Pods can only directly access ConfigMaps and Secrets inside their own namespace.

Lessons I Learned

Completing these two projects changed how I think about Kubernetes.

Some of my biggest takeaways were:

• Docker success doesn't guarantee Kubernetes success

• Networking is usually harder than writing YAML

• localhost means something different depending on where your code is running

• Namespaces provide isolation but require careful resource organization

• Stateful applications need persistent storage and should usually run as StatefulSets

• Debugging becomes much easier when you use kubectl describe, kubectl logs, and kubectl get events before changing configuration files

• Understanding why something failed is far more valuable than simply making it work

If You Want to Try This Yourself

Everything used in this article is available in my GitHub repository.

It includes:

• Dockerfiles

• Kubernetes manifests

• Architecture diagrams

• Deployment documentation

• Troubleshooting notes

To reproduce the projects, you'll need:

• Docker

• kubectl

• kind (for the local deployment)

• An AWS account (for the EKS deployment)

I recommend starting with the local kind deployment before moving to Amazon EKS.

Understanding the fundamentals locally makes debugging cloud deployments much easier.

Conclusion

When I started this project, I thought Kubernetes was mostly about writing YAML files.

After completing both deployments, I realized Kubernetes is really about understanding how containers, networking, storage, security, and cloud infrastructure work together.

The most valuable lessons didn't come from successful deployments.

They came from the failures.

Every ImagePullBackOff, every networking issue, every storage problem, and every debugging session taught me something new about how Kubernetes actually works.

If you're beginning your own Kubernetes journey, don't be afraid to break things.

Sometimes the fastest way to learn is by fixing your own mistakes.

Thank you for reading!

If you've faced similar Kubernetes challenges—or have suggestions for improving this setup—I’d love to hear from you in the comments.

Happy learning, and happy building! 🚀