Protecting The Wire - Semaphore Behind SSL Proxy

Mission Brief

Plain text communication is loud. It's bleeding data.
Prying eyes can see every bit in the wire.

You have to isolate the backend - the Semaphore UI and MySQL containers stay locked down. Unreachable for the external work.
Open a tiny hole on the stronghold to the world - the frontend is an NginX SSL proxy.

You use:

Podman pod for network and container isolation
The Semaphore and MySQL containers without exposing them to the world
An NginX proxy container with SSL

Requirements

Debian 12 (with hardened kernel preferred)
Podman installed and functional (v4 or later)
Working MySQL container
Functioning Semaphore UI container
X509 certificates for HTTPS
NginX configuration for the reverse proxy
Terminal access

Step 1: Create The Pod For The Configuration

The pod exposes only an HTTPS port for the NginX proxy:

podman pod create -p 4430:443 p_semaphore

Non-root containers use ports above 1000.

Step 2: Activate The MySQL Container

Run the database container assigned to the pod:

podman run --rm -d  \
       --pod p_semaphore  \
       -v semaphore_mysql:/var/lib/mysql  \
       --name semaphore_mysql  \
       -e  MYSQL_RANDOM_ROOT_PASSWORD= 'yes'  \
       -e  MYSQL_DATABASE=semaphore  \
       -e  MYSQL_USER=semaphore  \
       -e  MYSQL_PASSWORD=semaphore  #  DO NOT USE IN PROD \
       docker.io/mysql:lts

No open ports.
No exposure.

Step 3: Initiate The Semaphore Container

Assign and run Semaphore in the pod:

podman run --rm -d  \
       --pod p_semaphore  \
       --name semaphore  \
       -e  SEMAPHORE_DB_USER=semaphore  \
       -e  SEMAPHORE_DB_PASS=semaphore  #  DO NOT USE IN PROD \
       -e  SEMAPHORE_DB_HOST=127.0.0.1  \
       -e  SEMAPHORE_DB_PORT=3306  \
       -e  SEMAPHORE_DB_DIALECT=mysql  \
       -e  SEMAPHORE_DB=semaphore  \
       -e  SEMAPHORE_PLAYBOOK_PATH=/tmp/semaphore/  \
       -e  SEMAPHORE_ADMIN_PASSWORD=changeme  #  DO NOT USE IN PROD \
       -e  SEMAPHORE_ADMIN_NAME=admin  \
       -e  SEMAPHORE_ADMIN_EMAIL=admin@localhost  \
       -e  SEMAPHORE_ADMIN=admin  \
       -e  SEMAPHORE_ACCESS_KEY_ENCRYPTION= gs72mPntFATGJs9qK0pQ0rKtfidlexiMjYCH9gWKhTU=  \
       -e  SEMAPHORE_LDAP_ACTIVATED= 'no'  \
       -e  TZ=UTC  \
       docker.io/semaphoreui/semaphore:latest

No exposed ports.
The DB host is 127.0.0.1 in the pod.

Step 4: The Mask To The World - NginX Reverse Proxy

NginX Configuration

You set up the only exposure here. No mistakes.

The nginx-conf/nginx.conf file must be brief. No fluff.

Clients connect over SSL on port 4430 - they are forwarded to the NginX port 443 in Podman.
NginX from 443 forwards the connection to Semaphore's port 3000 (Default UI port). It's sealed. Never exposed.

server {
    listen 443 ssl;
    server_name semaphore.local;
 
    ssl_certificate /etc/nginx/cert.pem;
    ssl_certificate_key /etc/nginx/key.pem;
 
    location / {
        proxy_pass http://127.0.0.1:3000/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	      proxy_set_header X-Forwarded-Proto $scheme;
	      proxy_set_header X-Forwarded-Port $port;
	      proxy_set_header X-Forwarded-Host $host:$port;
        proxy_set_header X-Forwarded-Server $host;
    }
}

In a Podman pod the nework is separated. Containers "see" each other.

X509 Certificates

Always use valid certificates in production systems. They are your chain of trust.

For testing you can create your own files.
Never use self-signed certificates in prod. A warning in the browser is the killer of trust.

mkdir certs &&  cd certs
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
 cd ..

Proxy Container

Assign and run the reverse proxy in the pod:

podman run --rm -d  \
       --pod p_semaphore  \
       --name semaphore_proxy  \
       -v ./nginx-conf/nginx.conf:/etc/nginx/conf.d/default.conf:ro  \
       -v ./certs/cert.pem:/etc/nginx/cert.pem  \
       -v ./certs/key.pem:/etc/nginx/key.pem  \
       docker.io/library/nginx

Step 5: Verify The Setup

Log in.
Check the logs.
Verify your setup.

podman pod logs p_semaphore

Does it work? Seal it.

Step 6: Generating The Pod Configuration File

You built a system. Don't rebuild it by hand.
Create a Kubernetes-compatible pod configuration.

podman kube generate -f semaphore-pod.yaml p_semaphore

You may have to edit the file a bit. Remove the noise. Keep the data.

Step 7: Security Considerations - Hardening

Offload secrets to .env files or a secrets manager - never bake them into scripts.
Don't use self-signed certificates. Use valid Let's Encrypt or other ones.
Never trust. Verify. Monitor. Check.

Ghost Thoughts

Networks and systems are full of evil ghosts with prying eyes.
Unencrypted flows scream louder than logs. Silence them.

Secure the wire - use SSL, encrypt your connections.

"They left Semaphore open on port 3000. We tunneled in through that silence."

—

DeadSwitch | The Silent Architect
"Fear the silence. Fear the switch"