Complete OpenStack Installation on DigitalOcean with LEMP Stack

This guide documents the complete process of deploying OpenStack (via MicroStack) on a DigitalOcean droplet running Ubuntu 24.04 LTS, with a production-hardened LEMP stack serving as the front-end web infrastructure. It covers everything from initial server provisioning through launching your first VM, including all troubleshooting steps, bug fixes, and architectural decisions made along the way. Whether you are building a private cloud for development, testing, or production evaluation, this guide serves as a single source of truth.

1. Introduction and Overview

1.1 What is OpenStack?

OpenStack is an open-source cloud computing platform that provides infrastructure-as-a-service (IaaS) for provisioning and managing virtual machines, networks, block storage, and object storage. Originally developed by NASA and Rackspace in 2010, it has grown into one of the most widely deployed private cloud platforms, backed by a global community of developers and organizations.

OpenStack consists of multiple core services, each responsible for a different aspect of cloud infrastructure:

• Keystone — Identity and authentication service
• Nova — Compute service for managing VM instances
• Neutron — Networking service for defining networks, subnets, and routers
• Glance — Image service for storing and retrieving VM disk images
• Cinder — Block storage service for persistent volumes
• Horizon — Web-based dashboard for managing OpenStack resources
• Swift — Object storage service (optional, not used in this deployment)

1.2 Why Use OpenStack?

OpenStack gives you the power of a public cloud on infrastructure you control. Key advantages include:

• No vendor lock-in — Fully open source with a massive ecosystem
• Cost control — Run multiple VMs on a single droplet, optimizing resource usage
• API-driven automation — Every operation is scriptable via REST APIs or CLI
• Full network control — Define custom topologies with routers, subnets, and floating IPs
• Learning platform — Ideal for DevOps engineers studying cloud infrastructure

1.3 Architecture Overview

This deployment uses an all-in-one topology where the LEMP stack and OpenStack services coexist on a single DigitalOcean droplet. The following ASCII diagram illustrates the architecture:

+-------------------------------------------------------------+
|                  DigitalOcean Droplet                        |
|              4 vCPU / 8 GB RAM / 160 GB SSD                  |
|                    Ubuntu 24.04 LTS                          |
|                                                             |
|  +------------------+      +---------------------------+   |
|  |   LEMP Stack     |      |     OpenStack             |   |
|  |                  |      |     (MicroStack)          |   |
|  |  Nginx 1.24.0    |<---->|  Horizon Dashboard        |   |
|  |  (Reverse Proxy) |      |  http://10.20.20.1        |   |
|  |                  |      |                           |   |
|  |  MySQL 8.0       |      |  Keystone (Auth)          |   |
|  |  (App DB)        |      |  Nova (Compute)           |   |
|  |                  |      |  Neutron (Networking)     |   |
|  |  PHP 8.3-FPM     |      |  Glance (Images)          |   |
|  |                  |      |  Cinder (Block Storage)   |   |
|  +------------------+      +---------------------------+   |
|           |                              |                  |
|  +--------v------------------v-----------+                  |
|  |      UFW Firewall + Fail2ban          |                  |
|  +---------------------------------------+                  |
|           |                                                  |
+-----------v--------------------------------------------------+
            |
     openstack.kuyaops.com (A Record)
     HTTPS (Let's Encrypt SSL)

1.4 Prerequisites

2. Server Preparation

2.1 DigitalOcean Droplet Sizing

For an all-in-one OpenStack deployment, the droplet size directly determines how many concurrent VMs you can run and how responsive the control plane will be. DigitalOcean offers several suitable tiers:

2.2 Ubuntu 24.04 Initial Setup

After creating the droplet, perform these initial configuration steps:

2.2.1 Update the System

# Update package lists and upgrade all packages
sudo apt update && sudo apt full-upgrade -y

# Reboot if the kernel was updated
sudo reboot

2.2.2 Create a Non-Root User (Optional but Recommended)

# Create a new user and add to sudo group
sudo adduser devops
sudo usermod -aG sudo devops

# Switch to the new user
su - devops

2.2.3 Set the Hostname

# Set the hostname to match your domain
sudo hostnamectl set-hostname openstack.kuyaops.com

# Add to /etc/hosts
echo "$(hostname -I | awk '{print $1}') openstack.kuyaops.com openstack" | sudo tee -a /etc/hosts

2.3 Swap Configuration

OpenStack services, especially when running on a single node, consume substantial RAM. A 4 GB swap file prevents OOM (Out of Memory) kills during installation and normal operation. This is especially critical during the MicroStack initialization phase, when multiple services start simultaneously.

2.3.1 Check Current Swap Status

# Check if swap is already active
sudo swapon --show
free -h

2.3.2 Create a 4 GB Swap File

# Create a 4 GB swap file using fallocate (faster than dd)
sudo fallocate -l 4G /swapfile

# Set secure permissions
sudo chmod 600 /swapfile

# Format as swap
sudo mkswap /swapfile

# Enable the swap file
sudo swapon /swapfile

2.3.3 Make Swap Persistent

# Add to /etc/fstab so swap survives reboots
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

2.3.4 Tune Swappiness

The default vm.swappiness value of 60 is too aggressive for server workloads. A value of 10 keeps the system from swapping unless absolutely necessary:

# Set swappiness to 10 (applied immediately)
sudo sysctl vm.swappiness=10

# Make persistent across reboots
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf

2.4 DNS Configuration

Your domain must have an A record pointing to the droplet's public IP address before running the LEMP installation script (Certbot requires it for Let's Encrypt validation).

2.4.1 Configure A Record

Log in to your domain registrar or DNS provider and create the following record:

2.4.2 Verify DNS Propagation

# Check if the A record resolves correctly
dig +short openstack.kuyaops.com
nslookup openstack.kuyaops.com

# Verify from external DNS servers
dig @8.8.8.8 +short openstack.kuyaops.com

2.5 Verification Checklist

Before proceeding to Phase 1 (LEMP), confirm the following:

# Check Ubuntu version
lsb_release -a
# Expected: Ubuntu 24.04 LTS (Noble Numbat)

# Check available RAM
free -h
# Expected: Total RAM >= 8 GB, Swap >= 4 GB

# Check available disk space
df -h /
# Expected: At least 40 GB free (160 GB total recommended)

# Check CPU cores
nproc
# Expected: 4 or more

# Verify DNS resolution
ping -c 1 openstack.kuyaops.com
# Expected: Resolves to your droplet's public IP

# Verify internet connectivity
ping -c 3 1.1.1.1
# Expected: 0% packet loss

3. LEMP Stack Installation

3.1 What is LEMP?

LEMP is an acronym for a popular web service stack composed of four open-source components:

• Linux — The operating system (Ubuntu 24.04 LTS)
• Nginx (pronounced "Engine-X") — High-performance web server and reverse proxy
• MySQL — Relational database management system
• PHP — Server-side scripting language with FastCGI Process Manager (FPM)

In this deployment, Nginx serves as the reverse proxy for the OpenStack Horizon dashboard, while MySQL and PHP provide a foundation for any additional web applications you may deploy on the same droplet.

3.2 Script Overview: setup_lemp_improved.sh

Rather than installing each component manually, a single bash script automates the entire LEMP stack installation. The script is designed to be idempotent (safe to run multiple times) and includes extensive error handling, retry logic, and troubleshooting helpers.

3.2.1 Key Features

3.3 Configuration Variables

Edit these variables at the top of the script before running. They control the domain, SSL certificate email, database credentials, and PHP version.

# ==============================================
# CONFIGURATION VARIABLES
# ==============================================
DOMAIN="openstack.kuyaops.com"
EMAIL="admin@kuyaops.com"           # For Let's Encrypt SSL
DB_NAME="openstack_db"
DB_USER="openstack_user"
PHP_VERSION="8.3"                   # PHP 8.3 is the latest stable

# Auto-generated if not already present:
# DB_PASSWORD  — random 24-character string
# MYSQL_ROOT_PASSWORD  — random 24-character string

# Generated credentials are saved to:
# /root/.lemp_credentials (chmod 600)

3.4 Complete LEMP Installation Script

Save the following script as /root/setup_lemp_improved.sh:

#!/bin/bash
# =============================================================================
# LEMP Stack Installation Script for Ubuntu 24.04 LTS
# Version: 2.0 (Production-Ready with Full Error Handling)
# Description: Installs Nginx, MySQL, PHP-FPM with SSL, firewall, and security
# =============================================================================
set -euo pipefail

# -----------------------------------------------
# CONFIGURATION VARIABLES
# -----------------------------------------------
DOMAIN="openstack.kuyaops.com"
EMAIL="admin@kuyaops.com"
DB_NAME="openstack_db"
DB_USER="openstack_user"
PHP_VERSION="8.3"
CREDENTIALS_FILE="/root/.lemp_credentials"
BACKUP_DIR="/root/lemp-backups/$(date +%Y-%m-%d-%H%M%S)"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color

# -----------------------------------------------
# UTILITY FUNCTIONS
# -----------------------------------------------
log_info() { echo -e "${BLUE}[INFO]${NC}  $1"; }
log_ok()   { echo -e "${GREEN}[OK]${NC}    $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC}  $1"; }
log_err()  { echo -e "${RED}[ERROR]${NC} $1"; }

spinner() {
    local pid=$1 delay=0.1 spinstr='|/-\'
    while kill -0 "$pid" 2>/dev/null; do
        local temp=${spinstr#?}
        printf " [%c]  " "$spinstr"
        local spinstr=$temp${spinstr%"$temp"}
        sleep $delay
        printf "\b\b\b\b\b\b"
    done
    printf "    \b\b\b\b"
}

wait_for_apt_locks() {
    local timeout=300 elapsed=0
    log_info "Checking for apt lock contention..."
    while elapsed < timeout; do
        if ! lsof /var/lib/dpkg/lock-frontend &>/dev/null && \
           ! lsof /var/lib/apt/lists/lock &>/dev/null && \
           ! lsof /var/cache/apt/archives/lock &>/dev/null; then
            log_ok "No apt locks detected"
            return 0
        fi
        # Find and display the process holding the lock
        local lock_pid=$(lsof -t /var/lib/dpkg/lock-frontend 2>/dev/null | head -1)
        if [ -n "$lock_pid" ]; then
            log_warn "PID $lock_pid holding dpkg lock: $(ps -p $lock_pid -o comm= 2>/dev/null || echo 'unknown')"
        fi
        sleep 5
        elapsed=$((elapsed + 5))
    done
    # Timeout reached: attempt recovery
    log_err "Apt locks held for over 5 minutes. Attempting recovery..."
    rm -f /var/lib/dpkg/lock-frontend /var/lib/apt/lists/lock /var/cache/apt/archives/lock
    dpkg --configure -a 2>/dev/null || true
    apt-get update || true
}

apt_with_retry() {
    local retries=3 delay=5
    for i in $(seq 1 $retries); do
        if apt-get "$@"; then
            return 0
        fi
        log_warn "apt attempt $i/$retries failed. Retrying in ${delay}s..."
        sleep $delay
        delay=$((delay * 3))
        wait_for_apt_locks
    done
    log_err "apt failed after $retries attempts"
    return 1
}

verify_service() {
    local service=$1 max_wait=${2:-15}
    log_info "Verifying $service is running..."
    for i in $(seq 1 $max_wait); do
        if systemctl is-active --quiet "$service" 2>/dev/null; then
            log_ok "$service is running"
            return 0
        fi
        sleep 1
    done
    log_err "$service failed to start within ${max_wait}s"
    echo "--- Recent journal entries for $service ---"
    journalctl -u "$service" --no-pager -n 20 || true
    return 1
}

generate_password() {
    openssl rand -base64 18 | tr -dc 'a-zA-Z0-9' | head -c 24
}

# -----------------------------------------------
# PRE-FLIGHT CHECKS
# -----------------------------------------------
pre_flight_checks() {
    log_info "Running pre-flight checks..."

    # Check root
    if [ "$EUID" -ne 0 ]; then
        log_err "This script must be run as root"
        exit 1
    fi

    # Check OS
    if ! lsb_release -a 2>/dev/null | grep -q "Ubuntu 24.04"; then
        log_warn "This script is designed for Ubuntu 24.04 LTS"
    fi

    # Check RAM and create swap if needed
    local total_ram=$(free -m | awk '/^Mem:/{print $2}')
    local total_swap=$(free -m | awk '/^Swap:/{print $2}')
    log_info "RAM: ${total_ram}MB, Swap: ${total_swap}MB"

    if [ "$total_ram" -lt 2048 ] && [ "$total_swap" -eq 0 ]; then
        log_warn "Low RAM detected (${total_ram}MB) and no swap. Creating 4GB swap..."
        fallocate -l 4G /swapfile
        chmod 600 /swapfile
        mkswap /swapfile
        swapon /swapfile
        echo '/swapfile none swap sw 0 0' >> /etc/fstab
        sysctl vm.swappiness=10
        echo 'vm.swappiness=10' >> /etc/sysctl.conf
        log_ok "4GB swap created and activated"
    fi

    # Check disk space
    local free_disk=$(df -m / | awk 'NR==2{print $4}')
    if [ "$free_disk" -lt 10240 ]; then
        log_err "Insufficient disk space: ${free_disk}MB free (10GB minimum required)"
        exit 1
    fi
    log_ok "Disk space OK: ${free_disk}MB free"

    # Check for port conflicts
    if ss -tlnp | grep -q ':80 '; then
        log_warn "Port 80 is already in use: $(ss -tlnp | grep ':80 ')"
    fi
    if ss -tlnp | grep -q ':443 '; then
        log_warn "Port 443 is already in use: $(ss -tlnp | grep ':443 ')"
    fi

    # Create backup directory
    mkdir -p "$BACKUP_DIR"
    log_ok "Backup directory: $BACKUP_DIR"
}

# -----------------------------------------------
# BACKUP EXISTING CONFIGURATION
# -----------------------------------------------
backup_existing() {
    log_info "Backing up existing configuration..."
    [ -d /etc/nginx ] && cp -r /etc/nginx "$BACKUP_DIR/" 2>/dev/null || true
    [ -f /etc/mysql/my.cnf ] && cp /etc/mysql/my.cnf "$BACKUP_DIR/" 2>/dev/null || true
    [ -d /etc/php ] && cp -r /etc/php "$BACKUP_DIR/" 2>/dev/null || true
    [ -f /etc/ufw/before.rules ] && cp /etc/ufw/before.rules "$BACKUP_DIR/" 2>/dev/null || true
    log_ok "Backup completed"
}

# -----------------------------------------------
# LOAD OR GENERATE CREDENTIALS
# -----------------------------------------------
load_credentials() {
    if [ -f "$CREDENTIALS_FILE" ]; then
        log_info "Loading existing credentials from $CREDENTIALS_FILE"
        source "$CREDENTIALS_FILE"
    else
        DB_PASSWORD=$(generate_password)
        MYSQL_ROOT_PASSWORD=$(generate_password)
        cat > "$CREDENTIALS_FILE" << EOF
# LEMP Stack Credentials
# Generated: $(date)
# DO NOT COMMIT THIS FILE TO VERSION CONTROL
DB_NAME=$DB_NAME
DB_USER=$DB_USER
DB_PASSWORD=$DB_PASSWORD
MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
DOMAIN=$DOMAIN
PHP_VERSION=$PHP_VERSION
EOF
        chmod 600 "$CREDENTIALS_FILE"
        log_ok "New credentials generated and saved to $CREDENTIALS_FILE"
    fi
}

# -----------------------------------------------
# FIX BROKEN MYSQL STATE
# -----------------------------------------------
fix_broken_mysql() {
    log_info "Checking for broken MySQL installation..."
    local mysql_state=$(dpkg -l mysql-server 2>/dev/null | awk 'NR==6{print $1}' || echo "")
    if [ "$mysql_state" = "iU" ]; then
        log_warn "MySQL in 'installed but unconfigured' (iU) state. Purging and reinstalling..."
        dpkg --configure -a || true
        apt-get purge -y mysql-server mysql-client mysql-common || true
        apt-get autoremove -y || true
        rm -rf /var/lib/mysql /etc/mysql
        log_ok "Broken MySQL state cleaned up"
    fi
}

# -----------------------------------------------
# INSTALL NGINX
# -----------------------------------------------
install_nginx() {
    log_info "Installing Nginx..."
    apt_with_retry update
    apt_with_retry install -y nginx
    systemctl enable nginx
    verify_service nginx
    log_ok "Nginx $(nginx -v 2>&1 | awk -F'/' '{print $2}') installed"
}

# -----------------------------------------------
# INSTALL AND SECURE MYSQL
# -----------------------------------------------
install_mysql() {
    log_info "Installing MySQL 8.0..."
    fix_broken_mysql
    apt_with_retry install -y mysql-server mysql-client

    # Wait for MySQL to be ready (up to 30s)
    verify_service mysql 30

    # Secure MySQL installation
    log_info "Securing MySQL..."
    mysql -uroot << EOF
ALTER USER 'root'@'localhost' IDENTIFIED WITH caching_sha2_password BY '$MYSQL_ROOT_PASSWORD';
DELETE FROM mysql.user WHERE User='';
DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1');
DROP DATABASE IF EXISTS test;
DELETE FROM mysql.db WHERE Db='test' OR Db='test\\_%';
FLUSH PRIVILEGES;
EOF

    # Create application database and user
    mysql -uroot -p"$MYSQL_ROOT_PASSWORD" << EOF
DROP DATABASE IF EXISTS $DB_NAME;
CREATE DATABASE $DB_NAME CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
DROP USER IF EXISTS '$DB_USER'@'localhost';
CREATE USER '$DB_USER'@'localhost' IDENTIFIED WITH caching_sha2_password BY '$DB_PASSWORD';
GRANT ALL PRIVILEGES ON $DB_NAME.* TO '$DB_USER'@'localhost';
FLUSH PRIVILEGES;
EOF

    # Enable auth_socket for root (safer local access)
    mysql -uroot -p"$MYSQL_ROOT_PASSWORD" << 'EOF'
ALTER USER 'root'@'localhost' IDENTIFIED WITH auth_socket;
FLUSH PRIVILEGES;
EOF

    log_ok "MySQL 8.0 installed and secured"
}

# -----------------------------------------------
# INSTALL PHP-FPM
# -----------------------------------------------
install_php() {
    log_info "Installing PHP $PHP_VERSION-FPM..."

    # Clean up old PHP versions if PHP_VERSION changed
    local current_php=$(php -v 2>/dev/null | head -1 | grep -oP '\d+\.\d+' || echo "")
    if [ -n "$current_php" ] && [ "$current_php" != "$PHP_VERSION" ]; then
        log_warn "Detected PHP $current_php, disabling for upgrade to $PHP_VERSION..."
        systemctl stop "php${current_php}-fpm" 2>/dev/null || true
        systemctl disable "php${current_php}-fpm" 2>/dev/null || true
    fi

    apt_with_retry install -y \
        "php${PHP_VERSION}-fpm" \
        "php${PHP_VERSION}-mysql" \
        "php${PHP_VERSION}-curl" \
        "php${PHP_VERSION}-gd" \
        "php${PHP_VERSION}-mbstring" \
        "php${PHP_VERSION}-xml" \
        "php${PHP_VERSION}-zip" \
        "php${PHP_VERSION}-intl" \
        "php${PHP_VERSION}-opcache" \
        "php${PHP_VERSION}-bcmath"

    # Harden php.ini
    local php_ini="/etc/php/${PHP_VERSION}/fpm/php.ini"
    cp "$php_ini" "$BACKUP_DIR/php.ini.bak"
    sed -i 's/^;*cgi.fix_pathinfo=.*/cgi.fix_pathinfo=0/' "$php_ini"
    sed -i 's/^;*upload_max_filesize.*/upload_max_filesize = 64M/' "$php_ini"
    sed -i 's/^;*post_max_size.*/post_max_size = 64M/' "$php_ini"
    sed -i 's/^;*memory_limit.*/memory_limit = 256M/' "$php_ini"
    sed -i 's/^;*max_execution_time.*/max_execution_time = 300/' "$php_ini"

    # Tune OPcache for production
    local opcache_ini="/etc/php/${PHP_VERSION}/fpm/conf.d/10-opcache.ini"
    cat >> "$opcache_ini" << 'EOF'
opcache.enable=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=20000
opcache.revalidate_freq=60
opcache.fast_shutdown=1
opcache.validate_timestamps=0
EOF

    systemctl restart "php${PHP_VERSION}-fpm"
    verify_service "php${PHP_VERSION}-fpm"
    log_ok "PHP $PHP_VERSION-FPM installed and tuned"
}

# -----------------------------------------------
# CONFIGURE NGINX VIRTUAL HOST
# -----------------------------------------------
configure_nginx() {
    log_info "Configuring Nginx for $DOMAIN..."

    local nginx_conf="/etc/nginx/sites-available/${DOMAIN}"
    local nginx_enabled="/etc/nginx/sites-enabled/${DOMAIN}"

    cat > "$nginx_conf" << EOF
server {
    listen 80;
    listen [::]:80;
    server_name $DOMAIN;
    root /var/www/$DOMAIN;
    index index.php index.html;

    location / {
        try_files \$uri \$uri/ /index.php?\$query_string;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php${PHP_VERSION}-fpm.sock;
        fastcgi_param SCRIPT_FILENAME \$document_root\$fastcgi_script_name;
        include fastcgi_params;
    }

    location ~ /\.ht {
        deny all;
    }

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
}
EOF

    mkdir -p "/var/www/$DOMAIN"
    chown -R www-data:www-data "/var/www/$DOMAIN"
    chmod 755 "/var/www/$DOMAIN"

    # Create test PHP file
    cat > "/var/www/$DOMAIN/index.php" << 'EOF'
<?php phpinfo(); ?>
EOF

    # Enable site
    rm -f /etc/nginx/sites-enabled/default
    ln -sf "$nginx_conf" "$nginx_enabled"

    nginx -t && systemctl reload nginx
    log_ok "Nginx configured for $DOMAIN"
}

# -----------------------------------------------
# CONFIGURE SSL WITH CERTBOT
# -----------------------------------------------
configure_ssl() {
    log_info "Installing Certbot and configuring SSL..."
    apt_with_retry install -y certbot python3-certbot-nginx

    if ! certbot certificates 2>/dev/null | grep -q "$DOMAIN"; then
        certbot --nginx -d "$DOMAIN" --non-interactive --agree-tos -m "$EMAIL" --redirect
        log_ok "SSL certificate obtained for $DOMAIN"
    else
        log_info "SSL certificate for $DOMAIN already exists"
    fi

    # Auto-renewal test
    certbot renew --dry-run
    log_ok "SSL auto-renewal verified"
}

# -----------------------------------------------
# CONFIGURE FIREWALL (UFW)
# -----------------------------------------------
configure_firewall() {
    log_info "Configuring UFW firewall..."
    apt_with_retry install -y ufw

    # Reset to known state (idempotent)
    ufw --force reset
    ufw default deny incoming
    ufw default allow outgoing

    # Allow SSH (critical - do not lock yourself out)
    ufw allow 22/tcp

    # Allow Nginx (HTTP + HTTPS)
    ufw allow 'Nginx Full'

    # Enable firewall
    ufw --force enable

    log_ok "UFW configured: SSH + Nginx Full allowed"
    ufw status verbose
}

# -----------------------------------------------
# CONFIGURE FAIL2BAN
# -----------------------------------------------
configure_fail2ban() {
    log_info "Installing and configuring Fail2ban..."
    apt_with_retry install -y fail2ban

    # Main configuration
    cat > /etc/fail2ban/jail.local << 'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5
backend = systemd

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log

[nginx-badbots]
enabled = true
filter = nginx-badbots
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 2

[nginx-noscript]
enabled = true
filter = nginx-noscript
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 6
EOF

    systemctl enable fail2ban
    systemctl restart fail2ban
    verify_service fail2ban
    log_ok "Fail2ban configured"
    fail2ban-client status
}

# -----------------------------------------------
# PRINT SUMMARY
# -----------------------------------------------
print_summary() {
    echo ""
    echo "============================================="
    echo "  LEMP STACK INSTALLATION COMPLETE"
    echo "============================================="
    echo "Domain:          https://$DOMAIN"
    echo "Web Root:        /var/www/$DOMAIN"
    echo "Database:        $DB_NAME"
    echo "DB User:         $DB_USER"
    echo "PHP Version:     $PHP_VERSION"
    echo "Credentials:     $CREDENTIALS_FILE"
    echo "Backups:         $BACKUP_DIR"
    echo "============================================="
    echo ""
    echo "Services:"
    systemctl is-active nginx &>/dev/null && echo "  [OK] Nginx" || echo "  [FAIL] Nginx"
    systemctl is-active mysql &>/dev/null && echo "  [OK] MySQL" || echo "  [FAIL] MySQL"
    systemctl is-active "php${PHP_VERSION}-fpm" &>/dev/null && echo "  [OK] PHP-FPM" || echo "  [FAIL] PHP-FPM"
    systemctl is-active fail2ban &>/dev/null && echo "  [OK] Fail2ban" || echo "  [FAIL] Fail2ban"
    echo "============================================="
}

# -----------------------------------------------
# MAIN EXECUTION
# -----------------------------------------------
main() {
    log_info "Starting LEMP stack installation..."
    pre_flight_checks
    backup_existing
    load_credentials
    install_nginx
    install_mysql
    install_php
    configure_nginx
    configure_ssl
    configure_firewall
    configure_fail2ban
    print_summary
    log_ok "Installation complete!"
}

main "$@"

3.5 Running the Script

3.5.1 Basic Execution

# Download and run the script as root
curl -o /root/setup_lemp_improved.sh https://your-repo/setup_lemp_improved.sh
chmod +x /root/setup_lemp_improved.sh
sudo /root/setup_lemp_improved.sh

3.5.2 Dry-Run Mode (Simulate Without Making Changes)

For a dry run, wrap the script in a function or use the --dry-run flag if implemented. Alternatively, review the pre-flight checks section which safely reports system state without modifying anything:

# The script's pre_flight_checks() function reads-only and is safe to run
# It reports: RAM, disk, ports in use, and apt lock status
bash -c 'source /root/setup_lemp_improved.sh && pre_flight_checks'

3.5.3 Skip Flags (Partial Re-Runs)

Because the script is idempotent, you can safely re-run it in full. However, if you want to skip specific sections during development, create a wrapper script that sources functions selectively:

# Example: Re-run only PHP configuration after tuning
source /root/setup_lemp_improved.sh
pre_flight_checks
install_php
configure_nginx

3.6 Troubleshooting Common Issues

3.6.1 Dpkg Lock Errors

Symptom: Could not get lock /var/lib/dpkg/lock-frontend

Cause: Another process (unattended-upgrades, another apt instance, or a crashed process) is holding the lock file.

Solution:

# Identify the process holding the lock
sudo lsof /var/lib/dpkg/lock-frontend
# Output example: COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
#                 unattended-upgr 1234 root   12uW  REG   0,23        0 1234 /var/lib/dpkg/lock-frontend

# Option 1: Wait for the process to finish
sudo watch lsof /var/lib/dpkg/lock-frontend

# Option 2: Kill the stuck process (use with caution)
sudo kill -9 <PID>

# Option 3: Force-clear stale locks
sudo rm -f /var/lib/dpkg/lock-frontend /var/lib/apt/lists/lock /var/cache/apt/archives/lock
sudo dpkg --configure -a
sudo apt-get update

# Option 4: Reboot (nuclear option when all else fails)
sudo reboot

3.6.2 MySQL Broken Installation (iU State)

Symptom: mysql-server package shows iU in dpkg -l output; MySQL service fails to start.

Cause: A previous installation was interrupted (network failure, OOM kill, manual abort), leaving MySQL in an inconsistent state.

Solution:

# Check MySQL package state
dpkg -l | grep mysql-server
# ii = installed and configured (good)
# iU = installed but unconfigured (bad)

# Full purge and reinstall
sudo dpkg --configure -a || true
sudo apt-get purge -y mysql-server mysql-client mysql-common
sudo apt-get autoremove -y
sudo rm -rf /var/lib/mysql /etc/mysql /var/log/mysql*
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client

# Verify
sudo systemctl status mysql
sudo mysql -e "SELECT 1;"

3.6.3 OOM Kills (Out of Memory)

Symptom: Services randomly dying, system log shows Out of memory: Kill process <PID> (OOM killer invoked).

Cause: Insufficient RAM for running services. OpenStack control plane alone consumes approximately 3-4 GB.

Solution:

# Check OOM killer log entries
sudo dmesg | grep -i "out of memory\|oom\|killed process"
sudo grep -i "oom" /var/log/syslog

# Check current memory usage
free -h
vmstat -s | head -5

# Create swap if not present (see Section 2.3)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Reduce MySQL memory footprint (if needed)
sudo mysql -e "SET GLOBAL innodb_buffer_pool_size = 536870912;"  # 512MB

# Add to my.cnf for persistence
echo -e "[mysqld]\ninnodb_buffer_pool_size = 512M" | sudo tee /etc/mysql/mysql.conf.d/low-memory.cnf
sudo systemctl restart mysql

3.6.4 Port Conflicts

Symptom: Nginx fails to start with bind() to 0.0.0.0:80 failed (98: Address already in use)

Cause: Another process (Apache, another Nginx instance, or OpenStack service) is already bound to port 80 or 443.

Solution:

# Identify what is using ports 80 and 443
sudo ss -tlnp | grep -E ':80|:443'
sudo lsof -i :80
sudo lsof -i :443

# If Apache is running, stop and disable it
sudo systemctl stop apache2
sudo systemctl disable apache2

# Restart Nginx
sudo systemctl restart nginx
sudo systemctl status nginx

3.7 Verification Commands

After the LEMP installation completes, verify each component:

3.7.1 Nginx

# Check service status
sudo systemctl is-active nginx
sudo systemctl status nginx --no-pager

# Check configuration syntax
sudo nginx -t

# Check version
nginx -v

# Test HTTP response
curl -I http://openstack.kuyaops.com
# Expected: HTTP/1.1 200 OK or 301 redirect to HTTPS

# Test HTTPS response
curl -I https://openstack.kuyaops.com
# Expected: HTTP/2 200 with valid SSL certificate

3.7.2 MySQL

# Check service status
sudo systemctl is-active mysql

# Login as root (uses auth_socket - no password needed from root user)
sudo mysql -e "SELECT VERSION();"

# Login as application user
sudo mysql -u openstack_user -p -e "SHOW DATABASES;"

# Check user authentication methods
sudo mysql -e "SELECT user, host, plugin FROM mysql.user;"
# Expected: root@localhost -> auth_socket, openstack_user@localhost -> caching_sha2_password

3.7.3 PHP-FPM

# Check service status
sudo systemctl is-active php8.3-fpm

# Check version
php -v

# Test PHP processing via Nginx
curl -s https://openstack.kuyaops.com/index.php | grep -oP 'PHP Version \d+\.\d+'

# Verify OPcache is enabled
php -i | grep opcache

# Check FPM pool socket
ls -la /run/php/php8.3-fpm.sock

3.7.4 Firewall (UFW)

# Check firewall status
sudo ufw status verbose
# Expected: 22/tcp ALLOW, Nginx Full ALLOW, default deny incoming

3.7.5 Fail2ban

# Check service status
sudo systemctl is-active fail2ban

# View active jails
sudo fail2ban-client status

# View sshd jail details
sudo fail2ban-client status sshd

4. OpenStack Installation

4.1 MicroStack vs. Full OpenStack

MicroStack is Canonical's streamlined, snap-packaged distribution of OpenStack. It bundles all essential OpenStack services into a single, self-contained snap that can be installed on a single machine. This is in contrast to a full OpenStack deployment (using tools like Kolla-Ansible, OpenStack-Ansible, or Charms) which requires multiple servers and complex orchestration.

4.2 Why All-in-One for Development

An all-in-one deployment places the controller, compute, network, and storage services on a single node. While this violates production best practices, it is ideal for:

• Learning OpenStack — Understand all services on one machine
• CI/CD testing — Spin up VMs for integration tests
• Proof of concept — Validate OpenStack fits your workflow
• Development sandbox — Test applications in isolated VMs

4.3 Script Overview: install_openstack.sh

The OpenStack installation is automated via a bash script that handles pre-flight checks, MicroStack installation, networking configuration, image setup, and security group rules.

4.3.1 Key Features

4.4 Complete OpenStack Installation Script

Save the following script as /root/install_openstack.sh:

#!/bin/bash
# =============================================================================
# OpenStack (MicroStack) Installation Script for Ubuntu 24.04
# Version: 2.0 (Production-Ready with Full Error Handling)
# Description: All-in-one OpenStack deployment with LEMP integration
# =============================================================================
set -euo pipefail

# -----------------------------------------------
# CONFIGURATION VARIABLES
# -----------------------------------------------
DOMAIN="openstack.kuyaops.com"
EXTERNAL_IF="eth0"                    # Adjust to your primary interface
FLOATING_IP_START="10.20.20.100"      # Start of floating IP pool
FLOATING_IP_END="10.20.20.200"      # End of floating IP pool
GATEWAY_IP="10.20.20.1"             # MicroStack default gateway
DNS_SERVERS="8.8.8.8,1.1.1.1"       # External DNS for VMs
INTERNAL_CIDR="192.168.222.0/24"    # Private tenant network
IMAGE_URL="https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
IMAGE_NAME="ubuntu-22.04"
SSH_KEY="$HOME/.ssh/openstack_key"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log_info() { echo -e "${BLUE}[INFO]${NC}  $1"; }
log_ok()   { echo -e "${GREEN}[OK]${NC}    $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC}  $1"; }
log_err()  { echo -e "${RED}[ERROR]${NC} $1"; }

# -----------------------------------------------
# PRE-FLIGHT CHECKS
# -----------------------------------------------
pre_flight_checks() {
    log_info "Running OpenStack pre-flight checks..."

    if [ "$EUID" -ne 0 ]; then
        log_err "Must be run as root"
        exit 1
    fi

    # Check RAM (8GB minimum)
    local total_ram=$(free -m | awk '/^Mem:/{print $2}')
    if [ "$total_ram" -lt 7168 ]; then
        log_warn "RAM is ${total_ram}MB. 8GB+ strongly recommended for OpenStack."
        read -p "Continue anyway? (y/N) " -n 1 -r
        echo
        [[ ! $REPLY =~ ^[Yy]$ ]] && exit 1
    fi

    # Check disk space (40GB free minimum)
    local free_disk=$(df -m / | awk 'NR==2{print $4}')
    if [ "$free_disk" -lt 40960 ]; then
        log_err "Insufficient disk space: ${free_disk}MB free (40GB minimum)"
        exit 1
    fi

    # Check CPU
    local cpu_count=$(nproc)
    if [ "$cpu_count" -lt 2 ]; then
        log_err "At least 2 vCPUs required (found $cpu_count)"
        exit 1
    fi
    log_ok "Resources: ${cpu_count} vCPU, ${total_ram}MB RAM, ${free_disk}MB disk free"

    # Check nested virtualization
    if [ -e /dev/kvm ]; then
        log_ok "KVM nested virtualization available (/dev/kvm)"
    else
        log_warn "KVM not available. VMs will use QEMU (slower performance)."
        log_warn "To enable KVM: shutdown droplet, enable nested virt in DO panel, reboot."
    fi

    # Check for LEMP conflicts
    if ss -tlnp | grep -q ':80 '; then
        log_info "Port 80 in use by: $(ss -tlnp | grep ':80 ' | awk '{print $6}')"
        log_info "Nginx will proxy to Horizon; this is expected."
    fi

    # Verify snapd is available
    if ! command -v snap &>/dev/null; then
        log_err "snapd is not installed. Install with: apt install -y snapd"
        exit 1
    fi
    log_ok "snapd is available"
}

# -----------------------------------------------
# INSTALL MICROSTACK
# -----------------------------------------------
install_microstack() {
    log_info "Installing MicroStack via snap..."

    if snap list | grep -q microstack; then
        log_info "MicroStack already installed. Refreshing..."
        snap refresh microstack
    else
        snap install microstack --classic
    fi

    log_ok "MicroStack installed"
    snap list microstack
}

# -----------------------------------------------
# INITIALIZE MICROSTACK
# -----------------------------------------------
init_microstack() {
    log_info "Initializing MicroStack (this takes 10-20 minutes)..."

    if [ -f /var/snap/microstack/common/etc/nova/nova.conf ]; then
        log_info "MicroStack appears already initialized. Skipping init."
    else
        microstack init --auto --control
    fi

    log_ok "MicroStack initialized"

    # Verify key services
    local services="nova-compute neutron-server keystone glance-api"
    for svc in $services; do
        if snap services microstack | grep -q "$svc"; then
            log_ok "Service $svc is present"
        else
            log_warn "Service $svc not found in MicroStack"
        fi
    done
}

# -----------------------------------------------
# CONFIGURE NETWORKING
# -----------------------------------------------
configure_networking() {
    log_info "Configuring OpenStack networking..."

    local openstack="microstack.openstack"

    # Create external (provider) network
    if ! $openstack network list | grep -q external; then
        $openstack network create --external --provider-network-type flat \
            --provider-physical-network physnet1 external
        log_ok "External network created"
    else
        log_info "External network already exists"
    fi

    # Create external subnet
    if ! $openstack subnet list | grep -q external-subnet; then
        $openstack subnet create --network external --subnet-range 10.20.20.0/24 \
            --gateway 10.20.20.1 --allocation-pool start=$FLOATING_IP_START,end=$FLOATING_IP_END \
            --dns-nameserver 8.8.8.8 --dns-nameserver 1.1.1.1 external-subnet
        log_ok "External subnet created"
    else
        log_info "External subnet already exists"
    fi

    # Create internal (tenant) network
    if ! $openstack network list | grep -q internal; then
        $openstack network create internal
        log_ok "Internal network created"
    else
        log_info "Internal network already exists"
    fi

    # Create internal subnet
    if ! $openstack subnet list | grep -q internal-subnet; then
        $openstack subnet create --network internal --subnet-range $INTERNAL_CIDR \
            --gateway 192.168.222.1 --dns-nameserver 8.8.8.8 internal-subnet
        log_ok "Internal subnet created"
    else
        log_info "Internal subnet already exists"
    fi

    # Create router
    if ! $openstack router list | grep -q main-router; then
        $openstack router create main-router
        $openstack router set --external-gateway external main-router
        $openstack router add subnet main-router internal-subnet
        log_ok "Router 'main-router' created with NAT"
    else
        log_info "Router already exists"
    fi

    log_ok "Networking configured"
}

# -----------------------------------------------
# UPLOAD CLOUD IMAGE
# -----------------------------------------------
upload_image() {
    log_info "Uploading Ubuntu 22.04 cloud image..."
    local openstack="microstack.openstack"

    if $openstack image list | grep -q "$IMAGE_NAME"; then
        log_info "Image '$IMAGE_NAME' already exists"
        return 0
    fi

    local tmp_img="/tmp/${IMAGE_NAME}.img"
    if [ ! -f "$tmp_img" ]; then
        log_info "Downloading cloud image..."
        wget -O "$tmp_img" "$IMAGE_URL"
    fi

    $openstack image create --disk-format qcow2 --container-format bare \
        --file "$tmp_img" --public "$IMAGE_NAME"

    log_ok "Image '$IMAGE_NAME' uploaded to Glance"
    $openstack image list
}

# -----------------------------------------------
# CREATE FLAVORS
# -----------------------------------------------
create_flavors() {
    log_info "Creating VM flavors..."
    local openstack="microstack.openstack"

    declare -A flavors=(
        ["m1.tiny"]="512 1 10"
        ["m1.small"]="1024 1 20"
        ["m1.medium"]="2048 2 40"
        ["m1.large"]="4096 4 80"
        ["m1.xlarge"]="8192 8 160"
    )

    for name in "${!flavors[@]}"; do
        read -r ram vcpu disk <<< "${flavors[$name]}"
        if $openstack flavor list | grep -q "$name"; then
            log_info "Flavor '$name' already exists"
        else
            $openstack flavor create --ram $ram --vcpus $vcpu --disk $disk "$name"
            log_ok "Flavor '$name' created: ${ram}MB RAM, ${vcpu} vCPU, ${disk}GB disk"
        fi
    done
}

# -----------------------------------------------
# CONFIGURE SECURITY GROUPS
# -----------------------------------------------
configure_security_groups() {
    log_info "Configuring security groups..."
    local openstack="microstack.openstack"

    local default_sg=$($openstack security group list --project admin -c ID -f value | head -1)

    if [ -z "$default_sg" ]; then
        log_err "Could not find default security group"
        return 1
    fi

    # SSH (port 22)
    if ! $openstack security group rule list "$default_sg" | grep -q '22:22'; then
        $openstack security group rule create --protocol tcp --dst-port 22:22 \
            --remote-ip 0.0.0.0/0 "$default_sg"
        log_ok "SSH (22) rule added"
    fi

    # HTTP (port 80)
    if ! $openstack security group rule list "$default_sg" | grep -q '80:80'; then
        $openstack security group rule create --protocol tcp --dst-port 80:80 \
            --remote-ip 0.0.0.0/0 "$default_sg"
        log_ok "HTTP (80) rule added"
    fi

    # HTTPS (port 443)
    if ! $openstack security group rule list "$default_sg" | grep -q '443:443'; then
        $openstack security group rule create --protocol tcp --dst-port 443:443 \
            --remote-ip 0.0.0.0/0 "$default_sg"
        log_ok "HTTPS (443) rule added"
    fi

    # ICMP (ping)
    if ! $openstack security group rule list "$default_sg" | grep -q 'icmp'; then
        $openstack security group rule create --protocol icmp \
            --remote-ip 0.0.0.0/0 "$default_sg"
        log_ok "ICMP rule added"
    fi

    log_ok "Security groups configured"
}

# -----------------------------------------------
# GENERATE SSH KEYPAIR
# -----------------------------------------------
generate_ssh_key() {
    log_info "Generating SSH keypair..."

    if [ ! -f "$SSH_KEY" ]; then
        ssh-keygen -t ed25519 -f "$SSH_KEY" -N "" -C "openstack@$(hostname)"
        chmod 600 "$SSH_KEY"
        chmod 644 "${SSH_KEY}.pub"
        log_ok "SSH keypair generated: $SSH_KEY"
    else
        log_info "SSH keypair already exists: $SSH_KEY"
    fi

    # Import into OpenStack
    local openstack="microstack.openstack"
    if ! $openstack keypair list | grep -q openstack_key; then
        $openstack keypair create --public-key "${SSH_KEY}.pub" openstack_key
        log_ok "Public key imported into OpenStack as 'openstack_key'"
    else
        log_info "Keypair 'openstack_key' already exists"
    fi
}

# -----------------------------------------------
# LAUNCH TEST VM
# -----------------------------------------------
launch_test_vm() {
    log_info "Launching test VM to verify deployment..."
    local openstack="microstack.openstack"

    local vm_name="test-vm"

    # Delete existing test VM if present
    if $openstack server list | grep -q "$vm_name"; then
        log_info "Deleting existing test VM..."
        $openstack server delete "$vm_name"
        sleep 5
    fi

    # Launch VM
    $openstack server create \
        --image "$IMAGE_NAME" \
        --flavor m1.small \
        --network internal \
        --key-name openstack_key \
        --security-group default \
        "$vm_name"

    log_info "Waiting for VM to become active (this may take 2-5 minutes)..."
    local attempts=0
    while [ $attempts -lt 30 ]; do
        local status=$($openstack server show "$vm_name" -c status -f value)
        if [ "$status" = "ACTIVE" ]; then
            log_ok "VM '$vm_name' is ACTIVE"
            break
        fi
        log_info "VM status: $status (attempt $attempts/30)"
        sleep 10
        attempts=$((attempts + 1))
    done

    # Allocate and assign floating IP
    log_info "Allocating floating IP..."
    local fip=$($openstack floating ip create external -c floating_ip_address -f value)
    $openstack server add floating ip "$vm_name" "$fip"

    log_ok "Test VM launched with floating IP: $fip"
    echo ""
    echo "============================================="
    echo "  CONNECT TO TEST VM:"
    echo "  ssh -i $SSH_KEY ubuntu@$fip"
    echo "============================================="
}

# -----------------------------------------------
# CONFIGURE NGINX REVERSE PROXY FOR HORIZON
# -----------------------------------------------
configure_nginx_proxy() {
    log_info "Configuring Nginx reverse proxy for Horizon..."

    local horizon_conf="/etc/nginx/sites-available/openstack-horizon"

    cat > "$horizon_conf" << 'EOF'
# OpenStack Horizon Dashboard Proxy
server {
    listen 80;
    server_name openstack.kuyaops.com;

    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name openstack.kuyaops.com;

    # SSL certificates (managed by Certbot)
    ssl_certificate /etc/letsencrypt/live/openstack.kuyaops.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/openstack.kuyaops.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/openstack.kuyaops.com/chain.pem;

    # SSL hardening
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # Proxy to Horizon (MicroStack internal IP)
    location / {
        proxy_pass http://10.20.20.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support (for VNC console)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        proxy_read_timeout 300;
    }

    # Static files (optional optimization)
    location /static {
        proxy_pass http://10.20.20.1/static;
        proxy_cache_valid 200 1h;
    }
}
EOF

    ln -sf "$horizon_conf" /etc/nginx/sites-enabled/openstack-horizon
    nginx -t && systemctl reload nginx
    log_ok "Nginx proxy for Horizon configured"
}

# -----------------------------------------------
# PRINT SUMMARY
# -----------------------------------------------
print_summary() {
    local openstack="microstack.openstack"
    local admin_password=$(snap get microstack config.credentials.keystone-password 2>/dev/null || echo "unknown")

    echo ""
    echo "============================================="
    echo "  OPENSTACK INSTALLATION COMPLETE"
    echo "============================================="
    echo "Horizon Dashboard:  https://$DOMAIN"
    echo "Internal Horizon:   http://10.20.20.1"
    echo ""
    echo "Login Credentials:"
    echo "  Username: admin"
    echo "  Password: $admin_password"
    echo ""
    echo "Networks:"
    $openstack network list
    echo ""
    echo "Images:"
    $openstack image list
    echo ""
    echo "Flavors:"
    $openstack flavor list
    echo ""
    echo "SSH Key: $SSH_KEY"
    echo "============================================="
}

# -----------------------------------------------
# MAIN EXECUTION
# -----------------------------------------------
main() {
    log_info "Starting OpenStack installation..."
    pre_flight_checks
    install_microstack
    init_microstack
    configure_networking
    upload_image
    create_flavors
    configure_security_groups
    generate_ssh_key
    configure_nginx_proxy
    launch_test_vm
    print_summary
    log_ok "OpenStack installation complete!"
}

main "$@"

4.5 Running the Installation

# Download the script
curl -o /root/install_openstack.sh https://your-repo/install_openstack.sh
chmod +x /root/install_openstack.sh

# Run the installation (takes 20-40 minutes)
sudo /root/install_openstack.sh

# Monitor the installation in another terminal
sudo tail -f /var/log/syslog | grep -E "microstack|nova|neutron|glance"

4.6 Post-Installation Configuration

4.6.1 Networking Architecture

After installation, your OpenStack network topology looks like this:

Internet
    |
    | (Floating IPs: 10.20.20.100-200)
    v
+------------+         +------------+         +------------+
|  External  |<------->|   Router   |<------->|  Internal  |
|  Network   |  NAT    |  (main)    |  Route  |  Network   |
|10.20.20.0/24|       |            |         |192.168.222.0/24|
+------------+         +------------+         +------------+
                                                    |
                                              +-----+-----+
                                              |           |
                                          +---v---+   +---v---+
                                          | VM 1  |   | VM 2  |
                                          |10.20.20.150|  |192.168.222.5|
                                          +--------+   +--------+

4.6.2 Verify Networking

# List networks
microstack.openstack network list
# Expected: external (provider), internal (private tenant)

# List subnets
microstack.openstack subnet list
# Expected: external-subnet, internal-subnet

# List routers
microstack.openstack router list
# Expected: main-router with external gateway

# Verify router NAT
microstack.openstack router show main-router -c external_gateway_info

4.6.3 Verify Images

# List uploaded images
microstack.openstack image list
# Expected: ubuntu-22.04 in 'active' state

# Show image details
microstack.openstack image show ubuntu-22.04

4.6.4 Verify Flavors

# List flavors
microstack.openstack flavor list
# Expected: m1.tiny, m1.small, m1.medium, m1.large, m1.xlarge

4.6.5 Verify Security Groups

# List security group rules
microstack.openstack security group rule list default
# Expected: TCP 22, 80, 443 from 0.0.0.0/0; ICMP from 0.0.0.0/0

4.6.6 Horizon Dashboard Access

Once the installation completes, access the Horizon dashboard:

# URL (via Nginx proxy with SSL)
https://openstack.kuyaops.com

# URL (direct, internal only)
http://10.20.20.1

# Default credentials
Username: admin
Password: (auto-generated; retrieve with:)
snap get microstack config.credentials.keystone-password

4.7 Nginx Reverse Proxy and SSL Configuration

The Horizon dashboard listens on http://10.20.20.1 (MicroStack's internal IP). Nginx acts as a reverse proxy, providing SSL termination and public access through https://openstack.kuyaops.com.

4.7.1 Proxy Configuration Details

The Nginx proxy configuration handles:

• SSL termination — HTTPS from clients, HTTP to Horizon
• WebSocket proxying — Required for the VNC console in Horizon
• Security headers — X-Forwarded-* headers for proper request handling
• HTTP/2 support — For improved performance
• Connection timeouts — Extended to 300s for long-running operations

4.7.2 Renewing SSL Certificates

# Test auto-renewal
certbot renew --dry-run

# Force renewal if needed
certbot renew --force-renewal

# Reload Nginx after renewal
certbot renew --deploy-hook "systemctl reload nginx"

# Verify certificate expiry date
echo | openssl s_client -servername openstack.kuyaops.com -connect openstack.kuyaops.com:443 2>/dev/null | openssl x509 -noout -dates

4.8 Launching Your First VM

After the test VM is launched by the script, you can manually create additional VMs:

# Step 1: Create a VM
microstack.openstack server create \
    --image ubuntu-22.04 \
    --flavor m1.medium \
    --network internal \
    --key-name openstack_key \
    --security-group default \
    my-first-vm

# Step 2: Wait for ACTIVE status
watch microstack.openstack server list

# Step 3: Allocate a floating IP
FIP=$(microstack.openstack floating ip create external -c floating_ip_address -f value)
microstack.openstack server add floating ip my-first-vm "$FIP"
echo "Floating IP: $FIP"

# Step 4: Connect via SSH
ssh -i ~/.ssh/openstack_key ubuntu@$FIP

# Step 5: Verify VM can reach the internet
ping -c 3 8.8.8.8

curl -I http://archive.ubuntu.com

5. Architecture Deep Dive

5.1 Single Node vs. Multi-Node

The all-in-one deployment is excellent for learning and development, but production workloads require a distributed architecture. This section compares the two approaches.

5.1.1 All-in-One (Current Deployment)

+-----------------------------------------------------+
|              DigitalOcean Droplet                    |
|              4 vCPU / 8 GB RAM                       |
|                                                      |
|  +----------------+  +----------------+             |
|  |   Controller   |  |    Compute     |             |
|  |   Services     |  |    (KVM/QEMU)  |             |
|  |                |  |                |             |
|  |  Keystone      |  |  Nova Compute  |             |
|  |  Nova API      |  |  Libvirt       |             |
|  |  Neutron       |  |  QEMU/KVM      |             |
|  |  Glance        |  +----------------+             |
|  |  Horizon       |                                 |
|  |  MySQL (int.)  |  +----------------+             |
|  |  RabbitMQ      |  |   Networking   |             |
|  +----------------+  |   (OVS/Linux   |             |
|                      |    Bridge)     |             |
|                      +----------------+             |
+-----------------------------------------------------+

5.1.2 Production Multi-Node (Recommended for Scale)

+-----------------------------------------------------+
|                 CONTROLLER NODE                      |
|              2 vCPU / 4 GB RAM                       |
|                                                      |
|  Keystone + Glance + Nova API + Neutron Server       |
|  Horizon Dashboard + MySQL + RabbitMQ                |
+-----------------------------------------------------+
                         |
+-----------------------------------------------------+
|                 COMPUTE NODE 1                       |
|              4 vCPU / 8 GB RAM                       |
|                                                      |
|  Nova Compute + Libvirt + KVM + Neutron Agent        |
|  (Can run 4-6 small VMs)                             |
+-----------------------------------------------------+
                         |
+-----------------------------------------------------+
|                 COMPUTE NODE 2                       |
|              4 vCPU / 8 GB RAM                       |
|                                                      |
|  Nova Compute + Libvirt + KVM + Neutron Agent        |
|  (Additional capacity for horizontal scaling)        |
+-----------------------------------------------------+

5.2 When to Separate Components

5.3 Cost Analysis

5.4 Nested Virtualization Considerations

Nested virtualization allows VMs running inside OpenStack (which itself runs on a DigitalOcean droplet) to use hardware-accelerated CPU virtualization extensions. Without it, OpenStack falls back to QEMU software emulation, which is significantly slower.

5.4.1 Check Nested Virtualization Support

# Check if /dev/kvm exists
ls -la /dev/kvm
# If present: KVM is available

# Check CPU flags
grep -E "vmx|svm" /proc/cpuinfo
# vmx = Intel VT-x support
# svm = AMD-V support

# Check if KVM module is loaded
lsmod | grep kvm

5.4.2 Enabling Nested Virtualization on DigitalOcean

Nested virtualization is not enabled by default on all DigitalOcean droplet plans. It is typically available on:

• CPU-Optimized droplets (all sizes)
• General Purpose droplets (4 vCPU and above)

# If KVM is not available, you must:
# 1. Power off the droplet
sudo poweroff

# 2. In the DigitalOcean control panel:
#    - Navigate to Droplets > Your Droplet > Kernel
#    - Select a kernel with KVM support
#    - Power the droplet back on

# 3. After reboot, verify:
ls -la /dev/kvm

5.4.3 QEMU Fallback Performance

If KVM is not available, QEMU software emulation is used. Expect:

• 50-70% slower CPU performance
• Longer VM boot times (2-5x)
• Higher CPU usage on the host

For development and testing, QEMU is acceptable. For production workloads, KVM is mandatory.

5.5 RAM Budget Breakdown (8 GB Droplet)

Understanding how RAM is allocated helps you plan VM capacity and troubleshoot OOM issues.

With approximately 3-4 GB available for VMs, you can run:

• 1x m1.medium (2 GB) + 2x m1.small (1 GB each) = 4 GB (with some swap)
• 3x m1.small (1 GB each) = 3 GB
• 6x m1.tiny (512 MB each) = 3 GB

6. Access and Management

6.1 Horizon Dashboard Access

6.1.1 Logging Into Horizon

1. Navigate to https://openstack.kuyaops.com
2. Enter username: admin
3. Enter password (retrieve with snap command above)
4. Select domain: default
5. Click "Sign In"

6.2 OpenStack CLI Commands

MicroStack provides a wrapped OpenStack CLI via microstack.openstack. All standard OpenStack CLI commands work through this wrapper.

6.2.1 Authentication

# Set credentials for CLI (run once per session)
export OS_USERNAME=admin
export OS_PASSWORD=$(snap get microstack config.credentials.keystone-password)
export OS_AUTH_URL=http://10.20.20.1:5000/v3
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3

# Verify authentication
microstack.openstack token issue

6.2.2 Server (VM) Management

# List all VMs
microstack.openstack server list

# Show VM details
microstack.openstack server show <vm-name>

# Create a VM
microstack.openstack server create \
    --image ubuntu-22.04 \
    --flavor m1.small \
    --network internal \
    --key-name openstack_key \
    my-new-vm

# Start / stop / reboot a VM
microstack.openstack server start <vm-name>
microstack.openstack server stop <vm-name>
microstack.openstack server reboot <vm-name>

# Delete a VM
microstack.openstack server delete <vm-name>

# View VM console log
microstack.openstack console log show <vm-name>

# Get VNC console URL
microstack.openstack console url show --vnc <vm-name>

6.2.3 Floating IP Management

# List floating IPs
microstack.openstack floating ip list

# Allocate a new floating IP
microstack.openstack floating ip create external

# Associate floating IP with VM
microstack.openstack server add floating ip <vm-name> <floating-ip>

# Remove floating IP
microstack.openstack server remove floating ip <vm-name> <floating-ip>

# Release floating IP
microstack.openstack floating ip delete <floating-ip>

6.2.4 Image Management

# List images
microstack.openstack image list

# Upload a new image
microstack.openstack image create \
    --disk-format qcow2 \
    --container-format bare \
    --file /path/to/image.qcow2 \
    my-custom-image

# Delete an image
microstack.openstack image delete <image-name>

# Show image details
microstack.openstack image show <image-name>

6.2.5 Network and Security Management

# List networks
microstack.openstack network list

# List subnets
microstack.openstack subnet list

# List routers
microstack.openstack router list

# List security groups
microstack.openstack security group list

# List security group rules
microstack.openstack security group rule list <group-name>

# Create a new security group rule
microstack.openstack security group rule create \
    --protocol tcp --dst-port 8080:8080 \
    --remote-ip 0.0.0.0/0 \
    <security-group-id>

6.3 SSH Access to VMs

VMs on the internal network are accessible via their floating IPs using the SSH keypair generated during installation.

# Basic SSH connection (Ubuntu cloud images use 'ubuntu' user)
ssh -i ~/.ssh/openstack_key ubuntu@<floating-ip>

# With verbose output (for troubleshooting)
ssh -vvv -i ~/.ssh/openstack_key ubuntu@<floating-ip>

# If you get "UNPROTECTED PRIVATE KEY FILE" warning
chmod 600 ~/.ssh/openstack_key
ssh -i ~/.ssh/openstack_key ubuntu@<floating-ip>

# Using SSH config (add to ~/.ssh/config for convenience)
cat >> ~/.ssh/config << 'EOF'
Host openstack-vm
    HostName <floating-ip>
    User ubuntu
    IdentityFile ~/.ssh/openstack_key
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

# Then connect simply with:
ssh openstack-vm

6.4 Common Management Tasks

6.4.1 Check OpenStack Service Status

# List all MicroStack snap services
snap services microstack

# Check individual service status
snap services microstack | grep -E "nova|neutron|keystone|glance"

6.4.2 Restart OpenStack Services

# Restart a specific service
sudo snap restart microstack.nova-compute
sudo snap restart microstack.neutron-api
sudo snap restart microstack.keystone

# Restart all MicroStack services
sudo snap restart microstack

6.4.3 View Service Logs

# View logs for a specific service
sudo snap logs microstack.nova-compute
sudo snap logs microstack.neutron-server
sudo snap logs microstack.keystone

# Follow logs in real-time
sudo snap logs microstack.nova-compute -f

6.4.4 Resource Quotas

# Show current quotas
microstack.openstack quota show

# Update quotas (e.g., allow 20 instances)
microstack.openstack quota set --instances 20 admin

6.4.5 Backup and Export MicroStack

# Export MicroStack configuration
snap get microstack > /root/microstack-config-backup.txt

# Backup database
sudo cp /var/snap/microstack/common/var/lib/mysql /root/microstack-mysql-backup -r

# Backup Glance images
sudo cp /var/snap/microstack/common/var/lib/glance/images /root/microstack-images-backup -r

7. Troubleshooting Guide

This section provides systematic troubleshooting steps for the most common issues encountered during and after OpenStack deployment.

7.1 Service Not Starting

7.1.1 MicroStack Service Fails to Start

# 1. Check the service status
snap services microstack | grep -E "SERVICE|nova|neutron|keystone"

# 2. View recent logs for the failing service
sudo snap logs microstack.<service-name>
# Example:
sudo snap logs microstack.nova-compute

# 3. Check for port conflicts
sudo ss -tlnp | grep -E "8774|9696|5000|9292"

# 4. Restart the service
sudo snap restart microstack.<service-name>

# 5. If all else fails, restart all MicroStack services
sudo snap restart microstack

# 6. For persistent failures, check the system log
sudo journalctl -u snap.microstack.* --no-pager -n 50

7.1.2 MySQL Service Inside MicroStack Fails

# Check MySQL process
ps aux | grep mysql

# Check MySQL socket
ls -la /var/snap/microstack/common/var/lib/mysql/mysql.sock

# Restart MySQL within MicroStack
sudo snap restart microstack.mysql

# If corrupted, check disk space (MySQL fails if disk is full)
df -h /var/snap/microstack/

# Emergency: Reset MicroStack (destroys all data)
sudo snap remove microstack --purge
sudo snap install microstack --classic
microstack init --auto --control

7.1.3 Nginx Fails to Start

# Check Nginx configuration syntax
sudo nginx -t

# Check for port conflicts
sudo ss -tlnp | grep -E ':80|:443'

# Check Nginx error log
sudo tail -n 30 /var/log/nginx/error.log

# Check for SELinux/AppArmor denials
sudo dmesg | grep -i nginx
sudo aa-status | grep nginx

# Fix: If another service binds to port 80/443
sudo systemctl stop apache2  # or other conflicting service
sudo systemctl disable apache2
sudo systemctl restart nginx

7.2 Network Issues

7.2.1 VMs Cannot Reach the Internet

# 1. Verify the external network is configured correctly
microstack.openstack network show external
# Look for: provider:physical_network = physnet1
#           router:external = true

# 2. Verify the router has an external gateway
microstack.openstack router show main-router -c external_gateway_info
# Should show: "network_id": "<external-network-uuid>"

# 3. Verify the router is connected to the internal subnet
microstack.openstack router show main-router -c interfaces_info

# 4. Check if NAT is working on the host
sudo iptables -t nat -L -n -v | grep MASQUERADE

# 5. Check IP forwarding is enabled
sysctl net.ipv4.ip_forward
# Expected: net.ipv4.ip_forward = 1

# 6. If IP forwarding is disabled, enable it:
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf

# 7. Verify the VM has a default route
ssh -i ~/.ssh/openstack_key ubuntu@<floating-ip> 'ip route show'
# Expected: default via 192.168.222.1

7.2.2 Cannot Assign Floating IP

# 1. Check available floating IPs
microstack.openstack floating ip list

# 2. Check the external network's allocation pool
microstack.openstack subnet show external-subnet -c allocation_pools

# 3. If no IPs available, create more by expanding the pool
microstack.openstack subnet set --allocation-pool start=10.20.20.201,end=10.20.20.250 external-subnet

# 4. Verify the VM's port is on the internal network
microstack.openstack port list --device-id <vm-uuid>

7.2.3 DNS Resolution Fails Inside VMs

# 1. Check DNS configuration on the internal subnet
microstack.openstack subnet show internal-subnet -c dns_nameservers

# 2. Update DNS servers if needed
microstack.openstack subnet set --dns-nameserver 8.8.8.8 --dns-nameserver 1.1.1.1 internal-subnet

# 3. Inside the VM, test DNS resolution
ssh -i ~/.ssh/openstack_key ubuntu@<floating-ip>
systemd-resolve --status
cat /etc/resolv.conf
nslookup google.com 8.8.8.8

7.3 VM Won't Boot

7.3.1 VM Stuck in BUILD or ERROR State

# 1. Check VM status and fault message
microstack.openstack server show <vm-name> -c status -c fault

# 2. Check Nova compute logs
sudo snap logs microstack.nova-compute -n 50

# 3. Check if KVM is available
ls -la /dev/kvm
# If missing, QEMU will be used (slower but should still work)

# 4. Check disk space on the host
df -h /var/snap/microstack/common/

# 5. Check libvirt logs
sudo cat /var/snap/microstack/common/log/libvirt/qemu/<vm-name>.log 2>/dev/null

# 6. Check if the image is valid
microstack.openstack image show <image-name> -c status
# Must be: active

# 7. Delete and recreate the VM
microstack.openstack server delete <vm-name>
microstack.openstack server create \
    --image ubuntu-22.04 --flavor m1.small \
    --network internal --key-name openstack_key <vm-name>

7.3.2 VM Boots but Cannot Access via SSH

# 1. Check the VM's console log for boot messages
microstack.openstack console log show <vm-name>

# 2. Verify the security group allows SSH (port 22)
microstack.openstack security group rule list default

# 3. Verify a floating IP is assigned
microstack.openstack server show <vm-name> -c addresses

# 4. Test SSH with verbose output
ssh -vvv -i ~/.ssh/openstack_key ubuntu@<floating-ip>

# 5. Check if the SSH key was properly injected
# Look in console log for: "Authorized keys from /root/.ssh/authorized_keys"

# 6. Try accessing via VNC console through Horizon
# Navigate to Horizon > Instances > Click VM Name > Console Tab
# Login with: ubuntu / (no password, or check cloud-init output)

7.3.3 QEMU Performance Issues (Slow VM Boot)

# 1. Confirm KVM is NOT available
ls -la /dev/kvm  # Should return "No such file or directory"

# 2. Check CPU steal time (high = noisy neighbor)
top -bn1 | grep "Cpu(s)" | awk '{print $10}' | sed 's/%.*//'
# If steal time > 10%, upgrade to a dedicated CPU droplet

# 3. Reduce VM resource usage
# Use m1.tiny or m1.small flavors instead of larger ones

# 4. Enable KVM (requires droplet shutdown + DO panel change)
# See Section 5.4.2 for instructions

7.4 Horizon Not Accessible

7.4.1 Horizon Returns 502 Bad Gateway

# 1. Check if Horizon service is running inside MicroStack
sudo snap services microstack | grep -i horizon

# 2. Check if the Keystone authentication service is running
sudo snap services microstack | grep -i keystone

# 3. Verify Nginx can reach Horizon
sudo curl -s http://10.20.20.1 --connect-timeout 5
# Expected: Should return Horizon HTML (even if redirect)

# 4. Check Nginx error log
sudo tail -n 20 /var/log/nginx/error.log

# 5. Restart MicroStack services
sudo snap restart microstack.horizon 2>/dev/null || true
sudo snap restart microstack.keystone
sudo snap restart microstack

# 6. Check if the issue is SSL-related
curl -vk https://openstack.kuyaops.com 2>&1 | grep -E "SSL|error|connected"

7.4.2 Horizon Login Fails

# 1. Verify the admin password
snap get microstack config.credentials.keystone-password

# 2. Test authentication via CLI
microstack.openstack token issue

# 3. Check Keystone logs
sudo snap logs microstack.keystone -n 30

# 4. Reset admin password (if needed)
sudo snap set microstack config.credentials.keystone-password=<new-password>
sudo snap restart microstack.keystone

# 5. Clear browser cookies and cache, then retry

7.4.3 VNC Console Not Working in Horizon

# 1. VNC requires WebSocket proxying in Nginx
# Verify the Nginx config includes WebSocket support:
grep -A 3 "Upgrade" /etc/nginx/sites-enabled/openstack-horizon

# 2. The config should contain:
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";

# 3. Reload Nginx after confirming
sudo nginx -t && sudo systemctl reload nginx

# 4. Check Nova novncproxy logs
sudo snap logs microstack.nova-novncproxy -n 20

7.5 SSL Certificate Issues

7.5.1 Certificate Expired

# 1. Check certificate expiry
openssl x509 -in /etc/letsencrypt/live/openstack.kuyaops.com/cert.pem -noout -dates

# 2. Renew manually
sudo certbot renew --force-renewal

# 3. Reload Nginx
sudo systemctl reload nginx

# 4. Verify the new certificate
echo | openssl s_client -connect openstack.kuyaops.com:443 -servername openstack.kuyaops.com 2>/dev/null | openssl x509 -noout -dates

7.5.2 Certbot Fails Domain Validation

# 1. Verify domain resolves to this server
dig +short openstack.kuyaops.com
# Should return your droplet's public IP

# 2. Verify port 80 is accessible from the internet
# (Temporarily allow if UFW is blocking)
sudo ufw allow 80/tcp

# 3. Check Nginx is listening on port 80
sudo ss -tlnp | grep ':80'

# 4. Try Certbot with standalone mode (stops Nginx temporarily)
sudo certbot certonly --standalone -d openstack.kuyaops.com

# 5. Update Nginx to use the new certificate
# (Update paths in /etc/nginx/sites-enabled/openstack-horizon)
sudo nginx -t && sudo systemctl reload nginx

7.6 Full Reset and Reinstall Procedure

If the OpenStack installation becomes completely corrupted, follow this procedure to perform a clean reset. This destroys all VMs, networks, and data.

7.6.1 Preserve Important Data

# Export MicroStack configuration
snap get microstack > /root/microstack-config-backup-$(date +%Y%m%d).txt

# Export any custom images
microstack.openstack image list
# Download custom images manually if needed

# Note floating IPs (will be lost)
microstack.openstack floating ip list

7.6.2 Remove MicroStack

# Stop all MicroStack services first
sudo snap stop microstack

# Remove MicroStack (removes all data)
sudo snap remove microstack --purge

# Verify removal
snap list | grep microstack  # Should return nothing

7.6.3 Clean Up Remaining Artifacts

# Remove residual directories (optional)
sudo rm -rf /var/snap/microstack/

# Remove any leftover bridge interfaces
ip link show | grep brq
# sudo ip link delete <bridge-name>  # if needed

# Flush iptables NAT rules (CAUTION: affects all NAT)
sudo iptables -t nat -F
sudo iptables -t nat -X

# Re-enable IP forwarding
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

7.6.4 Reinstall MicroStack

# Reinstall
sudo snap install microstack --classic

# Reinitialize
sudo microstack init --auto --control

# Verify
microstack.openstack service list
microstack.openstack hypervisor list

7.6.5 Reconfigure Networking and Images

# Re-run the configuration sections of the script
# (Or run the full install_openstack.sh again)
sudo /root/install_openstack.sh

8. Security Best Practices

8.1 Security Architecture Overview

A single-node deployment concentrates all services on one machine, making security especially critical. The following layers provide defense in depth:

1. Network firewall (UFW) — Controls traffic at the OS level
2. Intrusion prevention (Fail2ban) — Detects and blocks malicious patterns
3. Transport encryption (SSL/TLS) — Protects data in transit
4. Authentication (OpenStack Keystone) — Identity management
5. Application security (Security Groups) — Per-VM firewall rules
6. Credential management — Secure storage of passwords and keys

8.2 Firewall Rules (UFW)

UFW (Uncomplicated Firewall) is the primary network-level defense. After the LEMP installation, the following rules are active:

8.2.1 Default UFW Configuration

# View current rules
sudo ufw status verbose

# Expected output:
# Status: active
# Logging: on (low)
# Default: deny (incoming), allow (outgoing), disabled (routed)
# To                         Action      From
# --                         ------      ----
# 22/tcp                     ALLOW IN    Anywhere
# 80/tcp                     ALLOW IN    Anywhere
# 443/tcp                    ALLOW IN    Anywhere
# 22/tcp (v6)                ALLOW IN    Anywhere (v6)
# 80/tcp (v6)                ALLOW IN    Anywhere (v6)
# 443/tcp (v6)               ALLOW IN    Anywhere (v6)

8.2.2 Restricting SSH Access (Recommended)

# Remove blanket SSH allow
sudo ufw delete allow 22/tcp

# Allow SSH only from your IP address
sudo ufw allow from <your-home-ip>/32 to any port 22 proto tcp

# If you have multiple admin IPs
sudo ufw allow from <office-ip>/32 to any port 22 proto tcp
sudo ufw allow from <vpn-ip>/32 to any port 22 proto tcp

# Verify
sudo ufw status numbered

8.2.3 Additional Rules for Production

# Rate limit SSH (built-in UFW feature)
sudo ufw limit 22/tcp

# Block a specific IP address
sudo ufw deny from <malicious-ip>

# Allow specific IP ranges (e.g., office network)
sudo ufw allow from 203.0.113.0/24

# Reload firewall
sudo ufw reload

8.3 Fail2ban Configuration

Fail2ban monitors log files and bans IP addresses that show malicious patterns (brute-force login attempts, vulnerability scans, etc.).

8.3.1 Active Jails

8.3.2 Fail2ban Management Commands

# View all active jails and their status
sudo fail2ban-client status

# View detailed status of a specific jail
sudo fail2ban-client status sshd
# Shows: number of currently banned IPs, total banned, file list

# List currently banned IPs
sudo fail2ban-client status sshd | grep "Banned IP list"

# Manually ban an IP
sudo fail2ban-client set sshd banip <ip-address>

# Manually unban an IP
sudo fail2ban-client set sshd unbanip <ip-address>

# Restart fail2ban after configuration changes
sudo systemctl restart fail2ban

# View fail2ban log
sudo tail -n 50 /var/log/fail2ban.log

8.4 SSL/TLS Encryption

8.4.1 SSL Configuration Details

The Nginx configuration enforces modern TLS settings:

# SSL protocols (TLS 1.2 minimum)
ssl_protocols TLSv1.2 TLSv1.3;

# Cipher suites (strong, forward-secret)
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers on;

# Session caching
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# Certificate paths (Let's Encrypt)
ssl_certificate /etc/letsencrypt/live/openstack.kuyaops.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/openstack.kuyaops.com/privkey.pem;

8.4.2 SSL Test and Verification

# Test SSL configuration
openssl s_client -connect openstack.kuyaops.com:443 -servername openstack.kuyaops.com < /dev/null

# Test with SSL Labs (from any browser)
# Visit: https://www.ssllabs.com/ssltest/analyze.html?d=openstack.kuyaops.com

# Check certificate expiry (set up monitoring for this)
echo | openssl s_client -connect openstack.kuyaops.com:443 2>/dev/null | \
    openssl x509 -noout -enddate

# Auto-renewal is configured via systemd timer
cat /etc/cron.d/certbot  # or
systemctl list-timers | grep certbot

8.4.3 HTTP Strict Transport Security (HSTS)

Add HSTS to force browsers to always use HTTPS:

# Add to the Nginx SSL server block
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# Reload Nginx
sudo nginx -t && sudo systemctl reload nginx

8.5 Strong Passwords and Credential Storage

8.5.1 Credential Files

# LEMP credentials
/root/.lemp_credentials
# Permissions: -rw------- (600)
# Owner: root

# Contents:
# DB_NAME=openstack_db
# DB_USER=openstack_user
# DB_PASSWORD=<random-24-char>
# MYSQL_ROOT_PASSWORD=<random-24-char>
# DOMAIN=openstack.kuyaops.com
# PHP_VERSION=8.3

# Verify permissions
ls -la /root/.lemp_credentials
# Should show: -rw------- 1 root root

8.5.2 OpenStack Admin Password

# Retrieve the admin password
snap get microstack config.credentials.keystone-password

# Change the admin password
snap set microstack config.credentials.keystone-password=<new-strong-password>
snap restart microstack.keystone

# Generate a strong password (24 characters)
openssl rand -base64 18 | tr -dc 'a-zA-Z0-9' | head -c 24

8.5.3 Security Recommendations

• Never commit credential files to version control
• Never paste credentials in chat, email, or logs
• Rotate passwords every 90 days
• Use a password manager for storing credentials
• Enable two-factor authentication for DigitalOcean account

8.6 OpenStack Security Group Rules

Security groups act as virtual firewalls for your VMs. The default security group configured during installation allows:

# View default security group rules
microstack.openstack security group rule list default

# Typical output:
# +--------------------------------------+-------------+-----------+------------+--------------------------------------+
# | ID                                   | IP Protocol | IP Range  | Port Range | Security Group                       |
# +--------------------------------------+-------------+-----------+------------+--------------------------------------+
# | xxxx                                 | tcp         | 0.0.0.0/0 | 22:22      | default                              |
# | xxxx                                 | tcp         | 0.0.0.0/0 | 80:80      | default                              |
# | xxxx                                 | tcp         | 0.0.0.0/0 | 443:443    | default                              |
# | xxxx                                 | icmp        | 0.0.0.0/0 | any        | default                              |
# +--------------------------------------+-------------+-----------+------------+--------------------------------------+

8.6.1 Restricting SSH to Specific IPs

# Remove the blanket SSH rule
microstack.openstack security group rule list default
# Note the ID of the SSH rule (port 22)
microstack.openstack security group rule delete <ssh-rule-id>

# Add a restricted SSH rule
microstack.openstack security group rule create \
    --protocol tcp --dst-port 22:22 \
    --remote-ip <your-office-ip>/32 \
    default

# Add additional IPs as needed
microstack.openstack security group rule create \
    --protocol tcp --dst-port 22:22 \
    --remote-ip <your-home-ip>/32 \
    default

8.6.2 Creating a Custom Security Group

# Create a new security group
microstack.openstack security group create web-servers \
    --description "Security group for web server VMs"

# Add rules
microstack.openstack security group rule create \
    --protocol tcp --dst-port 80:80 \
    --remote-ip 0.0.0.0/0 web-servers

microstack.openstack security group rule create \
    --protocol tcp --dst-port 443:443 \
    --remote-ip 0.0.0.0/0 web-servers

microstack.openstack security group rule create \
    --protocol tcp --dst-port 22:22 \
    --remote-ip <your-admin-ip>/32 web-servers

# Apply to a VM
microstack.openstack server add security group my-web-vm web-servers

# Remove default group if desired
microstack.openstack server remove security group my-web-vm default

8.7 Regular Updates

8.7.1 System Updates

# Update package lists
sudo apt update

# Review available upgrades
apt list --upgradable

# Apply upgrades
sudo apt upgrade -y

# Reboot if kernel was updated
[ -f /var/run/reboot-required ] && sudo reboot

8.7.2 MicroStack Updates

# Refresh MicroStack snap
sudo snap refresh microstack

# Check snap updates
snap list --all | grep microstack

# View snap change history
snap changes

8.7.3 Nginx and SSL Updates

# Update Nginx
sudo apt update && sudo apt install --only-upgrade nginx

# Test Nginx configuration
sudo nginx -t

# Reload if valid
sudo systemctl reload nginx

# Renew SSL certificate
certbot renew --dry-run

sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

9. Quick Reference

9.1 Important File Paths

9.2 Important Commands

9.2.1 System Commands

# System resource monitoring
free -h                              # RAM and swap usage
df -h                                # Disk usage
htop                                 # Interactive process viewer
vmstat 1                             # System activity (1-second intervals)
iostat -x 1                          # I/O statistics

# Check nested virtualization
ls -la /dev/kvm
grep -E "vmx|svm" /proc/cpuinfo

# IP forwarding check
sysctl net.ipv4.ip_forward

# Service management
systemctl status <service>
systemctl restart <service>
systemctl enable <service>
journalctl -u <service> --no-pager -n 50

9.2.2 Snap Commands

snap list                            # List installed snaps
snap list microstack                 # Show MicroStack version
snap services microstack             # List MicroStack services
snap restart microstack              # Restart all MicroStack services
snap restart microstack.<service>    # Restart specific service
snap logs microstack.<service>      # View service logs
snap logs microstack.<service> -f   # Follow logs in real-time
snap refresh microstack             # Update MicroStack
snap get microstack                 # Show MicroStack config
snap get microstack config.credentials.keystone-password
snap remove microstack --purge      # Uninstall MicroStack

9.2.3 OpenStack CLI Commands

# Authentication
microstack.openstack token issue

# Servers (VMs)
microstack.openstack server list
microstack.openstack server show <name>
microstack.openstack server create --image <img> --flavor <flv> --network internal <name>
microstack.openstack server start|stop|reboot|delete <name>
microstack.openstack server add floating ip <name> <ip>
microstack.openstack server add security group <name> <group>
microstack.openstack console log show <name>
microstack.openstack console url show --vnc <name>

# Images
microstack.openstack image list
microstack.openstack image show <name>
microstack.openstack image create --disk-format qcow2 --container-format bare --file <file> <name>
microstack.openstack image delete <name>

# Networks
microstack.openstack network list
microstack.openstack network create|delete <name>
microstack.openstack subnet list
microstack.openstack subnet create|delete <name>
microstack.openstack router list
microstack.openstack router create|delete <name>
microstack.openstack router add subnet <router> <subnet>
microstack.openstack router set --external-gateway <network> <router>

# Floating IPs
microstack.openstack floating ip list
microstack.openstack floating ip create external
microstack.openstack floating ip delete <ip>

# Flavors
microstack.openstack flavor list
microstack.openstack flavor create --ram <MB> --vcpus <n> --disk <GB> <name>

# Security Groups
microstack.openstack security group list
microstack.openstack security group rule list <group>
microstack.openstack security group rule create --protocol tcp --dst-port <port> --remote-ip <cidr> <group>
microstack.openstack security group rule delete <rule-id>

# Keypairs
microstack.openstack keypair list
microstack.openstack keypair create --public-key <pubkey-file> <name>

# Quotas
microstack.openstack quota show
microstack.openstack quota set --instances <n> admin

9.2.4 Nginx Commands

nginx -v                             # Show version
nginx -t                             # Test configuration
nginx -T                             # Show full config (with includes)
sudo systemctl start|stop|restart|reload nginx
sudo systemctl status nginx
curl -I https://openstack.kuyaops.com  # Test HTTPS response

9.2.5 MySQL Commands

sudo mysql -e "SELECT VERSION();"    # Show MySQL version
sudo mysql -u root -p -e "SHOW DATABASES;"
sudo mysql -u openstack_user -p -e "SHOW TABLES;" <database>
sudo systemctl status mysql
sudo systemctl restart mysql

9.2.6 PHP Commands

php -v                               # Show PHP version
php -i | grep opcache               # Check OPcache status
php -m                               # List loaded modules
sudo systemctl status php8.3-fpm
sudo systemctl restart php8.3-fpm

9.3 Port Reference Table

9.4 Log File Locations

10. Next Steps and Roadmap

10.1 Production Architecture

When ready to move beyond development, the recommended production architecture separates the controller and compute functions across multiple droplets:

10.1.1 Phase 1: Separate LEMP and OpenStack

+----------------------------+     +----------------------------+
|   LEMP Web Droplet ($12)   |     |  OpenStack Droplet ($48)   |
|   1 vCPU / 2 GB RAM        |     |  4 vCPU / 8 GB RAM         |
|                            |     |                            |
|  Nginx reverse proxy       |     |  MicroStack (all-in-one)   |
|  Let's Encrypt SSL         |     |  Controller + Compute      |
|  Static/error pages        |     |  VMs run here              |
+----------------------------+     +----------------------------+
        |                                        |
        +---------------+------------------------+
                        |
              openstack.kuyaops.com
              (A record -> LEMP droplet IP)
              
Total: $60/mo (vs $48/mo for combined)
Benefit: Public web services isolated from OpenStack; independent scaling

10.1.2 Phase 2: Dedicated Compute Node

+------------------------+    +---------------------------+
|  Controller ($48)      |    |  Compute Node 1 ($48)     |
|  4 vCPU / 8 GB RAM     |    |  4 vCPU / 8 GB RAM        |
|                        |    |                           |
|  Keystone              |    |  Nova Compute             |
|  Glance                |    |  Libvirt/KVM              |
|  Neutron Server        |    |  Neutron Agent            |
|  Horizon               |    |  (Runs VMs)               |
|  MySQL + RabbitMQ      |    |                           |
+------------------------+    +---------------------------+
           |
+----------v---------------------------------------------+
|  LEMP Droplet ($12) - Nginx proxy for Horizon          |
+--------------------------------------------------------+

10.2 Adding More Compute Nodes

To add additional compute capacity to a MicroStack deployment:

1. Provision a new Ubuntu 24.04 droplet with KVM support
2. Install MicroStack: snap install microstack --classic
3. Join it to the controller: microstack init --auto --compute --join <controller-ip>
4. Verify the hypervisor appears: microstack.openstack hypervisor list

10.3 Ceph Storage Backend

For production-grade persistent storage, integrate Ceph as the storage backend:

• Ceph RBD — Block storage for VM volumes (replaces Cinder LVM)
• CephFS — Shared filesystem for VM migration
• Ceph Object Gateway — S3-compatible object storage

A minimal Ceph cluster requires 3 storage nodes (OSD daemons) for data redundancy. Each node should have at least 100 GB of additional block storage.

10.4 Load Balancing

For high availability, add load balancers in front of the controller nodes:

# HAProxy configuration for OpenStack API endpoints
frontend openstack_api
    bind *:5000
    default_backend keystone_servers

backend keystone_servers
    balance roundrobin
    server controller1 10.20.20.10:5000 check
    server controller2 10.20.20.11:5000 check

10.5 Monitoring with Prometheus and Grafana

Set up comprehensive monitoring to track the health and performance of your OpenStack deployment:

10.5.1 Recommended Monitoring Stack

10.5.2 Key Metrics to Monitor

# System metrics
- CPU usage and steal time
- RAM usage and swap utilization
- Disk I/O and available space
- Network throughput and packet loss

# OpenStack metrics
- Number of VMs (active, building, error)
- Hypervisor capacity (free vCPUs, RAM)
- API response times (Keystone, Nova, Neutron)
- Network packet drops
- Image upload/download rates

# Application metrics
- Nginx request rate and error rate
- MySQL query latency and connection count
- SSL certificate expiry date

10.6 Backup Strategies

10.6.1 What to Backup

10.6.2 Automated Backup Script Example

#!/bin/bash
# /root/backup_openstack.sh
# Run via cron: 0 2 * * * /root/backup_openstack.sh

BACKUP_DIR="/backups/openstack/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Export MicroStack configuration
snap get microstack > "$BACKUP_DIR/microstack-config.txt"

# Backup MySQL (if using external MySQL)
mysqldump -u root --all-databases > "$BACKUP_DIR/mysql-all-databases.sql" 2>/dev/null || true

# Backup Glance images
rsync -a /var/snap/microstack/common/var/lib/glance/images/ "$BACKUP_DIR/glance-images/" 2>/dev/null || true

# Compress
 tar czf "$BACKUP_DIR.tar.gz" -C "$BACKUP_DIR" .
rm -rf "$BACKUP_DIR"

# Upload to DigitalOcean Spaces (S3-compatible) or offsite
# s3cmd put "$BACKUP_DIR.tar.gz" s3://my-backup-bucket/openstack/

# Keep only last 7 days
find /backups/openstack -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR.tar.gz"

10.7 Pre-Production Checklist

Before moving any workload to production, complete this checklist:

10.8 Additional Resources

Last updated: 2025 · Domain: openstack.kuyaops.com · OS: Ubuntu 24.04 LTS · OpenStack: MicroStack (latest/stable) · LEMP Stack: Nginx 1.24.0, MySQL 8.0, PHP 8.3-FPM