smileyface/ut4-server-ctl.sh

#!/bin/bash

# The 'All You Need' Unreal Tournament 4 Server (HUB) Script
# ==========================================================
# This script is everything you need for hosting your own UT4
# server! The way this works is you have a root server directory with this
# script inside it. There's separate directories for config files, maps,
# mutators, and rulesets for the provided game types. You always edit the files
# in those directories and then run this script to deploy the server. This
# includes:
# 
#   (*) Automatic downloading of the latest <server.zip> from Epic
#   (*) Installing the game configuration files, the mutators, the maps into the
#   instance
#   (*) Uploading all .pak package files to the redirect and automatic
#   generation of the required hashes for Game.ini
#   (*) The only thing that's left is for you to configure your gametypes,
#   add some maps, mutators, and get to fraggin'!!!
# 
# Remote and Multiple Instances
# -----------------------------
# This script is designed to take files (maps, mutators, config) from
# a project directory and deploy them to two remote servers, the game
# instance and the redirect. Alternatively, these could be the same
# computer but the redirect should be distinct from the game server
# or downloads will make everyone lag. There is no restriction on
# where you put the project directory but you need a place for
# this script, all the maps, mutators to reside. You put maps,
# rulesets, and configuration in this project directory and then run
# this script to generate the instance, deploy to the server instance,
# and deploy to the redirect. This is also used to manage multiple
# server instances without as much duplication of effort. You need
# enough hard drive space to store all the files for the server which
# could be 10-20GB if you put lots of stuff on your HUB. 

# Rulesets:
# ---------
# Management of the rulesets (which provide the pre-populated gametypes, including
# all the selected mutators, the map rotation, game options, and mutator
# options) is greatly improved. There's some provided rulesets and some Disable.json
# files which remove the pre-populated ones. You manage each rule as a .json configuration
# file in the ./files/rulesets directory. The main part that is not automated
# is the Resource Path for maps to put them in the rotation. You can get these from
# UTCC, or from hosting a lan game with that map on your local game client,
# 
# After you start a map, <alt>+<tab> to go to the UT4 console window and copy/paste
# the name from the logs. It will be the big line and look like:
# 
#     ...Game=/?
# 
# It will make sense when you see the files.
# 
# UPDATE: or better yet there is a script on the forums that will extract the required resource names
# from a directory of paks.
#
# What You Need:
# -------------
# (1) A Server
# What you need to actually host this somewhere is a server to run the hub.
# Ideally this is a machine with it's own internet line else it may slow
# down when you watch NetFlix. Cloud hosting providers will provide you
# a perfect machine for a fee/month.
#
# (2) A Redirect
# On the same note as giving the server it's own internet line, content
# from the server is sent to gamers from a third-party server. You additionally
# need a 'redirect' server to host the files. This can be another server or I've
# seen many public redirects which would be happy to spread some of that
# Unreal Tournament Love. In case, we use a second cloud server.
#
# (3) A good attitude. You're almost done. There's just a few commands to learn
# that this script provides. You need to put this script in the right directory
# and then edit that directory path to the right place. After that, you can
# invoke the script to install and configure your instance.
#
# Usage and Directions:
# --------------------- 
# 
# Edit the exported parameters at the top of this file. You
# certainly need to point this to your relevant servers or
# be prepared to perform the file transfer manually. All
# work is done in the PROJECT_DIR variable so you can see the files
# in there. Use the source code as reference.
#
#     Relevant directories (starting from PROJECT_DIR as base):
#
#       - base              Vanilla UT4 server as downloaded
# 
#       - files             Custom Maps, mutators, rulesets, and config
#
#       - instance          Generated UT4 server application
#                           (for transfer if moving to a remot eserver)
#                           (YOU shouldn't need to edit this folder as
#                           it gets updated by the script)
#
# Requirements: Linux server with installed dependencies, including wget,
# scp, unzip, and sed. These utilities are used for the various file transfers
# and operations, and will be installed on almost every Linux distribution by
# default.
#
# Credits: Script modeled after the code in the unrelated easy-rsa
# script which I found to be helpful in writing this script.
#
# Last tested with UT4 server: <version number>
# Last updated: 2019-05-05

# Configurable parameters (EDIT THESE):
# -------------------------------------
# Local directory to do everything in -- Point to a directory on the server
# to be the base directory, storing all the UT4 files. If you don't care,
# then just use /home/ut4/serv like provided.
# Some good places that make sense, though technically anywhere will work:
#     * /home/ut4
#     * /opt/ut4
#     * /srv/ut4

####
####
# BELOW has been moved to vars-dallas
# ---------------------------------------- 
# 
export PROJECT_DIR="/home/mathew/dev/zavage/ut4-server-ctl"
# 
# # For creating RedirectReferences= Lines
# export REDIRECT_PROTOCOL="https"
# export REDIRECT_URL="ut4-redirect.zavage.net"
# 
# # Remote server, if applicable, to upload instance and paks
# # Only used for upload-redirects and upload-server commands.
# # Format: username@host:/directory/on/remote/server
# export REMOTE_GAME_HOST="ut4-linode:/home/ut4/serv"
# export REMOTE_REDIRECT_HOST="mathewguest.com:/srv/ut4-redirect"

# BELOW HERE, DEFAULTS SHOULD BE FINE. FEEL FREE TO EDIT IF YOU
# KNOW WHAT YOU'RE DOING
# Latest Linux Server.zip:
export DOWNLOAD_URL="https://s3.amazonaws.com/unrealtournament/ShippedBuilds/%2B%2BUT%2BRelease-Next-CL-3525360/UnrealTournament-Server-XAN-3525360-Linux.zip"
export DOWNLOAD_FILENAME="UnrealTournament-Server-XAN-3525360-Linux.zip"
# If MD5 hash is provided, will verify the download:
export DOWNLOAD_MD5="cad730ad6793ba6261f9a341ad7396eb"

export SKIP_VALIDATE="false" # anything but 'true' will interactively ask
                             # the user for variables.

usage()
{
        cat <<"EOF"
Unreal Tournament 4 Server Build and Deploy Script

A list of commands is shown below. 

List commands and usage:
        ./ut4-server-ctl.sh 

Show help for a specific command:
        ./ut4-server-ctl.sh --help <COMMAND>

Here is the list of sub-commands with short syntax reminder:
        ./ut4-server-ctl.sh 1click-deploy
        ./ut4-server-ctl.sh clean-instance
        ./ut4-server-ctl.sh create-directories
        ./ut4-server-ctl.sh download-linux-server
        ./ut4-server-ctl.sh download-logs
        ./ut4-server-ctl.sh generate-instance
        ./ut4-server-ctl.sh start-server
        ./ut4-server-ctl.sh stop-server
        ./ut4-server-ctl.sh upload-redirects
        ./ut4-server-ctl.sh upload-server

Typical Usage:
        1) You need to either configure the remote server hostnames (both game server and remote redirect server)
        in the vars files. Edit vars with a text editor, or edit the defaults in this file, or manually type
        them interactively when prompted* (coming soon).

        e.g.:
                PROJECT_DIR="/path/to/this/script/directory/on/local/machine"
                REMOTE_GAME_HOST="54.123.456.10"
                REMOTE_GAME_DIR="/home/ut4/hub-instance"

                REMOTE_REDIRECT_HOST="45.321.654.10"
                REMOTE_REDIRECT_DIR="/srv/ut4-redirect/"

        2) You need to download the latest Linux Server release from Epic. Do
        this with the 'download-server' command.

        e.g.:
                ./ut4-server-ctl.sh download-server

        3) Add and configure custom maps, mutators, rulesets, and hub configuration to your *local* project folder.
        This is done by modifying the files in the project directory. With the current environment variables,
        this is set to be:
        
                "$PROJECT_DIR"

        This is the fun part! Connect with UTCC or UTZONE.DE (unaffiliated) to find custom content to put on
        your hub.

        4) 1click-deploy to update the remote hub and redirect with your latest content. You're done! Rinse and repeat.

        e.g.:
                ./ut4-server-ctl.sh 1click-deploy

        Alternatively, this command is equivalent to running the separate commands. If you'd like more fine-grained
        control, you can run them individually.
                ./ut4-server-ctl.sh generate-instance
                ./ut4-server-ctl.sh upload-redirects
                ./ut4-server-ctl.sh upload-server
                ./ut4-server-ctl.sh restart-server

EOF
}

cmd_help()
{
        cmd="$1"
        case "$cmd" in
                1click-deploy)
                        cat <<"EOF"
1click-deploy)
        Executes all the commands needed to update a remote game server. This is
        equivalent to running these 3 commands:
                1 - generate-instance (installing config, maps, and mutators)
                2 - upload-server (copying hub instance to remote game server)
                3 - upload-redirects (copying customized content to remote redirect server)
EOF
                        ;;

                clean-instance)
                        cat <<"EOF"
clean-instance)
        This command will delete all files on the generated instance on the LOCAL machine.

        Useful for if you ever delete a map or mutator and wan't it off the server. To do that,
        modify the files you need, clean-instance, and then generate-instance from scratch.
EOF
                        ;;
                create-directories)
                        cat <<"EOF"
EOF
                        ;;
                download-linux-server)
                        cat <<"EOF"
download-linux-server)
        Downloads the latest Linux server release from Epic.
EOF
                        ;;
                download-logs)
                        cat <<"EOF"
download-logs)
        Downloads the instance logs from the remote game server while clearing
        them on the server.
EOF
                        ;;
                generata-instance)
                        cat <<"EOF"
generate-instance)
        Installs the game configuration, maps, and mutators into the *local*
        instance directory. The next step after this has been done is to
        upload-server to copy the instance directory to the remote game server.
EOF
                        ;;
                restart-server)
                        cat <<"EOF"
restart-server)
        Equivalent to stop-server + start-server.
EOF
                        ;;
                start-server)
                        cat <<"EOF"
start-server)
        Turns on the remote game server Unreal Tournament 4 Linux Hub instance. Will run in
        an infinite loop until stopped.
EOF
                        ;;
                stop-server)
                        cat <<"EOF"
stop-server)
        Turns off and kills all remote game server instances.
EOF
                        ;;
                upload-redirects)
                        cat <<"EOF"
upload-redirects)
        Uploads customized content and sanitized server configuration to remote
        redirect server.
EOF
                        ;;
                upload-server)
                        cat <<"EOF"
upload-server)
        Installs the generated UT4 Hub instance to the configured remote game
        server.
EOF
                        ;;
                ""|help|-h|--help|--usage)
                        usage 
                        ;;
                *)
                        echo "Unknown command '$cmd'. Invoke script for usage help."
                        ;;
        esac
}

oneclick_deploy()
{
        generate_instance
        upload_redirects
        upload_server
}

clean_instance()
{
        _UT4_print Clearing .pak folder...
        (set -x # NOTE(MG) set -x echoes all commands that the script does.
                # The user will be able to see the computed commands as the
                # script runs them.

        rm -rv "$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Content/Paks/*

        )
}

create_directories()
{
        (set -x 

        mkdir -p "$PROJECT_DIR"/base
        mkdir -p "$PROJECT_DIR"/files/config
        mkdir -p "$PROJECT_DIR"/files/maps
        mkdir -p "$PROJECT_DIR"/files/mutators
        mkdir -p "$PROJECT_DIR"/files/rulesets
        mkdir -p "$PROJECT_DIR"/files/unused

        )
}

download_linux_server()
{
        cd "$PROJECT_DIR"/base
        
        # Only download server when hasn't already been downloaded
        if [ ! -f "$PROJECT_DIR"/base/"$DOWNLOAD_FILENAME" ]  ; then
                _UT4_print "Downloading Linux Server Archive..."
                (set -x
                        wget "$DOWNLOAD_URL" --show-progress 
                )

                _UT4_print "Verifying archive is the file we're expecting"
                md5=$(md5sum "$DOWNLOAD_FILENAME" | cut -d' ' -f1)
                if [ "$md5" != "$DOWNLOAD_MD5" ] ; then
                       server.zip  echo Downloaded '"$DOWNLOAD_FILENAME"' checksum FAILED.
                        echo The downloaded zip was not the exact file we\'re expecting.
                        echo Refusing to continue.
                        return 1
                else
                        echo The downloaded zip checksum matched what we expected: "$DOWNLOAD_MD5"
                fi
        else
                _UT4_print "UT4 Server .zip already downloaded"
                _UT4_print "You need to invoke this command manually if you want to delete these files. (Being extra safe)"
                _UT4_print "DELETE COMMAND (manual): cd "$PROJECT_DIR"/base && rm -rv *" && cd -
                cd - >/dev/null
                return 1
        fi
        
        # Extract ut4 linux server into ./base (with overwrite)
        _UT4_print "Extracting archive..."
        (set -x

        unzip -o "$DOWNLOAD_FILENAME"
        # rm "$DOWNLOAD_FILENAME"

        )
        
        cd - >/dev/null
}

download_logs()
{
        _UT4_print "Downloading the game logs from the server (clearing them also)"
        # Transfer log directory files from remote game server to local:
        (set -x

        rsync -ravzp "$REMOTE_GAME_HOST":"$REMOTE_GAME_DIR"/LinuxServer/UnrealTournament/Saved/Logs/ "$CONFIG_DIR"/downloaded-logs/

        )

        # Delete logs on remote game server if successfully transferred to local:
        if [ $? -eq 0 ] ; then
                (set -x

                :
                # ssh "$REMOTE_GAME_HOST" rm "$REMOTE_GAME_DIR"'/LinuxServer/UnrealTournament/Saved/Logs/* -r'

                )
        fi
}

generate_instance()
{
        _UT4_print "Generating server instance from custom files..."
        (set -x

        rsync -ravzp "$PROJECT_DIR"/base/LinuxServer "$PROJECT_DIR"/instance/
        cp "$PROJECT_DIR"/start-server.sh "$PROJECT_DIR"/instance/
        cp "$PROJECT_DIR"/stop-server.sh "$PROJECT_DIR"/instance/

        )
        # _first_run
        _install_config
        _install_paks
        _install_redirect_lines
        _install_rulesets
}

restart_server()
{
        stop_server
        start_server
}

start_server()
{
        _UT4_print "Starting server!!"
        (set -x

        nohup ssh "$REMOTE_GAME_HOST" "$REMOTE_GAME_DIR"/start-server.sh &

        )
}

stop_server()
{
        _UT4_print "Stopping all servers!"
        (set -x

        ssh "$REMOTE_GAME_HOST" "$REMOTE_GAME_DIR"/stop-server.sh

        )
}

upload_redirects()
{
        PAKS_DIR="$PROJECT_DIR"/files

        _UT4_print "Uploading redirects (maps, mutators, etc.) to download server"
        (set -x

        rsync -rvz --delete --exclude "*.md5" --exclude 'unused' --exclude ".KEEP" --exclude Mods.db "$PAKS_DIR"/ "$REMOTE_REDIRECT_HOST"

        )

        # _UT4_print "Uploading md5sum files to redirect"
        # mkdir /tmp/ut4-md5sums
        # for f in $(ls "$PAKS_DIR"); do
        #         md5sum "$PAKS_DIR"/"$f" | cut -d' ' -f1 > /tmp/ut4-md5sums/"$f".md5
        # done

        # rsync -rvz --exclude ".KEEP" /tmp/ut4-md5sums/ "$REMOTE_REDIRECT_HOST"


        # Hide admin password from redirect:
        # TODO(MG) hardcoded paths and hostname >.<

        # (on the server)
        gameini="/srv/ut4-redirect.zavage.net/config/Game.ini"
        engineini="/srv/ut4-redirect.zavage.net/config/Engine.ini"
        (set -x

        ssh mathewguest.com sed -i /ServerInstanceID=/c\ServerInstanceID=Hidden "$gameini"

        ssh mathewguest.com sed -i /RconPassword=/c\RconPassword=Hidden "$engineini"

        rsync -vz "$PROJECT_DIR"/ut4-server-ctl.sh "$REMOTE_REDIRECT_HOST"

        )

        # Fix permissions for redirect:
        (set -x

        ssh mathewguest.com 'chown http.http /srv/ut4-redirect.zavage.net -R'

        )
}

upload_server()
{
        _UT4_print "Uploading customized server"
        # Exclude everything we don't want to be
        # overwritten on the server instance.
        # (Hint: uncomment --dry-run to safely verify command works as intended)
        (set -x

        rsync -ravzp \
                --delete \
                --exclude ".KEEP" \
                --exclude "Mods.db" \
                --exclude "Mod.ini" \
                --exclude "Logs" \
                --exclude "ut4-server.log" \
                --exclude "Saved/*.ai" \
                --exclude "Saved/Crashes/*" \
                --exclude "Saved/Logs/*" \
                "$PROJECT_DIR"/instance/ \
                "$REMOTE_GAME_HOST":"$REMOTE_GAME_DIR"

        rsync -avzp "$PROJECT_DIR"/ut4-server-ctl.sh "$REMOTE_GAME_HOST":"$REMOTE_GAME_DIR"

        scp "$PROJECT_DIR"/instance/ut4-server.service "$REMOTE_GAME_HOST":/etc/systemd/system/

        ssh "$REMOTE_GAME_HOST" chown ut4.ut4 "$REMOTE_GAME_DIR" -R
        )
}

_first_run()
{
        _UT4_print "Starting instance once to get UID."
        _UT4_print "Unfortunately, this takes 20 seconds. Just wait."
        (set -x

        cd "$PROJECT_DIR"/instance/LinuxServer/Engine/Binaries/Linux
        chmod 770 UE4Server-Linux-Shipping
        ./UE4Server-Linux-Shipping UnrealTournament UT-Entry?Game=Lobby -log &>/dev/null &
        cd - >/dev/null

        )
        _UT4_print "sleeping 20 seconds and then we'll kill the server we started just now."
        sleep 20
        stop_server
        # TODO(MG) get uid and export
}

_install_config()
{
        f="Game.ini"
        _UT4_print Installing file: "$f"
        (set -x

        cp "$CONFIG_DIR"/"$f" "$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Saved/Config/LinuxServer/

        )

        f="Engine.ini"
        _UT4_print Installing file: "$f"
        (set -x

        cp "$CONFIG_DIR"/"$f" "$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Saved/Config/LinuxServer/

        )
}

_install_paks()
{
        _UT4_print Installing maps...

        (set -x

        rsync -ravzp "$PROJECT_DIR"/files/maps/ "$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Content/Paks/

        )


        _UT4_print Installing mutators...
        (set -x

        rsync -ravzp "$PROJECT_DIR"/files/mutators/ "$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Content/Paks/

        )
}

_install_redirect_lines()
{
        _UT4_print Generating redirect references...
        MOD_DIR="$PROJECT_DIR"/files
        shopt -s nocaseglob
        echo > /tmp/ut4-server-ctl-references
        for f in "$MOD_DIR"/**/*.pak ; do
                # f = full path
                RELATIVE_PATH=${f#"$MOD_DIR"/} # <- Trims /root/path/up/until/$MOD_DIR
                                               # example output: maps/DM-Rankin.pak
                                               # mutators/GodMode.pak
                BASENAME=$(basename "$f") # DM-Rankin.pak

                # Exclusions:
                #   - exclude the UnrealTournament pak
                if [ "$BASENAME" = "UnrealTournament-LinuxServer.pak" ] ; then
                        continue
                fi

                # Exclude any files in the 'unused' directory:
                FIRSTDIR=$(echo "$RELATIVE_PATH" | cut -d'/' -f1)
                if [ "$FIRSTDIR" = "unused" ] ; then
                        continue
                fi

                MD5=$(md5sum "$f"| cut -d' ' -f1)
                EXTENSION="${BASENAME##*.}" # .pak
                FILENAME="${BASENAME%.*}" # DM-Rankin
                # line continues that way         -->     -->    -->
	        LINE="RedirectReferences=(PackageName=\""$FILENAME"\",PackageURLProtocol=\"$REDIRECT_PROTOCOL\",PackageURL=\""$REDIRECT_URL"/"$RELATIVE_PATH"\",PackageChecksum=\""$MD5"\")"
                echo "$LINE"
                echo "$LINE" >> /tmp/ut4-server-ctl-references
        done

        _UT4_print Installing redirect references into Game.ini...
        FILENAME="$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Saved/Config/LinuxServer/Game.ini
        LINE_NUMBER=17 # TODO(MG) Not hard-coded

        # Delete 
        sed -i '/RedirectReferences/d' "$FILENAME"
        
        # Insert into Game.ini
        awk '1; NR==11 {system("cat /tmp/ut4-server-ctl-references")}' "$FILENAME" > /tmp/ut4-server-ctl-game.ini
        cp /tmp/ut4-server-ctl-game.ini "$FILENAME"
        cat "$FILENAME"
}

_install_rulesets()
{
        _UT4_print Concatenating rulesets for game modes...
        SRC_DIR="$PROJECT_DIR"/files/rulesets
        OUT_DIR="$PROJECT_DIR"/instance/LinuxServer/UnrealTournament/Saved/Config/Rulesets
        OUT_FILENAME="$OUT_DIR"/ruleset.json

        mkdir -pv "$OUT_DIR"

        echo OUT_FILENAME="$OUT_FILENAME"
        
        echo {\"rules\":[ > "$OUT_FILENAME"
        
        for f in "$SRC_DIR"/*.json ; do
        	cat "$f" >> $OUT_FILENAME
        done
        
        echo "]}" >> "$OUT_FILENAME"
        echo output ruleset is at "$OUT_FILENAME"
}

_validate_env_vars()
{
        # TODO(MG) Add which user we are running this under
        _UT4_print Are these environment variables correct?
        echo 'PROJECT_DIR:           '"$PROJECT_DIR"
        echo
        echo 'DOWNLOAD_URL:          '"$DOWNLOAD_URL"
        echo 'DOWNLOAD_FILENAME:     '"$DOWNLOAD_FILENAME"
        echo 'DOWNLOAD_MD5:          '"$DOWNLOAD_MD5"
        echo 'REDIRECT_PROTOCOL:     '"$REDIRECT_PROTOCOL"
        echo 'REDIRECT_URL:          '"$REDIRECT_URL"
        echo 'REMOTE_GAME_HOST:      '"$REMOTE_GAME_HOST"
        echo 'REMOTE_GAME_DIRECTORY: '"$REMOTE_GAME_DIR"
        echo 'REMOTE_REDIRECT_HOST:  '"$REMOTE_REDIRECT_HOST"
        echo


        read -p "Continue with above configuration? " -n 1 -r
        echo
        if [[ ! $REPLY =~ ^[Yy]$ ]] ; then
                exit 1
        fi
}

# Wrapper prints colored log messages which stand out to communicate what
# the script is doing to the user.
_UT4_print()
{
        RED='\033[0;31m'
        NC='\033[0m'
        echo -e "UT4>${RED} "$@" ${NC}"
}

main()
{
        cmd="$1"
        _validate_env_vars

        case "$cmd" in
                1click-instance)
                        shift
                        oneclick_deploy "$@"
                        exit "$?";;
                clean-instance)
                        shift
                        clean_instance "$@"
                        exit "$?";;
                create-directories)
                        shift
                        create_directories "$@"
                        exit "$?";;
                download-linux-server)
                        shift
                        download_linux_server "$@"
                        exit "$?";;
                download-logs)
                        shift
                        download_logs "$@"
                        exit "$?";;
                generate-instance)
                        shift
                        generate_instance "$@"
                        exit "$?";;
                restart-server)
                        shift
                        restart_server "$@"
                        exit "$?";;
                start-server)
                        shift
                        start_server "$@"
                        exit "$?";;
                stop-server)
                        shift
                        stop_server "$@"
                        exit "$?";;
                upload-redirects)
                        shift
                        upload_redirects "$@"
                        exit "$?";;
                upload-server)
                        shift
                        upload_server "$@"
                        exit "$?";;
                ""|help|-h|--help|--usage)
                        shift
                        cmd_help "$@"
                        exit 0;;
                *)
                        cmd_help "$@"
                        exit 1;;
        esac
}

main "$@"