Development Stack

This code defines the set of web applications which make up this website.  It hosts git repositories, builds and deploys them and allows the systems they run on to be accessed and monitored.

Applications

These application scan be accessed with the links at the top of this blog..

Gitea (git with a cup of tea) is an open source github like application.  It provides a web interface to git source code repositories.

Drone is a Continuous Delivery platform.  It automates the build, test, deployment and pretty much whatever you like of commits made to the git repositories hosted by gitea. When commits are made webhooks trigger pipelines in drone.  These pipelines are defined in the repository.  The stack builds and deploys itself.

Ghost is a blogging platform.  It's serving this post and is the main page of the website with links to the other applications.

Guacamole is clientless remote desktop gateway.  It translates protocols such as SSH, RDP and VNC into a protocol running over websockets allowing an HTML5 browser to act as a remote desktop.

Rocket Chat is a web chat platform.

Portainer  is a docker management system.

Commento is a commenting platform.

Matomo is a privacy focused web analytics platform.  

Zabbix is an infrastructure monitoring platform.

All these applications are open source.

Docker stack

These applications run as a stack of docker containers on a swarm.  This stack is defined in a yaml file.  Each container is an isolated linux environment defined by an image.

This is the yaml which defines the drone-server docker service which co-ordinates the execution of drone pipelines:

drone-server:
    # drone server application
    deploy:
      placement:
        constraints: [node.labels.com.sigyl.git-stack == yes]
      replicas: 1
      restart_policy:
        condition: any
    image: drone/drone:1.7.0
    volumes:
      - drone:/var/lib/drone
      - drone-data:/data
    depends_on:
      - gitea
    environment:
      - DRONE_LOGS_DEBUG=true
      - DRONE_LOGS_PRETTY=true
      - DRONE_GITEA_SERVER=${DRONE_GITEA_SERVER}
      - DRONE_GITEA_CLIENT_ID=${DRONE_GITEA_CLIENT_ID}
      - DRONE_GITEA_CLIENT_SECRET=${DRONE_GITEA_CLIENT_SECRET}
      - DRONE_SERVER_HOST=${DRONE_SERVER_HOST} # tunnel hostname       
      - DRONE_ADMIN=giles
      - DRONE_SERVER_PROTO=https # tunnel adds https on top
      - DRONE_SERVER_PORT=:8080
      - DRONE_RPC_SECRET=${DRONE_RPC_SECRET}
      - DRONE_USER_CREATE=username:giles,admin:true
      - DRONE_AGENTS_ENABLED=true
      - DRONE_CONVERT_PLUGIN_ENDPOINT=http://drone-starlark:3000
      - DRONE_CONVERT_PLUGIN_SECRET=${DRONE_CONVERT_SECRET}
    networks:
      - appnet

The build and deployment process creates the images which are based in images downloaded from the web (docker hub) and then deploys them.

The command that finally deploys the stack is:

docker stack deploy -c docker-compose.yml my-stack

Secrets

When a repository is configured in drone 'secrets' can be defined.  For it.  These are pieces of information which should not be in the repository, either because they should not be made public for security reasons or because they are specific to a deployment.  The pipeline steps can access these secrets but they cannot be seen externally once entered.

Starlark

Drone pipelines can be configured with a yaml file or written in Starlark.  Starlark is enabled by a drone-convert-starlark application running as a container in the stack.  The drone server uses this to convert starlark programs into pipeline configurations.  The drone-convert-starlark application is given a repos folder which contains library functions. This will continue to be developed.

The starlark for a pipeline to build and deploy the stack in the cloud then becomes as simple as:

load("@this//stack:drone.star", "drone")
load("@this//stack:stack-name.star", "stackName")
load("@this//stack:stack-root.star", "stackRoot")


def main(ctx):
  return drone(
    ctx,
    "do",
    stackRoot,
    stackName,
    [
      "docker service scale {name}_chat=1".format(name=stackName),
      "docker service scale {name}_letsencrypt-drone=1".format(name=stackName),
      "docker service scale {name}_letsencrypt-git=1".format(name=stackName),
    ]
  )

The 'drone' function then produces a set of pipeline definitions which drone runs.

Let's Encrypt

Two Nginx services act as reverse proxies to the web services in the stack.  Internally these services are exposed over http.  The Nginx services convert them to use TLS with certificates obtained from and renewed with Let's Encrypt.  To do this the NGINX images are built from a staticfloat/nginx-certbot image.

Ngrok

Where the stack is not hosted on an IP on the internet for example behind a home broadband router ngrok is used to tunnel to the internet.  This is done with a service based on wernight/ngrok.