.
continuous-integration/drone/push Build is passing Details

This commit is contained in:
Giles Bradshaw 2020-05-01 00:56:53 +01:00
commit 439f44f68d
18 changed files with 1313 additions and 278 deletions

129
README.md
View File

@ -52,12 +52,80 @@ Once installed and running the system can redeploy itself.
However initially you need to do this yourself. However initially you need to do this yourself.
### docker
you need a docker swarm set up with nodes with the following labels ### remove old versions of docker
(if it's a fresh install of linux there shouldn't be any)
```
sudo apt-get remove docker docker-engine docker.io
```
### install docker
```
sudo apt install docker.io
```
### add current user to docker group
logout and back in afterwards
```
sudo usermod -aG docker $USER
```
### start and enable docker
```
sudo systemctl start docker
sudo systemctl enable docker
```
### change ssh port to 2022
```
sudo vi /etc/ssh/sshd_config
```
change Port 2022
### allow root to ssh
```
sudo vi /etc/ssh/sshd_config
```
## set the root password
```
sudo passwd root
```
change PermitRootLogin yes
reboot
start a stack running gitea to host repository.
## stack
### labels
get nodes with
```
docker node ls
```
add label with
```
docker node update --label-add com.sigyl.git-stack=yes [node id]
```
* com.sigyl.git-stack=yes
* com.sigyl.git-stack-data=yes
### global environment ### global environment
@ -104,11 +172,18 @@ sh ca.sh $REGISTRY_DOMAIN:5003
sh make-cert.sh $REGISTRY_DOMAIN registry sh make-cert.sh $REGISTRY_DOMAIN registry
``` ```
copy the directory .ca/$REGISTRY_DOMAIN:5003 to /etc/docker/certs.d
```
sudo mkdir -p /etc/docker/certs.d/
sudo cp -r .ca/$REGISTRY_DOMAIN:5003 /etc/docker/certs.d/
```
### make environment variables ### make environment variables
``` ```
export TITLE=SiGyl Ltd export TITLE="SiGyl Ltd!"
export DESCRIPTION=Software Development export DESCRIPTION="Software Development"
export CERTBOT_EMAIL=giles.bradshaw@sigyl.com export CERTBOT_EMAIL=giles.bradshaw@sigyl.com
export DRONE_DOMAIN=drone.sigyl.com export DRONE_DOMAIN=drone.sigyl.com
export DRONE_GITEA_SERVER=https://sigyl.com/git export DRONE_GITEA_SERVER=https://sigyl.com/git
@ -341,10 +416,46 @@ Vist domain/ghost and set up admin user.
### chat ### chat
Admin user is automatically created according to configured secrets Admin user is automatically created according to configured secrets. Change the password!
### guacamole ### guacamole
sign in with guacadmin guacadmin Use admin user name and password you supplied when you set up the database.
change password asap!
## docker-exec-runner on windows
These instructions are not very good...
https://exec-runner.docs.drone.io/installation/windows/
download and unpack on linux with
```
curl -L https://github.com/drone-runners/drone-runner-exec/releases/latest/download/drone_runner_exec_windows_amd64.tar.gz | tar zx
```
rename drone-runner-exec to drone-runner-exec.exe
make directory c:\Drone\drone-runner-exec on windows
copy drone-runner-exec.exe to directory
make config file with
```
DRONE_RPC_PROTO=https
DRONE_RPC_HOST=drone.sigyl.com:443
DRONE_RPC_SECRET=[rpc secret]
DRONE_LOG_FILE=C:\Drone\drone-runner-exec\log.txt
DRONE_RUNNER_LABELS=web:true
```
install and start service with
```
drone-runner-exec service install
drone-runner-exec service start
```

View File

@ -67,12 +67,23 @@ services:
replicas: 1 replicas: 1
restart_policy: restart_policy:
condition: any condition: any
image: gitea/gitea:latest image: ${LOCAL_DOCKER_REGISTRY}gitea
environment: environment:
- USER_UID=1000 - USER_UID=1000
- USER_GID=1000 - USER_GID=1000
- ROOT_URL=https://${GIT_DOMAIN}/git - ROOT_URL=https://${GIT_DOMAIN}/git
- SSH_DOMAIN=${GIT_DOMAIN} - SSH_DOMAIN=${GIT_DOMAIN}
- GITEA_APP_NAME=${GITEA_APP_NAME}
- GIT_DOMAIN=${GIT_DOMAIN}
- GITEA_SERVER_LFS_JWT_SECRET=$GITEA_SERVER_LFS_JWT_SECRET
- GITEA_SECURITY_SECRET_KEY=$GITEA_SECURITY_SECRET_KEY
- GITEA_SECURITY_INTERNAL_TOKEN=$GITEA_SECURITY_INTERNAL_TOKEN
- GITEA_OAUTH2_JWT_SECRET=$GITEA_OAUTH2_JWT_SECRET
- GITEA_MAILER_HOST=$GITEA_MAILER_HOST
- GITEA_MAILER_USER=$GITEA_MAILER_USER
- GITEA_MAILER_FROM=$GITEA_MAILER_FROM
- GITEA_MAILER_PASSWD=$GITEA_MAILER_PASSWD
volumes: volumes:
- gitea-app:/data - gitea-app:/data
ports: ports:
@ -110,7 +121,7 @@ services:
replicas: 1 replicas: 1
restart_policy: restart_policy:
condition: any condition: any
image: drone/drone:latest image: drone/drone:1.7.0
volumes: volumes:
- drone:/var/lib/drone - drone:/var/lib/drone
- drone-data:/data - drone-data:/data
@ -303,7 +314,7 @@ services:
replicas: 0 # will scale after mongo initated replicas: 0 # will scale after mongo initated
restart_policy: restart_policy:
condition: any condition: any
image: rocketchat/rocket.chat:latest image: rocketchat/rocket.chat:3.0.7
networks: networks:
- appnet - appnet
environment: environment:
@ -311,7 +322,7 @@ services:
- ROOT_URL=https://${GIT_DOMAIN}/chat - ROOT_URL=https://${GIT_DOMAIN}/chat
- PORT=3000 - PORT=3000
- MONGO_URL=mongodb://chat-mongo:27017/rocketchat - MONGO_URL=mongodb://chat-mongo:27017/rocketchat
- ADMIN_USERNAME=${CHAT_ADMIN_USER} - ADMIN_USERNAME=${CHAT_ADMIN_NAME}
- ADMIN_PASS=${CHAT_ADMIN_PASSWORD} - ADMIN_PASS=${CHAT_ADMIN_PASSWORD}
- ADMIN_EMAIL=${CHAT_ADMIN_EMAIL} - ADMIN_EMAIL=${CHAT_ADMIN_EMAIL}
volumes: volumes:

View File

@ -1,4 +1,4 @@
FROM drone/drone-convert-starlark FROM drone/drone-convert-starlark:1.1.0-beta.1
COPY repos /repos COPY repos /repos
COPY run.sh / COPY run.sh /
USER root USER root

View File

@ -1,7 +1,7 @@
load("@this//:secret-to-environment.star", "secretToEnvironment") load("@this//:secret-to-environment.star", "secretToEnvironment")
def echoSecret(secret): def echoSecret(secret):
return 'echo "export {environment}=???" >> ***filename*** # {secret}'.format( return 'echo "export {environment}=???? ${environment}" >> ***filename*** # {secret}'.format(
secret = secret, secret = secret,
environment = secretToEnvironment(secret), environment = secretToEnvironment(secret),
) )

View File

@ -9,9 +9,9 @@ def printSecrets(filename, env, secretEnv):
return { return {
"name": "print secrets", "name": "print secrets",
"image": "appleboy/drone-ssh", "image": "appleboy/drone-ssh",
"environment": environment(env), "environment": environment(env + secretEnv),
"settings": { "settings": {
"envs": [x.replace("-", "_") for x in env ], "envs": [x.replace("-", "_") for x in env + secretEnv ],
"host": fromSecret("ssh-host"), "host": fromSecret("ssh-host"),
"port": fromSecret("ssh-port"), "port": fromSecret("ssh-port"),
"username": fromSecret("ssh-user"), "username": fromSecret("ssh-user"),

View File

@ -4,11 +4,12 @@ load("@this//:environment.star", "environment")
load("@this//:export.star", "export") load("@this//:export.star", "export")
def pull( def pull(
name,
images, images,
): ):
secrets = [ "local-docker-registry"] secrets = [ "local-docker-registry"]
return { return {
"name": "pull", "name": name,
"image": "appleboy/drone-ssh", "image": "appleboy/drone-ssh",
"environment": environment(secrets), "environment": environment(secrets),
"settings": { "settings": {

View File

@ -30,19 +30,16 @@ def drone(
pipeline( pipeline(
branch, branch,
[ [
wait(15, "wait"),
build("drone-starlark"),
printSecrets( printSecrets(
"env-stack", "env-stack",
publicSecrets, publicSecrets,
secretSecrets, secretSecrets,
), ),
wait(15, "wait"), build("gitea"),
build("drone-starlark"),
rescale(
"{name}_drone-starlark".format(name=name),
1,
),
build("ngrok-gitea"),
build("guacamole-postgresql"), build("guacamole-postgresql"),
build("ngrok-gitea"),
build("letsencrypt-nginx"), build("letsencrypt-nginx"),
build("ghost"), build("ghost"),
buildDockerFolder( buildDockerFolder(
@ -60,14 +57,18 @@ def drone(
"drone", "drone",
), ),
scp(base), scp(base),
pull([ pull(
"ghost", "pull images",
"ngrok-gitea", [
"letsencrypt-git", "drone-starlark",
"letsencrypt-drone", "gitea",
"drone-starlark", "ghost",
"guacamole-postgresql", "ngrok-gitea",
]), "letsencrypt-git",
"letsencrypt-drone",
"guacamole-postgresql",
],
),
deploy( deploy(
"docker-compose-home.yml", "docker-compose-home.yml",
name, name,

View File

@ -17,4 +17,8 @@ publicSecrets = [
"ghost-mail-user", "ghost-mail-user",
"chat-admin-name", "chat-admin-name",
"chat-admin-email", "chat-admin-email",
"gitea-mailer-host",
"gitea-mailer-from",
"gitea-mailer-user",
"gitea-app-name"
] ]

View File

@ -6,4 +6,9 @@ secretSecrets = [
"ngrok-auth-token", "ngrok-auth-token",
"ghost-mail-password", "ghost-mail-password",
"chat-admin-password", "chat-admin-password",
"gitea-server-lfs-jwt-secret",
"gitea-security-secret-key",
"gitea-security-internal-token",
"gitea-oauth2-jwt-secret",
"gitea-mailer-passwd",
] ]

View File

@ -1,4 +1,4 @@
FROM ghost FROM ghost:3.14.0
# USER root # USER root
RUN apt-get update RUN apt-get update
RUN apt-get install -y gettext RUN apt-get install -y gettext

View File

@ -1,2 +1,2 @@
envsubst < /var/lib/ghost/config.production.json > __tmp && mv __tmp /var/lib/ghost/config.production.json envsubst < /var/lib/ghost/config.production.json > __tmp && mv __tmp /var/lib/ghost/config.production.json
node "current/index.js" node "current/index.js"

5
gitea/Dockerfile Normal file
View File

@ -0,0 +1,5 @@
FROM gitea/gitea:1.11.4-linux-amd64
COPY app.ini /init/
COPY ./templates /init/templates/
COPY run.sh /
CMD ["sh", "/run.sh"]

89
gitea/app.ini Normal file
View File

@ -0,0 +1,89 @@
APP_NAME = ${GITEA_APP_NAME}
RUN_MODE = prod
RUN_USER = git
[repository]
ROOT = /data/git/repositories
[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo
[repository.upload]
TEMP_PATH = /data/gitea/uploads
[server]
APP_DATA_PATH = /data/gitea
SSH_DOMAIN = ${GIT_DOMAIN}
HTTP_PORT = 3000
ROOT_URL = https://${GIT_DOMAIN}/git/
DISABLE_SSH = false
SSH_PORT = 22
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
DOMAIN = ${GIT_DOMAIN}
LFS_JWT_SECRET = ${GITEA_SERVER_LFS_JWT_SECRET}
OFFLINE_MODE = false
[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD =
SCHEMA =
SSL_MODE = disable
CHARSET = utf8
[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve
[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file
[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true
[attachment]
PATH = /data/gitea/attachments
[log]
ROOT_PATH = /data/gitea/log
MODE = file
LEVEL = info
[security]
INSTALL_LOCK = true
SECRET_KEY = ${GITEA_SECURITY_SECRET_KEY}
INTERNAL_TOKEN = ${GITEA_SECURITY_INTERNAL_TOKEN}
PASSWORD_COMPLEXITY = off
[service]
DISABLE_REGISTRATION = false
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = true
ENABLE_NOTIFY_MAIL = true
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.${GIT_DOMAIN}
[oauth2]
JWT_SECRET = ${GITEA_OAUTH2_JWT_SECRET}
[mailer]
ENABLED = true
HOST = ${GITEA_MAILER_HOST}
FROM = ${GITEA_MAILER_FROM}
USER = ${GITEA_MAILER_USER}
PASSWD = ${GITEA_MAILER_PASSWD}
[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = true

1015
gitea/app.ini.sample Normal file

File diff suppressed because it is too large Load Diff

12
gitea/run.sh Normal file
View File

@ -0,0 +1,12 @@
envsubst < /init/app.ini > /data/gitea/conf/app.ini
mkdir /data/gitea/templates
for file in /init/templates/*
do
FILENAME=$(basename "$file")
echo "$file"
echo $FILENAME
envsubst < "$file" > "/data/gitea/templates/$FILENAME"
done
/bin/s6-svscan /etc/s6

17
gitea/templates/home.tmpl Normal file
View File

@ -0,0 +1,17 @@
{{template "base/head" .}}
<div class="home">
<div class="ui stackable middle very relaxed page grid">
<div class="sixteen wide center aligned centered column">
<div>
<img class="logo" src="{{StaticUrlPrefix}}/img/gitea-lg.png" />
</div>
<div class="hero">
<h1 class="ui icon header title">
{{AppName}}
</h1>
<h2>${DESCRIPTION}</h2>
</div>
</div>
</div>
</div>
{{template "base/footer" .}}

7
gitea/try.sh Normal file
View File

@ -0,0 +1,7 @@
for file in /home/giles/stack/gitea/*
do
FILENAME=$(basename "$file")
echo "$file"
echo $FILENAME
#envsubst < "$file" > /data/gitea/templates/$FILENAME
done

View File

@ -1,243 +0,0 @@
# gitea in a stack with drone and guacamole
Remote system support.
Consisting of
* gitea repository - github like self hosted git and web application
* drone ci system
* guacamole - rdp, vnc and ssh over the internet in the browser
## home
(nb when self deploying with drone it will will stick on started and have to be cancelled. This is because the deployment tears down the previous running drone).
NB each time you deploy this using drone it will leave an orphaned network for example:
eventually this will start to cause an error with message: Docker “ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network”
You need to keep clearing out orphaned networks.
## installing docker
Start with a fresh install of Ubuntu server 19.04 connected to the internet
### update packages
```
sudo apt-get update
sudo apt-get upgrade
```
### remove old versions of docker
(if it's a fresh install of linux there shouldn't be any)
```
sudo apt-get remove docker docker-engine docker.io
```
### install docker
```
sudo apt install docker.io
```
### add current user to docker group
logout and back in afterwards
```
sudo usermod -aG docker $USER
```
### start and enable docker
```
sudo systemctl start docker
sudo systemctl enable docker
```
### change ssh port to 2022
```
sudo vi /etc/ssh/sshd_config
```
change Port 2022
### allow root to ssh
```
sudo vi /etc/ssh/sshd_config
```
## set the root password
```
sudo passwd root
```
change PermitRootLogin yes
reboot
start a stack running gitea to host repository.
## stack
### labels
get nodes with
```
docker node ls
```
add label with
```
docker node update --label-add com.sigyl.git-stack=yes [node id]
```
### generate certificates
(rnd file) dd if=/dev/urandom of=~/.rnd bs=256 count=1
where [registry-domain] is the domain on which the registry will be served
sh ca.sh [registry-domain]:5000
sh make-cert.sh [registry-domain] registry
### ngrok auth
obtain ngrok auth token and place in .secrets in form
authtoken: [token]
### initial deploy
```shell
export LOCAL_DOCKER_REGISTRY=registry.local-domain:5000
export CERTBOT_EMAIL=giles.bradshaw@sigyl.com
export GIT_DOMAIN=git.sigyl.com
export REMOTE_DOMAIN=remote.sigyl.com
export DRONE_DOMAIN=drone.sigyl.com
export BLOG_DOMAIN=blog.sigyl.com
export DRONE_RPC_SECRET=$(openssl rand -base64 48)
docker stack deploy -c docker-compose.yml gitea
```
### create gitea drone app with client id and secret
### add secrets to repository in drone
![add secret](./add-secret.png)
* blog-domain
* certbot-email
* drone-domain
* drone-gitea-client-id
* drone-gitea-client-secret
* drone-rpc-secret
* git-domain
* local-docker-registry
* remote-domain
* ssh-password
* ssh-port
* ssh-root-password
* ssh-root-user
* ssh-user
* ssh-host
### kill orphan docker:dind containers
Wen the system is deployed by pushing to repository the docker:dind container will be orphaned and will run forever unless killed..
## guacamole
np no spaces in postgres password
docker stack for guacamole
adapted from https://digitalmccullough.com/posts/setting-up-apache-guacamole-with-docker-stack.html
### initialising
find id
```
docker ps
```
execute initdb.sql
```
docker exec -it $ID psql -U postgres -d guacamole_db -f /initdb.sql
docker exec -it $ID psql -U postgres -d guacamole_db -f /init-user.sql -v password='somepassword' -v user='guacamole_user'
```
initial admin is guacadmin:guacadmin
create a new admin and delete guacadmin
## docker-exec-runner on windows
These instructions are not very good...
https://exec-runner.docs.drone.io/installation/windows/
download and unpack on linux with
```
curl -L https://github.com/drone-runners/drone-runner-exec/releases/latest/download/drone_runner_exec_windows_amd64.tar.gz | tar zx
```
rename drone-runner-exec to drone-runner-exec.exe
make directory c:\Drone\drone-runner-exec on windows
copy drone-runner-exec.exe to directory
make config file with
```
DRONE_RPC_PROTO=https
DRONE_RPC_HOST=drone.sigyl.com:443
DRONE_RPC_SECRET=[rpc secret]
DRONE_LOG_FILE=C:\Drone\drone-runner-exec\log.txt
DRONE_RUNNER_LABELS=web:true
```
install and start service with
```
drone-runner-exec service install
drone-runner-exec service start
```
## chat
Once the chat-mongo container is up you need to get its id and do
```
docker exec -it [id] mongo --eval "printjson(rs.initiate({_id: 'rs0', members: [ { _id: 0, host: 'localhost:27017' } ]}))"
```
then scale up chat
```
docker service scale gitea_chat=1
```