diff --git a/content/network/IPAM.md b/content/network/IPAM.md
index 2f11f56..c00562b 100644
--- a/content/network/IPAM.md
+++ b/content/network/IPAM.md
@@ -35,12 +35,15 @@ IP address tables
|collector.dn42|172.20.129.4|fd42:4242:2601:ac12::1|Global Route Collector|
|pingable.burble.dn42|172.20.129.5|fd42:4242:2601:ac05::1|Pingable IP Address|
|nats.burble.dn42|172.20.129.6|fd42:4242:2601:ac06::1|nats.io Cluster|
-|rproxy.burble.dn42|172.20.129.7|fd42:4242:2601:acf0::1|Distributed NGINX Reverse Proxy|
+| |172.20.129.7| |_Unallocated_|
|whois.burble.dn42|172.20.129.8|fd42:4242:2601:ac43::1|Whois service|
|voip.burble.dn42|172.20.129.9|fd42:4242:2601:37:216:3eff:fe8f:6211|Asterisk VOIP Service|
|shell.burble.dn42|172.20.129.10|fd42:4242:2601:ac22::1|Shell service|
-|envoy.burble.dn42|172.20.129.11|fd42:4242:2601:ac81::1|Envoy load balancer & proxy|
-| |_172.20.129.12-19_| |_Unallocated_|
+| |172.20.129.11| |_Unallocated_|
+|traefik.burble.dn42|172.20.129.12|fd42:4242:2601:ac82::1|Global traefik cluster|
+|traefik-eu.burble.dn42|172.20.129.13|fd42:4242:2601:ac83::1|Europe traefik cluster|
+|traefik-na.burble.dn42|172.20.129.14|fd42:4242:2601:ac84::1|North America traefik cluster|
+| |_172.20.129.15-19_| |_Unallocated_|
||172.20.129.20/30|_n/a_|[Dialup Service](/retro/modem/) endpoints|
| |_172.20.129.24-26_| |_Unallocated_|
|shell.us-nyc2.burble.dn42|172.20.129.26|fd42:4242:2601:101d:216:3eff:fefc:722|us-nyc2 shell service|
@@ -55,28 +58,28 @@ IP address tables
|*unassigned* |172.20.129.164|fd42:4242:2601:3f::1||
|*unassigned* |172.20.129.165|fd42:4242:2601:3a::1||
|uk-lon2.burble.dn42 |172.20.129.166|fd42:4242:2601:2e::1|Private Node|
-|dn42-ca-bhs2.burble.dn42|172.20.129.167|fd42:4242:2601:2d::1||
+|ca-bhs2.burble.dn42 |172.20.129.167|fd42:4242:2601:2d::1|*being decommissioned 2024*|
|*unassigned* |172.20.129.168|fd42:4242:2601:34::1||
-|dn42-de-fra1.burble.dn42|172.20.129.169|fd42:4242:2601:31::1||
-|*unassigned* |172.20.129.170|fd42:4242:2601:2c::1||
-|*unassigned* |172.20.129.171|fd42:4242:2601:2b::1||
-|dn42-us-lax1.burble.dn42|172.20.129.172|fd42:4242:2601:2a::1||
+|de-fra1.burble.dn42 |172.20.129.169|fd42:4242:2601:31::1||
+|de-fra3.burble.dn42 |172.20.129.170|fd42:4242:2601:2c::1||
+|de-fra2.burble.d42 |172.20.129.171|fd42:4242:2601:2b::1|Private Node|
+|us-lax1.burble.dn42 |172.20.129.172|fd42:4242:2601:2a::1||
|ch-zur2.burble.dn42 |172.20.129.173|fd42:4242:2601:27::1|Private Node|
-|dn42-ch-zur1.burble.dn42|172.20.129.174|fd42:4242:2601:28::1||
-|dn42-us-nyc1.burble.dn42|172.20.129.175|fd42:4242:2601:29::1||
+|ch-zur1.burble.dn42 |172.20.129.174|fd42:4242:2601:28::1||
+|us-nyc1.burble.dn42 |172.20.129.175|fd42:4242:2601:29::1||
|us-nyc2.burble.dn42 |172.20.129.176|fd42:4242:2601:3d::1|Private Node|
|*unassigned* |172.20.129.177|fd42:4242:2601:25::1||
|*unassigned* |172.20.129.178|fd42:4242:2601:24::1||
|*unassigned* |172.20.129.179|fd42:4242:2601:23::1||
-|fr-par2.burble.dn42 |172.20.129.180|fd42:4242:2601:38::1|Private Node|
+|*unassigned* |172.20.129.180|fd42:4242:2601:38::1||
|*unassigned* |172.20.129.181|fd42:4242:2601:37::1||
|*unassigned* |172.20.129.182|fd42:4242:2601:3e::1||
|*unassigned* |172.20.129.183|fd42:4242:2601:3c::1||
|uk-lon3.burble.dn42 |172.20.129.184|fd42:4242:2601:30::1|Private Node|
-|dn42-no-trd1.burble.dn42|172.20.129.185|fd42:4242:2601:39::1||
+|no-trd1.burble.dn42 |172.20.129.185|fd42:4242:2601:39::1||
|nl-ams1.burble.dn42 |172.20.129.186|fd42:4242:2601:32::1|Private Node|
-|dn42-uk-lon1.burble.dn42|172.20.129.187|fd42:4242:2601:35::1||
-|dn42-fr-par1.burble.dn42|172.20.129.188|fd42:4242:2601:36::1||
+|uk-lon1.burble.dn42 |172.20.129.187|fd42:4242:2601:35::1||
+|fr-par1.burble.dn42 |172.20.129.188|fd42:4242:2601:36::1||
|fr-par3.burble.dn42 |172.20.129.189|fd42:4242:2601:26::1|Private Node|
|*reserved* |172.20.129.190|fd42:4242:2601:20::1|Private Node|
|*reserved* |172.20.129.191|fd42:4242:2601:20::1|Private Node|
diff --git a/content/network/nodes.md b/content/network/nodes.md
index 497fa13..09813be 100644
--- a/content/network/nodes.md
+++ b/content/network/nodes.md
@@ -72,6 +72,11 @@ weight: 20
## North America
### dn42-ca-bhs2
+
+{{}}
+being decommissioned 2024
+{{}}
+
|||
|---|---|
|**Location**|OVH (SoYouStart), Beauharnois, Canada|
diff --git a/content/services/acme.md b/content/services/acme.md
new file mode 100644
index 0000000..9d1a03d
--- /dev/null
+++ b/content/services/acme.md
@@ -0,0 +1,167 @@
+---
+title: "ACME"
+geekdocDescription: "ACME Service"
+weight: 55
+---
+burble.dn42 provides an [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment)
+service using an intermediate certificate issued by the
+[dn42 certificate authority](https://dn42.dev/services/Certificate-Authority) and implemented using
+a [HashiCorp Vault](https://vaultproject.io/) cluster to provide a highly available service.
+
+The following ACME challenge types are supported:
+
+ - http-01
+ - dns-01
+ - tls-alpn-01
+
+## dn42 endpoint
+
+-
+
+The dn42 endpoint serves certificates signed by an intermediate certificate issued by the
+[dn42 certificate authority](https://dn42.dev/services/Certificate-Authority).
+
+{{}}
+Note that certificates are issued with a validity period of **30 days**, which is
+shorter than most clearnet ACME services.
+
+The recommended interval to check for expiry is 5 days.
+{{}}
+
+## Staging endpoint
+
+-
+
+The staging endpoint can be used for testing and issues junk certificates.
+The service uses an internal certificate authority that is specific to the staging service
+and should not be trusted.
+
+The staging service issues short lived certificates with a validity period of a few days.
+
+## Certificate Transparency
+
+**TODO** A simpler process will be provided at a future stage, in the meantime the vault
+API can be queried manually to list issued certificates.
+
+--
+
+Vault provides an API for listing issued certificates, however the process for doing this
+is somewhat complicated if you have not used vault before. The instructions below detail
+how to interrogate the service using the vault CLI, however it is also possible to run
+through the same process via the
+[HTTP API](https://developer.hashicorp.com/vault/api-docs?product_intent=vault).
+
+```sh
+# The API endpoint to list issued certificates is an authenticated
+# endpoint that requires a vault token to access it.
+#
+# The burble.dn42 service includes an anonymous login that can be
+# used to obtain a suitable token.
+
+# set the VAULT_ADDR environment variable to the ACME service
+
+$ export VAULT_ADDR="https://acme.burble.dn42"
+
+# you can also set VAULT_SKIP_VERIFY=1 if you do not have the
+# dn42 certificate authority installed.
+
+# Issue an anonymous token and store it in the VAULT_TOKEN env variable
+
+$ export VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=anonymous)
+
+# now the vault API can be accessed
+
+
+# list issued certificates
+
+$ vault list dn42/certs
+
+Keys
+----
+06:72:54:74:02:eb:68:da:62:76:14:92:b4:84:19:36:b1:d1:d0:5c
+0c:bb:39:a0:0a:aa:9c:d9:06:e8:9e:87:ff:54:73:c4:a6:42:9c:f0
+13:91:4f:f7:3a:0b:ca:38:cd:c6:6e:7d:4d:fb:c5:7c:ed:b0:79:1b
+39:5c:46:16:27:d8:f7:30:cc:64:1a:3c:6c:ff:c4:ac:f9:3c:3c:9c
+4b:24:32:48:d0:64:55:3b:dd:b3:00:c6:33:2d:0f:3e:eb:d7:50:02
+4c:8f:ce:e6:18:7a:05:c1:a3:11:45:c9:3c:34:0f:50:e0:75:6d:fd
+5a:03:a9:5b:07:60:d0:fb:25:28:4b:e9:93:a8:22:cd:78:d1:29:b2
+5d:26:b4:47:59:0c:0a:e9:88:b6:97:1d:2a:2b:e5:cb:d2:90:34:9e
+65:c8:33:07:fc:9a:aa:fd:85:6b:fd:b4:de:29:71:e3:8e:6c:f2:11
+68:e1:a6:4a:e1:58:ee:71:c7:a6:12:48:e2:7a:c5:84:c1:7c:21:5e
+75:cf:16:f9:06:71:ea:86:1c:51:95:89:c9:1d:ea:a1:eb:f5:6f:83
+76:91:6e:6a:23:14:00:7c:5f:c7:de:91:c4:40:73:d9:51:b4:f8:4d
+
+# view an invidual certificate
+
+$ vault read -field certificate "dn42/cert/76:91:6e:6a:23:14:00:7c:5f:c7:de:91:c4:40:73:d9:51:b4:f8:4d"
+
+-----BEGIN CERTIFICATE-----
+MIIDTTCCAjWgAwIBAgIUdpFuaiMUAHxfx96RxEBz2VG0+E0wDQYJKoZIhvcNAQEL
+BQAwVTELMAkGA1UEBhMCWEQxDTALBgNVBAoTBGRuNDIxFDASBgNVBAsTC2J1cmJs
+
+...snip...
+
+yait1CFFq4g9/bvsNfIsvN6EJ/BGXqqww6BzKt/ioSLj
+-----END CERTIFICATE-----
+
+# human readable output using the step CLI (https://smallstep.com/)
+
+$ vault read -field certificate "dn42/cert/06:72:54:74:02:eb:68:da:62:76:14:92:b4:84:19:36:b1:d1:d0:5c" | step certificate inspect
+Certificate:
+ Data:
+ Version: 3 (0x2)
+ Serial Number: 36803586486229131299250018793512622456839458908 (0x672547402eb68da62761492b4841936b1d1d05c)
+ Signature Algorithm: SHA256-RSA
+ Issuer: C=XD,O=dn42,OU=burble.dn42,CN=burble.dn42 staging ACME
+ Validity
+ Not Before: Oct 2 18:21:36 2023 UTC
+ Not After : Nov 3 18:22:06 2023 UTC
+ Subject: CN=drone.git.dn42
+ Subject Public Key Info:
+ Public Key Algorithm: RSA
+ Public-Key: (4096 bit)
+ Modulus:
+...snip...
+ Exponent: 65537 (0x10001)
+ X509v3 extensions:
+ X509v3 Key Usage: critical
+ Digital Signature, Key Encipherment, Key Agreement
+ X509v3 Extended Key Usage:
+ Server Authentication
+ X509v3 Subject Key Identifier:
+ 01:4A:7E:02:F3:B7:78:03:66:F9:21:97:4B:31:34:7C:31:DE:BB:86
+ X509v3 Authority Key Identifier:
+ keyid:94:D1:C3:60:C7:88:81:A6:8C:37:AE:40:42:22:48:6B:5F:36:8F:CC
+ Authority Information Access:
+ OCSP - URI:https://acme.burble.dn42/v1/dn42/ocsp
+ CA Issuers - URI:https://acme.burble.dn42/v1/dn42/ca
+ X509v3 Subject Alternative Name:
+ DNS:drone.git.dn42
+ X509v3 CRL Distribution Points:
+ Full Name:
+ URI:https://acme.burble.dn42/v1/dn42/crl
+ Signature Algorithm: SHA256-RSA
+...snip...
+```
+
+## Implementation
+
+The ACME implementation is provided by a 3-node [HashiCorp Vault](https://www.vaultproject.io/)
+cluster behind the [burble.dn42 traefik load balancer](/services/internal/#traefik--traefik-eu--traefik-na). Together they provide a global,
+high availability service.
+
+The cluster currently runs on the following nodes:
+
+ - ch-zur2
+ - de-fra1
+ - fr-par1
+
+At any time the cluster has one leader which processes all requests and replicates state to the
+cluster members. The leader node automatically switches to one of the backup servers should
+a failure occur.
+
+The traefik load balancer runs health checks against the vault servers and automatically redirects
+users to the vault cluster leader.
+
+See the [vault HA reference architecture](https://developer.hashicorp.com/vault/tutorials/day-one-raft/raft-reference-architecture) for more details.
+
diff --git a/content/services/dn42.md b/content/services/dn42.md
index 7126973..9bf9ac2 100644
--- a/content/services/dn42.md
+++ b/content/services/dn42.md
@@ -32,14 +32,19 @@ read-only.
Please note that updates to the wiki may take several hours to sync with other mirrors.
-The service is provided by regional mirrors fronted by an nginx proxy that is itself
-anycasted across burble.dn42. The service is fully meshed and will continue to
-operate as long as at least one proxy and mirror is available.
+The wiki service is delivered using the [burble.dn42 nomad cluster](/services/internal/#nomadburbledn42).
-Mirrors are located in the following locations:
+## ACME
-* dn42-de-fra1
-* dn42-ca-bhs2
+ -
+ -
+
+burble.dn42 provides an [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment)
+service using an intermediate certificate issued by the
+[dn42 certificate authority](https://dn42.dev/services/Certificate-Authority) and implemented using
+a [HashiCorp Vault](https://vaultproject.io/) cluster to provide a highly available service.
+
+More details can be found on the [ACME service](/services/acme/) page.
## Whois Service
diff --git a/content/services/internal.md b/content/services/internal.md
index c689d33..8007eb6 100644
--- a/content/services/internal.md
+++ b/content/services/internal.md
@@ -6,27 +6,15 @@ weight: 30
This page provides some documenation on other services used within burble.dn42
that are not directly available for public use.
-## rproxy.burble.dn42
+## traefik / traefik-eu / traefik-na
-Core nodes run an [nginx](https://nginx.com) container that acts as a reverse proxy
-for services hosted in tier2.
+burble.dn42 runs a global [traefik](https://traefik.io/traefik/) cluster which
+acts as a reverse proxy and load balancer for burble.dn42 web services.
-The reverse proxy is distributed to improve local response times and is
-anycast as rproxy.burble.dn42. Most web services provided by burble.dn42 are
-simply CNAMEs to the reverse proxy which then balances and forwards the
-request to the actual service.
-
-As well as a reverse proxy, nginx also provides:
-
-- TLS termination
-- A local page cache to act as a poor man's CDN
-- Static content server
-
-## envoy.burble.dn42
-
-[Envoy Proxy](https://www.envoyproxy.io/) is being introduced as a health
-checking proxy and load balancer to replace anycasts and nginx for some
-workloads.
+The traefik instances are anycast globally (traefik.burble.dn42), but also
+have regional load balancing groups for Europe (traefik-eu.burble.dn42) and
+North America (traefik-na.burble.dn42). This regional split helps to direct
+users to local servcices where possible.
## vault.burble.dn42
@@ -69,11 +57,23 @@ user token during the deployment process. This ensures that even if access was
gained to the deployment server, secrets could still not be accessed without
also having access to a live user token.
+## nomad.burble.dn42
+
+burble.dn42 runs a global [HashiCorp Nomad](https://www.nomadproject.io/) cluster
+that is used primarily for web application workloads. Nomad integrates with
+[containerd](https://containerd.io/),
+[vault](/services/internal/#vaultburbledn42and) and
+[traefik](/services/internal/#traefik--traefik-eu--traefik-na)
+to provide resilient, globally available applications.
+
+The nomad configuration is publically available in the [burble.dn42 git](https://git.burble.dn42/burble.dn42/nomad) instance.
+
## nats.burble.dn42
burble.dn42 operates a [nats.io](https://nats.io/) cluster as a distributed,
-network wide, broadcast and RPC solution. The cluster uses Vault managed,
-ephemeral TLS certs for authentication and encryption.
+network wide, broadcast and RPC solution.
+
+The cluster uses decentralised JWT tokens for authentication.
## ci.burble.dn42
diff --git a/content/services/public.md b/content/services/public.md
index ba34b79..70fa5d3 100644
--- a/content/services/public.md
+++ b/content/services/public.md
@@ -29,14 +29,6 @@ Functionality includes:
- Changing your shell for the shell services
- Viewing peering information
-## Issue Log
-
-A public issue log is maintained on the [DN42 Registry](https://git.dn42.dev).
-
-- [Issue Log](https://git.dn42.dev/burble/burble.dn42/issues)
-
-Users are welcome to raise issues or enhancements via the log.
-
## Diagnostic Services
### Looking Glass
@@ -69,20 +61,6 @@ please be considerate and configure a reasonable test frequency.
In all cases, do not set the ping frequency to be higher than once a second.
{{}}
-### Speed Test Service
-
-A speed test service is available in a few select locations.
-Note that the service is currently available over IPv6 only at this time.
-
-|Location|URL|Speed|
-|:--|:--|:--|
-| Paris, France | [https://speedtest.fr-par1.burble.dn42](https://speedtest.fr-par1.burble.dn42) | 1gbit down / 1gbit up |
-| Beauharnois, Canada | [https://speedtest.ca-bhs2.burble.dn42](https://speedtest.ca-bhs2.burble.dn42) | 1gbit down / 500mbit up |
-
-{{}}
-Remember this service is provided for your benefit, use responsibly.
-{{}}
-
## Network Status and Reporting
### Grafana Dashboards
@@ -92,11 +70,6 @@ Remember this service is provided for your benefit, use responsibly.
### Uptime monitoring
-- [https://uptime.burble.dn42/status/bdn42](https://uptime.burble.dn42/status/bdn42) (dn42)
-- [https://uptime.burble.com/status/bdn42](https://uptime.burble.com/status/bdn42) (com)
-
-A self-hosted instance of [Uptime Kuma](https://github.com/louislam/uptime-kuma) provides the current status and alerts on many of the burble.dn42 services.
-
- [https://stats.uptimerobot.com/l2913c0R6](https://stats.uptimerobot.com/l2913c0R6)
Major nodes are also monitored off-network by [UptimeRobot](https://uptimerobot.com/).
@@ -123,14 +96,6 @@ The burble.dn42 shell service provides shell accounts for dn42 users who
have a burble.dn42 password or SSH auth methods in the registry.
See the [Shell Accounts](/services/shell/) page.
-## Web SSH Client
-
-- [https://sshwifty.burble.dn42/](https://sshwifty.burble.dn42/)
-
-[sshwifty](https://github.com/nirui/sshwifty) provides a web based terminal for telnet and SSH sessions.
-
-Configuration includes presets for the burble.dn42 shell servers and collector.dn42.
-
## S3 Compatible Object Store
- [https://minio.burble.dn42](https://minio.burble.dn42) - Web interface
@@ -259,3 +224,10 @@ Log in using your burble.dn42 username/password to lurk on
[#dn42](https://wiki.dn42.us/services/IRC).
(set a password using the [the burble.dn42 service portal](https://svc.burble.dn42/))
+## Invidious instance
+
+-
+
+burble.dn42 instance of [Invidious](https://invidious.io) the open source
+alternative front-end to YouTube.
+