diff --git a/.gitignore b/.gitignore index 3642500..179c44b 100644 --- a/.gitignore +++ b/.gitignore @@ -3,4 +3,8 @@ .#* # ignore public directory -site/public/* \ No newline at end of file +site/public/* + +# ignore hugo +.hugo_build.lock + diff --git a/content/network/IPAM.md b/content/network/IPAM.md index 288b681..5d4e7e8 100644 --- a/content/network/IPAM.md +++ b/content/network/IPAM.md @@ -51,7 +51,7 @@ IP address tables |dn42-ca-bhs2.burble.dn42|172.20.129.167|fd42:4242:2601:2d::1|| |*unassigned* |172.20.129.168|fd42:4242:2601:34::1|| |dn42-de-fra1.burble.dn42|172.20.129.169|fd42:4242:2601:31::1|| -|dn42-es-mad1.burble.dn42|172.20.129.170|fd42:4242:2601:2c::1|*Decommissioning - December 2021*| +|*unassigned* |172.20.129.170|fd42:4242:2601:2c::1|| |us-phx1.burble.dn42 |172.20.129.171|fd42:4242:2601:2b::1|Private Node| |dn42-us-dal3.burble.dn42|172.20.129.172|fd42:4242:2601:2a::1|*Decommissioning - April 2022*| |uk-lon3.burble.dn42 |172.20.129.173|fd42:4242:2601:27::1|Private Node| @@ -60,9 +60,9 @@ IP address tables |*unassigned* |172.20.129.176|fd42:4242:2601:3d::1|| |us-nyc1.burble.dn42 |172.20.129.177|fd42:4242:2601:25::1|Temporary Node| |*unassigned* |172.20.129.178|fd42:4242:2601:24::1|| -|dn42-hk-hkg1.burble.dn42|172.20.129.179|fd42:4242:2601:23::1|*Decommissioning - January 2022*| +|*unassigned* |172.20.129.179|fd42:4242:2601:23::1|| |*unassigned* |172.20.129.180|fd42:4242:2601:38::1|| -|dn42-sg-sin2.burble.dn42|172.20.129.181|fd42:4242:2601:37::1|*Decommissioning - January 2022*| +|*unassigned* |172.20.129.181|fd42:4242:2601:37::1|| |*unassigned* |172.20.129.182|fd42:4242:2601:3e::1|| |*unassigned* |172.20.129.183|fd42:4242:2601:3c::1|| |se-sto1.burble.dn42 |172.20.129.184|fd42:4242:2601:30::1|Private Node| @@ -71,8 +71,8 @@ IP address tables |dn42-uk-lon1.burble.dn42|172.20.129.187|fd42:4242:2601:35::1|| |dn42-fr-rbx1.burble.dn42|172.20.129.188|fd42:4242:2601:36::1|| |dn42-fr-rbx2.burble.dn42|172.20.129.189|fd42:4242:2601:26::1|| -|dn42-uk-bri1.burble.dn42|172.20.129.190|fd42:4242:2601:20::1|Private Node| -|  |172.20.129.191| |Reserved| +|*reserved* |172.20.129.190|fd42:4242:2601:20::1|Private Node| +|*reserved* |172.20.129.191|fd42:4242:2601:20::1|Private Node| ## burble.dn42 Nodes (Public Addressing) @@ -83,13 +83,10 @@ IP address tables |dn42-de-fra1.burble.com|193.41.237.149|2a0d:5941:1:17c::4e2a| |dn42-ch-zur1.burble.com|45.91.92.111|2a0e:dc0:6:8::1| |dn42-no-trd1.burble.com|217.168.87.226|2001:678:dd0:ffff::25| -|dn42-es-mad1.burble.com|45.132.74.100|2a0e:dc0:9:5::ab2d| |dn42-ca-bhs2.burble.com|192.99.6.65|2607:5300:60:3741::1| |dn42-us-nyc1.burble.com|107.173.23.178|*2001:470:1f06:56b::2*| |dn42-us-dal3.burble.com|144.172.126.201|2602:fe64:8::4| |dn42-us-lax1.burble.com|185.215.224.214|2a0b:ae40:1:4a0a::5a| -|dn42-sg-sin2.burble.com|139.99.89.157|2402:1f00:8000:800::3bc| -|dn42-hk-hkg1.burble.com|103.73.66.189|2406:ef80:1:1a::1| |DNS|IPv4|IPv6| |:--|:--|:--| diff --git a/content/network/communities.md b/content/network/communities.md index e6c2a41..3cee980 100644 --- a/content/network/communities.md +++ b/content/network/communities.md @@ -38,27 +38,6 @@ burble.dn42 implements [large BGP communities](http://largebgpcommunities.net/), | ( 4242422601 : 130 : 1 )   | Route is a direct peer | | ( 4242422601 : 140 : _DN42 region_ )   | Route learned in this DN42 region | -### Internal Ghost Route Protection - -In a large network like burble.dn42 it can take some time for route updates -to be distributed and for the network to settle following changes. As changes -ripple through the network it can create a cascade of ghost updates, each with -an increasing internal path length, or adjusted latency metric. Worse the -cascade of updates can also be re-distributed to peers, creating a significant -multiplier for the number of updates from a single change. - -To protect against ghost routes, a community is used to track how many internal -hops a route has. In a fully meshed network like burble.dn42, a genuine route should -never have more than two internal hops, so an export filter is used to prevent -distribution of longer paths. - -|Community  |Description| -|:---|:---| -| ( 4242422601 : 100 : 1 )   | Added on first internal re-distribution | -| ( 4242422601 : 100 : 2 )   | Added on second re-distribution and prevents further re-distribution within burble.dn42 | - -The ( 4242422601 : 100 : x ) communities are not exported to peers. - diff --git a/content/network/design.md b/content/network/design.md new file mode 100644 index 0000000..d290783 --- /dev/null +++ b/content/network/design.md @@ -0,0 +1,109 @@ +--- +title: "Network Design" +geekdocDescription: "burble.dn42 network design" +weight: 0 +--- +This page documents some key elements of the current burble.dn42 design. + + +## Tunnel Mesh + +{{
}} + +Hosts within the burble.dn42 network are joined using an IPsec/L2TP mesh. +Static, unmanaged, L2TP tunnels operate at the IP level and are configured +to create a full mesh between nodes. IPsec in transport mode protects the +L2TP protocol traffic. + +Using L2TP allows for a large, virtual MTU of 4310 between nodes; this is +chosen to spread the encapsulation costs of higher layers across packets. +L2TP also allows for multiple tunnels between hosts and these are sometimes +used to separate low level traffic without incurring the additional overheads +of VXLANs (e.g. for NFS cross mounting). + +The network also supports using point to point wireguard tunnels instead of the +IPsec/L2TP mesh. In this case, the large in-tunnel MTU requires UDP fragmentation +support between the hosts. + +Network configuration on hosts is managed by systemd-networkd. + +## BGP EVPN + +![EVPN diagram](/design/DN42-EVPN.svg) + +Overlaying the IPsec/L2TP mesh is a set of VXLANs managed by a BGP EVPN. + +The VXLANs are primarily designed to tag and isolate transit traffic, making +their use similar to MPLS. + +The Babel routing protocol is used to discover loopback addresses between nodes; +Babel is configured to operate across the point to point L2TP tunnels and with a static, +latency based metric that is applied during deployment. + +The BGP EVPN uses [FRR](https://frrouting.org/) with two global route reflectors +located on different continents, for redundency. Once overheads are taken in to account +the MTU within each VXLAN is 4260. + +## dn42 Core Routing + +![EVPN diagram](/design/DN42-Core.svg) + +Each host in the network runs an unprivileged LXD container that acts as a dn42 router +for that host. The container uses [Bird2](https://bird.network.cz/) and routes between +dn42 peer tunnels, local services on the same node and transit to the rest of the +burble.dn42 network via a single dn42 core VXLAN. + +Local services and peer networks are fully dual stack IPv4/IPv6 however the transit +VXLAN uses purely IPv6 link-local addressing, making use of BGP multiprotocol and +extended next hop capabilities for IPv4. + +The transit VXLAN and burble.dn42 services networks use an MTU of 4260, however the +dn42 BGP configuration includes internal communities to distribute desintation MTU across +the network for per-route MTUs. This helps ensure path mtu discovery +takes place as early and efficiently as possible. + +Local services on each host are provided by [LXD](https://linuxcontainers.org/lxd/introduction/) +containers or VMs connecting to internal network bridges. +These vary across hosts but typically include: + + - **tier1** - used for publically avialable services (DNS, web proxy, etc) + - **tier2** - used for internal services, with access restricted to burble.dn42 networks + +Other networks might include: + + - **dmz** - used for hosting untrusted services (e.g. the shell servers) + - **dn42 services** - for other networks, such as the registry services + +dn42 peer tunnels are created directly on the host and then injected in to the +container using a small script, allowing the router container itself to remain unprivileged. + +The routers also run nftables for managing access to each of the networks, +[bird_exporter](https://github.com/czerwonk/bird_exporter) for metrics and the +[bird-lg-go](https://github.com/xddxdd/bird-lg-go) proxy for the +burble.dn42 [looking glass](https://lg.burble.com). + +## Host Configuration + +![EVPN diagram](/design/DN42-Host.svg) + +burble.dn42 nodes are designed to have the minimum functionality at the host level, +with all major services being delivered via virtual networks, containers and VMs. + +Hosts have three main functions: + + - connecting in to the burble.dn42 IPsec/L2TP mesh and BGP EVPN + - providing internal bridges for virtual networks + - hosting [LXD](https://linuxcontainers.org/lxd/introduction/) containers and VMs + +Together these three capabilities allow for arbitary, isolated networks and services +to be created and hosted within the network. + +The hosts also provide a few ancillary services: + + - delivering clearnet access for internal containers/VMs using an internal bridge. +The host manages addressing and routing for the bridge to allow clearnet access independent +of the host capabilities (e.g. proxied vs routed IPv6 connectivity) + - creating dn42 peer tunnels and injecting them in to the dn42 router container + - monitoring via [netdata](https://www.netdata.cloud/) + - backup using [borg](https://borgbackup.readthedocs.io/en/stable/) + diff --git a/content/network/nodes.md b/content/network/nodes.md index 61a077b..5a4981b 100644 --- a/content/network/nodes.md +++ b/content/network/nodes.md @@ -85,21 +85,6 @@ weight: 20 ## Being Decommissioned -### dn42-es-mad1 - -{{}}Will be removed December 2021{{}} - -||| -|---|---| -|**Location**|HostHatch, Madrid, Spain| -|**Specs**|1 shared core, 2GB, 20GB SSD, 2TB bw| -|**Public Hostname**|dn42-es-mad1.burble.com| -|**Public IPv4 Address**|45.132.74.100| -|**Public IPv6 Address**|2a0e:dc0:9:5::ab2d| -|**Tunnel IPv4 Peer Address**|172.20.129.170/32| -|**Tunnel IPv6 Link Local**|fe80::42:2601:2c:1/64| -|**Tunnel IPv6 ULA**|fd42:4242:2601:2c::1/128| - ### dn42-us-dal3 {{}}Will be removed April 2022{{}} @@ -130,34 +115,3 @@ weight: 20 |**Tunnel IPv6 Link Local**|fe80::42:2601:3a:1/64| |**Tunnel IPv6 ULA**|fd42:4242:2601:3a::1/128| -### dn42-sg-sin2 - -{{}}Will be removed January 2022{{}} - -||| -|---|---| -|**Location**|OVH, Singapore| -|**Specs**|1 shared core, 2GB, 20GB SSD, 2TB bw| -|**Public Hostname**|dn42-sg-sin2.burble.com| -|**Public IPv4 Address**|139.99.89.157| -|**Public IPv6 Address**|2402:1f00:8000:800::3bc| -|**Tunnel IPv4 Peer Address**|172.20.129.181/32| -|**Tunnel IPv6 Link Local**|fe80::42:2601:37:1/64| -|**Tunnel IPv6 ULA**|fd42:4242:2601:37::1/128| - -### dn42-hk-hgk1 - -{{}}Will be removed January 2022{{}} - -||| -|---|---| -|**Location**|HostHatch, Hong Kong, CN| -|**Specs**|2 shared core, 8GB, 40GB SSD, 1TB bw| -|**Public Hostname**|dn42-hk-hkg1.burble.com| -|**Public IPv4 Address**|103.73.66.189| -|**Public IPv6 Address**|2406:ef80:1:1a::1| -|**Tunnel IPv4 Peer Address**|172.20.129.179/32| -|**Tunnel IPv6 Link Local**|fe80::42:2601:23:1/64| -|**Tunnel IPv6 ULA**|fd42:4242:2601:23::1/128| - - diff --git a/content/network/overview.md b/content/network/overview.md deleted file mode 100644 index 310db05..0000000 --- a/content/network/overview.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: "Overview" -geekdocDescription: "Introduction to the burble.dn42 network" -weight: 0 ---- -burble.dn42 is an experimental global network within [DN42](https://dn42.dev). - -The network is well [connected](https://map42.0x7f.cc/) with a large number of -peers, and hosts some of the [DN42 core infrastructure](/services/dn42). - -![logo](/burble-dn42-map.png) - -## Topology - -{{}} -Dec 2021 - The burble.dn42 network is being re-designed ! -More details coming soon. -{{}} - -*Previous design* - -All nodes in the burble.dn42 network are fully meshed with wireguard tunnels.
-iBGP with [BGP Confederations](https://en.wikipedia.org/wiki/BGP_confederation) and -a latency based metric are used as the interior routing protocol between nodes. -iBGP is also fully meshed. and the configuration for both iBGP and wireguard tunnels -is built using a number of [Ansible](https://www.ansible.com/) scripts. - -The current network design was introduced in December 2019; previous designs for -the network have included a VXLAN overlay over the wireguard mesh to create a -single layer 2 network, together with the use of OSPF as the -IGP. Other variations have included using BABEL, and tinc. - -## Core Technologies - -A selection of key technologies used within the network - -- [Ubuntu](https://ubuntu.com) - node operating system -- [Bird2](https://bird.network.cz/) - routing daemon -- [LXD](https://linuxcontainers.org/lxd/introduction/) - for virtualisation and containers -- [Packer](https://www.packer.io/) - for container builds -- [Alpine Linux](https://alpinelinux.org/) - used for containers -- [Ansible](https://www.ansible.com/) - for deploying configuration - - - diff --git a/content/services/dn42.md b/content/services/dn42.md index 23a27c1..ef88e45 100644 --- a/content/services/dn42.md +++ b/content/services/dn42.md @@ -40,8 +40,6 @@ Mirrors are located in the following locations: * dn42-de-fra1 * dn42-ca-bhs2 -* dn42-us-dal3 -* dn42-us-lax1 ## Whois Service diff --git a/content/services/dns.md b/content/services/dns.md index 32eb4b7..90da838 100644 --- a/content/services/dns.md +++ b/content/services/dns.md @@ -27,7 +27,7 @@ burble.dn42 runs one of the two master servers that support the DN42 DNS infrast See the [wiki](https://dn42.dev/services/New-DNS#instances_master-delegation-servers-dn42) for more information on the role of the master service. -The master is hosted on us-dal3, providing geographic and network redundancy against the other DN42 master service, hosted in Europe. +The master is hosted on ca-bhs2, providing geographic and network redundancy against the other DN42 master service, hosted in Europe. ## Authoritative DNS Service @@ -179,9 +179,6 @@ removed) using [GoBGP](https://github.com/osrg/gobgp) and a health checking scri |:--|:--|:--| | Europe | dns-slave.de-fra1.burble.dn42 | PHP Friends, Frankfurt, Germany | | Americas (East) | dns-slave.ca-bhs2.burble.dn42 | OVH, Beauharnois, Canada | -| Americas (Mid)   | dns-slave.us-dal3.burble.dn42 | DrServer, Dallas, USA | -| Americas (West)   | dns-slave.us-lax1.burble.dn42 | LetBox, Los Angeles, USA | -| Asia and Oceania | dns-slave.sg-sin2.burble.dn42 | OVH, Singapore | The slave nodes are implemented using [PowerDNS](https://www.powerdns.com/). diff --git a/extra/assets/DN42-Core.svg b/extra/assets/DN42-Core.svg new file mode 100644 index 0000000..fe4158c --- /dev/null +++ b/extra/assets/DN42-Core.svg @@ -0,0 +1,4 @@ + + + +
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Route Reflectors
Route Reflectors
dn42 routers
dn42 routers
MTU 4260
MTU 42...
MTU 1420 / various
MTU 1420 / various
dn42 peers
Wireguard
OVPN
dn42 peers...
Injected Tunnels
Injected...
Host Bridges
Host Br...
dn42 router lxd
dn42 rout...
svcs
svcs
svcs
svcs
svcs
svcs
svcs
svcs
tier1
public services
tier1...
tier2
internal services
tier2...
nftables
nftab...
MTU 4260
MTU 4260
dmz
dmz
dn42 VXLAN
dn42 VXLANViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/extra/assets/DN42-EVPN.svg b/extra/assets/DN42-EVPN.svg new file mode 100644 index 0000000..f72e5f8 --- /dev/null +++ b/extra/assets/DN42-EVPN.svg @@ -0,0 +1,4 @@ + + + +
BGP EVPN
BGP EVPN
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
Route Reflectors
Route Reflectors
Nodes
Nodes
VXLAN
VXLAN
MTU 4260
MTU 4260
MTU 4310
MTU 4310Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/extra/assets/DN42-Host.svg b/extra/assets/DN42-Host.svg new file mode 100644 index 0000000..3bf4ab5 --- /dev/null +++ b/extra/assets/DN42-Host.svg @@ -0,0 +1,4 @@ + + + +
Internet
Internet
IPSec/L2TP Tunnels
IPSec/L2TP Tunnels
VXLAN Endpoints
VXLAN Endpoints
host managed
networking
host managed...
Host Bridges
Host Bridges
dn42 router
dn42 router
dn42 service
dn42 service
dn42 service
dn42 service
non-dn42
non-dn42
dn42 tier1
dn42 tier1
dn42 tier2
dn42 tier2
direct clearnet access
direct clearnet access
dn42 routing VXLAN
dn42 routing VXLAN
dn42 peers
dn42 peers
tunnel injection
tunnel injection
FRR
FRR
nftables
nftablesViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/extra/assets/DN42-Tunnels.svg b/extra/assets/DN42-Tunnels.svg new file mode 100644 index 0000000..f8213b2 --- /dev/null +++ b/extra/assets/DN42-Tunnels.svg @@ -0,0 +1,4 @@ + + + +
IPSec/L2TP
Mesh
IPSec/L2TP...
MTU 4310
MTU 4310
Wireguard
Wireguard
MTU 4310
MTU 4310Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/static/design/DN42-Core.svg b/static/design/DN42-Core.svg new file mode 100644 index 0000000..fe4158c --- /dev/null +++ b/static/design/DN42-Core.svg @@ -0,0 +1,4 @@ + + + +
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Bird2
Route Reflectors
Route Reflectors
dn42 routers
dn42 routers
MTU 4260
MTU 42...
MTU 1420 / various
MTU 1420 / various
dn42 peers
Wireguard
OVPN
dn42 peers...
Injected Tunnels
Injected...
Host Bridges
Host Br...
dn42 router lxd
dn42 rout...
svcs
svcs
svcs
svcs
svcs
svcs
svcs
svcs
tier1
public services
tier1...
tier2
internal services
tier2...
nftables
nftab...
MTU 4260
MTU 4260
dmz
dmz
dn42 VXLAN
dn42 VXLANViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/static/design/DN42-EVPN.svg b/static/design/DN42-EVPN.svg new file mode 100644 index 0000000..f72e5f8 --- /dev/null +++ b/static/design/DN42-EVPN.svg @@ -0,0 +1,4 @@ + + + +
BGP EVPN
BGP EVPN
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
FRR
Route Reflectors
Route Reflectors
Nodes
Nodes
VXLAN
VXLAN
MTU 4260
MTU 4260
MTU 4310
MTU 4310Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/static/design/DN42-Host.svg b/static/design/DN42-Host.svg new file mode 100644 index 0000000..3bf4ab5 --- /dev/null +++ b/static/design/DN42-Host.svg @@ -0,0 +1,4 @@ + + + +
Internet
Internet
IPSec/L2TP Tunnels
IPSec/L2TP Tunnels
VXLAN Endpoints
VXLAN Endpoints
host managed
networking
host managed...
Host Bridges
Host Bridges
dn42 router
dn42 router
dn42 service
dn42 service
dn42 service
dn42 service
non-dn42
non-dn42
dn42 tier1
dn42 tier1
dn42 tier2
dn42 tier2
direct clearnet access
direct clearnet access
dn42 routing VXLAN
dn42 routing VXLAN
dn42 peers
dn42 peers
tunnel injection
tunnel injection
FRR
FRR
nftables
nftablesViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/static/design/DN42-Tunnels.svg b/static/design/DN42-Tunnels.svg new file mode 100644 index 0000000..f8213b2 --- /dev/null +++ b/static/design/DN42-Tunnels.svg @@ -0,0 +1,4 @@ + + + +
IPSec/L2TP
Mesh
IPSec/L2TP...
MTU 4310
MTU 4310
Wireguard
Wireguard
MTU 4310
MTU 4310Viewer does not support full SVG 1.1 \ No newline at end of file