This commit is contained in:
parent
a1d50ae958
commit
0be1840c4c
GPG Key ID:
0FCCD13AE1CF7ED8
6
.gitignore
vendored
6
.gitignore
vendored
@ -3,4 +3,8 @@
|
||||
.#*
|
||||
|
||||
# ignore public directory
|
||||
site/public/*
|
||||
site/public/*
|
||||
|
||||
# ignore hugo
|
||||
.hugo_build.lock
|
||||
|
||||
|
||||
@ -51,7 +51,7 @@ IP address tables
|
||||
|dn42-ca-bhs2.burble.dn42|172.20.129.167|fd42:4242:2601:2d::1||
|
||||
|*unassigned* |172.20.129.168|fd42:4242:2601:34::1||
|
||||
|dn42-de-fra1.burble.dn42|172.20.129.169|fd42:4242:2601:31::1||
|
||||
|dn42-es-mad1.burble.dn42|172.20.129.170|fd42:4242:2601:2c::1|*Decommissioning - December 2021*|
|
||||
|*unassigned* |172.20.129.170|fd42:4242:2601:2c::1||
|
||||
|us-phx1.burble.dn42 |172.20.129.171|fd42:4242:2601:2b::1|Private Node|
|
||||
|dn42-us-dal3.burble.dn42|172.20.129.172|fd42:4242:2601:2a::1|*Decommissioning - April 2022*|
|
||||
|uk-lon3.burble.dn42 |172.20.129.173|fd42:4242:2601:27::1|Private Node|
|
||||
@ -60,9 +60,9 @@ IP address tables
|
||||
|*unassigned* |172.20.129.176|fd42:4242:2601:3d::1||
|
||||
|us-nyc1.burble.dn42 |172.20.129.177|fd42:4242:2601:25::1|Temporary Node|
|
||||
|*unassigned* |172.20.129.178|fd42:4242:2601:24::1||
|
||||
|dn42-hk-hkg1.burble.dn42|172.20.129.179|fd42:4242:2601:23::1|*Decommissioning - January 2022*|
|
||||
|*unassigned* |172.20.129.179|fd42:4242:2601:23::1||
|
||||
|*unassigned* |172.20.129.180|fd42:4242:2601:38::1||
|
||||
|dn42-sg-sin2.burble.dn42|172.20.129.181|fd42:4242:2601:37::1|*Decommissioning - January 2022*|
|
||||
|*unassigned* |172.20.129.181|fd42:4242:2601:37::1||
|
||||
|*unassigned* |172.20.129.182|fd42:4242:2601:3e::1||
|
||||
|*unassigned* |172.20.129.183|fd42:4242:2601:3c::1||
|
||||
|se-sto1.burble.dn42 |172.20.129.184|fd42:4242:2601:30::1|Private Node|
|
||||
@ -71,8 +71,8 @@ IP address tables
|
||||
|dn42-uk-lon1.burble.dn42|172.20.129.187|fd42:4242:2601:35::1||
|
||||
|dn42-fr-rbx1.burble.dn42|172.20.129.188|fd42:4242:2601:36::1||
|
||||
|dn42-fr-rbx2.burble.dn42|172.20.129.189|fd42:4242:2601:26::1||
|
||||
|dn42-uk-bri1.burble.dn42|172.20.129.190|fd42:4242:2601:20::1|Private Node|
|
||||
| |172.20.129.191| |Reserved|
|
||||
|*reserved* |172.20.129.190|fd42:4242:2601:20::1|Private Node|
|
||||
|*reserved* |172.20.129.191|fd42:4242:2601:20::1|Private Node|
|
||||
|
||||
## burble.dn42 Nodes (Public Addressing)
|
||||
|
||||
@ -83,13 +83,10 @@ IP address tables
|
||||
|dn42-de-fra1.burble.com|193.41.237.149|2a0d:5941:1:17c::4e2a|
|
||||
|dn42-ch-zur1.burble.com|45.91.92.111|2a0e:dc0:6:8::1|
|
||||
|dn42-no-trd1.burble.com|217.168.87.226|2001:678:dd0:ffff::25|
|
||||
|dn42-es-mad1.burble.com|45.132.74.100|2a0e:dc0:9:5::ab2d|
|
||||
|dn42-ca-bhs2.burble.com|192.99.6.65|2607:5300:60:3741::1|
|
||||
|dn42-us-nyc1.burble.com|107.173.23.178|*2001:470:1f06:56b::2*|
|
||||
|dn42-us-dal3.burble.com|144.172.126.201|2602:fe64:8::4|
|
||||
|dn42-us-lax1.burble.com|185.215.224.214|2a0b:ae40:1:4a0a::5a|
|
||||
|dn42-sg-sin2.burble.com|139.99.89.157|2402:1f00:8000:800::3bc|
|
||||
|dn42-hk-hkg1.burble.com|103.73.66.189|2406:ef80:1:1a::1|
|
||||
|
||||
|DNS|IPv4|IPv6|
|
||||
|:--|:--|:--|
|
||||
|
||||
@ -38,27 +38,6 @@ burble.dn42 implements [large BGP communities](http://largebgpcommunities.net/),
|
||||
| ( 4242422601 : 130 : 1 ) | Route is a direct peer |
|
||||
| ( 4242422601 : 140 : _DN42 region_ ) | Route learned in this DN42 region |
|
||||
|
||||
### Internal Ghost Route Protection
|
||||
|
||||
In a large network like burble.dn42 it can take some time for route updates
|
||||
to be distributed and for the network to settle following changes. As changes
|
||||
ripple through the network it can create a cascade of ghost updates, each with
|
||||
an increasing internal path length, or adjusted latency metric. Worse the
|
||||
cascade of updates can also be re-distributed to peers, creating a significant
|
||||
multiplier for the number of updates from a single change.
|
||||
|
||||
To protect against ghost routes, a community is used to track how many internal
|
||||
hops a route has. In a fully meshed network like burble.dn42, a genuine route should
|
||||
never have more than two internal hops, so an export filter is used to prevent
|
||||
distribution of longer paths.
|
||||
|
||||
|Community |Description|
|
||||
|:---|:---|
|
||||
| ( 4242422601 : 100 : 1 ) | Added on first internal re-distribution |
|
||||
| ( 4242422601 : 100 : 2 ) | Added on second re-distribution and prevents further re-distribution within burble.dn42 |
|
||||
|
||||
The ( 4242422601 : 100 : x ) communities are not exported to peers.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
109
content/network/design.md
Normal file
109
content/network/design.md
Normal file
@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "Network Design"
|
||||
geekdocDescription: "burble.dn42 network design"
|
||||
weight: 0
|
||||
---
|
||||
This page documents some key elements of the current burble.dn42 design.
|
||||
|
||||
|
||||
## Tunnel Mesh
|
||||
|
||||
{{<figure src="/design/DN42-Tunnels.svg" width="80%">}}
|
||||
|
||||
Hosts within the burble.dn42 network are joined using an IPsec/L2TP mesh.
|
||||
Static, unmanaged, L2TP tunnels operate at the IP level and are configured
|
||||
to create a full mesh between nodes. IPsec in transport mode protects the
|
||||
L2TP protocol traffic.
|
||||
|
||||
Using L2TP allows for a large, virtual MTU of 4310 between nodes; this is
|
||||
chosen to spread the encapsulation costs of higher layers across packets.
|
||||
L2TP also allows for multiple tunnels between hosts and these are sometimes
|
||||
used to separate low level traffic without incurring the additional overheads
|
||||
of VXLANs (e.g. for NFS cross mounting).
|
||||
|
||||
The network also supports using point to point wireguard tunnels instead of the
|
||||
IPsec/L2TP mesh. In this case, the large in-tunnel MTU requires UDP fragmentation
|
||||
support between the hosts.
|
||||
|
||||
Network configuration on hosts is managed by systemd-networkd.
|
||||
|
||||
## BGP EVPN
|
||||
|
||||
|
||||
|
||||
Overlaying the IPsec/L2TP mesh is a set of VXLANs managed by a BGP EVPN.
|
||||
|
||||
The VXLANs are primarily designed to tag and isolate transit traffic, making
|
||||
their use similar to MPLS.
|
||||
|
||||
The Babel routing protocol is used to discover loopback addresses between nodes;
|
||||
Babel is configured to operate across the point to point L2TP tunnels and with a static,
|
||||
latency based metric that is applied during deployment.
|
||||
|
||||
The BGP EVPN uses [FRR](https://frrouting.org/) with two global route reflectors
|
||||
located on different continents, for redundency. Once overheads are taken in to account
|
||||
the MTU within each VXLAN is 4260.
|
||||
|
||||
## dn42 Core Routing
|
||||
|
||||
|
||||
|
||||
Each host in the network runs an unprivileged LXD container that acts as a dn42 router
|
||||
for that host. The container uses [Bird2](https://bird.network.cz/) and routes between
|
||||
dn42 peer tunnels, local services on the same node and transit to the rest of the
|
||||
burble.dn42 network via a single dn42 core VXLAN.
|
||||
|
||||
Local services and peer networks are fully dual stack IPv4/IPv6 however the transit
|
||||
VXLAN uses purely IPv6 link-local addressing, making use of BGP multiprotocol and
|
||||
extended next hop capabilities for IPv4.
|
||||
|
||||
The transit VXLAN and burble.dn42 services networks use an MTU of 4260, however the
|
||||
dn42 BGP configuration includes internal communities to distribute desintation MTU across
|
||||
the network for per-route MTUs. This helps ensure path mtu discovery
|
||||
takes place as early and efficiently as possible.
|
||||
|
||||
Local services on each host are provided by [LXD](https://linuxcontainers.org/lxd/introduction/)
|
||||
containers or VMs connecting to internal network bridges.
|
||||
These vary across hosts but typically include:
|
||||
|
||||
- **tier1** - used for publically avialable services (DNS, web proxy, etc)
|
||||
- **tier2** - used for internal services, with access restricted to burble.dn42 networks
|
||||
|
||||
Other networks might include:
|
||||
|
||||
- **dmz** - used for hosting untrusted services (e.g. the shell servers)
|
||||
- **dn42 services** - for other networks, such as the registry services
|
||||
|
||||
dn42 peer tunnels are created directly on the host and then injected in to the
|
||||
container using a small script, allowing the router container itself to remain unprivileged.
|
||||
|
||||
The routers also run nftables for managing access to each of the networks,
|
||||
[bird_exporter](https://github.com/czerwonk/bird_exporter) for metrics and the
|
||||
[bird-lg-go](https://github.com/xddxdd/bird-lg-go) proxy for the
|
||||
burble.dn42 [looking glass](https://lg.burble.com).
|
||||
|
||||
## Host Configuration
|
||||
|
||||
|
||||
|
||||
burble.dn42 nodes are designed to have the minimum functionality at the host level,
|
||||
with all major services being delivered via virtual networks, containers and VMs.
|
||||
|
||||
Hosts have three main functions:
|
||||
|
||||
- connecting in to the burble.dn42 IPsec/L2TP mesh and BGP EVPN
|
||||
- providing internal bridges for virtual networks
|
||||
- hosting [LXD](https://linuxcontainers.org/lxd/introduction/) containers and VMs
|
||||
|
||||
Together these three capabilities allow for arbitary, isolated networks and services
|
||||
to be created and hosted within the network.
|
||||
|
||||
The hosts also provide a few ancillary services:
|
||||
|
||||
- delivering clearnet access for internal containers/VMs using an internal bridge.
|
||||
The host manages addressing and routing for the bridge to allow clearnet access independent
|
||||
of the host capabilities (e.g. proxied vs routed IPv6 connectivity)
|
||||
- creating dn42 peer tunnels and injecting them in to the dn42 router container
|
||||
- monitoring via [netdata](https://www.netdata.cloud/)
|
||||
- backup using [borg](https://borgbackup.readthedocs.io/en/stable/)
|
||||
|
||||
@ -85,21 +85,6 @@ weight: 20
|
||||
|
||||
## Being Decommissioned
|
||||
|
||||
### dn42-es-mad1
|
||||
|
||||
{{<hint danger>}}Will be removed December 2021{{</hint>}}
|
||||
|
||||
|||
|
||||
|---|---|
|
||||
|**Location**|HostHatch, Madrid, Spain|
|
||||
|**Specs**|1 shared core, 2GB, 20GB SSD, 2TB bw|
|
||||
|**Public Hostname**|dn42-es-mad1.burble.com|
|
||||
|**Public IPv4 Address**|45.132.74.100|
|
||||
|**Public IPv6 Address**|2a0e:dc0:9:5::ab2d|
|
||||
|**Tunnel IPv4 Peer Address**|172.20.129.170/32|
|
||||
|**Tunnel IPv6 Link Local**|fe80::42:2601:2c:1/64|
|
||||
|**Tunnel IPv6 ULA**|fd42:4242:2601:2c::1/128|
|
||||
|
||||
### dn42-us-dal3
|
||||
|
||||
{{<hint danger>}}Will be removed April 2022{{</hint>}}
|
||||
@ -130,34 +115,3 @@ weight: 20
|
||||
|**Tunnel IPv6 Link Local**|fe80::42:2601:3a:1/64|
|
||||
|**Tunnel IPv6 ULA**|fd42:4242:2601:3a::1/128|
|
||||
|
||||
### dn42-sg-sin2
|
||||
|
||||
{{<hint danger>}}Will be removed January 2022{{</hint>}}
|
||||
|
||||
|||
|
||||
|---|---|
|
||||
|**Location**|OVH, Singapore|
|
||||
|**Specs**|1 shared core, 2GB, 20GB SSD, 2TB bw|
|
||||
|**Public Hostname**|dn42-sg-sin2.burble.com|
|
||||
|**Public IPv4 Address**|139.99.89.157|
|
||||
|**Public IPv6 Address**|2402:1f00:8000:800::3bc|
|
||||
|**Tunnel IPv4 Peer Address**|172.20.129.181/32|
|
||||
|**Tunnel IPv6 Link Local**|fe80::42:2601:37:1/64|
|
||||
|**Tunnel IPv6 ULA**|fd42:4242:2601:37::1/128|
|
||||
|
||||
### dn42-hk-hgk1
|
||||
|
||||
{{<hint danger>}}Will be removed January 2022{{</hint>}}
|
||||
|
||||
|||
|
||||
|---|---|
|
||||
|**Location**|HostHatch, Hong Kong, CN|
|
||||
|**Specs**|2 shared core, 8GB, 40GB SSD, 1TB bw|
|
||||
|**Public Hostname**|dn42-hk-hkg1.burble.com|
|
||||
|**Public IPv4 Address**|103.73.66.189|
|
||||
|**Public IPv6 Address**|2406:ef80:1:1a::1|
|
||||
|**Tunnel IPv4 Peer Address**|172.20.129.179/32|
|
||||
|**Tunnel IPv6 Link Local**|fe80::42:2601:23:1/64|
|
||||
|**Tunnel IPv6 ULA**|fd42:4242:2601:23::1/128|
|
||||
|
||||
|
||||
|
||||
@ -1,45 +0,0 @@
|
||||
---
|
||||
title: "Overview"
|
||||
geekdocDescription: "Introduction to the burble.dn42 network"
|
||||
weight: 0
|
||||
---
|
||||
burble.dn42 is an experimental global network within [DN42](https://dn42.dev).
|
||||
|
||||
The network is well [connected](https://map42.0x7f.cc/) with a large number of
|
||||
peers, and hosts some of the [DN42 core infrastructure](/services/dn42).
|
||||
|
||||
|
||||
|
||||
## Topology
|
||||
|
||||
{{<hint info>}}
|
||||
Dec 2021 - The burble.dn42 network is being re-designed !
|
||||
More details coming soon.
|
||||
{{</hint>}}
|
||||
|
||||
*Previous design*
|
||||
|
||||
All nodes in the burble.dn42 network are fully meshed with wireguard tunnels.<br/>
|
||||
iBGP with [BGP Confederations](https://en.wikipedia.org/wiki/BGP_confederation) and
|
||||
a latency based metric are used as the interior routing protocol between nodes.
|
||||
iBGP is also fully meshed. and the configuration for both iBGP and wireguard tunnels
|
||||
is built using a number of [Ansible](https://www.ansible.com/) scripts.
|
||||
|
||||
The current network design was introduced in December 2019; previous designs for
|
||||
the network have included a VXLAN overlay over the wireguard mesh to create a
|
||||
single layer 2 network, together with the use of OSPF as the
|
||||
IGP. Other variations have included using BABEL, and tinc.
|
||||
|
||||
## Core Technologies
|
||||
|
||||
A selection of key technologies used within the network
|
||||
|
||||
- [Ubuntu](https://ubuntu.com) - node operating system
|
||||
- [Bird2](https://bird.network.cz/) - routing daemon
|
||||
- [LXD](https://linuxcontainers.org/lxd/introduction/) - for virtualisation and containers
|
||||
- [Packer](https://www.packer.io/) - for container builds
|
||||
- [Alpine Linux](https://alpinelinux.org/) - used for containers
|
||||
- [Ansible](https://www.ansible.com/) - for deploying configuration
|
||||
|
||||
|
||||
|
||||
@ -40,8 +40,6 @@ Mirrors are located in the following locations:
|
||||
|
||||
* dn42-de-fra1
|
||||
* dn42-ca-bhs2
|
||||
* dn42-us-dal3
|
||||
* dn42-us-lax1
|
||||
|
||||
## Whois Service
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@ burble.dn42 runs one of the two master servers that support the DN42 DNS infrast
|
||||
See the [wiki](https://dn42.dev/services/New-DNS#instances_master-delegation-servers-dn42) for
|
||||
more information on the role of the master service.
|
||||
|
||||
The master is hosted on us-dal3, providing geographic and network redundancy against the other DN42 master service, hosted in Europe.
|
||||
The master is hosted on ca-bhs2, providing geographic and network redundancy against the other DN42 master service, hosted in Europe.
|
||||
|
||||
## Authoritative DNS Service
|
||||
|
||||
@ -179,9 +179,6 @@ removed) using [GoBGP](https://github.com/osrg/gobgp) and a health checking scri
|
||||
|:--|:--|:--|
|
||||
| Europe | dns-slave.de-fra1.burble.dn42 | PHP Friends, Frankfurt, Germany |
|
||||
| Americas (East) | dns-slave.ca-bhs2.burble.dn42 | OVH, Beauharnois, Canada |
|
||||
| Americas (Mid) | dns-slave.us-dal3.burble.dn42 | DrServer, Dallas, USA |
|
||||
| Americas (West) | dns-slave.us-lax1.burble.dn42 | LetBox, Los Angeles, USA |
|
||||
| Asia and Oceania | dns-slave.sg-sin2.burble.dn42 | OVH, Singapore |
|
||||
|
||||
The slave nodes are implemented using [PowerDNS](https://www.powerdns.com/).
|
||||
|
||||
|
||||
4
extra/assets/DN42-Core.svg
Normal file
4
extra/assets/DN42-Core.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 114 KiB |
4
extra/assets/DN42-EVPN.svg
Normal file
4
extra/assets/DN42-EVPN.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 86 KiB |
4
extra/assets/DN42-Host.svg
Normal file
4
extra/assets/DN42-Host.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 55 KiB |
4
extra/assets/DN42-Tunnels.svg
Normal file
4
extra/assets/DN42-Tunnels.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 144 KiB |
4
static/design/DN42-Core.svg
Normal file
4
static/design/DN42-Core.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 114 KiB |
4
static/design/DN42-EVPN.svg
Normal file
4
static/design/DN42-EVPN.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 86 KiB |
4
static/design/DN42-Host.svg
Normal file
4
static/design/DN42-Host.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 55 KiB |
4
static/design/DN42-Tunnels.svg
Normal file
4
static/design/DN42-Tunnels.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 144 KiB |
Loading…
x
Reference in New Issue
Block a user