+
+
+
+
+
+
+
+
+ Archive of changes made in 2018
+ +Migrated US anycast services from dn42-us-dal1 to dn42-us-dal3.
+ +Added Certificate Authority details.
+ +Upgraded the looking glass with Zhaofeng bird-lg fixes.
+ROA data is available through the burble.dn42 website, see the Services page.
+RPKI service is now replicated across regions to provide additional resiliency.
+New version of bird2 deployed, including RPKI fixes from JRB0001.
Added new peers:
+Updated the Services page to include more implementation details.
+Reworked intra-confederation peering to provide more resilience.
+Implemented ROA via RPKI updates using roasrv by Yamakaja and gortr
New node !
+dn42-jp-tyo1 has been commissioned and is open for new peers in Tokyo, Japan.
+ +Updated host information and network map with new nodes.
+ +New peers added:
+dn42-us-sea2 is now operational and available for peering.
+ +tinc + babeld is not a winning combination. Since introducing babeld, the burble.dn42 WAN overlay has experienced a number of periods of instability, with nodes dropping on and off the network.
+The WAN has been updated to use a Wireguard mesh with OSPF as IGP, and is now significantly more stable again.
+ +New peers added:
+Three new nodes will be available for peering soon:
+dn42-us-dal1 locked up, and has been restarted.
+ +dn42-uk-lon1, dn42-lt-vil1, dn42-sg-sin1 and dn42-us-mia1 all locked up at 03:00 UTC and have now been restarted.
+ +Black Friday has delivered four new nodes to the burble.dn42 network:
+All nodes are open to new peers, so just contact dn42@burble.com if you’d like to connect to the network.
+ +New peers added:
+Updates to reverse DNS.
+ +Added new peers
+The internal routing protocol (IGP) for burble.dn42 has moved from OSPF to using babeld.
+All nodes on the burble.dn42 network are inter-connected with a tinc mesh. Despite the network physically spanning across contintents, OSPF saw the tinc overlay network as being flat which prevented effective use of technologies such as anycast and forced the use of central resources. The hope is that babel, configured to use an RTT metric, will allow better use of regional services.
+Please let me know if you observe any issues due to the new IGP.
+ +New node in Istanbul, Turkey.
+dn42-tr-ist1 has been commissioned and is now open for new peers. See the peering page for more details.
+ + +
+
+
+ Archive of changes made in 2019
+ +The Christmas period has been a really busy period for burble.dn42, with integration +and transfer of services over to the new nodes. Primarily, this has meant moving services +from fr-rbx1 and sg-sin2 to fr-rbx2, fr-sbg1 and sg-sin1. As part of the rebuilding, +I’ve also taken the opportunity to re-create most of my ansible scripting, with the +intent that this will eventually be published.
+Most services are now moved, with the main exception of DNS and the GRC, both of which +need more significant work. The website also now needs major updates to reflect the +changes I’ve made.
+The following new nodes are also open for peering:
+Happy New Year
+ +The last month has been spent redesigning my WAN and introducting a latency based metric for +connectivity between nodes. This is now mostly complete, but not without its own follow on +problems that need to be resolved.
+Things still to do include:
+Another new node will also be added, dn42-fr-rbx2 and dn42-fr-rbx1 will be retired.
+Merry Christmas DN42
+ +Black friday is here and new nodes are on the way.
+Retired dn42-us-lax2, dn42-us-chi2, dn42-ca-bhs1, dn42-tr-ist1 and dn42-no-osl1.
+Restructured the internal confederations.
New experimental node added hosted in the Oracle Cloud environment in Mumbai, India.
+Users are welcome to peer and test the node, but should be aware there may be short notice changes +or interruptions to service.
+ +After a few weeks of outage and putting up with influx using up a vast amount of resources, the monitoring +service has finally moved to a federated prometheus architecture. Hopefully this will have better +performance than the influx architecture used previously. At some point I’ll update the monitoring page with +details of the new configuration.
+ +The burble.dn42 wiki service is now part of the global anycast for wiki.dn42.
+See the services page for more details.
+ +The recursive DNS service now supports clearnet queries
+ +Stop supporting IPsec tunnels
+ +Removed sg-sin3 and vn-han1
+ +Added DN42 wiki service editable via dn42, +readonly via clearnet.
+Issued new Certificate Authority root certificate with a longer expiry date.
+ +Added a couple of Python 3 updates for bird-lg that fixes +broken BGP map functionality in the looking glass.
+Influx ate all the memory (10gb!) on de-fra1, so is currently offline until it can be fixed.
+ +Add dn42-us-mia2, which will replace dn42-us-mia1
+ +Add pingable.burble.dn42
+ +Decommissioning of dn42-ru-mos1 and dn42-us-sea1
+ +DoH! The DNS Service now support DNS over HTTPS.
+ +Tidied up node information.
+ +A new host IRC web service has been added, based on thelounge.
+See the services page for more details.
+ +The recursive DNS service now uses parallel queries across all five regional master nodes.
+This approach takes advantage of the burble.dn42 global scale to reduce latencies,
+improve resilience and prevent local connectivity problems from impacting the results.
+See the DNS page for more info.
Moved and extended the DN42 monitoring so that it is more independent and also clustered.
+A writeup of the hosted grafana service and monitoring is available here.
+ +dn42-uk-lon1 is back again after being out of action for the day.
+The host server apparently threw a disk after being updated to cover the MDS vulnerability and the +provider has spent the day recovering the node.
+ +Some nodes may have outages over the next few days as providers deal with the recent MDS vulnerabilities.
+Added new peers
+Updated my fork of bird-lg by merging Zhaofeng’s Python2 to Python3 bird-lg updates and fixing a few outstanding problems.
+The updated code is now live on the burble.dn42 looking glass.
+ +Moved the looking glass to its own container, in anticipation of future website changes
+dn42-us-mia1 is offline again.
+ +dn42-us-chi2 was suspended by the provider on 8/5 due to ‘NTP reflection attacks’.
+This is a hazard of running a busy NTP server as part of the NTP Pool; +providers can get twitchy when they see a large amount of NTP traffic, due to the well publicised +vulnerabilities in stock NTPd.
+My network uses chronyd rather than NTPd and it is simply not vulnerable to abuse in the same way +as NTPd, I also regularly monitor and check the services. On the other hand, the server does see +a large amount of NTP traffic and it can sometimes be difficult demonstrating that I’m +specifically providing a service here and not under some kind of attack.
+Apologies that the server was offline for a few days, but it should now finally be back again.
+For info, here is the bandwidth graph of dn42-us-chi2 as it was suspended:
+
It’s trivial to see that an amplification attack was not occuring, as the inbound and outbound +traffic are both equal. It’s a shame some providers don’t consider this before suspending services, +but, understandable that the economics of providing VPS services can prohibt this.
+Added new peers:
+Added git service.
+See the services page for more details.`<
Seems traceroutes and some Europe Region, IPv4 related DNS lookups weren’t working.
+Both are fixed now.
Added new peers:
+ + +New node added and ready for peering
+With the addition of several new nodes, the internal BGP confederations
+have been re-orginised.
+This new organisation should provide better balance and allow for more local services.
Added new peers:
+New nodes added and ready for peering.
+Over the last week, and number of major changes have taken place to the burble.dn42 network.
+These include:
Over time, internal IPv4 services will be removed
+ +New prefix 172.20.129.0/27 registered to provide space for more nodes +and additional services.
+172.20.129.0/27 will be used as anycast addresses for services. +172.20.129.160/27 will be used for burble.dn42 nodes
+Added new peers:
+Added an old node in to the DN42 network, dn42-sg-sin2. +RPKI and DNS services have been moved to the node from dn42-sg-sin2 +which should improve diversification and stability.
+ +Added new peers:
+The DNS service has gone global, with every node in the burble.dn42 network
+now participating in the DNS Anycast service.
+More details can be found on the DNS page.
Added new peers:
+New node added dn42-de-fra1
+ +Added new peers:
+A new instance of the registry explorer has been created that references +the ‘object-fix’ branch of the DN42 registry. The main purpose of this +is to support the new DNS system being developed.
+ +A couple of the nodes on the network experienced some downtime over the week:
+Added new peers
+Initialised GRC website
+Added new peers
+The Looking Glass has been udpated to use +lgregmapper and data from +dn42regsrv.
+ +New peer added:
+The internal and public ROA service has been moved over to using dn42regsrv.
+See the services page for more details.
New peer added:
+New peers added:
+Updated the services to include new stuff::
+New peers added:
+New service !
+A burble.dn42 route collector has been added, together with +some interesting stats showing reachability of DN42 from the burble.dn42 network.
+A common, global route collector is in progress, see here
+ +New peer added:
+bgpmap updated to add MNT and prefix info for ASes.
+New peers added:
+The Looking Glass now supports bgpmap again.
+My bird-lg fixes are available on github.
New peer added:
+Some layout fixes to the Looking Glass, including fixing whois lookups.
+ +First new peers of 2019:
+Consolidated number of anycast sessions.
+ + +
+
+
+ Archive of changes made in 2020
+ +Issue Log
+burble.dn42 now has a public issue log, hosted on the DN42 Registry.
+Feel free to raise issues or enhancements on the log.
+Speedtest Service
+An experimental speed test service has been introduced:
+The two services are currently only accessible over IPv6 but are hosted on dedicated +servers with plenty of available bandwidth. If the service ends up loading or disrupting +the rest of the network then I may end up removing it, so use responsibly.
+n8n Automation
+The burble.dn42 network now has an instance of n8n to help
+automate internal workflows.
+Whilst this isn’t a public service the first visibile benefit is that the
+Explorer and
+ROA files now update
+immediately following registry changes. Previously changes were polled and could
+take up to an hour to be updated.
ca-bhs2 and fr-rbx1 have been migrated to their new servers. If you are peering with these +nodes please make sure you update any IP addresses on your side as required.
+git.dn42.dev is hosted on ca-bhs2 and so was also migrated and upgraded to v1.13.0.
+ +Black Friday has been been and gone and this means that a few nodes have now reached the end of their contract and are being retired:
+However, the good news is that Black Friday also delivered a few shiny new nodes for the burble.dn42 network:
+Users of ca-bhs2 will be migrated to the new node, with details to be confirmed.
+ +uk-lon1 has been upgraded. If you are peered on this node, please update +your IP address accordingly.
+burble.dn42 now includes some limited protection against ghost route updates. +See the communities page for more details.
+ +uk-lon1 will be upgraded this weekend (12/13th +September), but unfortunately this does mean that the IP address for the +server is going to change.
+The hostname will be changed to match the new address, but if you use the +IP address in your configuration (e.g. for firewall rules), you will need +to update them, as detailed below.
+dn42-uk-lon1.burble.com
+All other peering details, such as encryption keys and tunnel IP addresses +will remain the same, and the new server is in the same datacentre so there +should be no changes to connectivity or latency.
+ +Changes to the burble.dn42 network
+Over the next year the focus of the burble.dn42 network will change focus to providing high quality, reliable services for DN42. As part of this change, a number of the current ‘edge’ nodes will be decommissioned to reduce admin overhead and allow concentration on the core, service nodes.
+The following nodes will be decommissioned and are no longer available for new peerings:
+| Node | +Decommissioning Date | +
|---|---|
| us-mia2 | +Immediately | +
| sg-sin1 | +November 2020 | +
| us-sea2 | +November 2020 | +
| fr-sbg1 | +Nocember 2020 | +
| jp-tyo1 | +December 2020 | +
| au-syd1 | +January 2021 | +
| us-nyc1 | +April 2021 | +
| us-chi1 | +May 2021 | +
| + | + |
The current core nodes will continue to operate and some will also be upgraded. The number of services provided by the network will also expand.
+Core nodes
+| Node | +Future Plans | +
|---|---|
| fr-rbx1 | +Increase in services offered | +
| ca-bhs2 | +Upgrade to ssd disks ~November 2020 | +
| de-fra1 | +Upgraded in August to 4 x Epyc / 20G RAM / NVMe | +
| us-dal3 | +Increase in services offered / potential for upgrade | +
| us-lax1 | +Increase in services offered | +
| sg-sin2 | +Take over services from sg-sin1 | +
| + | + |
Other Nodes
+| Node | +Future Plans | +
|---|---|
| uk-lon1 | +Upgrade before January 2021 | +
| ch-zur1 | +No changes planned | +
| no-trd1 | +No changes planned | +
| + | + |
de-fra1 has been replaced with a shiny, upgraded, new node.
+If you are peered on de-fra1, please check your configuration and ensure +you are using the new IP addresses
+All other peering parameters remain the same.
+ +The DN42 registry now supports automated pipelines using Drone CI.
+Details will be published on the DN42 wiki.
us-mia1 and us-mia2 have been swapped. The provider for the old us-mia2 (Stockservers) +appears to have ceased trading, so the node has been swapped in case the original server +disappears at short notice.
+EDIT: seems the new provider wasn’t better, so us-mia2 is back on the old server +until it dies completely.
+no-trd1 has been added, courtesy of jastrup.
+lt-vil1 is being decommissioned and users will need to migrate to a different node to +maintain service.
+ +A busy weekend supporting the move of the DN42 registry to its new host.
+Remember to join the new mailing list at https://groups.io/g/dn42 +and create yourself an account on the new registry https://git.dn42.dev
+ +Website moved again, and new paste.burble.dn42 service added.
+ +The global route collector has had a long overdue upgrade. Please let me know +if you spot any residual issues.
+ +The new DNS implementation has been deployed across all nodes. +The DNS service now supports:
+All services support UDP, TCP, DNS over HTTPS and DNS over TLS queries. +See the DNS page for more info.
+ +A new implementation of the edge DNS service is currently being tested across +a few nodes, please let me know if you spot any DNS oddness.
+ +Added whois.burble.dn42 service, see the services +page for more details.
+https://explorer.burble.dn42 now has regional mirrors so should be significantly +faster for anyone not in Europe.
+ +Approximately 40 old or inactive peers have been deleted as part of a spring cleaning +exercise.
+If you’ve been accidently deleted and still want to peer with me, just give me a shout +and I will re-instate the configuration.
+ +Rate limiting on BGP sessions has been implemented to protect the network from major +route flapping events. The rate limiting should only kick in after 30+ minutes of +extremely high updates (or even longer for milder events), but please let me know if +this causes any issues.
+ +us-lax1 has been migrated. If you peer with me please remember to update the +clearnet IP addresses on your side:
+dn42-us-lax1
+IPv4: 185.215.224.214
+IPv6: 2a0b:ae40:1:4a0a::5a
us-lax1 is being upgraded !
+Apologies for the short notice, but us-lax1 will be upgraded over the weekend of 9th/10th +May. The upgrade will allow for more services to be provided from the node, to provide +enabling a better response for users in Asia and West Coast US.
+Unfortunately the upgrade means that IP address of the node will change and peers will need +to update their config accordingly. The encryption keys and tunnel addresses should not need +to change.
+ +Several of the burble.dn42 core nodes have been upgraded to Ubuntu 20.04. This required +a short outage, but will allow for a refactoring of a few services in the future.
+ +Bugs have been fixed and both instances of the burble.dn42 website are now running +in a new environment with the latest grav.
+The new website instance is the first burble.dn42 application running on Ubuntu 20.04 +(Focal Fossa).
+ +The clearnet version of this website is running with
+a new instance that has the latest grav.
+Please let me know if you spot any problems.
The DN42 instance continues to run with the previous version.
+ +Well, that was fun; burble.dn42 had a number of outages over this evening, caused by trying +to perform a rolling upgrade across the network. The biggest of these took out the burble.dn42 +DNS service for an extended period, impacting DNS resolution across DN42.
+The plan had been to perform a full upgrade and reboot for every burble.dn42 node. To minimise
+disruption I perform updates across groups of servers that are chosen to be independent
+so that service resilience should not be impacted.
+However, this time there were two key failures:
The provider configuration for ca-bhs2 meant that it could not mount all of its disks when +rebooted and it ended up in maintenance mode. The server needed to be recovered via the +IPMI console. Whilst global services continued to be provided by other nodes, peers on +ca-bhs2 lost connectivity whilst the node was recovered.
+The new pdns-recursor that was implemented at the end of March (see below) had a different +runtime path than the default OS install. This meant that when each of the core nodes was +restarted the pdns-recursor failed to restart as the runtime path was missing. Since the DNS +service is resilient, it continued to operate without problems until the last core node was +restarted, at which point the entire service failed. Without DNS, most of the remaining +burble.dn42 failed or could not be restarted and recovery was also hampered by having to +work without having DNS available.
+at-vie1 will be decommissioned by 14th April. +If you are peered on this node, please contact me to move the peering to another node.
+ +The patched pdns recursor is now deployed to all core nodes.
+Please let me know immediately if you notice odd DNS behaviour.
+ +fr-sbg1 (which hosts the europe region core DNS service) is currently testing a special pdns +recursor build in order to try and fix this issue.
+The server is likely to be used for most recursive DNS lookups across Europe that use the +new DNS anycast addresses, or my service directly. Please let me know immediately if you +notice odd DNS behaviour.
+ +This weekend has been a huge maintenance weekend for burble.dn42, with the following +updates taking place:
+| + | + |
|---|---|
| fr-rbx1 replaced by fr-rbx2 | +fr-rbx2 was a much faster node | +
| ca-bhs2 replaced with a new node | +the replacement is also much faster | +
| us-dal3 replaced by us-dal1 | +us-dal3 was a poor performer and has been replaced with a dedicated server | +
| au-syd1 replaced with a new node | +memory increased from 1G to 2G | +
| sg-sin2 replaced with a new node | +memory increased from 1G to 2G | +
Node renewals are now mostly sorted until November, which will be a nice break for my wallet.
+A bad decision around backups meant that I also had to re-create all the services on +fr-rbx2 and us-dal1 as they were swapped in to their new roles. As a result, the services +on these boxes were also flattened and rebuilt to the new disk layout.
+At some future point, fr-sbg1 will follow and also change to the new layout.
+The core network looked like this prior to November 2019:
+| Name | +CPU | +Memory | +Disk | +Network | +Descr | +
|---|---|---|---|---|---|
| fr-rbx1 | +i5-2400 (4/8 x 3.4Ghz) | +16G | +2TB Consumer HDD | +100mbps un-metered | +Kimsufi KS-10 | +
| ca-bhs2 | +i5-3570S (4/8 x 3.8Ghz) | +16G | +2TB Consumer HDD | +100mbps un-metered | +Kimsufi KS-10 | +
| sg-sin2 | +virtual (1 x 3.5Ghz) | +1G | +30GB HDD | +1TB @ 1gbit | +OVH VPS | +
| us-dal3 | +virtual (2 x 3.4Ghz) | +5G | +120GB HDD | +5TB @ 10gbit | +HostDoc VPS | +
Following the upgrades, the core now consists of the following servers:
+| Name | +CPU | +Memory | +Disk | +Network | +Descr | +
|---|---|---|---|---|---|
| fr-sbg1 | +E5-1620 (4/8 x 3.7Ghz) | +32G | +3 x 480GB SSD | +500mbps un-metered | +OVH SYS | +
| fr-rbx1 | +E3-1245 (4/8 x 3.4Ghz) | +32G | +2 x 480GB SSD | +500mbps un-metered | +OVH SYS | +
| uk-lon3 | +virtual (2 x 3Ghz) | +3G | +3TB HDD | +10TB @ 1gbit | +HostHatch | +
| ca-bhs2 | +E5-1620 (4/8 x 3.7Ghz) | +32G | +2 x 2TB Ent. HDD | +500mbps un-metered | +OVH SYS | +
| us-dal3 | +C2750 (8 x 2.4Ghz) | +8G | +240GB SSD | +100mbps un-metered | +drserver | +
| sg-sin1 | +virtual (4 x 2.2Ghz) | +4G | +24GB SSD | +1gbit un-metered | +ITLDC VPS | +
+
+
+ A log of changes to the burble.dn42 network.
+New static website built using Hugo.
+ +es-mad1 in Madrid, Spain has been deployed and is now open for peering.
+
+
+
+ What can you do in DN42 ? Ultimately, you’ll get out of DN42 what you put in to it, +but I’ve listed here a few ideas that may serve as inspiration and the spark an idea.
+This is deliberately not a set of instructions or a guide and it’s not a checklist of stuff you
+must do.
+If you are interested in something there is plenty of public information available on all these topics.
Congratulations, you’re connected to DN42 !
+ +
+
+
+ An experiment in global networking.
+ + +
+
+
+ IP address tables
+ + +| IPv4 Address Range | +Purpose | +
|---|---|
| 172.20.129.0/27 | +burble.dn42 services | +
| 172.20.129.160/27 | +burble.dn42 nodes | +
| IPv6 Address Range | +Purpose | +
|---|---|
| fd42:4242:2601:acXX::/64 | +Anycast services | +
| fd42:4242:2601:AA::/64 | +Public services for host AA | +
| fd42:4242:2601:AA00::/56 | +/56 routed to host AA | +
| fd42:4242:2601:AA02::/64 | +Tier2 services on host AA | +
| DNS | +IPv4 | +IPv6 | +Comment | +
|---|---|---|---|
| + | 172.20.129.0 | ++ | Reserved | +
| ns1.burble.dn42 | +172.20.129.1 | +fd42:4242:2601:ac53::1 | +Authoritative DNS Master | +
| dns.burble.dn42 | +172.20.129.2 | +fd42:4242:2601:ac53::53 | +Recursive DNS Resolver | +
| burble.dn42 www.burble.dn42 |
+172.20.129.3 | +fd42:4242:2601:ac80::1 | +Website | +
| collector.dn42 | +172.20.129.4 | +fd42:4242:2601:ac12::1 | +Global Route Collector | +
| pingable.burble.dn42 | +172.20.129.5 | +fd42:4242:2601:ac05::1 | +Pingable IP Address | +
| wiki.burble.dn42 | +172.20.129.6 | +fd42:4242:2601:ac81::1 | +DN42 Wiki Mirror | +
| rproxy.burble.dn42 | +172.20.129.7 | +fd42:4242:2601:acf0::1 | +Distributed NGINX Reverse Proxy | +
| whois.burble.dn42 | +172.20.129.8 | +fd42:4242:2601:ac43::1 | +Whois service | +
| + | 172.20.129.10-31 | ++ | Unallocated | +
| DNS | +IPv4 | +IPv6 | +Comment | +
|---|---|---|---|
| unassigned | +172.20.129.164 | +fd42:4242:2601:3f::1 | ++ |
| dn42-us-lax1.burble.dn42 | +172.20.129.165 | +fd42:4242:2601:3a::1 | ++ |
| dn42-us-chi1.burble.dn42 | +172.20.129.166 | +fd42:4242:2601:2e::1 | ++ Decom. May 2021 ++ |
+
| dn42-ca-bhs2.burble.dn42 | +172.20.129.167 | +fd42:4242:2601:2d::1 | ++ |
| dn42-us-nyc1.burble.dn42 | +172.20.129.168 | +fd42:4242:2601:34::1 | ++ Decom. April 2021 ++ |
+
| dn42-de-fra1.burble.dn42 | +172.20.129.169 | +fd42:4242:2601:31::1 | ++ |
| dn42-es-mad1 | +172.20.129.170 | +fd42:4242:2601:2c::1 | ++ |
| dn42-us-phx1.burble.dn42 | +172.20.129.171 | +fd42:4242:2601:2b::1 | +Private Node | +
| dn42-us-dal3.burble.dn42 | +172.20.129.172 | +fd42:4242:2601:2a::1 | ++ |
| unassigned | +172.20.129.173 | +fd42:4242:2601:3b::1 | ++ |
| dn42-ch-zur1.burble.dn42 | +172.20.129.174 | +fd42:4242:2601:28::1 | ++ |
| dn42-uk-lon4.burble.dn42 | +172.20.129.175 | +fd42:4242:2601:29::1 | +Private Node | +
| ca-bhs1.burble.dn42 | +172.20.129.176 | +fd42:4242:2601:26::1 | +Temporary replacement of ca-bhs2 | +
| unassigned | +172.20.129.177 | +fd42:4242:2601:25::1 | ++ |
| dn42-uk-lon2.burble.dn42 | +172.20.129.178 | +fd42:4242:2601:24::1 | +Private Node | +
| dn42-hk-hkg1 | +172.20.129.179 | +fd42:4242:2601:23::1 | +Available Q1 2021 | +
| dn42-au-syd1.burble.dn42 | +172.20.129.180 | +fd42:4242:2601:38::1 | ++ Decom. Jan 2021 ++ |
+
| dn42-sg-sin2.burble.dn42 | +172.20.129.181 | +fd42:4242:2601:37::1 | ++ |
| unassigned | +172.20.129.182 | +fd42:4242:2601:3e::1 | ++ |
| unassigned | +172.20.129.183 | +fd42:4242:2601:3c::1 | ++ |
| unassigned | +172.20.129.184 | +fd42:4242:2601:22::1 | ++ |
| dn42-no-trd1.burble.dn42 | +172.20.129.185 | +fd42:4242:2601:39::1 | ++ |
| dn42-fr-rbx2.burble.dn42 | +172.20.129.186 | +fd42:4242:2601:32::1 | +Use fr-rbx1 | +
| dn42-uk-lon1.burble.dn42 | +172.20.129.187 | +fd42:4242:2601:35::1 | ++ |
| dn42-fr-rbx1.burble.dn42 | +172.20.129.188 | +fd42:4242:2601:36::1 | ++ |
| unassigned | +172.20.129.189 | +fd42:4242:2601:3d::1 | ++ |
| dn42-uk-bri1.burble.dn42 | +172.20.129.190 | +fd42:4242:2601:20::1 | +Private Node | +
| + | 172.20.129.191 | ++ | Reserved | +
| dn42-uk-lon3.burble.dn42 | ++ | fd42:4242:2601:27::1 | +Private Node | +
| dn42-uk-lon4.burble.dn42 | ++ | fd42:4242:2601:29::1 | +Private Node | +
| dn42-uk-lon5.burble.dn42 | ++ | fd42:4242:2601:30::1 | +Private Node | +
| dn42-nl-ams1.burble.dn42 | ++ | fd42:4242:2601:33::1 | +Private Node | +
| DNS | +IPv4 | +IPv6 | +
|---|---|---|
| dn42-fr-rbx1.burble.com | +176.31.240.39 | +2001:41d0:8:127::1 | +
| dn42-uk-lon1.burble.com | +185.42.222.153 | +2a04:92c5:2::1 | +
| dn42-de-fra1.burble.com | +193.41.237.149 | +2a0d:5941:1:17c::4e2a | +
| dn42-ch-zur1.burble.com | +45.91.92.111 | +2a0e:dc0:6:8::1 | +
| dn42-no-trd1.burble.com | +217.168.87.226 | +2001:678:dd0:ffff::25 | +
| dn42-es-mad1.burble.com | +45.132.74.100 | +2a0e:dc0:9:5::ab2d | +
| dn42-ca-bhs2.burble.com | +192.99.6.65 | +2607:5300:60:3741::1 | +
| dn42-us-nyc1.burble.com | +185.213.26.143 | +2a0d:5600:33:b::1 | +
| dn42-us-chi1.burble.com | +193.29.63.150 | +2605:4840:3:10::ab2d | +
| dn42-us-dal3.burble.com | +144.172.126.201 | +2602:fe64:8::4 | +
| dn42-us-lax1.burble.com | +185.215.224.214 | +2a0b:ae40:1:4a0a::5a | +
| dn42-sg-sin2.burble.com | +139.99.89.157 | +2402:1f00:8000:800::3bc | +
| dn42-hk-hkg1.burble.com | +tbc | +tbc | +
| dn42-au-syd1.burble.com | +139.99.237.85 | +2402:1f00:8100:400::279 | +
+
+
+ This page describes the use of BGP communities within the network.
+ +DN42 Communities are applied both internally and externally, and are used to influence the Routing Policy.
+| Community | +Description | +
|---|---|
| ( 64511 : 0 < x < 21 ) | +Max latency | +
| ( 64511 : 20 < x < 30 ) | +Min bandwidth | +
| ( 64511 : 30 < x < 35 ) | +Min encryption | +
| ( 64511 : 40 < x < 54 ) | +Route Origin | +
The following well known communities are implemented.
+| Community | +Description | +Action | +
|---|---|---|
| ( 65535 : 65281 ) | +No Export | +Prefix should not be exported outside of AS4242422601 | +
| ( 65535 : 65282 ) | +No Advertise | +Prefix should not be exported to any peers | +
| ( 65535 : 65283 ) | +Local-AS | +Prefix should not be exported outside of region | +
burble.dn42 implements large BGP communities, with ISO 3166-1 / UNSD country, and UNSD region codes.
+ +| Community | +Description | +
|---|---|
| ( 4242422601 : 120 : host code ) | +Route learned on this host | +
| ( 4242422601 : 130 : 1 ) | +Route is a direct peer | +
| ( 4242422601 : 140 : DN42 region ) | +Route learned in this DN42 region | +
In a large network like burble.dn42 it can take some time for route updates +to be distributed and for the network to settle following changes. As changes +ripple through the network it can create a cascade of ghost updates, each with +an increasing internal path length, or adjusted latency metric. Worse the +cascade of updates can also be re-distributed to peers, creating a significant +multiplier for the number of updates from a single change.
+To protect against ghost routes, a community is used to track how many internal +hops a route has. In a fully meshed network like burble.dn42, a genuine route should +never have more than two internal hops, so an export filter is used to prevent +distribution of longer paths.
+| Community | +Description | +
|---|---|
| ( 4242422601 : 100 : 1 ) | +Added on first internal re-distribution | +
| ( 4242422601 : 100 : 2 ) | +Added on second re-distribution and prevents further re-distribution within burble.dn42 | +
The ( 4242422601 : 100 : x ) communities are not exported to peers.
+ + +
+
+
+ Information about the burble.dn42 network.
+ + + +
+
+
+ | + | + |
|---|---|
| Location | +OVH (SoYouStart), Roubaix, France | +
| Specs | +4 core/8 thread, 32GB, 2 x 960GB SSD, 500mbit unmetered | +
| Public Hostname | +dn42-fr-rbx1.burble.com | +
| Public IPv4 Address | +176.31.240.39 | +
| Public IPv6 Address | +2001:41d0:8:127::1 | +
| Tunnel IPv4 Peer Address | +172.20.129.188/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:36:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:36::1/128 | +
| + | + |
|---|---|
| Location | +Clouvider, London, UK | +
| Specs | +4 core/8 thread, 32GB, 2 x 240GB SSD, 10TB bw | +
| Public Hostname | +dn42-uk-lon1.burble.com | +
| Public IPv4 Address | +185.42.222.153 | +
| Public IPv6 Address | +2a04:92c5:2::1 | +
| Tunnel IPv4 Peer Address | +172.20.129.187/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:35:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:35::1/128 | +
| + | + |
|---|---|
| Location | +PHP Friends, Frankfurt, Germany | +
| Specs | +4 dedicated EPYC cores, 20GB, 160GB NVME, 5TB bw | +
| Public Hostname | +dn42-de-fra1.burble.com | +
| Public IPv4 Address | +193.41.237.149 | +
| Public IPv6 Address | +2a0d:5941:1:17c::4e2a | +
| Tunnel IPv4 Peer Address | +172.20.129.169/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:31:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:31::1/128 | +
| + | + |
|---|---|
| Location | +HostHatch, Zurich, Switzerland | +
| Specs | +2 shared cores, 8GB, 40GB NVME, 5TB bw | +
| Public Hostname | +dn42-ch-zur1.burble.com | +
| Public IPv4 Address | +45.91.92.111 | +
| Public IPv6 Address | +2a0e:dc0:6:8::1 | +
| Tunnel IPv4 Peer Address | +172.20.129.174/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:28:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:28::1/128 | +
| + | + |
|---|---|
| Location | +Trondheim, Norway | +
| Specs | +2 shared cores, 2GB, 16GB SSD, 1gbit unmetered | +
| Public Hostname | +dn42-no-trd1.burble.com | +
| Public IPv4 Address | +217.168.87.226 | +
| Public IPv6 Address | +2001:678:dd0:ffff::25 | +
| Tunnel IPv4 Peer Address | +172.20.129.185/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:39:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:39::1/128 | +
+ Available Q1 2021 ++ + +
| + | + |
|---|---|
| Location | +HostHatch, Madrid, Spain | +
| Specs | +1 shared core, 2GB, 20GB SSD, 2TB bw | +
| Public Hostname | +dn42-es-mad1.burble.com | +
| Public IPv4 Address | +45.132.74.100 | +
| Public IPv6 Address | +2a0e:dc0:9:5::ab2d | +
| Tunnel IPv4 Peer Address | +172.20.129.170/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:2c:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:2c::1/128 | +
| + | + |
|---|---|
| Location | +OVH (SoYouStart), Beauharnois, Canada | +
| Specs | +4 core/8 thread, 32GB, 2 x 960GB SSD, 500mbit unmetered | +
| Public Hostname | +dn42-ca-bhs2.burble.com | +
| Public IPv4 Address | +192.99.6.65 | +
| Public IPv6 Address | +2607:5300:60:3741::1 | +
| Tunnel IPv4 Peer Address | +172.20.129.167/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:2d:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:2d::1/128 | +
| + | + |
|---|---|
| Location | +drserver, Dallas, United States | +
| Specs | +8 core, 8GB, 240GB SSD, 100mbit unmetered | +
| Public Hostname | +dn42-us-dal3.burble.com | +
| Public IPv4 Address | +144.172.126.201 | +
| Public IPv6 Address | +2602:fe64:8::4 | +
| Tunnel IPv4 Peer Address | +172.20.129.172/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:2a:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:2a::1/128 | +
| + | + |
|---|---|
| Location | +LetBox, Los Angeles, United States | +
| Specs | +2 shared ryzen cores, 4GB, 15GB NVMe/250GB HDD, 5TB bw | +
| Public Hostname | +dn42-us-lax1.burble.com | +
| Public IPv4 Address | +185.215.224.214 | +
| Public IPv6 Address | +2a0b:ae40:1:4a0a::5a | +
| Tunnel IPv4 Peer Address | +172.20.129.165/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:3a:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:3a::1/128 | +
| + | + |
|---|---|
| Location | +OVH, Singapore | +
| Specs | +1 shared core, 2GB, 20GB SSD, 2TB bw | +
| Public Hostname | +dn42-sg-sin2.burble.com | +
| Public IPv4 Address | +139.99.89.157 | +
| Public IPv6 Address | +2402:1f00:8000:800::3bc | +
| Tunnel IPv4 Peer Address | +172.20.129.181/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:37:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:37::1/128 | +
+ Available Q1 2021 ++ + +
| + | + |
|---|---|
| Location | +HostHatch, Hong Kong, CN | +
| Specs | +2 shared core, 8GB, 40GB SSD, 1TB bw | +
| Public Hostname | +dn42-hk-hkg1.burble.com | +
| Public IPv4 Address | +tbc | +
| Public IPv6 Address | +tbc | +
| Tunnel IPv4 Peer Address | +172.20.129.179/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:23:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:23::1/128 | +
+ The following nodes are still active but are being decommissioned +and are no longer open for new peerings. ++ + +
| + | + |
|---|---|
| Location | +OVH, Sydney, Australia | +
| Public Hostname | +dn42-au-syd1.burble.com | +
+ Decom. Date ++ |
+January 2021 | +
| Public IPv4 Address | +139.99.237.85 | +
| Public IPv6 Address | +2402:1f00:8100:400::279 | +
| Tunnel IPv4 Peer Address | +172.20.129.180/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:38:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:38::1/128 | +
| + | + |
|---|---|
| Location | +HostHatch, New York, United States | +
| Public Hostname | +dn42-us-nyc1.burble.com | +
+ Decom. Date ++ |
+April 2021 | +
| Public IPv4 Address | +185.213.26.143 | +
| Public IPv6 Address | +2a0d:5600:33:b::1 | +
| Tunnel IPv4 Peer Address | +172.20.129.168/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:34:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:34::1/128 | +
| + | + |
|---|---|
| Location | +HostHatch, Chicago, United States | +
| Public Hostname | +dn42-us-chi1.burble.com | +
+ Decom. Date ++ |
+May 2021 | +
| Public IPv4 Address | +193.29.63.150 | +
| Public IPv6 Address | +2605:4840:3:10::ab2d | +
| Tunnel IPv4 Peer Address | +172.20.129.166/32 | +
| Tunnel IPv6 Link Local | +fe80::42:2601:2e:1/64 | +
| Tunnel IPv6 ULA | +fd42:4242:2601:2e::1/128 | +
+
+
+ burble.dn42 is an experimental global network within DN42.
+The network is well connected with a large number of +peers, and hosts some of the DN42 core infrastructure.
+
All nodes in the burble.dn42 network are fully meshed with wireguard tunnels.
+iBGP with BGP Confederations and
+a latency based metric are used as the interior routing protocol between nodes.
+iBGP is also fully meshed. and the configuration for both iBGP and wireguard tunnels
+is built using a number of Ansible scripts.
The current network design was introduced in December 2019; previous designs for +the network have included a VXLAN overlay over the wireguard mesh to create a +single layer 2 network, together with the use of OSPF as the +IGP. Other variations have included using BABEL, and tinc.
+ +A selection of key technologies used within the network
+ + + +
+
+
+ This page provides the information to get started on peering with the burble.dn42 network
+burble.dn42 is a set of global POPs integrated to the dn42 network, +and new peering requests are welcome. A description of the network is available in the +Overview page.
+burble.dn42 is a large network and there are some restrictions in place to protect the network and the rest of the DN42.
+Please ensure you read the information below before requesting to peer.
Please mail dn42@burble.com if you’d like to peer with me.
+ +To peer with burble.dn42, you must meet the following requirements:
+You must have at least two peerings already established with other DN42 networks
+Sorry, but burble.dn42 is not open to new starters. If you are a new starter in DN42 +please use the peerfinder or ask on +IRC; there are lots of other networks who will +be happy to peer with you, and some even offer automatic peering.
+This is a tough restriction, but one that is in place to promote network diversity.
+You must support IPv6
+You must implement ROA checks
+Contact information in the registry must always be up to date and admins must +respond when contacted
+Contacts must also be reachable in case of problems. In addition, the network +is ever evolving and failure to respond to change notices may result in your +peering being suspended.
+At a minimum, I’ll need to know the following in order to establish a peering:
++
All peerings will be configured as a full transit session.
+++ +Residential ISPs and Dynamic IP Addresses
+A 24/7 connection, with static IP addresses are the norm for DN42. If you are +connecting from a residential ISP or otherwise have a dynamic IP please let me +know so that I can configure my side appropriately. If you don’t do tell me, the +peering may stop working when your IP address changes.
+ +
++ + +Peering in Multiple Locations
+If you have multiple nodes, you are welcome to peer in several locations +to provide additional redundancy and route choice.
+It’s highly recommended to peer with multiple users DN42 users though, it’s +lots of fun and you should never rely on just one user for your connectivity.
+ +
I prefer to use wireguard, it’s simple to set up and just works. +I also support OpenVPN tunnels.
+ +My wireguard AllowedIPs are:
+AllowedIPs=fe80::/64
+AllowedIPs=fd00::/8
+AllowedIPs=0.0.0.0/0
+++ + +Use of wg-quick
+Using wg-quick is not recommended as it does not support adding a peer address. +If you want to use wg-quick you will need to delete and re-add the wireguard +interface IP address and configure it as a point to point address or you will +run in to next-hop problems when using BGP. You must read the +DN42 Wiki on how to set up wg-quick for use +within DN42.
+ +
By default I will configure the following OpenVPN parameters:
+comp-lzo
+cipher aes-256-cbc
+auth sha256
+Only the network ranges will be forwarded through the DN42 network, all other +traffic will be dropped.
+IPv4
+172.16.0.0/12
+10.0.0.0/8
+IPv6
+fd00::/8
++ BGP peer addresses are more permissive to allow for link local or non-DN42 +IP addresses within the tunnel, but these will not be forwarded through the +DN42 network. ++ + +
A typical BGP session in DN42 will use a trivial amount of traffic. However, for large networks like +burble.dn42 some transient events, such as BGP flapping, can generate multi MB/sec traffic flows that +damange the network and create instability across DN42.
+To protect the network from misconfigurations and prevent excessive updates from being propagated to the +rest of DN42, the burble.dn42 network implements rate limiting on direct BGP sessions. The rate limiting +activates when a large amount of BGP traffic is seen (typically 10’s or 100’s of thousands of +updates a second) over a sustained period and will typically reset automatically within an hour.
+There are no other controls applied to transit or non-BGP traffic.
+ +| + | + |
|---|---|
| Network Name | +BURBLE | +
| BURBLE-MNT | +dn42@burble.com | +
| ASN | +AS4242422601 | +
The burble.dn42 network uses a custom build of +bird 2, and the +following features are supported:
+The source code for the custom bird used on the network is available on +git.burble.dn42
+ +The network applies strict Route Origin Authorisation (ROA) filtering to all +received and exported routes. This means any advertised route that does +not have a corresponding route{,6} object in the DN42 registry will be dropped.
+++ + +ROA is implemented with updates through RPKI, using +dn42regsrv and +gortr.
+The DN42 ROA data is provided as a public service, see the Services page.
+ +
IPv4
+172.20.0.0/14+
+10.0.0.0/8+
+IPv6
+fd00::/8{44,64}
+Within the tunnel, hosts respond to ping and traceroute, but also have the +echo (port 7) and daytime (port 13) services enabled. These can be used to check +the tunnel is up and configured correctly.
+$ ping fe80::42:2601:32:1%wg0
+PING fe80::42:2601:32:1%wg0(fe80::42:2601:32:1%wg0) 56 data bytes
+64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=1 ttl=64 time=4.44 ms
+64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=2 ttl=64 time=4.52 ms
+64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=3 ttl=64 time=4.96 ms
+^C
+--- fe80::42:2601:32:1%wg0 ping statistics ---
+3 packets transmitted, 3 received, 0% packet loss, time 2003ms
+rtt min/avg/max/mdev = 4.445/4.643/4.961/0.233 ms
+$ netcat fe80::42:2601:32:1%wg0 13
+Sun Sep 23 09:57:26 2018
+^C
+$
+Once peering is established I have a BGP looking glass (public +internet link) and global route collector which can be used to +check routing configuration. Looking glasses are a key, self-service resource for you +to use when understanding how your routes are propogating around the DN42 network, please +take the time to learn how to use them.
+ +burble.dn42 operates two speed test servers on central, high bandwidth nodes.
+See the services pages for more info.
pingable.burble.dn42 (172.20.129.5 / fd42:4242:2601:ac05::1) is a dedicated address +that responds to ping and traceroute and may be used for automated reachability or +link quality testing.
+++ + + +Please be considerate when configuring automated tests and set a reasonable test +frequency.
+In all cases, the frequency must not be more than once a second.
+ +
+Please consider this if your router automatically pings its tunnel endpoint for +stats purposes.
+
+
+ With a global network and multiple peers, the burble.dn42 network typically has many alternative route paths for reaching a particular destination. The routing policy aims to keep route selection sane, and avoid sending traffic outside of a region where possible.
+ +The local_pref for routes is set on entry, and then propogated across the whole network. This forces the +network to prefer routes that, where possible, send traffic through the burble.dn42 network to a local peer, +rather than sending cross regional traffic through external peers (aka +Cold Potato Routing).
+| Local Pref | +Route Class | +
|---|---|
| 3000 | +burble.dn42 dynamic / anycast routes | +
| 2000 | +burble.dn42 internal networks | +
| 1000 | +Peer networks (AS path len = 1) | +
| 500 | +Route received in same DN42 region as it originated | +
| 100 | +Default | +
The med attribute is used to implement a latency based metric across the network. Scripts are used +to gather the latency between nodes (using ping) and this is then incorporated in to the ansible +scripting that generates the peer configuration for the internal mesh. The peer configuration +sets the med to be the latency in ms between nodes (in milliseconds * 10). A penalty of 500 is added +for each hop to encourange direct routing between nodes.
+med = (latency between nodes in ms * 10) + (500 per hop)
+The med metric is exported to external peers to help them decide how to route traffic to the burble.dn42 +network.
+ + +
+
+
+ All nodes are monitored using UptimeRobot.
+ +Netdata and Prometheus are +used to monitor the network and stats are presented using a series of +Grafana dashboards.
+ +BGP status can be found through my looking glass.
+ + +
+
+
+ In common with most websites, the burble.dn42 site and associated services +may log any access you make and these logs contain your source IP address +together with the page or service being accessed. If you are required to +log in to access a burble.dn42 service, you should assume that the user id +used for the service is also logged. Website and service logs are accessible +only by the network administrators and used purely for diagnostic reasons +and to prevent abuse. They are not shared in any way. Log retention varies +depending on the service, but is at most, 1 month.
+burble.dn42 services are provided by servers operating globally. Data +processing may take place in any country where the network has a pop or +presence.
+The services provided by burble.dn42 make use of data contained within the +DN42 Registry. This data may contain personal data +that has been provided voluntarily by users of DN42 and which is then made +public by this website or associated services. Please refer to the DN42 +registry privacy policy for more information.
+If you have any data privacy concerns or requests regarding burble.dn42 +services you may contact dn42@burble.com.
+ + +
+
+
+ burble.dn42 maintains a PKI infarstructure for its services, using +Hashicorp Vault
+ +| + | + |
|---|---|
| countryName | +GB | +
| stateOrProvinceName | +dn42 | +
| organizationName | +burble.dn2 | +
| commonName | +ca.burble.dn42 | +
| emailAddress | +dn42@burble.com | +
-----BEGIN CERTIFICATE-----
+MIIDrDCCApSgAwIBAgIJAIZWD8xmHTYFMA0GCSqGSIb3DQEBCwUAMGsxCzAJBgNV
+BAYTAkdCMQ0wCwYDVQQIDARkbjQyMRQwEgYDVQQKDAtidXJibGUuZG40MjEXMBUG
+A1UEAwwOY2EuYnVyYmxlLmRuNDIxHjAcBgkqhkiG9w0BCQEWD2RuNDJAYnVyYmxl
+LmNvbTAeFw0xODEyMjIwOTIxMDhaFw0yMDEyMjEwOTIxMDhaMGsxCzAJBgNVBAYT
+AkdCMQ0wCwYDVQQIDARkbjQyMRQwEgYDVQQKDAtidXJibGUuZG40MjEXMBUGA1UE
+AwwOY2EuYnVyYmxlLmRuNDIxHjAcBgkqhkiG9w0BCQEWD2RuNDJAYnVyYmxlLmNv
+bTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEsUm0KtuwZjrMeWl/x
+t8f5XCLdDdBAm9KWfJWl9fwxTFkwYEMaXMLjhsmoLKuyXejs7X72VAA/Ctz9KuiQ
+l/teuGKvt8gNbq3IXVH9KxW8uiSWJIUklZ801qLjUX4kzWJlCgug7Xd2Q0LsevvC
+QMSNa2Blfh6ieMtjeQNaRhoyy3xEn7t/CNkn5U+bVFTUYE31fREWyEJe2avX2KIs
+y55GxlkUmOZZPAsMs9at+NmfDWsxOYJSYBOeLsyzJnHWX0g+X9sBf14CDPL3KVxq
+NdGlPGYmJXr8Q5bNUv6diHSGd4nW/ft/IRGPpgXpwzcQNQHfneZUGSC9L+0B4LR4
+sqkCAwEAAaNTMFEwHQYDVR0OBBYEFN8+6wkXTKajsoezC1nzvBe+YZscMB8GA1Ud
+IwQYMBaAFN8+6wkXTKajsoezC1nzvBe+YZscMA8GA1UdEwEB/wQFMAMBAf8wDQYJ
+KoZIhvcNAQELBQADggEBAEdmCZyKaEk2AOcgVkQ9OMAR+AaGIUCazvWKGx2DXAJI
+2zmeEKx1tkRAkEQkzoUo7vor+X8/f3FVmtVF5bxbI9y7LsrXSNZB62z+Voyj36Id
+rOx05MN9FPbz6FAD5a7vTWCK7hRgGTaUGwYVyVXy9iiSA8Oqm8sqblvfk9jz3E5Y
+TJP6b8Y/Dq9BLpciozqSLo2zPOvl7kaN3kS3eufzA4O9LKfFIxXIcqfsSzFlbQyu
+afBrjiG18FVCNPQW3kCBk2oOWl7z/SJjB1oG/ZcDhSdHYPj+1gTsnzTKAB8qJikh
+gGwJMRLy1L5Bd0p63in5SNX9LXVsY+8YiA7sa3yAhWc=
+-----END CERTIFICATE-----
+
+ Certificate Expiry Date: May 12 10:08:49 2029 GMT
+
+
+
+
+
+
+
+ burble.dn42 hosts a number of DN42 infrastructure services.
+ +| Service | +Name | +IP | +
|---|---|---|
| DN42 Master | +b.master.delegation-servers.dn42 | +fd42:180:3de0:30::1 | +
| Authoritative Service | +b.delegation-servers.dn42 | +172.20.129.1 fd42:4242:2601:ac53::1 |
+
| Recursive Service | +b.recursive-servers.dn42 | +172.20.129.2 fd42:4242:2601:ac53::53 |
+
burble.dn42 provides a local, anycast, authoritative and recursive DNS service.
+The DNS Service has it’s own page.
| Mirror URLs | ++ |
|---|---|
| wiki.dn42 wiki.burble.dn42 |
+(editable via DN42) | +
| dn42.dev wiki.burble.com |
+(read-only via public internet) | +
burble.dn42 maintains a globally distributed mirror of the DN42 Wiki, and is part of +the wiki.dn42 anycast group. +The DN42 services (wiki.dn42 and +wiki.burble.dn42) are editable, whilst the public internet +views (dn42.dev and wiki.burble.com) are +read-only.
+Please note that updates to the wiki may take several hours to sync with other mirrors.
+The service is provided by regional mirrors fronted by an nginx proxy that is itself +anycasted across burble.dn42. The service is fully meshed and will continue to +operate as long as at least one proxy and mirror is available.
+Mirrors are located in the following locations:
+WHOIS service providing data from the +DN42 registry. The WHOIS service is also anycasted across the network.
+The source code for the service is available in the +burble.dn42 git.
+ +The global route collector provides a central
+bird instance that collects routes from peers across the DN42 network.
+All users are invited to join the collector and help provide stats for the network.
The route collector can currently be queried by using ssh to connect a bird shell
+or via a looking glass.
+Additional services and stats are expected to be developed in the future.
burble.dn42 hosts monitoring and alerting of key DN42 services, see the +grafana service for more details.
+ + +
+
+
+ burble.dn42 provides a suite of DNS services, including running one of the two +DN42 DNS master nodes that exports registry information to the DNS infrastructure.
+| Role | +Names | +
|---|---|
| DN42 DNS Master | +b.master.delegation-servers.dn42 | +
| Authoritative DNS Service | +b.delegation-servers.dn42 ns1.burble.dn42 |
+
| Recursive DNS Service | +b.recursive-servers.dn42 dns.burble.dn42 |
+
| dns64 Service | +dns64.burble.dn42 | +
Apart from the Master, all DNS services are anycast across every node to provide fast, +local responses network wide. The services support DNSSEC and are available over UDP, TCP, +DNS over HTTPs and DNS over TLS.
+ +| Name | +IP | +
|---|---|
| b.master.delegation-servers.dn42 | +fd42:180:3de0:30::1 | +
burble.dn42 runs one of the two master servers that support the DN42 DNS infrastructure.
+See the wiki for
+more information on the role of the master service.
The master is hosted on us-dal3, providing geographic and network redundancy against the other DN42 master service, hosted in Europe.
+ +| Name | +IP | +
|---|---|
| ns1.burble.dn42 b.delegation-servers.dn42 |
+172.20.129.1 fd42:4242:2601:ac53::1 |
+
ns1.burble.dn42 is slaved to master.delegation-servers.dn42, and provides +DNSSEC signed, authoritative data for DN42 related zones.
+The authoritative service may be used as the root for a local DNS resolver, with the assurance +that returned DNS records are traceable via DNSSEC to the DN42 registry. The service +also supports AXFR and may be used as a master to a local, slaved, root zone.
+Note that ns1.burble.dn42 will not forward DNS queries.
+Forwarding is provided by the recursive service, dns.burble.dn42.
| Zone | +Role | +
|---|---|
| burble.dn42 | +burble.dn42 forward zone | +
| collector.dn42 | +Global Route Collector forward zone | +
| 1.0.6.2.2.4.2.4.2.4.d.f.ip6.arpa | +burble.dn42 IPv6 reverse zone | +
| 0/27.129.20.172.in-addr.arpa | +burble.dn42 services IPv4 reverse zone | +
| 160/27.129.20.172.in-addr.arpa | +burble.dn42 nodes IPv4 reverse zone | +
| 0.3.0.0.0.e.d.3.0.8.1.0.2.4.d.f.ip6.arpa | +DNS Master reverse zone | +
| 0.0.1.0.0.e.d.3.0.8.1.0.2.4.d.f.ip6.arpa | +Registry services IPv6 reverse zone | +
| 0/28.63.22.172.in-addr.arpa | +Register services, IPv4 reverse zone | +
| Name | +IP | +
|---|---|
| dns.burble.dn42 b.recursive-servers.dn42 |
+172.20.129.2 fd42:4242:2601:ac53::53 |
+
dns.burble.dn42 is a caching, recursive DNS service that returns results for both DN42 +and clearnet domains. The service issues parallel queries from five regional masters, the +recursive service takes advantage of the burble.dn42 global scale to reduce latency and +avoid local connectivity problems.
+The recursor is DNSSEC enabled and validates all queries.
+ +Users are encouraged to consult recursive-servers.dn42 to obtain a list of +recursive DNS services and configure at least two independent resolvers +to obtain the best resilience.
+See also the DN42 Wiki for general guidelines and +best practice for setting up DNS in DN42.
+$ host -t SRV _dns._udp.recursive-servers.dn42
+_dns._udp.recursive-servers.dn42 has SRV record 10 10 53 a3.recursive-servers.dn42.
+_dns._udp.recursive-servers.dn42 has SRV record 20 10 53 b.recursive-servers.dn42.
+_dns._udp.recursive-servers.dn42 has SRV record 10 10 53 a0.recursive-servers.dn42.
+_dns._udp.recursive-servers.dn42 has SRV record 20 10 53 j.recursive-servers.dn42.
+_dns._udp.recursive-servers.dn42 has SRV record 20 10 53 k.recursive-servers.dn42.
+Example resolv.conf using IPv6 with IPv4 fallback
+# DN42 resolve.conf
+
+search dn42
+
+# burble.dn42 service
+# b.recursive-servers.dn42
+nameserver fd42:4242:2601:ac53::53
+
+# j.recursive-servers.dn42
+nameserver 172.20.1.19
+| Name | +IP | +
|---|---|
| dns64.burble.dn42 | +fd42:4242:2601:ac53::64 | +
The dns64 service operates in a similar way to the main recursive service but also provides +dns64 translation for hostnames that only have IPv4 addresses.
+The service will return IPv4 mapped to the rfc6052
+well-known prefix - 64:ff9b::/96
The burble.dn42 services support queries via DNS over HTTPS (on port 443) and +DNS over TLS (on port 843). The HTTPS service is signed by the burble.dn42 +Certificate Authority, and the CA certificate +will be required by the client in order to use the service.
+example
+$ doh burble.dn42 https://[fd42:4242:2601:ac53::53]/dns-query
+burble.dn42 from https://[fd42:4242:2601:ac53::53]/dns-query
+TTL: 3600 seconds
+A: 172.20.129.3
+AAAA: fd42:4242:2601:ac80:0000:0000:0000:0001
+The DNS service is implemented as a tiered, anycast service with each node +in the network providing a local cache in front of regional, slave nodes.
+ +Edge nodes provide a caching function for the slaves.
+Recursive services (dns.burble.dn42 and dns64.burble.dn42) are provided by +dnsmasq +configured using the ‘all-servers’ mode. DN42 queries are forwarded to all +regional slaves in parallel and the first response received is then returned. +This approach ensures users get the lowest latency results possible, regardless of +location, and that any local connectivity issues do not impact the results.
+The authoritive service as well as DNS over HTTPS and DNS over TLS services are +provided by dnsdist acting as a proxy. Requests are +forwarded to either the regional slaves or local recursor services as appropriate +and also cached.
+Clearnet queries are forwarded on the edge nodes to a combination of +Google and Cloudflare services.
+The edge services are monitored and anycast routes automatically injected (or +removed) using GoBGP and a health checking script.
+ +| Region | +Host | +Location | +
|---|---|---|
| Europe | +dns-slave.de-fra1.burble.dn42 | +PHP Friends, Frankfurt, Germany | +
| Americas (East) | +dns-slave.ca-bhs2.burble.dn42 | +OVH, Beauharnois, Canada | +
| Americas (Mid) | +dns-slave.us-dal3.burble.dn42 | +DrServer, Dallas, USA | +
| Americas (West) | +dns-slave.us-lax1.burble.dn42 | +LetBox, Los Angeles, USA | +
| Asia and Oceania | +dns-slave.sg-sin2.burble.dn42 | +OVH, Singapore | +
The slave nodes are implemented using PowerDNS.
+The Authoritative DNS servers are configured as slaves replicating from the +DN42 master for .dn42 related zones and a hidden master located on the private, +internal network for burble.dn42 zones.
+The recursive service is provided by the pdns-recursor configured with DNSSEC +validation and additional caching.
+ +The DN42 DNS master is a custom java program +running on us-dal3.
+ + +
+
+
+ Information about burble.dn42 services.
+ + + +
+
+
+ This page provides some documenation on other services used within burble.dn42 +that are not directly available for public use.
+ +Core nodes run an nginx container that acts as a reverse proxy +for services hosted in tier2.
+The reverse proxy is distributed to improve local response times and is +anycast as rproxy.burble.dn42. Most web services provided by burble.dn42 are +simply CNAMEs to the reverse proxy which then balances and forwards the +request to the actual service.
+As well as a reverse proxy, nginx also provides:
+n8n is used to provide an automation and workflow service.
+As an example, n8n is used to update dn42regsrv +and [ROA tables](/services/public#ROA Tables) when the +registry changes.
+
Hashicorp Vault is used to handle secrets
+across the burble.dn42 network.
+Vault is deployed as a 3 node cluster across the Europe core nodes
+and uses Consul as the cluster back end.
Vault acts as the main certificate authority for burble.dn42 +PKI, however there is also an intermediate ACME server based on +smallstep CA.
+Vault allows for regular, automated renewal of certificates on short timeframes +(typically a rolling week or monthly basis).
+ +Vault also acts as an SSH certificate authority, verifying both users and servers +within the network.
+Server certificates are generated during deployment, whilst user (or role) +certificates are short lived and generated on demand.
+ +Vault holds secrets used during node and service deployments.
+Most burble.dn42 are built as stateless container images and secrets are +pushed from vault in to the live containers at runtime. This ensures the +container images do not contain secrets and that secrets can be applied per +instance even when using a common image.
+Vault also manages database credentials (using the mysql/mariadb integration), +and these are also automatically generated and pushed in to container +instances on deployment.
+The authority to access deployment secrets is inherited, on demand, from the +user token during the deployment process. This ensures that even if access was +gained to the deployment server, secrets could still not be accessed without +also having access to a live user token.
+ +The burble.dn42 git has an associated CI/CD service +based on drone.
+The CI/CD service is used to manage DNS, build and publish applications and +the burble.dn42 website.
+ +min.io is used as an S3 compatible block storage service. +For example, min.io is used for storing build artifacts from CI pipelines.
+As well as a central storage server, min.io is deployed in ‘gateway’ mode +to provide local, regional caches for the block storage.
+The min.io services uses a global etcd cluster for credential +management.
+ +For lurking on #dn42 I use +thelounge, a web based IRC client.
+ + +
+
+
+ Services provided for use within DN42
+ +This website is built using Hugo and is +distributed across burble.dn42 +core nodes.
+The public internet site is hosted on de-fra1 behind +CloudFlare and the source for the website +is published in the burble.dn42 git.
+ +A public issue log is maintained on the DN42 Registry.
+Users are welcome to raise issues or enhancements via the log.
+ + +The burble.dn42 looking glass is based on +bird-lg with patches by +Zhaofeng, +tds and +myself to fix formating, bird2 +compatibility and other tweaks.
+A fork of sileht/bird-lg that includes all of our fixes is available on +GitHub.
+DN42 registry data in the BGP Map part of the looking glass uses +lgregmapper +to interface with dn42regsrv.
+The looking glass is hosted on de-fra1 and the public version is +behind CloudFlare.
+ +pingable.burble.dn42 is a single IP address that will respond to ping and +traceroute requests across the entire network.
+This address may be used for automated reachability or latency tests, however +please be considerate and configure a reasonable test frequency.
++ In all cases, do not set the ping frequency to be higher than once a second. ++ + +
A speed test service is available in France and Canada.
+Note that the service is currently available over IPv6 only at this time.
+ If the service ends up loading or disrupting the rest of the network then I may end up +removing it, so remember this service is provided for your benefit and use responsibly. ++ + + +
The hosted grafana service has it’s own page here.
+ + +Each node in the network is monitored by UptimeRobot with alerts +if a node becomes unavailable.
+ +Internally, nodes are measured by +netdata which provides a real time view of +each node. prometheus is then used to collect and +store that data for historical reporting. grafana is used +for visualisation.
+Syslogs are exported in real time to a central logging node on the internal network.
+ +| Service | +Name | +IP | +
|---|---|---|
| Authoritative Service | +ns1.burble.dn42 | +172.20.129.1 fd42:4242:2601:ac53::1 |
+
| Recursive Service | +dns.burble.dn42 | +172.20.129.2 fd42:4242:2601:ac53::53 |
+
| DNS64 Service | +dns64.burble.dn42 | +fd42:4242:2601:ac53::64 | +
burble.dn42 provides a local, anycast, authoritative and recursive DNS service.
+The DNS Service has it’s own page.
+ + ++ + +All services support DNS over HTTPs on port 443, and DNS over TLS on port 843. +
dn42regsrv is a REST API for the DN42 registry +that provides a bridge between interactive applications and the registry.
+As well as the main REST API to the DN42 registry, the server can also generate ROA tables +and provides a small web application for exploring registry data.
+ +Route Origin Authorisation (ROA) tables are generated using +dn42regsrv and published to the +dn42.burble.com website for general use.
+The JSON output file can be used with gortr
+to implement ROA checks via RPKI.
+The Bird files can be used directly with Bird to implement ROA checks as detailed
+in the DN42 Wiki (Bird1 / Bird2).
| URL | +IPv4/IPv6 | +Description | +
|---|---|---|
| https://dn42.burble.com/roa/dn42_roa_46.json | +Both | +DN42 ROA data in JSON format | +
| https://dn42.burble.com/roa/dn42_roa_bird1_46.conf | +Both | +DN42 ROA data for use with Bird1 | +
| https://dn42.burble.com/roa/dn42_roa_bird1_4.conf | +IPv4 Only | +DN42 ROA data for use with Bird1 | +
| https://dn42.burble.com/roa/dn42_roa_bird1_6.conf | +IPv6 Only | +DN42 ROA data for use with Bird1 | +
| https://dn42.burble.com/roa/dn42_roa_bird2_46.conf | +Both | +DN42 ROA data for use with Bird2 | +
| https://dn42.burble.com/roa/dn42_roa_bird2_4.conf | +IPv4 Only | +DN42 ROA data for use with Bird2 | +
| https://dn42.burble.com/roa/dn42_roa_bird2_6.conf | +IPv6 Only | +DN42 ROA data for use with Bird2 | +
ROA data is cached via Cloudflare to provide fast local access, and an +n8n script is used to update ROA data immediately following +registry changes.
+ +burble.dn42 related code and configuration is maintained in a local +gitea repository.
+ +burble.dn42 PrivateBin instance.
+ +All servers in burble.dn42 provide a stable, high stratum +NTP service using chrony.
+The NTP service is exposed over DN42, and users are welcome to use any server +in the burble.dn42 network as an NTP time server on either the public or DN42 networks.
+ + +