www/public/js/en.search-data.min.0ffb683be720d0aa9a952cdf7965f763bcca508eb33c869653150cce1dcacc46.js

"use strict";(function(){const t={};t.doc={id:"id",field:["title","content"],store:["title","href","parent"]};const e=FlexSearch.create(t);window.geekdocSearchIndex=e,e.add({id:0,href:"/additional/maintlog/2020/",title:"2020",parent:"Maint. Log Archive",content:`Archive of changes made in 2020
22nd December 2020 es-mad1 in Madrid, Spain has been deployed and is now open for peering.
13th December 2020 Issue Log
burble.dn42 now has a public issue log, hosted on the DN42 Registry.
Issue Log Feel free to raise issues or enhancements on the log.
Speedtest Service
An experimental speed test service has been introduced:
France: https://speedtest.fr-rbx1.burble.dn42 Canada: https://speedtest.ca-bhs2.burble.dn42 The two services are currently only accessible over IPv6 but are hosted on dedicated servers with plenty of available bandwidth. If the service ends up loading or disrupting the rest of the network then I may end up removing it, so use responsibly.
n8n Automation
The burble.dn42 network now has an instance of n8n to help automate internal workflows.
Whilst this isn&rsquo;t a public service the first visibile benefit is that the Explorer and ROA files now update immediately following registry changes. Previously changes were polled and could take up to an hour to be updated.
6th December 2020 ca-bhs2 and fr-rbx1 have been migrated to their new servers. If you are peering with these nodes please make sure you update any IP addresses on your side as required.
git.dn42.dev is hosted on ca-bhs2 and so was also migrated and upgraded to v1.13.0.
28th November 2020 Black Friday has been been and gone and this means that a few nodes have now reached the end of their contract and are being retired:
fr-sbg1 us-sea2 jp-tyo1 sg-sin1 us-mia2 However, the good news is that Black Friday also delivered a few shiny new nodes for the burble.dn42 network:
ca-bhs2 will be replaced with a new node that has SSD rather than HDD storage new nodes expected in Hong Kong and Madrid, eta Jan 2021 Users of ca-bhs2 will be migrated to the new node, with details to be confirmed.
12th September 2020 uk-lon1 has been upgraded. If you are peered on this node, please update your IP address accordingly.
burble.dn42 now includes some limited protection against ghost route updates. See the communities page for more details.
9th September 2020 uk-lon1 will be upgraded this weekend (12/13th September), but unfortunately this does mean that the IP address for the server is going to change.
The hostname will be changed to match the new address, but if you use the IP address in your configuration (e.g. for firewall rules), you will need to update them, as detailed below.
dn42-uk-lon1.burble.com
IPv4: 185.42.222.153 IPv6: 2a04:92c5:2::42 All other peering details, such as encryption keys and tunnel IP addresses will remain the same, and the new server is in the same datacentre so there should be no changes to connectivity or latency.
28th August 2020 Changes to the burble.dn42 network
Over the next year the focus of the burble.dn42 network will change focus to providing high quality, reliable services for DN42. As part of this change, a number of the current &rsquo;edge&rsquo; nodes will be decommissioned to reduce admin overhead and allow concentration on the core, service nodes.
The following nodes will be decommissioned and are no longer available for new peerings:
Node Decommissioning Date us-mia2 Immediately sg-sin1 November 2020 us-sea2 November 2020 fr-sbg1 Nocember 2020 jp-tyo1 December 2020 au-syd1 January 2021 us-nyc1 April 2021 us-chi1 May 2021 The current core nodes will continue to operate and some will also be upgraded. The number of services provided by the network will also expand.
Core nodes
Node Future Plans fr-rbx1 Increase in services offered ca-bhs2 Upgrade to ssd disks ~November 2020 de-fra1 Upgraded in August to 4 x Epyc / 20G RAM / NVMe us-dal3 Increase in services offered / potential for upgrade us-lax1 Increase in services offered sg-sin2 Take over services from sg-sin1 Other Nodes
Node Future Plans uk-lon1 Upgrade before January 2021 ch-zur1 No changes planned no-trd1 No changes planned 22nd August 2020 de-fra1 has been replaced with a shiny, upgraded, new node.
If you are peered on de-fra1, please check your configuration and ensure you are using the new IP addresses
IPv4: 193.41.237.149 IPv6: 2a0d:5941:1:17c::4e2a All other peering parameters remain the same.
15th August 2020 The DN42 registry now supports automated pipelines using Drone CI.
Details will be published on the DN42 wiki.
25th July 2020 us-mia1 and us-mia2 have been swapped. The provider for the old us-mia2 (Stockservers) appears to have ceased trading, so the node has been swapped in case the original server disappears at short notice.
EDIT: seems the new provider wasn&rsquo;t better, so us-mia2 is back on the old server until it dies completely.
no-trd1 has been added, courtesy of jastrup.
lt-vil1 is being decommissioned and users will need to migrate to a different node to maintain service.
5th July 2020 A busy weekend supporting the move of the DN42 registry to its new host.
Remember to join the new mailing list at https://groups.io/g/dn42 and create yourself an account on the new registry https://git.dn42.dev
10th June 2020 Website moved again, and new paste.burble.dn42 service added.
6th June 2020 The global route collector has had a long overdue upgrade. Please let me know if you spot any residual issues.
25th May 2020 The new DNS implementation has been deployed across all nodes. The DNS service now supports:
Authoritative DNS for DN42 domains (b.delegation-servers.dn42) Recursive DNS (b.recursive-servers.dn42) DNS64 (dns64.burble.dn42) All services support UDP, TCP, DNS over HTTPS and DNS over TLS queries. See the DNS page for more info.
23rd May 2020 A new implementation of the edge DNS service is currently being tested across a few nodes, please let me know if you spot any DNS oddness.
18th May 2020 Added whois.burble.dn42 service, see the services page for more details.
https://explorer.burble.dn42 now has regional mirrors so should be significantly faster for anyone not in Europe.
16th May 2020 Approximately 40 old or inactive peers have been deleted as part of a spring cleaning exercise.
If you&rsquo;ve been accidently deleted and still want to peer with me, just give me a shout and I will re-instate the configuration.
11th May 2020 Rate limiting on BGP sessions has been implemented to protect the network from major route flapping events. The rate limiting should only kick in after 30+ minutes of extremely high updates (or even longer for milder events), but please let me know if this causes any issues.
8th May 2020 us-lax1 has been migrated. If you peer with me please remember to update the clearnet IP addresses on your side:
dn42-us-lax1
IPv4: 185.215.224.214
IPv6: 2a0b:ae40:1:4a0a::5a
5th May 2020 us-lax1 is being upgraded !
Apologies for the short notice, but us-lax1 will be upgraded over the weekend of 9th/10th May. The upgrade will allow for more services to be provided from the node, to provide enabling a better response for users in Asia and West Coast US.
Unfortunately the upgrade means that IP address of the node will change and peers will need to update their config accordingly. The encryption keys and tunnel addresses should not need to change.
4th May 2020 Several of the burble.dn42 core nodes have been upgraded to Ubuntu 20.04. This required a short outage, but will allow for a refactoring of a few services in the future.
13th April 2020 Bugs have been fixed and both instances of the burble.dn42 website are now running in a new environment with the latest grav.
The new website instance is the first burble.dn42 application running on Ubuntu 20.04 (Focal Fossa).
11th April 2020 The clearnet version of this website is running with a new instance that has the latest grav.
Please let me know if you spot any problems.
The DN42 instance continues to run with the previous version.
4th April 2020 Well, that was fun; burble.dn42 had a number of outages over this evening, caused by trying to perform a rolling upgrade across the network. The biggest of these took out the burble.dn42 DNS service for an extended period, impacting DNS resolution across DN42.
The plan had been to perform a full upgrade and reboot for every burble.dn42 node. To minimise disruption I perform updates across groups of servers that are chosen to be independent so that service resilience should not be impacted.
However, this time there were two key failures:
The provider configuration for ca-bhs2 meant that it could not mount all of its disks when rebooted and it ended up in maintenance mode. The server needed to be recovered via the IPMI console. Whilst global services continued to be provided by other nodes, peers on ca-bhs2 lost connectivity whilst the node was recovered.
The new pdns-recursor that was implemented at the end of March (see below) had a different runtime path than the default OS install. This meant that when each of the core nodes was restarted the pdns-recursor failed to restart as the runtime path was missing. Since the DNS service is resilient, it continued to operate without problems until the last core node was restarted, at which point the entire service failed. Without DNS, most of the remaining burble.dn42 failed or could not be restarted and recovery was also hampered by having to work without having DNS available.
1st April 2020 at-vie1 will be decommissioned by 14th April. If you are peered on this node, please contact me to move the peering to another node.
28th March 2020 The patched pdns recursor is now deployed to all core nodes.
Please let me know immediately if you notice odd DNS behaviour.
24th March 2020 fr-sbg1 (which hosts the europe region core DNS service) is currently testing a special pdns recursor build in order to try and fix this issue.
The server is likely to be used for most recursive DNS lookups across Europe that use the new DNS anycast addresses, or my service directly. Please let me know immediately if you notice odd DNS behaviour.
26th January 2019 This weekend has been a huge maintenance weekend for burble.dn42, with the following updates taking place:
A number of nodes have been built and swapped in to the network to upgrade and manage renewals fr-rbx1 replaced by fr-rbx2 fr-rbx2 was a much faster node ca-bhs2 replaced with a new node the replacement is also much faster us-dal3 replaced by us-dal1 us-dal3 was a poor performer and has been replaced with a dedicated server au-syd1 replaced with a new node memory increased from 1G to 2G sg-sin2 replaced with a new node memory increased from 1G to 2G Node renewals are now mostly sorted until November, which will be a nice break for my wallet.
The build of ca-bhs2 introduced a new disk layout for my core nodes, which is intended to provide more flexibility for new features. uk-lon3, a private storage node, was also rebuilt for the new design. A bad decision around backups meant that I also had to re-create all the services on fr-rbx2 and us-dal1 as they were swapped in to their new roles. As a result, the services on these boxes were also flattened and rebuilt to the new disk layout.
At some future point, fr-sbg1 will follow and also change to the new layout.
The burble.dn42 is organised around a core network of servers in each region, the updates this weekend complete a series of changes to upgrade the core nodes that has been taking place since November 2019. A lot of the recent work has been to update the services so they are on, or point to, the new core nodes. The core network looked like this prior to November 2019:
Name CPU Memory Disk Network Descr fr-rbx1 i5-2400 (4/8 x 3.4Ghz) 16G 2TB Consumer HDD 100mbps un-metered Kimsufi KS-10 ca-bhs2 i5-3570S (4/8 x 3.8Ghz) 16G 2TB Consumer HDD 100mbps un-metered Kimsufi KS-10 sg-sin2 virtual (1 x 3.5Ghz) 1G 30GB HDD 1TB @ 1gbit OVH VPS us-dal3 virtual (2 x 3.4Ghz) 5G 120GB HDD 5TB @ 10gbit HostDoc VPS Following the upgrades, the core now consists of the following servers:
Name CPU Memory Disk Network Descr fr-sbg1 E5-1620 (4/8 x 3.7Ghz) 32G 3 x 480GB SSD 500mbps un-metered OVH SYS fr-rbx1 E3-1245 (4/8 x 3.4Ghz) 32G 2 x 480GB SSD 500mbps un-metered OVH SYS uk-lon3 virtual (2 x 3Ghz) 3G 3TB HDD 10TB @ 1gbit HostHatch ca-bhs2 E5-1620 (4/8 x 3.7Ghz) 32G 2 x 2TB Ent. HDD 500mbps un-metered OVH SYS us-dal3 C2750 (8 x 2.4Ghz) 8G 240GB SSD 100mbps un-metered drserver sg-sin1 virtual (4 x 2.2Ghz) 4G 24GB SSD 1gbit un-metered ITLDC VPS `}),e.add({id:1,href:"/additional/maintlog/2019/",title:"2019",parent:"Maint. Log Archive",content:`Archive of changes made in 2019
31st December 2019 The Christmas period has been a really busy period for burble.dn42, with integration and transfer of services over to the new nodes. Primarily, this has meant moving services from fr-rbx1 and sg-sin2 to fr-rbx2, fr-sbg1 and sg-sin1. As part of the rebuilding, I&rsquo;ve also taken the opportunity to re-create most of my ansible scripting, with the intent that this will eventually be published.
Most services are now moved, with the main exception of DNS and the GRC, both of which need more significant work. The website also now needs major updates to reflect the changes I&rsquo;ve made.
The following new nodes are also open for peering:
dn42-fr-rbx2 dn42-fr-sbg1 dn42-ch-zur1 dn42-sg-sin1 dn42-hk-hkg1 Happy New Year
24th December 2019 The last month has been spent redesigning my WAN and introducting a latency based metric for connectivity between nodes. This is now mostly complete, but not without its own follow on problems that need to be resolved.
Things still to do include:
Fixing the service delivery layer as a software upgrade breaks IPv6 connectivity Adding documentation to the website on the new design Opening new nodes for peering Making the config public Another new node will also be added, dn42-fr-rbx2 and dn42-fr-rbx1 will be retired.
Merry Christmas DN42
29th November 2019 Black friday is here and new nodes are on the way.
dn42-fr-sbg1 dn42-ch-zur1 dn42-sg-sin1 dn42-hk-hkg1 2nd November 2019 Retired dn42-us-lax2, dn42-us-chi2, dn42-ca-bhs1, dn42-tr-ist1 and dn42-no-osl1.
Restructured the internal confederations.
26th October 2019 New experimental node added hosted in the Oracle Cloud environment in Mumbai, India.
Users are welcome to peer and test the node, but should be aware there may be short notice changes or interruptions to service.
19th October 2019 After a few weeks of outage and putting up with influx using up a vast amount of resources, the monitoring service has finally moved to a federated prometheus architecture. Hopefully this will have better performance than the influx architecture used previously. At some point I&rsquo;ll update the monitoring page with details of the new configuration.
12th October 2019 The burble.dn42 wiki service is now part of the global anycast for wiki.dn42.
See the services page for more details.
2nd October 2019 The recursive DNS service now supports clearnet queries
15th September 2019 Stop supporting IPsec tunnels
21st August 2019 Removed sg-sin3 and vn-han1
13th August 2019 Added DN42 wiki service editable via dn42, readonly via clearnet.
Issued new Certificate Authority root certificate with a longer expiry date.
11th August 2019 Added a couple of Python 3 updates for bird-lg that fixes broken BGP map functionality in the looking glass.
Influx ate all the memory (10gb!) on de-fra1, so is currently offline until it can be fixed.
28th July 2019 Add dn42-us-mia2, which will replace dn42-us-mia1
25th July 2019 Add pingable.burble.dn42
21st July 2019 Decommissioning of dn42-ru-mos1 and dn42-us-sea1
17th July 2019 DoH! The DNS Service now support DNS over HTTPS.
22nd June 2019 Tidied up node information.
14th June 2019 A new host IRC web service has been added, based on thelounge.
See the services page for more details.
8th June 2019 The recursive DNS service now uses parallel queries across all five regional master nodes.
This approach takes advantage of the burble.dn42 global scale to reduce latencies, improve resilience and prevent local connectivity problems from impacting the results. See the DNS page for more info.
24th May 2019 Moved and extended the DN42 monitoring so that it is more independent and also clustered.
A writeup of the hosted grafana service and monitoring is available here.
21st May 2019 dn42-uk-lon1 is back again after being out of action for the day.
The host server apparently threw a disk after being updated to cover the MDS vulnerability and the provider has spent the day recovering the node.
20th May 2019 Some nodes may have outages over the next few days as providers deal with the recent MDS vulnerabilities.
Added new peers
AS4242421588 / TECH9 at dn42-us-lax2 AS4242421166 / MTR at dn42-fr-rbx1 and dn42-de-fra1 15th May 2019 Updated my fork of bird-lg by merging Zhaofeng&rsquo;s Python2 to Python3 bird-lg updates and fixing a few outstanding problems.
The updated code is now live on the burble.dn42 looking glass.
13th May 2019 Moved the looking glass to its own container, in anticipation of future website changes
dn42-us-mia1 is offline again.
10th May 2019 dn42-us-chi2 was suspended by the provider on 8/5 due to &lsquo;NTP reflection attacks&rsquo;.
This is a hazard of running a busy NTP server as part of the NTP Pool; providers can get twitchy when they see a large amount of NTP traffic, due to the well publicised vulnerabilities in stock NTPd.
My network uses chronyd rather than NTPd and it is simply not vulnerable to abuse in the same way as NTPd, I also regularly monitor and check the services. On the other hand, the server does see a large amount of NTP traffic and it can sometimes be difficult demonstrating that I&rsquo;m specifically providing a service here and not under some kind of attack.
Apologies that the server was offline for a few days, but it should now finally be back again.
For info, here is the bandwidth graph of dn42-us-chi2 as it was suspended:
It&rsquo;s trivial to see that an amplification attack was not occuring, as the inbound and outbound traffic are both equal. It&rsquo;s a shame some providers don&rsquo;t consider this before suspending services, but, understandable that the economics of providing VPS services can prohibt this.
Added new peers:
AS4242422322 / PLASMATRIX at dn42-de-fra1 5th May 2019 Added git service.
See the services page for more details.\`&lt;
1st May 2019 Seems traceroutes and some Europe Region, IPv4 related DNS lookups weren&rsquo;t working.
Both are fixed now.
Added new peers:
AS76140 / FEUERROT at dn42-de-fra1 30th April 2019 New node added and ready for peering
dn42-ca-bhs2 (Beauharnois, Canada) With the addition of several new nodes, the internal BGP confederations have been re-orginised.
This new organisation should provide better balance and allow for more local services.
The North American region has been split in two, becoming Central &amp; West Coast and East Coast. lt-vil1 and at-vie1 have been moved to the East Europe region. Added new peers:
AS4242423581 / CLOUDYSKIES at dn42-us-lax2 AS4242420141 / DEEPWATER at dn42-de-fra1 AS4242420246 / XESXEN at dn42-fr-rbx1 and dn42-uk-lon1 AS4242422543 / RESETTRAP at dn42-jp-tyo1 19th April 2019 New nodes added and ready for peering.
dn42-at-vie1 (Vienna, Austria) dn42-us-nyc1 (New York, United States) 18th April 2019 Over the last week, and number of major changes have taken place to the burble.dn42 network.
These include:
Configuring Jool to provide IPv4 to IPv6 SIIT for the new 172.20.129.0/27 prefix
The aim is for all internal services of the burble.dn42 network to be provided by IPv6, with SIIT taking place at the network edge for external IPv4 users. Configuring Jool to provide a NAT64 service
So that internal, IPv6 only, services can access external IPv4 networks Adding a new VXLAN to the WAN overlay
The new VXLAN segregates DN42 traffic from the internal traffic and enables a separate DN42 routing domain. As a side effect, this change also fixes the problem where internal IP addresses were being leaked and causing confusing traceroutes for DN42 users. Over time, internal IPv4 services will be removed
12th April 2019 New prefix 172.20.129.0/27 registered to provide space for more nodes and additional services.
172.20.129.0/27 will be used as anycast addresses for services. 172.20.129.160/27 will be used for burble.dn42 nodes
Added new peers:
AS4242421063 / ZIIS at dn42-uk-lon1 AS4242421475 / SIRMYSTERION at dn42-us-chi2 7th April 2019 Added an old node in to the DN42 network, dn42-sg-sin2. RPKI and DNS services have been moved to the node from dn42-sg-sin2 which should improve diversification and stability.
3rd April 2019 Added new peers:
AS4242423974 / GIGGA at dn42-sg-sin3 31st March 2019 The DNS service has gone global, with every node in the burble.dn42 network now participating in the DNS Anycast service.
More details can be found on the DNS page.
26rd March 2019 Added new peers:
AS4242420568 / MARSHY at dn42-au-syd1 AS4242423853 / CHENYAO2333 at dn42-ca-bhs1 AS4242423328 / DEBOERDN2000 at dn42-ca-bhs1 AS4242423924 / EVILZONE at dn42-sg-sin3 11th March 2019 New node added dn42-de-fra1
9th March 2019 Added new peers:
AS4242420101 / HEXA at dn42-fr-rbx1 AS4242423783 / OZARK at dn42-au-syd1 AS4242420571 / CAICAI at dn42-vn-han1 A new instance of the registry explorer has been created that references the &lsquo;object-fix&rsquo; branch of the DN42 registry. The main purpose of this is to support the new DNS system being developed.
http://grc.burble.dn42:8043/
A couple of the nodes on the network experienced some downtime over the week:
dn42-us-mia1 was down to 2 days and had to be rebuilt as my VPS provider&rsquo;s storage array crashed. dn42-us-dal3 was also down for a few hours, the provider accidently suspended the VPS due to a billing error on their side 7th March 2019 Added new peers
AS4242421955 / NOP at dn42-fr-rbx1 AS4242420161 / ZZZ at dn42-jp-tyo1 26th February 2019 Initialised GRC website
Added new peers
AS4242422626 / HANNIBAL at dn42-fr-rbx1 AS4242423156 / BUROA at dn42-us-chi2 21st February 2019 The Looking Glass has been udpated to use lgregmapper and data from dn42regsrv.
19th February 2019 New peer added:
AS4242423975 / FELIX at dn42-fr-rbx1 18th February 2019 The internal and public ROA service has been moved over to using dn42regsrv.
See the services page for more details.
New peer added:
AS4242423973 / TECHNOPOINT at dn42-sg-sin3 16th February 2019 New peers added:
AS4242420182 / JAN at dn42-uk-lon1 AS4242422042 / KLEEN at dn42-fr-rbx1 AS4242423201 / DDPO at dn42-uk-lon1 10th February 2019 Updated the services to include new stuff::
DNS Registry REST API and Explorer Global Route Collector New peers added:
AS4242420191 / TCDUE at dn42-uk-lon1 AS4242422019 / HENOKV at dn42-fr-rbx1 AS64713 / MARTIN89 at dn42-fr-rbx1 AS4242423000 / RELROD at dn42-ca-bhs1 AS4242421656 / PHIIVO at dn42-us-lax2 26th January 2019 New service !
A burble.dn42 route collector has been added, together with some interesting stats showing reachability of DN42 from the burble.dn42 network.
A common, global route collector is in progress, see here
21st January 2019 New peer added:
AS4242423306 / TIMK at dn42-au-syd1 13th January 2019 bgpmap updated to add MNT and prefix info for ASes.
New peers added:
AS4242420415 / TYLER at dn42-us-lax2 AS4242423569 / DHE at dn42-us-dal3 AS4242423585 / JD52RU at dn42-fr-rbx1 and dn42-uk-lon1 12th January 2019 The Looking Glass now supports bgpmap again.
My bird-lg fixes are available on github.
New peer added:
AS4242421501 / ADAMYI at dn42-au-syd1 11th January 2019 Some layout fixes to the Looking Glass, including fixing whois lookups.
3rd January 2019 First new peers of 2019:
AS4242420505 / 42ISLIFE at dn42-ca-bhs1 AS4242421114 / GRGR at dn42-us-chi2 AS4242421050 / NAPSTERBATER at dn42-us-chi2 2nd January 2019 Consolidated number of anycast sessions.
`}),e.add({id:2,href:"/additional/maintlog/2018/",title:"2018",parent:"Maint. Log Archive",content:`Archive of changes made in 2018
30th December 2018 Migrated US anycast services from dn42-us-dal1 to dn42-us-dal3.
27th December 2018 Added Certificate Authority details.
26th December 2018 Upgraded the looking glass with Zhaofeng bird-lg fixes.
ROA data is available through the burble.dn42 website, see the Services page.
RPKI service is now replicated across regions to provide additional resiliency.
New version of bird2 deployed, including RPKI fixes from JRB0001.
24th December 2018 Added new peers:
AS4242422255/LINUXGEMINI at dn42-tr-ist1 AS4242421191/YAMAKAJA at dn42-fr-rbx1 AS4242423230/RASP at dn42-au-syd1 Updated the Services page to include more implementation details.
Reworked intra-confederation peering to provide more resilience.
Implemented ROA via RPKI updates using roasrv by Yamakaja and gortr
16th December 2018 New node !
dn42-jp-tyo1 has been commissioned and is open for new peers in Tokyo, Japan.
14th December 2018 Updated host information and network map with new nodes.
10th December 2018 New peers added:
AS4242423090/HEIAS at dn42-fr-rbx1 AS4242421979/MDUCHARME at dn42-us-sea2 dn42-us-sea2 is now operational and available for peering.
2nd December 2018 tinc + babeld is not a winning combination. Since introducing babeld, the burble.dn42 WAN overlay has experienced a number of periods of instability, with nodes dropping on and off the network.
The WAN has been updated to use a Wireguard mesh with OSPF as IGP, and is now significantly more stable again.
1st December 2018 New peers added:
AS4242420260/GISH at dn42-au-syd1 AS4242421009/KLARA at dn42-no-osl1 AS4242420058/ILL at dn42-au-syd1 AS4242422547/LANTIAN at dn42-fr-rbx1 / dn42-us-lax2 / dn42-sg-sin3 30th November 2018 Three new nodes will be available for peering soon:
dn42-us-chi2 - Chicago, United States dn42-us-sea2 - Seattle, United States dn42-us-dal3 - Dallas, United States 29th November 2018 dn42-us-dal1 locked up, and has been restarted.
28th November 2018 dn42-uk-lon1, dn42-lt-vil1, dn42-sg-sin1 and dn42-us-mia1 all locked up at 03:00 UTC and have now been restarted.
23nd November 2018 Black Friday has delivered four new nodes to the burble.dn42 network:
dn42-vn-han1 - Hanoi, Vietnam dn42-no-osl1 - Oslo, Norway dn42-ca-bhs1 - Beauharnois, Canada dn42-us-lax2 - Los Angeles, United States dn42-sg-sin3 - Singapore All nodes are open to new peers, so just contact dn42@burble.com if you&rsquo;d like to connect to the network.
22nd November 2018 New peers added:
AS4242420165/ZAICA at dn42-fr-rbx1 AS42424222673/CORESTORAGE at dn42-uk-lon1 18th November 2018 Updates to reverse DNS.
17th November 2018 Added new peers
AS4242423640/HESSENET at dn42-fr-rbx1 AS4242420149/NIRF at dn42-lt-vil1 17th November 2018 The internal routing protocol (IGP) for burble.dn42 has moved from OSPF to using babeld.
All nodes on the burble.dn42 network are inter-connected with a tinc mesh. Despite the network physically spanning across contintents, OSPF saw the tinc overlay network as being flat which prevented effective use of technologies such as anycast and forced the use of central resources. The hope is that babel, configured to use an RTT metric, will allow better use of regional services.
Please let me know if you observe any issues due to the new IGP.
16th November 2018 New node in Istanbul, Turkey.
dn42-tr-ist1 has been commissioned and is now open for new peers. See the peering page for more details.
`}),e.add({id:3,href:"/retro/modem/",title:"Dialup Service",parent:"Retro42",content:`Connect in to dn42 using a real physical modem.
06/02/23 - The modem service is currently unavailable and will remain offline until later in the year.
In the meantime, the modem emulator service is still available.
Health warning: dialing in to dn42 can be rewarding and great fun, but using modems over VoIP is flakey at the best of times and getting it to work can be a frustrating experience. This is very much an experimental service. The burble.dn42 dialup service allows you to dial in to dn42 via a modem. The modem is reachable via VoIP and connected to a PPP server, allowing you to log in and obtain full dn42 IPv4 connectivity.
Endpoints The dialup service can be reached via the following endpoints:
Endpoint Network 424026019@voip.burble.dn42 dn42 tbc clearnet tbc PSTN PPP Server Log In Details Username: dn42 Password: dn42 See also the the modem emulator for more details on how the PPP server is configured, together with an automatic login script for windows variants.
Connection Tips Ensure that T.38 Fax Mode is disabled or set to passthrough Ensure that Re-invite after Fax Tone is disabled Ensure that all Echo Cancelation is disabled Ensure VAD (Voice Activity Detection) is disabled Disable all call features (voicemail, call waiting etc.) Use G.711 ALAW G.711 ULAW may work if ALAW isn&rsquo;t available, but do not use any other codecs Experiment with Disabling/Fixed/Adaptive Jitter buffers and different jitter buffer lengths Experiment with different gain settings Reduce modulation rates, you can&rsquo;t connect at anything above V.34 anyway Use AT+MS? to see the current modulation settings on your modem. Start at V.34 (AT+MS=V34) Try V.32bis (AT+MS=V32b) or even V.32 (AT+MS=V32) The modems are located on a residential DSL connection and will experience more jitter when the DSL is in use, try connecting at different times and specifically during UK night time or early mornings. The PPP log in process is the same as the modem emulator so you can use the emulator to check your process before connecting using the modem. Example connection from linux The video shows an example connecting using minicom and pppd on linux.
Using a software modem JerryXiao has created instructions for connecting using D-Modem, a software modem that can connect directly over VoIP. The instructions and D-Modem software are available in the dn42 gitea repository.
`}),e.add({id:4,href:"/network/",title:"Network",parent:"burble.dn42",content:`Information about the burble.dn42 network.
Network Design: burble.dn42 network design Peering with burble.dn42: How to peer with burble.dn42 Node Information: Detailed Node Information IPAM: IP Address Lists Routing Policy: Description of the network routing policy BGP Communities: BGP communities used in the network Realtime Status: Network Status Abuse Policy: Abuse Policy `}),e.add({id:5,href:"/network/peering/",title:"Peering with burble.dn42",parent:"Network",content:`This page provides the information to get started on peering with the burble.dn42 network
burble.dn42 is a set of global POPs integrated to the dn42 network, and new peering requests are welcome. Some details of the network are available in the Design page.
burble.dn42 is a large network and there are some restrictions in place to protect the network and the rest of the DN42. Please ensure you read the information below before requesting to peer.
Peering Requests Please mail dn42@burble.com if you&rsquo;d like to peer with me.
Peering Requirements To peer with burble.dn42, you must meet the following requirements:
You must support IPv6
You must implement ROA checks
Contact information in the registry must always be up to date and admins must respond when contacted
Contacts must also be reachable in case of problems. In addition, the network is ever evolving and failure to respond to change notices may result in your peering being suspended.
Latency to your node must be reasonable, typically this means a latency less than 40ms.
At a minimum, I&rsquo;ll need to know the following in order to establish a peering:
The burble.dn42 node you would like to peer with Your ASN The public address of your host The tunnel and BGP parameters, e.g. Port number, if using wireguard or OpenVPN Public key for wireguard Any special config you need that is different to my defaults (see the Supported Tunnel Types and BGP Feature Support sections) IP addresses of your end of the tunnel Typically these will be a single IPv4/32 and Link-Local IPv6 address All peerings will be configured as a full transit session.
Residential ISPs and Dynamic IP Addresses
A 24/7 connection, with static IP addresses are the norm for DN42. If you are connecting from a residential ISP or otherwise have a dynamic IP please let me know so that I can configure my side appropriately. If you don&rsquo;t do tell me, the peering may stop working when your IP address changes.
Peering in Multiple Locations
If you have multiple nodes, you are welcome to peer in several locations to provide additional redundancy and route choice. Routes exported from the network include a latency based MED attribute to help peers optimise their routing (See the Routing Policy)
It&rsquo;s highly recommended to peer with multiple DN42 users though, it&rsquo;s lots of fun and you should never rely on just one user for your connectivity.
Supported Tunnel Types I prefer to use wireguard, it&rsquo;s simple to set up and just works. I also support OpenVPN tunnels.
Wireguard The port number will be 2xxxx where xxxx is the last four digits of your ASN. Each peer is assigned a unique encryption key, pre-shared keys are also supported (but not enabled by default). Endpoint names and IP addresses are detailed in the nodes page. My wireguard AllowedIPs are:
AllowedIPs=fe80::/64 AllowedIPs=fd00::/8 AllowedIPs=0.0.0.0/0 Use of wg-quick
Using wg-quick is not recommended as it does not support adding a peer address. If you want to use wg-quick you will need to delete and re-add the wireguard interface IP address and configure it as a point to point address or you will run in to next-hop problems when using BGP. You must read the DN42 Wiki on how to set up wg-quick for use within DN42.
OpenVPN The port number will be 2xxxx where xxxx is the last four digits of your ASN. By default I will configure the following OpenVPN parameters:
comp-lzo cipher aes-256-cbc auth sha256 Tunnel Configuration Allowed Traffic Only the following network ranges will be forwarded through the DN42 network, all other traffic will be dropped.
IPv4
172.16.0.0/12 10.0.0.0/8 IPv6
fd00::/8 BGP peer addresses are more permissive to allow for link local or non-DN42 IP addresses within the tunnel, but these will not be forwarded through the DN42 network. Flow Control and BGP Rate Limiting A typical BGP session in DN42 will use a trivial amount of traffic. However, for large networks like burble.dn42 some transient events, such as BGP flapping, can generate multi MB/sec traffic flows that damage the network and create instability across DN42.
To protect the network from misconfigurations and prevent excessive updates from propagating to the rest of DN42, the burble.dn42 network implements rate limiting on BGP sessions. The rate limiting activates when a large amount of BGP traffic is seen (typically 10&rsquo;s or 100&rsquo;s of thousands of updates a second) over a sustained period and will typically reset automatically within an hour.
There are no other controls applied to transit or non-BGP traffic.
BGP Configuration Network Name BURBLE BURBLE-MNT dn42@burble.com ASN AS4242422601 BGP Feature Support The burble.dn42 network uses a custom build of bird 2, and the following features are supported:
Multiprotocol BGP - RFC 4760 BGP Large Communities - RFC 8092 BGP Confederations - RFC 5065 Extended Next Hop - RFC 5549 Extended Messages - RFC 8654 DN42 Route Origin Authorisation (ROA - see below section on Route Filtering) DN42 Route Origin Community Note that DN42 latency, bandwidth and encryption communities are not supported. burble.dn42 custom large communities burble.dn42 Routing Policy The source code for the custom bird used on the network is available on git.burble.dn42
Default Extensions Multiprotocol BGP is preferred, however it is not enabled by default as not all peers can support it. Please let me know when peering if you can support a multiprotocol BGP session.
Extended next hop and extended message support are both enabled by default.
Route Filtering The network applies strict Route Origin Authorisation (ROA) filtering to all received and exported routes. This means any advertised route that does not have a corresponding route{,6} object in the DN42 registry will be dropped.
ROA is implemented with updates through RPKI, using dn42regsrv and gortr.
The DN42 ROA data is provided as a public service, see the Services page.
Generic Allowed Prefixes: IPv4
172.20.0.0/14+ 10.0.0.0/8+ IPv6
fd00::/8{44,64} Testing Connectivity Testing Within the tunnel, hosts respond to ping and traceroute, but also have the echo (port 7) and daytime (port 13) services enabled. These can be used to check the tunnel is up and configured correctly.
$ ping fe80::42:2601:32:1%wg0 PING fe80::42:2601:32:1%wg0(fe80::42:2601:32:1%wg0) 56 data bytes 64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=1 ttl=64 time=4.44 ms 64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=2 ttl=64 time=4.52 ms 64 bytes from fe80::42:2601:32:1%wg0: icmp_seq=3 ttl=64 time=4.96 ms ^C --- fe80::42:2601:32:1%wg0 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 4.445/4.643/4.961/0.233 ms $ netcat fe80::42:2601:32:1%wg0 13 Sun Sep 23 09:57:26 2018 ^C $ Reachability Testing Once peering is established I have a BGP looking glass (public internet link) and global route collector which can be used to check routing configuration. Looking glasses are a key, self-service resource for you to use when understanding how your routes are propagating around the DN42 network, please take the time to learn how to use them.
Speed Test burble.dn42 operates a speed test service at speedtest.burble.dn42, see the services pages for more info.
Automated Tests pingable.burble.dn42 (172.20.129.5 / fd42:4242:2601:ac05::1) is a dedicated address that responds to ping and traceroute and may be used for automated reachability or link quality testing.
Please be considerate when configuring automated tests and set a reasonable test frequency.
In all cases, the frequency must not be more than once a second. Please consider this if your router automatically pings its tunnel endpoint for stats purposes.
`}),e.add({id:6,href:"/services/dn42/",title:"DN42",parent:"Services",content:`burble.dn42 hosts a number of DN42 infrastructure services.
DNS Service Name IP DN42 Master b.master.delegation-servers.dn42 fd42:180:3de0:30::1 Authoritative Service b.delegation-servers.dn42 172.20.129.1fd42:4242:2601:ac53::1 Recursive Service b.recursive-servers.dn42 172.20.129.2fd42:4242:2601:ac53::53 burble.dn42 provides a local, anycast, authoritative and recursive DNS service.
The DNS Service has it&rsquo;s own page.
DN42 Wiki Mirror Mirror URLs wiki.dn42wiki.burble.dn42 (editable via DN42) dn42.devwiki.burble.com (read-only via public internet) burble.dn42 maintains a globally distributed mirror of the DN42 Wiki, and is part of the wiki.dn42 anycast group. The DN42 services (wiki.dn42 and wiki.burble.dn42) are editable, whilst the public internet views (dn42.dev and wiki.burble.com) are read-only.
Please note that updates to the wiki may take several hours to sync with other mirrors.
The wiki service is delivered using the burble.dn42 nomad cluster.
ACME https://acme.burble.dn42/v1/dn42/acme/directory https://acme.burble.dn42/v1/staging/acme/directory burble.dn42 provides an ACME service using an intermediate certificate issued by the dn42 certificate authority and implemented using a HashiCorp Vault cluster to provide a highly available service.
More details can be found on the ACME service page.
Whois Service whois.burble.dn42 fd42:4242:2601:ac43::1 172.20.129.8 WHOIS service providing data from the DN42 registry. The WHOIS service is also anycasted across the network.
The source code for the service is available in the burble.dn42 git.
Global Route Collector https://collector.dn42/ (DN42 link) https://lg.collector.dn42/ (DN42 link) ssh shell@collector.dn42 The global route collector provides a central bird instance that collects routes from peers across the DN42 network.
All users are invited to join the collector and help provide stats for the network.
The route collector can currently be queried by using ssh to connect a bird shell or via a looking glass.
Additional services and stats are expected to be developed in the future.
DN42 Infrastructure Monitoring burble.dn42 hosts monitoring and alerting of key DN42 services at https://grafana.burble.dn42.
`}),e.add({id:7,href:"/retro/fake/",title:"Modem Emulator",parent:"Retro42",content:`No modem ? no problem.
If you don&rsquo;t have a physical modem you can still get that retro vibe by accessing dn42 using the modem emulator service. This service presents a series of TCP/IP ports that emulate a subset of the Hayes AT command set and which are speed limited to specific baud rates. The modem can dial directly to the fr-par1 shell service for that CLI experience, or get full dn42 access via a PPP session.
The service is intended to be used as a fake serial port in a retro VM or connected to directly using a serial terminal.
Accessing the modem emulator service dialup.burble.dn42:(10000 + baud/100) The modem emulator is accessed by connecting to dialup.burble.dn42 on ports corresponding to specific baud rates, as detailed in the table below.
Port BAUD Port BAUD Port BAUD 10003 300 10012 1200 10024 2400 10048 4800 10096 9600 10144 14400 10192 19200 10288 28800 10336 33600 10560 56000 11150 115000 Low baud rates give a great experience when using a serial terminal and the shell service but aren&rsquo;t so much fun when using PPP. Conversely, using a serial terminal and higher baud rates is a bit dull and they are better suited to the PPP service. The emulated modem is able to dial two services depending on the number used.
Dial Number Service 54311 fr-par1 shell server 4242 PPP dialup service Dialling the fr-par1 shell server Use a terminal emulator and dial 54311 to access the fr-par1 shell server.
Once the modem connects, use your burble.dn42 username and password to log in.
You can set a burble.dn42 password using the services portal. Don&rsquo;t forget to set a terminal type once you&rsquo;ve logged in, e.g. export TERM=vt102 Dialing the PPP server Dial 4242 to access the PPP server. Use username: dn42, password: dn42 to log in. Once you see the PPP initiation string, connect your PPP client. PPP authentication should not be required. Example shell session using socat and minicom Example on how to connect to the shell server using socat and minicom at 2400 baud
$ socat -d -d pty,rawer tcp:dialup.burble.dn42:10024 &amp; ... ... socat[xxx] N PTY is /dev/pts/2 ... $ minicom -D /dev/pts/2 Within minicom type ATDT54311 to dial the shell server.
Welcome to minicom 2.8 OPTIONS: I18n Port /dev/pts/2, 14:01:58 Press CTRL-A Z for help on special keys ATDT54311 CONNECT 2400 Ubuntu 22.04.1 LTS shell-fr-par1 login: You can now log in using your burble.dn42 username and password.
Example PPP connection using QEMU and Windows XP Add a virtual serial port to QEMU using -chardev and -device.
e.g. for windows XP connecting at 115k
qemu-system-i386 -enable-kvm -cpu host -m 1024 -vga std -device AC97 -hda winxp.img -chardev socket,id=bdn42,port=11150,host=dialup.burble.dn42,server=off -device pci-serial,chardev=bdn42 Create a PPP dialup connection with username dn42 and password dn42, enable the option to prompt for username and password on connect.
Start the connection and when then chat box opens:
Type ATDT4242 to dial the PPP service Log in using username dn42 / password dn42 Hit continue when you see the PPP connect string The following chat script provided by 04dco should provide automatic login for win95 onwards:
; ; This is a script file that automatically logs you in ; to establish a dn42 dial-up PPP connection. ; ; Main entry point to script ; proc main integer nTries = 3 ; This is the login prompt and timeout values string szLogin = &#34;modem-server Login: &#34; integer nLoginTimeout = 5 ; This is the password prompt and timeout values string szPW = &#34;Password: &#34; integer nPWTimeout = 5 ; This is the prompt once your password is verified string szPrompt = &#34;Last login: &#34; ; Attempt to login at most &#39;nTries&#39; times while 0 &lt; nTries do ; Wait for the login prompt before entering ; the user ID, timeout after x seconds waitfor szLogin then DoLogin until nLoginTimeout TryAgain: transmit &#34;^M&#34; ; ping nTries = nTries - 1 endwhile goto BailOut DoLogin: ; Enter user ID transmit $USERID, raw transmit &#34;^M&#34; ; Wait for the password prompt waitfor szPW until nPWTimeout if FALSE == $SUCCESS then goto TryAgain endif ; Send the password transmit $PASSWORD, raw transmit &#34;^M&#34; ; Wait for the prompt waitfor szPrompt goto Done BailOut: ; Something isn&#39;t responding. Halt the script ; and let the user handle it manually. set screen keyboard on halt Done: endproc Source code The modem emulator is a small golang server and the source code is available on the burble.dn42 git server
`}),e.add({id:8,href:"/network/nodes/",title:"Node Information",parent:"Network",content:`Public peering nodes
Europe dn42-fr-par1 Location Scaleway/Online.net, Paris, France Specs E3-1230v2 - 4c/8t, 32GB, 2 x 1TB SSD, 1Gbit unmetered Public Hostname dn42-fr-par1.burble.com Public IPv4 Address 51.159.194.131 Public IPv6 Address 2001:bc8:1201:50d:d6ae:52ff:fecc:c97 Tunnel IPv4 Peer Address 172.20.129.181/32 Tunnel IPv6 Link Local fe80::42:2601:37:1/64 Tunnel IPv6 ULA fd42:4242:2601:37::1/128 dn42-uk-lon1 Location HostHatch, London, UK Specs 6 x EPYC 7513, 24GB, 270GB NVME, 40TB @ 10gbit Public Hostname dn42-uk-lon1.burble.com Public IPv4 Address 45.91.93.104 Public IPv6 Address 2a0e:dc0:5:5::1 Tunnel IPv4 Peer Address 172.20.129.187/32 Tunnel IPv6 Link Local fe80::42:2601:35:1/64 Tunnel IPv6 ULA fd42:4242:2601:35::1/128 dn42-de-fra1 Location Bero Host, Frankfurt, Germany Specs 6 x EPYC 7443P, 24GB, 300GB NVME, 25TB @ 10gbit Public Hostname dn42-de-fra1.burble.com Public IPv4 Address 5.180.253.211 Public IPv6 Address 2a0e:6a80:3:430::1 Tunnel IPv4 Peer Address 172.20.129.169/32 Tunnel IPv6 Link Local fe80::42:2601:31:1/64 Tunnel IPv6 ULA fd42:4242:2601:31::1/128 dn42-ch-zur1 Location HostHatch, Zurich, Switzerland Specs 2 x E5-2690v2, 12GB, 60GB SSD, 10TB @ 10gbit Public Hostname dn42-ch-zur1.burble.com Public IPv4 Address 45.91.92.111 Public IPv6 Address 2a0e:dc0:6:8::1 Tunnel IPv4 Peer Address 172.20.129.174/32 Tunnel IPv6 Link Local fe80::42:2601:28:1/64 Tunnel IPv6 ULA fd42:4242:2601:28::1/128 dn42-no-trd1 Location Trondheim, Norway Specs 2 shared cores, 2GB, 16GB SSD, 1gbit unmetered Public Hostname dn42-no-trd1.burble.com Public IPv4 Address 217.168.87.226 Public IPv6 Address 2001:678:dd0:ffff::25 Tunnel IPv4 Peer Address 172.20.129.185/32 Tunnel IPv6 Link Local fe80::42:2601:39:1/64 Tunnel IPv6 ULA fd42:4242:2601:39::1/128 North America dn42-us-lax1 Location HostHatch, Los Angeles, United States Specs 2 x EPYC 7413, 8GB, 60GB NVME, 10TB @ 10gbit Public Hostname dn42-us-lax1.burble.com Public IPv4 Address 45.67.219.3 Public IPv6 Address 2a04:bdc7:100:4656::1 Tunnel IPv4 Peer Address 172.20.129.172/32 Tunnel IPv6 Link Local fe80::42:2601:2a:1/64 Tunnel IPv6 ULA fd42:4242:2601:2a::1/128 dn42-us-nyc1 Location HostHatch, New York, United States Specs 2 x EPYC 7443P, 8GB, 60GB NVME, 10TB @ 10gbit Public Hostname dn42-us-nyc1.burble.com Public IPv4 Address 109.205.61.36 Public IPv6 Address 2605:4840:2:eed4::1 Tunnel IPv4 Peer Address 172.20.129.175/32 Tunnel IPv6 Link Local fe80::42:2601:29:1/64 Tunnel IPv6 ULA fd42:4242:2601:29::1/128 `}),e.add({id:9,href:"/services/public/",title:"Public Services",parent:"Services",content:`Services provided for use within DN42
Website burble.dn42 (dn42 link) dn42.burble.com (public internet link) This website is built using Hugo and is distributed across burble.dn42 core nodes.
The public internet site is a CloudFlare pages application and the source is published in the burble.dn42 git.
Service Administration Portal svc.burble.dn42 The service portal allows you to configure your burble.dn42 services.
Functionality includes:
Setting or changing a burble.dn42 LDAP password Changing your shell for the shell services Viewing peering information Diagnostic Services Looking Glass lg.burble.com (public internet link) lg.burble.dn42 (dn42 link) The burble.dn42 looking glass is based on bird-lg-go with some local customisations. The source code for the looking glass is available on the burble.dn42 git.
The looking glass is replicated via the nomad cluster, with the public version behind CloudFlare.
Pingable IP address pingable.burble.dn42 172.20.129.5 fd42:4242:2601:ac05::1 pingable.burble.dn42 is a single IP address that will respond to ping and traceroute requests across the entire network.
This address may be used for automated reachability or latency tests, however please be considerate and configure a reasonable test frequency.
In all cases, do not set the ping frequency to be higher than once a second. Speed Test speedtest.burble.dn42 The speed test service provides a way for you to measure your latency and bandwidth to the core burble.dn42 service clusters.
The service is based on is OpenSpeedTest.
The speedtest service is delivered via the internal traefik and nomad clusters which are anycasted and have internal load balancing. This means that the particular server providing the speedtest is non-deterministic and you may get different results across speedtest runs. The speedtest service is provided for your benefit, please don&rsquo;t abuse the service or using excessive bandwidth. Network Status and Reporting Grafana Dashboards https://grafana.burble.dn42 dn42 link https://grafana.burble.com public internet link Uptime monitoring https://stats.uptimerobot.com/l2913c0R6 Major nodes are also monitored off-network by UptimeRobot.
Anycast / Dynamic Route Monitoring https://anycast.burble.dn42 Realtime dashboard for anycast / dynamic services. Anycast status is reported every minute to the nats.io cluster and the dashboard is a live visualtion of the current status.
Internal monitoring Internally, nodes are measured by netdata which provides a real time view of each node. prometheus is then used to collect and store that data for historical reporting. grafana is used for visualisation.
Syslogs are exported in real time to a central logging node on the internal network.
Shell Accounts The burble.dn42 shell service provides shell accounts for dn42 users who have a burble.dn42 password or SSH auth methods in the registry.
See the Shell Accounts page.
S3 Compatible Object Store https://minio.burble.dn42 - Web interface s3.burble.dn42 - S3 compatible interface A min.io instance provides S3 compatible storage space for users that have registered a burble.dn42 password.
See the S3 Object Storage page for more information and examples.
DNS Service Name IP Authoritative Service ns1.burble.dn42 172.20.129.1fd42:4242:2601:ac53::1 Recursive Service dns.burble.dn42 172.20.129.2fd42:4242:2601:ac53::53 DNS64 Service dns64.burble.dn42 fd42:4242:2601:ac53::64 burble.dn42 provides a local, anycast, authoritative and recursive DNS service.
The DNS Service has it&rsquo;s own page.
Registry API Service and Explorer https://explorer.burble.com/ (public internet link) https://explorer.burble.dn42/ (DN42 link) dn42regsrv is a REST API for the DN42 registry that provides a bridge between interactive applications and the registry.
As well as the main REST API to the DN42 registry, the server can also generate ROA tables and provides a small web application for exploring registry data.
ROA Data Route Origin Authorisation (ROA) tables are generated using dn42regsrv and published to the dn42.burble.com website for general use.
The JSON output file can be used with gortr to implement ROA checks via RPKI.
The Bird files can be used directly with Bird to implement ROA checks as detailed in the DN42 Wiki (Bird1 / Bird2).
The OpenBGPD files can be used directly with OpenBGPD to implement ROA checks as details in the DN42 Wiki OpenBGPD
URL IPv4/IPv6 Description https://dn42.burble.com/roa/dn42_roa_46.json Both DN42 ROA data in JSON format https://dn42.burble.com/roa/dn42_roa_bird1_46.conf Both DN42 ROA data for use with Bird1 https://dn42.burble.com/roa/dn42_roa_bird1_4.conf IPv4 Only DN42 ROA data for use with Bird1 https://dn42.burble.com/roa/dn42_roa_bird1_6.conf IPv6 Only DN42 ROA data for use with Bird1 https://dn42.burble.com/roa/dn42_roa_bird2_46.conf Both DN42 ROA data for use with Bird2 https://dn42.burble.com/roa/dn42_roa_bird2_4.conf IPv4 Only DN42 ROA data for use with Bird2 https://dn42.burble.com/roa/dn42_roa_bird2_6.conf IPv6 Only DN42 ROA data for use with Bird2 https://dn42.burble.com/roa/dn42_roa_obgpd_46.conf Both DN42 ROA data for use with OpenBGPD https://dn42.burble.com/roa/dn42_roa_obgpd_4.conf IPv4 Only DN42 ROA data for use with OpenBGPD https://dn42.burble.com/roa/dn42_roa_obgpd_6.conf IPv6 Only DN42 ROA data for use with OpenBGPD ROA data is cached via Cloudflare to provide fast local access.
Git git.burble.dn42 (dn42 link) git.burble.com (public internet link) burble.dn42 related code and configuration is maintained in a local gitea repository.
VOIP voip.burble.dn42 burble.dn42 runs an asterisk based VOIP service, primarily to support the Retro42 service.
The VOIP service has a number of public endpoints available for testing:
SIP Endpoint Service 424026010@voip.burble.dn42 Time Announcement 424026011@voip.burble.dn42 Announces your caller ID 424026012@voip.burble.dn42 Echo Service 424026013@voip.burble.dn42 Randomised music 424026017@voip.burble.dn42 Dials Modem 1 directly (currently offline and used for testing) 424026018@voip.burble.dn42 Dials Modem 2 directly 424026019@voip.burble.dn42 Dialup service endpoint - dials both modems PrivateBin Instance paste.burble.dn42 (dn42 link) paste.burble.com (public internet link) burble.dn42 PrivateBin instance.
NTP Service All servers in burble.dn42 provide a stable, high stratum NTP service using chrony.
The NTP service is exposed over DN42, and users are welcome to use any server in the burble.dn42 network as an NTP time server on either the public or DN42 networks.
clicker.burble.dn42 https://clicker.burble.dn42/ Waste your time with this dn42 themed, browser based idle game.
lounge.burble.dn42 https://lounge.burble.dn42/ (dn42 link) https://lounge.burble.com/ (public internet link) thelounge is a web based IRC client that is integrated with the LDAP backend.
Log in using your burble.dn42 username/password to lurk on #dn42.
(set a password using the the burble.dn42 service portal)
Invidious instance https://invidious.burble.dn42/ burble.dn42 instance of Invidious the open source alternative front-end to YouTube.
`}),e.add({id:10,href:"/services/",title:"Services",parent:"burble.dn42",content:`Information about burble.dn42 services.
DN42: DN42 Infrastructure Services Public Services: List of public services Internal Services: Documentation for Non-public applications DNS: DNS services ACME: ACME Service Shell Accounts: burble.dn42 Shell Services S3 Object Storage: S3 Object Service Certificate Authority: burble.dn42 certificate authority Ping Challenge: The burble.dn42 Ping Challenge `}),e.add({id:11,href:"/retro/",title:"Retro42",parent:"burble.dn42",content:`dn42 like its the 1990&rsquo;s
Dialup Service burble.dn42 has a full dialup service using real physical modems served over VOIP.
Modem Emulator If you don&rsquo;t have a physical modem you can use the modem emulator to get a retro terminal experience or connect in to dn42 from your retro VMs.
`}),e.add({id:12,href:"/additional/",title:"Additional Info",parent:"burble.dn42",content:" Maint. Log Archive: A log of changes to the burble.dn42 network 2020 2019 2018 Other stuff: Other random stuff Things to do in DN42: Stuck for inspiration ? "}),e.add({id:13,href:"/services/internal/",title:"Internal Services",parent:"Services",content:`This page provides some documenation on other services used within burble.dn42 that are not directly available for public use.
traefik / traefik-eu / traefik-na burble.dn42 runs a global traefik cluster which acts as a reverse proxy and load balancer for burble.dn42 web services.
The traefik instances are anycast globally (traefik.burble.dn42), but also have regional load balancing groups for Europe (traefik-eu.burble.dn42) and North America (traefik-na.burble.dn42). This regional split helps to direct users to local services where possible.
vault.burble.dn42 Hashicorp Vault is used to handle secrets across the burble.dn42 network. Vault is deployed as a 3 node cluster across the Europe core nodes and uses the internal raft database as a back end.
TLS Certificate Authority Vault acts as the main certificate authority for burble.dn42 PKI.
Vault allows for regular, automated renewal of certificates on short timeframes (typically a rolling week or monthly basis).
SSH Certificate Authority Vault also acts as an SSH certificate authority, verifying both users and servers within the network.
Server certificates are generated during deployment, whilst user (or role) certificates are short lived and generated on demand.
Deployment Secrets Vault holds secrets used during node and service deployments.
Most burble.dn42 are built as stateless container images and secrets are pushed from vault in to the live containers at runtime. This ensures the container images do not contain secrets and that secrets can be applied per instance even when using a common image.
Vault also manages database credentials (using the mysql/mariadb integration), and these are also automatically generated and pushed in to container instances on deployment.
The authority to access deployment secrets is inherited, on demand, from the user token during the deployment process. This ensures that even if access was gained to the deployment server, secrets could still not be accessed without also having access to a live user token.
nomad.burble.dn42 burble.dn42 runs a global HashiCorp Nomad cluster that is used primarily for web application workloads. Nomad integrates with containerd, vault and traefik to provide resilient, globally available applications.
nats.burble.dn42 burble.dn42 operates a nats.io cluster as a distributed, network wide, broadcast and RPC solution.
The cluster uses decentralised JWT tokens for authentication.
ci.burble.dn42 The burble.dn42 git has an associated CI/CD service based on drone.
The CI/CD service is used to manage DNS, build and publish applications and the burble.dn42 website.
`}),e.add({id:14,href:"/network/IPAM/",title:"IPAM",parent:"Network",content:`IP address tables
General Ranges IPv4 IPv4 Address Range Purpose 172.20.129.0/27 burble.dn42 services 172.20.129.160/27 burble.dn42 nodes IPv6 IPv6 Address Range Purpose fd42:4242:2601:acXX::/64 Anycast services fd42:4242:2601:AA::/64 Public services for host AA fd42:4242:2601:AA00::/56 /56 routed to host AA fd42:4242:2601:AA02::/64 Tier2 services on host AA fd42:4242:2601:AA64::/64 NAT64 service related to host AA burble.dn42 Services DNS IPv4 IPv6 Comment 172.20.129.0 Reserved ns1.burble.dn42 172.20.129.1 fd42:4242:2601:ac53::1 Authoritative DNS Master dns.burble.dn42 172.20.129.2 fd42:4242:2601:ac53::53 Recursive DNS Resolver burble.dn42www.burble.dn42 172.20.129.3 fd42:4242:2601:ac80::1 Website collector.dn42 172.20.129.4 fd42:4242:2601:ac12::1 Global Route Collector pingable.burble.dn42 172.20.129.5 fd42:4242:2601:ac05::1 Pingable IP Address nats.burble.dn42 172.20.129.6 fd42:4242:2601:ac06::1 nats.io Cluster 172.20.129.7 Unallocated whois.burble.dn42 172.20.129.8 fd42:4242:2601:ac43::1 Whois service voip.burble.dn42 172.20.129.9 fd42:4242:2601:37:216:3eff:fe8f:6211 Asterisk VOIP Service shell.burble.dn42 172.20.129.10 fd42:4242:2601:ac22::1 Shell service 172.20.129.11 Unallocated traefik.burble.dn42 172.20.129.12 fd42:4242:2601:ac82::1 Global traefik cluster traefik-eu.burble.dn42 172.20.129.13 fd42:4242:2601:ac83::1 Europe traefik cluster traefik-na.burble.dn42 172.20.129.14 fd42:4242:2601:ac84::1 North America traefik cluster 172.20.129.15-19 Unallocated 172.20.129.20/30 n/a Dialup Service endpoints shell.de.burble.dn42 172.20.129.24 fd42:4242:2601:100b:216:3eff:fe32:1a21 DE shell service shell.uk.burble.dn42 172.20.129.25 fd42:4242:2601:1015:216:3eff:fe91:b943 UK shell service shell.nyc.burble.dn42 172.20.129.26 fd42:4242:2601:101d:216:3eff:fefc:722 NYC shell service shell.lax.burble.dn42 172.20.129.27 fd42:4242:2601:1018:216:3eff:feaa:7249 LAX shell service shell.fr.burble.dn42 172.20.129.28 fd42:4242:2601:1017:216:3eff:fe01:2f1f FR shell service 172.20.129.30 shell service unassigned 172.20.129.31 unassigned burble.dn42 Nodes (DN42 Addressing) DNS IPv4 IPv6 Comment unassigned 172.20.129.164 fd42:4242:2601:3f::1 us-nyc3.burble.dn42 172.20.129.165 fd42:4242:2601:3a::1 NYC Cluster uk-lon2.burble.dn42 172.20.129.166 fd42:4242:2601:2e::1 UK Storage unassigned 172.20.129.167 fd42:4242:2601:2d::1 nl-ams2.burble.dn42 172.20.129.168 fd42:4242:2601:34::1 Private Node de-fra1.burble.dn42 172.20.129.169 fd42:4242:2601:31::1 DE Public Peering de-fra3.burble.dn42 172.20.129.170 fd42:4242:2601:2c::1 DE Dev Cluster de-fra2.burble.dn42 172.20.129.171 fd42:4242:2601:2b::1 DE Cluster us-lax1.burble.dn42 172.20.129.172 fd42:4242:2601:2a::1 LAX Public Peering ch-zur2.burble.dn42 172.20.129.173 fd42:4242:2601:27::1 CH Cluster ch-zur1.burble.dn42 172.20.129.174 fd42:4242:2601:28::1 CH Public Peering us-nyc1.burble.dn42 172.20.129.175 fd42:4242:2601:29::1 NYC Public Peering us-nyc2.burble.dn42 172.20.129.176 fd42:4242:2601:3d::1 NYC Cluster uk-lon4.burble.dn42 172.20.129.177 fd42:4242:2601:25::1 UK Cluster uk-lon5.burble.dn42 172.20.129.178 fd42:4242:2601:24::1 UK Cluster de-fra4.burble.dn42 172.20.129.179 fd42:4242:2601:23::1 DE Dev Cluster us-lax2.burble.dn42 172.20.129.180 fd42:4242:2601:38::1 LAX Cluster de-nue1.burble.dn42 172.20.129.181 fd42:4242:2601:37::1 DE Cluster unassigned 172.20.129.182 fd42:4242:2601:3e::1 nl-ams3.burble.dn42 172.20.129.183 fd42:4242:2601:3c::1 Experimental uk-lon3.burble.dn42 172.20.129.184 fd42:4242:2601:30::1 Private Node no-trd1.burble.dn42 172.20.129.185 fd42:4242:2601:39::1 NO Public Peering nl-ams1.burble.dn42 172.20.129.186 fd42:4242:2601:32::1 Private Node uk-lon1.burble.dn42 172.20.129.187 fd42:4242:2601:35::1 UK Public Peering fr-par1.burble.dn42 172.20.129.188 fd42:4242:2601:36::1 FR Public Peering unassigned 172.20.129.189 fd42:4242:2601:26::1 reserved 172.20.129.190 fd42:4242:2601:20::1 Private Node reserved 172.20.129.191 fd42:4242:2601:20::1 Private Node burble.dn42 Nodes (Public Addressing) DNS IPv4 IPv6 dn42-fr-par1.burble.com 51.159.194.131 2001:bc8:1201:50d:d6ae:52ff:fecc:c97 dn42-uk-lon1.burble.com 45.91.93.104 2a0e:dc0:5:5::1 dn42-de-fra1.burble.com 5.180.253.211 2a0e:6a80:3:430::1 dn42-ch-zur1.burble.com 45.91.92.111 2a0e:dc0:6:8::1 dn42-no-trd1.burble.com 217.168.87.226 2001:678:dd0:ffff::25 dn42-us-lax1.burble.com 45.67.219.3 2a04:bdc7:100:4656::1 dn42-us-nyc1.burble.com 109.205.61.36 2605:4840:2:eed4::1 DNS IPv4 IPv6 git.dn42.dev 45.91.93.104 2a0e:dc0:5:5:425d:6eff:feb4:6e2d `}),e.add({id:15,href:"/network/routing-policy/",title:"Routing Policy",parent:"Network",content:`With a global network and multiple peers, the burble.dn42 network typically has many alternative route paths for reaching a particular destination. The routing policy aims to keep route selection sane, and avoid sending traffic outside of a region where possible.
Policy Objectives Direct routes for prefixes belonging to a peer should be given the highest priority. So that traffic to peer networks is routed over the burble.dn42 network directly to the peer and not via an external 3rd party Where prefixes are tagged with a DN42 region, they should be routed locally or within the burble.dn42 network. To avoid sending traffic across regions when this could have been avoided. The AS path length is also increased between regions to pursuade external routers to also prefer local hosts. Prioritise by shortest path, then lowest latency Policy Implementation bgp local_pref The local_pref for routes is set on entry, and then propogated across the whole network. This forces the network to prefer routes that, where possible, send traffic through the burble.dn42 network to a local peer, rather than sending cross regional traffic through external peers (aka Cold Potato Routing).
Local Pref Route Class 3000 burble.dn42 dynamic / anycast routes 2000 burble.dn42 internal networks 1000 Peer networks (AS path len = 1) 500 Route received in same DN42 region as it originated 100 Default bgp med The med attribute is used to implement a latency based metric across the network. Scripts are used to gather the latency between nodes (using ping) and this is then incorporated in to the ansible scripting that generates the peer configuration for the internal mesh. The peer configuration sets the med to be the latency in ms between nodes (in milliseconds * 10). A penalty of 500 is added for each hop to encourange direct routing between nodes.
med = (latency between nodes in ms * 10) + (500 per hop) The med metric is exported to external peers to help them decide how to route traffic to the burble.dn42 network.
collector.dn42 The global route collector is treated as a special case and does not follow the normal routing policy. The objective of the collector policy is to steer traffic directly to fr-par1 and reduce transit of GRC traffic over the burble.dn42 network. This is aggressively implemented for certain low bandwidth nodes to prevent any GRC traffic going through these nodes.
To accomplish this, each node announces the GRC prefix in to DN42 with different levels of path prepending depending on the available bandwidth of that node. Some nodes with low bandwidth allocations do not announce the GRC prefix at all.
Node capability Action Taken fr-par1 Prefix announced without any prepending Other nodes with unlimited BW allowance ASN prepended once Nodes with BW limits ASN prepended twice Nodes with low BW limits GRC prefix not announced As well as the external traffic engineering, the internal mesh policy also prevents transit of GRC traffic via low bandwidth nodes.
`}),e.add({id:16,href:"/network/communities/",title:"BGP Communities",parent:"Network",content:`This page describes the use of BGP communities within the network.
DN42 Communities The DN42 Route Origin Community is applied both internally and externally, and is used to influence the Routing Policy.
Community Description ( 64511 : 40 &lt; x &lt; 54 ) Route Origin The other DN42 communities are not used in burble.dn42:
Community Description ( 64511 : 0 &lt; x &lt; 21 ) Max latency ( 64511 : 20 &lt; x &lt; 30 ) Min bandwidth ( 64511 : 30 &lt; x &lt; 35 ) Min encryption Well Known BGP Communities The following well known communities are implemented.
Community Description Action ( 65535 : 65281 ) No Export Prefix should not be exported outside of AS4242422601 ( 65535 : 65282 ) No Advertise Prefix should not be exported to any peers ( 65535 : 65283 ) Local-AS Prefix should not be exported outside of region burble.dn42 Specific Communities burble.dn42 implements large BGP communities, with ISO 3166-1 / UNSD country, and UNSD region codes.
Informational Communities Community Description ( 4242422601 : 120 : host code ) Route learned on this host ( 4242422601 : 130 : 1 ) Route is a direct peer ( 4242422601 : 140 : DN42 region ) Route learned in this DN42 region `}),e.add({id:17,href:"/services/dns/",title:"DNS",parent:"Services",content:`burble.dn42 provides a suite of DNS services, including running one of the two DN42 DNS master nodes that exports registry information to the DNS infrastructure.
Role Names DN42 DNS Master b.master.delegation-servers.dn42 Authoritative DNS Service b.delegation-servers.dn42
ns1.burble.dn42 Recursive DNS Service b.recursive-servers.dn42dns.burble.dn42 dns64 Service dns64.burble.dn42 Apart from the Master, all DNS services are anycast across every node to provide fast, local responses network wide. The services support DNSSEC and are available over UDP, TCP, DNS over HTTPs and DNS over TLS.
DN42 DNS Master Name IP b.master.delegation-servers.dn42 fd42:180:3de0:30::1 burble.dn42 runs one of the two master servers that support the DN42 DNS infrastructure.
See the wiki for more information on the role of the master service.
The master is hosted on ca-bhs2, providing geographic and network redundancy against the other DN42 master service, hosted in Europe.
Authoritative DNS Service Name IP ns1.burble.dn42b.delegation-servers.dn42 172.20.129.1fd42:4242:2601:ac53::1 ns1.burble.dn42 is slaved to master.delegation-servers.dn42, and provides DNSSEC signed, authoritative data for DN42 related zones.
The authoritative service may be used as the root for a local DNS resolver, with the assurance that returned DNS records are traceable via DNSSEC to the DN42 registry. The service also supports AXFR and may be used as a master to a local, slaved, root zone.
Note that ns1.burble.dn42 will not forward DNS queries.
Forwarding is provided by the recursive service, dns.burble.dn42.
Slaved DN42 zones .dn42 .recursive-servers.dn42 .delegation-servers.dn42 .registry-sync.dn42 d.f.ip6.arpa. 20.172.in-addr.arpa. 21.172.in-addr.arpa. 22.172.in-addr.arpa. 23.172.in-addr.arpa. 31.172.in-addr.arpa. 10.in-addr.arpa. Mastered Zones Zone Role burble.dn42 burble.dn42 forward zone collector.dn42 Global Route Collector forward zone 1.0.6.2.2.4.2.4.2.4.d.f.ip6.arpa burble.dn42 IPv6 reverse zone 0/27.129.20.172.in-addr.arpa burble.dn42 services IPv4 reverse zone 160/27.129.20.172.in-addr.arpa burble.dn42 nodes IPv4 reverse zone 0.3.0.0.0.e.d.3.0.8.1.0.2.4.d.f.ip6.arpa DNS Master reverse zone 0.0.1.0.0.e.d.3.0.8.1.0.2.4.d.f.ip6.arpa Registry services IPv6 reverse zone 0/28.63.22.172.in-addr.arpa Register services, IPv4 reverse zone Recursive DNS Service Name IP dns.burble.dn42b.recursive-servers.dn42 172.20.129.2fd42:4242:2601:ac53::53 dns.burble.dn42 is a caching, recursive DNS service that returns results for both DN42 and clearnet domains. The service issues parallel queries from regional masters, the recursive service takes advantage of the burble.dn42 global scale to reduce latency and avoid local connectivity problems.
The recursor is DNSSEC enabled and validates all queries.
Using the recursive DNS service Users are encouraged to consult recursive-servers.dn42 to obtain a list of recursive DNS services and configure at least two independent resolvers to obtain the best resilience.
See also the DN42 Wiki for general guidelines and best practice for setting up DNS in DN42.
$ host -t SRV _dns._udp.recursive-servers.dn42 _dns._udp.recursive-servers.dn42 has SRV record 10 10 53 a3.recursive-servers.dn42. _dns._udp.recursive-servers.dn42 has SRV record 20 10 53 b.recursive-servers.dn42. _dns._udp.recursive-servers.dn42 has SRV record 10 10 53 a0.recursive-servers.dn42. _dns._udp.recursive-servers.dn42 has SRV record 20 10 53 j.recursive-servers.dn42. _dns._udp.recursive-servers.dn42 has SRV record 20 10 53 k.recursive-servers.dn42. Example resolv.conf using IPv6 with IPv4 fallback
# DN42 resolve.conf search dn42 # burble.dn42 service # b.recursive-servers.dn42 nameserver fd42:4242:2601:ac53::53 # j.recursive-servers.dn42 nameserver 172.20.1.19 DNS64 Service Name IP dns64.burble.dn42 fd42:4242:2601:ac53::64 The dns64 service operates in a similar way to the main recursive service but also provides dns64 translation for hostnames that only have IPv4 addresses.
The service will return IPv4 mapped to the rfc6052 well-known prefix - 64:ff9b::/96
DNS over HTTPS (DoH) DNS over TLS The burble.dn42 services support queries via DNS over HTTPS (on port 443) and DNS over TLS (on port 843). The HTTPS service is signed by the burble.dn42 Certificate Authority, and the CA certificate will be required by the client in order to use the service.
example
$ doh burble.dn42 https://[fd42:4242:2601:ac53::53]/dns-query burble.dn42 from https://[fd42:4242:2601:ac53::53]/dns-query TTL: 3600 seconds A: 172.20.129.3 AAAA: fd42:4242:2601:ac80:0000:0000:0000:0001 Implementation The DNS service is implemented as a tiered, anycast service with each node in the network providing a local cache in front of regional, slave nodes.
dns-edge Edge nodes provide a caching function for the slaves.
Recursive services (dns.burble.dn42 and dns64.burble.dn42) are provided by dnsmasq configured using the &lsquo;all-servers&rsquo; mode. DN42 queries are forwarded to all regional slaves in parallel and the first response received is then returned. This approach ensures users get the lowest latency results possible, regardless of location, and that any local connectivity issues do not impact the results.
The authoritive service as well as DNS over HTTPS and DNS over TLS services are provided by dnsdist acting as a proxy. Requests are forwarded to either the regional slaves or local recursor services as appropriate and also cached.
Clearnet queries are forwarded on the edge nodes to a combination of Google and Cloudflare services.
The edge services are monitored and anycast routes automatically injected (or removed) with a health checking script.
dns-slave Region Host Location Europe dns-slave.de-fra1.burble.dn42 PHP Friends, Frankfurt, Germany Americas (East) dns-slave.ca-bhs2.burble.dn42 OVH, Beauharnois, Canada Americas (West) dns-slave.us-lax1.burble.dn42 Alvin Servers, Los Angeles, USA The slave nodes are implemented using PowerDNS.
The Authoritative DNS servers are configured as slaves replicating from the DN42 master for .dn42 related zones and a hidden master located on the private, internal network for burble.dn42 zones.
The recursive service is provided by the pdns-recursor configured with DNSSEC validation and additional caching.
dns-master The DN42 DNS master is a custom java program running on us-dal3.
`}),e.add({id:18,href:"/services/acme/",title:"ACME",parent:"Services",content:`burble.dn42 provides an ACME service using an intermediate certificate issued by the dn42 certificate authority and implemented using a HashiCorp Vault cluster to provide a highly available service.
The following ACME challenge types are supported:
http-01 dns-01 tls-alpn-01 dn42 endpoint https://acme.burble.dn42/v1/dn42/acme/directory The dn42 endpoint serves certificates signed by an intermediate certificate issued by the dn42 certificate authority.
Note that certificates are issued with a validity period of 30 days, which is shorter than most clearnet ACME services.
The recommended interval to check for expiry is 5 days.
Staging endpoint https://acme.burble.dn42/v1/staging/acme/directory The staging endpoint can be used for testing and issues junk certificates. The service uses an internal certificate authority that is specific to the staging service and should not be trusted.
The staging service issues short lived certificates with a validity period of a few days.
Certificate Transparency TODO A simpler process will be provided at a future stage, in the meantime the vault API can be queried manually to list issued certificates.
&ndash;
Vault provides an API for listing issued certificates, however the process for doing this is somewhat complicated if you have not used vault before. The instructions below detail how to interrogate the service using the vault CLI, however it is also possible to run through the same process via the HTTP API.
# The API endpoint to list issued certificates is an authenticated # endpoint that requires a vault token to access it. # # The burble.dn42 service includes an anonymous login that can be # used to obtain a suitable token. # set the VAULT_ADDR environment variable to the ACME service $ export VAULT_ADDR=&#34;https://acme.burble.dn42&#34; # you can also set VAULT_SKIP_VERIFY=1 if you do not have the # dn42 certificate authority installed. # Issue an anonymous token and store it in the VAULT_TOKEN env variable $ export VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=anonymous) # now the vault API can be accessed # list issued certificates $ vault list dn42/certs Keys ---- 06:72:54:74:02:eb:68:da:62:76:14:92:b4:84:19:36:b1:d1:d0:5c 0c:bb:39:a0:0a:aa:9c:d9:06:e8:9e:87:ff:54:73:c4:a6:42:9c:f0 13:91:4f:f7:3a:0b:ca:38:cd:c6:6e:7d:4d:fb:c5:7c:ed:b0:79:1b 39:5c:46:16:27:d8:f7:30:cc:64:1a:3c:6c:ff:c4:ac:f9:3c:3c:9c 4b:24:32:48:d0:64:55:3b:dd:b3:00:c6:33:2d:0f:3e:eb:d7:50:02 4c:8f:ce:e6:18:7a:05:c1:a3:11:45:c9:3c:34:0f:50:e0:75:6d:fd 5a:03:a9:5b:07:60:d0:fb:25:28:4b:e9:93:a8:22:cd:78:d1:29:b2 5d:26:b4:47:59:0c:0a:e9:88:b6:97:1d:2a:2b:e5:cb:d2:90:34:9e 65:c8:33:07:fc:9a:aa:fd:85:6b:fd:b4:de:29:71:e3:8e:6c:f2:11 68:e1:a6:4a:e1:58:ee:71:c7:a6:12:48:e2:7a:c5:84:c1:7c:21:5e 75:cf:16:f9:06:71:ea:86:1c:51:95:89:c9:1d:ea:a1:eb:f5:6f:83 76:91:6e:6a:23:14:00:7c:5f:c7:de:91:c4:40:73:d9:51:b4:f8:4d # view an invidual certificate $ vault read -field certificate &#34;dn42/cert/76:91:6e:6a:23:14:00:7c:5f:c7:de:91:c4:40:73:d9:51:b4:f8:4d&#34; -----BEGIN CERTIFICATE----- MIIDTTCCAjWgAwIBAgIUdpFuaiMUAHxfx96RxEBz2VG0+E0wDQYJKoZIhvcNAQEL BQAwVTELMAkGA1UEBhMCWEQxDTALBgNVBAoTBGRuNDIxFDASBgNVBAsTC2J1cmJs ...snip... yait1CFFq4g9/bvsNfIsvN6EJ/BGXqqww6BzKt/ioSLj -----END CERTIFICATE----- # human readable output using the step CLI (https://smallstep.com/) $ vault read -field certificate &#34;dn42/cert/06:72:54:74:02:eb:68:da:62:76:14:92:b4:84:19:36:b1:d1:d0:5c&#34; | step certificate inspect Certificate: Data: Version: 3 (0x2) Serial Number: 36803586486229131299250018793512622456839458908 (0x672547402eb68da62761492b4841936b1d1d05c) Signature Algorithm: SHA256-RSA Issuer: C=XD,O=dn42,OU=burble.dn42,CN=burble.dn42 staging ACME Validity Not Before: Oct 2 18:21:36 2023 UTC Not After : Nov 3 18:22:06 2023 UTC Subject: CN=drone.git.dn42 Subject Public Key Info: Public Key Algorithm: RSA Public-Key: (4096 bit) Modulus: ...snip... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Key Agreement X509v3 Extended Key Usage: Server Authentication X509v3 Subject Key Identifier: 01:4A:7E:02:F3:B7:78:03:66:F9:21:97:4B:31:34:7C:31:DE:BB:86 X509v3 Authority Key Identifier: keyid:94:D1:C3:60:C7:88:81:A6:8C:37:AE:40:42:22:48:6B:5F:36:8F:CC Authority Information Access: OCSP - URI:https://acme.burble.dn42/v1/dn42/ocsp CA Issuers - URI:https://acme.burble.dn42/v1/dn42/ca X509v3 Subject Alternative Name: DNS:drone.git.dn42 X509v3 CRL Distribution Points: Full Name: URI:https://acme.burble.dn42/v1/dn42/crl Signature Algorithm: SHA256-RSA ...snip... Implementation The ACME implementation is provided by a 3-node HashiCorp Vault cluster behind the burble.dn42 traefik load balancer. Together they provide a global, high availability service.
The cluster currently runs on the following nodes:
ch-zur2 de-fra1 fr-par1 At any time the cluster has one leader which processes all requests and replicates state to the cluster members. The leader node automatically switches to one of the backup servers should a failure occur.
The traefik load balancer runs health checks against the vault servers and automatically redirects users to the vault cluster leader.
See the vault HA reference architecture for more details.
`}),e.add({id:19,href:"/services/shell/",title:"Shell Accounts",parent:"Services",content:`burble.dn42 provides shell accounts on the following servers:
shell.fr.burble.dn42 shell.uk.burble.dn42 shell.de.burble.dn42 shell.lax.burble.dn42 shell.nyc.burble.dn42 There is also an anycast address shell.burble.dn42 that will route to the closest server.
Accessing the Service The shell service imports user information from the dn42 registry allowing any MNTNER to log in to the servers. Usernames are constructed by lowercasing and removing the &lsquo;-MNT&rsquo; suffix.
Using an SSH public key If you have an auth attribute with an SSH public key, this will be imported from the registry and you can use the SSH key to log in to the shell server.
Using a password MNTNERs without an SSH key must first use the burble.dn42 service portal to set an account password.
Connection Example For mntner FOO-MNT
Log in to the closest server using your ssh key or burble.dn42 password:
ssh foo@shell.burble.dn42 or log in to a specific server:
ssh foo@shell.fr.burble.dn42 Your home directory is created automatically on first access and will then persist across logins.
Home directories are not replicated across servers.
Key Services Installed Packages A broad set of command line tools, applications and games are available; the aim is to provide a comprehensive environment that is useful and fun to tinker with.
The current set of packages can be found in the shell config repo:
https://git.dn42.dev/burble/config-shell/src/branch/main/roles/user_apps/tasks/main.yml Requests for additional packages are welcome, please raise these as issues in the repo.
Webserver The shell servers include a webserver with user directories (~/public_html/) and CGI (~/public_html/cgi-bin/) enabled. The webserver is accessed over https and has a dn42 certificate.
https://shell.fr.burble.dn42/~&lt;username&gt;/ https://shell.uk.burble.dn42/~&lt;username&gt;/ https://shell.de.burble.dn42/~&lt;username&gt;/ https://shell.lax.burble.dn42/~&lt;username&gt;/ https://shell.nyc.burble.dn42/~&lt;username&gt;/ https://shell.burble.dn42/~&lt;username&gt;/ Remember that any files need to be accessible by the webserver, which runs as user/group www-data/www-data; CGI scripts must also be executable. The default umask of 077 means that the webserver won&rsquo;t be able to read your public_html folder or any files within it without changing permissions.
A simplistic approach would be to make your home directory, and public_html directory world readable:
chmod a+x ~ chmod -R a+rX ~/public_html chmod -R a+rx ~/public_html/cgi-bin A more secure way of allowing access would be to use posix ACLs:
setfacl -m &#34;u:www-data:x&#34; ~ setfacl -Rdm &#34;u:www-data:rX&#34; ~/public_html setfacl -Rdm &#34;u:www-data:rx&#34; ~/public_html/cgi-bin Note also that home directories are not replicated across each shell server.
If you want to provide services using the anycast address you must copy your code between servers yourself.
Login Shell You can change your login shell using the burble.dn42 service portal.
Classic Games The shells have a number of classic text games installed:
Colossal Cave Adventure
Get lost in a twisty little maze of passages
$ adventure
Trek
$ trek
NetHack
The original time sink
$ nethack
Zork
Zork 1, 2 and 3 are available in /usr/local/frotz/
$ frotz /usr/local/frotz/ZORK1.DAT
$ frotz /usr/local/frotz/ZORK2.DAT
$ frotz /usr/local/frotz/ZORK3.DAT
Take a look in /usr/games for more text games.
Cron, Batch and Services Cron and other batch or long running tasks are ok, but be a nice neighbour and prioritise other users&rsquo; interactive use.
Schedule crons to run at random or obscure times to avoid stampeding herds and control your resource usage using tools like nice and ionice.
Clearnet Clearnet access is provided. Rate limiting allows for a small amount of burst traffic, but then bandwidth is quickly limited to 10mbit/sec. In general, you should be better off using your own clearnet access for large downloads.
Connection Forwarding SSH forwarding is enabled on the servers.
For example, this means you are able to use the shell servers as a resilient, anycast jump host:
ssh -J shell.burble.dn42 my.other.host.dn42 There are also a small number of X11 apps installed on the servers:
ssh -X shell.burble.dn42 -f &#39;xterm &amp; xeyes&#39; Integration with S3 object storage The shell servers include rclone which can be used to access the S3 object storage service.
See the S3 storage example for details.
Acceptable Use In general, as long as you are not risking the service or other users you should be ok. These services are provided free for your benefit, and the objective is to provide a fun, open environment for dn42 users. Please be considerate in your usage and remember that abusing the service just spoils it for everyone else.
See also the main Abuse Policy.
Source Code and Configuration Configuration for the shell servers is maintained in a git repo:
https://git.dn42.dev/burble/config-shell The repository may be used for raising issues or requesting additional software to be installed.
`}),e.add({id:20,href:"/services/minio/",title:"S3 Object Storage",parent:"Services",content:`burble.dn42 provides an S3 compatible storage service based on min.io.
https://minio.burble.dn42 - Web interface s3.burble.dn42 - S3 compatible interface Remember that the storage service is provided for the dn42 community and the burble.dn42 abuse policy applies.
Be considerate in your usage so that others will get the same benefits as you are enjoying now. Clean up files that you no longer need, do not use the service for illegal or objectional content and don&rsquo;t be stupid.
Accessing the Service The object storage service is available for all users who have set a burble.dn42 password via the service portal.
Web Interface https://minio.burble.dn42 The minio web console can be accessed by logging in using your burble.dn42 username (lowercase of your MNTNER without the -MNT suffix) and password.
From the web interface users can create your own buckets, upload or download files, and create access keys for the S3 service.
Any buckets you create must start with your lowercase MNTNER name followed by a dash.
For example, for FOOBAR-MNT buckets must start with &lsquo;foobar-&rsquo;
Valid bucket names
foobar-mybucket foobar-bucket1 Invalid bucket names
foobar foobarmnt-bucket notfoobar-bucket S3 API s3.burble.dn42 An S3 compatible API is accessible via https://s3.burble.dn42. The S3 region is uk-lon2
Don&rsquo;t use your burble.dn42 username and password.
Use the web interface on https://minio.burble.dn42 to create a unique access and secret key when using the S3 API. SFTP sftp -P 8022 @minio.burble.dn42 The minio instance can also be accessed via SFTP on port 8022. Log in to the SFTP server using your burble.dn42 username (MNTNER name in lowercase) and password.
Example usage Using the minio client utility Install the minio client Log in to the web console at https://minio.burble.dn42 using your burble.dn42 username and password Create yourself a bucket, remember to start the bucket with your mntner e.g. &lsquo;foobar-&rsquo; Create an acccess and secret key pair Log out the web interface and return back to the cli Create an alias in the client using the access and secret key pair generated in step 4 mc alias set burble https://s3.burble.dn42 ACCESS_KEY SECRET KEY You can now use the client to access your bucket mc ls burble/foobar-mybucket Using the S3 API to mount buckets using rclone The burble.dn42 shell servers include rclone which you can use to mount buckets directly in your shell account.
Log in to the web console at https://minio.burble.dn42 using your burble.dn42 username and password Create yourself a bucket, remember to start the bucket with your mntner e.g. &lsquo;foobar-&rsquo; Create an acccess and secret key pair Log out the web interface and and log in to your shell account Use rclone config to configure rclone Use https://s3.burble.dn42 as the endpoint and uk-lon2 as the region Use the access and secret key pair created in step 3 for authentication You can now use rclone to mount your buckets in the shell
$ mkdir tmp $ rclone mount burble: tmp Using SFTP Simply use your burble.dn42 username and password to access the SFTP server on port 8022.
$ sftp -P 8022 burble@minio.burble.dn42 burble@minio.burble.dn42&#39;s password: Connected to minio.burble.dn42. sftp&gt; dir burble-images sftp&gt; exit `}),e.add({id:21,href:"/services/ca/",title:"Certificate Authority",parent:"Services",content:`burble.dn42 maintains a PKI infarstructure for its services, using Hashicorp Vault
CA details countryName GB stateOrProvinceName dn42 organizationName burble.dn2 commonName ca.burble.dn42 emailAddress dn42@burble.com CA Download burble-dn42-ca.pem
-----BEGIN CERTIFICATE----- MIIDtzCCAp+gAwIBAgIUBIkK5f6OppmInBEKnG0xiNTn7lIwDQYJKoZIhvcNAQEL BQAwazELMAkGA1UEBhMCR0IxDTALBgNVBAgMBGRuNDIxFDASBgNVBAoMC2J1cmJs ZS5kbjQyMRcwFQYDVQQDDA5jYS5idXJibGUuZG40MjEeMBwGCSqGSIb3DQEJARYP ZG40MkBidXJibGUuY29tMB4XDTE5MDgxMzEwMDg0OVoXDTI5MDUxMjEwMDg0OVow azELMAkGA1UEBhMCR0IxDTALBgNVBAgMBGRuNDIxFDASBgNVBAoMC2J1cmJsZS5k bjQyMRcwFQYDVQQDDA5jYS5idXJibGUuZG40MjEeMBwGCSqGSIb3DQEJARYPZG40 MkBidXJibGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsSxS bQq27BmOsx5aX/G3x/lcIt0N0ECb0pZ8laX1/DFMWTBgQxpcwuOGyagsq7Jd6Ozt fvZUAD8K3P0q6JCX+164Yq+3yA1urchdUf0rFby6JJYkhSSVnzTWouNRfiTNYmUK C6Dtd3ZDQux6+8JAxI1rYGV+HqJ4y2N5A1pGGjLLfESfu38I2SflT5tUVNRgTfV9 ERbIQl7Zq9fYoizLnkbGWRSY5lk8Cwyz1q342Z8NazE5glJgE54uzLMmcdZfSD5f 2wF/XgIM8vcpXGo10aU8ZiYlevxDls1S/p2IdIZ3idb9+38hEY+mBenDNxA1Ad+d 5lQZIL0v7QHgtHiyqQIDAQABo1MwUTAdBgNVHQ4EFgQU3z7rCRdMpqOyh7MLWfO8 F75hmxwwHwYDVR0jBBgwFoAU3z7rCRdMpqOyh7MLWfO8F75hmxwwDwYDVR0TAQH/ BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAm2hdaRx7CESeeWpvQIUmWysYWZ5r tLsDkU0PdM2vQSB42MbngKNGi2GU1CXgQhNiYDieoIMhtfMf1Zl6PBm13XE11COg HKCbs828G5X6HrlCnVAfqQiaa0HLZ0vFO94RVknn/wqr2VoHKNLdA46gMD67FWGm KGyeD480JbIOiM69r/4JGzuJSh384CxH6KPCX7dUywxgI9zbW99SaKTQNJ8Z+O04 qPtmh+qW0L8a7lTuaR/SEbloDA+ztDYyEPsgmVpvM+e2PtJdmygGaUc8Y15TevJR PGauY9oPLMXdxRYsqjQeKjBXv7Ms0CQB8XNHQ7zPsM83EZD5Eq68wLIvUA== -----END CERTIFICATE----- Certificate Expiry Date: May 12 10:08:49 2029 GMT `}),e.add({id:22,href:"/services/ping/",title:"Ping Challenge",parent:"Services",content:`Can you solve the burble.dn42 ping challenge ?
To start $ ping -s 0 -c 1 fd42:4242:2601:31f0::1 Hints Use -c 1 to only return one ping, unless instructed otherwise Use -s 0 to create a zero sized ping, unless instructed otherwise Packet capture is your friend You may also find a CyberChef useful The server maintains state based on your source IP and will time out idle clients after several days. There are ways to check the status (revealed during the challenge) but if your state times out you will need to restart from the nearest checkpoint. `}),e.add({id:23,href:"/network/status/",title:"Realtime Status",parent:"Network",content:`Uptime Kuma https://uptime.burble.dn42/status/bdn42 (dn42) https://uptime.burble.com/status/bdn42 (com) A self-hosted instance of Uptime Kuma provides the current status on many of the burble.dn42 services.
Uptime Robot https://stats.uptimerobot.com/l2913c0R6 (public internet link) Major nodes are also monitored off-network using UptimeRobot.
Grafana Dashboards https://grafana.burble.com (public internet link) https://grafana.burble.dn42 (DN42 link) Netdata and Prometheus are used to monitor the network and stats are presented using a series of Grafana dashboards.
BGP and Peering Status https://lg.burble.com/ (public internet link) https://lg.burble.dn42/ (DN42 link) BGP status can be found through my looking glass.
`}),e.add({id:24,href:"/network/abuse/",title:"Abuse Policy",parent:"Network",content:`The burble.dn42 network has a zero tolerance policy on network abuse, both within burble.dn42 and across the wider DN42 network.
Abuse could include, but is not limited to:
Excessive use of resources Hacking, viruses, trojans etc or any other disruption that could harm or create risk to the services or its users Distribution of objectional content that could create a civil or criminal liability The burble.dn42 network owner is the sole arbiter for determining what is deemed abuse and for any mitigations taken.
The owner may be contacted via dn42@burble.com.
Complaints Please direct any reports of abuse originating from the burble.dn42 network to the network owner: dn42@burble.com.
Abuse of DN42 and burble.dn42 resources Remember that burble.dn42 network services (which includes peering and transit) cost real time and money to deliver, and yet are provided free so you may benefit from them. You have no rights to access the services, so be considerate in your usage such that other users can continue to get the same opportunities as you have.
Whilst the aim is to provide an open and neutral network, any actions that could be considered a risk or a threat to services, either directly against burble.dn42 or within the rest of the DN42 network may result in your use being unconditionally blocked, without warning, explanation or appeal. This could include blocking transit traffic, or removing access to the wider DN42 network, if deemed necessary.
`}),e.add({id:25,href:"/",title:"burble.dn42",parent:"",content:`An experiment in global networking.
`}),e.add({id:26,href:"/privacy/",title:"Privacy Policy",parent:"burble.dn42",content:`In common with most websites, the burble.dn42 site and associated services may log any access you make and these logs contain your source IP address together with the page or service being accessed. If you are required to log in to access a burble.dn42 service, you should assume that the user id used for the service is also logged. Website and service logs are accessible only by the network administrators and used purely for diagnostic reasons and to prevent abuse. They are not shared in any way. Log retention varies depending on the service, but is at most, 1 month.
burble.dn42 services are provided by servers operating globally. Data processing may take place in any country where the network has a pop or presence.
The services provided by burble.dn42 make use of data contained within the DN42 Registry. This data may contain personal data that has been provided voluntarily by users of DN42 and which is then made public by this website or associated services. Please refer to the DN42 registry privacy policy for more information.
If you have any data privacy concerns or requests regarding burble.dn42 services you may contact dn42@burble.com.
`}),e.add({id:27,href:"/categories/",title:"Categories",parent:"burble.dn42",content:""}),e.add({id:28,href:"/additional/maintlog/",title:"Maint. Log Archive",parent:"Additional Info",content:`A log of changes to the burble.dn42 network.
6th December 2021 As you may have noticed, the maintenance log is no longer being kept up to date !
17th July 2021 A new dedicated server us-nyc1 has joined burble.dn42, hosted in Virmach&rsquo;s Buffalo, NY datacentre.
The server is not open for peering, however it does host a shell server that you are free to use. See the Shell Accounts service page for more info.
27th June 2021 The shell servers now have apache2 installed to provide home directory public_html access.
See the Shell Accounts service page for more info.
23rd June 2021 Added shell.ca-bhs2.burble.dn42 as a new shell server.
21st June 2021 Added new Shell Accounts service
19th May 2021 dn42regsrv now supports publishing ROA in OpenBGPD format.
The ROA are now published at the following links:
https://dn42.burble.com/roa/dn42_roa_obgpd_46.conf https://dn42.burble.com/roa/dn42_roa_obgpd_4.conf https://dn42.burble.com/roa/dn42_roa_obgpd_6.conf us-nyc1 and us-chi1 have been removed from service and decommissioned.
If you were peered on these nodes, please contact me if you wish to re-peer on a different node. 10th April 2021 The b.recursive-servers.dn42 DNS resovler is running an experimental build of pdns-recursor to test a fix of this issue.
Please let me know if you spot any strange problems.
burble.dn42 websites are now using a TLS certificate issued by the DN42 ACME service.
A number of significant changes have been implemented for the global route collector
Downstream peerings have been stopped, in favour of parsing the MRT dumps The collector has moved from de-fra1 to fr-rbx1, where bandwidth is no issue A special routing policy has been implemented for the collector to encourage traffic to go directly to fr-rbx1 and not transit through burble.dn42 nodes. See also the Routing Policy page. Internal rate limits on BGP sessions have been relaxed 3rd April 2021 The collector is now using a TLS certificate issued by the DN42 ACME service. The collector is behind an anycasted reverse proxy, so a normal ACME challenge will not work. Instead, the certificate is managed using dnscontrol to respond to an ACME DNS challenge.
DNSSEC has been enabled on all edge nodes.
2nd April 2021 There was a major DNS outage today as a minor change took out the entire service.
What should have been a trivial config change actually upgraded the container from Alpine 3.11 to Alpine 3.13 and caused a number of the DNS applications to stop working due to incompatibilities.
The lack of working DNS meant it was more complicated to bootstrap the service back again, leading to a long delay in restoring service.
27th March 2021 Fixed a bug in bird that was preventing MRT dumps from the collector working. Hopefully the dumps can now be successfully parsed: https://mrt.collector.dn42
25th March 2021 Bird 2.0.8 has been deployed across the network. Please let me know if you see problems.
burble.dn42 uses a custom bird build that includes additional debugging. The source code for the build is available on git.burble.dn42.
Advanced Notice
us-nyc1 will be decommissioned before 15/04/21 us-chi1 will be decommissioned before 14/05/21 23rd February 2021 Updated IPv6 address for hk-hkg1
10th January 2021 Upgraded the looking glass to use bird-lg-go.
The main benefit of the go version is that it executes queries in parallel, greatly improving response times with a large number of nodes.
6th January 2021 hk-hkg1 is now open for IPv4 peering; see the node information for details.
IPv6 connectivity is expected ~February.
4th January 2021 Happy New Year DN42.
New Website The new year brings a new website for burble.dn42 built using Hugo and statically delivered from each core node for speed. As always, the source for the website is available in the gitea repo.
Anycast MTU The MTU for anycast services has been reduced to 1280 after a problem was seen with IPv6 path MTU discovery.
The problem was due to an asymmetric path, where a request to the wiki went to one node but the return path was via a different node. The other node also hosted a wiki instance, which meant that pmtud ICMP messages on the return path were being picked up by the wrong node. To fix this, the MTU has been clamped to the minimum allowable size of 1280.
Interestingly, Cloudflare also recognised the same type of issue and wrote up what they did in their blog.
The following services were impacted by the changes.
DNS Services NGINX Reverse Proxy (and therefore also all websites, including the Wiki mirrors) WHOIS Service New Nodes es-mad1 in Madrid, Spain has already been delivered and is now open for peerings.
The new node in Hong Kong, hk-hkg1 has also been delivered and I&rsquo;m now just waiting for IPv6 to be available before it too will also be ready for peering.
Historical changes from previous years 2020 2019 2018 `}),e.add({id:29,href:"/network/design/",title:"Network Design",parent:"Network",content:`This page documents some key elements of the current burble.dn42 design.
Tunnel Mesh Hosts within the burble.dn42 network are joined using an Wireguard/L2TP mesh. Static, unmanaged, L2TP tunnels operate at the IP level and are configured to create a full mesh between nodes. Wireguard is used to provide encryption and encapsulate L2TP traffic in plain UDP such that it hides fragmentation and allows packets to be processed within intermediate routers&rsquo; fast path.
Using L2TP allows for a large, virtual MTU of 4310 between nodes; this is chosen to spread the encapsulation costs of higher layers across packets. L2TP also allows for multiple tunnels between hosts and this can also be used to separate low level traffic without incurring the additional overheads of VXLANs (e.g. for NFS cross mounting).
Network configuration on hosts is managed by systemd-networkd and applied with Ansible.
Real Life Networks and Fragmentation.
Earlier designs for the burble.dn42 relied on passing fragmented packets directly down to the clearnet layer (e.g. via ESP IPsec fragementation, or UDP fragmentation with wireguard). In practice it was observed that clearnet ISPs could struggle with uncommon packet types, with packet loss seen particularly in the IPv6 case. It seems likely that some providers&rsquo; anti-DDOS and load balancing platforms had a particular impact at magnifying this problem.
To resolve this, the network was re-designed to ensure fragmentation took place at the L2TP layer such that all traffic gets encapsulated in to standard sized UDP packets. This design ensures all traffic is &rsquo;normal&rsquo; and can remain within intermediate routers' fast path.
ISP Rate Limiting
The burble.dn42 network uses jumbo sized packets that are fragemented by L2TP before being encapsulated by wireguard. This means a single packet in the overlay layers can generate multiple wireguard UDP packets in quick succession, appearing to be a high bandwidth, burst of traffic on the outgoing clearnet interface. It&rsquo;s vital that all these packets arrive at the destination, or the entire overlay packet will be corrupted. For most networks this is not a problem and generally the approach works very well.
However, if you have bandwidth limits with your ISP (e.g. a 100mbit bandwidth allowance provided on a 1gbit port) packets may be generated at a high bit rate and then decimated by the ISP to match the bandwidth allowance. This would normally be fine, but if a fragmented packet is sent, the burst of smaller packets is highly likely to exceed the bandwidth allowance and the impact on upper layer traffic is brutal, causing nearly all packets to get dropped.
The burble.dn42 network manages this issue by implementing traffic shaping on the outgoing traffic using linux tc (via FireQOS). This allows outgoing packets to be queued at the correct rate, rather than being arbitrarily decimated by the ISP.
BGP EVPN Overlaying the Wireguard/L2TP mesh is a set of VXLANs managed by a BGP EVPN.
The VXLANs are primarily designed to tag and isolate transit traffic, making their use similar to MPLS.
The Babel routing protocol is used to discover loopback addresses between nodes; Babel is configured to operate across the point to point L2TP tunnels and with a static, latency based metric that is applied during deployment.
The BGP EVPN uses FRR with two global route reflectors located on different continents, for redundency. Once overheads are taken in to account the MTU within each VXLAN is 4260.
dn42 Core Routing Each host in the network runs an unprivileged LXD container that acts as a dn42 router for that host. The container uses Bird2 and routes between dn42 peer tunnels, local services on the same node and transit to the rest of the burble.dn42 network via a single dn42 core VXLAN.
Local services and peer networks are fully dual stack IPv4/IPv6 however the transit VXLAN uses purely IPv6 link-local addressing, making use of BGP multiprotocol and extended next hop capabilities for IPv4.
The transit VXLAN and burble.dn42 services networks use an MTU of 4260, however the dn42 BGP configuration includes internal communities to distribute destination MTU across the network for per-route MTUs. This helps ensure path mtu discovery takes place as early and efficiently as possible.
Local services on each host are provided by LXD containers or VMs connecting to internal network bridges.
These vary across hosts but typically include:
tier1 - used for publically avialable services (DNS, web proxy, etc) tier2 - used for internal services, with access restricted to burble.dn42 networks Other networks might include:
dmz - used for hosting untrusted services (e.g. the shell servers) dn42 services - for other networks, such as the registry services dn42 peer tunnels are created directly on the host and then injected in to the container using a small script, allowing the router container itself to remain unprivileged.
The routers also run nftables for managing access to each of the networks, bird_exporter for metrics and the bird-lg-go proxy for the burble.dn42 looking glass.
Host Configuration burble.dn42 nodes are designed to have the minimum functionality at the host level, with all major services being delivered via virtual networks, containers and VMs.
Hosts have three main functions:
connecting in to the burble.dn42 Wireguard/L2TP mesh and BGP EVPN providing internal bridges for virtual networks hosting LXD containers and VMs Together these three capabilities allow for arbitary, isolated networks and services to be created and hosted within the network.
The hosts also provide a few ancillary services:
delivering clearnet access for internal containers/VMs using an internal bridge. The host manages addressing and routing for the bridge to allow clearnet access independent of the host capabilities (e.g. proxied vs routed IPv6 connectivity) creating dn42 peer tunnels and injecting them in to the dn42 router container monitoring via netdata backup using borg `}),e.add({id:30,href:"/additional/other-stuff/",title:"Other stuff",parent:"Additional Info",content:`A collection of other stuff that may or may not be dn42 related, or even interesting.
An atomic clock NTP server Rubidium Atomic Clock + GPS + Odroid N2+
Homelab The burble.dn42 homelab:
32 cores, 156GB RAM, 6TB NVMe/SSD, 56TB HDD, ~70 watts
The two hanging USB devices are USB modems for the burble.dn42 dial up service.
`}),e.add({id:31,href:"/tags/",title:"Tags",parent:"burble.dn42",content:""}),e.add({id:32,href:"/additional/things-to-do/",title:"Things to do in DN42",parent:"Additional Info",content:`What can you do in DN42 ? Ultimately, you&rsquo;ll get out of DN42 what you put in to it, but I&rsquo;ve listed here a few ideas that may serve as inspiration and the spark an idea.
This is deliberately not a set of instructions or a guide and it&rsquo;s not a checklist of stuff you must do. If you are interested in something there is plenty of public information available on all these topics.
Getting Started Read up on how Internet peering works, and the tools and protocols that are used Register your details in the DN42 registry Do read the DN42 getting started guide Do browse through the registry itself and use what other people did as examples Do look through recent Pull Requests to see what is required and how to do it Join the mailing list and #dn42 on hackint DN42 is a great community with many knowledgable members. You can learn a lot from what other people are doing, or the problems they have, as well as getting your own network working Get your first peer use the peerfinder to find peers close to you, or ask on IRC ping something on DN42 use a DN42 service Congratulations, you&rsquo;re connected to DN42 !
The Basics Get more peers Add 4 or 5 different peers having several peers prevents having a dependency on a single peer and adds redundancy provides you with a variety of different routes learn how different peers manage their networks How do you see which routes are being advertised and selected ? Change route metrics and see how this influences selected routes Optimise your routes across your peers What is an optimal route anyway ? How is your network being distributed across DN42 ? How do you find out ? Change how your routes are advertised to peers to influence the routing to your network across DN42 Set up DNS and resolve a host in the .dn42 hierarchy Set up your own DNS server Register a domain; set up forward and reverse DNS Set up a blog/wiki and document your network Make the pages available over DN42 and the Internet Add your network to the peerfinder Learn something new and add it to the DN42 Wiki Intermediate Secure your network Distribute DN42 routes to another, internal node Learn how to use an IGP and iBGP Add two or more nodes to DN42 and peer with multiple AS Distribute routes from all peers across the nodes in your network How do you decide which routes are &lsquo;best&rsquo; across the network ? Optimise your routes to DN42 How do you manage multiple entry points to your network ? What do other networks see ? How do other networks decide which node to route to ? Configure your network so that one node is preferred Optimise how DN42 sees your network Add two or more nodes in different continents Why is that different ? How do you optimise your network now ? Implement ROA Implement BGP communities Help a new joiner connect to DN42 Resolve someone elses DN42 problem Set up a looking glass Set up a service that can be used by the rest of the DN42 community Make it a &lsquo;production&rsquo; service, add HA, monitoring and alerting Secure your service Use the DN42 CA Add it to the wiki Complex Isolate your network Using VRFs Using Namespaces Connect multiple nodes to the same peer AS in different geographic locations Optimise the routes to the AS Optimise the routes that the peer AS has to you Monitor your network How do you know it&rsquo;s working well ? (what does &lsquo;working well&rsquo; mean ?) provide public metrics Create a virtual environmment to test changes How do you make your virtual environment representative of DN42 ? Volunteer to help with DN42 core services System administration and automation Patching and maintenance Implement backups and DR Automate the set up and configuration of your nodes Automate adding peers Even more Make something new What&rsquo;s the latest software or network trend ? implement it and learn how to use it Make something experimental Try out a cutting edge service or network technology and see if you can get it to work What are the current challenges faced by the Internet ? How are they being solved ? Can you replicate the problem and potential solutions in DN42 ? Make something social Create and share a community resource Make something stable Fine tune your network and nodes DN42 changes all the time, how do you protect your network from other people breaking things ? Make something small Take something huge (DNS, CDNs, gmail, distributed computing, serverless, AI &hellip;) and shrink it down to your network, but using the same techniques as the global players to manage things scale it up, and down Make something corporate Replicate a multi-datacentre corporate/organisation design Grab your next job based on your experience ;) Make it all again There&rsquo;s more than one way to do it; rebuild your network using a different design How can you rebuild everything whilst minising the impact to everyone else ? What did you learn ? What works better now ? What is worse ? How can you do it again better next time ? Final words What can you do in DN42 ?
Do whatever is fun and interesting for you `})})()