www/content/additional/maintlog/2020.md
Simon Marsh 3970981f1b
All checks were successful
continuous-integration/drone/push Build is passing
test pipeline
2021-12-27 14:15:48 +00:00

13 KiB

title weight
2020 -2020

Archive of changes made in 2020

22nd December 2020

es-mad1 in Madrid, Spain has been deployed and is now open for peering.

13th December 2020

Issue Log

burble.dn42 now has a public issue log, hosted on the DN42 Registry.

Feel free to raise issues or enhancements on the log.

Speedtest Service

An experimental speed test service has been introduced:

The two services are currently only accessible over IPv6 but are hosted on dedicated servers with plenty of available bandwidth. If the service ends up loading or disrupting the rest of the network then I may end up removing it, so use responsibly.

n8n Automation

The burble.dn42 network now has an instance of n8n to help automate internal workflows.
Whilst this isn't a public service the first visibile benefit is that the Explorer and ROA files now update immediately following registry changes. Previously changes were polled and could take up to an hour to be updated.

6th December 2020

ca-bhs2 and fr-rbx1 have been migrated to their new servers. If you are peering with these nodes please make sure you update any IP addresses on your side as required.

git.dn42.dev is hosted on ca-bhs2 and so was also migrated and upgraded to v1.13.0.

28th November 2020

Black Friday has been been and gone and this means that a few nodes have now reached the end of their contract and are being retired:

  • fr-sbg1
  • us-sea2
  • jp-tyo1
  • sg-sin1
  • us-mia2

However, the good news is that Black Friday also delivered a few shiny new nodes for the burble.dn42 network:

  • ca-bhs2 will be replaced with a new node that has SSD rather than HDD storage
  • new nodes expected in Hong Kong and Madrid, eta Jan 2021

Users of ca-bhs2 will be migrated to the new node, with details to be confirmed.

12th September 2020

uk-lon1 has been upgraded. If you are peered on this node, please update your IP address accordingly.

burble.dn42 now includes some limited protection against ghost route updates. See the communities page for more details.

9th September 2020

uk-lon1 will be upgraded this weekend (12/13th September), but unfortunately this does mean that the IP address for the server is going to change.

The hostname will be changed to match the new address, but if you use the IP address in your configuration (e.g. for firewall rules), you will need to update them, as detailed below.

dn42-uk-lon1.burble.com

  • IPv4: 185.42.222.153
  • IPv6: 2a04:92c5:2::42

All other peering details, such as encryption keys and tunnel IP addresses will remain the same, and the new server is in the same datacentre so there should be no changes to connectivity or latency.

28th August 2020

Changes to the burble.dn42 network

Over the next year the focus of the burble.dn42 network will change focus to providing high quality, reliable services for DN42. As part of this change, a number of the current 'edge' nodes will be decommissioned to reduce admin overhead and allow concentration on the core, service nodes.

The following nodes will be decommissioned and are no longer available for new peerings:

Node Decommissioning Date
us-mia2 Immediately
sg-sin1 November 2020
us-sea2 November 2020
fr-sbg1 Nocember 2020
jp-tyo1 December 2020
au-syd1 January 2021
us-nyc1 April 2021
us-chi1 May 2021
 

The current core nodes will continue to operate and some will also be upgraded. The number of services provided by the network will also expand.

Core nodes

Node Future Plans
fr-rbx1 Increase in services offered
ca-bhs2 Upgrade to ssd disks ~November 2020
de-fra1 Upgraded in August to 4 x Epyc / 20G RAM / NVMe
us-dal3 Increase in services offered / potential for upgrade
us-lax1 Increase in services offered
sg-sin2 Take over services from sg-sin1
 

Other Nodes

Node Future Plans
uk-lon1 Upgrade before January 2021
ch-zur1 No changes planned
no-trd1 No changes planned
 

22nd August 2020

de-fra1 has been replaced with a shiny, upgraded, new node.

If you are peered on de-fra1, please check your configuration and ensure you are using the new IP addresses

  • IPv4: 193.41.237.149
  • IPv6: 2a0d:5941:1:17c::4e2a

All other peering parameters remain the same.

15th August 2020

The DN42 registry now supports automated pipelines using Drone CI.
Details will be published on the DN42 wiki.

25th July 2020

us-mia1 and us-mia2 have been swapped. The provider for the old us-mia2 (Stockservers) appears to have ceased trading, so the node has been swapped in case the original server disappears at short notice.

EDIT: seems the new provider wasn't better, so us-mia2 is back on the old server until it dies completely.

no-trd1 has been added, courtesy of jastrup.

lt-vil1 is being decommissioned and users will need to migrate to a different node to maintain service.

5th July 2020

A busy weekend supporting the move of the DN42 registry to its new host.

Remember to join the new mailing list at https://groups.io/g/dn42 and create yourself an account on the new registry https://git.dn42.dev

10th June 2020

Website moved again, and new paste.burble.dn42 service added.

6th June 2020

The global route collector has had a long overdue upgrade. Please let me know if you spot any residual issues.

25th May 2020

The new DNS implementation has been deployed across all nodes. The DNS service now supports:

  • Authoritative DNS for DN42 domains (b.delegation-servers.dn42)
  • Recursive DNS (b.recursive-servers.dn42)
  • DNS64 (dns64.burble.dn42)

All services support UDP, TCP, DNS over HTTPS and DNS over TLS queries. See the DNS page for more info.

23rd May 2020

A new implementation of the edge DNS service is currently being tested across a few nodes, please let me know if you spot any DNS oddness.

18th May 2020

Added whois.burble.dn42 service, see the services page for more details.

https://explorer.burble.dn42 now has regional mirrors so should be significantly faster for anyone not in Europe.

16th May 2020

Approximately 40 old or inactive peers have been deleted as part of a spring cleaning exercise.

If you've been accidently deleted and still want to peer with me, just give me a shout and I will re-instate the configuration.

11th May 2020

Rate limiting on BGP sessions has been implemented to protect the network from major route flapping events. The rate limiting should only kick in after 30+ minutes of extremely high updates (or even longer for milder events), but please let me know if this causes any issues.

8th May 2020

us-lax1 has been migrated. If you peer with me please remember to update the clearnet IP addresses on your side:

dn42-us-lax1
IPv4: 185.215.224.214
IPv6: 2a0b:ae40:1:4a0a::5a

5th May 2020

us-lax1 is being upgraded !

Apologies for the short notice, but us-lax1 will be upgraded over the weekend of 9th/10th May. The upgrade will allow for more services to be provided from the node, to provide enabling a better response for users in Asia and West Coast US.

Unfortunately the upgrade means that IP address of the node will change and peers will need to update their config accordingly. The encryption keys and tunnel addresses should not need to change.

4th May 2020

Several of the burble.dn42 core nodes have been upgraded to Ubuntu 20.04. This required a short outage, but will allow for a refactoring of a few services in the future.

13th April 2020

Bugs have been fixed and both instances of the burble.dn42 website are now running in a new environment with the latest grav.

The new website instance is the first burble.dn42 application running on Ubuntu 20.04 (Focal Fossa).

11th April 2020

The clearnet version of this website is running with a new instance that has the latest grav.
Please let me know if you spot any problems.

The DN42 instance continues to run with the previous version.

4th April 2020

Well, that was fun; burble.dn42 had a number of outages over this evening, caused by trying to perform a rolling upgrade across the network. The biggest of these took out the burble.dn42 DNS service for an extended period, impacting DNS resolution across DN42.

The plan had been to perform a full upgrade and reboot for every burble.dn42 node. To minimise disruption I perform updates across groups of servers that are chosen to be independent so that service resilience should not be impacted.
However, this time there were two key failures:

  • The provider configuration for ca-bhs2 meant that it could not mount all of its disks when rebooted and it ended up in maintenance mode. The server needed to be recovered via the IPMI console. Whilst global services continued to be provided by other nodes, peers on ca-bhs2 lost connectivity whilst the node was recovered.

  • The new pdns-recursor that was implemented at the end of March (see below) had a different runtime path than the default OS install. This meant that when each of the core nodes was restarted the pdns-recursor failed to restart as the runtime path was missing. Since the DNS service is resilient, it continued to operate without problems until the last core node was restarted, at which point the entire service failed. Without DNS, most of the remaining burble.dn42 failed or could not be restarted and recovery was also hampered by having to work without having DNS available.

1st April 2020

at-vie1 will be decommissioned by 14th April. If you are peered on this node, please contact me to move the peering to another node.

28th March 2020

The patched pdns recursor is now deployed to all core nodes.

Please let me know immediately if you notice odd DNS behaviour.

24th March 2020

fr-sbg1 (which hosts the europe region core DNS service) is currently testing a special pdns recursor build in order to try and fix this issue.

The server is likely to be used for most recursive DNS lookups across Europe that use the new DNS anycast addresses, or my service directly. Please let me know immediately if you notice odd DNS behaviour.

26th January 2019

This weekend has been a huge maintenance weekend for burble.dn42, with the following updates taking place:

  • A number of nodes have been built and swapped in to the network to upgrade and manage renewals
fr-rbx1 replaced by fr-rbx2 fr-rbx2 was a much faster node
ca-bhs2 replaced with a new node the replacement is also much faster
us-dal3 replaced by us-dal1 us-dal3 was a poor performer and has been replaced with a dedicated server
au-syd1 replaced with a new node memory increased from 1G to 2G
sg-sin2 replaced with a new node memory increased from 1G to 2G

Node renewals are now mostly sorted until November, which will be a nice break for my wallet.

  • The build of ca-bhs2 introduced a new disk layout for my core nodes, which is intended to provide more flexibility for new features. uk-lon3, a private storage node, was also rebuilt for the new design.

A bad decision around backups meant that I also had to re-create all the services on fr-rbx2 and us-dal1 as they were swapped in to their new roles. As a result, the services on these boxes were also flattened and rebuilt to the new disk layout.

At some future point, fr-sbg1 will follow and also change to the new layout.

  • The burble.dn42 is organised around a core network of servers in each region, the updates this weekend complete a series of changes to upgrade the core nodes that has been taking place since November 2019. A lot of the recent work has been to update the services so they are on, or point to, the new core nodes.

The core network looked like this prior to November 2019:

Name CPU Memory Disk Network Descr
fr-rbx1 i5-2400 (4/8 x 3.4Ghz) 16G 2TB Consumer HDD 100mbps un-metered Kimsufi KS-10
ca-bhs2 i5-3570S (4/8 x 3.8Ghz) 16G 2TB Consumer HDD 100mbps un-metered Kimsufi KS-10
sg-sin2 virtual (1 x 3.5Ghz) 1G 30GB HDD 1TB @ 1gbit OVH VPS
us-dal3 virtual (2 x 3.4Ghz) 5G 120GB HDD 5TB @ 10gbit HostDoc VPS

Following the upgrades, the core now consists of the following servers:

Name CPU Memory Disk Network Descr
fr-sbg1 E5-1620 (4/8 x 3.7Ghz) 32G 3 x 480GB SSD 500mbps un-metered OVH SYS
fr-rbx1 E3-1245 (4/8 x 3.4Ghz) 32G 2 x 480GB SSD 500mbps un-metered OVH SYS
uk-lon3 virtual (2 x 3Ghz) 3G 3TB HDD 10TB @ 1gbit HostHatch
ca-bhs2 E5-1620 (4/8 x 3.7Ghz) 32G 2 x 2TB Ent. HDD 500mbps un-metered OVH SYS
us-dal3 C2750 (8 x 2.4Ghz) 8G 240GB SSD 100mbps un-metered drserver
sg-sin1 virtual (4 x 2.2Ghz) 4G 24GB SSD 1gbit un-metered ITLDC VPS