www/content/additional/maintlog/2020.md

---
title: "2020"
weight: -2020
---
Archive of changes made in 2020

#### 22nd December 2020

es-mad1 in Madrid, Spain has been deployed and is now open for peering.

#### 13th December 2020

**Issue Log**

burble.dn42 now has a public issue log, hosted on the [DN42 Registry](https://git.dn42.dev).

- [Issue Log](https://git.dn42.dev/burble/burble.dn42/issues)

Feel free to raise issues or enhancements on the log. 

**Speedtest Service**

An experimental speed test service has been introduced:

- France: [https://speedtest.fr-rbx1.burble.dn42](https://speedtest.fr-rbx1.burble.dn42)
- Canada: [https://speedtest.ca-bhs2.burble.dn42](https://speedtest.ca-bhs2.burble.dn42)

The two services are currently only accessible over IPv6 but are hosted on dedicated
servers with plenty of available bandwidth. If the service ends up loading or disrupting
the rest of the network then I may end up removing it, so use responsibly.

**n8n Automation**

The burble.dn42 network now has an instance of [n8n](https://n8n.io) to help
automate internal workflows.  
Whilst this isn't a public service the first visibile benefit is that the
[Explorer](https://explorer.burble.com) and
[ROA files](https://dn42.dev/howto/Bird#route-origin-authorization) now update
immediately following registry changes. Previously changes were polled and could
take up to an hour to be updated.

#### 6th December 2020

ca-bhs2 and fr-rbx1 have been migrated to their new servers. If you are peering with these
nodes please make sure you update any IP addresses on your side as required.

git.dn42.dev is hosted on ca-bhs2 and so was also migrated and upgraded to v1.13.0.


#### 28th November 2020

Black Friday has been been and gone and this means that a few nodes have now reached the end of their contract and are being retired:

 - fr-sbg1
 - us-sea2
 - jp-tyo1
 - sg-sin1
 - us-mia2

However, the good news is that Black Friday also delivered a few shiny new nodes for the burble.dn42 network:

 - ca-bhs2 will be replaced with a new node that has SSD rather than HDD storage
 - new nodes expected in Hong Kong and Madrid, eta Jan 2021

Users of ca-bhs2 will be migrated to the new node, with details to be confirmed.
 

#### 12th September 2020

uk-lon1 has been upgraded. If you are peered on this node, please update
your IP address accordingly.

burble.dn42 now includes some limited protection against ghost route updates.
See the [communities](/home/bgp-communities) page for more details.

#### 9th September 2020

uk-lon1 will be upgraded this weekend (12/13th
September), but unfortunately this does mean that the IP address for the
server is going to change.

The hostname will be changed to match the new address, but if you use the
IP address in your configuration (e.g. for firewall rules), you will need
to update them, as detailed below.

*dn42-uk-lon1.burble.com*
- IPv4: 185.42.222.153
- IPv6: 2a04:92c5:2::42

All other peering details, such as encryption keys and tunnel IP addresses
will remain the same, and the new server is in the same datacentre so there
should be no changes to connectivity or latency.


#### 28th August 2020

**Changes to the burble.dn42 network**

Over the next year the focus of the burble.dn42 network will change focus to providing high quality, reliable services for DN42. As part of this change, a number of the current 'edge' nodes will be decommissioned to reduce admin overhead and allow concentration on the core, service nodes. 

The following nodes will be decommissioned and are no longer available for new peerings:

| Node | Decommissioning Date |
|:--|:--|
| us-mia2 | Immediately |
| sg-sin1 | November 2020 |
| us-sea2 | November 2020 |
| fr-sbg1 | Nocember 2020 |
| jp-tyo1 | December 2020 |
| au-syd1 | January 2021 |
| us-nyc1 | April 2021 |
| us-chi1 | May 2021 |
 
  

The current core nodes will continue to operate and some will also be upgraded. The number of services provided by the network will also expand.

**Core nodes**

| Node | Future Plans |
|:--|:--|
| fr-rbx1 | Increase in services offered |
| ca-bhs2 | Upgrade to ssd disks ~November 2020 |
| de-fra1 | Upgraded in August to 4 x Epyc / 20G RAM / NVMe |
| us-dal3 | Increase in services offered / potential for upgrade |
| us-lax1 | Increase in services offered |
| sg-sin2 | Take over services from sg-sin1 |
 

**Other Nodes**

| Node | Future Plans |
|:--|:--|
| uk-lon1 | Upgrade before January 2021 |
| ch-zur1 | No changes planned |
| no-trd1 | No changes planned |
 


#### 22nd August 2020

de-fra1 has been replaced with a shiny, upgraded, new node.

If you are peered on de-fra1, please check your configuration and ensure
you are using the new IP addresses

- IPv4: 193.41.237.149
- IPv6: 2a0d:5941:1:17c::4e2a

All other peering parameters remain the same.

#### 15th August 2020

The DN42 registry now supports automated pipelines using Drone CI.  
Details will be published on the DN42 wiki.

#### 25th July 2020

us-mia1 and us-mia2 have been swapped. The provider for the old us-mia2 (Stockservers)
appears to have ceased trading, so the node has been swapped in case the original server
disappears at short notice.

EDIT: seems the new provider wasn't better, so us-mia2 is back on the old server
until it dies completely.

no-trd1 has been added, courtesy of jastrup.

lt-vil1 is being decommissioned and users will need to migrate to a different node to
maintain service.

#### 5th July 2020

A busy weekend supporting the move of the DN42 registry to its new host.

Remember to join the new mailing list at [https://groups.io/g/dn42](https://groups.io/g/dn42)
and create yourself an account on the new registry [https://git.dn42.dev](https://git.dn42.dev)

#### 10th June 2020

Website moved again, and new paste.burble.dn42 service added.

#### 6th June 2020

The global route collector has had a long overdue upgrade. Please let me know
if you spot any residual issues.

#### 25th May 2020

The new DNS implementation has been deployed across all nodes.
The DNS service now supports:

- Authoritative DNS for DN42 domains (b.delegation-servers.dn42)
- Recursive DNS (b.recursive-servers.dn42)
- DNS64 (dns64.burble.dn42)

All services support UDP, TCP, DNS over HTTPS and DNS over TLS queries.
See the [DNS](/home/dns) page for more info.

#### 23rd May 2020

A new implementation of the edge DNS service is currently being tested across
a few nodes, please let me know if you spot any DNS oddness.

#### 18th May 2020

Added whois.burble.dn42 service, see the [services](/home/burble-dn42-services)
page for more details.

https://explorer.burble.dn42 now has regional mirrors so should be significantly
faster for anyone not in Europe.

#### 16th May 2020

Approximately 40 old or inactive peers have been deleted as part of a spring cleaning
exercise.

If you've been accidently deleted and still want to peer with me, just give me a shout
and I will re-instate the configuration. 


#### 11th May 2020

Rate limiting on BGP sessions has been implemented to protect the network from major
route flapping events. The rate limiting should only kick in after 30+ minutes of
extremely high updates (or even longer for milder events), but please let me know if
this causes any issues.

#### 8th May 2020

us-lax1 has been migrated. If you peer with me please remember to update the
clearnet IP addresses on your side:

dn42-us-lax1  
**IPv4**: 185.215.224.214  
**IPv6**: 2a0b:ae40:1:4a0a::5a

#### 5th May 2020

us-lax1 is being upgraded !

Apologies for the short notice, but us-lax1 will be upgraded over the weekend of 9th/10th
May. The upgrade will allow for more services to be provided from the node, to provide
enabling a better response for users in Asia and West Coast US.

Unfortunately the upgrade means that IP address of the node will change and peers will need
to update their config accordingly. The encryption keys and tunnel addresses should not need
to change. 


#### 4th May 2020

Several of the burble.dn42 core nodes have been upgraded to Ubuntu 20.04. This required
a short outage, but will allow for a refactoring of a few services in the future. 


#### 13th April 2020

Bugs have been fixed and both instances of the burble.dn42 website are now running
in a new environment with the latest grav. 

The new website instance is the first burble.dn42 application running on Ubuntu 20.04
(Focal Fossa). 

#### 11th April 2020

The clearnet version of this [website](https://dn42.burble.com) is running with
a new instance that has the latest grav.  
Please let me know if you spot any problems.

The DN42 instance continues to run with the previous version.

#### 4th April 2020

Well, that was fun; burble.dn42 had a number of outages over this evening, caused by trying
to perform a rolling upgrade across the network. The biggest of these took out the burble.dn42
DNS service for an extended period, impacting DNS resolution across DN42.

The plan had been to perform a full upgrade and reboot for every burble.dn42 node. To minimise
disruption I perform updates across groups of servers that are chosen to be independent
so that service resilience should not be impacted.  
However, this time there were two key failures:

- The provider configuration for ca-bhs2 meant that it could not mount all of its disks when
rebooted and it ended up in maintenance mode. The server needed to be recovered via the
IPMI console. Whilst global services continued to be provided by other nodes, peers on
ca-bhs2 lost connectivity whilst the node was recovered.

- The new pdns-recursor that was implemented at the end of March (see below) had a different
runtime path than the default OS install. This meant that when each of the core nodes was
restarted the pdns-recursor failed to restart as the runtime path was missing. Since the DNS
service is resilient, it continued to operate without problems until the last core node was
restarted, at which point the entire service failed. Without DNS, most of the remaining
burble.dn42 failed or could not be restarted and recovery was also hampered by having to
work without having DNS available.

#### 1st April 2020

at-vie1 will be decommissioned by 14th April.
If you are peered on this node, please contact me to move the peering to another node.

#### 28th March 2020

The patched pdns recursor is now deployed to all core nodes.

Please let me know immediately if you notice odd DNS behaviour.


#### 24th March 2020

fr-sbg1 (which hosts the europe region core DNS service) is currently testing a special pdns
recursor build in order to try and fix [this issue](https://github.com/PowerDNS/pdns/issues/8106).

The server is likely to be used for most recursive DNS lookups across Europe that use the
new DNS anycast addresses, or my service directly. Please let me know immediately if you
notice odd DNS behaviour.


#### 26th January 2019

This weekend has been a huge maintenance weekend for burble.dn42, with the following
updates taking place:

- A number of nodes have been built and swapped in to the network to upgrade and manage renewals

|||
|---|---|
| fr-rbx1 replaced by fr-rbx2 | *fr-rbx2 was a much faster node* |
| ca-bhs2 replaced with a new node | *the replacement is also much faster* |
| us-dal3 replaced by us-dal1 | *us-dal3 was a poor performer and has been replaced with a dedicated server* |
| au-syd1 replaced with a new node | *memory increased from 1G to 2G* |
| sg-sin2 replaced with a new node | *memory increased from 1G to 2G* |

Node renewals are now mostly sorted until November, which will be a nice break for my wallet.

- The build of ca-bhs2 introduced a new disk layout for my core nodes, which is intended
to provide more flexibility for new features. uk-lon3, a private storage
node, was also rebuilt for the new design.

A bad decision around backups meant that I also had to re-create all the services on
fr-rbx2 and us-dal1 as they were swapped in to their new roles. As a result, the services
on these boxes were also flattened and rebuilt to the new disk layout.

At some future point, fr-sbg1 will follow and also change to the new layout.
  
- The burble.dn42 is organised around a core network of servers in each region, the updates
this weekend complete a series of changes to upgrade the core nodes that has been taking
place since November 2019. A lot of the recent work has been to update the services so they
are on, or point to, the new core nodes.

The core network looked like this prior to November 2019:


|Name|CPU|Memory|Disk|Network|Descr|
|---|---|---|---|---|---|
| fr-rbx1 | i5-2400 (4/8 x 3.4Ghz) | 16G | 2TB Consumer HDD | 100mbps un-metered | Kimsufi KS-10 |
| ca-bhs2 | i5-3570S (4/8 x 3.8Ghz) | 16G | 2TB Consumer HDD | 100mbps un-metered | Kimsufi KS-10 |
| sg-sin2 | virtual (1 x 3.5Ghz) | 1G | 30GB HDD | 1TB @ 1gbit | OVH VPS |
| us-dal3 | virtual (2 x 3.4Ghz) | 5G | 120GB HDD | 5TB @ 10gbit | HostDoc VPS |

Following the upgrades, the core now consists of the following servers:

|Name|CPU|Memory|Disk|Network|Descr|
|---|---|---|---|---|---|
| fr-sbg1 | E5-1620 (4/8 x 3.7Ghz) | 32G | 3 x 480GB SSD | 500mbps un-metered | OVH SYS |
| fr-rbx1 | E3-1245 (4/8 x 3.4Ghz) | 32G | 2 x 480GB SSD | 500mbps un-metered | OVH SYS |
| uk-lon3 | virtual (2 x 3Ghz) | 3G | 3TB HDD | 10TB @ 1gbit | HostHatch |
| ca-bhs2 | E5-1620 (4/8 x 3.7Ghz) | 32G | 2 x 2TB Ent. HDD | 500mbps un-metered | OVH SYS |
| us-dal3 | C2750 (8 x 2.4Ghz) | 8G | 240GB SSD | 100mbps un-metered | drserver |
| sg-sin1 | virtual (4 x 2.2Ghz) | 4G | 24GB SSD | 1gbit un-metered | ITLDC VPS |