Browse Source

Initial Commit

master
Simon Marsh 1 year ago
commit
66ca94dccc
Signed by: burble <[email protected]> GPG Key ID: 7B9FE8780CFB6593
11 changed files with 1720 additions and 0 deletions
  1. +83
    -0
      API.md
  2. +66
    -0
      README.md
  3. +1
    -0
      StaticRoot/anchorme.min.js
  4. +12
    -0
      StaticRoot/bootstrap.min.css
  5. BIN
      StaticRoot/dn42_logo.png
  6. +245
    -0
      StaticRoot/explorer.js
  7. +184
    -0
      StaticRoot/index.html
  8. +169
    -0
      dn42regsrv.go
  9. +267
    -0
      regapi.go
  10. +638
    -0
      registry.go
  11. +55
    -0
      static.go

+ 83
- 0
API.md View File

@@ -0,0 +1,83 @@
# dn42regsrv API Description

## GET /<file>

If the StaticRoot configuration option points to a readable directory, files from
the directory will be served under /

The git repository contains a sample StaticRoot directory with a simple registry
explorer web app.

## GET /api/registry/

Returns a JSON object, with keys for each registry type and values containing a count
of the number of registry objects for each type.

Example:
```
http://localhost:8042/api/registry/

# sample output
{"as-block":8,"as-set":34,"aut-num":1482,"domain":451,"inet6num":744,"inetnum":1270,"key-cert":7,"mntner":1378,"organisation":275,"person":1387,"registry":4,"role":14,"route":886,"route-set":2,"route6":594,"schema":18,"tinc-key":25,"tinc-keyset":3}
```


## GET /api/registry/<type>?match

Returns a JSON object listing all objects for the matched types.

Keys for the returned object are registry types, the value for each type is an
array of object names

If the match parameter is provided, the <type> is substring matched against
all registry types, otherwise an exact type name is required.

A special type of '*' returns all types and objects in the registry.

Example:
```
http://localhost:8042/api/registry/aut-num # list aut-num objects
http://localhost:8042/api/registry/* # list all types and objects
http://localhost:8042/api/registry/route?match # list route and route6 objects

# sample output
{"role":["ALENAN-DN42","FLHB-ABUSE-DN42","ORG-SHACK-ADMIN-DN42","PACKETPUSHERS-DN42","CCCHB-ABUSE-DN42","ORG-NETRAVNEN-DN42","ORG-SHACK-ABUSE-DN42","MAGLAB-DN42","NIXNODES-DN42","SOURIS-DN42","CCCKC-DN42","NL-ZUID-DN42","ORG-SHACK-TECH-DN42","ORG-YANE-DN42"]}

```

## GET /api/registry/<type>/<object>?match&raw

Return a JSON object with the registry data for each matching object.

The keys for the object are the object paths in the form <type>/<object name>. The values depends on the raw parameter.

if the raw parameter is provided, the returned object consists of a single key 'Attributes'
which will be an array of key/value pairs exactly as held within the registry.

If the raw parameter is not provided, the returned Attributes are decorated with markdown
style links depending the relations defined in the DN42 schema. In addition a
'Backlinks' key is added which provides an array of registry objects that
reference this one.

If the match parameter is provided, the <object> is substring matched against all
objects in the <type>. Matching is case insensitive.

If the match parameter is not provided, an exact, case sensitive object name is required.

A special object of '*' returns all objects in the type

Example:
```
http://localhost:8042/api/registry/domain/burble.dn42?raw # return object in raw format
http://localhost:8042/api/registry/mntner/BURBLE-MNT # return object in decorated format
http://localhost:8042/api/registry/aut-num/2601?match # return all aut-num objects matching 2601
http://localhost:8042/api/registry/schema/* # return all schema objects

# sample output (raw)
{"domain/burble.dn42":[["domain","burble.dn42"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","BURBLE-DN42"],["tech-c","BURBLE-DN42"],["mnt-by","BURBLE-MNT"],["nserver","ns1.burble.dn42 172.20.129.161"],["nserver","ns1.burble.dn42 fd42:4242:2601:ac53::1"],["ds-rdata","61857 13 2 bd35e3efe3325d2029fb652e01604a48b677cc2f44226eeabee54b456c67680c"],["source","DN42"]]}

# sample output (decorated)
{"mntner/BURBLE-MNT":{"Attributes":[["mntner","BURBLE-MNT"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","[BURBLE-DN42](person/BURBLE-DN42)"],["tech-c","[BURBLE-DN42](person/BURBLE-DN42)"],["auth","pgp-fingerprint 1C08F282095CCDA432AECC657B9FE8780CFB6593"],["mnt-by","[BURBLE-MNT](mntner/BURBLE-MNT)"],["source","[DN42](registry/DN42)"]],"Backlinks":["as-set/AS4242422601:AS-DOWNSTREAM","as-set/AS4242422601:AS-TRANSIT","inetnum/172.20.129.160_27","person/BURBLE-DN42","route/172.20.129.160_27","inet6num/fd42:4242:2601::_48","mntner/BURBLE-MNT","aut-num/AS4242422601","aut-num/AS4242422602","route6/fd42:4242:2601::_48","domain/collector.dn42","domain/burble.dn42"]}}

```


+ 66
- 0
README.md View File

@@ -0,0 +1,66 @@
# dn42regsrv

A REST API for the DN42 registry, written in Go, to provide a bridge between
interactive applications and the DN42 registry.

## Features

- REST API for querying DN42 registry objects
- Able to decorate objects with relationship information based on SCHEMA type definitions
- Includes a simple webserver for delivering static files which can be used to deliver
basic web applications utilising the API (such as the included DN42 Registry Explorer)
- Automatic pull from the DN42 git repository to keep the registry up to date
- Included responsive web app for exploring the registry

## Building

Requires [git](https://git-scm.com/) and [go](https://golang.org)

```
go get https://git.dn42.us/burble/dn42regsrv
```

## Running

Use --help to view configurable options
```
./dn42regsrv --help
```

The server requires access to a clone of the DN42 registry and for the git executable
to be accessible.
If you want to use the auto pull feature then the registry must
also be writable by the server.

```
cd ${GOROOT}/src/dn42regsrv
git clone http://git.dn42.us/dn42/registry.git
./dn42regsrv --help
./dn42regsrv
```

A sample service file is included for running the server under systemd

## Using

By default the server will be listening on port 8042.
See the [API.md](API.md) file for a detailed description of the API.


## Support

Please feel free to raise issues or create pull requests for the project git repository.

## #ToDo

### Server

- Add WHOIS interface
- Add endpoints for ROA data
- Add attribute searches

### DN42 Registry Explorer Web App

- Add search history and fix going back
- Allow for attribute searches


+ 1
- 0
StaticRoot/anchorme.min.js
File diff suppressed because it is too large
View File


+ 12
- 0
StaticRoot/bootstrap.min.css
File diff suppressed because it is too large
View File


BIN
StaticRoot/dn42_logo.png View File

Before After
Width: 173  |  Height: 60  |  Size: 2.7KB

+ 245
- 0
StaticRoot/explorer.js View File

@@ -0,0 +1,245 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry Explorer
//////////////////////////////////////////////////////////////////////////

//////////////////////////////////////////////////////////////////////////
// registry-stats component

Vue.component('registry-stats', {
template: '#registry-stats-template',
data() {
return {
state: "loading",
error: "",
types: null,
}
},
methods: {
updateSearch: function(str) {
vm.updateSearch(str)
},
reload: function(event) {
this.types = null,
this.state = "loading"

axios
.get('/api/registry/')
.then(response => {
this.types = response.data
this.state = 'complete'
})
.catch(error => {
this.error = error
this.state = 'error'
console.log(error)
})
}
},
mounted() {
this.reload()
}
})

//////////////////////////////////////////////////////////////////////////
// registry object component

Vue.component('reg-object', {
template: '#reg-object-template',
props: [ 'link' ],
data() {
return { }
},
methods: {
updateSearch: function(str) {
vm.updateSearch(str)
}
},
computed: {
rtype: function() {
var ix = this.link.indexOf("/")
return this.link.substring(0, ix)
},
obj: function() {
var ix = this.link.indexOf("/")
return this.link.substring(ix + 1)
}
}
})

//////////////////////////////////////////////////////////////////////////
// reg-attribute component

Vue.component('reg-attribute', {
template: '#reg-attribute-template',
props: [ 'content' ],
data() {
return { }
},
methods: {
isRegObject: function(str) {
return (this.content.match(/^\[.*?\]\(.*?\)/) != null)
}
},
computed: {
objectLink: function() {
reg = this.content.match(/^\[(.*?)\]\((.*?)\)/)
return reg[2]
},
decorated: function() {
return anchorme(this.content, {
truncate: 40,
ips: false,
attributes: [ { name: "target", value: "_blank" } ]
})
}
}
})

//////////////////////////////////////////////////////////////////////////
// construct a search URL from a search term

function matchObjects(objects, rtype, term) {

var results = [ ]
for (const obj in objects) {
var s = objects[obj].toLowerCase()
var pos = s.indexOf(term)
if (pos != -1) {
if ((pos == 0) && (s == term)) {
// exact match, return just this result
return [[ rtype, objects[obj] ]]
}
results.push([ rtype, objects[obj] ])
}
}
return results
}


function searchFilter(index, term) {

var results = [ ]

// comparisons are lowercase
term = term.toLowerCase()

// includes a '/' ? search only in that type
var slash = term.indexOf('/')
if (slash != -1) {
var rtype = term.substring(0, slash)
var term = term.substring(slash + 1)
objects = index[rtype]
if (objects != null) {
results = matchObjects(objects, rtype, term)
}
}
else {
// walk though the entire index
for (const rtype in index) {
results = results.concat(matchObjects(index[rtype], rtype, term))
}
}

return results
}

//////////////////////////////////////////////////////////////////////////
// main application

// application data
var appData = {
searchInput: '',
searchTimeout: 0,
state: '',
debug: "",
index: null,
filtered: null,
result: null
}

// methods
var appMethods = {

loadIndex: function(event) {
axios
.get('/api/registry/*')
.then(response => {
this.index = response.data
})
.catch(error => {
// what to do here ?
console.log(error)
})
},

// called on every search input change
debounceSearchInput: function(value) {
if (this.search_timeout) {
clearTimeout(this.search_timeout)
}

// reset if searchbox is empty
if (value == "") {
this.state = ""
this.searchInput = ""
this.filtered = null
this.results = null
return
}
this.search_timeout =
setTimeout(this.updateSearch.bind(this,value),500)
},

// called after the search input has been debounced
updateSearch: function(value) {
this.searchInput = value
this.filtered = searchFilter(this.index, value)
if (this.filtered.length == 0) {
this.state = "noresults"
}
else if (this.filtered.length == 1) {
this.state = "loading"
var details = this.filtered[0]

query = '/api/registry/' + details[0] + '/' + details[1]
axios
.get(query)
.then(response => {
this.state = 'result'
this.result = response.data
})
.catch(error => {
this.error = error
this.state = 'error'
})
}
else {
this.state = "resultlist"
this.result = this.filtered
}
}

}


// intialise Vue instance

var vm = new Vue({
el: '#explorer',
data: appData,
methods: appMethods,
mounted() {
this.loadIndex()
}
})



//////////////////////////////////////////////////////////////////////////
// end of code

+ 184
- 0
StaticRoot/index.html View File

@@ -0,0 +1,184 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="bootstrap.min.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
<style>
.material-icons { display:inline-flex;vertical-align:middle }
.table th, .table td { padding: 0.2rem; border: none }
pre { margin-bottom: 0px }
.regref span {
padding: 0.3em 1em 0.3em 1em; margin: 0.4em;
white-space: nowrap; display:inline-block;
}
a { cursor: pointer }
</style>
<title>DN42 Registry Explorer</title>
</head>
<body style="box-shadow: inset 0 2em 10em rgba(0,0,0,0.4); min-height: 100vh">
<div id="explorer">
<nav class="navbar navbar-fixed-top navbar-expand-md navbar-dark bg-dark">
<form class="form-inline" id="SearchForm">
<input v-bind:value="searchInput"
v-on:input="debounceSearchInput($event.target.value)"
class="form-control-lg" size="30" type="search"
placeholder="Search the registry" aria-label="Search"/>
</form>
<div class="collapse navbar-collapse w-100 ml-auto">
<div class="ml-auto"><a class="navbar-brand"
href="/">Registry Explorer</a>&nbsp;<a class="pull-right navbar-brand"
href="https://dn42.us/"><img src="/dn42_logo.png" width="173"
height="60"/></a></div>
</nav>
<div style="padding: 1em">

<div v-show="searchInput == ''">

<div class="jumbotron">
<h1>DN42 Registry Explorer</h1>
<p class="lead">Just start typing in the search box to start searching the registry</p>
<hr/>
<p>
<p>Search Tips</p>
<ul>
<li>Searches are case independent
<li>No need to hit enter, searches will start immediately
<li>Prefixing the search by a registry type followed by / will narrow the search to
just that type (e.g. <a v-on:click="updateSearch('domain/.dn42')"
class="text-success">domain/.dn42</a>&nbsp;)
<li>Searching for <b>type/</b> will return all the objects for that type (e.g.
<a v-on:click="updateSearch('schema/')" class="text-success">schema/</a>&nbsp;)
<li>A blank search box will return you to these instructions
<li>Searches are made on object names; searching the content of objects
is not supported (yet!).
<li>Going back (or any search history) is also not supported yet.
</ul>
<hr/>
<p>The registry explorer is a simple web app using
<a href="https://git.dn42.us/burble/dn42regsrv">dn42regsrv</a>;
a REST API for the DN42 registry built with <a href="https://golang.org/">Go</a>
</div>
<registry-stats/>

</div>

<section v-if="state == 'loading'">
<div class="alert alert-info" role="alert">
Loading data ...
</div>
</section>

<section v-else-if="state == 'error'">
<div class="alert alert-primary clearfix" role="alert">
An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
</div>
</section>

<section v-else-if="state == 'noresults'">
<h2>Searching for "{{ searchInput }}" ...</h2>
<div class="alert alert-dark" role="alert">
Sorry, no results found
</div>
</section>

<section v-else-if="state == 'resultlist'">
<h2>Listing results for "{{ searchInput }}" ...</h2>
<div class="container d-flex flex-row flex-wrap">
<div style="text-align: center">
<span v-for="value in result" style="margin: 0.5em 1em 0.5em 1em; display:inline-block">
<reg-object v-bind:link="value[0] + '/' + value[1]"></reg-object>
</span>
</div>
</div>
</section>

<section v-else-if="state == 'result'">
<div v-for="(val, key) in result">
<h2><reg-object v-bind:link="key"></reg-object></h2>
<div style="padding-left: 2em">
<table class="table">
<thead>
<tr><th scope="col">Key</th><th scope="col">Value</th></tr>
</thead>
<tbody>
<tr v-for="a in val.Attributes">
<th scope="row" class="text-primary" style="white-space:nowrap">{{ a[0] }}</th>
<td><reg-attribute v-bind:content="a[1]"></reg-attribute></td>
</tr>
</tbody>
</table>
</div>
<section v-if="val.Backlinks.length != 0">
<p>Referenced by</p>
<div style="padding-left: 2em">
<table class="table">
<tbody>
<tr v-for="r in val.Backlinks">
<td><reg-object v-bind:link="r"></reg-object></td>
</tr>
</tbody>
</table>
</div>
</section>
</div>
</section>

</div>
</div>
<footer class="page-footer font-small">
<div style="margin-top: 20px; padding: 5px">
<a href="https://git.dn42.us/burble/dn42regsrv">Source Code</a>.
Powered by
<a href="https://getbootstrap.com/">Bootstrap</a>,
<a href="https://vuejs.org">Vue.js</a>,
<a href="https://github.com/axios/axios">axios</a>,
<a href="http://alexcorvi.github.io/anchorme.js/">Anchorme</a>.
</div>
</footer>
</div>

<script type="text/x-template" id="registry-stats-template">
<div class="container d-flex flex-column w-75">
<h5>Registry Stats</h5>
<section v-if="state == 'error'">
<div class="alert alert-primary clearfix" role="alert">
An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
</div>
</section>
<section v-else-if="state == 'loading'">
<div class="alert alert-info" role="alert">
Loading data ...
</div>
</section>
<section v-else>
<p style="padding:1em;text-align:center">
<span v-for="(value, key) in types" style="margin-left:0.5em;margin-right:0.5em;white-space:nowrap;display:inline-block"><a v-on:click="updateSearch(key + '/')" class="text-success">{{ key }}</a>:&nbsp;<b>{{ value }}</b>&nbsp;records</span>
</p>
</section>
</div>
</script>

<script type="text/x-template" id="reg-object-template">
<span class="regref"><a v-on:click="updateSearch(rtype + '/' + obj)" class="text-success"
style="margin-right: 0.4em">{{ obj }}</a>&nbsp;<span
class="badge badge-pill badge-dark text-muted">{{ rtype }}</span></span></span>
</script>

<script type="text/x-template" id="reg-attribute-template">
<span style="word-break: break-all">
<reg-object v-if="isRegObject()" v-bind:link="objectLink"></reg-object>
<span v-else class="text-monospace" v-html="decorated"></span>
</span>
</script>

<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js" integrity="sha384-B0UglyR+jN6CkvvICOB2joaf5I4l3gm9GU6Hc1og6Ls7i6U/mkkaduKaBhlAXv9k" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.js"></script>
<!-- <script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.min.js"></script> -->
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="anchorme.min.js"></script>
<script src="explorer.js"></script>
</body>
</html>

+ 169
- 0
dn42regsrv.go View File

@@ -0,0 +1,169 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////

package main

//////////////////////////////////////////////////////////////////////////

import (
"context"
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
flag "github.com/spf13/pflag"
"net/http"
"os"
"os/signal"
"time"
)

//////////////////////////////////////////////////////////////////////////
// list of API endpoints

type InitEndpointFunc = func(route *mux.Router)

var apiEndpoints = make([]InitEndpointFunc, 0)

func RegisterAPIEndpoint(f InitEndpointFunc) {
apiEndpoints = append(apiEndpoints, f)
}

//////////////////////////////////////////////////////////////////////////
// utility function to set the log level

func setLogLevel(levelStr string) {

if level, err := log.ParseLevel(levelStr); err != nil {
// failed to set the level

// set a sensible default and, of course, log the error
log.SetLevel(log.InfoLevel)
log.WithFields(log.Fields{
"loglevel": levelStr,
"error": err,
}).Error("Failed to set requested log level")

} else {

// set the requested level
log.SetLevel(level)

}
}

//////////////////////////////////////////////////////////////////////////
// http request logger

func requestLogger(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

log.WithFields(log.Fields{
"method": r.Method,
"URL": r.URL.String(),
"Remote": r.RemoteAddr,
}).Debug("HTTP Request")

next.ServeHTTP(w, r)
})
}

//////////////////////////////////////////////////////////////////////////
// everything starts here

func main() {

// set a default log level, so that logging can be used immediately
// the level will be overidden later on once the command line
// options are loaded
log.SetLevel(log.InfoLevel)
log.Info("DN42 Registry API Server Starting")

// declare cmd line options
var (
logLevel = flag.StringP("LogLevel", "l", "Info", "Log level")
regDir = flag.StringP("RegDir", "d", "registry", "Registry data directory")
bindAddress = flag.StringP("BindAddress", "b", "[::]:8042", "Server bind address")
staticRoot = flag.StringP("StaticRoot", "s", "StaticRoot", "Static page directory")
refreshInterval = flag.StringP("Refresh", "i", "60m", "Refresh interval")
gitPath = flag.StringP("GitPath", "g", "/usr/bin/git", "Path to git executable")
autoPull = flag.BoolP("AutoPull", "a", true, "Automatically pull the registry")
pullURL = flag.StringP("PullURL", "p", "origin", "URL to auto pull")
)
flag.Parse()

// now initialise logging properly based on the cmd line options
setLogLevel(*logLevel)

// parse the refreshInterval and start data collection
interval, err := time.ParseDuration(*refreshInterval)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"interval": *refreshInterval,
}).Fatal("Unable to parse registry refresh interval")
}

InitialiseRegistryData(*regDir, interval,
*gitPath, *autoPull, *pullURL)

// initialise router
router := mux.NewRouter()
// log all access
router.Use(requestLogger)

// initialise API routes
subr := router.PathPrefix("/api").Subrouter()
for _, epInit := range apiEndpoints {
epInit(subr)
}

// initialise static routes
InstallStaticRoutes(router, *staticRoot)

// initialise http server
server := &http.Server{
Addr: *bindAddress,
WriteTimeout: time.Second * 15,
ReadTimeout: time.Second * 15,
IdleTimeout: time.Second * 60,
Handler: router,
}

// run the server in a non-blocking goroutine

log.WithFields(log.Fields{
"BindAddress": *bindAddress,
}).Info("Starting server")

go func() {
if err := server.ListenAndServe(); err != nil {
log.WithFields(log.Fields{
"error": err,
"BindAddress": *bindAddress,
}).Fatal("Unable to start server")
}
}()

// graceful shutdown via SIGINT (^C)
csig := make(chan os.Signal, 1)
signal.Notify(csig, os.Interrupt)

// and block
<-csig

log.Info("Server shutting down")

// deadline for server to shutdown
ctx, cancel := context.WithTimeout(context.Background(), 10)
defer cancel()

// shutdown the server
server.Shutdown(ctx)

// nothing left to do
log.Info("Shutdown complete, all done")
os.Exit(0)
}

//////////////////////////////////////////////////////////////////////////
// end of code

+ 267
- 0
regapi.go View File

@@ -0,0 +1,267 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////

package main

//////////////////////////////////////////////////////////////////////////

import (
"encoding/json"
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
"net/http"
"strings"
// "time"
)

//////////////////////////////////////////////////////////////////////////
// register the api

func init() {
RegisterAPIEndpoint(InitRegAPI)
}

//////////////////////////////////////////////////////////////////////////
// called from main to initialise the API routing

func InitRegAPI(router *mux.Router) {

s := router.
Methods("GET").
PathPrefix("/registry").
Subrouter()

s.HandleFunc("/", regRootHandler)
//s.HandleFunc("/.schema", rTypeListHandler)
//s.HandleFunc("/.meta/", rTypeListHandler)

s.HandleFunc("/{type}", regTypeHandler)
s.HandleFunc("/{type}/{object}", regObjectHandler)

log.Info("Registry API installed")
}

//////////////////////////////////////////////////////////////////////////
// handler utility funcs

func responseJSON(w http.ResponseWriter, v interface{}) {

// for response time testing
//time.Sleep(time.Second)

// marshal the JSON string
data, err := json.Marshal(v)
if err != nil {
log.WithFields(log.Fields{
"error": err,
}).Error("Failed to marshal JSON")
}

// write back to http handler
w.Header().Set("Content-Type", "application/json")
w.Write(data)
}

//////////////////////////////////////////////////////////////////////////
// root handler, lists all types within the registry

func regRootHandler(w http.ResponseWriter, r *http.Request) {

response := make(map[string]int)
for _, rType := range RegistryData.Types {
response[rType.Ref] = len(rType.Objects)
}
responseJSON(w, response)

}

//////////////////////////////////////////////////////////////////////////
// type handler returns list of objects that match the type

func regTypeHandler(w http.ResponseWriter, r *http.Request) {

// request parameters
vars := mux.Vars(r)
query := r.URL.Query()

typeName := vars["type"] // type name to list
match := query["match"] // single query or match

// special case to return all types
all := false
if typeName == "*" {
match = []string{}
all = true
}

// results will hold the types to return
var results []*RegType

// check match type
if match == nil {
// exact match

// check the type object exists
rType := RegistryData.Types[typeName]
if rType == nil {
http.Error(w, "No types matching '"+typeName+"' found", http.StatusNotFound)
return
}

// return just a single result
results = []*RegType{rType}

} else {
// substring match

// comparisons are lower case
typeName = strings.ToLower(typeName)

// walk through the types and filter to the results list
results = make([]*RegType, 0)
for key, rType := range RegistryData.Types {
if all || strings.Contains(strings.ToLower(key), typeName) {
// match found, add to the list
results = append(results, rType)
}
}

}

// construct the response
response := make(map[string][]string)
for _, rType := range results {

objects := make([]string, 0, len(rType.Objects))
for key := range rType.Objects {
objects = append(objects, key)
}

response[rType.Ref] = objects
}

responseJSON(w, response)
}

//////////////////////////////////////////////////////////////////////////
// object handler returns object data

// per object response structure
type RegObjectResponse struct {
Attributes [][2]string
Backlinks []string
}

func regObjectHandler(w http.ResponseWriter, r *http.Request) {

// request parameters
vars := mux.Vars(r)
query := r.URL.Query()

typeName := vars["type"] // object type
objName := vars["object"] // object name or match
match := query["match"] // single query or match
raw := query["raw"] // raw or decorated results

// special case to return all objects
all := false
if objName == "*" {
match = []string{}
all = true
}

// verify the type exists
rType := RegistryData.Types[typeName]
if rType == nil {
http.Error(w, "No types matching '"+typeName+"' found",
http.StatusNotFound)
return
}

// results will hold the objects to return
var results []*RegObject

// check match type
if match == nil {
// exact match

// check the object exists
object := rType.Objects[objName]
if object == nil {
http.Error(w, "No objects matching '"+objName+"' found",
http.StatusNotFound)
return
}

// then just create a results list with one object
results = []*RegObject{object}

} else {
// substring matching

// comparisons are lower case
objName = strings.ToLower(objName)

// walk through the type objects and filter to the results list
results = make([]*RegObject, 0)
for key, object := range rType.Objects {
if all || strings.Contains(strings.ToLower(key), objName) {
// match found, add to the list
results = append(results, object)
}
}
}

// collate the results in to the response data
if raw == nil {
// provide a decorated response
response := make(map[string]RegObjectResponse)

// for each object in the results
for _, object := range results {

// copy the raw attributes
attributes := make([][2]string, len(object.Data))
for ix, attribute := range object.Data {
attributes[ix] = [2]string{attribute.Key, attribute.Value}
}

// construct the backlinks
backlinks := make([]string, len(object.Backlinks))
for ix, object := range object.Backlinks {
backlinks[ix] = object.Ref
}

// add to the response
response[object.Ref] = RegObjectResponse{
Attributes: attributes,
Backlinks: backlinks,
}
}

responseJSON(w, response)

} else {
// provide a response with just the raw registry data
response := make(map[string][][2]string)

// for each object in the results
for _, object := range results {

attributes := make([][2]string, len(object.Data))
response[object.Ref] = attributes

// copy the raw attributes
for ix, attribute := range object.Data {
attributes[ix] = [2]string{attribute.Key, attribute.RawValue}
}
}

responseJSON(w, response)
}

}

//////////////////////////////////////////////////////////////////////////
// end of code

+ 638
- 0
registry.go View File

@@ -0,0 +1,638 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////

package main

//////////////////////////////////////////////////////////////////////////

import (
"bufio"
// "errors"
"fmt"
log "github.com/sirupsen/logrus"
"io/ioutil"
"os"
"os/exec"
"strings"
"time"
)

//////////////////////////////////////////////////////////////////////////
// registry data model

// registry data

// Attributes within Objects
type RegAttribute struct {
Key string
Value string // this is a post-processed, or decorated value
RawValue string // the raw value as read from the registry
}

type RegObject struct {
Ref string // the ref contains the full path for this object
Data []*RegAttribute // the key/value data for this object
Backlinks []*RegObject // other objects that reference this one
}

// types are collections of objects
type RegType struct {
Ref string // full path for this type
Objects map[string]*RegObject // the objects in this type
}

// registry meta data

type RegAttributeSchema struct {
Fields []string
Relations []*RegType
}

type RegTypeSchema struct {
Ref string
Attributes map[string]*RegAttributeSchema
}

// the registry itself

type Registry struct {
Schema map[string]*RegTypeSchema
Types map[string]*RegType
}

// and a variable for the actual data
var RegistryData *Registry

// store the current commit has
var previousCommit string

//////////////////////////////////////////////////////////////////////////
// utility and manipulation functions

// general functions

func RegistryMakePath(t string, o string) string {
return t + "/" + o
}

// attribute functions

// return a pointer to a RegType from a decorated attribute value
func (*RegAttribute) ExtractRegType() *RegType {
return nil
}

// object functions

// return attributes exactly matching a specific key
func (object *RegObject) GetKey(key string) []*RegAttribute {

attributes := make([]*RegAttribute, 0)
for _, a := range object.Data {
if a.Key == key {
attributes = append(attributes, a)
}
}

return attributes
}

// return a single key
func (object *RegObject) GetSingleKey(key string) *RegAttribute {

attributes := object.GetKey(key)
if len(attributes) != 1 {
log.WithFields(log.Fields{
"key": key,
"object": object.Ref,
}).Error("Unable to find unique key in object")

// can't register the object
return nil
}
return attributes[0]
}

// schema functions

// validate a set of attributes against a schema
func (schema *RegTypeSchema) validate(attributes []*RegAttribute) []*RegAttribute {

validated := make([]*RegAttribute, 0, len(attributes))
for _, attribute := range attributes {

// keys beginning with 'x-' are user defined, skip validation
if !strings.HasPrefix(attribute.Key, "x-") {
if schema.Attributes[attribute.Key] == nil {
// couldn't find a schema attribute

log.WithFields(log.Fields{
"key": attribute.Key,
"schema": schema.Ref,
}).Error("Schema validation failed")

// don't add to the validated list
continue
}
}

// all ok
validated = append(validated, attribute)
}

return validated
}

//////////////////////////////////////////////////////////////////////////
// reload the registry

func reloadRegistry(path string) {

log.Debug("Reloading registry")

// r will become the new registry data
registry := &Registry{
Schema: make(map[string]*RegTypeSchema),
Types: make(map[string]*RegType),
}

// bootstrap the schema registry type
registry.Types["schema"] = &RegType{
Ref: "schema",
Objects: make(map[string]*RegObject),
}
registry.loadType("schema", path)

// and parse the schema to get the remaining types
registry.parseSchema()

// now load the remaining types
for _, rType := range registry.Types {
registry.loadType(rType.Ref, path)
}

// mark relationships
registry.decorate()

// swap in the new registry data
RegistryData = registry
}

//////////////////////////////////////////////////////////////////////////
// create and load the raw data for a registry type

func (registry *Registry) loadType(typeName string, path string) {

// the type will already have been created
rType := registry.Types[typeName]

// as will the schema (unless attempting to load the schema itself)
schema := registry.Schema[typeName]

// special case for DNS as the directory
// doesn't match the type name
if typeName == "domain" {
path += "/dns"
} else {
path += "/" + typeName
}

// and load all the objects in this type
rType.loadObjects(schema, path)

}

//////////////////////////////////////////////////////////////////////////
// load all the objects associated with a type

func (rType *RegType) loadObjects(schema *RegTypeSchema, path string) {

entries, err := ioutil.ReadDir(path)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": path,
"type": rType.Ref,
}).Error("Failed to read registry type directory")
return
}

// for each entry in the directory
for _, entry := range entries {

// each file maps to a registry object
if !entry.IsDir() {

filename := entry.Name()
// ignore dotfiles
if !strings.HasPrefix(filename, ".") {

// load the attributes from file
attributes := loadAttributes(path + "/" + filename)

// basic validation of attributes against the schema
// schema may be nil if we are actually loading the schema itself
if schema != nil {
attributes = schema.validate(attributes)
}

// make the object
object := &RegObject{
Ref: RegistryMakePath(rType.Ref, filename),
Data: attributes,
Backlinks: make([]*RegObject, 0),
}

// add to type
rType.Objects[filename] = object
}
}
}

log.WithFields(log.Fields{
"ref": rType.Ref,
"path": path,
"count": len(rType.Objects),
}).Debug("Loaded registry type")

}

//////////////////////////////////////////////////////////////////////////
// read attributes from a file

func loadAttributes(path string) []*RegAttribute {

attributes := make([]*RegAttribute, 0)

// open the file to start reading it
file, err := os.Open(path)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": path,
}).Error("Failed to read attributes from file")
return attributes
}
defer file.Close()

// read the file line by line using the bufio scanner
scanner := bufio.NewScanner(file)
for scanner.Scan() {

line := strings.TrimRight(scanner.Text(), "\r\n")
runes := []rune(line)

// lines starting with '+' denote an empty line
if runes[0] == rune('+') {

// concatenate a \n on to the previous attribute value
attributes[len(attributes)-1].RawValue += "\n"

} else {

// look for a : separator in the first 20 characters
ix := strings.IndexByte(line, ':')
if ix == -1 || ix >= 20 {
// couldn't find one

if len(runes) <= 20 {
// hmmm, the line was shorter than 20 characters
// something is amiss

log.WithFields(log.Fields{
"length": len(runes),
"path": path,
"line": line,
}).Warn("Short line detected")

} else {

// line is a continuation of the previous line, so
// concatenate the value on to the previous attribute value
attributes[len(attributes)-1].RawValue +=
"\n" + string(runes[20:])

}
} else {
// found a key and : separator

// is there actually a value ?
var value string
if len(runes) <= 20 {
// blank value
value = ""
} else {
value = string(runes[20:])
}

// create a new attribute
a := &RegAttribute{
Key: string(runes[:ix]),
RawValue: value,
}
attributes = append(attributes, a)
}
}
}

return attributes
}

//////////////////////////////////////////////////////////////////////////
// parse schema files to extract keys and for attribute relations

func (registry *Registry) parseSchema() {

// for each object in the schema type
for _, object := range registry.Types["schema"].Objects {

// look up the ref attribute
ref := object.GetSingleKey("ref")
if ref == nil {
log.WithFields(log.Fields{
"object": object.Ref,
}).Error("Schema record without ref")

// can't process this object
continue
}

// create the type schema object
typeName := strings.TrimPrefix(ref.RawValue, "dn42.")
typeSchema := &RegTypeSchema{
Ref: typeName,
Attributes: make(map[string]*RegAttributeSchema),
}

// ensure the type exists
rType := registry.Types[typeName]
if rType == nil {
rType := &RegType{
Ref: typeName,
Objects: make(map[string]*RegObject),
}
registry.Types[typeName] = rType
}

// for each key attribute in the schema
attributes := object.GetKey("key")
for _, attribute := range attributes {

// split the value on whitespace
fields := strings.Fields(attribute.RawValue)
keyName := fields[0]

typeSchema.Attributes[keyName] = &RegAttributeSchema{
Fields: fields[1:],
}
}

// register the type schema
registry.Schema[typeName] = typeSchema

}

// scan the fields of each schema attribute to determine relationships
// this needs to be second step to allow pre-creation of the types
for _, typeSchema := range registry.Schema {
for attribName, attribSchema := range typeSchema.Attributes {
for _, field := range attribSchema.Fields {
if strings.HasPrefix(field, "lookup=") {

// the relationships may be a multivalue, separated by ,
rels := strings.Split(strings.
TrimPrefix(field, "lookup="), ",")

// map to a regtype
relations := make([]*RegType, 0, len(rels))
for ix := range rels {
relName := strings.TrimPrefix(rels[ix], "dn42.")
relation := registry.Types[relName]

// log if unable to look up the type
if relation == nil {
// log unless this is the schema def lookup=str '>' [spec]...
if typeSchema.Ref != "schema" {
log.WithFields(log.Fields{
"relation": relName,
"attribute": attribName,
"type": typeSchema.Ref,
}).Error("Relation to type that does not exist")
}

} else {
// store the relationship
relations = append(relations, relation)
}
}

// register the relations
attribSchema.Relations = relations

// assume only 1 lookup= per key
break
}
}
}
}

log.Debug("Schema parsing complete")
}

//////////////////////////////////////////////////////////////////////////
// parse all attributes and decorate them

func (registry *Registry) decorate() {

cattribs := 0
cmatched := 0

// walk each attribute value
for _, rType := range registry.Types {
schema := registry.Schema[rType.Ref]
for _, object := range rType.Objects {
for _, attribute := range object.Data {
cattribs += 1

attribSchema := schema.Attributes[attribute.Key]
// are there relations defined for this attribute ?
// attribSchema may be null if this attribute is user defined (x-*)
if (attribSchema != nil) && attribute.matchRelation(object,
attribSchema.Relations) {
// matched
cmatched += 1
} else {
// no match, just copy the attribute data
attribute.Value = attribute.RawValue
}
}
}
}

log.WithFields(log.Fields{
"attributes": cattribs,
"matched": cmatched,
}).Debug("Decoration complete")

}

//////////////////////////////////////////////////////////////////////////
// match an attribute against schema relations

func (attribute *RegAttribute) matchRelation(parent *RegObject,
relations []*RegType) bool {

// it's not going to match if relations is empty
if relations == nil {
return false
}

// check each relation
for _, relation := range relations {

object := relation.Objects[attribute.RawValue]
if object != nil {
// found a match !

// decorate the attribute value
attribute.Value = fmt.Sprintf("[%s](%s)",
attribute.RawValue, object.Ref)

// and add a back reference to the related object
object.Backlinks = append(object.Backlinks, parent)
return true
}

}

// didn't find anything
return false
}

//////////////////////////////////////////////////////////////////////////
// fetch the current commit hash

func getCommitHash(regDir string, gitPath string) string {

// run git to get the latest commit hash
cmd := exec.Command(gitPath, "log", "-1", "--format=%H")
cmd.Dir = regDir
// execute
out, err := cmd.Output()
if err != nil {
log.WithFields(log.Fields{
"error": err,
"gitPath": gitPath,
"regDir": regDir,
}).Error("Failed to execute git log")
}

return strings.TrimSpace(string(out))
}

//////////////////////////////////////////////////////////////////////////
// refresh the registry

func refreshRegistry(regDir string, gitPath string, pullURL string) {

// run git to get the latest commit hash
cmd := exec.Command(gitPath, "pull", pullURL)
cmd.Dir = regDir
// execute
out, err := cmd.Output()
if err != nil {
log.WithFields(log.Fields{
"error": err,
"gitPath": gitPath,
"regDir": regDir,
"pullURL": pullURL,
}).Error("Failed to execute git log")
}

fmt.Println(string(out))
}

//////////////////////////////////////////////////////////////////////////
// called from main to initialse the registry data and syncing

func InitialiseRegistryData(regDir string, refresh time.Duration,
gitPath string, autoPull bool, pullURL string) {

// validate that the regDir/data path exists
dataPath := regDir + "/data"
regStat, err := os.Stat(dataPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": dataPath,
}).Fatal("Unable to find registry directory")
}

// and it is a directory
if !regStat.IsDir() {
log.WithFields(log.Fields{
"error": err,
"path": dataPath,
}).Fatal("Registry path is not a directory")
}

// check that git exists
_, err = os.Stat(gitPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": gitPath,
}).Fatal("Unable to find git executable")
}

// enforce a minimum update time
minTime := 10 * time.Minute
if refresh < minTime {
log.WithFields(log.Fields{
"interval": refresh,
}).Error("Enforcing minimum update time of 10 minutes")

refresh = minTime
}

// initialise the previous commit hash
// and do initial load from registry
previousCommit = getCommitHash(regDir, gitPath)
reloadRegistry(dataPath)

go func() {

// every refresh interval
for range time.Tick(refresh) {
log.Debug("Refresh Timer")

// automatically try to refresh the registry ?
if autoPull {
refreshRegistry(regDir, gitPath, pullURL)
}

// get the latest hash
currentCommit := getCommitHash(regDir, gitPath)

// has the registry been updated ?
if currentCommit != previousCommit {
log.WithFields(log.Fields{
"current": currentCommit,
"previous": previousCommit,
}).Info("Registry has changed, refresh started")

// refresh
reloadRegistry(dataPath)

// update commit
previousCommit = currentCommit
}

}
}()

}

//////////////////////////////////////////////////////////////////////////
// end of code

+ 55
- 0
static.go View File

@@ -0,0 +1,55 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////

package main

//////////////////////////////////////////////////////////////////////////

import (
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
"net/http"
"os"
)

//////////////////////////////////////////////////////////////////////////
// called from main to initialise the API routing

func InstallStaticRoutes(router *mux.Router, staticPath string) {

// an empty path disables static route serving
if staticPath == "" {
log.Info("Disabling static route serving")
return
}

// validate that the staticPath exists
stat, err := os.Stat(staticPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": staticPath,
}).Fatal("Unable to find static page directory")
}

// and it is a directory
if !stat.IsDir() {
log.WithFields(log.Fields{
"error": err,
"path": staticPath,
}).Fatal("Static path is not a directory")
}

// install a file server for the static route
router.PathPrefix("/").Handler(http.StripPrefix("/",
http.FileServer(http.Dir(staticPath)))).Methods("GET")

log.WithFields(log.Fields{
"path": staticPath,
}).Info("Static route installed")

}

//////////////////////////////////////////////////////////////////////////
// end of code

Loading…
Cancel
Save