burble.dn42/dn42regsrv
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Simon Marsh 2019-02-09 21:15:18 +00:00
commit 66ca94dccc
Signed by: burble
GPG Key ID: 7B9FE8780CFB6593
11 changed files with 1720 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
API.md Normal file
View File

@ -0,0 +1,83 @@
# dn42regsrv API Description
## GET /<file>
If the StaticRoot configuration option points to a readable directory, files from
the directory will be served under /
The git repository contains a sample StaticRoot directory with a simple registry
explorer web app.
## GET /api/registry/
Returns a JSON object, with keys for each registry type and values containing a count
of the number of registry objects for each type.
Example:
```
http://localhost:8042/api/registry/
# sample output
{"as-block":8,"as-set":34,"aut-num":1482,"domain":451,"inet6num":744,"inetnum":1270,"key-cert":7,"mntner":1378,"organisation":275,"person":1387,"registry":4,"role":14,"route":886,"route-set":2,"route6":594,"schema":18,"tinc-key":25,"tinc-keyset":3}
```
## GET /api/registry/<type>?match
Returns a JSON object listing all objects for the matched types.
Keys for the returned object are registry types, the value for each type is an
array of object names
If the match parameter is provided, the <type> is substring matched against
all registry types, otherwise an exact type name is required.
A special type of '*' returns all types and objects in the registry.
Example:
```
http://localhost:8042/api/registry/aut-num # list aut-num objects
http://localhost:8042/api/registry/* # list all types and objects
http://localhost:8042/api/registry/route?match # list route and route6 objects
# sample output
{"role":["ALENAN-DN42","FLHB-ABUSE-DN42","ORG-SHACK-ADMIN-DN42","PACKETPUSHERS-DN42","CCCHB-ABUSE-DN42","ORG-NETRAVNEN-DN42","ORG-SHACK-ABUSE-DN42","MAGLAB-DN42","NIXNODES-DN42","SOURIS-DN42","CCCKC-DN42","NL-ZUID-DN42","ORG-SHACK-TECH-DN42","ORG-YANE-DN42"]}
```
## GET /api/registry/<type>/<object>?match&raw
Return a JSON object with the registry data for each matching object.
The keys for the object are the object paths in the form <type>/<object name>. The values depends on the raw parameter.
if the raw parameter is provided, the returned object consists of a single key 'Attributes'
which will be an array of key/value pairs exactly as held within the registry.
If the raw parameter is not provided, the returned Attributes are decorated with markdown
style links depending the relations defined in the DN42 schema. In addition a
'Backlinks' key is added which provides an array of registry objects that
reference this one.
If the match parameter is provided, the <object> is substring matched against all
objects in the <type>. Matching is case insensitive.
If the match parameter is not provided, an exact, case sensitive object name is required.
A special object of '*' returns all objects in the type
Example:
```
http://localhost:8042/api/registry/domain/burble.dn42?raw # return object in raw format
http://localhost:8042/api/registry/mntner/BURBLE-MNT # return object in decorated format
http://localhost:8042/api/registry/aut-num/2601?match # return all aut-num objects matching 2601
http://localhost:8042/api/registry/schema/* # return all schema objects
# sample output (raw)
{"domain/burble.dn42":[["domain","burble.dn42"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","BURBLE-DN42"],["tech-c","BURBLE-DN42"],["mnt-by","BURBLE-MNT"],["nserver","ns1.burble.dn42 172.20.129.161"],["nserver","ns1.burble.dn42 fd42:4242:2601:ac53::1"],["ds-rdata","61857 13 2 bd35e3efe3325d2029fb652e01604a48b677cc2f44226eeabee54b456c67680c"],["source","DN42"]]}
# sample output (decorated)
{"mntner/BURBLE-MNT":{"Attributes":[["mntner","BURBLE-MNT"],["descr","burble.dn42 https://dn42.burble.com/"],["admin-c","[BURBLE-DN42](person/BURBLE-DN42)"],["tech-c","[BURBLE-DN42](person/BURBLE-DN42)"],["auth","pgp-fingerprint 1C08F282095CCDA432AECC657B9FE8780CFB6593"],["mnt-by","[BURBLE-MNT](mntner/BURBLE-MNT)"],["source","[DN42](registry/DN42)"]],"Backlinks":["as-set/AS4242422601:AS-DOWNSTREAM","as-set/AS4242422601:AS-TRANSIT","inetnum/172.20.129.160_27","person/BURBLE-DN42","route/172.20.129.160_27","inet6num/fd42:4242:2601::_48","mntner/BURBLE-MNT","aut-num/AS4242422601","aut-num/AS4242422602","route6/fd42:4242:2601::_48","domain/collector.dn42","domain/burble.dn42"]}}
```

66
README.md Normal file
View File

@ -0,0 +1,66 @@
# dn42regsrv
A REST API for the DN42 registry, written in Go, to provide a bridge between
interactive applications and the DN42 registry.
## Features
- REST API for querying DN42 registry objects
- Able to decorate objects with relationship information based on SCHEMA type definitions
- Includes a simple webserver for delivering static files which can be used to deliver
basic web applications utilising the API (such as the included DN42 Registry Explorer)
- Automatic pull from the DN42 git repository to keep the registry up to date
- Included responsive web app for exploring the registry
## Building
Requires [git](https://git-scm.com/) and [go](https://golang.org)
```
go get https://git.dn42.us/burble/dn42regsrv
```
## Running
Use --help to view configurable options
```
./dn42regsrv --help
```
The server requires access to a clone of the DN42 registry and for the git executable
to be accessible.
If you want to use the auto pull feature then the registry must
also be writable by the server.
```
cd ${GOROOT}/src/dn42regsrv
git clone http://git.dn42.us/dn42/registry.git
./dn42regsrv --help
./dn42regsrv
```
A sample service file is included for running the server under systemd
## Using
By default the server will be listening on port 8042.
See the [API.md](API.md) file for a detailed description of the API.
## Support
Please feel free to raise issues or create pull requests for the project git repository.
## #ToDo
### Server
- Add WHOIS interface
- Add endpoints for ROA data
- Add attribute searches
### DN42 Registry Explorer Web App
- Add search history and fix going back
- Allow for attribute searches

1
StaticRoot/anchorme.min.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

12
StaticRoot/bootstrap.min.css vendored Normal file
View File

File diff suppressed because one or more lines are too long

BIN
StaticRoot/dn42_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

245
StaticRoot/explorer.js Normal file
View File

@ -0,0 +1,245 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry Explorer
//////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////
// registry-stats component
Vue.component('registry-stats', {
template: '#registry-stats-template',
data() {
return {
state: "loading",
error: "",
types: null,
}
},
methods: {
updateSearch: function(str) {
vm.updateSearch(str)
},
reload: function(event) {
this.types = null,
this.state = "loading"
axios
.get('/api/registry/')
.then(response => {
this.types = response.data
this.state = 'complete'
})
.catch(error => {
this.error = error
this.state = 'error'
console.log(error)
})
}
},
mounted() {
this.reload()
}
})
//////////////////////////////////////////////////////////////////////////
// registry object component
Vue.component('reg-object', {
template: '#reg-object-template',
props: [ 'link' ],
data() {
return { }
},
methods: {
updateSearch: function(str) {
vm.updateSearch(str)
}
},
computed: {
rtype: function() {
var ix = this.link.indexOf("/")
return this.link.substring(0, ix)
},
obj: function() {
var ix = this.link.indexOf("/")
return this.link.substring(ix + 1)
}
}
})
//////////////////////////////////////////////////////////////////////////
// reg-attribute component
Vue.component('reg-attribute', {
template: '#reg-attribute-template',
props: [ 'content' ],
data() {
return { }
},
methods: {
isRegObject: function(str) {
return (this.content.match(/^\[.*?\]\(.*?\)/) != null)
}
},
computed: {
objectLink: function() {
reg = this.content.match(/^\[(.*?)\]\((.*?)\)/)
return reg[2]
},
decorated: function() {
return anchorme(this.content, {
truncate: 40,
ips: false,
attributes: [ { name: "target", value: "_blank" } ]
})
}
}
})
//////////////////////////////////////////////////////////////////////////
// construct a search URL from a search term
function matchObjects(objects, rtype, term) {
var results = [ ]
for (const obj in objects) {
var s = objects[obj].toLowerCase()
var pos = s.indexOf(term)
if (pos != -1) {
if ((pos == 0) && (s == term)) {
// exact match, return just this result
return [[ rtype, objects[obj] ]]
}
results.push([ rtype, objects[obj] ])
}
}
return results
}
function searchFilter(index, term) {
var results = [ ]
// comparisons are lowercase
term = term.toLowerCase()
// includes a '/' ? search only in that type
var slash = term.indexOf('/')
if (slash != -1) {
var rtype = term.substring(0, slash)
var term = term.substring(slash + 1)
objects = index[rtype]
if (objects != null) {
results = matchObjects(objects, rtype, term)
}
}
else {
// walk though the entire index
for (const rtype in index) {
results = results.concat(matchObjects(index[rtype], rtype, term))
}
}
return results
}
//////////////////////////////////////////////////////////////////////////
// main application
// application data
var appData = {
searchInput: '',
searchTimeout: 0,
state: '',
debug: "",
index: null,
filtered: null,
result: null
}
// methods
var appMethods = {
loadIndex: function(event) {
axios
.get('/api/registry/*')
.then(response => {
this.index = response.data
})
.catch(error => {
// what to do here ?
console.log(error)
})
},
// called on every search input change
debounceSearchInput: function(value) {
if (this.search_timeout) {
clearTimeout(this.search_timeout)
}
// reset if searchbox is empty
if (value == "") {
this.state = ""
this.searchInput = ""
this.filtered = null
this.results = null
return
}
this.search_timeout =
setTimeout(this.updateSearch.bind(this,value),500)
},
// called after the search input has been debounced
updateSearch: function(value) {
this.searchInput = value
this.filtered = searchFilter(this.index, value)
if (this.filtered.length == 0) {
this.state = "noresults"
}
else if (this.filtered.length == 1) {
this.state = "loading"
var details = this.filtered[0]
query = '/api/registry/' + details[0] + '/' + details[1]
axios
.get(query)
.then(response => {
this.state = 'result'
this.result = response.data
})
.catch(error => {
this.error = error
this.state = 'error'
})
}
else {
this.state = "resultlist"
this.result = this.filtered
}
}
}
// intialise Vue instance
var vm = new Vue({
el: '#explorer',
data: appData,
methods: appMethods,
mounted() {
this.loadIndex()
}
})
//////////////////////////////////////////////////////////////////////////
// end of code

184
StaticRoot/index.html Normal file
View File

@ -0,0 +1,184 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="bootstrap.min.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
<style>
.material-icons { display:inline-flex;vertical-align:middle }
.table th, .table td { padding: 0.2rem; border: none }
pre { margin-bottom: 0px }
.regref span {
padding: 0.3em 1em 0.3em 1em; margin: 0.4em;
white-space: nowrap; display:inline-block;
}
a { cursor: pointer }
</style>
<title>DN42 Registry Explorer</title>
</head>
<body style="box-shadow: inset 0 2em 10em rgba(0,0,0,0.4); min-height: 100vh">
<div id="explorer">
<nav class="navbar navbar-fixed-top navbar-expand-md navbar-dark bg-dark">
<form class="form-inline" id="SearchForm">
<input v-bind:value="searchInput"
v-on:input="debounceSearchInput($event.target.value)"
class="form-control-lg" size="30" type="search"
placeholder="Search the registry" aria-label="Search"/>
</form>
<div class="collapse navbar-collapse w-100 ml-auto">
<div class="ml-auto"><a class="navbar-brand"
href="/">Registry Explorer</a>&nbsp;<a class="pull-right navbar-brand"
href="https://dn42.us/"><img src="/dn42_logo.png" width="173"
height="60"/></a></div>
</nav>
<div style="padding: 1em">
<div v-show="searchInput == ''">
<div class="jumbotron">
<h1>DN42 Registry Explorer</h1>
<p class="lead">Just start typing in the search box to start searching the registry</p>
<hr/>
<p>
<p>Search Tips</p>
<ul>
<li>Searches are case independent
<li>No need to hit enter, searches will start immediately
<li>Prefixing the search by a registry type followed by / will narrow the search to
just that type (e.g. <a v-on:click="updateSearch('domain/.dn42')"
class="text-success">domain/.dn42</a>&nbsp;)
<li>Searching for <b>type/</b> will return all the objects for that type (e.g.
<a v-on:click="updateSearch('schema/')" class="text-success">schema/</a>&nbsp;)
<li>A blank search box will return you to these instructions
<li>Searches are made on object names; searching the content of objects
is not supported (yet!).
<li>Going back (or any search history) is also not supported yet.
</ul>
<hr/>
<p>The registry explorer is a simple web app using
<a href="https://git.dn42.us/burble/dn42regsrv">dn42regsrv</a>;
a REST API for the DN42 registry built with <a href="https://golang.org/">Go</a>
</div>
<registry-stats/>
</div>
<section v-if="state == 'loading'">
<div class="alert alert-info" role="alert">
Loading data ...
</div>
</section>
<section v-else-if="state == 'error'">
<div class="alert alert-primary clearfix" role="alert">
An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
</div>
</section>
<section v-else-if="state == 'noresults'">
<h2>Searching for "{{ searchInput }}" ...</h2>
<div class="alert alert-dark" role="alert">
Sorry, no results found
</div>
</section>
<section v-else-if="state == 'resultlist'">
<h2>Listing results for "{{ searchInput }}" ...</h2>
<div class="container d-flex flex-row flex-wrap">
<div style="text-align: center">
<span v-for="value in result" style="margin: 0.5em 1em 0.5em 1em; display:inline-block">
<reg-object v-bind:link="value[0] + '/' + value[1]"></reg-object>
</span>
</div>
</div>
</section>
<section v-else-if="state == 'result'">
<div v-for="(val, key) in result">
<h2><reg-object v-bind:link="key"></reg-object></h2>
<div style="padding-left: 2em">
<table class="table">
<thead>
<tr><th scope="col">Key</th><th scope="col">Value</th></tr>
</thead>
<tbody>
<tr v-for="a in val.Attributes">
<th scope="row" class="text-primary" style="white-space:nowrap">{{ a[0] }}</th>
<td><reg-attribute v-bind:content="a[1]"></reg-attribute></td>
</tr>
</tbody>
</table>
</div>
<section v-if="val.Backlinks.length != 0">
<p>Referenced by</p>
<div style="padding-left: 2em">
<table class="table">
<tbody>
<tr v-for="r in val.Backlinks">
<td><reg-object v-bind:link="r"></reg-object></td>
</tr>
</tbody>
</table>
</div>
</section>
</div>
</section>
</div>
</div>
<footer class="page-footer font-small">
<div style="margin-top: 20px; padding: 5px">
<a href="https://git.dn42.us/burble/dn42regsrv">Source Code</a>.
Powered by
<a href="https://getbootstrap.com/">Bootstrap</a>,
<a href="https://vuejs.org">Vue.js</a>,
<a href="https://github.com/axios/axios">axios</a>,
<a href="http://alexcorvi.github.io/anchorme.js/">Anchorme</a>.
</div>
</footer>
</div>
<script type="text/x-template" id="registry-stats-template">
<div class="container d-flex flex-column w-75">
<h5>Registry Stats</h5>
<section v-if="state == 'error'">
<div class="alert alert-primary clearfix" role="alert">
An error recurred whilst retrieving data <button type="button" class="float-right btn btn-warning" v-on:click="reload"><span class="material-icons">refresh</span>&nbsp;Refresh</button>
</div>
</section>
<section v-else-if="state == 'loading'">
<div class="alert alert-info" role="alert">
Loading data ...
</div>
</section>
<section v-else>
<p style="padding:1em;text-align:center">
<span v-for="(value, key) in types" style="margin-left:0.5em;margin-right:0.5em;white-space:nowrap;display:inline-block"><a v-on:click="updateSearch(key + '/')" class="text-success">{{ key }}</a>:&nbsp;<b>{{ value }}</b>&nbsp;records</span>
</p>
</section>
</div>
</script>
<script type="text/x-template" id="reg-object-template">
<span class="regref"><a v-on:click="updateSearch(rtype + '/' + obj)" class="text-success"
style="margin-right: 0.4em">{{ obj }}</a>&nbsp;<span
class="badge badge-pill badge-dark text-muted">{{ rtype }}</span></span></span>
</script>
<script type="text/x-template" id="reg-attribute-template">
<span style="word-break: break-all">
<reg-object v-if="isRegObject()" v-bind:link="objectLink"></reg-object>
<span v-else class="text-monospace" v-html="decorated"></span>
</span>
</script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js" integrity="sha384-B0UglyR+jN6CkvvICOB2joaf5I4l3gm9GU6Hc1og6Ls7i6U/mkkaduKaBhlAXv9k" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/vue@2.6.2/dist/vue.js"></script>
<!-- <script src="https://cdn.jsdelivr.net/npm/vue@2.6.2/dist/vue.min.js"></script> -->
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="anchorme.min.js"></script>
<script src="explorer.js"></script>
</body>
</html>

169
dn42regsrv.go Normal file
View File

@ -0,0 +1,169 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////
package main
//////////////////////////////////////////////////////////////////////////
import (
"context"
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
flag "github.com/spf13/pflag"
"net/http"
"os"
"os/signal"
"time"
)
//////////////////////////////////////////////////////////////////////////
// list of API endpoints
type InitEndpointFunc = func(route *mux.Router)
var apiEndpoints = make([]InitEndpointFunc, 0)
func RegisterAPIEndpoint(f InitEndpointFunc) {
apiEndpoints = append(apiEndpoints, f)
}
//////////////////////////////////////////////////////////////////////////
// utility function to set the log level
func setLogLevel(levelStr string) {
if level, err := log.ParseLevel(levelStr); err != nil {
// failed to set the level
// set a sensible default and, of course, log the error
log.SetLevel(log.InfoLevel)
log.WithFields(log.Fields{
"loglevel": levelStr,
"error": err,
}).Error("Failed to set requested log level")
} else {
// set the requested level
log.SetLevel(level)
}
}
//////////////////////////////////////////////////////////////////////////
// http request logger
func requestLogger(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
log.WithFields(log.Fields{
"method": r.Method,
"URL": r.URL.String(),
"Remote": r.RemoteAddr,
}).Debug("HTTP Request")
next.ServeHTTP(w, r)
})
}
//////////////////////////////////////////////////////////////////////////
// everything starts here
func main() {
// set a default log level, so that logging can be used immediately
// the level will be overidden later on once the command line
// options are loaded
log.SetLevel(log.InfoLevel)
log.Info("DN42 Registry API Server Starting")
// declare cmd line options
var (
logLevel = flag.StringP("LogLevel", "l", "Info", "Log level")
regDir = flag.StringP("RegDir", "d", "registry", "Registry data directory")
bindAddress = flag.StringP("BindAddress", "b", "[::]:8042", "Server bind address")
staticRoot = flag.StringP("StaticRoot", "s", "StaticRoot", "Static page directory")
refreshInterval = flag.StringP("Refresh", "i", "60m", "Refresh interval")
gitPath = flag.StringP("GitPath", "g", "/usr/bin/git", "Path to git executable")
autoPull = flag.BoolP("AutoPull", "a", true, "Automatically pull the registry")
pullURL = flag.StringP("PullURL", "p", "origin", "URL to auto pull")
)
flag.Parse()
// now initialise logging properly based on the cmd line options
setLogLevel(*logLevel)
// parse the refreshInterval and start data collection
interval, err := time.ParseDuration(*refreshInterval)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"interval": *refreshInterval,
}).Fatal("Unable to parse registry refresh interval")
}
InitialiseRegistryData(*regDir, interval,
*gitPath, *autoPull, *pullURL)
// initialise router
router := mux.NewRouter()
// log all access
router.Use(requestLogger)
// initialise API routes
subr := router.PathPrefix("/api").Subrouter()
for _, epInit := range apiEndpoints {
epInit(subr)
}
// initialise static routes
InstallStaticRoutes(router, *staticRoot)
// initialise http server
server := &http.Server{
Addr: *bindAddress,
WriteTimeout: time.Second * 15,
ReadTimeout: time.Second * 15,
IdleTimeout: time.Second * 60,
Handler: router,
}
// run the server in a non-blocking goroutine
log.WithFields(log.Fields{
"BindAddress": *bindAddress,
}).Info("Starting server")
go func() {
if err := server.ListenAndServe(); err != nil {
log.WithFields(log.Fields{
"error": err,
"BindAddress": *bindAddress,
}).Fatal("Unable to start server")
}
}()
// graceful shutdown via SIGINT (^C)
csig := make(chan os.Signal, 1)
signal.Notify(csig, os.Interrupt)
// and block
<-csig
log.Info("Server shutting down")
// deadline for server to shutdown
ctx, cancel := context.WithTimeout(context.Background(), 10)
defer cancel()
// shutdown the server
server.Shutdown(ctx)
// nothing left to do
log.Info("Shutdown complete, all done")
os.Exit(0)
}
//////////////////////////////////////////////////////////////////////////
// end of code

267
regapi.go Normal file
View File

@ -0,0 +1,267 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////
package main
//////////////////////////////////////////////////////////////////////////
import (
"encoding/json"
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
"net/http"
"strings"
// "time"
)
//////////////////////////////////////////////////////////////////////////
// register the api
func init() {
RegisterAPIEndpoint(InitRegAPI)
}
//////////////////////////////////////////////////////////////////////////
// called from main to initialise the API routing
func InitRegAPI(router *mux.Router) {
s := router.
Methods("GET").
PathPrefix("/registry").
Subrouter()
s.HandleFunc("/", regRootHandler)
//s.HandleFunc("/.schema", rTypeListHandler)
//s.HandleFunc("/.meta/", rTypeListHandler)
s.HandleFunc("/{type}", regTypeHandler)
s.HandleFunc("/{type}/{object}", regObjectHandler)
log.Info("Registry API installed")
}
//////////////////////////////////////////////////////////////////////////
// handler utility funcs
func responseJSON(w http.ResponseWriter, v interface{}) {
// for response time testing
//time.Sleep(time.Second)
// marshal the JSON string
data, err := json.Marshal(v)
if err != nil {
log.WithFields(log.Fields{
"error": err,
}).Error("Failed to marshal JSON")
}
// write back to http handler
w.Header().Set("Content-Type", "application/json")
w.Write(data)
}
//////////////////////////////////////////////////////////////////////////
// root handler, lists all types within the registry
func regRootHandler(w http.ResponseWriter, r *http.Request) {
response := make(map[string]int)
for _, rType := range RegistryData.Types {
response[rType.Ref] = len(rType.Objects)
}
responseJSON(w, response)
}
//////////////////////////////////////////////////////////////////////////
// type handler returns list of objects that match the type
func regTypeHandler(w http.ResponseWriter, r *http.Request) {
// request parameters
vars := mux.Vars(r)
query := r.URL.Query()
typeName := vars["type"] // type name to list
match := query["match"] // single query or match
// special case to return all types
all := false
if typeName == "*" {
match = []string{}
all = true
}
// results will hold the types to return
var results []*RegType
// check match type
if match == nil {
// exact match
// check the type object exists
rType := RegistryData.Types[typeName]
if rType == nil {
http.Error(w, "No types matching '"+typeName+"' found", http.StatusNotFound)
return
}
// return just a single result
results = []*RegType{rType}
} else {
// substring match
// comparisons are lower case
typeName = strings.ToLower(typeName)
// walk through the types and filter to the results list
results = make([]*RegType, 0)
for key, rType := range RegistryData.Types {
if all || strings.Contains(strings.ToLower(key), typeName) {
// match found, add to the list
results = append(results, rType)
}
}
}
// construct the response
response := make(map[string][]string)
for _, rType := range results {
objects := make([]string, 0, len(rType.Objects))
for key := range rType.Objects {
objects = append(objects, key)
}
response[rType.Ref] = objects
}
responseJSON(w, response)
}
//////////////////////////////////////////////////////////////////////////
// object handler returns object data
// per object response structure
type RegObjectResponse struct {
Attributes [][2]string
Backlinks []string
}
func regObjectHandler(w http.ResponseWriter, r *http.Request) {
// request parameters
vars := mux.Vars(r)
query := r.URL.Query()
typeName := vars["type"] // object type
objName := vars["object"] // object name or match
match := query["match"] // single query or match
raw := query["raw"] // raw or decorated results
// special case to return all objects
all := false
if objName == "*" {
match = []string{}
all = true
}
// verify the type exists
rType := RegistryData.Types[typeName]
if rType == nil {
http.Error(w, "No types matching '"+typeName+"' found",
http.StatusNotFound)
return
}
// results will hold the objects to return
var results []*RegObject
// check match type
if match == nil {
// exact match
// check the object exists
object := rType.Objects[objName]
if object == nil {
http.Error(w, "No objects matching '"+objName+"' found",
http.StatusNotFound)
return
}
// then just create a results list with one object
results = []*RegObject{object}
} else {
// substring matching
// comparisons are lower case
objName = strings.ToLower(objName)
// walk through the type objects and filter to the results list
results = make([]*RegObject, 0)
for key, object := range rType.Objects {
if all || strings.Contains(strings.ToLower(key), objName) {
// match found, add to the list
results = append(results, object)
}
}
}
// collate the results in to the response data
if raw == nil {
// provide a decorated response
response := make(map[string]RegObjectResponse)
// for each object in the results
for _, object := range results {
// copy the raw attributes
attributes := make([][2]string, len(object.Data))
for ix, attribute := range object.Data {
attributes[ix] = [2]string{attribute.Key, attribute.Value}
}
// construct the backlinks
backlinks := make([]string, len(object.Backlinks))
for ix, object := range object.Backlinks {
backlinks[ix] = object.Ref
}
// add to the response
response[object.Ref] = RegObjectResponse{
Attributes: attributes,
Backlinks: backlinks,
}
}
responseJSON(w, response)
} else {
// provide a response with just the raw registry data
response := make(map[string][][2]string)
// for each object in the results
for _, object := range results {
attributes := make([][2]string, len(object.Data))
response[object.Ref] = attributes
// copy the raw attributes
for ix, attribute := range object.Data {
attributes[ix] = [2]string{attribute.Key, attribute.RawValue}
}
}
responseJSON(w, response)
}
}
//////////////////////////////////////////////////////////////////////////
// end of code

638
registry.go Normal file
View File

@ -0,0 +1,638 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////
package main
//////////////////////////////////////////////////////////////////////////
import (
"bufio"
// "errors"
"fmt"
log "github.com/sirupsen/logrus"
"io/ioutil"
"os"
"os/exec"
"strings"
"time"
)
//////////////////////////////////////////////////////////////////////////
// registry data model
// registry data
// Attributes within Objects
type RegAttribute struct {
Key string
Value string // this is a post-processed, or decorated value
RawValue string // the raw value as read from the registry
}
type RegObject struct {
Ref string // the ref contains the full path for this object
Data []*RegAttribute // the key/value data for this object
Backlinks []*RegObject // other objects that reference this one
}
// types are collections of objects
type RegType struct {
Ref string // full path for this type
Objects map[string]*RegObject // the objects in this type
}
// registry meta data
type RegAttributeSchema struct {
Fields []string
Relations []*RegType
}
type RegTypeSchema struct {
Ref string
Attributes map[string]*RegAttributeSchema
}
// the registry itself
type Registry struct {
Schema map[string]*RegTypeSchema
Types map[string]*RegType
}
// and a variable for the actual data
var RegistryData *Registry
// store the current commit has
var previousCommit string
//////////////////////////////////////////////////////////////////////////
// utility and manipulation functions
// general functions
func RegistryMakePath(t string, o string) string {
return t + "/" + o
}
// attribute functions
// return a pointer to a RegType from a decorated attribute value
func (*RegAttribute) ExtractRegType() *RegType {
return nil
}
// object functions
// return attributes exactly matching a specific key
func (object *RegObject) GetKey(key string) []*RegAttribute {
attributes := make([]*RegAttribute, 0)
for _, a := range object.Data {
if a.Key == key {
attributes = append(attributes, a)
}
}
return attributes
}
// return a single key
func (object *RegObject) GetSingleKey(key string) *RegAttribute {
attributes := object.GetKey(key)
if len(attributes) != 1 {
log.WithFields(log.Fields{
"key": key,
"object": object.Ref,
}).Error("Unable to find unique key in object")
// can't register the object
return nil
}
return attributes[0]
}
// schema functions
// validate a set of attributes against a schema
func (schema *RegTypeSchema) validate(attributes []*RegAttribute) []*RegAttribute {
validated := make([]*RegAttribute, 0, len(attributes))
for _, attribute := range attributes {
// keys beginning with 'x-' are user defined, skip validation
if !strings.HasPrefix(attribute.Key, "x-") {
if schema.Attributes[attribute.Key] == nil {
// couldn't find a schema attribute
log.WithFields(log.Fields{
"key": attribute.Key,
"schema": schema.Ref,
}).Error("Schema validation failed")
// don't add to the validated list
continue
}
}
// all ok
validated = append(validated, attribute)
}
return validated
}
//////////////////////////////////////////////////////////////////////////
// reload the registry
func reloadRegistry(path string) {
log.Debug("Reloading registry")
// r will become the new registry data
registry := &Registry{
Schema: make(map[string]*RegTypeSchema),
Types: make(map[string]*RegType),
}
// bootstrap the schema registry type
registry.Types["schema"] = &RegType{
Ref: "schema",
Objects: make(map[string]*RegObject),
}
registry.loadType("schema", path)
// and parse the schema to get the remaining types
registry.parseSchema()
// now load the remaining types
for _, rType := range registry.Types {
registry.loadType(rType.Ref, path)
}
// mark relationships
registry.decorate()
// swap in the new registry data
RegistryData = registry
}
//////////////////////////////////////////////////////////////////////////
// create and load the raw data for a registry type
func (registry *Registry) loadType(typeName string, path string) {
// the type will already have been created
rType := registry.Types[typeName]
// as will the schema (unless attempting to load the schema itself)
schema := registry.Schema[typeName]
// special case for DNS as the directory
// doesn't match the type name
if typeName == "domain" {
path += "/dns"
} else {
path += "/" + typeName
}
// and load all the objects in this type
rType.loadObjects(schema, path)
}
//////////////////////////////////////////////////////////////////////////
// load all the objects associated with a type
func (rType *RegType) loadObjects(schema *RegTypeSchema, path string) {
entries, err := ioutil.ReadDir(path)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": path,
"type": rType.Ref,
}).Error("Failed to read registry type directory")
return
}
// for each entry in the directory
for _, entry := range entries {
// each file maps to a registry object
if !entry.IsDir() {
filename := entry.Name()
// ignore dotfiles
if !strings.HasPrefix(filename, ".") {
// load the attributes from file
attributes := loadAttributes(path + "/" + filename)
// basic validation of attributes against the schema
// schema may be nil if we are actually loading the schema itself
if schema != nil {
attributes = schema.validate(attributes)
}
// make the object
object := &RegObject{
Ref: RegistryMakePath(rType.Ref, filename),
Data: attributes,
Backlinks: make([]*RegObject, 0),
}
// add to type
rType.Objects[filename] = object
}
}
}
log.WithFields(log.Fields{
"ref": rType.Ref,
"path": path,
"count": len(rType.Objects),
}).Debug("Loaded registry type")
}
//////////////////////////////////////////////////////////////////////////
// read attributes from a file
func loadAttributes(path string) []*RegAttribute {
attributes := make([]*RegAttribute, 0)
// open the file to start reading it
file, err := os.Open(path)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": path,
}).Error("Failed to read attributes from file")
return attributes
}
defer file.Close()
// read the file line by line using the bufio scanner
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := strings.TrimRight(scanner.Text(), "\r\n")
runes := []rune(line)
// lines starting with '+' denote an empty line
if runes[0] == rune('+') {
// concatenate a \n on to the previous attribute value
attributes[len(attributes)-1].RawValue += "\n"
} else {
// look for a : separator in the first 20 characters
ix := strings.IndexByte(line, ':')
if ix == -1 || ix >= 20 {
// couldn't find one
if len(runes) <= 20 {
// hmmm, the line was shorter than 20 characters
// something is amiss
log.WithFields(log.Fields{
"length": len(runes),
"path": path,
"line": line,
}).Warn("Short line detected")
} else {
// line is a continuation of the previous line, so
// concatenate the value on to the previous attribute value
attributes[len(attributes)-1].RawValue +=
"\n" + string(runes[20:])
}
} else {
// found a key and : separator
// is there actually a value ?
var value string
if len(runes) <= 20 {
// blank value
value = ""
} else {
value = string(runes[20:])
}
// create a new attribute
a := &RegAttribute{
Key: string(runes[:ix]),
RawValue: value,
}
attributes = append(attributes, a)
}
}
}
return attributes
}
//////////////////////////////////////////////////////////////////////////
// parse schema files to extract keys and for attribute relations
func (registry *Registry) parseSchema() {
// for each object in the schema type
for _, object := range registry.Types["schema"].Objects {
// look up the ref attribute
ref := object.GetSingleKey("ref")
if ref == nil {
log.WithFields(log.Fields{
"object": object.Ref,
}).Error("Schema record without ref")
// can't process this object
continue
}
// create the type schema object
typeName := strings.TrimPrefix(ref.RawValue, "dn42.")
typeSchema := &RegTypeSchema{
Ref: typeName,
Attributes: make(map[string]*RegAttributeSchema),
}
// ensure the type exists
rType := registry.Types[typeName]
if rType == nil {
rType := &RegType{
Ref: typeName,
Objects: make(map[string]*RegObject),
}
registry.Types[typeName] = rType
}
// for each key attribute in the schema
attributes := object.GetKey("key")
for _, attribute := range attributes {
// split the value on whitespace
fields := strings.Fields(attribute.RawValue)
keyName := fields[0]
typeSchema.Attributes[keyName] = &RegAttributeSchema{
Fields: fields[1:],
}
}
// register the type schema
registry.Schema[typeName] = typeSchema
}
// scan the fields of each schema attribute to determine relationships
// this needs to be second step to allow pre-creation of the types
for _, typeSchema := range registry.Schema {
for attribName, attribSchema := range typeSchema.Attributes {
for _, field := range attribSchema.Fields {
if strings.HasPrefix(field, "lookup=") {
// the relationships may be a multivalue, separated by ,
rels := strings.Split(strings.
TrimPrefix(field, "lookup="), ",")
// map to a regtype
relations := make([]*RegType, 0, len(rels))
for ix := range rels {
relName := strings.TrimPrefix(rels[ix], "dn42.")
relation := registry.Types[relName]
// log if unable to look up the type
if relation == nil {
// log unless this is the schema def lookup=str '>' [spec]...
if typeSchema.Ref != "schema" {
log.WithFields(log.Fields{
"relation": relName,
"attribute": attribName,
"type": typeSchema.Ref,
}).Error("Relation to type that does not exist")
}
} else {
// store the relationship
relations = append(relations, relation)
}
}
// register the relations
attribSchema.Relations = relations
// assume only 1 lookup= per key
break
}
}
}
}
log.Debug("Schema parsing complete")
}
//////////////////////////////////////////////////////////////////////////
// parse all attributes and decorate them
func (registry *Registry) decorate() {
cattribs := 0
cmatched := 0
// walk each attribute value
for _, rType := range registry.Types {
schema := registry.Schema[rType.Ref]
for _, object := range rType.Objects {
for _, attribute := range object.Data {
cattribs += 1
attribSchema := schema.Attributes[attribute.Key]
// are there relations defined for this attribute ?
// attribSchema may be null if this attribute is user defined (x-*)
if (attribSchema != nil) && attribute.matchRelation(object,
attribSchema.Relations) {
// matched
cmatched += 1
} else {
// no match, just copy the attribute data
attribute.Value = attribute.RawValue
}
}
}
}
log.WithFields(log.Fields{
"attributes": cattribs,
"matched": cmatched,
}).Debug("Decoration complete")
}
//////////////////////////////////////////////////////////////////////////
// match an attribute against schema relations
func (attribute *RegAttribute) matchRelation(parent *RegObject,
relations []*RegType) bool {
// it's not going to match if relations is empty
if relations == nil {
return false
}
// check each relation
for _, relation := range relations {
object := relation.Objects[attribute.RawValue]
if object != nil {
// found a match !
// decorate the attribute value
attribute.Value = fmt.Sprintf("[%s](%s)",
attribute.RawValue, object.Ref)
// and add a back reference to the related object
object.Backlinks = append(object.Backlinks, parent)
return true
}
}
// didn't find anything
return false
}
//////////////////////////////////////////////////////////////////////////
// fetch the current commit hash
func getCommitHash(regDir string, gitPath string) string {
// run git to get the latest commit hash
cmd := exec.Command(gitPath, "log", "-1", "--format=%H")
cmd.Dir = regDir
// execute
out, err := cmd.Output()
if err != nil {
log.WithFields(log.Fields{
"error": err,
"gitPath": gitPath,
"regDir": regDir,
}).Error("Failed to execute git log")
}
return strings.TrimSpace(string(out))
}
//////////////////////////////////////////////////////////////////////////
// refresh the registry
func refreshRegistry(regDir string, gitPath string, pullURL string) {
// run git to get the latest commit hash
cmd := exec.Command(gitPath, "pull", pullURL)
cmd.Dir = regDir
// execute
out, err := cmd.Output()
if err != nil {
log.WithFields(log.Fields{
"error": err,
"gitPath": gitPath,
"regDir": regDir,
"pullURL": pullURL,
}).Error("Failed to execute git log")
}
fmt.Println(string(out))
}
//////////////////////////////////////////////////////////////////////////
// called from main to initialse the registry data and syncing
func InitialiseRegistryData(regDir string, refresh time.Duration,
gitPath string, autoPull bool, pullURL string) {
// validate that the regDir/data path exists
dataPath := regDir + "/data"
regStat, err := os.Stat(dataPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": dataPath,
}).Fatal("Unable to find registry directory")
}
// and it is a directory
if !regStat.IsDir() {
log.WithFields(log.Fields{
"error": err,
"path": dataPath,
}).Fatal("Registry path is not a directory")
}
// check that git exists
_, err = os.Stat(gitPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": gitPath,
}).Fatal("Unable to find git executable")
}
// enforce a minimum update time
minTime := 10 * time.Minute
if refresh < minTime {
log.WithFields(log.Fields{
"interval": refresh,
}).Error("Enforcing minimum update time of 10 minutes")
refresh = minTime
}
// initialise the previous commit hash
// and do initial load from registry
previousCommit = getCommitHash(regDir, gitPath)
reloadRegistry(dataPath)
go func() {
// every refresh interval
for range time.Tick(refresh) {
log.Debug("Refresh Timer")
// automatically try to refresh the registry ?
if autoPull {
refreshRegistry(regDir, gitPath, pullURL)
}
// get the latest hash
currentCommit := getCommitHash(regDir, gitPath)
// has the registry been updated ?
if currentCommit != previousCommit {
log.WithFields(log.Fields{
"current": currentCommit,
"previous": previousCommit,
}).Info("Registry has changed, refresh started")
// refresh
reloadRegistry(dataPath)
// update commit
previousCommit = currentCommit
}
}
}()
}
//////////////////////////////////////////////////////////////////////////
// end of code

55
static.go Normal file
View File

@ -0,0 +1,55 @@
//////////////////////////////////////////////////////////////////////////
// DN42 Registry API Server
//////////////////////////////////////////////////////////////////////////
package main
//////////////////////////////////////////////////////////////////////////
import (
"github.com/gorilla/mux"
log "github.com/sirupsen/logrus"
"net/http"
"os"
)
//////////////////////////////////////////////////////////////////////////
// called from main to initialise the API routing
func InstallStaticRoutes(router *mux.Router, staticPath string) {
// an empty path disables static route serving
if staticPath == "" {
log.Info("Disabling static route serving")
return
}
// validate that the staticPath exists
stat, err := os.Stat(staticPath)
if err != nil {
log.WithFields(log.Fields{
"error": err,
"path": staticPath,
}).Fatal("Unable to find static page directory")
}
// and it is a directory
if !stat.IsDir() {
log.WithFields(log.Fields{
"error": err,
"path": staticPath,
}).Fatal("Static path is not a directory")
}
// install a file server for the static route
router.PathPrefix("/").Handler(http.StripPrefix("/",
http.FileServer(http.Dir(staticPath)))).Methods("GET")
log.WithFields(log.Fields{
"path": staticPath,
}).Info("Static route installed")
}
//////////////////////////////////////////////////////////////////////////
// end of code