diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..bb70059 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,28 @@ +# How to Contribute + +We'd love to accept your patches and contributions to this project. There are +just a few small guidelines you need to follow. + +## Contributor License Agreement + +Contributions to this project must be accompanied by a Contributor License +Agreement. You (or your employer) retain the copyright to your contribution, +this simply gives us permission to use and redistribute your contributions as +part of the project. Head over to to see +your current agreements on file or to sign a new one. + +You generally only need to submit a CLA once, so if you've already submitted one +(even if it was for a different project), you probably don't need to do it +again. + +## Code reviews + +All submissions, including submissions by project members, require review. We +use GitHub pull requests for this purpose. Consult +[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more +information on using pull requests. + +## Community Guidelines + +This project follows [Google's Open Source Community +Guidelines](https://opensource.google.com/conduct/). diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..deb8c02 --- /dev/null +++ b/README.md @@ -0,0 +1,78 @@ +# Alertmanager IRC Relay + +Alertmanager IRC Relay is a bot relaying [Prometheus](https://prometheus.io/) alerts to IRC. +Alerts are received from Prometheus using +[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-) +and are relayed to an IRC channel. + +### Configuring and running the bot + +To configure and run the bot you need to create a YAML configuration file and +pass it to the service. Running the service without a configuration will use +the default test values and connect to a default IRC channel, which you +probably do not want to do. + +Example configuration: +``` +# Start the HTTP server receiving alerts from Prometheus Webhook binding to +# this host/port. +# +http_host: localhost +http_port: 8000 + +# Connect to this IRC host/port. +# +# Note: SSL is enabled by default, use "irc_use_ssl: no" to disable. +irc_host: irc.example.com +irc_port: 7000 + +# Use this IRC nickname. +irc_nickname: myalertbot +# Password used to identify with NickServ +irc_nickname_password: mynickserv_key +# Use this IRC real name +irc_realname: myrealname + +# Optionally pre-join certain channels. +# +# Note: If an alert is sent to a non # pre-joined channel the bot will join +# that channel anyway before sending the notice. Of course this cannot work +# with password-protected channels. +irc_channels: + - name: "#mychannel" + - name: "#myprivatechannel" + password: myprivatechannel_key + +# Define how IRC messages should be sent. +# +# Send only one notice when webhook data is received. +# Note: By default a notice is sent for each alert in the webhook data. +notice_once_per_alert_group: no + +# Define how IRC messages should be formatted. +# +# The formatting is based on golang's text/template . +notice_template: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}" +# Note: When sending only one notice per alert group the default +# notice_template is set to +# "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}" +``` + +Running the bot (assuming *$GOPATH* and *$PATH* are properly setup for go): +``` +$ go install github.com/google/alertmanager-irc-relay +$ alertmanager-irc-relay --config /path/to/your/config/file +``` + +### Prometheus configuration + +Prometheus can be configured following the official +[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-) +documentation. The `url` must specify the IRC channel name that alerts should +be sent to: +``` +send_resolved: false +url: http://localhost:8000/mychannel +``` + + diff --git a/backoff.go b/backoff.go new file mode 100644 index 0000000..5873365 --- /dev/null +++ b/backoff.go @@ -0,0 +1,102 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "log" + "math" + "math/rand" + "time" +) + +type JitterFunc func(int) int + +type TimeFunc func() time.Time + +type Delayer interface { + Delay() +} + +type Backoff struct { + step float64 + maxBackoff float64 + resetDelta float64 + lastAttempt time.Time + durationUnit time.Duration + jitterer JitterFunc + timeGetter TimeFunc +} + +func jitterFunc(input int) int { + if input == 0 { + return 0 + } + return rand.Intn(input) +} + +func NewBackoff(maxBackoff float64, resetDelta float64, + durationUnit time.Duration) *Backoff { + return NewBackoffForTesting( + maxBackoff, resetDelta, durationUnit, jitterFunc, time.Now) +} + +func NewBackoffForTesting(maxBackoff float64, resetDelta float64, + durationUnit time.Duration, jitterer JitterFunc, timeGetter TimeFunc) *Backoff { + return &Backoff{ + step: 0, + maxBackoff: maxBackoff, + resetDelta: resetDelta, + lastAttempt: timeGetter(), + durationUnit: durationUnit, + jitterer: jitterer, + timeGetter: timeGetter, + } +} + +func (b *Backoff) maybeReset() { + now := b.timeGetter() + lastAttemptDelta := float64(now.Sub(b.lastAttempt) / b.durationUnit) + b.lastAttempt = now + + if lastAttemptDelta >= b.resetDelta { + b.step = 0 + } +} + +func (b *Backoff) GetDelay() time.Duration { + b.maybeReset() + + var synchronizedDuration float64 + // Do not add any delay the first time. + if b.step == 0 { + synchronizedDuration = 0 + } else { + synchronizedDuration = math.Pow(2, b.step) + } + + if synchronizedDuration < b.maxBackoff { + b.step++ + } else { + synchronizedDuration = b.maxBackoff + } + duration := time.Duration(b.jitterer(int(synchronizedDuration))) + return duration * b.durationUnit +} + +func (b *Backoff) Delay() { + delay := b.GetDelay() + log.Printf("Backoff for %s", delay) + time.Sleep(delay) +} diff --git a/backoff_test.go b/backoff_test.go new file mode 100644 index 0000000..f6448e7 --- /dev/null +++ b/backoff_test.go @@ -0,0 +1,80 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "testing" + "time" +) + +type FakeTime struct { + timeseries []int + lastIndex int + durationUnit time.Duration +} + +func (f *FakeTime) GetTime() time.Time { + timeDelta := time.Duration(f.timeseries[f.lastIndex]) * f.durationUnit + fakeTime := time.Unix(0, 0).Add(timeDelta) + f.lastIndex++ + return fakeTime +} + +func FakeJitter(input int) int { + return input +} + +func RunBackoffTest(t *testing.T, + maxBackoff float64, resetDelta float64, + elapsedTime []int, expectedDelays []int) { + fakeTime := &FakeTime{ + timeseries: elapsedTime, + lastIndex: 0, + durationUnit: time.Millisecond, + } + backoff := NewBackoffForTesting(maxBackoff, resetDelta, time.Millisecond, + FakeJitter, fakeTime.GetTime) + + for i, value := range expectedDelays { + expected_delay := time.Duration(value) * time.Millisecond + delay := backoff.GetDelay() + if expected_delay != delay { + t.Errorf("Call #%d of GetDelay returned %s (expected %s)", + i, delay, expected_delay) + } + } +} + +func TestBackoffIncreasesAndReachesMax(t *testing.T) { + RunBackoffTest(t, + 8, + 32, + // Simple sequential time + []int{0, 0, 1, 2, 3, 4, 5, 6, 7}, + // Exponential ramp-up to max, then keep max. + []int{0, 2, 4, 8, 8, 8, 8, 8}, + ) +} + +func TestBackoffReset(t *testing.T) { + RunBackoffTest(t, + 8, + 32, + // Simulate two intervals bigger than resetDelta + []int{0, 0, 1, 2, 50, 51, 100, 101, 102}, + // Delays get reset each time + []int{0, 2, 4, 0, 2, 0, 2, 4}, + ) +} diff --git a/config.go b/config.go new file mode 100644 index 0000000..f0897e1 --- /dev/null +++ b/config.go @@ -0,0 +1,80 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "gopkg.in/yaml.v2" + "io/ioutil" +) + +const ( + defaultNoticeOnceTemplate = "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}" + defaultNoticeTemplate = "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}" +) + +type IRCChannel struct { + Name string `yaml:"name"` + Password string `yaml:"password"` +} + +type Config struct { + HTTPHost string `yaml:"http_host"` + HTTPPort int `yaml:"http_port"` + IRCNick string `yaml:"irc_nickname"` + IRCNickPass string `yaml:"irc_nickname_password"` + IRCRealName string `yaml:"irc_realname"` + IRCHost string `yaml:"irc_host"` + IRCPort int `yaml:"irc_port"` + IRCUseSSL bool `yaml:"irc_use_ssl"` + IRCChannels []IRCChannel `yaml:"irc_channels"` + NoticeTemplate string `yaml:"notice_template"` + NoticeOnce bool `yaml:"notice_once_per_alert_group"` +} + +func LoadConfig(configFile string) (*Config, error) { + config := &Config{ + HTTPHost: "localhost", + HTTPPort: 8000, + IRCNick: "alertmanager-irc-relay", + IRCNickPass: "", + IRCRealName: "Alertmanager IRC Relay", + IRCHost: "irc.freenode.net", + IRCPort: 7000, + IRCUseSSL: true, + IRCChannels: []IRCChannel{IRCChannel{Name: "#airtest"}}, + NoticeOnce: false, + } + + if configFile != "" { + data, err := ioutil.ReadFile(configFile) + if err != nil { + return nil, err + } + if err := yaml.Unmarshal(data, config); err != nil { + return nil, err + } + } + + // Set default template if config does not have one. + if config.NoticeTemplate == "" { + if config.NoticeOnce { + config.NoticeTemplate = defaultNoticeOnceTemplate + } else { + config.NoticeTemplate = defaultNoticeTemplate + } + } + + return config, nil +} diff --git a/config_test.go b/config_test.go new file mode 100644 index 0000000..525c93a --- /dev/null +++ b/config_test.go @@ -0,0 +1,180 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "fmt" + "gopkg.in/yaml.v2" + "io/ioutil" + "os" + "testing" +) + +func TestNoConfig(t *testing.T) { + noConfigFile := "" + + config, err := LoadConfig(noConfigFile) + if config == nil { + t.Errorf("Expected a default config, got: %s", err) + } + +} + +func TestLoadGoodConfig(t *testing.T) { + expectedConfig := &Config{ + HTTPHost: "test.web", + HTTPPort: 8888, + IRCNick: "foo", + IRCHost: "irc.example.com", + IRCPort: 1234, + IRCUseSSL: true, + IRCChannels: []IRCChannel{IRCChannel{Name: "#foobar"}}, + NoticeTemplate: defaultNoticeTemplate, + NoticeOnce: false, + } + expectedData, err := yaml.Marshal(expectedConfig) + if err != nil { + t.Errorf("Could not serialize test data: %s", err) + } + + tmpfile, err := ioutil.TempFile("", "airtestconfig") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + defer os.Remove(tmpfile.Name()) + + if _, err := tmpfile.Write(expectedData); err != nil { + t.Errorf("Could not write test data in tmpfile: %s", err) + } + if err := tmpfile.Close(); err != nil { + t.Errorf("Could not close tmpfile: %s", err) + } + + config, err := LoadConfig(tmpfile.Name()) + if config == nil { + t.Errorf("Expected a config, got: %s", err) + } + + configData, err := yaml.Marshal(config) + if err != nil { + t.Errorf("Could not serialize loaded config") + } + + if string(expectedData) != string(configData) { + t.Errorf("Loaded config does not match expected config: %s", configData) + } +} + +func TestLoadBadFile(t *testing.T) { + tmpfile, err := ioutil.TempFile("", "airtestbadfile") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + tmpfile.Close() + os.Remove(tmpfile.Name()) + + config, err := LoadConfig(tmpfile.Name()) + if config != nil { + t.Errorf("Expected no config upon non-existent file.") + } +} + +func TestLoadBadConfig(t *testing.T) { + tmpfile, err := ioutil.TempFile("", "airtestbadconfig") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + defer os.Remove(tmpfile.Name()) + + badConfigData := []byte("footest\nbarbaz\n") + if _, err := tmpfile.Write(badConfigData); err != nil { + t.Errorf("Could not write test data in tmpfile: %s", err) + } + tmpfile.Close() + + config, err := LoadConfig(tmpfile.Name()) + if config != nil { + t.Errorf("Expected no config upon bad config.") + } +} + +func TestNoticeOnceDefaultTemplate(t *testing.T) { + tmpfile, err := ioutil.TempFile("", "airtesttemmplateonceconfig") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + defer os.Remove(tmpfile.Name()) + + noticeOnceConfigData := []byte("notice_once_per_alert_group: yes") + if _, err := tmpfile.Write(noticeOnceConfigData); err != nil { + t.Errorf("Could not write test data in tmpfile: %s", err) + } + tmpfile.Close() + + config, err := LoadConfig(tmpfile.Name()) + if config == nil { + t.Errorf("Expected a config, got: %s", err) + } + + if config.NoticeTemplate != defaultNoticeOnceTemplate { + t.Errorf("Expecting defaultNoticeOnceTemplate when NoticeOnce is true") + } +} + +func TestNoticeDefaultTemplate(t *testing.T) { + tmpfile, err := ioutil.TempFile("", "airtesttemmplateconfig") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + defer os.Remove(tmpfile.Name()) + + if _, err := tmpfile.Write([]byte("")); err != nil { + t.Errorf("Could not write test data in tmpfile: %s", err) + } + tmpfile.Close() + + config, err := LoadConfig(tmpfile.Name()) + if config == nil { + t.Errorf("Expected a config, got: %s", err) + } + + if config.NoticeTemplate != defaultNoticeTemplate { + t.Errorf("Expecting defaultNoticeTemplate when NoticeOnce is false") + } +} + +func TestGivenTemplateNotOverwritten(t *testing.T) { + tmpfile, err := ioutil.TempFile("", "airtestexpectedtemmplate") + if err != nil { + t.Errorf("Could not create tmpfile for testing: %s", err) + } + defer os.Remove(tmpfile.Name()) + + expectedTemplate := "Alert {{ .Status }}: {{ .Annotations.SUMMARY }}" + configData := []byte(fmt.Sprintf("notice_template: \"%s\"", expectedTemplate)) + if _, err := tmpfile.Write(configData); err != nil { + t.Errorf("Could not write test data in tmpfile: %s", err) + } + tmpfile.Close() + + config, err := LoadConfig(tmpfile.Name()) + if config == nil { + t.Errorf("Expected a config, got: %s", err) + } + + if config.NoticeTemplate != expectedTemplate { + t.Errorf("Template does not match configuration") + } +} diff --git a/data.go b/data.go new file mode 100644 index 0000000..6f32633 --- /dev/null +++ b/data.go @@ -0,0 +1,19 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +type AlertNotice struct { + Channel, Alert string +} diff --git a/http.go b/http.go new file mode 100644 index 0000000..ba03178 --- /dev/null +++ b/http.go @@ -0,0 +1,148 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "bytes" + "encoding/json" + "io" + "io/ioutil" + "log" + "net/http" + + "github.com/gorilla/mux" + promtmpl "github.com/prometheus/alertmanager/template" + "strconv" + "strings" + "text/template" +) + +type HTTPListener func(string, http.Handler) error + +type HTTPServer struct { + StoppedRunning chan bool + Addr string + Port int + NoticeTemplate *template.Template + NoticeOnce bool + AlertNotices chan AlertNotice + httpListener HTTPListener +} + +func NewHTTPServer(config *Config, alertNotices chan AlertNotice) ( + *HTTPServer, error) { + return NewHTTPServerForTesting(config, alertNotices, http.ListenAndServe) +} + +func NewHTTPServerForTesting(config *Config, alertNotices chan AlertNotice, + httpListener HTTPListener) (*HTTPServer, error) { + tmpl, err := template.New("notice").Parse(config.NoticeTemplate) + if err != nil { + return nil, err + } + server := &HTTPServer{ + StoppedRunning: make(chan bool), + Addr: config.HTTPHost, + Port: config.HTTPPort, + NoticeTemplate: tmpl, + NoticeOnce: config.NoticeOnce, + AlertNotices: alertNotices, + httpListener: httpListener, + } + + return server, nil +} + +func (server *HTTPServer) FormatNotice(data interface{}) string { + output := bytes.Buffer{} + var msg string + if err := server.NoticeTemplate.Execute(&output, data); err != nil { + msg_bytes, _ := json.Marshal(data) + msg = string(msg_bytes) + log.Printf("Could not apply notice template on alert (%s): %s", + err, msg) + log.Printf("Sending raw alert") + } else { + msg = output.String() + } + return msg +} + +func (server *HTTPServer) GetNoticesFromAlertMessage(ircChannel string, + data *promtmpl.Data) []AlertNotice { + notices := []AlertNotice{} + if server.NoticeOnce { + msg := server.FormatNotice(data) + notices = append(notices, + AlertNotice{Channel: ircChannel, Alert: msg}) + } else { + for _, alert := range data.Alerts { + msg := server.FormatNotice(alert) + notices = append(notices, + AlertNotice{Channel: ircChannel, Alert: msg}) + } + } + return notices +} + +func (server *HTTPServer) RelayAlert(w http.ResponseWriter, r *http.Request) { + vars := mux.Vars(r) + ircChannel := "#" + vars["IRCChannel"] + + body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1024*1024*1024)) + if err != nil { + log.Printf("Could not get body: %s", err) + return + } + + var alertMessage = promtmpl.Data{} + if err := json.Unmarshal(body, &alertMessage); err != nil { + log.Printf("Could not decode request body (%s): %s", err, body) + + w.Header().Set("Content-Type", "application/json; charset=UTF-8") + w.WriteHeader(422) // Unprocessable entity + if err := json.NewEncoder(w).Encode(err); err != nil { + log.Printf("Could not write decoding error: %s", err) + return + } + return + } + for _, alertNotice := range server.GetNoticesFromAlertMessage( + ircChannel, &alertMessage) { + select { + case server.AlertNotices <- alertNotice: + default: + log.Printf("Could not send this alert to the IRC routine: %s", + alertNotice) + } + } +} + +func (server *HTTPServer) Run() { + router := mux.NewRouter().StrictSlash(true) + + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + server.RelayAlert(w, r) + }) + router.Path("/{IRCChannel}").Handler(handler).Methods("POST") + + listenAddr := strings.Join( + []string{server.Addr, strconv.Itoa(server.Port)}, ":") + log.Printf("Starting HTTP server") + if err := server.httpListener(listenAddr, router); err != nil { + log.Printf("Could not start http server: %s", err) + } + server.StoppedRunning <- true +} diff --git a/http_test.go b/http_test.go new file mode 100644 index 0000000..843d2e9 --- /dev/null +++ b/http_test.go @@ -0,0 +1,218 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "fmt" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" +) + +type FakeHTTPListener struct { + StartedServing chan bool + StopServing chan bool + AlertNotices chan AlertNotice // kinda ugly putting it here, but convenient + router http.Handler +} + +func (listener *FakeHTTPListener) Serve(_ string, router http.Handler) error { + listener.router = router + + listener.StartedServing <- true + <-listener.StopServing + return nil +} + +func NewFakeHTTPListener() *FakeHTTPListener { + return &FakeHTTPListener{ + StartedServing: make(chan bool), + StopServing: make(chan bool), + AlertNotices: make(chan AlertNotice, 10), + } +} + +func MakeHTTPTestingConfig() *Config { + return &Config{ + HTTPHost: "test.web", + HTTPPort: 8888, + NoticeTemplate: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}", + } +} + +func RunHTTPTest(t *testing.T, + alertData string, url string, + testingConfig *Config, listener *FakeHTTPListener) *http.Response { + httpServer, err := NewHTTPServerForTesting(testingConfig, + listener.AlertNotices, listener.Serve) + if err != nil { + t.Fatal(fmt.Sprintf("Could not create HTTP server: %s", err)) + } + + go httpServer.Run() + + <-listener.StartedServing + + alertDataReader := strings.NewReader(alertData) + request, err := http.NewRequest("POST", url, alertDataReader) + if err != nil { + t.Fatal(fmt.Sprintf("Could not create HTTP request: %s", err)) + } + responseRecorder := httptest.NewRecorder() + + listener.router.ServeHTTP(responseRecorder, request) + + listener.StopServing <- true + <-httpServer.StoppedRunning + return responseRecorder.Result() +} + +func TestAlertsDispatched(t *testing.T) { + listener := NewFakeHTTPListener() + testingConfig := MakeHTTPTestingConfig() + + expectedAlertNotices := []AlertNotice{ + AlertNotice{ + Channel: "#somechannel", + Alert: "Alert airDown on instance1:3456 is resolved", + }, + AlertNotice{ + Channel: "#somechannel", + Alert: "Alert airDown on instance2:7890 is resolved", + }, + } + expectedStatusCode := 200 + + response := RunHTTPTest( + t, testdataSimpleAlertJson, "/somechannel", + testingConfig, listener) + + if expectedStatusCode != response.StatusCode { + t.Error(fmt.Sprintf("Expected %d status in response, got %d", + expectedStatusCode, response.StatusCode)) + } + + for _, expectedAlertNotice := range expectedAlertNotices { + alertNotice := <-listener.AlertNotices + if !reflect.DeepEqual(expectedAlertNotice, alertNotice) { + t.Error(fmt.Sprintf( + "Unexpected alert notice.\nExpected: %s\nActual: %s", + expectedAlertNotice, alertNotice)) + } + } +} + +func TestAlertsDispatchedOnce(t *testing.T) { + listener := NewFakeHTTPListener() + testingConfig := MakeHTTPTestingConfig() + testingConfig.NoticeOnce = true + testingConfig.NoticeTemplate = "Alert {{ .GroupLabels.alertname }} is {{ .Status }}" + + expectedAlertNotices := []AlertNotice{ + AlertNotice{ + Channel: "#somechannel", + Alert: "Alert airDown is resolved", + }, + } + expectedStatusCode := 200 + + response := RunHTTPTest( + t, testdataSimpleAlertJson, "/somechannel", + testingConfig, listener) + + if expectedStatusCode != response.StatusCode { + t.Error(fmt.Sprintf("Expected %d status in response, got %d", + expectedStatusCode, response.StatusCode)) + } + + for _, expectedAlertNotice := range expectedAlertNotices { + alertNotice := <-listener.AlertNotices + if !reflect.DeepEqual(expectedAlertNotice, alertNotice) { + t.Error(fmt.Sprintf( + "Unexpected alert notice.\nExpected: %s\nActual: %s", + expectedAlertNotice, alertNotice)) + } + } +} + +func TestRootReturnsError(t *testing.T) { + listener := NewFakeHTTPListener() + testingConfig := MakeHTTPTestingConfig() + + expectedStatusCode := 404 + + response := RunHTTPTest( + t, testdataSimpleAlertJson, "/", + testingConfig, listener) + + if expectedStatusCode != response.StatusCode { + t.Error(fmt.Sprintf("Expected %d status in response, got %d", + expectedStatusCode, response.StatusCode)) + } +} + +func TestInvalidDataReturnsError(t *testing.T) { + listener := NewFakeHTTPListener() + testingConfig := MakeHTTPTestingConfig() + + expectedStatusCode := 422 + + response := RunHTTPTest( + t, testdataBogusAlertJson, "/somechannel", + testingConfig, listener) + + if expectedStatusCode != response.StatusCode { + t.Error(fmt.Sprintf("Expected %d status in response, got %d", + expectedStatusCode, response.StatusCode)) + } +} + +func TestTemplateErrorsCreateRawAlertNotice(t *testing.T) { + listener := NewFakeHTTPListener() + testingConfig := MakeHTTPTestingConfig() + testingConfig.NoticeTemplate = "Bogus template {{ nil }}" + + expectedAlertNotices := []AlertNotice{ + AlertNotice{ + Channel: "#somechannel", + Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance1:3456","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance1","SUMMARY":"service /prometheus air down on instance1"},"startsAt":"2017-05-15T13:49:37.834Z","endsAt":"2017-05-15T13:50:37.835Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`, + }, + AlertNotice{ + Channel: "#somechannel", + Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance2:7890","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance2","SUMMARY":"service /prometheus air down on instance2"},"startsAt":"2017-05-15T11:47:37.834Z","endsAt":"2017-05-15T11:48:37.834Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`, + }, + } + expectedStatusCode := 200 + + response := RunHTTPTest( + t, testdataSimpleAlertJson, "/somechannel", + testingConfig, listener) + + if expectedStatusCode != response.StatusCode { + t.Error(fmt.Sprintf("Expected %d status in response, got %d", + expectedStatusCode, response.StatusCode)) + } + + for _, expectedAlertNotice := range expectedAlertNotices { + alertNotice := <-listener.AlertNotices + if !reflect.DeepEqual(expectedAlertNotice, alertNotice) { + t.Error(fmt.Sprintf( + "Unexpected alert notice.\nExpected: %s\nActual: %s", + expectedAlertNotice, alertNotice)) + } + } +} diff --git a/irc.go b/irc.go new file mode 100644 index 0000000..ee27006 --- /dev/null +++ b/irc.go @@ -0,0 +1,251 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "crypto/tls" + irc "github.com/fluffle/goirc/client" + "log" + "strconv" + "strings" + "time" +) + +const ( + pingFrequencySecs = 60 + connectionTimeoutSecs = 30 + nickservWaitSecs = 10 + ircConnectMaxBackoffSecs = 300 + ircConnectBackoffResetSecs = 1800 +) + +func loggerHandler(_ *irc.Conn, line *irc.Line) { + log.Printf("Received: '%s'", line.Raw) +} + +type ChannelState struct { + Channel IRCChannel + BackoffCounter Delayer +} + +type IRCNotifier struct { + // Nick stores the nickname specified in the config, because irc.Client + // might change its copy. + Nick string + NickPassword string + Client *irc.Conn + StopRunning chan bool + StoppedRunning chan bool + AlertNotices chan AlertNotice + + // irc.Conn has a Connected() method that can tell us wether the TCP + // connection is up, and thus if we should trigger connect/disconnect. + // We need to track the session establishment also at a higher level to + // understand when the server has accepted us and thus when we can join + // channels, send notices, etc. + sessionUp bool + sessionUpSignal chan bool + sessionDownSignal chan bool + + PreJoinChannels []IRCChannel + JoinedChannels map[string]ChannelState + + NickservDelayWait time.Duration + BackoffCounter Delayer +} + +func NewIRCNotifier(config *Config, alertNotices chan AlertNotice) (*IRCNotifier, error) { + + ircConfig := irc.NewConfig(config.IRCNick) + ircConfig.Me.Ident = config.IRCNick + ircConfig.Me.Name = config.IRCRealName + ircConfig.Server = strings.Join( + []string{config.IRCHost, strconv.Itoa(config.IRCPort)}, ":") + ircConfig.SSL = config.IRCUseSSL + ircConfig.SSLConfig = &tls.Config{ServerName: config.IRCHost} + ircConfig.PingFreq = pingFrequencySecs * time.Second + ircConfig.Timeout = connectionTimeoutSecs * time.Second + ircConfig.NewNick = func(n string) string { return n + "^" } + + backoffCounter := NewBackoff( + ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs, + time.Second) + + notifier := &IRCNotifier{ + Nick: config.IRCNick, + NickPassword: config.IRCNickPass, + Client: irc.Client(ircConfig), + StopRunning: make(chan bool), + StoppedRunning: make(chan bool), + AlertNotices: alertNotices, + sessionUpSignal: make(chan bool), + sessionDownSignal: make(chan bool), + PreJoinChannels: config.IRCChannels, + JoinedChannels: make(map[string]ChannelState), + NickservDelayWait: nickservWaitSecs * time.Second, + BackoffCounter: backoffCounter, + } + + notifier.Client.HandleFunc(irc.CONNECTED, + func(*irc.Conn, *irc.Line) { + log.Printf("Session established") + notifier.sessionUpSignal <- true + }) + + notifier.Client.HandleFunc(irc.DISCONNECTED, + func(*irc.Conn, *irc.Line) { + log.Printf("Disconnected from IRC") + notifier.sessionDownSignal <- false + }) + + notifier.Client.HandleFunc(irc.KICK, + func(_ *irc.Conn, line *irc.Line) { + notifier.HandleKick(line.Args[1], line.Args[0]) + }) + + for _, event := range []string{irc.NOTICE, "433"} { + notifier.Client.HandleFunc(event, loggerHandler) + } + + return notifier, nil +} + +func (notifier *IRCNotifier) HandleKick(nick string, channel string) { + if nick != notifier.Client.Me().Nick { + // received kick info for somebody else + return + } + state, ok := notifier.JoinedChannels[channel] + if ok == false { + log.Printf("Being kicked out of non-joined channel (%s), ignoring", channel) + return + } + log.Printf("Being kicked out of %s, re-joining", channel) + go func() { + state.BackoffCounter.Delay() + notifier.Client.Join(state.Channel.Name, state.Channel.Password) + }() + +} + +func (notifier *IRCNotifier) CleanupChannels() { + log.Printf("Deregistering all channels.") + notifier.JoinedChannels = make(map[string]ChannelState) +} + +func (notifier *IRCNotifier) JoinChannel(channel *IRCChannel) { + if _, joined := notifier.JoinedChannels[channel.Name]; joined == true { + return + } + log.Printf("Joining %s", channel.Name) + notifier.Client.Join(channel.Name, channel.Password) + state := ChannelState{ + Channel: *channel, + BackoffCounter: NewBackoff( + ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs, + time.Second), + } + notifier.JoinedChannels[channel.Name] = state +} + +func (notifier *IRCNotifier) JoinChannels() { + for _, channel := range notifier.PreJoinChannels { + notifier.JoinChannel(&channel) + } +} + +func (notifier *IRCNotifier) MaybeIdentifyNick() { + if notifier.NickPassword == "" { + return + } + + // Very lazy/optimistic, but this is good enough for my irssi config, + // so it should work here as well. + currentNick := notifier.Client.Me().Nick + if currentNick != notifier.Nick { + log.Printf("My nick is '%s', sending GHOST to NickServ to get '%s'", + currentNick, notifier.Nick) + notifier.Client.Privmsgf("NickServ", "GHOST %s %s", notifier.Nick, + notifier.NickPassword) + time.Sleep(notifier.NickservDelayWait) + + log.Printf("Changing nick to '%s'", notifier.Nick) + notifier.Client.Nick(notifier.Nick) + } + log.Printf("Sending IDENTIFY to NickServ") + notifier.Client.Privmsgf("NickServ", "IDENTIFY %s", notifier.NickPassword) + time.Sleep(notifier.NickservDelayWait) +} + +func (notifier *IRCNotifier) MaybeSendAlertNotice(alertNotice *AlertNotice) { + if !notifier.sessionUp { + log.Printf("Cannot send alert to %s : IRC not connected", + alertNotice.Channel) + return + } + notifier.JoinChannel(&IRCChannel{Name: alertNotice.Channel}) + notifier.Client.Notice(alertNotice.Channel, alertNotice.Alert) +} + +func (notifier *IRCNotifier) Run() { + keepGoing := true + for keepGoing { + if !notifier.Client.Connected() { + log.Printf("Connecting to IRC") + notifier.BackoffCounter.Delay() + if err := notifier.Client.Connect(); err != nil { + log.Printf("Could not connect to IRC: %s", err) + select { + case <-notifier.StopRunning: + log.Printf("IRC routine not connected but asked to terminate") + keepGoing = false + default: + } + continue + } + log.Printf("Connected to IRC server, waiting to establish session") + } + + select { + case alertNotice := <-notifier.AlertNotices: + notifier.MaybeSendAlertNotice(&alertNotice) + case <-notifier.sessionUpSignal: + notifier.sessionUp = true + notifier.MaybeIdentifyNick() + notifier.JoinChannels() + case <-notifier.sessionDownSignal: + notifier.sessionUp = false + notifier.CleanupChannels() + notifier.Client.Quit("see ya") + case <-notifier.StopRunning: + log.Printf("IRC routine asked to terminate") + keepGoing = false + } + } + if notifier.Client.Connected() { + log.Printf("IRC client connected, quitting") + notifier.Client.Quit("see ya") + + if notifier.sessionUp { + log.Printf("Session is up, wait for IRC disconnect to complete") + select { + case <-notifier.sessionDownSignal: + case <-time.After(notifier.Client.Config().Timeout): + log.Printf("Timeout while waiting for IRC disconnect to complete, stopping anyway") + } + } + } + notifier.StoppedRunning <- true +} diff --git a/irc_test.go b/irc_test.go new file mode 100644 index 0000000..d6b549f --- /dev/null +++ b/irc_test.go @@ -0,0 +1,720 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "bufio" + "fmt" + irc "github.com/fluffle/goirc/client" + "io" + "log" + "net" + "reflect" + "strings" + "sync" + "testing" + "time" +) + +type LineHandlerFunc func(*bufio.ReadWriter, *irc.Line) error + +func h_USER(conn *bufio.ReadWriter, line *irc.Line) error { + r := fmt.Sprintf(":example.com 001 %s :Welcome\n", line.Args[0]) + conn.WriteString(r) + return nil +} + +func h_QUIT(conn *bufio.ReadWriter, line *irc.Line) error { + return fmt.Errorf("client asked to terminate") +} + +type closeEarlyHandler func() + +type testServer struct { + net.Listener + Client net.Conn + + ServingWaitGroup sync.WaitGroup + ConnectionsWaitGroup sync.WaitGroup + + lineHandlersMu sync.Mutex + lineHandlers map[string]LineHandlerFunc + + Log []string + + closeEarlyMu sync.Mutex + closeEarlyHandler +} + +func (s *testServer) setDefaultHandlers() { + if s.lineHandlers == nil { + s.lineHandlers = make(map[string]LineHandlerFunc) + } + s.lineHandlers["USER"] = h_USER + s.lineHandlers["QUIT"] = h_QUIT +} + +func (s *testServer) getHandler(cmd string) LineHandlerFunc { + s.lineHandlersMu.Lock() + defer s.lineHandlersMu.Unlock() + return s.lineHandlers[cmd] +} + +func (s *testServer) SetHandler(cmd string, h LineHandlerFunc) { + s.lineHandlersMu.Lock() + defer s.lineHandlersMu.Unlock() + if h == nil { + delete(s.lineHandlers, cmd) + } else { + s.lineHandlers[cmd] = h + } +} + +func (s *testServer) handleLine(conn *bufio.ReadWriter, line *irc.Line) error { + s.Log = append(s.Log, strings.Trim(line.Raw, " \r\n")) + handler := s.getHandler(line.Cmd) + if handler == nil { + log.Printf("=Server= No handler for command '%s', skipping", line.Cmd) + return nil + } + return handler(conn, line) +} + +func (s *testServer) handleConnection(conn net.Conn) { + defer func() { + s.Client = nil + conn.Close() + s.ConnectionsWaitGroup.Done() + }() + bufConn := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn)) + for { + msg, err := bufConn.ReadBytes('\n') + if err != nil { + if err == io.EOF { + log.Printf("=Server= Client %s disconnected", conn.RemoteAddr().String()) + } else { + log.Printf("=Server= Could not read from %s: %s", conn.RemoteAddr().String(), err) + } + return + } + log.Printf("=Server= Received %s", msg) + line := irc.ParseLine(string(msg)) + if line == nil { + log.Printf("=Server= Could not parse received line") + continue + } + err = s.handleLine(bufConn, line) + if err != nil { + log.Printf("=Server= Closing connection: %s", err) + return + } + bufConn.Flush() + } +} + +func (s *testServer) SetCloseEarly(h closeEarlyHandler) { + s.closeEarlyMu.Lock() + defer s.closeEarlyMu.Unlock() + s.closeEarlyHandler = h +} + +func (s *testServer) handleCloseEarly(conn net.Conn) bool { + s.closeEarlyMu.Lock() + defer s.closeEarlyMu.Unlock() + if s.closeEarlyHandler == nil { + return false + } + log.Printf("=Server= Closing connection early") + conn.Close() + s.closeEarlyHandler() + return true +} + +func (s *testServer) Serve() { + defer s.ServingWaitGroup.Done() + for { + conn, err := s.Listener.Accept() + if err != nil { + log.Printf("=Server= Stopped accepting new connections") + return + } + log.Printf("=Server= New client connected from %s", conn.RemoteAddr().String()) + if s.handleCloseEarly(conn) { + continue + } + s.Client = conn + s.ConnectionsWaitGroup.Add(1) + s.handleConnection(conn) + } +} + +func (s *testServer) Stop() { + s.Listener.Close() + s.ServingWaitGroup.Wait() + s.ConnectionsWaitGroup.Wait() +} + +func makeTestServer(t *testing.T) (*testServer, int) { + server := new(testServer) + server.Log = make([]string, 0) + server.setDefaultHandlers() + + addr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("=Server= Could not resolve tcp addr: %s", err) + } + listener, err := net.ListenTCP("tcp", addr) + if err != nil { + t.Fatalf("=Server= Could not create listener: %s", err) + } + addr = listener.Addr().(*net.TCPAddr) + log.Printf("=Server= Test server listening on %s", addr.String()) + + server.Listener = listener + + server.ServingWaitGroup.Add(1) + go func() { + server.Serve() + }() + + addr = listener.Addr().(*net.TCPAddr) + return server, addr.Port +} + +type FakeDelayer struct { +} + +func (f *FakeDelayer) Delay() { + log.Printf("Faking Backoff") +} + +func makeTestIRCConfig(IRCPort int) *Config { + return &Config{ + IRCNick: "foo", + IRCNickPass: "", + IRCHost: "127.0.0.1", + IRCPort: IRCPort, + IRCUseSSL: false, + IRCChannels: []IRCChannel{ + IRCChannel{Name: "#foo"}, + IRCChannel{Name: "#bar"}, + IRCChannel{Name: "#baz"}, + }, + } +} + +func makeTestNotifier(t *testing.T, config *Config) (*IRCNotifier, chan AlertNotice) { + alertNotices := make(chan AlertNotice) + notifier, err := NewIRCNotifier(config, alertNotices) + if err != nil { + t.Fatal(fmt.Sprintf("Could not create IRC notifier: %s", err)) + } + notifier.Client.Config().Flood = true + notifier.BackoffCounter = &FakeDelayer{} + + return notifier, alertNotices +} + +func TestPreJoinChannels(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, _ := makeTestNotifier(t, config) + + var testStep sync.WaitGroup + + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Add(1) + go notifier.Run() + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Did not pre-join channels") + } +} + +func TestSendAlertOnPreJoinedChannel(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, alertNotices := makeTestNotifier(t, config) + + var testStep sync.WaitGroup + + testChannel := "#foo" + testMessage := "test message" + + // Send the alert after configured channels have joined, to ensure we + // check for no re-join attempt. + joinedHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + if line.Args[0] == testChannel { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinedHandler) + + testStep.Add(1) + go notifier.Run() + + testStep.Wait() + + server.SetHandler("JOIN", nil) + + noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + testStep.Done() + return nil + } + server.SetHandler("NOTICE", noticeHandler) + + testStep.Add(1) + alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage} + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "NOTICE #foo :test message", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestSendAlertAndJoinChannel(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, alertNotices := makeTestNotifier(t, config) + + var testStep sync.WaitGroup + + testChannel := "#foobar" + testMessage := "test message" + + // Send the alert after configured channels have joined, to ensure log + // ordering. + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Add(1) + go notifier.Run() + + testStep.Wait() + + server.SetHandler("JOIN", nil) + + noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + testStep.Done() + return nil + } + server.SetHandler("NOTICE", noticeHandler) + + testStep.Add(1) + alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage} + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + // #foobar joined before sending message + "JOIN #foobar", + "NOTICE #foobar :test message", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestSendAlertDisconnected(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, alertNotices := makeTestNotifier(t, config) + + var testStep, holdUserStep sync.WaitGroup + + testChannel := "#foo" + disconnectedTestMessage := "disconnected test message" + connectedTestMessage := "connected test message" + + // First send an alert while the session is not established. + testStep.Add(1) + holdUserStep.Add(1) + holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error { + log.Printf("=Server= Wait before completing session") + testStep.Wait() + log.Printf("=Server= Completing session") + holdUserStep.Done() + return h_USER(conn, line) + } + server.SetHandler("USER", holdUser) + + go notifier.Run() + + alertNotices <- AlertNotice{Channel: testChannel, Alert: disconnectedTestMessage} + + testStep.Done() + holdUserStep.Wait() + + // Make sure session is established by checking that pre-joined + // channels are there. + testStep.Add(1) + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Wait() + + // Now send and wait until a notice has been received. + testStep.Add(1) + noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + testStep.Done() + return nil + } + server.SetHandler("NOTICE", noticeHandler) + + alertNotices <- AlertNotice{Channel: testChannel, Alert: connectedTestMessage} + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + // Only message sent while being connected is received. + "NOTICE #foo :connected test message", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestReconnect(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, _ := makeTestNotifier(t, config) + + var testStep sync.WaitGroup + + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Add(1) + go notifier.Run() + + // Wait until the last pre-joined channel is seen. + testStep.Wait() + + // Simulate disconnection. + testStep.Add(1) + server.Client.Close() + + // Wait again until the last pre-joined channel is seen. + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + // Commands from first connection + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + // Commands from reconnection + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestConnectErrorRetry(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + // Attempt SSL handshake. The server does not support it, resulting in + // a connection error. + config.IRCUseSSL = true + notifier, _ := makeTestNotifier(t, config) + + var testStep, joinStep sync.WaitGroup + + testStep.Add(1) + earlyHandler := func() { + testStep.Done() + } + + server.SetCloseEarly(earlyHandler) + + go notifier.Run() + + testStep.Wait() + + // We have caused a connection failure, now check for a reconnection + notifier.Client.Config().SSL = false + joinStep.Add(1) + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + joinStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + server.SetCloseEarly(nil) + + joinStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestIdentify(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + config.IRCNickPass = "nickpassword" + notifier, _ := makeTestNotifier(t, config) + notifier.NickservDelayWait = 0 * time.Second + + var testStep sync.WaitGroup + + // Wait until the last pre-joined channel is seen (joining happens + // after identification). + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Add(1) + go notifier.Run() + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "PRIVMSG NickServ :IDENTIFY nickpassword", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Identification did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestGhostAndIdentify(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + config.IRCNickPass = "nickpassword" + notifier, _ := makeTestNotifier(t, config) + notifier.NickservDelayWait = 0 * time.Second + + var testStep, usedNick, unregisteredNickHandler sync.WaitGroup + + // Trigger 433 for first nick + usedNick.Add(1) + unregisteredNickHandler.Add(1) + nickHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + if line.Args[0] == "foo" { + conn.WriteString(":example.com 433 * foo :nick in use\n") + } + usedNick.Done() + unregisteredNickHandler.Wait() + return nil + } + server.SetHandler("NICK", nickHandler) + + // Wait until the last pre-joined channel is seen (joining happens + // after identification). + joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error { + // #baz is configured as the last channel to pre-join + if line.Args[0] == "#baz" { + testStep.Done() + } + return nil + } + server.SetHandler("JOIN", joinHandler) + + testStep.Add(1) + go notifier.Run() + + usedNick.Wait() + server.SetHandler("NICK", nil) + unregisteredNickHandler.Done() + + testStep.Wait() + + notifier.StopRunning <- true + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "NICK foo^", + "PRIVMSG NickServ :GHOST foo nickpassword", + "NICK foo", + "PRIVMSG NickServ :IDENTIFY nickpassword", + "JOIN #foo", + "JOIN #bar", + "JOIN #baz", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Ghosting did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} + +func TestStopRunningWhenHalfConnected(t *testing.T) { + server, port := makeTestServer(t) + config := makeTestIRCConfig(port) + notifier, _ := makeTestNotifier(t, config) + + var testStep, holdQuitWait sync.WaitGroup + + // Send a StopRunning request while the client is connected but the + // session is not up + testStep.Add(1) + holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error { + log.Printf("=Server= NOT completing session") + testStep.Done() + return nil + } + server.SetHandler("USER", holdUser) + + // Ignore quit, but wait for it to have deterministic test commands + holdQuitWait.Add(1) + holdQuit := func(conn *bufio.ReadWriter, line *irc.Line) error { + log.Printf("=Server= Ignoring quit") + holdQuitWait.Done() + return nil + } + server.SetHandler("QUIT", holdQuit) + + go notifier.Run() + + testStep.Wait() + + notifier.StopRunning <- true + + <-notifier.StoppedRunning + + holdQuitWait.Wait() + + // Client has left, cleanup the server side before stopping + server.Client.Close() + + server.Stop() + + expectedCommands := []string{ + "NICK foo", + "USER foo 12 * :", + "QUIT :see ya", + } + + if !reflect.DeepEqual(expectedCommands, server.Log) { + t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n")) + } +} diff --git a/main.go b/main.go new file mode 100644 index 0000000..9f02190 --- /dev/null +++ b/main.go @@ -0,0 +1,67 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "flag" + "log" + "os" + "os/signal" + "syscall" +) + +func main() { + + configFile := flag.String("config", "", "Config file path.") + + flag.Parse() + + signals := make(chan os.Signal, 1) + signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM) + + config, err := LoadConfig(*configFile) + if err != nil { + log.Printf("Could not load config: %s", err) + return + } + + alertNotices := make(chan AlertNotice, 10) + + ircNotifier, err := NewIRCNotifier(config, alertNotices) + if err != nil { + log.Printf("Could not create IRC notifier: %s", err) + return + } + go ircNotifier.Run() + + httpServer, err := NewHTTPServer(config, alertNotices) + if err != nil { + log.Printf("Could not create HTTP server: %s", err) + return + } + go httpServer.Run() + + select { + case <-httpServer.StoppedRunning: + log.Printf("Http server terminated, exiting") + case <-ircNotifier.StoppedRunning: + log.Printf("IRC notifier stopped running, exiting") + case s := <-signals: + log.Printf("Received %s, exiting", s) + ircNotifier.StopRunning <- true + log.Printf("Waiting for IRC to quit") + <-ircNotifier.StoppedRunning + } +} diff --git a/testdata.go b/testdata.go new file mode 100644 index 0000000..ce5981c --- /dev/null +++ b/testdata.go @@ -0,0 +1,77 @@ +// Copyright 2018 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +const ( + testdataSimpleAlertJson = ` +{ + "status": "resolved", + "receiver": "example_receiver", + "groupLabels": { + "alertname": "airDown", + "service": "prometheus" + }, + "commonLabels": { + "alertname": "airDown", + "job": "air", + "service": "prometheus", + "severity": "ticket", + "zone": "global" + }, + "commonAnnotations": {}, + "externalURL": "https://prometheus.example.com/alertmanager", + "alerts": [ + { + "annotations": { + "SUMMARY": "service /prometheus air down on instance1", + "DESCRIPTION": "service /prometheus has irc gateway down on instance1" + }, + "endsAt": "2017-05-15T13:50:37.835Z", + "generatorURL": "https://prometheus.example.com/prometheus/...", + "labels": { + "alertname": "airDown", + "instance": "instance1:3456", + "job": "air", + "service": "prometheus", + "severity": "ticket", + "zone": "global" + }, + "startsAt": "2017-05-15T13:49:37.834Z", + "status": "resolved" + }, + { + "annotations": { + "SUMMARY": "service /prometheus air down on instance2", + "DESCRIPTION": "service /prometheus has irc gateway down on instance2" + }, + "endsAt": "2017-05-15T11:48:37.834Z", + "generatorURL": "https://prometheus.example.com/prometheus/...", + "labels": { + "alertname": "airDown", + "instance": "instance2:7890", + "job": "air", + "service": "prometheus", + "severity": "ticket", + "zone": "global" + }, + "startsAt": "2017-05-15T11:47:37.834Z", + "status": "resolved" + } + ] +} +` + + testdataBogusAlertJson = `{"this is not": "a valid alert",}` +)