<commit_before>{
  "created_at": 1522790106000,
  "deleted": null,
  "message": "The go-routers are complaining about TLS certs being invalid. Are the certificates out of sync between the control plane and cell block deployments?\n\nWarning: This alert has proven to generate false alarms in the past! (read more for details)\n\nThe reason the alert causes confusion is that it is not specified anywhere that each go-router is reporting its monotonically increasing count of witnessed TLS handshake failures. This means the metric will only grow. Additionally, the metric can appear spiky due to router count differences and less than synchronized router checkin intervals.\n\nAlso, the alert triggers in a strange way. It adds up all counts over a 5 minute window, while this alone is not an issue; the rate of reports is about 1 a second across all routers, so if a couple routers have seen one or two errors this thing will go off. (And stay that way until you restart the routers, even if you have fixed the underlying issue.) This also makes it look like the metric is far under the alert threshold on the graph...\n\n[BW]\n- I am attempting to create a [better metric](https://pivotalcloudopspws.datadoghq.com/monitors#10784390) but it is unclear if it will actually feel better during a time of stress (aka RED alert).\n- This new metric is also less sensitive; so, I wanted to err on the safe-side and vet it before we committed to it. If these help you during a RED alert, maybe consider replacing the \"Backend TLS invalid cert errors are spiking\" metric with this new one.\n\nThere is a story to track improving this here: https://www.pivotaltracker.com/story/show/167662632",
  "multi": false,
  "name": "Backend TLS invalid cert errors are spiking",
  "options": {
    "escalation_message": "",
    "include_tags": false,
    "locked": false,
    "new_host_delay": 300,
    "no_data_timeframe": null,
    "notify_audit": false,
    "notify_no_data": false,
    "renotify_interval": 0,
    "require_full_window": true,
    "silenced": {
    },
    "thresholds": {
      "critical": 10.0
    },
    "timeout_h": 0
  },
  "org_id": 4242,
  "overall_state": "OK",
  "overall_state_modified": "2019-08-02T00:21:19+00:00",
  "query": "sum(last_5m):avg:datadog.nozzle.gorouter.backend_invalid_tls_cert{*} >= 10",
  "tags": [

  ],
  "type": "metric alert"
}<commit_msg>[Backend TLS invalid cert] improve monitor<commit_after>{
  "created_at": 1522790106000,
  "deleted": null,
  "message": "The go-routers are complaining about backend TLS certs being invalid. Are the certificates out of sync between the control plane and cell block deployments? Or could there be one application/service that is causing all of the errors?\n\nOne thing that has bitten us in the past was one team testing a new service with a failing certificate and it was enough to trigger our previously-set threshold.\n\nFor historical context, see [this doc](https://docs.google.com/document/d/1kP64UvsBX42nvUJ03dJrDZRdycOb0EegHar5laVFid8).",
  "multi": false,
  "name": "Backend TLS invalid cert errors are spiking",
  "options": {
    "escalation_message": "",
    "include_tags": false,
    "locked": false,
    "new_host_delay": 300,
    "no_data_timeframe": null,
    "notify_audit": false,
    "notify_no_data": false,
    "renotify_interval": 0,
    "require_full_window": true,
    "silenced": {
    },
    "thresholds": {
      "critical": 10.0
    },
    "timeout_h": 0
  },
  "org_id": 4242,
  "overall_state": "Alert",
  "overall_state_modified": "2019-08-05T23:23:18+00:00",
  "query": "sum(last_5m):per_minute(sum:datadog.nozzle.gorouter.backend_invalid_tls_cert{*}.rollup(sum, 60)) >= 10",
  "tags": [

  ],
  "type": "query alert"
}
{'fraction_non_alphanumeric': 0.10541845493562232, 'fraction_numerical': 0.03299356223175966, 'mean_word_length': 4.2081005586592175, 'pattern_counts': {'":': 50, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '368608', 'n_tokens_mistral': 1243, 'n_tokens_neox': 1140, 'n_words': 439}