Prometheus Alertmanager

using:

• flux api
• helm api
• kustomize

TODO:

• separate baseoverlay
• create gitlab helm repository
• split repo prometheus-community-lcars
  • k3d createdestroy
  • kustomizefluxhelm
• Flux UIs / GUIs
• wasdx
• wasdx
•  

Additional links:

• Additional Scrape Configuration
• Grafana Alertmanager blog
• alermanager bug no slack api
• helm.toolkit.fluxcd.io/v2beta1
• flux/helm/values
• kube-prometheus-stack hacks
• HelmReleaseSpec
• fluxcd.io: are-there-two-kustomization-types
• flux troubleshooting cheatsheet
• fluxcd.io:Manage Helm Releases
• kuztomise: configMapGenerator
• kuztomize-configmap-generatorsCustomizing Slack notifications
• sync_prometheus_rules.py
• Slack app support ?
• Customizing Slack notifications
• prom avatar
• opsgenie_config
• we
•  
• lcars
• we

Create a confige map for the  kube-stack-prometheus-values.yaml using kustomize: configMapGenerator.

				
					❯ cat ./base/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: kube-prometheus-stack-values
  namespace: prometheus
  files:
  - ./kube-prometheus-stack-values.yaml
 ❯ kustomize build ./base/ | kubectl apply -f - 
configmap/kube-prometheus-stack-values-2799bk595k created
# proposed dir structure:
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── dev
    │   ├── deployment-dev.yaml
    │   ├── kustomization.yaml
    │   └── service-dev.yaml
    ├── generators
    │   ├── deployment.yaml
    │   ├── files
    │   │   └── index.html
    │   ├── kustomization.yaml
    │   └── service.yaml
    └── prod
        ├── deployment-prod.yaml
        ├── kustomization.yaml
        └── service-prod.yaml
				
			

Sending a manual alert

				
					curl -H 'Content-Type: application/json' -d '[{"labels":{"alertname":"demo-alert","namespace":"demo","service":"Demo alert"}}]' http://alertmanager.localhost/api/v1/alerts
{"status":"success"}%
❯ curl -H 'Content-Type: application/json' -d '[{"labels":{"alertname":"InstanceDown","namespace":"kube-system","service":"Demo alert", "severity":"critical"}}]' http://alertmanager.localhost/api/v1/alerts
{"status":"success"}%
				
			

Fancy slack message

				
					global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/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'
route:
  receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#monitoring-instances'
    send_resolved: true
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    title: |-
     [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
     {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
       {{" "}}(
       {{- with .CommonLabels.Remove .GroupLabels.Names }}
         {{- range $index, $label := .SortedPairs -}}
           {{ if $index }}, {{ end }}
           {{- $label.Name }}="{{ $label.Value -}}"
         {{- end }}
       {{- end -}}
       )
     {{- end }}
    text: >-
     {{ range .Alerts -}}
     *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
     *Description:* {{ .Annotations.description }}
     *Details:*
       {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
       {{ end }}
     {{ end }}