Monitoring Ingress with Prometheus and Grafana
Prometheus is an open-source monitoring and alerting system. When used along with Grafana, you can create a dynamic dashboard for monitoring ingress into our Kubernetes cluster.
Deployment
This guide will focus on deploying Prometheus and Grafana alongside Ambassador Edge Stack in Kubernetes using the Prometheus Operator.
Note: Both Prometheus and Grafana can be deployed as standalone applications outside of Kubernetes. This process is well-documented within the website and docs within their respective projects.
Ambassador Edge Stack
Ambassador Edge Stack makes it easy to output Envoy-generated statistics to Prometheus. For the remainder of this guide, it is assumed that you have installed and configured Ambassador Edge Stack into your Kubernetes cluster, and that it is possible for you to modify the global configuration of the Ambassador Edge Stack deployment.
Starting with Ambassador 0.71.0
, Prometheus can scrape stats/metrics directly from Envoy's /metrics
endpoint, removing the need to configure Ambassador Edge Stack to output stats to StatsD.
The /metrics
endpoint can be accessed internally via the Ambassador Edge Stack admin port (default 8877):
http(s)://ambassador:8877/metrics
or externally by creating a Mapping
similar to below:
---apiVersion: getambassador.io/v2kind: Mappingmetadata:name: metricsspec:prefix: /metricsrewrite: ""service: localhost:8877
Note: Since /metrics
in an endpoint on Ambassador Edge Stack itself, the service
field can just reference the admin port on localhost.
Prometheus Operator with Standard YAML
In this section, we will deploy the Prometheus Operator using the standard YAML files. Alternatively, you can install it with Helm if you prefer.
Deploy the Prometheus Operator
To deploy the Prometheus Operator, you can clone the repository and follow the instructions in the README, or simply apply the published YAML with
kubectl
.kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yamlDeploy Prometheus by creating a
Prometheus
CRDFirst, create RBAC resources for your Prometheus instance
kubectl apply -f https://www.getambassador.io/yaml/monitoring/prometheus-rbac.yamlThen, copy the YAML below, and save it in a file called
prometheus.yaml
---apiVersion: v1kind: Servicemetadata:name: prometheusspec:type: ClusterIPports:- name: webport: 9090protocol: TCPtargetPort: 9090selector:prometheus: prometheus---apiVersion: monitoring.coreos.com/v1kind: Prometheusmetadata:name: prometheusspec:ruleSelector:matchLabels:app: prometheus-operatorserviceAccountName: prometheusserviceMonitorSelector:matchLabels:app: ambassadorresources:requests:memory: 400Mikubectl apply -f prometheus.yaml
Create a
ServiceMonitor
Finally, we need to tell Prometheus where to scrape metrics from. The Prometheus Operator easily manages this using a
ServiceMonitor
CRD. To tell Prometheus to scrape metrics from Ambassador Edge Stack's/metrics
endpoint, copy the following YAML to a file calledambassador-monitor.yaml
, and apply it withkubectl
.If you are running an Ambassador version higher than 0.71.0 and want to scrape metrics directly from the
/metrics
endpoint of Ambassador Edge Stack running in theambassador
namespace:---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:name: ambassador-monitorlabels:app: ambassadorspec:namespaceSelector:matchNames:- ambassadorselector:matchLabels:service: ambassador-adminendpoints:- port: ambassador-admin
Prometheus is now configured to gather metrics from Ambassador Edge Stack.
Prometheus Operator with Helm
In this section, we will deploy the Prometheus Operator using Helm. Alternatively, you can install it with kubectl YAML if you prefer.
The default Helm Chart will install Prometheus and configure it to monitor your Kubernetes cluster.
This section will focus on setting up Prometheus to scrape stats from Ambassador Edge Stack. Configuration of the Helm Chart and analysis of stats from other cluster components is outside of the scope of this documentation.
Install the Prometheus Operator from the helm chart
helm install -n prometheus stable/prometheus-operatorCreate a
ServiceMonitor
The Prometheus Operator Helm chart creates a Prometheus instance that is looking for
ServiceMonitor
s withlabel: release=prometheus
.If you are running an Ambassador version higher than 0.71.0 and want to scrape metrics directly from the
/metrics
endpoint of Ambassador Edge Stack running in thedefault
namespace:---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:name: ambassador-monitornamespace: monitoringlabels:release: prometheusspec:namespaceSelector:matchNames:- defaultselector:matchLabels:service: ambassador-adminendpoints:- port: ambassador-adminIf you are scraping metrics from a
statsd-sink
deployment:---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:name: statsd-monitornamespace: monitoringlabels:release: prometheusspec:namespaceSelector:matchNames:- defaultselector:matchLabels:service: statsd-sinkendpoints:- port: prometheus-metrics
Prometheus is now configured to gather metrics from Ambassador Edge Stack.
Prometheus Operator CRDs
The Prometheus Operator creates a series of Kubernetes Custom Resource Definitions (CRDs) for managing Prometheus in Kubernetes.
Custom Resource Definition | Description |
---|---|
AlertManager | An AlertManager handles alerts sent by the Prometheus server. |
PrometheusRule | Registers altering and reporting rules with Prometheus. |
Prometheus | Creates a Prometheus instance. |
ServiceMonitor | Tells Prometheus where to scrape metrics from. |
CoreOS has published a full API reference to these different CRDs.
Grafana
Grafana is an open-source graphing tool for plotting data points. Grafana allows you to create dynamic dashboards for monitoring your ingress traffic stats collected from Prometheus.
We have published a sample dashboard you can use for monitoring your ingress traffic. Since the stats from the /metrics
and /stats
endpoints are different, you will see a section in the dashboard for each use case.
Note: If you deployed the Prometheus Operator via the Helm Chart, a Grafana dashboard is created by default. You can use this dashboard or set grafana.enabled: false
and follow the instructions below.
To deploy Grafana behind Ambassador Edge Stack: replace {{AMBASSADOR_IP}}
with the IP address or DNS name of your Ambassador Edge Stack service, copy the YAML below, and apply it with kubectl
:
---kind: DeploymentapiVersion: apps/v1metadata:name: grafanalabels:app: grafanacomponent: corespec:replicas: 1selector:matchLabels:app: grafanacomponent: coretemplate:metadata:creationTimestamp: nulllabels:app: grafanacomponent: coreannotations:sidecar.istio.io/inject: 'false'spec:volumes:- name: dataemptyDir: {}containers:- name: grafanaimage: 'grafana/grafana:6.4.3'ports:- containerPort: 3000protocol: TCPenv:- name: GF_SERVER_ROOT_URLvalue: {{AMBASSADOR_IP}}/grafana- name: GRAFANA_PORTvalue: '3000'- name: GF_AUTH_BASIC_ENABLEDvalue: 'false'- name: GF_AUTH_ANONYMOUS_ENABLEDvalue: 'true'- name: GF_AUTH_ANONYMOUS_ORG_ROLEvalue: Admin- name: GF_PATHS_DATAvalue: /data/grafanaresources:requests:cpu: 10mvolumeMounts:- name: datamountPath: /data/grafanareadinessProbe:httpGet:path: /api/healthport: 3000scheme: HTTPtimeoutSeconds: 1periodSeconds: 10successThreshold: 1failureThreshold: 3imagePullPolicy: IfNotPresentrestartPolicy: Always---apiVersion: v1kind: Servicemetadata:name: grafanaspec:ports:- port: 80targetPort: 3000selector:app: grafanacomponent: core
Now, create a service and Mapping
to expose Grafana behind Ambassador Edge Stack:
---apiVersion: getambassador.io/v2kind: Mappingmetadata:name: grafanaspec:prefix: /grafana/service: grafana.{{GRAFANA_NAMESPACE}}
Now, access Grafana by going to {AMBASSADOR_IP}/grafana/
and logging in with username: admin
: password: admin
.
Import the provided dashboard by clicking the plus sign in the left side-bar, clicking New Dashboard
in the top left, selecting Import Dashboard
, and entering the dashboard ID(10434).
Viewing Stats/Metrics
Above, you have created an environment where Ambassador is handling ingress traffic, Prometheus is scraping and collecting statistics from Envoy, and Grafana is displaying these statistics in a dashboard.
You can easily view a sample of these statistics via the Grafana dashboard at {AMBASSADOR_IP}/grafana/
and logging in with the credentials above.
The example dashboard you installed above displays 'top line' statistics about the API response codes, number of connections, connection length, and number of registered services.
To view the full set of stats available to Prometheus you can access the Prometheus UI by running:
kubectl port-forward -n monitoring service/prometheus 9090
and going to http://localhost:9090/
from a web browser
In the UI, click the dropdown and see all of the stats Prometheus is able to scrape from Ambassador Edge Stack.
The Prometheus data model is, at its core, time-series based. Therefore, it makes it easy to represent rates, averages, peaks, minimums, and histograms. Review the Prometheus documentation for a full reference on how to work with this data model.
Additional Install Options
StatsD Exporter: Output Statistics to Ambassador Edge Stack
If running a pre-0.71.0
version of Ambassador, you will need to configure Envoy to output stats to a separate collector before being scraped by Prometheus. You will use the Prometheus StatsD Exporter to do this.
Deploy the StatsD Exporter in the
default
namespacekubectl apply -f https://www.getambassador.io/yaml/monitoring/statsd-sink.yamlConfigure Ambassador Edge Stack to output statistics to
statsd
In the Ambassador Edge Stack deployment, add the
STATSD_ENABLED
andSTATSD_HOST
environment variables to tell Ambassador Edge Stack where to output stats....env:- name: AMBASSADOR_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: STATSD_ENABLEDvalue: "true"- name: STATSD_HOSTvalue: "statsd-sink.default.svc.cluster.local"...
Ambassador Edge Stack is now configured to output statistics to the Prometheus StatsD exporter.
ServiceMonitor
If you are scraping metrics from a statsd-sink
deployment, you will configure the ServiceMonitor
to scrape from that deployment.
---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:name: statsd-monitorlabels:app: ambassadorspec:namespaceSelector:matchNames:- defaultselector:matchLabels:service: statsd-sinkendpoints:- port: prometheus-metrics
Questions?
We’re here to help. If you have questions, join our Slack or contact us.