> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mithunai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# DevOps employee: investigates alerts and drafts postmortems

> The DevOps AI employee investigates an alert in Sentry, PagerDuty, Datadog or Grafana, states the likely cause and its confidence, and drafts postmortems.

The **DevOps** employee investigates an alert. It reads what you have connected -- metrics and logs,
the error behind the alert, recent deploys and the on-call incident -- and writes the **likely
cause**, with each piece of evidence numbered and a plain statement of **how sure it is**. It
proposes next steps: posting the investigation to the incident, a note and an acknowledgement on the
PagerDuty incident, and a rollback or restart that your team runs. After an incident is resolved it
drafts the **postmortem**.

**A fix is never run on your systems**, even when approved, and **nothing goes out without a
person's approval**.

> [!WARNING]
>
> In **beta**, off unless your deployment has switched it on. Grafana Cloud, Datadog, Sentry,
> GitHub, PagerDuty and Jira can be connected with a token you create. **None of them has yet been
> run against a real account.** See [AI employees](https://docs.mithunai.com/team/overview.md).

## How work reaches it

* **Sentry** sends an alert rule's notification, signed with your integration's client secret.
* **PagerDuty** sends `incident.triggered` (investigated) and `incident.resolved` (a postmortem to
  draft), signed with your webhook subscription's secret.
* **Your monitoring**, such as Grafana alerting, sends an alert with an API key
  ([Send an alert](https://docs.mithunai.com/api-reference/employees/send-alert.md)).
* **A person** hands one over on the DevOps page, or asks for a postmortem there.

The Sentry and PagerDuty addresses are shown beside each connection on the DevOps page. Each address
names a connection, not your organisation, and only a request carrying that vendor's valid signature
is read. See [Connect your tools](https://docs.mithunai.com/team/connect-your-tools.md#sentry-devops-errors).

## What an investigation does

1. **Reads the evidence** for the service the alert names: error rate and latency, recent log lines
   and recent deploys; for an alert from Sentry, the issue and its latest event -- exception,
   release and stack frames; for one from PagerDuty, the incident and its timeline. Reads need no
   approval.

2. **Numbers each piece of evidence** (`E1`, `E2` ...), quoting what the tool reported.

3. **Names a likely cause and says how sure it is.** Confidence comes from which structured signals
   agree, never from what any text says:

   | Likely cause | Confidence |
   | - | - |
   | A deploy in the 60 minutes before the error began | **High** when two independent signals agree (the error-rate rise, a Sentry error first seen within ten minutes, its release matching the deploy); otherwise **medium** |
   | A new error in your code (Sentry), no deploy close | **Medium** with an error-rate rise, otherwise **low** |
   | A pattern in the logs, no deploy close | **Medium** with an error-rate rise, otherwise **low** -- log text can be a symptom |
   | Nothing fits | No cause is named: *a person needs to look* |

   Even "high" is written as a hypothesis, with what would confirm or rule it out.

4. **Says what it did not read** -- a tool that is not connected, a service with no data -- so a
   missing tool is never mistaken for a clean bill.

5. **Asks your knowledge for the runbook**, when an assistant is set.

6. **Proposes next steps**, each waiting for your approval.

Log lines, exception messages, file names and incident titles are quoted and never followed: a log
line that says "roll back" is just a log line.

## What it proposes

| Action | What you can edit | Waits for a person |
| - | - | - |
| **Read logs, metrics, errors, deploys, incidents** | -- | Never: reads only |
| **Post the investigation to the incident ticket** | The summary | By default; you can switch it off |
| **Add the investigation to the incident as a note** | The note | By default; you can switch it off |
| **Acknowledge the incident** | Nothing | **Always** |
| **Roll back a deploy** | Nothing | **Always** |
| **Restart a service** | Nothing | **Always** |
| **Publish the postmortem** | The draft | **Always** |

> [!NOTE]
>
> **An approved rollback or restart is recorded and nothing is sent.** No connection carries a fix
> to your systems. **Nothing here resolves a PagerDuty incident**: only a note and an
> acknowledgement, each approved by a person.

## Postmortems

When PagerDuty reports an incident resolved -- or when you choose **Draft a postmortem** beside an
alert on the DevOps page -- it drafts one:

* **Summary** and **impact**: duration from the incident's timeline, the peak error rate in the
  hour before the draft, urgency, and who was affected (left for you to fill in).
* **Timeline** in UTC: deploys, the error-rate rise, when Sentry first saw the error, and PagerDuty's
  own log entries.
* **Cause**, with the same confidence statement, and the **evidence**.
* **Action items** for your team to agree.

The draft waits in **Approvals**. Edit it there; approving files it as a Jira issue when Jira is
connected. Without Jira nothing is sent, and you can **copy** or **download** it as Markdown.

## Tools it can connect

| Tool | Works with today | Not yet |
| - | - | - |
| Monitoring and deploys | **Grafana Cloud** or **Datadog** (metrics and logs), **Sentry** (errors) and **GitHub** (deploys). All read-only | -- |
| On-call incidents | **PagerDuty**: read, a note and an acknowledgement | -- |
| Incident ticket | **Jira** | ServiceNow |

[Connect your tools](https://docs.mithunai.com/team/connect-your-tools.md) has the steps for each.

## Limits

* **It never runs a fix**, and never resolves an incident.
* **The cause is a heuristic** over structured signals. It says so, and says how sure it is.
* **Metrics and logs cover the last hour** before the investigation or draft.
* **A Sentry issue is read only when Sentry sent the alert.** An alert from elsewhere does not search
  Sentry.
* **PagerDuty's US service region only** (`api.pagerduty.com`); an EU account cannot be connected
  yet.
* **A self-hosted Sentry must be reachable from the internet** over https; a private address is
  refused.

## Related

* [Approvals](https://docs.mithunai.com/team/approvals.md)
* [Connect your tools](https://docs.mithunai.com/team/connect-your-tools.md)
* [Work that arrives by itself](https://docs.mithunai.com/team/work-that-arrives-by-itself.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.