# Connecting telemetry sources AWS DevOps Agent provides three ways to connect to your telemetry sources. ## Built-in, 2-way integration Currently, AWS DevOps Agent supports Dynatrace users with a built-in, 2-way integration enabling the following: + **Topology resource mapping** - AWS DevOps Agent will augment your DevOps Agent Space Topology with entities and relationships available to it via a AWS DevOps Agent-hosted Dynatrace MCP server. + **Automated Investigation triggering** - Dynatrace Workflows can be configured to trigger incident resolution Investigations from Dynatrace Problems. + **Telemetry introspection** - AWS DevOps Agent can introspect Dynatrace telemetry as it investigates an issue via the AWS DevOps Agent-hosted Dynatrace MCP server. + **Status updates** - AWS DevOps Agent will publish key investigation findings, root cause analyses, and generated mitigation plans to the Dynatrace user interface. To learn about 2-way integrations, see + [Connecting Dynatrace](connecting-telemetry-sources-connecting-dynatrace.md) ## Built-in, 1-way integration Currently, AWS DevOps Agent supports AWS CloudWatch, Datadog, Grafana, New Relic, and Splunk users with built-in, 1 way integrations. **Security best practice:** When configuring credentials for built-in 1-way integrations, we recommend scoping API keys and tokens to read-only access. AWS DevOps Agent uses these credentials for telemetry introspection only and does not require write access to your telemetry provider. The AWS CloudWatch built-in, 1-way integration requires no additional setup and enables the following: + **Topology resource mapping** - AWS DevOps Agent will augment your DevOps Agent Space Topology with entities and relationships available to it via your configured primary and secondary AWS cloud accounts. + **Telemetry introspection** - AWS DevOps Agent can introspect AWS CloudWatch telemetry as it investigates an issue via the IAM role(s) provided during primary and secondary AWS cloud account configuration. The Datadog, Grafana, New Relic, and Splunk built-in, 1 way integrations require setup and enable the following: + **Automated Investigation triggering** - Datadog, Grafana, New Relic, and Splunk events can be configured to trigger AWS DevOps Agent incident resolution Investigations via AWS DevOps Agent webhooks. + **Telemetry introspection** - AWS DevOps Agent can introspect Datadog, Grafana, New Relic, and Splunk telemetry as it investigates an issue via each provider's remote MCP server. To learn about 1-way integrations, see the following: + [Connecting DataDog](connecting-telemetry-sources-connecting-datadog.md) + [Connecting Grafana](connecting-telemetry-sources-connecting-grafana.md) + [Connecting New Relic](connecting-telemetry-sources-connecting-new-relic.md) + [Connecting Splunk](connecting-telemetry-sources-connecting-splunk.md) ## Bring-your-own telemetry sources For any other telemetry source, including Prometheus metrics, you can leverage AWS DevOps Agent’s support for both webhook and MCP server integration. To learn about bring-your-own integrations, see the following + [Invoking DevOps Agent through Webhook](configuring-capabilities-for-aws-devops-agent-invoking-devops-agent-through-webhook.md) + [Connecting MCP Servers](configuring-capabilities-for-aws-devops-agent-connecting-mcp-servers.md) # Connecting Dynatrace ## Built-in, 2-way integration Currently, AWS DevOps Agent supports Dynatrace users with a built-in, 2-way integration enabling the following: + **Topology resource mapping** - AWS DevOps Agent will augment your DevOps Agent Space Topology with entities and relationships available to it from your Dynatrace environment. + **Automated Investigation triggering** - Dynatrace Workflows can be configured to trigger incident resolution Investigations from Dynatrace Problems. + **Telemetry introspection** - AWS DevOps Agent can introspect Dynatrace telemetry as it investigates an issue via the AWS DevOps Agent-hosted Dynatrace MCP server. + **Status updates** - AWS DevOps Agent will publish key investigation findings, root cause analyses, and generated mitigation plans to the Dynatrace user interface. ## Onboarding ### Onboarding Process Onboarding your Dynatrace observability system involves three stages: 1. **Connect** - Establish connection to Dynatrace by configuring account access credentials, with all the environments you may need 1. **Enable** - Activate Dynatrace in specific Agent spaces with specific Dynatrace environments 1. **Configure your Dynatrace environment** - download the workflows and dashboard and import into Dynatrace, making a note of the webhook details to trigger investigations in designated Agent spaces ### Step 1: Connect Establish connection to your Dynatrace environment #### Configuration 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Find **Dynatrace** in the **Available** providers section under **Telemetry** and click **Register** 1. **Create OAuth client in Dynatrace, with the detailed permissions.** 1. See [Dynatrace documentation](https://docs.dynatrace.com/docs/shortlink/aws-devops-agent) 1. When ready press next 1. You can connect multiple Dynatrace environments and later scope to specific ones for each DevOps Agent Space you may have. 1. Enter your Dynatrace details from the OAuth client setup: + **Client Name** + **Client ID** + **Client Secret** + **Account URN** 1. Click Next 1. Review and add ### Step 2: Enable Activate Dynatrace in a specific Agent space and configure appropriate scoping #### Configuration 1. From the agent spaces page, select an agent space and press view details 1. Select the Capabilities tab 1. Locate the Telemetry section, Press Add 1. You will notice Dynatrace with ‘Registered’ status. Click on add to add this to your agent space 1. Dynatrace Environment ID - Provide the Dynatrace environment ID you would like to associate with this DevOps agent space. 1. Enter one or more Dynatrace Entity IDs - these help DevOps agent discover your most important resources, examples might be services or applications. **If you are unsure you can press remove.** 1. Review and press Save 1. Copy the Webhook URL and Webhook Secret. See [Dynatrace documentation](https://docs.dynatrace.com/docs/shortlink/aws-devops-agent) to add these credentials to Dynatrace. ### Step 3: Configure your Dynatrace environment To complete your Dynatrace set up you will need to perform certain setup steps in your Dynatrace environment. Follow the instructions in the [Dynatrace documentation](https://docs.dynatrace.com/docs/shortlink/aws-devops-agent). #### Supported Event Schemas AWS DevOps Agent supports two types of events from Dynatrace using webhooks. The supported event schemas are documented below: ##### Incident Event Incident events are used to trigger an investigation. The event schema is: ``` { "event.id": string; "event.status": "ACTIVE" | "CLOSED"; "event.status_transition": string; "event.description": string; "event.name": string; "event.category": "AVAILABILITY" | "ERROR" | "SLOWDOWN" | "RESOURCE_CONTENTION" | "CUSTOM_ALERT" | "MONITORING_UNAVAILABLE" | "INFO"; "event.start"?: string; "affected_entity_ids"?: string[]; } ``` ##### Mitigation Event Mitigation events are used to trigger generating a mitigation report for the investigation on next steps. The event schema is: ``` { "task_id": string; "task_version": number; "event.type": "mitigation_request"; } ``` ## Removal The telemetry source is connected at two levels at the agent space level and at account level. To completely remove it you must first remove from all agent spaces where it is used and then it can be unregistered. ### Step 1: Remove from agent space 1. From the agent spaces page, select an agent space and press view details 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Select Dynatrace 1. Press remove ### Step 2: Deregister from account 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Scroll to the **Currently registered** section. 1. Check the agent space count is zero (if not repeat Step 1 above in your other agent spaces) 1. Press Deregister next to Dynatrace # Connecting DataDog ## Built-in, 1 way integration Currently, AWS DevOps Agent supports Datadog users with built-in, 1 way integration, enabling the following: + **Automated Investigation triggering** - Datadog events can be configured to trigger AWS DevOps Agent incident resolution Investigations via AWS DevOps Agent webhooks. + **Telemetry introspection** - AWS DevOps Agent can introspect Datadog telemetry as it investigates an issue via each provider's remote MCP server. ## Onboarding ### Step 1: Connect Establish connection to your Datadog remote MCP endpoint with account access credentials #### Configuration 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Find **Datadog** in the **Available** providers section under **Telemetry** and click **Register** 1. Enter your Datadog MCP server details: + **Server Name** - Unique identifier (e.g., my-datadog-server) + **Endpoint URL** - Your Datadog MCP server endpoint. The endpoint URL varies depending on your Datadog site. See the Datadog site endpoint table below. + **Description** - Optional server description 1. Click Next 1. Review and submit #### Datadog site endpoints The MCP endpoint URL varies depending on your Datadog site. To identify your site, check the URL in your browser when logged into Datadog, or see [Access the Datadog site](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site). | Datadog Site | Site Domain | MCP Endpoint URL | | --- | --- | --- | | US1 (default) | datadoghq.com | https://mcp.datadoghq.com/api/unstable/mcp-server/mcp | | US3 | us3.datadoghq.com | https://mcp.us3.datadoghq.com/api/unstable/mcp-server/mcp | | US5 | us5.datadoghq.com | https://mcp.us5.datadoghq.com/api/unstable/mcp-server/mcp | | EU1 | datadoghq.eu | https://mcp.datadoghq.eu/api/unstable/mcp-server/mcp | | AP1 | ap1.datadoghq.com | https://mcp.ap1.datadoghq.com/api/unstable/mcp-server/mcp | | AP2 | ap2.datadoghq.com | https://mcp.ap2.datadoghq.com/api/unstable/mcp-server/mcp | #### Authorization Complete OAuth authorization by: + Authorizing as your user on the Datadog OAuth page + If not logged in, click Allow, login, then authorize Once configured, Datadog becomes available across all Agent spaces. ### Step 2: Enable Activate DataDog in a specific Agent space and configure appropriate scoping #### Configuration 1. From the agent spaces page, select an agent space and press view details (if you have not yet created an agent space see [Creating an Agent Space](getting-started-with-aws-devops-agent-creating-an-agent-space.md)) 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Press Add 1. Select Datadog 1. Next 1. Review and press Save 1. Copy the Webhook URL and API Key ### Step 3: Configure webhooks Using the Webhook URL and API Key you can configure Datadog to send events to trigger an investigation, for example from an alarm. To ensure that events sent can be used by the DevOps Agent, make sure that the data transmitted to the webhook matches the data schema specified below. Events that do not match this schema may be ignored by DevOps Agent. Set the method and the headers ``` method: "POST", headers: { "Content-Type": "application/json", "Authorization": "Bearer ", }, ``` Send the body as a JSON string. ``` { eventType: 'incident'; incidentId: string; action: 'created' | 'updated' | 'closed' | 'resolved'; priority: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL"; title: string; description?: string; timestamp?: string; service?: string; // The original event generated by service is attached here. data?: object; } ``` Send webhooks with Datadog [https://docs.datadoghq.com/integrations/webhooks/](https://docs.datadoghq.com/integrations/webhooks/) (note select no authorization and instead use the custom header option). Learn more: [Datadog Remote MCP Server](https://www.datadoghq.com/blog/datadog-remote-mcp-server/) ## Removal The telemetry source is connected at two levels at the agent space level and at account level. To completely remove it you must first remove from all agent spaces where it is used and then it can be unregistered. ### Step 1: Remove from agent space 1. From the agent spaces page, select an agent space and press view details 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Select Datadog 1. Press remove ### Step 2: Deregister from account 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Scroll to the **Currently registered** section. 1. Check the agent space count is zero (if not repeat Step 1 above in your other agent spaces) 1. Press Deregister next to Datadog # Connecting Grafana Grafana integration enables AWS DevOps Agent to query metrics, dashboards, and alerting data from your Grafana instance during incident investigations. This integration follows a two-step process: account-level registration of Grafana, followed by connecting it to individual Agent Spaces. To improve security, the Grafana integration only enables read-only tools. Write tools are disabled and cannot be enabled. This means the agent can query and read data from your Grafana instance but cannot create, modify, or delete any Grafana resources such as dashboards, alerts, or annotations. For more information, see [Security in AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/aws-devops-agent-security.html). ## Grafana requirements Before connecting Grafana, ensure you have: + Grafana version 9.0 or later. Some features, particularly datasource-related operations, may not work correctly with earlier versions due to missing API endpoints. + A Grafana instance accessible over HTTPS. Both public and private network endpoints are supported. With private network connectivity, your Grafana instance can be hosted inside a VPC with no public internet access. For details, see [Connecting to privately hosted tools](configuring-capabilities-for-aws-devops-agent-connecting-to-privately-hosted-tools.md). + A Grafana service account with an access token that has appropriate read permissions ## Registering Grafana (account-level) Grafana is registered at the AWS account level and shared among all Agent Spaces in that account. ### Step 1: Configure Grafana 1. Sign in to the AWS Management Console 1. Navigate to the AWS DevOps Agent console 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Find **Grafana** in the **Available** providers section under **Telemetry** and click **Register** 1. On the **Configure Grafana** page, enter the following information: + **Service Name** (required) – Enter a descriptive name for your Grafana server using alphanumeric characters, hyphens, and underscores only. For example, `my-grafana-server`. + **Grafana URL** (required) – Enter the full HTTPS URL of your Grafana instance. For example, `https://myinstance.grafana.net`. + **Service Account Access Token** (required) – Enter a Grafana service account access token. Tokens typically start with `glsa_`. To create a service account token, navigate to your Grafana instance, go to **Administration > Service accounts**, create a service account with Viewer role, and generate a token. + **Description** (optional) – Add a description to help identify the server's purpose. For example, `Production Grafana server for monitoring`. 1. (Optional) Add AWS tags to the registration for organizational purposes. 1. Click **Next** ### Step 2: Review and submit Grafana registration 1. Review all the Grafana configuration details 1. Click **Submit** to complete the registration 1. Upon successful registration, Grafana appears in the **Currently registered** section of the Capability Providers page ## Adding Grafana to an Agent Space After registering Grafana at the account level, you can connect it to individual Agent Spaces: 1. In the AWS DevOps Agent console, select your Agent Space 1. Go to the **Capabilities** tab 1. In the **Telemetry** section, click **Add** 1. Select **Grafana** from the list of available providers 1. Click **Save** ## Configuring Grafana alert webhooks You can configure Grafana to automatically trigger AWS DevOps Agent investigations when alerts fire by sending webhooks through Grafana contact points. For details on webhook authentication methods and credential management, see [Invoking DevOps Agent through Webhook](configuring-capabilities-for-aws-devops-agent-invoking-devops-agent-through-webhook.md). ### Step 1: Create a custom notification template In your Grafana instance, navigate to **Alerting > Contact points > Notification templates** and create a new template with the following content: ``` {{ define "devops-agent-payload" }} { "eventType": "incident", "incidentId": "{{ (index .Alerts 0).Labels.alertname }}-{{ (index .Alerts 0).Fingerprint }}", "action": "{{ if eq .Status "resolved" }}resolved{{ else }}created{{ end }}", "priority": "{{ if eq .Status "resolved" }}MEDIUM{{ else }}HIGH{{ end }}", "title": "{{ (index .Alerts 0).Labels.alertname }}", "description": "{{ (index .Alerts 0).Annotations.summary }}", "service": "{{ if (index .Alerts 0).Labels.job }}{{ (index .Alerts 0).Labels.job }}{{ else }}grafana{{ end }}", "timestamp": "{{ (index .Alerts 0).StartsAt }}", "data": { "metadata": { {{ range $k, $v := (index .Alerts 0).Labels }} "{{ $k }}": "{{ $v }}", {{ end }} "_source": "grafana" } } } {{ end }} ``` This template formats Grafana alerts into the webhook payload structure expected by AWS DevOps Agent. It maps alert labels, annotations, and status into the appropriate fields, and includes all alert labels as metadata. **Note:** This template processes only the first alert in a group. Grafana groups multiple firing alerts into a single notification by default. To ensure each alert is sent individually, configure your notification policies to group by `alertname`. Additionally, this template does not escape special JSON characters in label values or annotations. Ensure that alert labels and the `summary` annotation do not contain characters such as double quotes or newlines, which would produce invalid JSON. ### Step 2: Create a webhook contact point 1. In Grafana, navigate to **Alerting > Contact points** and click **Add contact point** 1. Select **Webhook** as the integration type 1. Set the **URL** to your AWS DevOps Agent webhook endpoint 1. Under **Optional Webhook settings**, configure the authentication headers based on your webhook type. See [Webhook authentication methods](configuring-capabilities-for-aws-devops-agent-invoking-devops-agent-through-webhook.md) for details. 1. Set the **Message** field to use your custom template: `{{ template "devops-agent-payload" . }}` 1. Click **Save contact point** ### Step 3: Assign the contact point to a notification policy 1. Navigate to **Alerting > Notification policies** 1. Edit an existing policy or create a new one 1. Set the contact point to the webhook contact point you created 1. Click **Save policy** When a matching alert fires, Grafana will send the formatted payload to AWS DevOps Agent, which will start an investigation automatically. ## Limitations + **ClickHouse data source tools** – ClickHouse data source tools are not currently supported. + **Proactive incident prevention** – [Proactive incident prevention](working-with-devops-agent-proactive-incident-prevention.md) does not currently use Grafana tools. Support is planned for a future release. ### Amazon Managed Grafana considerations If you are using [Amazon Managed Grafana](https://aws.amazon.com/grafana/) (AMG), be aware of the following limitations: + **Webhook contact points are not supported** – AMG does not currently support webhook contact points in its alerting configuration. You cannot use AMG to send alert webhooks directly to AWS DevOps Agent. For details, see [Alerting contact points in Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/v9-alerting-explore-contacts.html). + **Service account token expiration** – AMG service account tokens have a maximum expiration of 30 days. You will need to rotate tokens and update your Grafana registration in AWS DevOps Agent before they expire. See [Managing Grafana connections](#managing-grafana-connections) for how to update credentials. For details on AMG token limits, see [Service accounts in Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/service-accounts.html). ## Managing Grafana connections + **Updating credentials** – If your service account token expires or needs to be updated, deregister Grafana from the Capability Providers page and re-register with the new token. + **Viewing connected instances** – In the AWS DevOps Agent console, select your Agent Space and go to the Capabilities tab to view connected telemetry sources. + **Removing Grafana** – To disconnect Grafana from an Agent Space, select it in the Telemetry section and click **Remove**. To completely remove the registration, remove it from all Agent Spaces first, then deregister from the Capability Providers page. # Connecting New Relic ## Built-in, 1 way integration Currently, AWS DevOps Agent supports New Relic users with built-in, 1 way integration, enabling the following: + **Automated Investigation triggering** - New Relic events can be configured to trigger AWS DevOps Agent incident resolution Investigations via AWS DevOps Agent webhooks. + **Telemetry introspection** - AWS DevOps Agent can introspect New Relic telemetry as it investigates an issue via each provider's remote MCP server. ## Onboarding ### Step 1: Connect Establish connection to your New Relic remote MCP endpoint with account access credentials Please use a Full Platform User (not Basic/Core) in New relic to enable New Relic MCP tools. #### Configuration 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Find **New Relic** in the **Available** providers section under **Telemetry** and click **Register** 1. Follow the instructions to obtain your New Relic API Key 1. Enter your New Relic MCP server API Key details: + **Account ID:** Enter your New Relic account ID obtained above + **API Key:** Enter the API Key obtained above + **Select US or EU region** based on where your New Relic account is. 1. Click Add ### Step 2: Enable Activate New Relic in a specific Agent space and configure appropriate scoping #### Configuration 1. From the agent spaces page, select an agent space and press view details (if you have not yet created an agent space see [Creating an Agent Space](getting-started-with-aws-devops-agent-creating-an-agent-space.md)) 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Press Add 1. Select New Relic 1. Next 1. Review and press Save 1. Copy the Webhook URL and API Key ### Step 3: Configure webhooks Using the Webhook URL and API Key you can configure New Relic to send events to trigger an investigation, for example from an alarm. For more details on setting up webhooks, see [Change tracking webhooks](https://docs.newrelic.com/docs/change-tracking/change-tracking-webhooks/). To ensure that events sent can be used by the DevOps Agent, make sure that the data transmitted to the webhook matches the data schema specified below. Events that do not match this schema may be ignored by DevOps Agent. Set the method and the headers ``` method: "POST", headers: { "Content-Type": "application/json", "Authorization": "Bearer ", }, ``` Send the body as a JSON string. ``` { eventType: 'incident'; incidentId: string; action: 'created' | 'updated' | 'closed' | 'resolved'; priority: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL"; title: string; description?: string; timestamp?: string; service?: string; // The original event generated by service is attached here. data?: object; } ``` Send webhooks with New Relic [https://newrelic.com/instant-observability/webhook-notifications](https://newrelic.com/instant-observability/webhook-notifications). You can either select Bearer token for the authorization type, or select no authorization and add the `Authorization: Bearer ` as a custom header instead. Learn more: [https://docs.newrelic.com/docs/agentic-ai/mcp/overview/](https://docs.newrelic.com/docs/agentic-ai/mcp/overview/) ## Removal The telemetry source is connected at two levels at the agent space level and at account level. To completely remove it you must first remove from all agent spaces where it is used and then it can be unregistered. ### Step 1: Remove from agent space 1. From the agent spaces page, select an agent space and press view details 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Select New Relic 1. Press remove ### Step 2: Deregister from account 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Scroll to the **Currently registered** section. 1. Check the agent space count is zero (if not repeat Step 1 above in your other agent spaces) 1. Press Deregister next to New Relic # Connecting Splunk ## Built-in, 1 way integration Currently, AWS DevOps Agent supports Splunk users with built-in, 1 way integration, enabling the following: + **Automated Investigation triggering** - Splunk events can be configured to trigger AWS DevOps Agent incident resolution Investigations via AWS DevOps Agent webhooks. + **Telemetry introspection** - AWS DevOps Agent can introspect Splunk telemetry as it investigates an issue via each provider's remote MCP server. ## Prerequisites ### Getting a Splunk API token You will need an MCP URL and token to connect Splunk. ### Splunk Administrator steps Your Splunk Administrator needs to perform the following steps: + enable [REST API access ](https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud) + [enable token authentication ](https://help.splunk.com/en/splunk-cloud-platform/administer/manage-users-and-security/9.2.2406/authenticate-into-the-splunk-platform-with-tokens/enable-or-disable-token-authentication) on the deployment. + create a new role 'mcp\$1user', the new role does not need to have any capabilities. + assign the role 'mcp\$1user' to any users on the deployment who are authorized to use the MCP server. + create the token for the authorized users with audience as 'mcp' and set the appropriate expiration, if the user does not have the permission to create tokens themselves. ### Splunk User steps A Splunk user needs to perform the following steps: + Get an appropriate token from the Splunk Administrator or create one themselves, if they have the permission. The audience for the token must be 'mcp'. ## Onboarding ### Step 1: Connect Establish connection to your Splunk remote MCP endpoint with account access credentials #### Configuration 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Find **Splunk** in the **Available** providers section under **Telemetry** and click **Register** 1. Enter your Splunk MCP server details: + **Server Name** - Unique identifier (e.g., my-splunk-server) + **Endpoint URL** - Your Splunk MCP server endpoint: `https://.api.scs.splunk.com//mcp/v1/` + **Description** - Optional server description + **Token Name** - The name of the bearer token for authentication: `my-splunk-token` + **Token Value** The bearer token value for authentication ### Step 2: Enable Activate Splunk in a specific Agent space and configure appropriate scoping #### Configuration 1. From the agent spaces page, select an agent space and press view details (if you have not yet created an agent space see [Creating an Agent Space](getting-started-with-aws-devops-agent-creating-an-agent-space.md)) 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Press Add 1. Select Splunk 1. Next 1. Review and press Save 1. Copy the Webhook URL and API Key ### Step 3: Configure webhooks Using the Webhook URL and API Key you can configure Splunk to send events to trigger an investigation, for example from an alarm. To ensure that events sent can be used by the DevOps Agent, make sure that the data transmitted to the webhook matches the data schema specified below. Events that do not match this schema may be ignored by DevOps Agent. Set the method and the headers ``` method: "POST", headers: { "Content-Type": "application/json", "Authorization": "Bearer ", }, ``` Send the body as a JSON string. ``` { eventType: 'incident'; incidentId: string; action: 'created' | 'updated' | 'closed' | 'resolved'; priority: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL"; title: string; description?: string; timestamp?: string; service?: string; // The original event generated by service is attached here. data?: object; } ``` Send webhooks with Splunk [https://help.splunk.com/en/splunk-enterprise/alert-and-respond/alerting-manual/9.4/configure-alert-actions/use-a-webhook-alert-action](https://help.splunk.com/en/splunk-enterprise/alert-and-respond/alerting-manual/9.4/configure-alert-actions/use-a-webhook-alert-action) (note select no authorization and instead use the custom header option) ### Learn more: + Splunk's MCP Server Documentation: [https://help.splunk.com/en/splunk-cloud-platform/mcp-server-for-splunk-platform/about-mcp-server-for-splunk-platform ](https://help.splunk.com/en/splunk-cloud-platform/mcp-server-for-splunk-platform/about-mcp-server-for-splunk-platform) + Access requirements and limitations for the Splunk Cloud Platform REST API: [https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud ](https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud) + Manage authentication tokens in Splunk Cloud Platform: [https://help.splunk.com/en/splunk-cloud-platform/administer/manage-users-and-security/9.3.2411/authenticate-into-the-splunk-platform-with-tokens/manage-or-delete-authentication-tokens ](https://help.splunk.com/en/splunk-cloud-platform/administer/manage-users-and-security/9.3.2411/authenticate-into-the-splunk-platform-with-tokens/manage-or-delete-authentication-tokens) + Create and manage roles with Splunk Web: [https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Addandeditroles ](https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Addandeditroles) ## Removal The telemetry source is connected at two levels at the agent space level and at account level. To completely remove it you must first remove it from all agent spaces where it is used and then it can be unregistered. ### Step 1: Remove from agent space 1. From the agent spaces page, select an agent space and press view details 1. Select the Capabilities tab 1. Scroll down to the Telemetry section 1. Select Splunk 1. Press remove ### Step 2: Deregister from account 1. Go to the **Capability Providers** page (accessible from the side navigation) 1. Scroll to the **Currently registered** section. 1. Check the agent space count is zero (if not repeat Step 1 above in your other agent spaces) 1. Press Deregister next to Splunk