Connecting Splunk
Built-in, 1 way integration
Currently, AWS DevOps Agent supports Splunk users with built-in, 1 way integration, enabling the following:
Automated Investigation triggering - Splunk events can be configured to trigger AWS DevOps Agent incident resolution Investigations via AWS DevOps Agent webhooks.
Telemetry introspection - AWS DevOps Agent can introspect Splunk telemetry as it investigates an issue via the each providers remote MCP server.
Prerequisites
Getting a Splunk API token
You will need an MCP URL and token to connect Splunk.
Splunk Administrator steps
Your Splunk Administrator needs to perform the following steps:
enable REST API access
enable token authentication
on the deployment. create a new role 'mcp_user', the new role does not need to have any capabilities.
assign the role 'mcp_user' to any users on the deployment who are authorized to use the MCP server.
create the token for the authorized users with audience as 'mcp' and set the appropriate expiration, if the user does not have the permission to create tokens themselves.
Splunk User steps
A Splunk user needs to perform the following steps:
Get an appropriate token from the Splunk Administrator or create one themselves, if they have the permission. The audience for the token must be 'mcp'.
Onboarding
Step 1: Connect
Establish connection to your Splunk remote MCP endpoint with account access credentials
Configuration
Open the hamburger menu and select Settings
Scroll to the Available - Telemetry section. Press Register next to Splunk
Enter your Splunk MCP server details:
Server Name - Unique identifier (e.g., my-splunk-server)
Endpoint URL - Your Splunk MCP server endpoint:
https://<YOUR_SPLUNK_DEPLOYMENT_NAME>.api.scs.splunk.com/<YOUR_SPLUNK_DEPLOYMENT_NAME>/mcp/v1/
Description - Optional server description
Token Name - The name of the bearer token for authentication:
my-splunk-tokenToken Value The bearer token value for authentication
Step 2: Enable
Activate Splunk in a specific Agent space and configure appropriate scoping
Configuration
From the agent spaces page, select an agent space and press view details (if you have not yet created an agent space see Creating an Agent Space)
Select the Capabilities tab
Scroll down to the Telemetry section
Press Add
Select Splunk
Next
Review and press Save
Copy the Webhook URL and API Key
Step 3: Configure webhooks
Using the Webhook URL and API Key you can configure Splunk to send events to trigger an investigation, for example from an alarm.
To ensure that events sent can be used by the DevOps Agent, make sure that the data transmitted to the webhook matches the data schema specified below. Events that do not match this schema may be ignored by DevOps Agent.
Set the method and the headers
method: "POST", headers: { "Content-Type": "application/json", "Authorization": "Bearer <Token>", },
Send the body as a JSON string.
{ eventType: 'incident'; incidentId: string; action: 'created' | 'updated' | 'closed' | 'resolved'; priority: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL"; title: string; description?: string; timestamp?: string; service?: string; // The original event generated by service is attached here. data?: object; }
Send webhooks with Splunk https://help.splunk.com/en/splunk-enterprise/alert-and-respond/alerting-manual/9.4/configure-alert-actions/use-a-webhook-alert-action
Learn more:
Splunk's MCP Server Documentation: https://help.splunk.com/en/splunk-cloud-platform/mcp-server-for-splunk-platform/about-mcp-server-for-splunk-platform
Access requirements and limitations for the Splunk Cloud Platform REST API: https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud
Manage authentication tokens in Splunk Cloud Platform: https://help.splunk.com/en/splunk-cloud-platform/administer/manage-users-and-security/9.3.2411/authenticate-into-the-splunk-platform-with-tokens/manage-or-delete-authentication-tokens
Create and manage roles with Splunk Web: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/Addandeditroles
Removal
The telemetry source is connected at two levels at the agent space level and at account level. To completely remove it you must first remove from all agentspaces where it is used and then it can be unregistered.
Step 1: Remove from agent space
From the agent spaces page, select an agent space and press view details
Select the Capabilities tab
Scroll down to the Telemetry section
Select Splunk
Press remove
Step 2: Remove from agent space
Open the hamburger menu and select Settings
Scroll to the Currently registered section.
Check the agent space count is zero (if not repeat Step 1 above in your other agent spaces)
Press Deregister next to Splunk
