Incidents API (Beta)
Introduction
Incidents are a concept used to flag particular Data Assets as being in an unhealthy state. Each incident has an independent lifecycle and details including a state (active, resolved), a title, a description, & more.
A couple scenarios in which incidents can be useful are
- Pipeline Circuit Breaking: You can use Incidents as the basis for intelligent data pipelines that verify upstream inputs (e.g. datasets) are free of any active incidents before executing.
- [Coming Soon] Announcing Known-Bad Assets: You can mark a known-bad data asset as under an ongoing incident so consumers and stakeholders can be informed about the health status of a data asset via the DataHub UI. Moreover, they can follow the incident as it progresses toward resolution.
In the next section, we'll show you how to
- Create a new incident
- Fetch all incidents for a data asset
- Resolve an incident
for Datasets using the Acryl GraphQL API.
Let's get started!
Creating an Incident
Creating incidents is currently only supported against Dataset assets.
To create (i.e. raise) a new incident for a data asset, simply create a GraphQL request using the raiseIncident mutation.
type Mutation {
"""
Raise a new incident for a data asset
"""
raiseIncident(input: RaiseIncidentInput!): String! # Returns new Incident URN.
}
input RaiseIncidentInput {
"""
The type of incident, e.g. OPERATIONAL
"""
type: IncidentType!
"""
A custom type of incident. Present only if type is 'CUSTOM'
"""
customType: String
"""
An optional title associated with the incident
"""
title: String
"""
An optional description associated with the incident
"""
description: String
"""
The resource (dataset, dashboard, chart, dataFlow, etc) that the incident is associated with.
"""
resourceUrn: String!
"""
The source of the incident, i.e. how it was generated
"""
source: IncidentSourceInput
}
Examples
First, we'll create a demo GraphQL query, then show how to represent it via CURL & Python.
Imagine we want to raise a new incident on a Dataset with URN urn:li:dataset:(abc) because it's failed automated quality checks. To do so, we could make the following GraphQL query:
Request
mutation raiseIncident {
raiseIncident(input: {
type: OPERATIONAL
title: "Dataset Failed Quality Checks"
description: "Dataset failed 2/6 Quality Checks for suite run id xy123mksj812pk23."
resourceUrn: "urn:li:dataset:(urn:li:dataPlatform:kafka,SampleKafkaDataset,PROD)"
})
}
After we make this query, we will get back a unique URN for the incident.
Response
{
"data": {
"raiseIncident": "urn:li:incident:bfecab62-dc10-49a6-a305-78ce0cc6e5b1"
}
}
Now we'll see how to issue this query using a CURL or Python.
CURL
To issue the above GraphQL as a CURL:
curl --location --request POST 'https://your-account.acryl.io/api/graphql' \
--header 'Authorization: Bearer your-access-token' \
--header 'Content-Type: application/json' \
--data-raw '{"query":"mutation raiseIncident {\n raiseIncident(input: {\n type: OPERATIONAL\n title: \"Dataset Failed Quality Checks\"\n description: \"Dataset failed 2/6 Quality Checks for suite run id xy123mksj812pk23.\"\n resourceUrn: \"urn:li:dataset:(urn:li:dataPlatform:kafka,SampleKafkaDataset,PROD)\"\n })\n}","variables":{}}'
Python
To issue the above GraphQL query in Python (requests):
import requests
datahub_session = requests.Session()
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer your-personal-access-token",
}
json = {
"query": """mutation raiseIncident {\n
raiseIncident(input: {\n
type: OPERATIONAL\n
resourceUrn: \"urn:li:dataset:(urn:li:dataPlatform:kafka,SampleKafkaDataset,PROD)\"\n
})}""",
"variables": {},
}
response = datahub_session.post(f"https://your-account.acryl.io/api/graphql", headers=headers, json=json)
response.raise_for_status()
res_data = response.json() # Get result as JSON
Retrieving Active Incidents
To fetch the the ongoing incidents for a data asset, we can use the incidents GraphQL field on the entity of interest.
Datasets
To retrieve all incidents for a Dataset with a particular URN, you can reference the 'incidents' field of the Dataset type:
type Dataset {
....
"""
Incidents associated with the Dataset
"""
incidents(
"""
Optional incident state to filter by, defaults to any state.
"""
state: IncidentState,
"""
Optional start offset, defaults to 0.
"""
start: Int,
"""
Optional start offset, defaults to 20.
"""
count: Int): EntityIncidentsResult # Returns a list of incidents.
}
Examples
Now that we've raised an incident on it, imagine we want to fetch the first 10 "active" incidents for the Dataset with URN urn:li:dataset:(abc). To do so, we could issue the following request:
Request
query dataset {
dataset(urn: "urn:li:dataset:(abc)") {
incidents(state: ACTIVE, start: 0, count: 10) {
total
incidents {
urn
title
description
status {
state
}
}
}
}
}
After we make this query, we will get back a unique URN for the incident.
Response
{
"data": {
"dataset": {
"incidents": {
"total": 1,
"incidents": [
{
"urn": "urn:li:incident:bfecab62-dc10-49a6-a305-78ce0cc6e5b1",
"title": "Dataset Failed Quality Check",
"description": "Dataset failed 2/6 Quality Checks for suite run id xy123mksj812pk23.",
"status": {
"state": "ACTIVE"
}
}
]
}
}
}
}
Now we'll see how to issue this query using a CURL or Python.
CURL
To issue the above GraphQL as a CURL:
curl --location --request POST 'https://your-account.acryl.io/api/graphql' \
--header 'Authorization: Bearer your-access-token' \
--header 'Content-Type: application/json' \
--data-raw '{"query":"query dataset {\n dataset(urn: "urn:li:dataset:(abc)") {\n incidents(state: ACTIVE, start: 0, count: 10) {\n total\n incidents {\n urn\n title\n description\n status {\n state\n }\n }\n }\n }\n}","variables":{}}'Python
To issue the above GraphQL query in Python (requests):
import requests
datahub_session = requests.Session()
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer your-personal-access-token",
}
json = {
"query": """query dataset {\n
dataset(urn: "urn:li:dataset:(abc)") {\n
incidents(state: ACTIVE, start: 0, count: 10) {\n
total\n
incidents {\n
urn\n
title\n
description\n
status {\n
state\n
}\n
}\n
}\n
}\n
}""",
"variables": {},
}
response = datahub_session.post(f"https://your-account.acryl.io/api/graphql", headers=headers, json=json)
response.raise_for_status()
res_data = response.json() # Get result as JSON
Resolving an Incident
To resolve an incident for a data asset, simply create a GraphQL request using the updateIncidentStatus mutation. To mark an incident as resolved, simply update its state to RESOLVED.
type Mutation {
"""
Update an existing incident for a resource (asset)
"""
updateIncidentStatus(
"""
The urn for an existing incident
"""
urn: String!
"""
Input required to update the state of an existing incident
"""
input: UpdateIncidentStatusInput!): String
}
"""
Input required to update status of an existing incident
"""
input UpdateIncidentStatusInput {
"""
The new state of the incident
"""
state: IncidentState!
"""
An optional message associated with the new state
"""
message: String
}
Examples
Imagine that we've fixed our Dataset with urn urn:li:dataset:(abc) so that it's passing validation. Now we want to mark the Dataset as healthy, so stakeholders and downstream consumers know it's ready to use.
To do so, we need the URN of the Incident that we raised previously.
Request
mutation updateIncidentStatus {
updateIncidentStatus(urn: "urn:li:incident:bfecab62-dc10-49a6-a305-78ce0cc6e5b1",
input: {
state: RESOLVED
message: "Dataset is now passing validations. Verified by John Joyce on Data Platform eng."
})
}
Response
{
"data": {
"updateIncidentStatus": "true"
}
}
True is returned if the incident's was successfully marked as resolved.
CURL
To issue the above GraphQL as a CURL:
curl --location --request POST 'https://your-account.acryl.io/api/graphql' \
--header 'Authorization: Bearer your-access-token' \
--header 'Content-Type: application/json' \
--data-raw '{"query":"mutation updateIncidentStatus {\n updateIncidentStatus(urn: "urn:li:incident:bfecab62-dc10-49a6-a305-78ce0cc6e5b1", \n input: {\n state: RESOLVED\n message: "Dataset is now passing validations. Verified by John Joyce on Data Platform eng."\n })\n}","variables":{}}'Python
To issue the above GraphQL query in Python (requests):
import requests
datahub_session = requests.Session()
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer your-personal-access-token",
}
json = {
"query": """mutation updateIncidentStatus {\n
updateIncidentStatus(urn: \"urn:li:incident:bfecab62-dc10-49a6-a305-78ce0cc6e5b1\",\n
input: {\n
state: RESOLVED\n
message: \"Dataset is now passing validations. Verified by John Joyce on Data Platform eng.\"\n
})\n
}""",
"variables": {},
}
response = datahub_session.post(f"https://your-account.acryl.io/api/graphql", headers=headers, json=json)
response.raise_for_status()
res_data = response.json() # Get result as JSON
Tips
Authorization
Remember to always provide a DataHub Personal Access Token when calling the GraphQL API. To do so, just add the 'Authorization' header as follows:
Authorization: Bearer <personal-access-token>
Exploring GraphQL API
Also, remember that you can play with an interactive version of the Acryl GraphQL API at https://your-account-id.acryl.io/api/graphiql
Enabling Slack Notifications
You can configure Acryl to send slack notifications to a specific channel when incidents are raised or their status is changed.
These notifications are also able to tag the immediate asset's owners, along with the owners of downstream assets consuming it.
To do so, simply follow the Slack Integration Guide and contact your Acryl customer success team to enable the feature!