xMatters FAQs
xMatters's Frequently Asked Questions page is a central hub where its customers can always go to with their most common questions. These are the 386 most popular questions xMatters receives.
Frequently Asked Questions About xMatters
Here's a list of all our upcoming workshops.
We've split the calendars into regions to make it easy to find the courses closest to your time zone. But if you're a night owl or an early riser, feel free to join a workshop in any region that suits your schedule.
See something you like? Click the link in the Topic column to find out more about the course, or click the link in the Register column to sign up for it right now.
Courses by region:
North America
Europe, Middle East, and Africa
Asia, Pacific, Australia
(Want to know whenever this list gets updated? Just click the Follow button at the top of this article.)
If you have any questions about the workshops, or want to connect with the instructors and fellow attendees before or after the session, visit our forum and join the conversation!
Workshops and Special Events community forum
North AmericaAll times in New York / Eastern time and observe Day Light Savings
Date & Time
Topic
Register
January 29
9:00 AM - 11:00 AM
Integration Workshop: ServiceNow
Finished
January 30
9:00 AM - 12:00 PM
Developer Workshop: REST APIs
Finished
February 10
10:00 AM - 10:30 AM
xMatters 101: Getting Started
Sign up now!
February 11
9:00 AM - 12:00 PM
Developer Workshop: Flow Designer Workshop
Sign up now!
February 12
10:00 AM - 11:00 AM
Integration Workshop: Slack
Sign up now!
February 13
9:00 AM - 12:00 PM
Developer Workshop: REST APIs
Sign up now!
February 17
11:00 AM - 1:00 PM
xMatters 201: Overview & Admin Review
Sign up now!
February 18
11:00 AM - 2:00 PM
xMatters 202: Users
Sign up now!
February 19
11:00 AM - 3:00 PM
xMatters 203: Groups & Teams
Sign up now!
February 20
11:00 AM - 3:00 PM
xMatters 204: Workflows
Sign up now!
February 21
11:00 AM - 1:30 PM
xMatters 205: Reporting and Workflow Add-Ons
Sign up now!
February 5, 2020 05:01 AM
EMEA
All times in London BST/GMT and observe Day Light Savings
Date & Time
Topic
Register
January 29
9:00 AM - 12:00 PM
Developer Workshop: Flow Designer Workshop
Finished
2:00 PM - 4:00 PM
Integration Workshop: ServiceNow
Finished
January 30
11:00 AM - 12:00 PM
Integration Workshop: Slack
Finished
2:00 PM - 5:00 PM
Developer Workshop: REST APIs
Finished
February 10
3:00 PM - 3:30 PM
xMatters 101: Getting Started
Sign up now!
February 11
2:00 PM - 5:00 PM
Developer Workshop: Flow Designer Workshop
Sign up now!
February 12
3:00 PM - 4:00 PM
Integration Workshop: Slack
Sign up now!
February 13
2:00 PM - 5:00 PM
Developer Workshop: REST APIs
Sign up now!
March 2 to March 6 (5-day course)
9:00 AM - 1:00 PM
New Administrators Workshop
Sign up now!
February 5, 2020 05:01 AM
Asia Pacific
All times in Australia Eastern and observe Day Light Savings
Date & Time
Topic
Register
March 2 to March 6 (5-day course)
1:00 PM - 5:00 PM
New Administrators Workshop
Sign up now!
February 5, 2020 05:01 AM
View ArticleJump to Class Schedules:
xMatters 101
North America
Europe, Middle East, and Africa
Asia, Pacific, Australia
xMatters 201, 202, 203, 204 205
North America
Europe, Middle East, and Africa
Asia, Pacific, Australia
xMatters 101
Got questions about xMatters?
Whetheryou're new to xMatters or looking to brush up, you're probably wondering what's the best way to get started. Maybe you find yourself asking someone any of the following questions:
What resources are available to new users?
Where can new admins go to find out about configuration options?
Who do I ask about this feature I want to use?
Where can I find instructional videos?
Who took my cheese?
Where can I find training courses?
We've got answers!
If you have 30 minutes to spare, come join a free online workshop, led by a real live person! (We don't bite!) We'll show you all of the resources available to you, including online documentation, community forums, learning materials, online training, and even our xMatters Certification options.
North AmericaAll times in New York / Eastern time and observe Day Light Savings
Date & Time
Topic
Register
February 10
10:00 AM - 10:30 AM
xMatters 101: Getting Started
Sign up now!
February 5, 2020 05:01 AM
EMEA
All times in London BST/GMT and observe Day Light Savings
Date & Time
Topic
Register
February 10
3:00 PM - 3:30 PM
xMatters 101: Getting Started
Sign up now!
February 5, 2020 05:01 AM
Asia Pacific
All times in Australia Eastern and observe Day Light Savings
Date & Time
Topic
Register
February 5, 2020 05:01 AM
xMatters 201, 202, 203, 204, 205
xMatters: 201 Overview & Admin Review (2 hours)
Admin Review
xMatters Overview
UI Walkthrough
xMatters: 202 Users (3 hours)
User Profile Overview
User Management / Role Review
User Upload Process
Device Setup including the xMatters Mobile App
xMatters: 203 Groups & Teams (4 hours)
Dynamic teams
Group & Team Management Overview
Group & Dynamic Team Configuration
xMatters: 204 Workflows (Previously Communication Plans) (4 hours)
Best Practice Workflows
Workflow Forms - Layout, Messages & Responses
Conference Bridges
xMatters: 205 - Reporting and Workflow add-ons (2.5 hours)
Reporting
Scenarios
Subscriptions
Communication Center
North America
All times in New York / Eastern time and observe Day Light Savings
Date & Time
Topic
Register
February 17
11:00 AM - 1:00 PM
xMatters 201: Overview & Admin Review
Sign up now!
February 18
11:00 AM - 2:00 PM
xMatters 202: Users
Sign up now!
February 19
11:00 AM - 3:00 PM
xMatters 203: Groups & Teams
Sign up now!
February 20
11:00 AM - 3:00 PM
xMatters 204: Workflows
Sign up now!
February 21
11:00 AM - 1:30 PM
xMatters 205: Reporting and Workflow Add-Ons
Sign up now!
February 5, 2020 05:01 AM
EMEA
All times in London BST/GMT and observe Day Light Savings
Date & Time
Topic
Register
February 5, 2020 05:01 AM
Asia Pacific
All times in Australia Eastern and observe Day Light Savings
Date & Time
Topic
Register
February 5, 2020 05:01 AM
View ArticleContents
Introduction
Configure xMatters
Set up an integration user
Create users and groups that will receive notifications
Import the communication plan
Configure endpoints
Configure inbound integrations
Configure Zendesk
Install the Zendesk app
Add an xMatters user
Populate the xMatters User ID field
Add triggers
Test the integration
Troubleshooting
Add properties
Download resources
The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
Introduction
This article provides installation, configuration, and implementation details when integrating xMatters On-Demand with Zendesk.
Are you using the Zendesk steps in Flow Designer?
Flow Designer - our drag-and-drop, visual workflow builder - includes steps to easily add Zendesk to your incident management workflow. You can find help on using the Zendesk steps in the xMatters On-Demand help.
How it works
Zendesk is a cloud based customer service platform that gives substantial power with little to no coding. Coupled with the power of xMatters alerts, this integration:
Quickly notifies the on-call resource on their preferred devices.
Allows for voice, SMS, and push messages to agents.
Allows agents to reply with "Assign to me" from their device and take ownership of the ticket.
The integration to Zendesk is initiated by Triggers in Zendesk, which decide when to send an event to xMatters. These triggers build a payload and send to an Extension that holds the URL and authentication information for communicating to the xMatters Integration Builder.
Once the payload reaches the Integration Builder, xMatters creates an event and notifies the target group. Or, if the Assignee form was targeted, xMatters makes an API call to the Zendesk Users API to retrieve the xMatters user name and map it to the Assignee's User fields, and then creates the event. In either case, current active events for this ticket ID are terminated.
After the event is created, users are notified:
If a group was targeted, members have the option to respond with "Assign to me", which looks up the xMatters user name and uses it to set the Assignee to the user's Zendesk ID and then ends the event. Recipients can also choose to respond with "Escalate", which immediately notifies the next user in the escalation.
If the event targeted the Assignee form, recipients can respond with "Accept" which ends the event.
All three outbound integration types (event status, delivery status, and response) are active by default and authenticate with the xMatters user set up in Zendesk.
A note about this integration
The xMatters app for Zendesk was built on an earlier version of the Zendesk Application framework. Zendesk has deprecated this version of the framework and removed apps using it from their app marketplace. We're working on updating our app to use the latest version. In the meantime, we've attached a communication plan that you can use to initiate events in xMatters when a Zendesk trigger is initiated.
The Engage with xMatters and Who's On Call features depend on the xMatters app, so they are currently unavailable.
Features and updates
This version of the xMatters On-Demand and Zendesk integration includes a Certified Zendesk App that you can install into your Zendesk instance to help streamline the installation and configuration process. The app includes packaged extensions to use with your Zendesk triggers that will communicate with the xMatters inbound integrations, and a new sidebar application that launches the Engage with xMatters and Who's On Call features.
Engage with xMatters
The Engage with xMatters feature allows users to provide updates, request help, or start a conference bridge with colleagues from other teams to assist with an existing ticket.
For example, a support team member has taken ownership of a ticket, but needs help from the database team to resolve part of the problem. Instead of assigning the ticket to someone else and losing the benefits of ownership, the support person can use the Engage with xMatters feature to contact the primary on-call person in the database group. xMatters continues to update the internal notes of the ticket within Zendesk so the the support team member still receives real-time updates on whether someone else has started work - and who that person is. If there is a group or person that does not exist in Zendesk (but does in xMatters), the support team member can still find them using the Engage with xMatters feature and bring them up to speed directly.
The support team member could also choose to use the Engage with xMatters conference bridge feature to connect with multiple groups through hosted or external conference bridges and work on a solution together.
Who's On Call
The Who's On Call feature displays current xMatters shift information within the Zendesk interface - no need to open a new browser window or login to xMatters.
For example, a support team member wants to place a direct call to someone on the database team to confirm a solution. Instead of looking up the team in a company directory or calling a data center and hoping someone picks up, the support person can use the Who's On Call feature to see who is the right primary on-call person in the database group to contact.
Add an xMatters user in Zendesk
Before you get started, you need to configure an user in Zendesk for the integration. You'll use this later when configuring the Zendesk endpoint in xMatters. You can use an existing administrator account in Zendesk, or create a specific user for xMatters to use when authenticating into Zendesk to update tickets.
To create a user in Zendesk:
In the Manage menu, click People.
Click add user, and then type a name and email address in the Name and Email fields. Because this user will be displayed in Zendesk as the authenticating user, you may want to identify the user as specific to xMatters; for example, use "xMatters-Zendesk" as the name.
The email address must not be currently in use and must be valid. A link for resetting the password will be sent to this address.
In the Role drop-down list, select Agent, and then click Save.
Configure xMatters
The first step in setting up your integration is to configure xMatters.
Download the integration package
To begin, download the communication plan attached to this article; it contains everything needed for this integration.
Set up an integration user
This integration requires a user who can authenticate REST web service calls when working with events these permissions are provided by the "REST Web Service User" role in xMatters. See Create an integration user for more information.
For this integration, you also need to add the Group Supervisor and Person Supervisor roles to this user.
Note: Make sure you keep the user ID and password of this user handy. You'll need them when configuring other parts of this integration.
Create users and groups to receive notifications
The integration notifies the Assignment Group defined in the ticket unless an Assignee is designated, in which case the Assignee is notified. The group names in Zendesk must match the group names in xMatters; however, if the user name in Zendesk does not match the user name in xMatters, you can configure an "xMatters User Name" user field as described below.
To create a new group, see Manage groups.
To create a new user, see Manage users.
Import the communication plan
The next step is to import the communication plan into xMatters.
To import the communication plan:
In the target xMatters system, click Import Plan from the Developer tab.
Click Choose File, and then locate the downloaded communication plan (.zip file).
Click Import Plan.
Click the Edit drop-down list for the plan, and select Access Permissions.
Add the integration user, and then click Save Changes.
In the Edit drop-down list, select Forms.
For the Incident Alert - Notify Assignee form, in the Web Service drop-down list, click Sender Permissions.
Enter the integration user, and then click Save Changes.
Repeat steps 6-8 for the remaining forms in the plan.
Configure endpoints
After you have imported the communication plan, you need to set the authentication for the xMatters and Zendesk endpoints.
From the communication plan, click the Integration Builder tab, and then click Edit Endpoints.
Add a new "Zendesk" endpoint.
Enter your Zendesk instance in the Base URL field.
Set Authentication Type to Basic, and then enter the credentials (email address and password) of the user you created in Zendesk for this integration.
Click Save Changes.
You can delete the existing Zendesk endpoint.
Determine the integration URLs
Each integration has its own URL that you need to copy so you can target it from Zendesk.
To get the integration URL:
Since these URLs are only used to configure the xMatters app, you can skip to the next section.
In the communication plan, on the Forms tab, click the the Web Service drop-down list beside a form, and then select Access Web Service URL.
Copy the highlighted URL from the dialog box:
Configure inbound integrations
You need to update the authentication for each inbound integrations, then retrieve the URLs to configure Zendesk. (The following steps assume that you are using the URL Authentication option. For more information about these options, see Inbound integration service authentication.)
To configure an inbound integration and retrieve its URL:
In the Integration Builder, expand the list of inbound integrations.
Click the name of the integration to view its details.
Scroll down to How to trigger the integration at the bottom of the page, select URL Authentication then select the integration user as the authenticating user. The URL trigger is updated to reflect the new user.
To be able to select the integration user, you need to be a supervisor of that user and the user needs to be assigned to the REST Web Services role.
Click Copy beside the Trigger field:
Configure Zendesk
Now that you've configured xMatters to integrate with your system, it's time to configure Zendesk to integrate with xMatters. The following sections require you to log into Zendesk with an administrator account and access the Admin settings page.
Install the Zendesk App
Note: The xMatters app is currently not available for download from the Zendesk app marketplace. We're working on updating the app and getting it back in the marketplace. For the interim, skip ahead to the next section.
Log in to Zendesk as an administrator, and open the Admin menu (click the gear icon on the lower left of the screen).
Under Apps, click Marketplace
Search the marketplace for "xMatters"; in the results list, click the xMatters badge.
Click Install App.
Configure the settings on the Installation screen as shown in the table below (you can adjust these settings once the app is installed).
Click Install to save your settings.
ZenDesk setting
xMatters setting
xMatters Base URL
The URL of your xMatters instance; do not include the https:// portion.
Example: mycompany.abc.xmatters.com
Engage with xMatters - Provide Update URL
The web service URL of the "Engage with xMatters - Provide Update" form.
Example:
https://mycompany.abc.xmatters.com/api/integration/1/functions/ed0369d3-d7b4-433b-8736-94da4a0fa33d/triggers
Engage with xMatters - Request Assistance URL
The web service URL of the "Engage with xMatters - Request Assistance" form.
Engage with xMatters - Conference Bridge URL
The web service URL of the "Engage with xMatters - Conference Bridge" form.
xMatters Incident - Notify Assignee URL
The integration URL of the "Incident - Notify Asignee" inbound integration
xMatters Incident - Notify Group URL
The integration URL of the "Incident - Notify Group" inbound integration
xMatters - Terminate Events Endpoint URL
The integration URL of the "Terminate Events Endpoint" inbound integration
xMatters Inbound Integration Username
The username used to authenticate inbound integrations (the username of the integration user ).
xMatters Inbound Integration Password
The password used to authenticate inbound integrations (the password of the integration user).
xMatters User Basic Authentication Credentials
The username and password of the xMatters REST user in Zendesk, in the format username:password and Base64-encoded. You can create this value by following the instructions below:
Go to https://www.base64encode.org/.
In the “Encode to Base64 format” area, type the username and password of the xMatters REST user account, separated by a colon in the format username:password, for example, restUser:UDQw9awK
Click Encode, and copy the result.
Example: emVuZGXzayp7ZW5kZXnRmTE=
Note: After the app is installed, you can update or modify settings using the Manage option in the Apps menu. When updating settings, you may need to disable and then re-enable the integration for the packaged Zendesk targets to use your new settings.
Populate the xMatters User ID field
In cases where your Zendesk usernames (email addresses) do not match usernames in xMatters, you can update the new "xMatters User ID" field to store the corresponding xMatters username for your Zendesk users. xMatters User ID is a custom user field in Zendesk.
When an event targets the Incident - Notify Assignee integration, a call is made back to Zendesk to retrieve the xMatters User ID. Additionally, when a user responds with "Assign to me" to a notification from xMatters, the response callback integration service will look up the Zendesk user's ID based on the xMatters User ID field.
For details on how to populate the xMatters User ID with values, see the Zendesk Support site.
Add triggers
Zendesk uses triggers to hold the rules for when to initiate the web service calls detailed in the extensions. These are logical conditions that build a payload and will vary from help desk to help desk.
Note: When developing your triggers, be careful to ensure the alerts are being fired at the relevant times.
To begin creating triggers, click the Triggers option under the Business Rules menu on the Admin page.
The following examples illustrate the settings for some potential triggers.
Trigger One
Title: Send Incident to Group via xMatters
Meet all of the following conditions:
Ticket: is Updated
Ticket: Priority is Urgent
Other: Current user is not xmatters
Ticket: Group changed
Ticket: Status is not Solved
Perform these actions:
Notifications: Notify target - xMatters Group
JSON Body:
{
"properties": {
"Description": "{{ticket.description}}",
"ID": "{{ticket.id}}",
"Status": "{{ticket.status}}",
"Title": "{{ticket.title}}",
"Assignee": "{{ticket.assignee.name}}",
"Assignee ID": "{{ticket.assignee.id}}",
"Group Name": "{{ticket.group.name}}",
"Priority": "{{ticket.priority}}"
}
}
NOTE: The structure of this JSON body is very important. It is used by the Integration Builder to pass through the properties. You can add additional fields, but you must also add them to the layout of the relevant communication plan form. This payload adheres to the POST /trigger payload.
Trigger Two
Title: Terminate xMatters Events for this Incident
Meet all of the following conditions:
Ticket: is Updated
Ticket: Status Greater than Pending
Perform these actions:
Notifications: Notify target - xMatters Terminate Event
JSON Body:
{
"properties": {
"ID": "{{ticket.id}}"
}
}
Note: The xMatters User Name field for the assignee can be referenced as
"Assignee xM User Name": "{{ticket.assignee.custom_fields.xmatters_user_name}}"
Test the integration
To test the integration, create or update a ticket that will satisfy the trigger rules defined above.
Engage with xMatters
Note: The Engage with xMatters feature relies on the xMatters app, which is currently not available for download from the Zendesk app marketplace. We're working on updating the app and getting it back in the marketplace.
You can test the Engage with xMatters feature using an existing incident.
To engage with xMatters:
With an incident open in Zendesk, click Launch xMatters.
In the Engage with xMatters dialog box, choose the engagement type (Request Assistance or Provide Update), use the Recipients area to add the people or groups you want to notify and add the Subject and Message you would like to send.
Click Send to send your message.
Who's On Call
Note: The Who's On Call feature relies on the xMatters app, which is currently not available for download from the Zendesk app marketplace. We're working on updating the app and getting it back in the marketplace.
To test the Who's On Call feature, open an existing incident, and then click Launch xMatters. In the dialog box, click Who's On Call:
To view the contact details for a particular user or group, click the Details link beside their name.
Troubleshooting
There are several places you can inspect when troubleshooting why an event is not sent to xMatters.
Inbound to xMatters
First, in Zendesk, click the Show all events tab on a ticket that was expected to create an event. This will display an audit of all the actions that happened on each update. In particular, it will show what triggers fired.
When expanded, the entry shows the Assignee was removed and the "Send Incident to Assignee via xMatters - Update" trigger fired:
The next place to look is the Integration Builder Activity Stream. While on the Integration Builder tab, expand the Inbound Integrations section, click the gear icon beside the intended integration service, and then click Activity Stream.
The Activity Stream contains the incoming (and for outbound integrations, the outgoing) request, any logging statements as well as the final event creation messages.
Outbound from xMatters
For the two-way integration from xMatters to Zendesk, the primary source of logging is the Integration Builder Activity Stream for that particular integration.
For example, troubleshooting why a ticket was not assigned to a user after they replied with "Assign to me", check the Activity Steam for the Response Callback outbound integration.
Add properties
Occasionally, business use cases will require additional properties to be included in the end notifications. Since the "properties" JSON structure is created in the trigger, the Integration Builder is used as a pass through and no additional work needs to be done there.
To add additional properties (key/value pairs) follow these steps:
In the communication plan in xMatters, create the property with a sensible name.
We recommend that you create these as text properties. Otherwise, care needs to be taken in ensuring the trigger generates the correct JSON format.
Add the new property to the layout of the appropriate form.
There is no limit to the number of properties on a layout so it won't hurt to add it to all forms.
Add the entry to the trigger JSON body.
Refer to the Zendesk trigger docs for details on how to refer to specific fields in the ticket.
For example, if the new property was titled "VIP User" and the field was ticket.custom_fields.vip_user, then you would add a new line to the "properties" element:
"VIP User": "{{ticket.custom_fields.vip_user}}",
Download resources
View ArticleThe information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
Introduction
We're hard at work on our next quarterly release, Hogan's Alley, which will be rolling out in the Jan/Feb 2020 time frame. Please follow this article and its comments to receive updates throughout the quarter about new features and functionality that we're working on for the Hogan's Alley release.
Customers interested in previewing new features before they're released can opt into our Early Access Program (EAP), which delivers new features into non-production environments as they're rolled off the assembly line.
We'll also use this document to capture highlights of additions and updates to our mobile platforms and the xMatters REST API.
Deployment Details
For more information on how a feature in this article works, click the online help links embedded in each topic. As with any of our Community articles, you can click any image or animated GIF to see a larger version.
Support Notes
For a list of additional changes made in each deployment that are not covered in the Development Highlights, see the On-Demand support notes included below each highlight. While most features will be added to production environments as part of our quarterly releases, the support notes typically describe fixes, which can sometimes affect product behavior.
January 27-31, 2020
xMatters REST API: GET /people updates
Availability: Now in the xMatters REST API
We've just equipped the GET /people endpoint with the following new query parameters that you can use to find users who meet particular criteria:
status - users that are inactive or active. For example, GET /people?status=ACTIVE
supervisors.exists - users that have a supervisor, or those that don't. For example, GET /people?supervisors.exists=TRUE
devices.exists - users that have devices in their profile, or not. For example: GET /people?devices.exists=TRUE
devices.testStatus - users with devices in a particular test status, such as those with untested devices. For example: GET /people?devices.testStatus={UNTESTED}
xMatters REST API: New GET /audits records
Availability: Now in the xMatters REST API
The GET /audits endpoint allows you to access audit records for actions that occur in your system, such as how recipients responded to an event and the comments they made. We've been building out this endpoint over time, and you can now use it to retrieve the following auditType records for your events:
EVENT_CREATED - when an event was created
EVENT_SUSPENDED - when an event was suspended
EVENT_RESUMED - when an event was resumed
EVENT_COMPLETED - when an event was completed
EVENT_TERMINATED - when an event was terminated
Support notes (week of Jan 27-31)
xMatters Internal Reference No.
Summary
COR-21829
Subscriptions - notification delay: Fixed an issue where the option to set a notification delay was still displayed when creating a subscription, even if it was hidden on the subscription form.
COR-23115
Flow Designer - Activity stream (cosmetic fix): Corrected an issue where flow trigger icons were missing from the activity stream.
COR-23116
Flow Designer - custom steps: Fixed an issue that could prevent users from accessing the usage permissions for some custom steps.
COR-23166
xMatters REST API - POST /subscriptions: Corrected an issue that prevented xMatters from saving subscriptions with a large number of subscribers.
COR-32328
(SUP-21113)
Messaging - scheduled messages: Fixed an issue that was preventing the delivery of some scheduled messages.
January 20-24, 2020
xMatters REST API: Get all temporary absences
Availability: Now in the xMatters REST API
You can now use GET /temporary-absences to return the scheduled absences for all users you have access to in xMatters (instead of just a single user). We've also added two new search parameters that you can use to narrow down your results:
absenceType - search for absences that have a replacement user, or for those that don't.
groups - search for temporary absences for a particular group, or groups.
xMatters: An enhanced navigation experience
Availability: Early Access Program
In case you missed our recent announcement, as part of the upcoming Hogan's Alley Quarterly Release we're introducing an enhanced navigation experience:
On-Demand Deployment Process & Early Access
The main benefit of the new navigation design, with its collapsible side menu, is that it frees up a lot of extra screen real estate for you to work with your data in xMatters. With all the new data columns we're adding to the All Events report, we think you're really going to appreciate the extra elbow room!
For the full details, see our announcement of the enhanced navigation experience. Or better yet, if you're enrolled in our Early Access Program, you can experience it first-hand in your non-production environment right now!
Messaging: New recipient selector
Availability: Early Access Program
Another feature from the Hogan's Alley Quarterly Release that's now available in the Early Access Program is a new and improved recipient selector for message and subscription forms. Our new design makes it waaay easier to search for and select multiple recipients at the same time:
Lions, and tigers, and devices - oh my!
Just like our previous recipient selector, you can filter by the type of recipient you want to search for, including All, Users, Groups, and Dynamic Teams:
You'll notice that when you select 'Users' from the available recipients drop-down, you now have a separate option to search for Devices:
Separating users and devices prevents you from unintentionally selecting a user's device - like their home phone - as a recipient, instead of their username (which is what you'd want to do the majority of the time so that xMatters contacts them according to their configured device settings and timeframes).
Improved response counts
As part of our work on the recipient selector, we also improved the usability of specifying response counts. In case you're not aware, you can use response counts to specify the number of recipients that must respond to a message before xMatters stops attempting to notify a group.
With a little rearranging, more descriptive labels, and helpful tooltips, we think you'll now find this feature more straightforward and intuitive to use:
Help Resources: Getting started guide for new users
Are you new to xMatters? Do you wish someone could give you a quick run-through of how to get started so that you're ready to receive and respond to notifications?
Well, you're in luck! We've just released a no-nonsense, easy-to-follow getting started guide for new xMatters users. It'll walk you through the basics of logging into the web application and mobile app, lead you through setting up your devices and checking your on-call schedule, and show you what to expect when getting a notification.
It also includes additional recommendations for more in-depth documentation and resources that are available to help launch you from a newbie to an 'xPert' in no time!
Support notes (week of Jan 20-24)
xMatters Internal Reference No.
Summary
COR-23113
(SUP-21059)
Flow Designer - Webhook step: Fixed an issue that resulted in the step not using the optional endpoint suffix provided in the step configuration.
COR-22975
(SUP-21037)
xMatters REST API - GET /audits: Updated the processing of the GET /audits endpoint to address an issue with requests timing out.
COR-22945
(SUP-21039)
Historical On Call report: Fixed an issue causing an error when trying to export the report.
COR-22918
(SUP-20997)
Events - message subject: Updated the code to properly display URL-encoded special characters in the message subject.
COR-22769
(SUP-20998)
Flow Designer: Fixed an issue where using the browser's back button when in a canvas then clicking a empty flow would cause the steps from the first flow to display.
COR-22760
(SUP-20976)
Workflows - editor permissions: Updated the code for retrieving editor permissions to address a discrepancy between the permissions set and those displayed in the dialog.
COR-22609
COR-22608
COR-22554
Web user interface (cosmetic fixes): Made a number of minor updates to the web user interface to improve usability and readability of text.
COR-22597
xMatters REST API - POST /devices: Improved the handling of the request when it provides a UUID for the new device but that UUID is not valid.
COR-22551
xMatters REST API - plans: Updated the /plans/{planId}/endpoints endpoint to use the same permissions as the root /plans endpoint.
COR-22809
COR-22227
COR-22119
iOS mobile app (cosmetic and usability enhancements): Made a number of minor changes to the mobile app interface to improve usability and consistency.
COR-22056
(SUP-20791)
Integrations - Remedy: Updated the deduplication settings to fix an issue where unsuccessful requests to create events were not being re-sent.
January 13-17, 2020
Hogan's Alley Quarterly Release
Our next quarterly release is coming up fast! Here are the dates you'll want to mark on your calendar:
Non-production environment access: Tuesday, February 18
Production environment access: Tuesday, March 3 (enabled between 10:00-10:30 am Pacific)
Keep an eye out for the complete Hogan's Alley feature overview, coming on Tuesday, February 18!
Reporting: Historical on-call report
Availability: Now in xMatters
The Historical On Call report announced in the Galaga Release is now available! Located on the Groups tab, the report makes it easy for supervisors to download information for on-call users in their organization and use it for compensation and planning purposes.
Use the 'From' and 'To' fields to export up to 100 days of historical data to a spreadsheet (.xls) file. The date selectors allow you to go as far back as data availability and your pricing plan allow (we started populating the report with data on January 1, 2020).
Helpful resources:
Interpret the Historical On Call report - An overview of the data included in the report and how to interpret it.
Making the most of your Historical On Call report data - How to use pivot tables to analyze and extract the on-call data that's most important to you.
Mobile: Android 2.22
Android users, you'll want to head over to the the Google Play store to get the latest version of the xMatters app, which includes all of these new goodies:
See in the dark - The app now supports Night mode when it's enabled on your device.
Give your thumbs a break - You don't have to type .xmatters.com when you log into the app.
More accurate group search - The app now returns groups that matchall ofyour search criteria.
Android 10 Support - Get this version of the app to keep things running smoothly on Android’s latest OS.
Reporting: Export All Events report
Availability: Now in xMatters
The All Events report now includes an Export option that you can use to export either 'All Columns', or just the 'Current Columns' you've selected to include in the on-screen report:
Export like a pro
To save yourself extra time formatting the exported spreadsheet, you can first filter, sort, and arrange the report in the web user interface. Then, when you're ready to export, xMatters preserves your formatting in the exported file!
If you selected the option to export all columns in the report, xMatters exports the current results as you've arranged them in the web user interface, and then appends any additional columns in alphabetical heading order to the right of the last column of current results.
You'll notice two tabs on the exported file:
Export Details - a summary of the report, including the date range, initiator of the export, and search filters applied to the report.
Events - the list of events for the specified timeframe.
Integrations: xMatters Actionable Alerts for Splunk updates
Availability: Now from splunkbase
A new version of xMatters Actionable Alerts for Splunk is now available for Cloud and Enterprise Splunk customers. If you're updating to the latest and greatest version of Splunk (v7.2-8.0), then you'll want this update because it's compatible with Python 3.
Support notes (week of Jan 13-17)
xMatters Internal Reference No.
Summary
COR-22690
COR-22749
Workflows tab - xMatters Agents (cosmetic fix):Updated the styling and layout of the xMatters Agents page in the web user interface.
COR-222595
Flow Designer canvas (usability fix): Fixed an issue where creating a webhook integration did not properly display the mini-map in the Flow Designer canvas.
COR-22752
(SUP-20989)
Communication Center - Recent Events widget: Corrected an issue where special character encoding in email subject lines was not being converted to proper punctuation on the dashboard.
COR-22426
(SUP-20603)
xMatters Agent: Corrected an issue where the xMatters Agent would sporadically process the same job twice.
COR-22638
(SUP-21005)
Admin tab: Fixed an issue where attempting to access the Admin tab would cause an error.
COR-22406
Workflows - scenarios: Corrected an issue that allowed scenarios to be saved without a required name.
COR-22396
Communication Center - saving dashboards: Fixed an issue that prevented users from saving changes to the dashboard name and description.
COR-22653
(SUP-20961)
All Events report: Fixed an issue that was causing the All Events report to display different results than the Recent Events report.
COR-20349
Android mobile app (cosmetic fix): Corrected an issue that caused the hamburger menu icons to distort when viewed on Google Pixel 2 and 3 devices.
COR-22165
COR-22164
iOS mobile app - My Schedule (usability fix): Fixed an issue that displayed a blank schedule screen when reloading data.
January 6-10, 2020
Messaging: 'Alert Context' property
Availability: Now in xMatters
As a message recipient, it's often useful to have more context about why you're receiving a notification. By knowing if you were targeted directly, because of your membership in a group, or from one of your subscriptions, you'll have better insight into your role and responsibilities in the incident resolution process.
To help with this, we've added a new "Alert Context" property to our message editors that you can include in your messages to show recipients how they were notified for an event:
Can I get more context, please?
When the message is sent, xMatters replaces the Alert Context property with the path used to notify you, as shown in the the examples below:
Directly targeted: "Direct Notification"
As part of a group: "DBAs > P.M. Shift"
As part of sub-group: "Campus D > Default > DBAs > P.M. Shift" etc.
Subscription notifications: "Product Recalls Subscription"
Group subscription notifications: "Product Recalls Subscription > DBAs > P.M. Shift"
As shown below, even though Mary McBride is a member of several different groups in her organization (like IT, First Responders, and Party Planning) she knows she's been invited to the conference bridge for her IT expertise in developing mobile apps - not to help with this year's Holiday party:
All Events report: 'Source' column
Availability: Now in xMatters
Need more context about your past events? The All Events report now includes a Source column that displays the name of the built-in or custom workflow used to create the event:
xMatters REST API: Get people by role
Availability: Now in the xMatters REST API
Ever wished you could programmatically look up the users in your system with a particular role (or set of roles) in xMatters? Like the Incident Managers? Or users that are Incident Managers andalso have the Developer role?
Well, the great news is that you can now use GET /people to do just that! When you query this endpoint, specify which role or roles you're interested in by their name or ID. For example:
GET /people?roles={Incident Manager}
GET /people?roles={Incident Manager,Developer}
xMatters REST API: Details for read-only access forms
Availability: Now in the xMatters REST API
Like we implemented last week for plans (AKA workflows), now when you use the xMatters REST API to request information about a form that you don't have access to, instead of getting a 403 Forbidden error you'll receive basic details about the form including its UUID, name, description, as well as the ID and name of the plan (AKA workflow) it belongs to.
Support notes (week of Jan 6-10)
xMatters Internal Reference No.
Summary
COR-22424
(SUP-20932)
Trials/Free - expiry warnings: Fixed an issue where Free customers were receiving incorrect email warnings about their instance expiring, and some Trial users were seeing a banner at the top of the page indicating that their trial had expired.
COR-20193
xMatters REST API - GET /people: Fixed an issue where attempting to embed devices when using the GET /people endpoint to retrieve a large number of records would result in a 504 error.
COR-22100
xMatters iOS app - Reports tab: Updated the app to prevent the Reports tab from being incorrectly relabeled when navigating in the interface.
COR-22125
xMatters iOS app - My Devices: Fixed an issue where navigating to a device timeline while the My Devices screen was still loading would cause the app to show a "No Devices" message.
COR-22169
xMatters iOS app - Tracking Report: Updated the behavior when terminating an event on the Tracking Report to immediately update the progress bar to show the termination.
COR-22556
Groups - unscheduled shifts: Fixed an issue where attempting to edit a recently unscheduled shift would cause an error.
COR-22103
Communication Center (cosmetic fix): Updated the layout and appearance for some of the items and widgets displayed in Communication Center dashboards.
COR-22284
(SUP-20788)
xMatters REST API - POST /subscriptions: Updated the POST /subscriptions endpoint to prevent users from inadvertently creating subscriptions with empty property values.
COR-21477
Flow Designer - email initiation: Fixed an issue where some users were encountering a 404 error when attempting to initiate an email trigger.
December 30, 2019 - January 3, 2020
xMatters REST API: Get targeted recipients
Availability: Now in the xMatters REST API
If you're using the GET /events endpoint to return a list of events, you now have the option to embed targeted recipients. For example: GET /events?embed=targetedRecipients
To retrieve events that targeted a specific user, you can explicitly search for just the events targeting that user by identifying them by target name or UUID. For example:
GET /events?targetedRecipients=mmcbride
GET /events?targetedRecipients=c56730a9-1435-4ae2-8c7e-b2539e635ac6
xMatters REST API: Embed text property translations
Availability: Now in the xMatters REST API
Do you have text properties configured in more than one language? If you're interested in looking up the values assigned to text properties for different languages as you're working with scenarios, you now have the option to embed translations:
GET /scenarios/{scenarioId}?embed=properties,plan,form,properties.translations
GET /plans/{planId}/forms/{formId}/scenarios?embed=properties.translations
When you include these optional embeds in your request to GET /scenarios or GET /plans/{planId}/forms/{formId}/scenarios, the xMatters REST API returns the translated text for each property with a two-letter code for the corresponding language:
"myTextProperty": {
"propertyType": "TEXT",
"value": "This is urgent. Please respond.",
"translations": {
"en": "This is urgent. Please respond.",
"fr": "C'est urgent. S'il vous plat, rpondez."
}
},
xMatters REST API: Details for read-only access plans
Availability: Now in the xMatters REST API
Even if you don't have edit permissions to all the plans (AKA workflows) in your xMatters instance, it's still useful to know that they exist and to be able to see basic information about them. Now when you use GET /plans or GET /plans/{planId} for plans that you don't have edit permissions for, instead of getting a 403 Forbidden error you'll instead receive details about each plan including its UUID, name, description, and whether it's enabled.
Support notes (week of Dec 30-Jan 3)
xMatters Internal Reference No.
Summary
COR-22054
Users - profile: Updated what's displayed the web user interface when a user without permission to see user profiles clicks on a user in the UI.
COR-22252
xMatters REST API: Updated a subset of endpoints that were not using the complete targetName when referencing a device.
COR-22071
Messaging - preview: Fixed an issue where certain characters in a message subject prevented the subject from displaying in the preview.
COR-21977
(SUP-20727)
EPIC upload - custom properties: Fixed an issue where the EPIC upload process was not properly handling updates to users custom field and custom attribute properties.
COR-22331
(SUP-20909)
Workflows - integrated properties: Fixed an issue where some integrated properties were not appearing in event reports.
COR-22141
(SUP-20886)
Groups - shifts: Fixed an issue where the proper time zone was not applied when changing a one-time shift into a recurring shift.
COR-22245
Groups - shifts (cosmetic fix): Improved the styles on the web user interface to prevent the shift name field from overlapping the description.
COR-22151
Groups - shifts: Updated the message displayed when trying to create a shift with the same name as an existing one.
COR-22174
(SUP-20861)
xMatters Android app: Updated the app to fix an issue where some umlaut () characters were not being presented correctly.
COR-21963
Free/Trial - Invite Users (usability enhancement): Updated the phone number validation to allow users to continue after only changing the country code.
COR-22102
Users - devices (cosmetic fix): Updated the Devices screen to prevent a specific tooltip from staying open when you move the mouse away.
COR-22250
(SUP-20904)
xMatters Free: Fixed an issue that caused users to see an error after logging in.
COR-22247
All Events report: Updated the response highlights section to display the 'time to first response' information correctly.
December 23-27, 2019
xMatters REST API: Updated Python code samples
Availability: Now in the xMatters REST API
We're pleased to announce that we've just finished updating all the Python code samples in our xMatters REST API docs to be compliant with Python 3.6! Just in time, too, because they'll be sunsetting Python 2 on January 1, 2020.
Support notes (week of Dec 23-27)
xMatters Internal Reference No.
Summary
COR-22207
(SUP-20908)
User Upload: Corrected an issue where an intentionally blank field caused the uploaded file to fail.
COR-21874
Temporary Absences (usability fix): Fixed an issue where clearing the "All Groups" check box did not remove the check marks from the listed groups.
December 16-20, 2019
Flow Designer: Merge flow paths
Availability: Now in xMatters
It's now possible to connect multiple paths of a flow to the input port of a single step. This is useful when different segments of your flow follow the same series of steps. By merging them into a common path, you reduce duplicated configuration and simplify the layout of your canvas:
In the example above, the flow uses a switch step to split the flow based on a customer_impacting event property. A value of "Yes" creates a Statuspage incident and then posts to a NOC dashboard, while any other value just posts straight to the dashboard. Joining both flow paths directly to the same 'Post to NOC' step removes a duplicate step from the canvas and saves configuration time!
Workflows: Send Alerts
Availability: Now in xMatters
A new Send Alerts workflow is now available from our collection of Workflow Templates. We've designed this pre-built workflow template to help you start notifying users and groups with minimal setup. It lets you create an event and send notifications by simply sending an email or HTTP request to xMatters.
You can use this workflow to quickly send test messages, learn how workflows work, and as a starting point for creating your own workflows. To get up to speed quickly, see our Send Alerts workflow guide.
Mobile: iOS 3.31
We've just released a new version of the xMatters iOS app, which includes a TON of great new updates! Here's an overview:
Duplicate events - When you manage events from the app, you can now copy an existing event, modify some of its properties, and resend it as a new event.
Reply to push messages - Long press or pull down on a push message from your lock screen or the notification center to reply or to join a conference bridge.
Edit shifts when you’re on the go - Tap on a shift in My Schedule to change that shift occurrence’s duration or its members.
On-call schedule at-a-glance - A fresh new look for the on-call widget makes it even easier to tell when you’re next on call for a shift.
Star or Send with ease - We’ve improved the placement of the Star and Send controls so fingers of all sizes can reliably select one without tapping the other.
Support notes (week of Dec 16-20)
xMatters Internal Reference No.
Summary
COR-21778
Workflows (usability fix): Fixed an issue where attempting to delete a workflow or flood control rule in Internet Explorer 11 would cause the page to stop responding.
COR-21836
Flow Designer - custom steps: Fixed an issue where new custom steps would be incorrectly flagged as in use on a canvas.
COR-18532
(SUP-20066)
Web user interface (usability fix): Updated the web user interface to improve messaging, response times, and prompts when a back-end change requires the user to refresh the page.
COR-21838
(SUP-20285)
Admin tab - Company Details: Fixed an issue where the Admin tab would show two Company Details menu options for some permission and role combinations.
COR-21033
Flow Designer - HTTP Triggers (usability fix): Fixed an issue where the hover-over tooltip for HTTP triggers in the Flow Designer palette would sometime appear in a way that caused them to be truncated by the edge of the screen.
COR-21850
COR-21777
xMatters REST API - /groups: Updated the xMatters REST API to prevent groups from being accidentally created without any supervisors, and to prevent false naming collisions when creating groups with similar but non-identical names.
COR-19384
xMatters iOS app (usability fix): Improved the layout and spacing of the Send and Favorite (Star) buttons to prevent inadvertent tapping of the wrong one.
COR-21362
xMatters iOS app - sending messages: Fixed an issue where the app would prompt to confirm changes to a messaging form before sending when no changes had been made.
COR-21571
xMatters iOS app - terminating events: Fixed an issue where the app would display the option to suspend or terminate already terminated events.
COR-21555
xMatters iOS app (usability fix): Fixed an issue where users were not being directed to the correct event when tapping on a push message.
COR-21831
(SUP-20847)
User Upload - updating users: Updated user target names to be case-insensitive.
COR-21827
All Event - Response Statistics (cosmetic fix): Updated the layout of the Response Statistics section on the All Events report so that long response options won't overlap the next item in the list.
COR-21931
(SUP-20837)
Flow Designer (performance enhancement): Improved the responsiveness and performance of Flow Designer when a workflow contains a very large number of forms.
COR-21794
(SUP-20834)
User profiles - Roles tab: Updated the styling for the Roles tab to address a minor cosmetic issue when adding or editing roles for a user.
COR-20971
Flow Designer - custom steps: Fixed an issue where the Endpoints section would still appear on the Script tab of a custom step even when an endpoint was not defined for that step.
COR-21768
Workflows - importing: Fixed an issue where importing an exported workflow that included a "Send a webhook" outbound integration would cause an error.
COR-21764
(SUP-20836)
Messaging tab: Fixed an issue where the Messaging tab would not display any content.
COR-21699
(SUP-20822)
Flow Designer - email initiation: Fixed an issue where the canvas would not load in Flow Designer if the workflow included email form initiation.
COR-21849
(SUP-20840)
xMatters mobile apps: Fixed an issue that was preventing the xMatters mobile apps from retrieving and displaying data from some instances.
COR-20582
xMatters REST API - POST /groups: Updated the POST /groups endpoint to automatically delete any holiday shifts from a group if the site associated with the group is removed. (This mimics the behavior in the web user interface.)
December 9-13, 2019
All Events report: New columns
Availability: Now in xMatters
We've just added two new columns to the All Events report. You can use Targeted Recipients and Form to get more context about the users, devices, groups, or dynamic teams targeted to receive notifications for each of your events, and the form that was used to initiate them:
We've also included these new columns in the handy Columns selector that lets you choose which data to display in the report:
Help Resources: Be the best group supervisor ever!
Have you just been assigned the Group Supervisor role in xMatters? Are you wondering what that means and what kinds of things you'll be able to do?
Well, we've gotgreat news: we just released an awesome new resource to help you out! Our guide on How to be the best group supervisor ever gives you the run-down on the Group Supervisor role’s capabilities, as well as links to handy resources that will help introduce you to working with groups, shifts, escalations, rosters, and everything else you need to know.
It also includes some first-hand tips and tricks contributed from our advisors and consultants in the field that you won't find anywhere else - so be sure to check it out!
Support notes (week of Dec 9-13)
xMatters Internal Reference No.
Summary
COR-21311
COR-21312
Flow Designer canvas - switch step: Implemented usability fixes that make moving switch steps on the canvas easier, and that place the cursor in the proper position when editing text fields.
COR-21365
Users - Schedule (usability fix): Corrected an issue where the "Filter by" drop down menu is still visible after navigating away from a user's schedule.
COR-21729
Shifts - copying a shift: Fixed an issue where focus did not switch to the copied shift as expected.
COR-20965
xMatters Online Documentation:Corrected the documentation so all links point to appropriate help pages.
December 2-6, 2019
Flow Designer: Palette styling improvements
Availability: Now in xMatters
You'll notice that we've made some styling updates to the Flow Designer palette to tighten things up and make it easier on the eyes. The changes are subtle, but we think you'll enjoy working with this more refined version:
Support notes (week of Dec 2-6)
xMatters Internal Reference No.
Summary
COR-21231
COR-21263
User Upload - error messages (usability fix): Improved some of the error messages displayed in the User Upload report to be more helpful when a column header is missing, or when all rows in a file are invalid.
COR-21478
(SUP-20690)
xMatters REST API - GET /groups/{groupId}/shifts: Corrected an issue that prevented users from editing a shift when multiple shifts have the same name.
COR-21193
All Events report (usability fix): Corrected an issue that displayed incorrect search result counts.
COR-21190
(SUP-20693)
Integrations - Activity Stream logging:Corrected an issue where the Activity Stream was not properly logging outbound integrations.
COR-21502
(SUP-20788)
xMatters REST API - POST /subscriptions: Corrected an issue where empty values prevented the subscription from operating properly.
COR-20743
xMatters Agent - permissions: Fixed an issue so users without the proper permissions could no longer make changes to the list of Status Alert Recipients. The changes these users made prior to the fix were not saved.
COR-21274
(SUP-20723)
Groups - incorrect shift members displayed: Fixed an issue where a past shift incorrectly shows the current shift's members.
COR-21356
(SUP-20766)
Web user interface (usability fix): Corrected an issue where users saw a blank screen when xMatters was upgrading the web user interface.
COR-21425
Reporting - access and permissions:Fixed an issue where users with appropriate permissions were prevented from viewing events in the Recent Events report.
COR-21525
(SUP-20825)
Groups - shifts: Fixed an issue where the proper timezone is not applied when changing a one-time shift into a recurring shift.
COR-21333
(SUP-20774)
Workflows - deleting a workflow: Corrected an issue where a subscription form was still available after the workflow was deleted.
COR-21305
(SUP-20676)
xMatters REST API (usability fix): Fixed an issue that caused requests to timeout.
November 25-29, 2019
xMatters REST API: Get user delivery data
Availability: Now in the xMatters REST API
We've added a new endpoint to the xMatters REST API that lets you look up information about who was notified for an event. Some of our customers like to use this data to create reports in other systems such as Splunk or Tableau. Here's the new endpoint:
GET /events/{eventID}/user-deliveries?at={timestamp} - returns detailed information on who was notified for a specific event, the notification delivery status, the date and time of the notification, and which devices were contacted.
Since this endpoint queries historical data, you'll need to include the at parameter in your request. You can also optionally embed the person.properties query parameter to include the values of the custom fields and attributes assigned to each person (for example:
GET /events/{eventID}/user-deliveries?at={timestamp}?embed=person.properties).
All Events report: View events for the groups you supervise
Availability: Now in On-Demand
We've added a new permission that allows you to view events for the groups you supervise. This makes it easier for Group Supervisors to stay informed and take action on any events targeting the groups that they supervise. For more information about this permission, and to add it to your company's Group Supervisor role, contact xMatters Support.
Support notes (week of Nov 25-29)
xMatters Internal Reference No.
Summary
COR-21024
(SUP-20714)
Subscriptions: Fixed an issue where subscription owners were automatically being added to subscriptions without any other recipients.
COR-21253
(SUP-20741)
Integration Builder: Fixed an issue where users with a specific form configuration found that events were not created and flows not executed when a request was sent to an integration associated with the form.
COR-21032
Workflows (usability fix): Updated the validation of workflow names to check for all disallowed characters.
COR-21075
(SUP-20668)
xMatters REST API - GET /groups: Updated embed=supervisors to correctly process the limit parameter (if included in the query).
COR-21205
(SUP-20751)
xMatters Agent - installation: Fixed an issue where the agent installation information was not being displayed if a user selected manual proxy configuration.
COR-21118
(SUP-20730)
Integration Builder: Fixed an issue where initiating a particular integration using the xMatters REST API would not result in any notifications.
COR-21011
(SUP-20679)
All Events report: Fixed an issue where the selected date range did not persist after logging out and logging back in, even though data displayed in the report used the correct range.
COR-21160
(SUP-20745)
Subscriptions - subscribers: Fixed an issue where an incomplete list of subscribers was returned when opening a subscription, and was then saved when the subscription was saved.
COR-21070
(SUP-20623)
Groups - shifts: Fixed an issue where editing a shift would change the shift's time zone to the group's time zone.
November 18-22, 2019
xMatters REST API: Get, create, and modify scenarios
Availability: Now in the xMatters REST API
Scenarios allow you to save multiple versions of a form with predefined values for common or anticipated situations, making it easier to send relevant messages quickly when those situations arise.You can now use the following xMatters REST API endpoints to look up information about scenarios, and to create and modify them:
GET /scenarios/{scenarioId} - return a specific scenario, based on its unique identifier.
GET /plans/{planId}/forms/{formId}/scenarios - return a list of scenarios for a form in a workflow.
POST /forms/{formId}/scenarios - create or modify scenarios for a form.
Flow Designer: Update Jira status
Availability: Now in On-Demand
Flow Designer's built-in steps to update issues in Jira Cloud or Jira Server now include the option to change the status of the issue. This means that as an incident progresses, you can keep its status up-to-date in Jira (for example, you could change its status from "To Do" to "In Progress" to "Resolved").
Flow Designer: ServiceNow endpoint updates
Availability: Now in On-Demand
It's now easier to include ServiceNow in your incident management workflows! You're no longer required to have the xMatters app installed in your ServiceNow instance to connect your steps to the ServiceNow endpoint, which makes it a lot more convenient to get ServiceNow steps up-and-running in Flow Designer.
Support notes (week of Nov 18-22)
xMatters Internal Reference No.
Summary
COR-20898
(SUP-20677)
Messaging tab - creating messages:Corrected an issue that prevented messages from using their multi-select lists of properties.
COR-20922
(SUP-206520)
User uploads - voice devices: Corrected an issue where Slovakian phone numbers were not properly formatted.
COR-20905
Flow Designer - custom step images: Corrected the message when loading an icon that is too large to properly state the maximum image size is 250KB.
COR-20925
(SUP-20636)
SSO access: Corrected an issue where users saw only a blank screen after logging on using Single Sign-On.
COR-20744
(SUP-20626)
Messaging - text missing from subject line:Fixed an issue where text surrounded by angle brackets was removed from the subject line of the outgoing email.
COR-20882
(SUP-20675)
Groups - editing recurring shifts: Corrected an issue where the Edit Shift button would not work when the shift had a slash in its name.
COR-20831
(SUP-20654)
EPIC data sync processing: Corrected an issue where EPIC was blocked by the length of time taken to remove orphan events.
COR-20943
(SUP-20685)
xMatters REST API - POST /subscriptions: Corrected an issue where the timezone id prevents users from creating subscriptions.
COR-19850
(SUP-20429)
Flow Designer - logging:Fixed an issue where comments submitted with user responses were not displayed in the event log.
November 11-15, 2019
Mobile: Android 2.21
A new version of the xMatters Android app is now available! Head on over to the Google Play store to take advantage of the following updates:
My Schedule calendar view
There's a new calendar view for My Schedule that you can use to quickly and easily look up which days you're on call - just click the calendar icon at the top of the My Schedule screen:
The calendar displays a green dot on the days where you're scheduled to be on call, and you can tap on any day to see your on-call shifts. As you scroll your list of shifts, the monthly calendar view collapses to a weekly view to give more screen real-estate to your shift information:
You can use the calendar to view your on-call schedule for up to the next 90 days. A handy "Today" shortcut makes it easy to quickly return to today's on-call schedule.You can also use the shift filter to choose which on-call shifts to display - where you're primary on-call, primary or secondary on-call, or all of your shifts.
Available update notices
Take advantage of the latest app updates as soon as they're released! The xMatters Android app now lets you know when a new version of the app becomes available.
To get the latest and greatest updates, just tap to download and to relaunch the app. If it's not a convenient time, tap 'Remind me later' and we'll notify you again the next time you open the app.
Support notes (week of November 11-15)
xMatters Internal Reference No.
Summary
COR-20556
EPIC data synchronization - Synchronization report: Fixed an issue that could prevent the Synchronization report from loading properly for a company with an ampersand (&) in its name.
COR-20751
All Events report (usability fix): Updated the All Events report to properly display some special characters in the Message Subject column and on the Message tab.
COR-20844
Flow Designer - run locations: Fixed an issue that was preventing the Run Location tab from appearing on some custom steps.
COR-20553
(SUP-20604)
Groups - permissions: Fixed an issue that was preventing some Group Supervisors from being able to view and edit shift details for their groups.
COR-20830
(SUP-20668)
xMatters REST API - GET /group: Fixed an issue where the /groups endpoint would return an error when attempting to retrieve a list of groups with embedded supervisors.
COR-20806
(SUP-20653)
All Events report - responses: Fixed an issue that was preventing some responses from being displayed when viewing event details on the All Events report.
COR-20565
(SUP-20603)
xMatters Agent - duplicate events: Fixed an issue where the xMatters Agent would sometimes create multiple events if the connection was disrupted during initial processing.
About the early access delivery process
Customers can choose to opt their non-production environments into anEarly Access Program (EAP)to see new functionality before it's released. If you're considering signing up for the early access program, there are a couple of important things to know:
Enrollment inEAPcan be turned on any time through a support request.
Exiting EAPcan only be done on quarterly boundaries (you cannot opt in and out between quarterly releases).
For full details, refer to the officialarticle.
View ArticleThe information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
xMatters is proud to introduce a game-changer for incident management: Flow Designer!A stunningly simple, visual workflow builder that’s as easy as drag, drop, and done.
Modern IT teams need to manage the complexity of synchronizing data, tools, and people at the speed of customer demand. From minor issues to the largest of customer-facing incidents, problems need to be addressed efficiently. With Flow Designer, any authorized user can automate resolution with the simplicity of an elegant drag-and-drop experience.
Join us on a tour through the key features in Flow Designer and see the latest innovation from xMatters one that promises to revolutionize the way you integrate, synchronize, and automate toolchains.
First, a little history
The last major addition to our integration platform, the Integration Builder, introduced the ability for integrators to create automated, closed-loop, cloud-to-cloud integrations between xMatters and other products or systems. It allowed integrations to be created, customized, and deployed from within the xMatters user interface creating one-stop shopping for building your communication processes and toolchains.
Next we added the xMatters Agent, which allowed users to process Integration Builder scripts behind their company’s firewall, letting them trigger events in xMatters with enriched information from local network resources.
Then we made it easier for clients to install integrations with our Integration Directory, a new mechanism for pre-configuring many of our most popular integrations to help get events into xMatters.
So, why Flow Designer?
The greatest power of xMatters is its ability to chain your tools together. Sure, being able to reply to a push message and automatically assign yourself a help desk ticket is pretty cool, but we've long been proponents of having that response do so much more.
To truly drive incident resolution and reach the pinnacle of digital service availability, you need to consider the next steps: what happens after someone receives an initial event? What do they do during the first triage steps? How do they perform their incident management activities?
We've always known that xMatters can streamline, enhance, and automate those processes. Clients with the necessary resources and expertise have been able to harness the power of theIntegration Builder to design and implement toolchains linking their processes and applications.
But, it hasn't been something we've seen across our client base. The Integration Builder requires a certain level of coding acumen the integrations rely on scripts to process requests, and scripts often need to be modified or tweaked to meet different customer requirements. We want to make sure that it's easy to design and create these workflows and that any client can do it (including our xMatters Free customers).
Flow Designer offers:
Codeless design with built-in steps for the most requested apps to flatten the learning curve for integrated toolchains.
A visual designer for at-a-glance analysis of workflows that lets you focus on incident response orchestration.
Enhancement of your existing communication plans to extend your current process.
A toolkit for building custom steps to support any business process.
With Flow Designer, simply drag the steps you need to create a toolchain into place, connect them together, and let it rip!
Let’s see it in action!
Less talk, more action, right? With Flow Designer it's all about action!
community
Flow Designer has three main components (highlighted above):
Canvas - a large central area to build your flows.
Palette - a sidebar that holds steps you can use in your flows.
Activity - a panel you can expand to check on the runtime state of your flows.
To build a toolchain, simply drag the system activity that will start your flow (the "trigger") and the steps for applications that you're targeting from the palette to the canvas (click images to embigify):
Connect your steps in the order you want information to flow:
Specify the information that's passed between each step and any authentication details required to communicate with your other systems:
And, you don't have to set each and every input: any information in the flow is available to all of your downstream steps, including any of the outputs from steps in your flow. Don't need the detailed event summary when creating the chat channel? No problem just grab it when you're ready to post it to the ticket. Want to name your chat channel with the new incident ticket you just created? Take the incident ID from your "Create Incident" step and feed it into your "Create Channel" step.
Finally, test your flow! With Flow Designer's visual representation of your toolchain and the Activity panel, it's fast and easy to identify where and how to fix configuration issues with your flow:
In the example above, we created a toolchain to help resolve a major incident by turning a user's response into a ServiceNow ticket, then creating a Slack channel and posting relevant details about the event to the channel to help our incident resolution team. We did that in just a matter of minutes and without a single line of code!
Let's get fancy
If, then... say what? You don't need to know how to code conditional statements to create complex workflows that branch based on incident details. Want to post a notice to a status page, but only if the incident is customer-facing? Flow Designer includes a built-in Switch stepthat you can use to easily branch your flow based on the value of a property:
What about my existing integrations?
The introduction of Flow Designer doesn't affect the functionality of your existing integrations, and you'll find them all in the same place as they were before. You'll also now see your outbound integrations as triggers on the canvas when you open Flow Designer for one of your existing communication plans:
What's really cool about this is that you can then use Flow Designer to extend these integrations just connect additional steps to your outbound triggers:
Something for everyone, novice to ninja
The initial release of Flow Designer, with its codeless drag-and-drop functionality, makes it easy for anyone to set up a toolchain that performs common actions in the most-requested incident management applications. And this is just the beginning:over time we'll be adding more apps, more tools, and more ways for you to enhance your toolchains with ease.
If you've got some technical chops and want to use Flow Designer to take your toolchains to the next level right now, you certainly can. We've included the ability for you to create your own Custom steps, where you can use your coding abilities to connect with any tool or perform any action that you can dream up.
Custom steps are easy to build simply define the inputs and outputs for your step and the logic you'd like to run. Once you're done, anybody can use your step with the same drag-and-drop ease as built-in steps:
Availability
Alright, alright! Now for the answer to the most important question when can you get your hands on Flow Designer?
Come to the launch!
We'll be officially launching Flow Designer at Google Cloud Next in San Francisco on April 9. If you're at the conference or in the area, we'd love for you to come by and say hi!
Here are the dates you need to know:
Non-production environment access: April 2
Production environment access: April 9 - and we're live!
Flow Designer is now available in all production environments!
We're excited to see what you make with it! Join us in the and share your experience.
View Articlepricing plan
We've heard loud and clear that you want more access to more of your xMatters data to support root cause analyses, address compliance requirements, and other important processes. With the Centipede release in October 2018, we launched our new data storage capabilities, and we're now able to begin giving you access to the data that you need!
Historical data access
Our new capabilities allow us to greatly extend customer data retention, providing you with improved visibility into your xMatters data via better reports and access to historical on-call, event, and notification history.
It's the start of the xMatters Time Machine which gives new point-in-time behaviors for viewing historical data, such as:
What email address was that dude in Ops using again?
What was that group doing all weekend and who was in it?
Who was on call that time we had that very bad day where the terrible thing happened?
We started populating your historical database beginning with the Centipede release (October 2018), so for now the new All Eventsreport will display a limited set of your event data. As you create more data and add it to your database, you'll also see additional features surfaced into this new report.
Starting with the Defender release (February 2019), we'll enhance the xMatters REST API to access historical data, beginning with on-call schedule information, so stay tuned for more information there.
Data licensing
The new feature willretain all of your data, but limits visibility to that data based on your so you only pay for the data you need.
The licensing breakdown is as follows:
Free: 90 days
Starter: 365 days
Base: 365 days
Advanced: Unlimited cosmic power!
You can see that even the Free plan is way better than the old default settings which limited most environments to 30 days of runtime data and 60 days of archived.
And the even better news is that if you decide to upgrade your pricing plan to a higher level, you'll instantly gain access to all that delicious, life-affirming data. So if you upgrade from Free to the Base plan, for example, you'll immediately be able to look at your historical data for the previous year instead of just the past 90 days.
Where are these data limits applied?
The data licensing limits are only applicable to the new historical data capabilities. This means that existing features, reporting, and APIs will continue to be based on your existing database.
Also, note that we had to actually build the system first, and couldn't start populating it with data until October 2018. Anything that was purged from your archive before then will not be available in your historical data.
View ArticleQuestion
Is it possible to add/remove/change the device types seen in xMatter? For example, can I add more e-mail devices? Can I rename Office Phone to Office Mobile?
Environment
All versions of xMatters On-Demand
Answer
By opening a Support ticket, our Customer Support team can assist in making these kinds of modifications (except on Free and Trial environments). There are a few rules however:
1) We cannot rename a device type. We can add one or remove one and it can be named most anything reasonable.
2) Before we can remove a device type, all users that have a device associated to them of that type must have that device removed from their profile. The most efficient way to look at this would be to do a user export from the Users tab, sort the data by device type and see how many exist or search the CSV for the device type in question. This can be updated to remove the devices from useres then use the User Import task to update the environment. Note - if users are brought in via a user sync this would be a situation where the devices need to be removed by the user sync then removed from any configurations to set them.
3) The types of devices we can create are still limited by the main types: SMS, Phone, e-mail, pager
View ArticleQuestion
When using the API in an integration or just as a manual CURL request can you affect how many records return at a time? The default seems to suggest 100 records.
Answer
Yes you can! Most endpoints support the limit parameter so by specifying the number of records to return per page you can get significantly more data.
For example if you take the event audit endpoint, the default looks like this:
"https://acmeco.xmatters.com/api/xm/1/audits?at=2018-11-02T08:00:00.000Z&from=2018-01-27T08:00:00.000Z&to=2018-06-30T08:00:00.000Z
If we add the Limit parameter it would look like this:
"https://acmeco.xmatters.com/api/xm/1/audits?at=2018-11-02T08:00:00.000Z&from=2018-01-27T08:00:00.000Z&to=2018-06-30T08:00:00.000Z&limit=500
This should increase the amount of records fetched.
For more information about our API see this page
Overview
Put in additional context, add in other keywords, phrases or variants of terms to increase findability
View ArticleThis KBA is intended to help client assistance workers resolve SSL*-related problems with minimal delays and complication. The focus of this article is on troubleshooting SSL connections between the integration agent and a management system using information and techniques garnered from past successes, but with some interpretation the information can be applied in other situations.
1. Overview
This document will make extensive use of the terms "client" and "server". From here on, a server is a device that is connected to the internet and that is capable of responding to an HTTPS request, and a clientis a device that initiates the HTTPS request.
For example, your web browser is a client that allows you to send requests to the website of a bank or an online store. Similarly, an Integration Agent acts as a client when it sends notification event data to an xMatters hosted instance. The Integration Agent can also act as a server if it is configured to receive notification data from a management system such as BMC Remedy. Note that in the second example, the client is actually transmitting data, not requesting it.
So the critical distinction between a client and a server is that the client initiates the communication with the server by making a request - even if the purpose of the request is to send data to the server.
If the purpose of the request is to start an online banking session, then the client must protect the user's data by ensuring that the response is actually coming from the bank and not an imposter, and by encrypting the user's data so that it can't be stolen by a third party. Encryption and Identity verification are performed through the use of an SSL certificate, which is received from the client as part of the response from the server.
So: an SSL certificate consists of a software component that is installed on a server, and that allows clients to verify the server's identity and encrypt data that is exchanged between the server and the client.
Plenty of information is available pertaining to setting up a certificate on a server, but relatively little thatapplies to clients. Here's a quick overview of why a client (eg, the IA) needs SSL certificates when interacting with a server (eg, a management system such as Remedy.)
When a client (such as the IA) sends an HTTPS request to a server, the client expects to receive information aboutthe server's SSL certificates in the response.
Unless the IA has a way of authenticating the server's certificate, it will nottrust the server.
Therefore, the IA has a file called a "trust store", which contains all of the certificates that it knows can be trusted. This includes certificates from commercial vendors such as Verisign. The trust store is included with the Java Runtime Environment that ships with the IA.
The IA compares the certificate from the server with the certificates in its trust store to see if there is a match. If there is, then the IA continues the dialog with the server and receives thedata that it initially requested.
If you're new to the world of SSL certificates, there's a great introduction to the topic here ; scroll down to the section headed "How Does the SSL Certificate Create aSecure Connection?"
1.1Commercial vs. Self-Signed Certificates
Commercial SSL certificates (purchased from a recognized vendor such as Digicert, Verisign, etc) allow any client to verify that the information it is receiving is trustworthy, in the sense that the information is being provided by the proper server. Because these commercial certificates can be algorithmically linked to a trusted certificate vendor, and no client configuration is required, they are essential to businesses providing online banking, credit card payments, and exchange of confidential information.
However, there is a disadvantage to these commercial certificates: they cost money.
Many of the advantages of a commercial certificate can be obtained through use of the self-generated (aka "self-signed") certificates. These certificates provide most of the benefits of a commercial certificate and they are free, but there is a disadvantage: a client cannot trust a self-signed certificate unless a copy of it is manually added to the client's trust store. This is obviously unappealing in an online banking situation, but might be perfectly practical when transferring data securely between a management system and an IA.
Adding a self-signed certificate to the IA's trust store is oneof the topics of this KBA.
1.2 Certificate Chains
When the owner of a server buys an SSL certificate from a vendor such as Verisign, they will receive several certificate files. There will be a copy of the vendor's certificate, a certificate that is unique and specific to the purchaser, and usually an "intermediate" certificate. The vendor's certificate does not have to be directly trusted by the client (eg, the browser that you are using to do your online banking). The concept of certificate chaining allows the client to trust a certificate even if it is not in the client's trust store.
The client's Trust Store will contain a relatively small number of "Root" certificates that are considered trustworthy by the provider of the user's browser, JRE, or operating system.
A certificate chain is a set of certificates that links a server's certificate to one of the "root" certificates that the client trusts. When examining the certificate data received from a server, a client will first test the server's certificate to see if it matches one of the certs in the client's trust store. If it doesn't, then the client will examine the first intermediate certificate in the data. This intermediate certificate is digitally linked to the server's own certificate, so the client tests the link between these two to ensure that it is valid. The client verifies that each intermediate certificate in the chain is authentically linked to the next one. When it gets to the end of the chain, it expects to find a certificate that matches one of the Root certificates in its own trust store. Assuming that is the case, then the server's certificate is considered trustworthy and the client continues the exchange.
Certificates on the Server
Typically, when an SSL certificate is purchased (eg, from Verisign,) the purchaser will receive a unique certificate that contains data about the server and the organization that owns it. The purchaser will generally receive one or more "intermediate" certificates and a Root certificate. The purchaser will install the root, intermediate, and server certificates in the keystore file on the server.
Once the certificates are properly installed on the server, the server is able to accept requests starting "https://". When an HTTPS request comes in (eg, from a browser or an Integration Agent), the server will provide aresponse that includes the server certificateand the others described above.
If the certificates are not installed correctly in the server's keystore, then the requester will see a warning that the chain is not valid, and the site is not to be trusted. This will also be the case if the server certificate has expired or has been revoked.
A good way to test the validity of the certificate chain is to use a browser to access the server (eg, https://www.xmatters.com). You can use the browser to display the "certification path" if desired, but simply browsing to the URL and looking at the browser's green padlock icon will tell you whether the certificate is valid.
https://maulwuff.de/research/ssl-debugging.html
In the Google Chrome browser's "Certification Path" screen you can view the certificate chain. At the top is the issuer's Root certificate, and at the bottom is the certificate belonging to the server. Each certificate is verifiably linked to the one above it in the chain.
2. Adding a Certificate to the IA's Trust Store
For SSL communication to work, the JRE may need to possess a local copy of the server's public certificate. This will be necessary if
the server's certificate is self-signed, or
there is a problem with the server's commercial certificate.
The client'scopy is kept in a "trust store", which by default is a file called cacerts in the <IAHOME>\jre\lib\security folder.
To add a certificate to the JRE's trust store, make a backup copy of the cacerts file, then open a command window in the Integration Agent's jre\bin folder and run the following command:
keytool -importcert -keystore ../lib/security/cacerts -storepass changeit -file /temp/somecert.cer -alias somecert
<IAHOME>/jre/lib/security/cacerts is the JRE's default trust store. You can use a different trust store, but that will require specifying the path in the IA's wrapper.conf. It is very unlikely that you'll need to do this.
"changeit" is the default password for any JRE's trust store
-file is followed by the path to the certificate that you will be adding to the trust store. The certificate should be in either DER (binary) format, or else X.509 Base-64 encoded text format. Certificates exported from Chrome, for example, will be DER by default. JRE 8 will also trust PKCS12-formatted certificates in "compatibility" mode.
-alias allows you to specify an alias for the certificate, which makes it more convenient to list and manipulate the certificates in the store.
You can view certificates in the store with the keytool -list command. To see only a single certificate, use the -alias argument. For more detail, use -v:
keytool -list -keystore../lib/security/cacerts -storepass changeit -alias somecert -v
Where to get a copy of the certificate? The client will probably be able to provide one, but you can also export a copy from a browser. Simply browse to the server's URL and then use the browser's certificate export tool. Browser-specific instructions are available via Google search.
3. Configuring Integrations For SSL
Integrations that are designed to allow SSL communication with the management system may include configurable parameters forthe trust store's location and filename, password and so on. Refer to the integration document for an example.
This capability is provided by XMIO.js (used by newer integrations.) Older integrations may instead use wsutil.js, which is part of the integration agent utilities package. If this isn't made clear by the integration document, search the integration's script files for references to XMIO.js or wsutil.js, and follow the path to determine which library is providing the functionality. XMIO.js uses the JRE's global settings, including the cacerts trust store mentioned above. Wsutil.js uses a different trust store. See the comments in wsutil.js for details.
4. Diagnosing SSL Problems
Diagnosing SSL connection failures is difficult for several reasons:
The information that is logged by the Java libraries tends to be vague and not very helpful
SSL connections can fail for a variety of reasons, some of which are difficult to identify
the server's certificate may have expired, been revoked, may not be correctly installed, may notbe compatible with the TLSv1.2 protocol, or its "Valid from" date may be in the future.
SSL communication is not well understood by many IT workers
In this chapter we will describe some troubleshooting techniques and some tools that we have found to be useful.
If the terminology in this chapter is unfamiliar, read the Background Informationchapter near the end of this document.
4.1Messages in the IA Log
The following message in an IA log indicates that you have encountered an SSL certificate problem:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
The "certification path" in the message is a reference to the certificate chain mentioned earlier on. The "requested target" is theorganization name in the certificate, which must match the URL to which the request was sent.
4.2Ensure that the Trust Store has a Valid Certificate for theServer
If the server's SSL certificate is self-signed, then a copy of it must be installed in the IA's trust store. Ensure that the certificate has notexpired, has not been revoked, and has the correct information in its Certificate Name ("CN=") field. The Certificate Name information must match the URL to which the IA is sending its requests. Note that in Java 8, it is necessary for the certificate to include a Server Alternate Name (SAN) that matches theURL to which the IA is sending its request.
Even if the server certificate is not self-signed, it might need to be installed in the IA's trust store. The certificate might not be properly installed on the server. You can use a browser to perform a quick check - if the browser doesn't display a green padlock icon, then the server's certificate will not be accepted by the IA.
Even if the browser shows the certificate to be valid, it is still possiblethat the IA's JRE willnot trust it. For example, there was a period during which GoDaddy's root certificate was not included in the Java trust store, and therefore certs issued by GoDaddywere not recognized by Java applications. Keep in mind that the JRE and the browser were produced by different development teams, and that the JRE and the browser do not use the same trust store. If in doubt, install a copy of the server certificate in the IA's trust store (<IAHOME>\lib\security\cacerts by default).
If the certificate is self-signed, then you can import itinto the browser's trust store. A Google search will get you directions specific to your browser (IE, Firefox, Chrome, etc) and operating system. The purpose of doing this is to verify whether the problem lies in the certificate itself, or in the way the certificate has been imported into the IA's trust store. If the certificate is not trusted by the browser even after being imported, then it will not work with the IA either.
4.3 Inspect The Certificate with Keytool
After importing a certificate, you can inspect it to determine whether it has obvious flaws (and to verify that it was imported correctly.) To inspect a certificate inthe IA's trust store, use a utility such as keytool.exe. Keytool is included with the Integration Agent and can be found in <IAHome>\jre\bin. In this way you will be able to determine who issued the certificate and who it was issued to, as well as the certificate's expiry and issuance dates.
Considerations when using keytool:
Use the keytool executable that was shippedwith the integration agent. This is important because the JRE may not be able to read certificates that were added with a different version ofkeytool.
to inspect the certificate, use the following keytool command:
keytool -list -v -keystore <storeName> -alias <theAlias>
where
<storeName> is the path to the trust store (eg, ..\lib\security\cacerts) and
<theAlias> is the alias that was used when the certificate was added. (If you don't know the alias, omit the "-alias" parameter from the command.)
You will probably want to direct the output to a text file so you can examine it with an editor, since the keystore may contain dozens of certificates.
4.4 What to look for when inspecting a certificate
The first few lines will contain most of the information that you need. For example, looking at the xMatters.com certificate:
C:\xmatters\integrationagent-5.1.8\jre\bin>keytool -list -v -storepass changeit -keystore /temp/cacerts -alias xmatters
Alias name: xmatters
Creation date: Feb 27, 2017
Entry type: trustedCertEntry
Owner: CN=*.xmatters.com, O="xMatters, inc.", L=San Ramon, ST=CA, C=US
Issuer: CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US
Valid from: Thu Feb 18 16:00:00 PST 2016 until: Fri Feb 22 04:00:00 PST 2019
Also pay attention to the "SubjectAlternativeName" section later on in the report:
SubjectAlternativeName [
DNSName: *.xmatters.com
DNSName: *.hosted.xmatters.com
DNSName: *.api.xmatters.com
DNSName: *.na1.xmatters.com
DNSName: *.na2.xmatters.com
DNSName: *.na3.xmatters.com
DNSName: *.na4.xmatters.com
DNSName: *.na5.xmatters.com
DNSName: *.cs1.xmatters.com
DNSName: *.eu1.xmatters.com
DNSName: xmatters.com
]
This information tells us that the certificate is valid for any URL that ends with ".xmatters.com", as well as "hosted.xmatters.com" and so on. It also tells when the certificate will expire, and who issued it.
The connection will not succeed if the URL is " https://35.201.109.29 ", unless the SubjectAlternativeName field includes an entry "ip:35.201.109.29" -even though www.xmatters.com resolves to this address. Nor will the connection succeed if the JRE cannot resolve the name www.xmatters.com (if this is the case, add an entry in the OS's hosts file,) or if the JRE cannot reach the address because of interference from a firewall or proxy server.
Note that this is a so-called "wildcard" certificate, which can be installed on a number of servers. More typically, a certificate will be valid for only one server.
Note that the server's IP address must be present in thea Subject Alternative Name (SAN) field, if the https request uses the server's IP address. If the IP address is present in the Certificate Name (CN) field but not the SAN field, then the connection will fail with "java.security.cert.CertificateException: No subject alternative names present".
If the"Valid Until" date has expired, the connection will not succeed. Similarly, if there is a "Revocation Time:", the certificate will not be accepted. Certificate issuers occasionally revoke certificates if there is a problem with them - for example, if a vulnerability such as heartbleed is discovered. In such cases, the server administrator should be able to get a replacement certificate from the issuer.
4.5 Testing the Connection
4.5.1 Using Ssltest
If the certificate in the trust store is valid, and the integration is configured to use the correct trust store, then the connection should work. If it appears not to be working, try ssltest will let you test the connection using the IA's JRE, but having to use the IA. This is desirable when the IA is being used in production, and also lets you avoid the clutter of the IA's console and/or logs.
To do this:
use ssltest.class, which is attached to this KBA:
download ssltest.class and copy it to <IAHOME>\jre\bin (to ensure that the test uses the same JRE as the integration agent)
open a command window in<IAHOME>\jre\bin
invoke ssltest with the following command:
java ssltest <url>
for example,
java ssltest https://www.xmatters.com
if the certificate is in a different trust store, use the following syntax to specify the trust store and password:
java -Djavax.net.ssl.trustStore=</path/to/cacerts> -Djavax.net.ssl.trustStorePassword=<password> ssltest <url>
for example,
java -Djavax.net.ssl.trustStore=..\lib\security\cacerts -Djavax.net.ssl.trustStorePassword=changeit ssltest https://www.xmatters.com
you will likely find an appropriate URL in the integration's Javascript files. It is likely that the integration is using the URL to connect to the management system's web services. If this is the case, use the WSDL to test the connection. Ask the management system administrator for the URL that provides access to the WSDL.
The expected output is:
C:\xMatters\integrationagent-5.1.8.1\jre\bin>java ssltest https://www.xmatters.com
[INFO] Received host address https://www.xmatters.com
[INFO] Setting connection timeout to 5 second(s).
[INFO] Trying to connect to https://www.xmatters.com
[INFO] Great! It worked.
If the certificate has expired, the result will be
[INFO] Could not connect to the host address https://expired.badssl.com
[INFO] The error is: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed.
If the certificate specifies a key that is not secure enough to comply with TLSv1.2, the result will be
[INFO] Could not connect to the host address https://dh512.badssl.com
[INFO] The error is: DHPublicKey does not comply to algorithm constraints
[INFO] Here are the details:
[SEVERE] DHPublicKey does not comply to algorithm constraints.
There is a variety of bad certificate types accessible at bad-ssl.com. By testing your troubleshooting tool against these certificates, you can reproduce the error that you are experiencing and thereby identify the cause of the error you are diagnosing.
4.5.2 Using Analyze-ssl.pl
This is another excellent tool for SSL connection diagnosis. Unfortunately, it has a couple of disadvantages:
it requires a perl interpreter, which does not come pre-installed on most servers. Some clients might balk at installing additional software on their IA server.
it isn't compatible with the older version of perl that installs by default on CentOS (presently 5.16.3)
Analyze-ssl.plis compatible with the current version of ActiveState perl (5.24.1), which is available for Windows, OS X and Linux, though. You can download ActiveState perl fromhttps://www.perl.org/get.html
If you need to diagnose an SSL connection and have access to a suitable version of perl, then this tool is well worth the trouble.
You can download the script fromhttps://github.com/noxxi/p5-ssl-tools/blob/master/analyze-ssl.pl or you can use the copy that is attached to this document.
Syntax: perl analyze-ssl.pl <with no arguments> - display the help screeen
perl analyze-ssl.pl(options) <address> -analyze the SSL/TLS connection to the address
Example:perl d:\tools\analyze-ssl.pl --show-chain xmsupport.hosted.xmatters.com
Output:
-- xmsupport.hosted.xmatters.com port 443
* maximum SSL version : TLSv1_2 (SSLv23)
* supported SSL versions with handshake used and preferred cipher(s):
* handshake protocols ciphers
* SSLv23 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_2 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_1 TLSv1_1 ECDHE-RSA-AES128-SHA
* TLSv1 TLSv1 ECDHE-RSA-AES128-SHA
* cipher order by : server
* SNI supported : ok
* certificate verified : ok
* chain on 65.151.10.10
* [0/0] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/ST=CA/L=San Ramon/O=xMatters, inc./CN=*.hosted.xmatters.com SAN=DNS:*.hosted.xmatters.com,DNS:hosted.xmatters.com
* [1/1] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
* [-/2] bits=2048, ocsp_uri=, /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
* OCSP stapling : no stapled response
* OCSP status : good
This option, which analyzes the certificate chain, is one of the most useful features of the analyze-ssl tool (particularly because this functionality doesn't seem available at all in the Java world.) There are other options, plus a verbose mode. See the help mode for details, by invoking analyze-ssl.pl with no arguments.
If the client won't let you install this tool on the IA server, it works just as well from a laptop. Ideally, you will use a wired connection to the same switch as the IA is using, so that you won't get false results due to proxy servers, etc.
Keep in mind that this tool uses a different certificate store and different algorithms from the IA's JRE. So, as with a browser, there is a chance that you will get different results from this tool than you get from the IA or ssltest.
4.5.3 Using ssllabs.com
If the server is internet-accessible, then you can paste its URL into www.ssllabs.com to get a comprehensive report on its SSL certificate. Despite the excellent information available from this site, there are a couple of deficiencies:
It only works with publicly-accessible servers, so if you are trying to diagnose a certificate belonging to a server on a LAN or a proxy server, then ssllabs will probably not be able to help you.
As with alalyze-ssl.pl and the browser, ssllabs.com uses a different certificate store and different algorithms from the IA's JRE. So there is a chance that you will get different results from this tool than you get from the ssltest or from the IA itself.
4.5.4 Using curl
curl is available for Linux and Windows, and it is a great tool for testing SSL certificates. It provides detailed information about the certificate, and you can provide arguments to specify the TLS (or SSL) level, the proxy parameters (if applicable), and a great variety of other parameters. User "curl -h" to see them. The help is terse, but more details and examples are available online.
The basic command is
curl -v https://www.xmatters.com
Note that syntax is different for Windows than for Linux.
You can redirect the actual page content to a file, so that your console shows only the connection data. For example,
curl -v -o c:\temp\deleteme.txt https://www.xmatters.com
Disadvantages to using curl are:
It isn't a Java program, so it can provide false positives. In other words, just because a connection succeeds with curl, it won't necessarily succeed for the IA.
It isn't installed by default on Windows servers (or on some Linux systems,) and so a customer may not allow it to be installed on their server.
However, curl provides great diagnostic details that are not available from Java (and therefore they aren't available from the IA itself.) So if you are looking for detailed troubleshooting information, curl provides some of the best that you will find.
Tools such as curl (or a browser) are also useful in that they can demonstrate to a customer that the IA is not the source of connection problems. If you can demonstrate to the customer that connections fail with curl as well with the IA, then the IA itself is less likely to be blamed for the problem.
Note that recent versions of curl for Windows do not seem to display certificate information. The message to watch for is
schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
This was encountered with curl for Windows 7.5.5, so use an earlier version instead. curl 7.4.0 is recommended.
5 Advanced Diagnostic Procedures
If the above tools fail to help youresolve the problem, then there are still a few things you can do.
5.1 Re-checking the Basics:
ensure that the certificate is valid (and hasn't expired)
ensure that you are testing with a URL that matches the Certificate Name shown in the trust store
ensure that the integration agent server can resolve the management server's network name, exactly as it appears in the Certificate Name
ensure that there is no interference from a firewall or a proxy server between the integration agent server and the management system
5.2 Using Ssltest in Debug Mode
If you pass the debug=all flag to the JRE while invoking ssltool, the output will be verbose but potentially very helpful in diagnosing the connection failure. By comparing the tool's debug-level output with the attached file, you will be able to determine exactly which step failed in the SSL exchange.
Open the attached ssltest.debug.log file with a text editor to see examples of the following information from a successful exchange:
truststore name and path
list of certificates found in the truststore (abbreviated in this example)
SSL handshake (initial "write" from client and "read" response from server "vic-vw-remedy8.invoqsystems.com")
analysis of the certificate chain sent by the server
"Found trusted certificate" message indicating successful match of the server certificate with one found in the truststore
request for data, using the secure connection
decrypted information received from the server
"close" message sent by client
client closes the socket connection
Compare the information in this file with the output from your own session.
Use the following syntax to run ssltest in debug mode:
<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest <URL>
To simplify inspection of the output, redirect it to a text file. For example:
<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest https://www.xmatters.com/> \%temp\%\ssltest.txt
Hint: when opening the output file in a text editor, search for "***". This string is used as a marker to indicate milestones in the establishment of an SSL connection.
Here are some notes on interpreting the verbose output:
The messages do not discriminate between TLS v. 1.0, TLS v. 1.1 and TLS v. 1.2, so don't spend time investigating the possibility that TLS 1.0 is being used instead of TLS 1.2. Assuming that you are using IA 5.1.8 (which ships with JRE 8) or later, TLS 1.2 is used by default.
Don't be deceived by the messages saying "adding as trusted cert:". Those messages indicate that the JRE is reading its trust store file and making a list of candidate certificates to be potentially matched against the ones submitted by the server. The messages don't indicate that aserver's certificate will automatically be trusted just because it hasthe same "CN=" name.
The "*** Certificate chain" messages are worth paying attention to. Typically the certificate chain will be an array of one or more certificates which have been submitted by the server, and the client will attempt to match those certificates to the ones in its trust store. If the chain has multiple certificates, then the first one (labeled "chain [0]") will be the server's own (eg, "CN=*.xmatters.com") and the rest ("chain [1]", etc) will be a "root" from the certificate authority (the vendor) who provided the certificates.
The "certificate chain" messages should be followed by "***
Found trusted certificate: "
If the "certificate chain" messages are followed instead by messages like these, then there is a problem with the certificates sent by the server:
"\%\% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]"
"TLSv1 ALERT: fatal, description = certificate_unknown "
The problem is likely to be one of:
The "root" certificate from the server is not recognized as a Certificate Authority. The root cert should include a record saying "CA:true", and should match one of the certs in the IA's trust store.
The IA's JRE could not create a valid link from the server certificate to the root certificate. For example, if the certificate chain is intended to provide an intermediate certificate as well as a root and a server certificate (which is often the case,) but the intermediate certificate is absent, then theX509 Trust Manager will not be able to validate the server certificate and the connection will fail. In other words, the certificate was not properly installed on the server. A browser should be able to confirm this diagnosis.
The server's certificate was not added correctly to the IA's trust store. One likely cause is that the certificate was not in a form that the IA's JRE could accept (using keytool - list should tell you if this is the case.)
Another possibility is that the server's entire certificate chain was installed in the IA's trust store. It's a mistake to install the server's intermediate and root certificate in the IA's trust store. If you install the server's certificate alone (whether or not it is self-signed,) the IA will trust it - assuming that the certificate is in a format that is acceptable to the IA's JRE. Again, keystore -list will tell you this.
If these notes don't help you to resolve the problem, then take a look at the attached Oracle "Debugging SSL/TLS Connections" document for further information about the certificate validation and handshaking process.
5.3 Compiling Ssltest on the Server
If the client will not agree to installing the ssltestbinary file on the server, they may agree to allowing the ssltest tool to be compiled from the source code attached to this KBA. Provide the file ssltest.java to the client - it is a very simple piece of code that can be inspected with a text editor. Once it has been inspected, the client can compile it with the following command (assuming that the Java JDK is installed):
javac ssltest.java
This will create a new copy of the ssltest.class file which can be used as documented above.
6. Background Information
All these terms can be so confusing! Troubleshooting is hard enough without a bunch ofjargon getting in the way. Here are explanations of a few of the terms used in this KBA.
SSL("secure sockets layer") is a mechanism that allows HTTP requests and responses to be transmitted securely. HTTP plus SSL is whereHTTPScomes from. SSL is also used in other forms of communication, such as SSH (which allows people to log on to and control remote servers,) and SCP (which allows files to be transferred securely.)
Two major parts of SSL communications areencryptionandverification. Encryption ensures that a third party who intercepts a request or response will not be able to understand the content. Verification ensures that a client can trust the response that they receive from a server, by verifying the responder's identity. That is what SSL certificates are for.
*TLSis a newer version of the Secure Sockets Layer specification. In this KBA I've avoided referring to TLS because the term "SSL" is woven all through the terminology (e.g., no-one uses the term "TLS certificates"), and it wouldn't have added clarity.
Akeystoreis just a file or other storage medium that contains SSLkeys and/or certificates. Thekeys and certificates have to be added via software such as the Javakeytool.
Wait a minute... so far we've been talking about putting "certificates",not "keys", into keystores. So what's this "key" business? Answer: an SSL key (actually, a key pair) is a unique two-part digital document that is requiredto createan SSL certificate (either commercial or self-signed.) The SSL key pair consists of a public key, which is included in the certificate, and a private key, which is kept on the server and is never shared.
For you keeners, there is a good overview of SSL/TLS encryption here: https://www.digicert.com/ssl.htm
Aserver'skeystore holds itsprivate keyand itsSSL certificate. When the client receives a response from the server, it usesthe certificate information in the response to verify the server's authenticity. The client also uses the public key from the certificate to encrypt its next request. The server uses its private key to decrypt the request.
Atrust storeis also a file that contains certificates. The main difference between a keystore and a trust store is that a trust store is used by a client (i.e., the device that initiates a request to a server), whereas a keystore is used by a server. Another difference is that a keystore contains the server's private key and its certificate, whereas a trust store contains only certificates. When the client receives an HTTPSresponse, it compares the certificate information provided by the server with the certificates in its trust store, to see if there is a match. If none of the certificates match, then the response will not be trusted.
Different devices and systems store their trusted certificates in different ways. The Windows OS stores its trusted certificates in the Registry. Some browsers such as Chrome and Internet Explorer use the trusted certificates provided by the OS. Other browsers such as Firefox and Opera keep their own separate trust store, in a file. And the Java JRE also keeps its trusted certificates in a file, as described above.
Since the JRE can operate as a server or a client, it needs a keystore and also a trust store. By default, the JRE's keystore and trust store are the same file.
In the JRE's default trust store ("cacerts"), you will see a number of certificates that were shipped with the JRE. These certificates are in the trust store because they are issued byCertificate Authorities; generally, these are the companies that sell SSL certificates.
If a server sends a certificate that was supplied by a commercial signing authority, but the certificate was not installed correctly on the management system, then it may not be recognized by the JRE. In other words, it will be treated like a self-signed certificate. And, like a self-signed certificate, the IA's JRE will trust an improperly-installed server certificate,ifit is added to theIA's trust store.
7. SSL Inspection Proxy Servers
You may encounter difficulties connecting to xMatters On Demand through an SSL inspection proxy server. A likely cause is that the proxy server removes the certificate to inspect the traffic and then puts a new certificate on the data to deliver it to the Integration Agent. The agent will not accept the dataif the proxy server's certificate is not trusted (eg, if the certificate is self-signed.)
One solution is to switch off SSL inspection on the proxy for that traffic. If that's not possible, then adding the proxy server's certificate to the IA's trust store (as described in section 2) solves the problem.
(Thanks to Adam Watson for contributing this field-tested information.)
8. Further Reading
This is an excellent resource for SSL/TLS troubleshooting:
If you want to understand SSL cryptography better, here's a great starting point:
https://www.digicert.com/ssl-cryptography.htm
xMatters Reference
JDN-4346 Originally created by Jeremy Brown
View ArticleBy default, the integration agent uses the same timezone as the host operating system when writing timestamped messages to its log files. If this default does not meet your needs, you can specify a different time zone.
Tospecify a different timezone:
Download apache-log4j-extras-1.2.17.jar from http://archive.apache.org/dist/logging/log4j/extras/1.2.17/ and put it in <IAHOME>\lib\mule-1.4.3\lib\opt.
In <IAHOME>\conf\log4j.xml replace:
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="\%d{yyyy/MM/dd HH:mm:ss.SSS Z z} [\%t] \%p - \%m\%n"/>
with:
<layout class="org.apache.log4j.EnhancedPatternLayout">
<param name="ConversionPattern" value="\%d{yyyy/MM/dd HH:mm:ss.SSS Z z}{GMT} [\%t] \%p - \%m\%n"/>
Replace {GMT}with the timezone that you want (eg, {GMT+2}).
Save the file and restart the integration agent.
Note that the above instructions will only affect the timestamps in the IA log file. If you use the IA in console mode and want the console messages to use the same timestamps, then make the same change in the "consoleAppender" section of log4j.xml
This configuration has been validated with integration agent 5.2.2 but has not been subjected to rigorous testing, so we recommend that you verify correct operation in a staging environment before installing it on a production system.
xMatters Reference
DOC-4308, COR-3050
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
Issue overview
Many integrations allow the management system to send a request to xMatters to delete pending notifications for a specified incident ID. For example, if an incident is reassigned or its priority is raised, the management system will send a message to the integration agent requesting that pending notifications be cancelled for that incident ID.
Older integrations often use the DEL integration service to request these cancellations. This mechanism is acceptable in most instancesif no new notifications are to be requested; however, if the DEL is followed by an ADD, then new notifications may not be created. This failure occurs if the DEL and the ADD are not sent to xMatters in the correct order, in which case the DEL negates the ADD.
The underlying problem is that messages for different integrations are sent to xMatters via individual outbound queues belonging to the integration agent. For example, the BMC BPPM integration has its own outbound queues, and so does the DEL integration. Messages that are added to a given queue are guaranteed to be sent to xMatters in the order in which they were received, but there is no such guarantee if the messages are in different queues.
For this reason, we recommend that an integration not use the DEL integration service. Instead, notification requests (ADDs) and cancellations (DELs) can be sent to the same integration service, ensuring that they will be delivered to xMatters in the correct order. This is accomplished by specifying the appropriate method (ADD or DEL) in the integration script while composing the APXML request to xMatters.
In the sections below, you'll find the following:
Configure the management system toallow the sending of both ADD and DELmessages to the same integration.
Configure the integration scripts to enable themto distinguish between an ADD and a DEL injection.
Ensure synchronous deletions byadding code that sends a DEL message to xMatters when a matching injection is received from the management system.
While doing the work described below, you may need to refer to the integration agent guide for detailed information about the concepts involved.
Examples
This article uses version 1.1.2 of the BMC BPPM integration throughout as an example of the processes required. Your integration may differ slightly - or significantly - based on the trigger and initiation processes involved.
Configure the management system
Each integration uses its own mechanism to initiate an injection into the integration agent: some use a batch file to call the APClient.bin, while others call it directly. Otherintegrations are capable of injecting directly into the integration agent directly, without using the APClient.bin function.
Example
As in many integrations, the BPPM integration relies upon the management system using a batch script to inject ADDs and DELs into the integration agent. The integration does this by using batch script files to invoke the xMatters-supplied command-line tool APClient.bin.
To configure the management system, append an extra parameter to the xm_send_notification.bat script and set it to "false". The DEL script will be replaced with a copy (essentially) of the ADD script, with the following attributes:
the DEL script has the same number of parameters as the ADD script
the DEL script invokes the same integration service
all of the parameters can be empty, except for the incident ID and final parameter, which will be set to "true"
xm_send_notification.bat
"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm \%1 \%mc_ueid\%
false ...[parameters omitted]... "\%msg\%"
xm_delete_notification.bat
"C:\Program Files\xmatters\Integrationagent\bin\APClient.bin.exe" --map-data del "\%mc_ueid\%"
Note that these two scripts both invoke APClient.bin in the same way, but they provide different integration names after the "--map-data" argument. This indicates that the ADDs are sent to the bmcbppm integration service, and the DELs are sent to the del integration service.
Append an extra parameter to the ADD script (xm_send_notification.bat, in this example) indicating whether the integration should send a DEL to xMatters.
Updated xm_send_notification.bat
"C:\Program Files\xmatters\integrationagent\bin\apclient.bin" --map-data bmcbppm20 \%1 \%mc_ueid\% "false" ... "\%msg\%" "false"
Replace the contents of the "del" batch script so that it has the same number of arguments as the "add" script, and the final parameter is "true":
Updated xm_delete_notification.bat
"C:\Program Files\xmatters\integrationagent\bin\apclient.bin" --map-data bmcbppm \%1 \%mc_ueid\% "" ... "" "true"
Note: In this specific integration, the \%mc_ueid\% parameter contains the incident ID that designates the notifications to be deleted.
Configure the integration scripts
This stage of the conversion is more complex than the previous one, and requires the following steps:
Configure the integration's XML definition file
Provide code to process the "delete_existing" token
Provide code to send DEL requests to xMatters
Configure the integration's XML definition file
The XML file defines aspects of the integration service that are used by the integration agent at startup time. Among these are the parameters that the integration service expects when a message is injected from the management system.
Example
To configure the XML definition file, we add a parameter that matches the one appended in step 1 above. Using bmcbppm.xml as an example, we see a list of parameters that match the ones in the xm_send_notification.batshown above:
parameters in bmcbppm.xml
<parameter type=>recipients</parameter>
<parameter type="string">incident_id</parameter>
...[parameters omitted]...
<parameter type="string">message</parameter>
We will add a parameter to contain the true/false information indicating whether the integration should send a DEL to xMatters:
<parameter type="string">recipients</parameter>
<parameter type="string">incident_id</parameter>
...[parameters omitted]...
<parameter type="string">message</parameter>
<parameter type="string">delete_existing</parameter>
This parameter creates a token called "delete_existing" that will be included in the APXML message created by the integration service. The integration script uses this token to determine whether an ADD or a DEL request is to be sent to xMatters.
Provide code to process the delete_existing token
Next, we will update the integration service script containing the actual instructions for processing injections from the management system. If the integration uses APClient.bin, then the processing instructions will be in a function called apia_input (APXML), which accepts an APXML message from the integration agent's inbound queues and typically returns an APXML message to the integration agent's outbound queues. In its most basic form, this function consists of the single instruction "return apxml;".
If the integration does not use APClient.bin for incident injections, a different function handles the incoming injection. For example, apia_http handles injections sent via direct request to the integration agent's HTTP endpoint.
Example
The BMC BPPM integration uses apia_input, although it uses a variable to send a copy of the APXML request to the outbound queues, rather than sending the original message.
The objective of this step is to add code that will test the delete_existing token in the incoming APXML message. We will create a variable called should_delete to contain the result of this test.
We will test the delete_existing token before any others, because other tokens may be null if this is a DEL injection:
// test whether the "delete_existing" token is true or false in the incoming APXML
var should_delete = apxml.getValue("delete_existing")
var msEpochTimestampSec = apxml.getValue("date_reception");
Provide code to send DEL requests to xMatters
Now that we have a variable to tell us whether the incoming APXML message contains an ADD or a DEL request, we can modify the integration script to prepare and send the appropriate outgoing message.
In the simplest case, we will want to send a request to xMatters only to cancel pending notifications for a specified incident ID.
Example
Because our inbound APXML includes the incident ID, we need only set the setMethod and setSubclasss tokens in the outbound APXML message:
apxml.setMethod("DEL");
apxml.setSubclass("action");
To prevent this code from affecting all injections (including ADDs) we will need to put a conditional block around the existing code. The result will be similar to the following:
Revised apia_input to detect delete_existing flag and send del request to xMatters
function apia_input(apxml) {
var result = null;
// test whether the "delete_existing" token is true or false in the incoming APXML
var should_delete = apxml.getValue("delete_existing");
if (should_delete)
{
apxml.setMethod("DEL");
apxml.setSubclass("action");
return (apxml);
}
else
{
var msEpochTimestampSec = apxml.getValue("date_reception");
var msEpochTimestampMilli = Long.parseLong(msEpochTimestampSec) * 1000;
apxml.setToken("bppm_epoch_timestamp_ms", Long.toString(msEpochTimestampMilli), APXMLToken.Type.NUMERIC);
apxml.removeToken("date_reception");
// Forward enriched APXML to xMatters, subject to deduplication
var webService = new xMattersWS();
var recentlyReceived = EventDeduplicator.getInstance().recentOccurrenceCount(apxml, DEDUPLICATOR_FILTER);
if (recentlyReceived <= 1)
{
result = apxml;
}
else
{
log.warn("***Deduplicator Suppressed Notification ***: An event with tokens " + webService.convertTokensToString(apxml) + " has been injected into the " +
apxml.getValue("agent_client_id") + " event domain " + recentlyReceived + " times within the configured suppression period. It has been suppressed." );
}
return result;
}
}
Note that this will only delete any pending notifications for the supplied incident ID, and no notifications at all will result from the DEL injections from the management system. ADD injections will be handled exactly the same as previously.
If this is not the desired behaviour (in other words, if the integration is expected to send notifications even when a DEL injection is received from the management system), then it is possible to respond to the DEL injection by sending a pair of requests to xMatters: a DEL and then a subsequent ADD. In other words, a single injection from the management system results in two request messages to xMatters.
This functionality is provided by the integration agent utilities software package, which is already included with many integrations. The utilities package is an optional add-on to the integration agent which provides capabilities such as a deduplicator filter to prevent event storms. It also provides the capability to send a DEL request to xMatters, either alone or in conjunction with a subsequent ADD request.
You can tell whether an integration includes the utilities package by looking for a reference to "xmattersws.js" in the "load" statements at the top of the main integration script. For example, in bmcbppm.js we see:
load("integrationservices/bmcbppm/xmattersws.js");
The BMC BPPM integration already includes the utilities package, but does not use it to enqueue messages for delivery to xMatters. (The utilities package was initially added to this integration purely so that event deduplication could be performed.) The utilities package can be added to any integration agent-based integration, but the process is outside the scope of this document.
To modify an integration so that it responds to a DEL injection by deleting old notifications and then creating new ones, use the utilities package's submitAPXML() method instead of the return APXML instruction shown previously. For example:
var sentSuccessfully = submitAPXML(apxml, shouldDelete);
This code will test the value of its second parameter and, if it is true, will send a DEL request message to xMatters before sending an ADD. It will ensure that the incident_id and apxml_process_group tokens are the same in the ADD and the DEL request, thereby guaranteeing that the messages are sent to xMatters in the correct order.
Ensuring synchronous deletions
In this document, the term "synchronous deletions" signifies that a DEL request and a subsequent ADD request will be sent from the integration agent to xMatters in the order in which they were received from the management system.
The integration agent's queueing mechanism ensures that messages will be sent synchronously as long as three conditions are met:
the request messages have the same integration name
they have the same apia_priority token
they have the same apia_process_group token
The instructions provided so far will ensure that messages have the same integration name, but they will not address the need for the same apia_priority token or the same apia_process_group token.
It is possible to set all messages to have "normal" priority (the options are "normal" and "high").
Example
When added to the BPPM integration's apia_input function, this instruction will ensure that all APXML messages sent to xMatters will have the same apia_priority.
apxml.setToken("apia_priority", "normal");
Similarly, it is simple to ensure that all DEL and ADD messages pertaining to a given incident ID have the same apia_process_group token. To do so, assign this token the same value as the incident_id token:
apxml.setToken( "apia_process_group", apxml.getToken("incident_id") );
Many older integrations were not designed to use these tokens, and so there is no possibility of conflict between the above code and the designed behaviour of the integration. However, it is possible that an integration will have existing code that uses the apxml_priority token to force the message into the integration's "high" or "normal" outbound queue.
The actual benefit of forcing a message into the "high" outbound queue is generally negligible, however, especially when compared to the cost of failed notifications if a DEL and an ADD reach xMatters out of order. Therefore it is recommended that you force all notifications to use the same apxml_priority token unless you know of a compelling reason for doing otherwise.
Further information
See the xMatters integration agent guide for more detailed information about the processes described in this article.
xMatters internal reference: DTN-4101, DTN-3922
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
Information supplied by Robert Hawk and the xMatters Security Office.
On Tuesday, March 1, 2016, a group of independent Internet security professionals identified and announced the Decrypting RSA with Obsolete and Weakened eNcryption (DROWN) Attack, and created a website dedicated to the vulnerability [ https://drownattack.com/ ]. The DROWN Attack exploits a protected Hyper-Text Transfer Protocol (HTTP) security mechanism such as Transport Layer Security (TLS) by using an existing vulnerability in the Secure Socket Layer (SSL) v2 code in the same library.
The xMatters cloud-based Software-as-a-Service (SaaS) does not use or enable SSL v2, and has no exposure to this vulnerability.
The DROWN Attack is a new form of cross-protocol, Bleichenbacher padding oracle attack. It allows an attacker to decrypt intercepted TLS connections by making specially crafted connections to an SSL v2 server that uses the same private key for all symmetric cipher suites. The xMatters Security Office analyzed exposure to CVE-2016-800 and found that while xMatters uses the affected code, the system configuration mitigates the vulnerability. The SSL v2 required by the vulnerability is not used by or enabled in the xMatters SaaS.
Resources:
https://drownattack.com/
https://www.openssl.org/news/secadv/20160301.txt
https://drownattack.com/drown-attack-paper.pdf
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0800
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-0800
The information in this article is proprietary and confidential toxMatters and xMatters customers. Do not distribute or print.
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
Information supplied by Robert Hawk and the xMatters Security Office.
On Friday, September 18, 2015, security researchers from Palo Alto Networks announced that they had found compiler malware in Apple Xcode, a popular Apple programming tool. They determined that this malware, known as XcodeGhost, had been introduced and uploaded to mirror sites hosted on the Baidu cloud sharing service used by many Chinese iOS/OS X developers.
The xMatters mobile app for the Apple iOS platform is not compromised by the XcodeGhost malware.
xMatters developers access Xcode only at the official Apple source, and development of the xMatters iOS platform is carried out only in North America. When the news of XcodeGhost was announced, the xMatters development teams checked their versions of Xcode and confirmed that the xMatters mobile app is not susceptible to XcodeGhost.
Resources:
http://researchcenter.paloaltonetworks.com/2015/09/malware-xcodeghost-infects-39-ios-apps-including-wechat-affecting-hundreds-of-millions-of-users/
http://researchcenter.paloaltonetworks.com/2015/09/novel-malware-xcodeghost-modifies-xcode-infects-apple-ios-apps-and-hits-app-store/#
http://www.bbc.com/news/technology-34311203
The information in this article is proprietary and confidential toxMatters and xMatters customers. Do not distribute or print.
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
This vulnerability was investigated and resolved in
https://xmatters.atlassian.net/browse/SCO-10911
Information supplied by Robin Percy, and Robert Hawk and the xMatters Security Office.
On Tuesday, November 10, 2015, the National Vulnerability Database (NVD) announced the identification of the Java Object Deserialization Vulnerability. This vulnerability was assigned the identifier CVE-2015-4852 by the NVD at the National Institute of Standards and Technology.
The xMatters cloud-based service is not negatively affected by the CVE-2015-4852 vulnerability.
The xMatters Operations team analyzed exposure to CVE-2015-4852 and determined that while the xMatters cloud-based software-as-a-service application uses the affected code, it does not use the code for primary service delivery. The affected code is used only for carrying out specific operations within the protected sub-networks of the cloud environment, and presents no exposure to the public networks with which the xMatters cloud interfaces, such as the Internet and telecommunications providers.
Technical Analysis
When serializing a Java object (for example, when caching the object in a session), the java.lang.Serializable interface must be implemented by initializing an ObjectOutputStream and writing the object to that stream using oos.writeObject(myObject). When an object is deserialized, the serialized representation is loaded into an ObjectInputStream and (MyClass) ois.readObject() is called. This operation is standard, and not considered particularly unsafe.
The risk in the procedure is that Java allows custom behavior to be defined during serialization and deserialization. If a class overrides the readObject() and writeObject() methods, they will be invoked by the corresponding object stream methods. There is no indication to the ObjectInputStream what type of class to deserialize; it invokes the readObject() method on whichever class it determines the stream contains. This means that a vulnerable class just has to be on the class path (and therefore discoverable by ObjectInputStream) for its readObject() method to be invoked.
Fortunately, the object streams only work with a specific binary representation for objects. This exposure exists in places where an application (or an underlying library) tries to decode that format.
Simplified vulnerability and exposure
The vulnerability with Apache Commons Collections is due to a class' readObject() method expecting some serialized values to be method invocations (for developers, it may help to think of GoF Command Pattern). When handling these values, it resolves three values: class name, method name, and args. It then uses reflection to look up the class and invoke the method with the given args.
On its own, this does not allow arbitrary code execution, merely the execution of any method accessible on the class path. Some libraries, however, such as Groovy Runtime and some Sprint components, may have risky methods available, such as eval(). It is for this reason that security alerts make a point of calling out the added danger of some library combinations.
Summary of analysis
The first step in analyzing the vulnerability of the xMatters cloud-based platform to this type of exploit was determining which ports were exposed to the public network (i.e., the Internet). Unsurprisingly, the only exposed port is port 8888, required for HTTP/S traffic.
The next task was to determine if and where readObject() is called on serialized representations. A grep of the On-Demand and Integration Agent codebases returned only one call to readObject(), located within the CompressionUtility class. Further investigation revealed that this class is used only on internally-generated values for optimizing transit of messages, and is not called on raw user input.
Finally, third-party libraries were examined to identify any that might be using readObject() for deserialization of user data. While this seemed unlikely, the following areas of the application were marked as higher risks than others:
SOAP API: As SOAP deals with unmarshalling XML input, it was necessary to ensure that embedded binary objects are disallowed in requests. For example, IBM's WebSphere product allows JMX objects to be embedded in its SOAP requests. Given that the xMatters API is defined by and validated against a WSDL, the investigation focused on the object types and formats allowed by the WSDL. The Operations team extracted all parameter type names into one list, and all complex type definitions into another. By comparing these lists, it was confirmed that all of the internally-defined complex types are composed of primitive types (strings, inits, etc.) and no types are defined by third parties. There is therefore no apparent way that xMatters would accept binary data in the SOAP API.
REST API: The same concerns existed for the REST API as for SOAP, with the added concern that the REST API does not have a WSDL. The REST API is small enough, however, that it was possible to manually inspect and confirm that it does not accept binary content.
GWT: Some concern existed over the fact that GWT defines its own protocol, and it was unknown whether it used Java serialization to pass data between the client and server. Other investigations have reverse-engineered the protocol and confirmed that it uses a custom serialization format. The Operations team used the Burp Suite proxy/sniffer to inspect client-server communication while navigating and submitting GWT pages to verify consistency with the spec. For more information, see https://docs.google.com/document/d/1eG0YocsYYbNAtivkLtcaiEE5IOF5u4LUol8-LL0TIKU/edit
Cocoon: While Cocoon did not seem to be at risk, the Operations team decided, given its complex implementation, to capture a sample request and verify that all arguments were regular HTTP form data.
Ping.jsp: This page was raised as a concern prior to a full understanding of the nature and mechanics of this vulnerability. Although the page responds to anonymous requests, it does not perform custom serialization of user input, meaning there is no apparent vector of attack.
Conclusions
The risk of vulnerability to CVE-2015-4852 is likely very low for xMatters. Despite this assessment, it is recommended that the Operations team and xMatters as a whole remain cognizant and alert to any further developments related to the vulnerability, and re-evaulate should any further information, patches, and tools become available.
Resources
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-4852
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-4852
http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in- common-this-vulnerability/
http://www.oracle.com/technetwork/topics/security/alert-cve-2015-4852-2763333.html
https://blogs.oracle.com/security/entry/security_alert_cve_2015_4852
https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread
http://security.stackexchange.com/questions/105475/can-cve-2015-4852-be-exploited-against-weblogic-servers-after-a-load-balancer
https://docs.google.com/document/d/1eG0YocsYYbNAtivkLtcaiEE5IOF5u4LUol8-LL0TIKU/edit
The information in this article is proprietary and confidential toxMatters and xMatters customers. Do not distribute or print.
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
Note: This vulnerability was investigated as documented in https://xmatters.atlassian.net/browse/COREL-2592 which has been marked as "Closed" and "Fixed".
Information supplied by Robert Hawk and the xMatters Security Office.
On Friday, February 19, 2016, the National Vulnerability Database (NVD) announced multiple stack-based buffer overflows in the send_dg and send_vc functions in the libresolv library of the GNU C library. This vulnerability was assigned the identifier CVE-2015-7547 by the NVDat the National Institute of Standards and Technology.
The xMatters cloud-based service operations have low exposure to the CVE-2015-7547 vulnerability. Monitoring will be used until a patch is deployed to detect system anomalies and respond accordingly.
The xMatters Operations team analyzed exposure to CVE-2015-7547 and determined that while xMatters uses the affected code, the system's configuration mitigates the vulnerability. The affected code is used for DNS name resolution for A/AAAA records. Exposure allows remote attackers to cause a denial of service (crash), or possibly execute code via a crafted DNS response that triggers a call to the ‘getaddrinfo’ function with the AF_UNSPEC or AF_INET6 address family, related to performing ‘dual A/AAAA DNS queries’ and the libnss_dns.so.2 NSS module. xMatters network architecture has Next Generation (NG) Deep Packet Inspection (DPI) firewalls before all internal assets, including DNS servers, as well as trusted DNS servers in a different network segment as processing and storage assets. xMatters will deploy the code fix in compliance to the Patch Management Policy and Procedures. The minimal exposure to the CVE-2015-7547 vulnerability will be monitored and dealt with by Operations.
Resources:
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7547
https://access.redhat.com/security/cve/cve-2015-7547
https://access.redhat.com/articles/2161461
http://www.cyberciti.biz/faq/linux-patch-cve-2015-7547-glibc-getaddrinfo-stack-based-buffer-overflow/
The information in this article is proprietary and confidential toxMatters and xMatters customers. Do not distribute or print.
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
This article describes a known issue with previous versions of the xMatters - ServiceNow integration. This issue has been resolved in the latestversion of the integration.
Batch sync process issues
Some versions of the ServiceNow integration were subject to group batch sync failures accompanied by the following message in the ServiceNow user interface:
"Transaction canceled - maximum number of outbound HTTP requests exceeded"
This issue occurswhen the background worker responsible for the batch group sync exceeds limitations. To resolve this issue, you can modify the integration to spread the load among multiple background workers.
To resolve this problem:
Log on to the ServiceNow user interface.
Click the Type filter text field, and then type xMatters.
In the Integration-xMatters section of the returned search results, under the Script Includes heading, click xMattersBackgroundBatchSync.
Find the line that starts "this.MAX_SOAP_CALLS_PER_PROCESS" and change the value to 10.
Click Save.
(xMatters reference: DTN-5156 Originally by Jeremy Brown)
Dynamic sync process issues
Some versions of the ServiceNow integration may also have been subject exceeding the HTTP request limits while syncing users dynamically - a process that occurred in the foreground. To avoid this limitation, ensure that all syncing operations are performed in background worker processes.
Step One
Add the following method at the bottom of the xMattersUserSync Script Include (x_xma_xmatters.xMattersUserSync):
/**
* Synchronize the user with xMatters
* @param currentName name of the user being modified
* @param userName name of the user being modified (could be previous)
* @param triggerRule String defining the action to take
* @param updateDevices If true, then devices need to be updated
* @return none
*/
syncUserAndDevicesFromBR : function(currentName, userName, triggerRule, updateDevices) {
this.log.debug("syncUserAndDevicesFromBR - currentName: " + currentName);
var userRec = new GlideRecord('sys_user');
userRec.get('user_name', currentName);
if (userRec) {
this.log.debug("syncUserAndDevicesFromBR - Found User: " + userRec.getUniqueValue());
var syncd = this.syncUser(userRec, userName, triggerRule);
if (syncd == "true" && updateDevices) {
this.log.debug("syncUserAndDevicesFromBR - About to updateDevices for User: " + currentName);
this.syncDevices(userRec);
}
} else {
this.log.debug("syncUserAndDevicesFromBR - Could not find user with currentName: " + currentName);
}
}
StepTwo
Add the following helper function to the bottom of the xMatters User Sync business rule:
/**
* Synchronize the user in a separate worker process.
* @param record the current user record
* @param userName name of the user being modified
* @param triggerRule String defining the action to take
* @param updateDevices boolean is true if devices need to be updated
* @return nothing
*/
function syncUserAndDevicesAsync(record, userName, triggerRule, updateDevices) {
var worker = new GlideScriptedHierarchicalWorker();
// This is the name of the worker in the sys_progress_worker table.
worker.setProgressName("User Sync Bus Rule Worker");
// Instantiate the ScriptInclude and indicate which method to fire
// The parameters are passed in order, not by name, so the first
// value of that putMethodArg is not used.
worker.setScriptIncludeName(appPrefix + '.' + "xMattersUserSync");
worker.setScriptIncludeMethod("syncUserAndDevicesFromBR");
worker.putMethodArg("currentName", record.user_name);
worker.putMethodArg("userName", userName);
worker.putMethodArg("triggerRule", triggerRule);
worker.putMethodArg("updateDevices", updateDevices);
worker.setBackground(true);
xMlog.debug("syncUserAndDevicesAsync - About to start background worker for: " + record.user_name);
worker.start();
return worker.getProgressID();
}
Step Three
Change the body of the User Sync business rule to call the helper method:
/*
// Update the user in xMatters
if (current.operation() == "update") {
syncd = userSync.syncUser(current, previous.user_name, "UpdateUser");
if (!seedOnly && syncd == "true") {
syncDevices(current, previous, userSync);
} else {
xMlog.debug("Users devices will not be synchronized as the user was not synchronized");
}
} else if (current.operation() == "insert") {
// Inject the user into xMatters
syncd = userSync.syncUser(current, current.user_name, "AddUser");
if (syncd == "true") {
syncDevices(current, previous, userSync);
} else {
xMlog.debug("Users devices will not be synchronized as the user was not synchronized");
}
} else if (current.operation() == "delete") {
// Delete the user from xMatters
userSync.syncUser(current, current.user_name, "DeleteUser");
}
*/
// Update the user in xMatters
var jobId;
if (current.operation() == "update") {
var updateDevices = current.email.changes() || current.phone.changes() || current.mobile_phone.changes();
jobId = syncUserAndDevicesAsync(current, previous.user_name, "UpdateUser", updateDevices);
xMlog.debug("jobId from call to syncUserAndDevicesAsync: " + jobId);
} else if (current.operation() == "insert") {
// Inject the user into xMatters
jobId = syncUserAndDevicesAsync(current, current.user_name, "AddUser", true);
xMlog.debug("jobId from call to syncUserAndDevicesAsync: " + jobId);
} else if (current.operation() == "delete") {
// Delete the user from xMattersjobId = syncUserAndDevicesAsync(current, current.user_name, "DeleteUser", false);
xMlog.debug("jobId from call to syncUserAndDevicesAsync: " + jobId);
}
View ArticleThis article describes how to open an exported list of users in Excel and properly display special characters, such as umlauts and accents used in people's names.
When you export a .csv file of users from xMatters and open it directly in Excel, entries with special characters will be displayed incorrectly. For example, AdladeMhlhuser will be displayed as AdladeMhlhuser.
This occurs because Excel does not properly detect UTF-8 encoded .csv files, which is the format exported by xMatters. To display these characters correctly, you must import the file into Excel and manually select UTF-8 encoding as the file origin.
To properly display special characters in the User Export file in Excel:
Export your list of users from xMatters. See Manage Users for more information about exporting a list of users.
Open a new blank workbook in Excel.
Select to import the user export .csv file. (Go to Data Tab and click From Text button)
Select the file you would like to load in excel.
In the import wizard, select Unicode (UTF-8) as the file origin:
Continue the import process, being sure to delimit the file by the 'Comma' character.
Note: If you use the User Export file to upload users with special characters in their names into xMatters, you must save the file in Excel with UTF-8 encoding. This option is available in the Save As dialog, by selecting: Tools > Web Options > Encoding > Save this document as: Unicode (UTF-8).
xMatters internal reference: DTN-5162, DTN-5296, SUP-13130
View ArticleThe 'multi-line' setting for a text property controls whether multi-line formatting is enabled for that property on a message form. This setting is enabled by default for new text properties. If the multi-line setting is turned off for a property, then its value will display as a continuous string of text.
For example, here's how the same default text would appear on a message form for a text property using multi-line formatting, and a property where this setting is not enabled:
A message sender can edit the default text provided for either property. The multi-line text property allows the sender to add line breaks to their text, while the text property with multi-line turned off forces the text to be entered into a single-line field.
In the resulting message, the multi-line formatting of the first property is preserved, while the text of the second property is displayed as a continuous string:
You can adjust the multi-line setting for a text property on a form-by-form basis after creating the text property. This setting is controlled at the form-level because a single property can be used on multiple forms in a workflow and you may want different formatting on separate forms.
To change the 'Multi-Line' setting for a text property:
Add the text property to your form's layout.
On the form's Layout tab, click Settings(the gear icon) for the property, and select or clearthe Multi-Line option.
Click Save Changes.
xMatters reference: DOC-5760
View ArticleThis article outlines possible causes of the following common SMS notification error that you may encounter in the xMatters log files:
Notification Error; [351] Invalid destination address value
Possible causes of this error include:
The phone number belongs to a prepaid user who does not have sufficient funds to receive an SMS message.
The end user is not provisioned to receive SMS from all short codes or specific short codes.
Note: T-Mobile does not deliver short-code SMS to any of its resellers' numbers.
The phone number is deactivated.
The destination address does not belong to the mobile operator. This often happens when a user switches carriers, such as from T-Mobile to AT&T, and the systems haven't fully updated yet.
We recommend contacting your carrier to discuss issues about receiving SMS, in general, or specifically from the xMatters short code.
Have you considered using the xMatters mobile app?
Although we continue to work with carriers, providers, and aggregators to lobby for the best end-user experience over SMS, we recommend the xMatters mobile app as the most reliable, easy-to-use, and cheapest solution for our clients.
For more information on the advantages of the mobile app, see our article: Mobile Clients vs. SMS - Why SMS STINKS.
xMatters internal reference:
DTN-6361; DTN-7824
View ArticleProblem
Email notifications seem significantly delayed. What's happening? Is it on the xMatters side or ours?
Environment
All versions of xMatters On-Demand; all mail server and client versions
Resolution
To resolve this issue, we'll need to collect a few samples of the messages that were slow to be delivered. We need the original messages so we can check the email headers - fortips on how to review e-mail headers, this site contains instructions on how to do just that.
http://www.levinecentral.com/mail_parse/
If the email messages are in msg format and you use a Mac, you have two options:
Use another app like MailReader; or,
Log into Office 365 and create a draft uploading the samples. Once the draft is saved you will be able to see the samples in the online version of Outlook under the Drafts folder.
Copy the email headers (also known as raw source or message details) from one of the samples.
Use a website like https://mxtoolbox.com/EmailHeaders.aspx or to analyze or parse the headers.
Check for any columns that contain the word "Delay" to determine which hop experienced the delay. In the following example we can see on the third row under Hop Delay that it was over 1 day and 1 hour. That indicates the issue was between the source and destination servers where see the longest hop delays and in that case you need to contact that provider or the mail server's administrator team who probably will need to check their logs to determine the root cause.
Cause
An external hop in the SMTP delivery chain was delayed (outside of xMatters and the local mail server) and that hop needs to be contacted and alerted about the delay. This can be anything from antivirus scanning or anti-spam or email security solutions.
View ArticleContents
Where and how do I log in?
What should I do when I log in for the first time?
Get going with the xMatters mobile app
How can I check my on-call schedule?
What should I do when I receive a notification?
Additional tips & tricks
The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
New to xMatters? Wish someone could give you a quick run-through of how to get started so that you're ready and able to receive and respond to notifications?
Great, because that's exactly what this guide will show you! We'll walk you through the basics of logging into our web application and mobile app, lead you through setting up your devices and checking your on-call schedule, and show you what to expect when getting a notification.
We'll also include additional recommendations to more in-depth documentation and resources that are available to help launch you from a newbie to an 'xPert' in no time.
Where and how do I log in?
There's a couple of different ways you can log into your xMatters account. To log into the xMatters web application, all you need is an internet connection and a web browser. And when you're on the go, the easiest way to stay connected is with the xMatters mobile app.
Web application
To log into the xMatters web user interface, go to your-company.xmatters.com (for example, if your organization's name is ACME Corp, its xMatters login URL would be something like acmecorp.xmatters.com).
Depending on how xMatters is configured, you'll log into xMatters using your corporate credentials or with xMatters credentials provided by your administrator.
Free Online Training
Additional resources:
Sign in to xMatters - step-by-step sign in instructions, browser recommendations, and troubleshooting.
What should I do when I log in for the first time?
After logging into the web application, you'll want to get your devices set up so that you can receive notifications from xMatters.
Set up your devices:
Click your name in the upper-right corner and select Devices.
Click Add Device and select the type of device you want to add.
Enter your device's contact information and then send yourself a test message to make sure you've configured it correctly.
Once you've successfully added your devices to xMatters, you can set delays and timeframes for when you're available to receive notifications on each device. For example, you may want to first receive notifications on your email and the mobile app, then after 5 mins on your mobile phone.
Externally-owned devices
If your organization uses a data sync process to add device data into xMatters from another system, then you may already have some devices configured in your profile. If the contact details for any of these devices are incorrect, let your xMatters Admin know so they can update the source system. You can add additional devices to your account without affecting the data sync.
Additional resources:
Manage your devices - how to add and remove devices and customize how xMatters contacts you.
Get going with the mobile app (iOS/Android)
If you don't already have the xMatters mobile app installed on your iOS or Android device, you can get it by going to the App Store or Google Play store and searching for "xMatters".
Once you've downloaded and installed the app on your device, there are two ways you can log in: manually using your host name, or by scanning a QR code displayed in the web user interface.
Log in using host name
Enter your hostname. For example, if the URL you use to access the xMatters web user interface is your-company.xmatters.com, then enter "your-company".
Tap Continue.
Enter your username and password.
Tap LOGIN.
Additional resources:
Log in with host name - instructions for manually logging into the mobile app with your host name and credentials.
Scan a QR code
Log into the xMatters web application, go to the Devices tab in your profile, and add a "Mobile App" device.
On the mobile app, tap Log in with QR Code.
Scan the QR code that's displayed in the web user interface.
Additional resources:
Log in with QR code - detailed instructions for logging in by scanning a QR code with your mobile
How can I check my on-call schedule?
You can use your schedule to check your current on-call status and your upcoming on-call schedule. The schedule includes on-call information for all of your groups and shifts.
Web application
Click your name in the upper-right corner, and then select Schedule.
You can filter the schedule to show when you're the primary on-call resource (by default, the schedule shows all of your shifts):
Temporary Absences
If you're unable to receive notifications for a period of time (such as if you're going on vacation or you're out sick), you can click Add Absence to add a temporary absence to your schedule, and optionally appoint someone else to receive notifications on your behalf while you're away:
Additional resources:
Using your schedule - checking your on-call schedule, filtering by escalation order, shift details, and temporary absences from your on-call duties.
Set a temporary absence - view, add, and manage temporary absences in your schedule.
Mobile App
From the iOS or Android app menu, tap My Schedule.
The xMatters mobile app allows you to view your schedule in a daily or monthly calendar view. Like the web application, you can filter the schedule to display only shifts where you're primary on-call - just tap the filter icon in the upper-right corner of the screen.
Temporary Absences
To set a temporary absence in your schedule from the mobile app, tap the stopwatch icon. (Note: The temporary absence feature is currently available on our iOS platform, and it's coming very soon to Android.)
What should I do when I receive a notification?
After you've set up your devices and your supervisor has assigned you to one or more shifts, you can expect to receive notifications. The messages you receive on any of your configured devices will include information about the event, and it may also include options for you to respond. Response options are highly customizable, so you should read or listen to all available options before you select your response.
How to respond
Email - Scroll to the bottom of the email to click on a response link. A dialog box may appear that allows you to add a comment with your response.
Mobile app - Tap one of the response options. The mobile app asks if you'd like to respond now, or respond with comment. Respond with a comment if you'd like to add additional context with your chosen response.
SMS (Text) - Reply with the number associated with your response choice.
Escalations
Depending on how the shift schedule is defined, if you don't respond within a pre-configured amount of time, xMatters automatically escalates the notification to the next on-call resource in the shift.
Additional resources:
Receive and respond to alerts - resources for receiving and responding to alerts on different device types.
Additional tips & tricks
Context-sensitive help - If you're not sure how a feature works, just click the icon in the top right corner of the screen to see help topics specific to the screen you're on.
Subscriptions - In xMatters, you can subscribe to receive notifications that keep you informed when certain types of events occur in your system. For more information, see:
How to use subscriptions
Find subscriptions and subscribers
Other helpful resources
xMatters Support & Community
xMatters Online Help
Video Tutorials
View ArticleThe Repeat Alerts feature available in our mobile apps is a great way to make sure that you never miss a critical notification. This feature allows you to repeat notifications sent to your the mobile app up to ten times, at an interval of every one to ten minutes. You can also set a threshold priority level that a notification must meet to be repeated (for example, you can specify that only high priority messages are repeated).
This article provides instructions for how to enable the Repeat Alerts feature in the iOS and Android mobile apps.
To enable repeat alerts on the iOS mobile app:
In the xMatters app menu, tap More.
Tap Accounts.
Tap Edit Settings.
Tap Notification Settings.
Tap Repeat Alerts.
Tap thetoggle to enable the feature.
Tap Repeat to set the number of times the notification will be repeated after the original alert occurs (one, two, three, five, or ten times).
Tap Interval to set the time between repetitions of the notification alert (one, two, three, five, or ten minutes).
Tap Priority to set the minimum priority level that notifications must meet or exceed for the notification alert to be repeated (high, medium, or low).
To enable repeat alerts on the Android mobile app:
In the xMatters app menu, tap Settings (the gear icon).
Tap Notification Settings.
Tap Repeating Alerts.
Tap the toggle to enable the feature.
Tap Repeat to set the number of times the notification will be repeated after the original alert occurs (one, two, three, five, or ten times).
Tap Interval to set the time between repetitions of the notification alert (one, two, three, five, or ten minutes).
Tap Priority to set the minimum priority level that notifications must meet or exceed for the notification alert to be repeated (high, medium, or low).
Note: Repeat notifications do not stop regular group escalations. Depending on the number and length of the repeat alert settings, group escalations may occur before a message finishes its repetition cycle.
xMatters reference:
DTN-6360 Originally by Daniel Thomas
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
Debugging Integration Agent Scripts
The Integration Agent has a built-in debugger with a graphical user interface that allows you to trace through Javascript code and watch variables. If you are familiar with the use of debuggers, you might find this to be a useful tool for developing and troubleshooting Integration Agent scripts.
The debugger is particularly well-suited to troubleshooting complex code such as that found in datasync integrations.
The debugger is not suitable for troubleshooting code that relies upon imported Java objects or methods.
Accessing the Debugger
To access the debugger, copy the batch script attached to this article into the Integration Agent's tools folder, and then invoke it from a command prompt. (The batch script is in Windows format, but if you are using another operating system, you should be able to easily port the commands to work with your OS shell.)
Using the Debugger
Once you have invoked the debugger, load the main integration script via the Run command in the File menu (or pressCtrl-N).
The "Go" button will execute all the code in the main script, as well as any other scripts that are invoked via "load" instructions in the main file.
To trace through a script, use the "Step Into" button instead of the "Go" button.
To monitor the value of a variable, type its name in the "expression" field of the "Watch" window.
Thisonly works while you are stepping through code (ie, it doesn't work if you use the Go button.) It's best to wait until you've stepped through the "var" instruction that instantiates the variable before trying to access its value.
You can navigate from one script to another by using the "Window" menu, when more than one script has been loaded.
Limitations and Hints
Imported Java code
The Rhino debugger doesn't handle imported Java code well. The Import statements themselves won't cause trouble, but neither will they make Java objects and methods available in debugging sessions in the same way that they are available when the script is executed by the Integration Agent.
In particular, this will cause errors to be thrown when log instructions ("log.debug", "log.info", "IALOG.debug", etc) are encountered, since these instructions invoke imported Java code. Once this happens, the best thing to do is to exit the debugger and start it again.
Binary code
Before loading the integration script, it's best to comment out any instructions that invoke binary code (such as the log instructions mentioned earlier.) This is easily done with a text editor's search-and-replace tool. Keep a copy of the original script so you can revert after the debugging session.
Step Over
Use the Step Over button to avoid tracing through code that you don't want to debug. For example, if you step over a "load" statement, the target script will not replace the one that you are debugging. The stepped-over code will still have been executed; you just avoid seeing the individual instructions.
Testing code
You can also use the debugger to test code on the fly. As soon as the debugger starts up, you will see a "Javascript Console" window. You can type Javascript code into this window and the results will appear when you click the "Go" or "Step Into" button. For example,
Rhino 1.7 release 4 2012 06 18
js> typeof {"name": "john"};
object
js> typeof "name";
string
js>
Forcing methods to process data
Most Integration Agent scripts don't do anything until they receive input, typically from the apclient gateway or the service gateway (the "/http" interface.) These gateways are not available when you use the debugger, and so the scripts will not automatically produce any output after being loaded into the debugger. To test the script with production data, you can extract the data from the Integration Agent log and then add an instruction to the end of the script to send the data to the entry point.
For example, to forcethe script's apia_http() method to process data:
If the data hasn't already been written to the Integration Agent log, add an instruction to the apia_http() function in the main integration script; e.g.:
log.info("Entering apia_http(); httpRequestProperties is: \n" + httpRequestProperties.toString() );
Start the Integration Agent and send the data to it in the usual way (eg, from the management system).
Stop the Integration Agent and extract the data from the Integration Agent log file.
Add an instruction that includes the data to the very end of the main integration script file, similar to the following:
apia_http( "{Content-Type=... (actual data ommitted)..., Host=127.0.0.1:8081}", "" );
When sending data to the apia_http function, you need to wrap the data from the log file in double-quotes, and escape any double-quotes in the data with a backslash ('\'). Note the secondempty parameter, since the apia_http function expects two parameters.
Comment out any instructions that invoke Java code (such as log.debug, etc.) and save the file.
Start the debugger and run the file as described above.
xMatters reference: DTN-5414; information supplied by Jeremy Brown
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
This article is intended to help anyone working with Java scripts while modifying or troubleshooting integration scripts. The Integration Agent's JavaScript interpreter ( Rhino ) allows Java objects to be imported in integration scripts. This can add versatility to your integration, but can also cause some scripts to become error-prone.
Specifically, Java String objects have caused several reported script defects. This article will describe the differences between JavaScript strings and Java String objects, and will recommend techniques for avoiding some of the problems that can result when using Java String objects in integration scripts.
Java String objects versus JavaScript strings
This article will refer to Java String objects (with an uppercase "S") and JavaScript strings (with the "s" in lowercase.) This is purely for clarity, considering the similarity between "Java" and "JavaScript": like the names of the languages, JavaScript strings and objects are superficially very similar to, and easily confused with, their equivalents in Java, which can cause important differences to be overlooked. Hopefully,the lowercase and uppercase "S" will make it easier to distinguish between them.
Each language has an object class called "String" (with an uppercase "S",) but we will try to use the term "Java String object" consistently to distinguish between the Java and JavaScript string entities.
Java String objects and JavaScript strings have many similarities and a few important differences, including:
Their class names: "java.lang.String" in Java, and "String" in JavaScript.
Some of their methods (which are the functions that determine what an object can do, in object-oriented programming.)For example, Java String objects have a "replaceAll()" method, and JavaScript strings do not.
Because of the differences in methods between JavaScript strings and Java String objects, integration scripts are prone to errors like the following when Java String objects are used in integrations:
Exception - name: [InternalError], message [The choice of Java method java.lang.String.replace matching JavaScript argument types (function,string) is ambiguous; candidate methods are:
class java.lang.String replace(java.lang.CharSequence,java.lang.CharSequence)
class java.lang.String replace(char,char)]
The reason for this particular error is that the integration script has called a Java String object's "replace" method with an argument that it cannot accept. Specifically, a regular expression was passed as an argument. The JavaScript string's "replace" method accepts regular expressions, but the Java String object's "replace()" method does not.
Java String objects have a "replace()" method that accepts only string arguments. For example, the following code would execute without error:
var oldString = new java.lang.String("Working with strings is easy");
var newString = oldString.replace("easy", "hard");
The replace operation would fail if it were written to use a regular expression instead of a string argument, though:
var oldString = new java.lang.String("Working with strings is easy");
var newString = oldString.replace(/e[a-z]+y/g, "hard");
Based on this information, we can infer that the above error message resulted from a script that was written to work with JavaScript strings, but failed when it encountered a Java String object.
Where Java String objects come from
Integration scripts are written in JavaScript, which is distinct from the Java language. In the simplest integration scripts there will be no Java objects.
However, JavaScript has limitations in terms of the functionality that it offers. For example, JavaScript offers very little capability for working with dates and times. Fortunately, the Rhino JavaScript interpreter allows Java classes to be imported, so if we wantto work with dates and times, for example, we can import Java classes that will allow us to manipulate and compare timestamps much more easily than we could do in pure JavaScript.
Since many of these Java functions provide output in string format (as opposed to integers, etc,) we shouldn't be surprised to find that the imported Java classes introduce Java String objects into our integrations. However, while we might not expect these Java String objects to replace existing JavaScript string literals, this is exactly what can happen. The problem is complicated by the fact that JavaScript is not a strongly typed language (in fact it is untyped.)
This means that the following chain of events can occur in a JavaScript code fragment:
var jsString = "this is a JavaScript string literal."; // create a new JavaScript string
jsString = someJavaFunction(); // send the output from imported Java code to the JavaScript string
jsString = jsString.replace(/a[a-z]+c/g, "def"); // expect an error here because jsString has been replaced by a Java String object
In the above code fragment, a JavaScript string has been replaced by a Java String object. Any JavaScript-specific functionality will no longer be available from this object.
In a strongly typed language, we would expect an error if we tried to assign a value of the wrong type to a variable, but in JavaScript this is tolerated.
An example of a JavaScript function that is broken by Java String objects is formatStringForE4X(), which is common in Integration scripts:
formatStringForE4X: function(string)
{
string = string.replace(/<\?(.*?)\?>/g,''); // remove XML processing instructions
return string;
}
When implemented in this way, the function will throw a runtime error whenever a Java String object is sent to it (because the function attempts to send a regex to the String object's replace() method.)
To prevent these runtime errors, it is best to replace Java String objects with JavaScript strings wherever feasible.
Converting Java Strings to JavaScript strings
As noted above, it's easy to inadvertently create a Java String object (i.e., when a JavaScript String is expected,) or to convert a JavaScript string to a Java String object without meaning to. Converting back to JavaScript is simple, but not intuitive.
To create a new JavaScript string (as opposed to a Java String object,) use single- or double quotes to define an initial value (empty or otherwise) when declaring the variable, or use the "new String()" syntax:
var jsString1 = '';
var jsString1 = "";
var jsString3 = "I am a JavaScript string literal";
var jsString4 = new String("I am a JavaScript String object");
Confusingly, a variable created with the "new String" syntax will be identified as an "object" by the typeof operator. This makes it difficult to distinguish between a Java String object and a JavaScript String object (since the typeof operator identifies them both as "object". JavaScript string literals, on the other hand, will be identified as "string" by the typeof operator.
typeof jsString1;
(output:) string
typeof jsString3;
(output:) string
typeof jsString4;
(output:) object
More comprehensive information about String literals and String objects in JavaScript is here
To convert a Java String object to its JavaScript equivalent, concatenate the Java String object to an empty string:
var javaString = new java.lang.String("I am a Java object");
typeof javaString;
(output:) object
var jsString5 = "" + javaString;
typeof jsString5;
(output:) string
In the above example, a Java String object has been converted to a JavaScript string by concatenating it to an empty JavaScript string.
Best practice
Converting a potential Java String object to a JavaScript string (ie, by creating a new JavaScript string and then concatenating the Java object to it) has proven to be a reliable way of handling strings of unknown origin. This mechanism has been used successfully with the formatStringForE4X() function mentioned earlier:
formatStringForE4X: function(string)
{
string = "" + string; // Replace the string with one that is known to be a JavaScript object
string = string.replace(/<\?(.*?)\?>/g,''); // remove XML processing instructions
return string;
}
In general, whenever working with Java String objects (or when receiving data from imported Java classes,) convert the string data to a JavaScript string (as described above) as soon as possible.
Information created and supplied by Jeremy Brown.
xMatters reference: DTN-5482
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
What's the problem?
The POODLE vulnerability isa security issue within the SSL 3.0 protocol, identified by Google in October of 2014. The researchers found that an attacker with control over network connections (such as on a public WiFi network) could trick a web browser into leaking personal cookies. In turn, these cookies could be used to assume another identity on secure web services.
The good news is that the SSL 3.0 protocol has been superseded by more secure communication protocols for at least a decade. The bad news is that the same vulnerability exists on some implementations of the Transport Layer Security protocol.
This means that xMatters will require all customers to update their local deployments to ensure that they are not attempting to communicate usingSSL v3.0 or TLS v1.0.
What are we doing about it?
xMatters has determined that the bestway to ensure that our users are protected from this vulnerability is to do the following:
First, in accordance with network security recommendations, we will bedisabling SSL 3.0 support on all of our servers. This effectively prevents older and out-of-date browsers from inadvertently falling prey to the vulnerability by preventingthem fromaccessing xMatters using SSL 3.0.
Second, we must ensure that the only applications (such as xMatters integration agents and EPIC clients) communicating with xMatters are upgraded to use protocols other than SSL 3.0. (Previous versions of the integration agent had SSL 3.0 support enabled as a back-up protocol in the event that TLS was not available, which means they may be subject to this vulnerability.)
How does it affect you?
This will affect you in one or both of the following ways:
If you have a local xMatters integration agent in your deployment, you will need to ensure that it has been upgraded to a minimum of version 5.1.3.
AIX integration agent deployments require a minimum version of 5.1.4.
If you are using an EPIC client to sync your data, you must ensure it isusing a minimum JRE version of 1.7.0_45.
How long do youhave to do this?
Due to the critical nature of thisvulnerability, xMatters will be disabling these older versions by region, according to thefollowing schedule:
Region
Scheduled Update
Australia, Japan, Asia Pacific
Monday, May 25, 2015
Europe, Middle East, Africa
Monday, June 1, 2015
North America
Monday, June 8, 2015
On these dates, a new protocol suite will be implemented in all xMatters On-Demand deployments for each region. If you have not updated your environments at that time, you may not be able to communicate with xMatters.
How can you get more help?
If you have any questions, or need help with the upgrade, contact xMatters Support using the Submit a Request form.
How do youcheck yourintegration agent version?
On the machine hosting the integration agent, open a command window and navigate to the <IAHOME>/bin folder. Then, executethe following command:
./iadmin.sh get status
If the integration agent is running, you should see an output that resembles the following:
Version: 5.1.3-SNAPSHOT r79677
Release date: 07-Jan-2015
Agent started: 10-Apr-2015 13:07:12
Agent ID: vic-vm-mbennett/10.2.1.168:8081
Integration Services:
Event domain: applications
Name: sample-relevance-engine
Clients: [MG, APCLIENT]
URL: http://10.2.1.168:8081/applications_sample-relevance-engine
Started: 10-Apr-2015 13:07:13
Last request: none
Status: ACTIVE
Pending request count: 0
Normal priority inbound APXML queue size: 0
High priority inbound APXML queue size: 0
Normal priority outbound APXML queue size: 0
High priority outbound APXML queue size: 0
Event domain: default
Name: sample
Clients: [MG, APCLIENT]
URL: http://10.2.1.168:8081/default_sample
Started: 10-Apr-2015 13:07:13
Last request: none
Status: ACTIVE
Pending request count: 0
Normal priority inbound APXML queue size: 0
High priority inbound APXML queue size: 0
Normal priority outbound APXML queue size: 0
High priority outbound APXML queue size: 0
Event domain: del
Name: del
Clients: [APCLIENT]
URL: http://10.2.1.168:8081/del_del
Started: 10-Apr-2015 13:07:14
Last request: none
Status: ACTIVE
Pending request count: 0
Normal priority inbound APXML queue size: 0
High priority inbound APXML queue size: 0
Normal priority outbound APXML queue size: 0
High priority outbound APXML queue size: 0
Event domain: generic
Name: generic
Clients: [MG, APCLIENT]
URL: http://10.2.1.168:8081/generic_generic
Started: 10-Apr-2015 13:07:14
Last request: none
Status: ACTIVE
Pending request count: 0
Normal priority inbound APXML queue size: 0
High priority inbound APXML queue size: 0
Normal priority outbound APXML queue size: 0
High priority outbound APXML queue size: 0
Event domain: ping
Name: ping
Clients: [MG, APCLIENT]
URL: http://10.2.1.168:8081/ping_ping
Started: 10-Apr-2015 13:07:13
Last request: none
Status: ACTIVE
Pending request count: 0
Normal priority inbound APXML queue size: 0
High priority inbound APXML queue size: 0
Normal priority outbound APXML queue size: 0
High priority outbound APXML queue size: 0
xMatters Servers:
URL: http://10.2.1.36:8888/api/services/AlarmPointWebService
Connectivity status: PRIMARY_ACCEPTED
Last heartbeat attempt: 10-Apr-2015 13:52:58
URL: http://10.2.1.36:8888/api/services/AlarmPointWebService
Connectivity status: UNKNOWN
Last heartbeat attempt: none
The version of the integration agent is reported in the first line of the output; it must be "VERSION: 5.1.3-" or higher.
How do you verify that your integration agent is working properly?
First, you can use the output you generatedto check the health of the integration agent; if the integration agent is properly connected, you should see PRIMARY_ACCEPTED on at least one of the defined xMatters servers.
Second, check the <IAHOME>/log/AlarmPoint.txt file and ensure it does not contain any errors.
Third, inspect the log file for activity: if the integration agent is active and in use, you should see log entries for events coming in and appearing in xMatters.
If there is no activity or the integration agent isn't currently in use, you can run a test injection using the out-of-box ping integration (if the ping endpoint has not been disabled):
In the <IAHOME>/bin folder, run the following command (replacing <xm_user> with the user ID of a user with at least one active device in xMatters, and <server_ip> with the IP address of a server that can be pinged from the integration agent machine):
APClient.bin --map-data ping <xm_user> Test <server_ip> INCOMTEST0001
Check the <IAHOME>/log/AlarmPoint.txt filetoensure there are no errors, and that the integration agent created the event.
Confirm that the event is created in xMatters, and the user is notified.
Respond to the notification on the device, and confirm that the integration agent processes the response (it should return the ping results to the user).
Note: Make sure you repeat these steps for all integration agents in your deployment. If the ping integration endpoint has been disabled on your system, replace the above command line with an appropriate one for your integration.
How do you upgrade your integration agent?
Start by downloading the latest version from the integration agent product page.
Complete instructions for upgrading your integration agent are contained within the release notes for each version:
integration agent 5.1.4 release notes
integration agent 5.1.5 release notes
The basic steps are:
Back up your existing files, including any integrations or other customizations.
Extract the integration agent download package to create a new integration agent installation folder.
Copy the necessary configuration files to the new integration agent folder.
Merge any integrations and customizations.
Note:
Some older integrations relied on a specific version of the IAUtils package that was bundled with versions of the integration agent prior to 5.1.3. If you are upgrading an integration agent version 5.1.2 or lower, and have an existing integration, you may need to restore your IAUtils file. For more information, refer to Integration agent utilities.
How do you make sure you are protected?
First, stop your existing integration agent if you haven't already, and start the new version. Then check the <IAHOME>/log/AlarmPoint.txt file, and ensure there are no errors.
Next, run the get-status command as explained above and check the version number (must be 5.1.3 or later!) and ensure at least one of the xMatters servers identified at the bottom of the output contains PRIMARY_ACCEPTED. You can also follow the other steps outlined in the verification process above.
What about EPIC data sync?
If you are using EPIC to synchronize user and group data, we recommend that you use the version of EPIC that matches your version of xMatters On-Demand.
You should also ensure that you are using a minimum JRE of version 1.7.0_45, which is compatible with TLS version 1.2.
Note: It is possible to have two versions of JRE installed on the same machine. Use the following instructions to determine which version your EPIC client is using.
The first step is to determine whether the JAVA_HOME environment variable is set for your EPIC client. If this is set, the EPIC client will always use the JRE set for JAVA_HOME. On the machine where your EPIC client is installed, run one of the following commands:
On Windows:
echo \%JAVA_HOME\%
On Unix:
echo $JAVA_HOME
If itreturns a value, run
\%JAVA_HOME\%/bin/java -version
or
$JAVA_HOME/bin/java -version
The version specified will be the one used by the EPIC client.
If the echo command does NOT return a value, it means that the JAVA_HOME environment variable is not set. To get the JRE version on your machine, run:
java -version
The JRE version returned by either command should resemble the following:
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
If the Java version identified in the first line of the output is not at LEAST 1.7.0_45, you will need to update to a minimum of Java 7 ; we recommend the latest update available.
xMatters internal references
DTN-4292, COR-2362, COR-2410, SUP-9967, COREL-112
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
The 5.x (Premises) version of the xMatters web server may be subject to the following identified security vulnerabilities:
CVE-2013-2566, CVE-2015-2808: SSL/TLS use of weak RC4 cipher port 8443/tcp over SSL
CVE-2014-3566: SSLv3 Padding Oracle Attack Information Disclosure Vulnerability (POODLE) port 8443/tcp over SSL
CVE-2011-3389: SSLv3.0/TLSv1.0 Protocol Weak CBC Mode Server Side Vulnerability (BEAST) port 8443/tcp over SSL
Resolution
Eliminating these vulnerabilities depends on your deployment:
The xMatter 5.1 patch 011 and Integration Agent 5.1.8 releases include the recommended changes and updates to Jetty that are described in this article.
New installations of 5.1 patch 011 using the 5.1.8 version of the Integration Agent that have completed the SSL configuration described in the xMatters installation and administration guide need to complete only the "Disallow CBC algorithms" section below.
Deployments that have been upgraded to 5.1. patch 011 and are using the Integration Agent 5.1.8 should review the "Configure Jetty" and "Disallow CBC algorithms" sections below for instructions.
Deployments using any xMatters 5.1 patch prior to patch 011 should review and follow all instructions below.
All customers are recommended to use a proxy server in front of the xMatters web container. For more information about configuring a proxy server with the required facilities, such as Apache and IIS, refer to the xMatters installation and administration guide.
Recommendations
The following recommendations are suggested for any customers wanting to eliminate these issues on versions of xMatters 5.1 patch 010 and prior.
Configure Jetty to use TLS version 1.2 or 1.1
You can reconfigure Jetty to force the SSL sockets to support only TLSv1.1 and TLSv1.2.
NOTE: If you are configuring a deployment running 5.1 patch 010 (or earlier), install the 5.1 patch 011 update first OR do the following:
Stop the xMatters webserver.,
Navigate to <xMHOME>/webserver/lib/ext. Move the com.xmatters.jetty.handler.jar file to a backup location, and make a backup copy of thejetty-ssl.xml file.
Download the jetty-redirector.jar file attached to this article and copy it into the<xMHOME>/webserver/lib/ext folder.
To configure Jetty:
Open the <xMHOME>/webserver/lib/ext/jetty-ssl.xml file in a text editor, and then locate the following line:
<New class="org.mortbay.jetty.security.SslSocketConnector">
Change it to the following:
<New class="com.xmatters.jetty.security.SecureSslSocketConnector">
Save and close the file.
Restart the xMatters web server.
NOTE: Upgrading your web server (i.e., applying an xMatters patch) may overwrite configuration changes. Newer patches may also require different steps to configure Jetty. See the "Applying product patches" section below for more information.
Disallow CBC algorithms
Disallowing CBC algorithms will remediate issues with the BEAST vulnerability. This step is required on all
To disallow CBC algorithms:
Navigate to <xMHOME>\jre\lib\security and open the java.security file in a text editor.
Locate the following line:
# jdk.tls.disabledAlgorithms=MD5, SHA1, DSA, RSA keySize < 2048
Change it to the following:
# jdk.tls.disabledAlgorithms=MD5, SHA1, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=CBC
Save and close the file.
Restart the xMatters web server.
Integration Agent
The Integration Agent supports secure connections to the web server over TLSv1 only. This is the only protocol enabled by the JRE 1.7 that is shipped with the Integration Agent. To allow the Integration Agent to communicate with the web server, you must allow TLSv1 connections on the web server, or deploy the work-around attached to this article.
To configure the Integration Agent (version 5.1.7 only):
Stop the Integration Agent.
Navigate to the<IAHOME>/lib folder and make a back up copy of the com.alarmpoint.apex.integrationagent.jar file ina separate directory
Download the com.alarmpoint.apex.integrationagent.jar file attached to this article and copy it into <IAHOME>/lib.
Restart the Integration Agent.
NOTE: The 5.1.8 release of the Integration Agent includes an updated JRE (1.89) that is not subject to this issue.
Further information
Applying product patches
When applying an xMatters patch, any changes made to configuration file parameters may not be retained after install. Each patch may overwrite modifications to some or all of the following files.
node\assets\resources\spring\integration*.xml
webserver\webapps\cocoon\WEB-INF\classes\resources\spring\integration*.xml
webserver\webapps\axis2\WEB-INF\classes\resources\spring\integration*.xml
webserver\webapps\mobilegateway\WEB-INF\classes\resources\spring\integration*.xml
node.sh
webserver.sh
node-start.conf
webserver-start.conf
c3p0.properties
c3p0-config.xml
Backup your configuration files before applying the patch; any changes you have made prior to the patch will need to be reapplied. For more information, consult the release notes for the patch you are applying, or refer to the xMatters installation and administration guide.
Additional Jetty configuration information
How to secure the xMatters web server for OnPremise installations
xMatters internal reference: DTN-5547, SUP-13893
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
This article describes how to configure, enable, and use SSL keys and certificates with the Jetty web server for xMatters customers using our on-premises product version 5.1.
NOTE: This article is not intended to be a comprehensive security document, and does not include information related to the strength of ciphers or security standards. You should always consider the key strength and size that is required by your organization. xMatters always follows industry best practices and standards published by security authorities. It is recommended that premise-based clients use an enterprise-grade proxy server to front and load-balance the xMatters user interface.
Preparation
When following the information in this article, ensure that you back up relevant files before overwriting or deleting them. Work with your network staff to obtain the following information for production installations:
DNS Domain for your URL
Internet-facing IP Address
Internet-facing DNS Entry for the latter IP address in the desired DNS Domain
You must also install the Java Development Kit (5.0 or later), which provides the keytool command referenced in this article. It is a best practice to place the JDK binaries on the classpath so that the keytool command is accessible from any directory. To learn more about keytool, click here.
NOTE: if you only want to renew a certificate, click here.
Generating a keystore
To generate a keystore, run the following command (the parameters are explained below):
keytool-genkey-keystore<your.keystore.file>-storepass<password>-keypass<password>-keyalgRSA-alias<your.alias>-dname
"CN=<your.internet.facing.url>,OU=<your.org.unit>,O=<your.company>,L=<your.city>,ST=<your.state>,C=<your.country>"
Replace the placeholder parameters in the command with the actual values, as explained in the following points:
- This will be the file that will hold the certificate/key pair and any certificate chains. Do not lose this file. If you have a revision control or document management system you should add this file to it when the process is done.
(-storepass)- Password to protect access to the keystore file.
(-keypass) - Password to protect access to the private key of the generate key pair.
- Alias you chose to store your key/cert pair.
- Internet-facing URL to which the certificate will be bound. If the URL used to navigate your site and the URL in the certificate are not identical, you will receive certificate warnings.
-dname values - Distinguished name values. For full details, see the following excerpt from the keytool URL referenced in the previous section.
X.500 Distinguished Names are used to identify entities, such as those which are named by thesubject and issuer (signer) fields of X.509 certificates. keytool supports the following subparts
commonName - common name of a person, e.g., "Susan Jones"
organizationUnit - small organization (e.g, department or division) name, e.g., "Purchasing"
organizationName - large organization name, e.g., "ABCSystems, Inc."
localityName - locality (city) name, e.g., "Palo Alto"
stateName - state or province name, e.g., "California"
country - two-letter country code, e.g., "CH"
When supplying a distinguished name string as the value of a -dname option, as for the -genkey or -selfcert commands, the string must be in the following format:
CN=cName, OU=orgUnit, O=org, L=city, S=state, C=countryCod
where all the italicized items represent actual values and the above keywords are abbreviations for the following:
CN=commonName OU=organizationUnit O=organizationName L=localityName S=stateName C=country
A sample distinguished name string is
CN=Mark Smith, OU=JavaSoft, O=Sun, L=Cupertino, S=California, C=US
and a sample command using such a string is
keytool -genkey -dname "CN=Mark Smith, OU=JavaSoft, O=Sun, L=Cupertino, S=California, C=US" -alias mark
Case does not matter for the keyword abbreviations. For example, "CN", "cn", and "Cn" are all treated the same.
Order matters; each subcomponent must appear in the designated order. However, it is not necessary to have all the subcomponents. You may use a subset, for example:
CN=Steve Meier, OU=SunSoft, O=Sun, C=US
If a distinguished name string value contains a comma, the comma must be escaped by a "\" character when you specify the string on a command line, as in
cn=peter schuster, o=Sun Microsystems\, Inc., o=sun, c=us
It is never necessary to specify a distinguished name string on a command line. If it is needed for a command, but not supplied on the command line, the user is prompted for each of the subcomponents. In this case, a comma does not need to be escaped by a "\".
NOTES:
For -dname, do not leave any fields empty (empty fields the CA will consider the CSR as invalid).
Provide the full name for ST field of -dname (e.g., use New York instead of NY).
Use the two-letter country code in the C field of -dname.
Example
The following command shows an example command with parameter placeholders replaced with sample parameters:
keytool-genkey-keystorekeystore-storepass123456ABC-keypass123456ABC-keyalgRSA-aliasjetty-dname
"CN=www.company.com,OU=information_services,O=ABC,L=NewYork,ST=NewYork,C=US"
Generate Certificate Signing Request (CSR)
This command generates a Certificate Signing Request (CSR) that is used to request a certificate from a certificate authority, based on your generated key:
keytool-certreq-keyalgRSA-keystore<your.keystore.file>-storepass<password>-alias<your.alias>-filecertreq.csr
Submit Certificate Signing Request to Certificate Authority (CA)
Purchase a certificate from a trusted certificate authority such as VeriSign, Network Solutions, or Thawte.
Ensure that the vendor's certificates are trusted by most modern web browsers. As part of this process, the certificate vendor will attempt to verify your identity, your company's identity, and that you are an authorized representative of the company. The vendor should also verify that the URL you are requesting for the certificate is valid and owned by your company. In your planning and scheduling, you should allow at least 3-10 business days for this process. For planning purposes, it is recommended that you discuss time lines directly with the certificate vendor.
Make a note of the certificate's expiration date so that it can be renewed before expiry (tip: set a calendar reminder well before the certificate expiry date).
When you are prompted for an email address for certificate management, it is recommended that you set this address to a group account (i.e., so that several people will be notified about any certificate issues such as expiration).
Add certificates to keystore
After you have received the certificate (and possibly the certificate chain files) you need to add them to the keystore you used to generate the CSR. After adding the certificate, the keystore will be ready for use by Jetty. Back up the keystore and record all related information (e.g., password).
To add the certificates, run the following commands:
keytool-import-alias<your.root.alias>-keystore<your.keystore.filename>-trustcacerts-file<filename_of_the_chain_certificate>
keytool-import-alias<your.alias>-keystore<your.keystore.filename>-trustcacerts-file<your_certificate_filename>
Example
Assume that you receive a trial SSL certificate from VeriSign using the generated .csr file; you will receive three certificates: your SSL certificate, a root CA certificate and an intermediate CA certificate. Save them to jetty.crt, rootCA.cer and intermediateCA.crt respectively and import these certificates into a keystore named "keystore".
keytool-import-aliasjetty_int-keystorekeystore-trustcacerts-fileintermedicateCA.crt
keytool-import-aliasjetty_root-keystorekeystore-trustcacerts-filerootCA.cer
keytool-import-aliasjetty-keystorekeystore-trustcacerts-filejetty.crt
After you have imported these certificates, your keystore should look similar to the following:
Keystoretype:jks
Keystoreprovider:SUN
Yourkeystorecontains3entries
jetty,Apr7,2009,keyEntry,
Certificatefingerprint(MD5):D2:EB:4E:8F:C5:D9:9E:7E:63:68:21:64:47:19:43:E0
jetty_root,Apr7,2009,trustedCertEntry,
Certificatefingerprint(MD5):B6:9D:A4:40:52:02:50:0D:D5:9C:E1:B8:4B:66:C4:AC
jetty_int,Apr7,2009,trustedCertEntry,
Certificatefingerprint(MD5):8D:E9:89:DB:7F:CC:5E:3B:FD:DE:2C:42:08:13:EF:43
Configure Jetty
After you have prepared your keystore file, you must configure Jetty to use it.
To configure Jetty:
Back up the original keystore.
Copy the keystore file to webserver/resources and remove the default keystore file that comes with xMatters.
Navigate to the webserver/etc/jetty-ssl.xml file and open it in a text editor.
Locate the following text and replace <your.keystore.filename>with the name of your keystore file (note that you must also replace<your.store.password>, <your.key.password>, and <your.trust.password> with the actual values from the initial keystore creation in the "Generating a Keystore" section):
<Call name="addConnector">
<Arg>
<New class="org.mortbay.jetty.security.SslSocketConnector">
<Set name="Port">8443</Set>
<Set name="maxIdleTime">30000</Set>
<Set name="handshakeTimeout">2000</Set>
<Set name="keystore">
<SystemProperty name="jetty.home" default="." />/resources/<your.keystore.filename>
</Set>
<Set name="password">
<your.store.password>
</Set>
<Set name="keyPassword">
<your.key.password>
</Set>
<Set name="truststore">
<SystemProperty name="jetty.home" default="." />/resources/<your.keystore.filename>
</Set>
<Set name="trustPassword"><your.trust.password></Set>
</New>
</Arg>
</Call>
Restart your web server after making these changes.
Add root CA certificate to your browser
The final step is to add the root CA certificate to all web browsers. The following sections describe this process for the two most popular browsers, Firefox and Internet Explorer.
Firefox
Start Firefox.
Navigate to Tools > Options > Advanced > Encryption > View Certificates > Authorities.
Click Import.
Navigate to your Root Certificate, and then click Open.
Select Trust this CA to identify web sites.
Click OK.
Internet Explorer
Start Internet Explorer.
Navigate to Tools > Internet Options > Content > Certificates.
Click Import.
On the Certificate Import Wizard, click Next.
Click Browse and navigate to your Root Certificate, and then click Open.
Click Next.
Select Automatically select the certificate store based on the type of the certificate.
Click Next, and then click Finish.
When you are prompted to add the certificate to the root store, click Yes.
Renewing your SSL certificate
When your certificate expires, use the following steps to renew it.
To renew your SSL certificate:
Generate a Certificate Signing Request (CSR) from your current keystore.
Send the CSR to your CA.
You will receive a new SSL certificate, and a number of new chain certificates. For example, from VeriSign, you should receive a new SSL Certificate, a new intermediate certificate, and a new root certificate.
Remove the old chain certificates from the current keystore.
For example, for certificates from VeriSign, use the following commands:
keytool-delete-aliasjetty_int-keystorekeystore
keytool-delete-aliasjetty_root-keystorekeystore
Import the new chain certificates.
Import the new SSL certificates.
Additional Jetty configuration information
Premises Web Server Security Vulnerabilities
xMatters Reference
DTN-2283, SUP-3931, SUP-2603, JDN-1228
Originally created by Don Clark
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
A client recently encountered a problem when attempting to start a newly-installed Integration Agent. When the Integration Agent failed to start, the client checked the log files and found the following error message:
ERROR - The script for Integration Service (applications,bmcbppm30) could not be created due to an exception.
com.alarmpoint.integrationagent.exceptions.ScriptCreationException: The script for Integration Service
(applications,bmcbppm30) could not be created due to an exception.
...
Caused by: org.mozilla.javascript.EvaluatorException: Cannot import "Base64" since a property by that name
is already defined. (integrationservices/bmcbppm30/lib/javascript/webservices/wsutil.js#10)
Cause
This issue occurred because theIntegration Agent version 5.1.8 uses Java 1.8, which offersnew functionality, and includesa class called Base64. Previous Integration Agents used earlier versions of Java which did not include this class, and therefore had to explicitly import Base64 from another source.
In Integration Agent 5.1.8, the explicit "import" instruction fails because the class already exists, and this prevents the Integration Agent from starting up.
This problem is most likely to occur after upgrading an existing Integration Agent to version 5.1.8, or when installing a new integration using Integration Agent 5.1.8.
Resolution
If you encounter this issue, use the following steps to resolve it and start your Integration Agent:
Locate the Base64 error message in your Integration Agent log and note the name of the file that is causing the exception.
In the above error message, the problem file isintegrationservices/bmcbppm30/lib/javascript/webservices/wsutil.js
Open the file in a text editor and find the instruction or import command causing the BASE64 class to be loaded.
In the error message above, the line number is appended to the file name: wsutil.js#10
Insert "// " at the beginning of the line to comment out the instruction.
For example, replace:
importClass(Packages.org.apache.commons.codec.binary.Base64);
with:
// importClass(Packages.org.apache.commons.codec.binary.Base64);
Save the file, and then restart the Integration Agent.
If Your Integration Script Requires Base64Encode
If your find that your integration script requires the Base64Encode function, you can use JRE 8'sbuilt-in Base64 functionality instead of the version imported from Apache. Recall that JRE 8 is included with IA 5.1.8 and later only.
Using the HP NNMi integration as an example, you can replace
this.authorization = this.base64Encode(NNMI_USER+":"+NNMI_PASSWORD);
with
var nnmiCred = NNMI_USER + ":" + NNMI_PASSWORD;
this.authorization = "" + java.util.Base64.getEncoder().encodeToString( new java.lang.String(nnmiCred).getBytes() );
xMatters internal reference: DOC-5878, SUP-14304
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377 )
This article describes how web service requests from the Integration Agent to a hosted xMatters instance may fail due to a 202 response, and provides instructions for updating integration scripts to handle 202 responses.
Issue details
Many integration scripts and Integration Agent utility scripts do not include logic to handle 202 responses. This can cause the script to stop processing the current notification request, even though the request has been accepted by xMatters and notifications can be expected to succeed.
Requests that may fail due to this condition include:
submitApxml
getUserDetails
getEvents
setEventStatus
A 202 response does not signify that the request has failed; rather, it indicates the request content has been accepted for processing, and processing is not yet complete. Incorrect handing of the 202 response by the integration service can cause the following undesirable consequences:
Subsequent processing of the notification request (such as an annotation being sent to the management system) may be abandoned.
The client may be concerned about resulting error messages in the Integration Agent log files.
Updating integration scripts to handle 202 responses
In most integrations, the code that handles sending requests to xMatters is encapsulated in the XMRESTAPI functions. The exception is the submitApxml function, which is typically found in the main integration scripts rather than the XMRESTAPI files.
Therefore, you will usually need to modify the XMRESTAPI script file and the main integration files to resolve the problem.
To update the XMRESTAPI functions to handle 202 responses:
Back up <IAHome>\lib\integrationservices\xmrestapi.js and then open it with a text editor.
Search for:
XMRESTAPI.RESPONSE_SUCCESS = "200";
On a separate line, append:
XMRESTAPI.RESPONSE_SUCCESS_ACCEPTED = "202";
Next, search for:
if ( response.status !== undefined && response.status != XMRESTAPI.RESPONSE_SUCCESS ) {
Replace it with:
if ( ( response.status !== undefined && response.status != XMRESTAPI.RESPONSE_SUCCESS ) && ( response.status != XMRESTAPI.RESPONSE_SUCCESS_ACCEPTED ) )
To update the main integration scripts to handle 202 responses:
Search the integration script files for any instance of "XMRESTAPI.RESPONSE_SUCCESS".
Typically, the code will be very similar to:
if (response.status == XMRESTAPI.RESPONSE_SUCCESS) {
Back up any such script files and open them with a text editor.
Append "|| (response.status == XMRESTAPI.RESPONSE_SUCCESS_ACCEPTED)"
For example, replace:
if (response.status == XMRESTAPI.RESPONSE_SUCCESS) {
with:
if ( (response.status == XMRESTAPI.RESPONSE_SUCCESS) || (response.status == XMRESTAPI.RESPONSE_SUCCESS_ACCEPTED) ) {
Search for and update any other instances of the above code (typically there will be just one).
Save the files and restart the Integration Agent.
xMatters internal reference:
DTN-6369 Originally by Jeremy Brown
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377 )
Introduction
Most xMatters integrations are written to be single-threaded: they process one incident injection at a time, and subsequent incidents remain in the integration's inbound queue until they can be processed.
However, if increased throughput is required, the integration agent has the ability to process more than one incident at a time - this capability is referred as "concurrent processing," or "multi-threading". A client might need to exploit this capability if the number of incident injections from the management system is very large, or if incident processing involves time-consuming operations such as requests to external web services.
For an integration to be multi-threaded, the integration scripts must be configured appropriately. This article describes the process of adding multi-threading capability to an existing integration service.
Prerequisites
xMatters integration agent 4.1 patch 5 or higher.
"Del" injections from the management system must be processed by the same integration service as "Add" injections. If the integration uses the "DEL" integration service to process "Del" injections, then the integration must be updated according to the instructions in https://support.xmatters.com/hc/en-us/articles/203632939.
Note: If the management system does not issue "Del" messages to the integration, then the integration is ready to be configured for multi-threading.
Configuring the integration service
Assign a value to the apia_process_group
The "apia_process_group" token is used to identify messages which must be processed sequentially. For example, many management systems send a Del message to the integration agent when a ticket is reassigned or re-prioritized, and then an Add is sent immediately afterward:
The Del message is used to tell xMatters to terminate any pending notifications for the specified incident.
The Add message causes a new round of notifications to be initiated.
Clearly, these two messages must be delivered to xMatters in the correct order, or the Del will cause the new notifications to be immediately terminated.
Unrelated incident messages can be safely sent to xMatters without the need to preserve order. This behavior is preferredin a multi-threaded integration because it allows messages to be delivered to xMatters without having to wait for earlier messages which might have been blocked during some processing step. To enforce this behavior, the apia_process_group is used to associate related messages with each other so they will be processed in the correct order.
The apia_process_group is also used to distinguish unrelated messages so that they can be processed concurrently.
To assign an appropriate value to the apia_process_group token:
Configure the management system to assign the incident ID to the apia_process_group token. The details of this step will vary according to the specific management system and the way it sends incidents to the Integration Agent. If the management system uses apclient.bin to send incidents to the IA:
In the command string that invokes apclient.bin, find the parameter that represents the incident ID and duplicate it. For example, in the BMC BPPM integration, a script invokes apclient.bin using the \%mc_ueid\% environment variable to represent the incident ID:
"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm20 \%1 \%mc_ueid\% "false" "\%severity\%" ...
Modify this script to include a second instance of the incident ID as follows:
"C:\Program Files\xmatters\Integrationagent\bin\apclient.bin" --map-data bmcbppm20 \%1 \%mc_ueid\% \%mc_ueid\% "false" "\%severity\%" ...
Modify the integration's XML configuration script to associate the new parameter with the apia_process_group token. For example, in the BPPM integration's configuration file "bmcbppm.xml" replace:
<parameter type="string">recipients</parameter>
<parameter type="string">incident_id</parameter>
<parameter type="string">fyi</parameter>
with:
<parameter type="string">recipients</parameter>
<parameter type="string">incident_id</parameter>
<parameter type="string">apia_process_group</parameter>
<parameter type="string">fyi</parameter>
To determine how to modify your integration, refer to the specific xMatters integration guide to find out how your management system indicates the incident ID, and then add a second instance of this parameter to the apclient invocation and the integration configuration file as shown in the example above.
If your integration uses a mechanism other than apclient.bin to send incidents to the Integration Agent, then the Integration Agent's inbound queues are probably not being used, and so multiple threads cannot be managed by the apia_process_group token.
Next, search your integration's Javascript files for "apia_process_group" and delete any code that overwrites this token.
To summarize, the desired outcomes are:
The management system will send all adds and dels to the same integration service (i.e., dels will not be sent to the "del" integration service).
All injections (whether "add" or "del") from the management system will have an apia_process_group token that contains the incident_id before the injection reaches the Integration Agent's inbound queue.
Ensure that all events have the same priority
If an Add and a Del injection have been assigned different priorities by the management system, then they will not be sent to the same outbound queue and may reach xMatters out of order.
To prevent this from happening, open the integration's XML configuration file and add a constant that will overwrite any priority tokens to ensure that all injections are processed in the correct order.
If the XML configuration file does not already have a "constants" section, add one with the following contents:
<constants>
<constant name="apia_priority" type="string" overwrite="true">normal</constant>
</constants>
If the file already has a constants section, then just add the above "apia_priority" line to it.
Refer to the xMatters Integration Agent documentationfor further information about the apia_priority token.
Configure the the concurrency setting for the integration service
The concurrency setting is configured in the integration's XML definition file, located within the IntegrationServices folder. For example, in the CA Spectrum integration, the file is located in the integrationservices\caspectrum folder and is called caspectrum.xml
The concurrency setting is governed by code similar to the following:
<concurrency>
<normal-priority-thread-count>3</normal-priority-thread-count>
<high-priority-thread-count>3</high-priority-thread-count>
</concurrency>
This code can be added, if it isn't already present. Refer to the Ping integration definition file for an example.
In this case, the integration service will be able to process three separate incident injections concurrently (assuming that all injections have the default "normal" apia_priority). The "High" apia_priority queue, despite its name, does not provide any preferential treatment; it serves only to segregate particular messages from other messages. This segregation is to be avoided if Add and Del messages are to be processed in the correct order.
Test and tune
Once the integration service has been configured as described above, restart the integration agent and test basic functionality (for example, ensure that notifications are still being delivered by xMatters.)
Validation and tuning can be done rigorously or informally. For example, if the integration performs a time-consuming operation such as making a web service request before forwarding the incident information to xMatters, and the operation is known to take an average of five seconds, then you can inject three incidents to the integration simultaneously and then check the messages in the integration agent log file (or the xMatters Event Log.) If the incidents were forwarded to xMatters at roughly the same time, then concurrency is working correctly; if they are delivered five seconds apart, then some troubleshooting may be required.
To tune the concurrency setting informally:
Use the command "iadmin get-status" while the management system is sending incidents to the integration agent. In the output from the iadmin script, look for the message "Normal priority inbound APXML queue size:" (note that there is one such message for each integration service, so be sure you are looking at the correct one.) If the number of messages in the inbound queue becomes large, increase the concurrency setting.
To tune the concurrency setting more rigorously:
Enable logging of the individual queues by adding the following lines to <IAHOME>\conf\log4j.xml:
<logger name="org.apache.activemq.broker">
<level value="DEBUG"/>
</logger>
You will now be able to monitor the size of the inbound queues in the integration agent log file as messages are accepted from the management system and processed by the integration agent. Ideally, there will always be an available thread waiting to process a message when it is added, so the number of messages in the inbound queue should always be very small. Occasional spikes may result when the management system injects a large number of incidents, but the queued messages should be processed promptly. If the number of messages in the queue continues to grow over seconds or minutes, then the concurrency setting should be increased.
You can isolate the relevant messages by opening a command window in the <IAHOME>\log folder and issuing a command similar to the following (replace "test.test" with the name of your integration service):
find "inbound.test.test.normal" alarmpoint.txt | find "[BrokerService[localhost] Task]"
By monitoring these messages over a suitable period, you can gain an accurate understanding of the integration agent's ability to process injected incidents in a timely manner, and by updating the concurrency setting you can adjust the configuration if necessary.
Filtering and screening
If the integration's inbound queues remain large even when the integration is multi-threaded, then individual messages may be taking a long time to process. This commonly occurs when the integration must communicate with the management system via web service request.
An obvious source of incoming messages is the management system. Management systems sometimes inject large volumes of spurious messages (such as those which do not result in desirable notifications). These messages can generally be suppressed by the integration agent's Portable Filtering and Suppression module. If it's feasible, prevent spurious messages from reaching the integration agent by configuring the management system appropriately.
A less obvious source of incoming messages is the xMatters server. For example, if xMatters is configured to send an annotation to the management system whenever a notification is delivered, then the annotations will be handled by the same incoming queues as the injections from the management system, and each annotation will consume a thread during processing. Annotations are commonly delivered to the management system via web service request, and the corresponding thread will be unavailable for other purposes until the management system provides a response. Therefore, it is advisable to configure xMatters to not send delivery annotations to the management system, since deliveries can be tracked via the xMatters user interface, if necessary.
xMatters reference
DOC-4179 Originally created by Jeremy Brown
View ArticleMarked as obsolete (ref: https://xmatters.atlassian.net/browse/DOC-10377)
We've come across a couple of minor but persistent issues that can occur when using the xMatters for iOS mobile app on devices that have been updated to use iOS 11. Both of these issues are related to situations where the xMatters app is running in the background.
Repeat alerts not working
Users should be able to configure the xMatters app to play repeating alerts if a push notification arrives, and they don't answer it right away. However, some mobile app users have reported that if the app is running in the background and they don't respond to a push notification, they don't get the repeating alert.
With iOS 11, we've noticed that it's possible for repeating alerts to stop firing up to 50\% of the time. We found that the root cause of this issue is related to the way that iOS 11 allocates background processing time. There's no guarantee that the app will get the processing time it needs, and there's no accessible way for us to sort out the throttling heuristic that Apple uses to determine the allocation.
We're currently investigating this issue and looking into possible solutions.
Badge updates not appearing
Similar to the issue described above, some users have reported that the badge count displayed on the app is not being updated correctly if a notification arrives while the app is running in the background.
We believe this is also related to the allocation of background processing time, and we're currently investigating possible solutions.
Notifications not showing up
Several users have reported that when a message comes in, the app will chirp and display the push notification (the banner) only briefly. On other iOS versions, the push notification would remain visible and "tappable".
The fix for this is to open the iOS Notifications settings, and set the notifications for the xMatters app to "Persistent". (See the discussion in the comments below for more.)
Need to know more?
If you'd like to be automatically notified about any updates to these issues, head to the top of this section, click the Followdrop-down list beside the title and then select New articles and comments; we'll keep you posted whenever we have more information we can share.
xMatters internal reference:
APO-13393, BUG-11050
View ArticleThe information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
community
Welcome to a new xMatters! As part of the upcoming Hogan's Alley Quarterly Release, we're rolling out an improved navigation experience for our web application.
Here's when you can expect to see the new nav hitting your xMatters instance:
Non-production environment access: February 18
Production environment access: March 3
But you don't have to wait until then to see the updates! This article gives you a sneak preview of some of the improvements that are coming your way:
Left sidebar navigation
Right off the bat, you'll notice we've moved that main navigation to a collapsible menu on the left side of the screen. In its most streamlined view, the menu displays icons for the main areas of the product. You can move your pointer over the icons to view their labels, or if you'd prefer you can opt to expand the menu to include labels:
Getting around
To view the contents of a menu item, just click on it. You can then click on an item in the list to be redirected, or you can click the 'x' icon to remain on the current screen:
Admin controls
If you have administrative privileges in xMatters, you'll notice the Admin settings are now located under the gear icon at the bottom of the sidebar menu. This separates these settings from the other parts of the menu that you're more likely to use on a day-to-day basis.
More screen real estate
The main benefit of the new navigation design - besides its slick new look and feel - is that it frees up a lot of extra screen space, giving you more room to work with your data. With all the new data columns that we've been adding to the All Events report, we think you're going to appreciate the extra elbow room!
Active alert badges
For easy access to your Inbox, we've added a link to the top right corner of the screen, next to the user drop-down menu. We're currently in the process of adding a badge that lets you know when you've got active alerts in your Inbox - this includes a handy counter that lets you see at-a-glance if new alerts come in or others are resolved.
Let us know what you think!
We love getting your feedback, so if you have thoughts on our new design or ideas for future usability improvements, drop us a line in the !
View ArticleStarting in version 3.21 of the xMatters iOS app ( available now! ), you'll be able to view and add comments to an event directly from the mobile app.
Comments added to an event, either when a user responds to a notification or when they comment directly on the Tracking report, can enrich an event with additional details or context. For example, users can use comments to give status updates during a major incident, or during an emergency they can provide information on their current situation and safety.
For incident managers, the ability to manage comments from the mobile app means they can stay informed and take follow-up actions when they’re on-the-move without having to periodically log into the web user interface on a laptop to view users’ comments on an event!
View comments
The Recent Events screen in the mobile app displays a comment icon and count of the number of comments that have been added to an event. For events with comments, you’ll also see this information in a new Comments section when you drill-through to the event.
App Store.
Tap the Comments section on the Event report to read the comments added to the event. The most-recent comment is listed at the bottom of the screen, and you can scroll back to view earlier comments.
Add a comment
To add a comment to the event, tap into the Add Comment box at the bottom of the screen and type away! You can comment up to 1000 characters each time.
Availability
Eager to get your hands on this feature? Here's when it'll be available for iOS and Android:
iOS: Version 3.21 - available now from the
Android: Coming soon - exact date to be determined.
View ArticleIf you've everusedtables to designthelayoutof anxMatters email notification, you may have become frustrated with the padding, lines, and background colors that are automatically applied to each cell. This article discusses howto remove the default formatting soyou can use tables to create rows and columns in your email design.
This technique involves using CSS specificity rules tocreate new styles thatrank higher thanthe default xMatters styles. You can use this technique not only to override default table styles, but also other default styles that are applied to your email notifications.
The following image shows an emailthatusestablestocreatetwo rows, wherethe bottom row is split into three columns. When thistable is styledwith the default table styles, you can see that padding and borders are applied toeach cell(which might not be something you want in your design):
this article
Using CSS specificity rules to override the default styling, you can style the table in the above message withoutpadding or borders so it lookslike this instead:
Where are thetable styles defined?
The tablestylesare defined in a Cascading Style Sheet (CSS) that is appliedto all of the email messages in your company.These styles are also used by the email message designer'sWYSIWYG editor to formatcontent.Thetable linesmake it possible to see and work with tables in the WYSIWYG editor.
The examples in this article show you how tooverride thedefault xMatters table styles, however, if your company uses a customized stylesheet you may need to adjust some of the values.
Overriding the styles using CSS specificity rules
To override the default table styles, wecreate new table styles that have a higher specificity than the default table styles and apply the new table styles to our messages.Email readers then use the new styles to format the tables.
CSS specificityrules statethat when there are severaldifferentstyles defined for a particular element, the most specificstyle is applied. One CSS rule could be "textis red", but a more specific CSS rule would be "text within a table cell is blue". In this case, text within table cells would be styled blue because that ruleis more specific and wins out over the "text is red" rule.
Determiningthespecificity of a stylecan bekind of complicated, because someHTML components are weighted higherthan others.Fortunately, there are plenty of resources on the internet that can help you understand CSS specificity, such this article. The thing that you need to remember is to create a new style that has higher specificitythan the default style, and then apply the new style to your element.
Why you MUST usehigher specificityto overridestyles
You may be tempted to overwritetable styles (table, th, td) directly without usinghigher specificity as describedin this example. This may initially appear to work when you view the messageinthe email message editor and some email readers. However, it creates an ambiguous situation where the email containsconflicting styles of equal importance and forces the email reader to choose which oneto use.
When presented with two styles of equal importance, most email readers choose the lateststyle (the one you added to youremail message), but other email readers choose the first style (the one that isin thedefault stylesheet).
It's important to previewyour emails in a variety of email readers before you send it out to ensure that yourstyling doesn't rely on the quirks of a particular email reader(don't forget to test your email on Microsoft Outlook on Windows.)
Example
For this example we're going to keep it simple and override the table styles bycreating new tablestyles thathave slightly higher specificitythan the default table styles.
To do this, we'll wrap the email message content in a <div> tag that is styled withthe "plainTable" class. We'll then create new styles that apply to tables that are also located within an element styledwith the "plainTable" class.Email message readers will apply our table styles instead of the default styles because they are more specific.
We'll workdirectly with the email's underlying HTML to create and apply these styles.
Step 1: Openthe email in source view
Navigate to the communication plan and form that contains your email message, and open the email message editor. Toggle theShow Sourcebutton to view the email's underlying HTML in source view.
Tip: If you're working with an existing email message, the HTML in the source view might notbe that easy to read if it doesn't contain whitespace. In this case, copy it into an HTML editorsuch as the one at http://codepen.io, format it, and paste it back in to the editor.
Step 2: Add atable to the email message content
If you're not already inthe source view of the email message designer, click Show Source. You can then paste the following table into the source view. (You may want to return to the WYSYIWYG view to see how this email looks with the default styles.)
<center>
<div>
<table class="" id="table-5">
<tbody>
<tr>
<td tabindex="0">
<img src="http://placehold.it/600x300">
</td>
</tr>
<tr>
<td tabindex="0">
<table class="" id="table-6">
<tbody>
<tr>
<td tabindex="0">
<table class="" id="table-7">
<tbody>
<tr>
<td tabindex="0">
<img src="http://placehold.it/200">
</td>
</tr>
</tbody>
</table>
</td>
<td tabindex="0">
<table class="" id="table-8">
<tbody>
<tr>
<td tabindex="0">
<img src="http://placehold.it/200">
</td>
</tr>
</tbody>
</table>
</td>
<td tabindex="0">
<table class="" id="table-9">
<tbody>
<tr>
<td tabindex="0">
<img src="http://placehold.it/200">
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<div>
<br>
</div>
</center>
The WYSIWYG view of the email message designer shows what this table looks like whenstyled with the default table styles.
Step 3: Create a newtable style
Next, we'llcreate a new style that formats tables withoutpadding, background colors, or borders. For this example we'll create a style fortables that are also in the "plainTable" class.
Paste the following code to the top of the email message in source view, and then toggle the Show Source button to verifythat these styles are not yet applied tothe table.
<style>
.plainTable table
{
border-spacing: 0px;
}
.plainTable table th
{
table th
{
background: white;
font-weight: normal;
}
}
.plainTable table th,
.plainTable table td
{
border: none;
padding: 0px;
margin: 0px;
}
.plainTable table
{
border: 0px;
margin: 0 0 0 0;
border-radius: 0px;
border-bottom:none;
text-align: center;
font-size: inherit;
}
</style>
Step 4:Apply the styleto the table
The styles are not yet applied to the tablebecause the table is not styled with the "plainTable" class. The easiest way to apply the "plainTable"class to thetable is to apply it to the entire email message body.
To apply this class to the entireemail message body, wrap the HTML content in the tag <div class="plainTable"></div>.
Step 5: Preview themessage
You can now toggle the Show Source button to see thatthe new styles applyto the table.
Step 6: Test themessage
Before you put your message into production, send a test notification and view it in a variety of email readers on various platforms, focusing on the ones that people in your company use the most. In particular, be sure to test your email on Microsoft Outlook for Windows and web clients such as the Gmail web client.
Be aware that advanced CSS features may not be supported by allemailreaders. For a overviewof which features are supported on different browsers,see.
Download this example
If you'd rather just see the differences in action, or want to work directly with this example, you can download the communication plan attached to this article and import it into your xMatters instance.
View ArticleDeduplication allows an integration service to prevent "event storms," which can occur when a management system sends multiple notification requests pertaining to a single incident. By detecting and blocking duplicate notification requests, the Integration Agent can prevent event storms from affecting xMatters users.
The Integration Agent's deduplicator has three components:
A "filter" which specifies the conditions that will cause incident requests to be discarded.
Script instructions that match incident requests to the filter conditions.
Deduplication capability in the Integration Agent, which is invoked by the script instructions.
The filter and script instructions must be understood and used properly for the deduplicator to perform correctly.
Deduplication Filters
Deduplication filters are specified in <IAHome>\conf\deduplicator-filter.xml. In order for an integration service to perform deduplication, that integration service must specify a filter that exists in deduplicator-filter.xml. The filter will typically have the same name as the integration service. For example, the "sample-plan" integration might have a filter that is specified as follows:
<<filter name="sample-plan">
<predicates>
<predicate>building</predicate>
<predicate>city</predicate>
</predicates>
<suppression_period>60</suppression_period>
<window_size>10</window_size>
</filter>
In addition to its name, a filter has three components:
Predicates: Each predicate must correspond to a property* that is present in the data that will be forwarded to xMatters by the Integration Agent. Based on the values of these predicates, each message will be categorized as "unique" or "duplicate" by the Integration Agent. See below for examples and explanation.
Suppression Period: After each unique notification request is received by the Integration Agent, a record of its predicate values isstored in memory for this number of seconds, and any subsequent notification request is discarded if all its predicates match the stored values. When the suppression period expires for those predicate values, the Integration Agent "forgets" that combination of predicates.
Window Size: If this number of unique notification requests is received within the suppression period, then the oldest one is "forgotten", meaning that a second such request will be allowed (even if the suppression period has not yet expired).
*Deduplication works with newer JSON-based integrations as well as the older APXML based ones, but behind the scenes the event properties are translated to APXML for the purposes of deduplication.
Here are a couple of helpful examples using the "sample-plan" integration service and filter:
Example 1: Requests are deduplicated until the suppression period expires
At 10:10:10, the sample-plan integration service receives a notification request with properties:
"building": "Building A",
"city": "Victoria"
This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:10:10", "Building A", "Victoria" in memory.
At 10:10:11, a second request is received that has the same three properties. The second request is suppressed.
At 10:11:10, the 60-second suppression period expires and the "10:10:10", "Building A", "Victoria" record is purged from the deduplicator's memory.
At 10:11:11, another request is received with tokens:
"building": "Building A",
"city": "Victoria"
This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:11:11", "Building A", "Victoria" in memory.
Example 2: Requests are deduplicated until the window size is exceeded
At 10:10:10, the sample-plan integration service receives a notification request with properties:
"building": "Building A",
"city": "Victoria"
This request is processed, and the deduplicator stores "10:10:10", "Building A", "Victoria" in memory.
At 10:10:11, nine more requests are received, all of which are different from each other as well as from the original request. All the requests are processed and a time-stamped record of each is stored in memory.
At 10:10:12, one more unique request is received. A time-stamped record of it is stored in memory. Since eleven unique records have now been received, the window size is exceeded, causing the first record ("10:10:10", "Building A", "Victoria") to be purged from the deduplicator's memory.
At 10:10:13, the sample-plan integration service receives another notification request with properties:
"building": "Building A",
"city": "Victoria"
This request is processed, the Event request is delivered to xMatters, and the deduplicator stores "10:10:13", "Building A", "Victoria" in memory.
Script Instructions
The following script instructions are required for the deduplicator to perform correctly:
DEDUPLICATION_FILTER_NAME = "sample-plan";
This instruction creates a constant whose value matches the name of the deduplication filter.
It is typically found in the integration's configuration.js script file (if there is one); otherwise it should be at the top of the main integration script (e.g., sample-plan.js) immediately after the "importPackage", "importClass", and "load" instructions.
The constant is used by the utility functions in the XMUtil script package that ships with the Integration Agent.
load("lib/integrationservices/javascript/event.js");
This instruction is typically found at the top of an integration service's main script, e.g., sample-plan.js
event.js contains instructions to load the other utility script files that ship with the Integration Agent, including xmutil.js
If you don't want your integration to load event.js, you can load xmutil.js directly (i.e., by replacing "event.js" with "xmutil.js" in the above instruction). However, due to dependencies, if you load xmutil.js directly, you will have to load log.js first. See the code in the "Example" section below for details.
if (XMUtil.deduplicator.isDuplicate(event.properties)) {
XMUtil.deduplicate(event.properties);
return;
}
After the integration script has prepared an Event request to be sent to xMatters, the above code determines whether the request should be deduplicated or delivered.
The argument can be either an APXML message (typically called "apxml"), or the properties component of an Event object (e.g., "event.properties") as shown above. If it receives an event object, the xmutil deduplicator will convert it internally to APXML.
XMUtil.deduplicator.incrementCount(event.properties);
If the notification data is an APXML message, then this code should immediately precede the instruction that sends the notification request to xMatters.
If the notification data is an event object, then this code should immediately follow the instruction that sends the notification request to xMatters.
The reason for this distinction is:
Event objects are sent from the Integration Agent to xMatters via an HTTP POST (typically "XMIO.post"), which may fail and throw an exception. In this case, the Integration Agent will retry the HTTP POST, but if the XMUtil.deduplicator.incrementCount instruction has already been executed, then the retry will be suppressed by the deduplicator.
APXML messages are sent to xMatters via the "return" statement at the end of an apia_input() or apia_http() function. The XMUtil.deduplicator.incrementCount instruction will never be executed if it follows the "return" statement.
Example: Adding deduplication to the generic integration
As of Integration Agent 5.2.0, all of the included sample integrations do include deduplication. The "sample-plan" integration provides a clear example with comments, in sample-plan.js.
The "generic" integration script provides an uncomplicated example that does not include deduplication. To add deduplication to the generic integration, make the following changes in generic.js:
Near the top of the file, replace:
importClass(Packages.com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider);
/* Created by Adam Dahlquist, [email protected]
with:
importClass(Packages.com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider);
// Declare the name of the deduplicator filter
DEDUPLICATION_FILTER_NAME = "generic";
// Load the Integration Agent script packages that include the deduplicator code
load("lib/integrationservices/javascript/log.js");
load("lib/integrationservices/javascript/xmutil.js");
load("lib/integrationservices/javascript/apxml.js");
load("lib/integrationservices/javascript/xmio.js");
/* Created by Adam Dahlquist, [email protected]
Replace the apia_input function with the following code:
function apia_input(apxml)
{
IALOG.info("Apia_input received the following APXML message: \n" + apxml);
// Event processing code goes here, ie, before the deduplication code
// Check whether the processed event message is a duplicate, and discard it if so
if (XMUtil.deduplicator.isDuplicate(apxml)) {
XMUtil.deduplicate(apxml);
return;
}
// Increment the deduplicator. Note that this instruction goes AFTER the call to XMIO.post in REST-based integrations.
XMUtil.deduplicator.incrementCount(apxml);
// Send the message to xMatters and terminate apia_input
return apxml;
}
You will also need to add a filter to <IAHome>\conf\deduplicator-filter.xml. You can use the example filter from this KBA, exactly as it appears above.
Save the files and restart the Integration Agent.
A couple of things to note:
If the filter includes a predicate that does not match any property in the notification request, then that notification request will be ignored by the deduplicator and will never be suppressed.
Some early integration scripts used the deduplicator instructions incorrectly, so if the above information contradicts the released code, then the released code is probably at fault.
xMatters reference: DTN-6286 Originally by Jeremy Brown
View ArticleWe've had some questions recently about people getting Invalid_Response emails from xMatters.We know you get enough emails, and we want to help make sure you don't get any from us that you can't act on.
These Invalid_Response emails are sent when xMatters receives an email response to a notification that it already has a response for. There are a few scenarios that might cause this, each with its own unique solution.
Your security team is being (wisely) cautious
It's possible that your security team has put in place software to try to protect you from malicious links. This is good. But it can also muck up the URL response in the response email, adding entries to the URL which cause multiple responses to be sent back to xMatters.xMatters generally accepts the first response, but sends that Invalid_Response email for the additionalresponses.
Ask your security team to whitelist emails from xMatters so that these additions to the response URL are not added and xMatters just receives the one response.
You've already responded to the notification
We know you're keen to resolve any issues that cross your path, but you might be responding to the same event from different devices. If you don't have any delays set up between your devices in xMatters, they ALL receive the same notification at the same time.
Keep being you, and keep responding to events. But only respond to the notification on one device. Better yet, set up delays between your devices in xMatters that way we'll only notify you on more than one device if you don't respond on the first one.
One of your teammates is already on it
It could be that one of your teammates has already responded to the event in a way that ends the event escalation, meaning you don't need to respond. If you do, you get that Invalid_Response email.
There are a couple of things you can try to stop this. First, add delays between the people in your group. This gives the first person on the list a chance to respond before xMatters notifies you. If that doesn't work, check how the Response Options/Actions of a Communication Plan are set up it might be that you need to tweak these to let your team respond in a way that does not cause others to get Invalid_Response emails.
View ArticleQuestion
I was able to login previously but now I get an error.
xMatters Customer Support
Under Single Sign On URL I have something like:
https://ident.mycompany.org/idpssoinit?metaAlias=/company/idp&RelayState=12345&spEntityID=https\%3A\%2F\%2Fmycompany.hosted.xmatters.com
Environment
All versions of xMatters On-Demand using SAML authentication.
Answer
Your SAML configuration appears to be correct, however you are getting a message that indicate a bad URL is being submitted.
Recently we enabled a feature to support the RelayState parameter in a login URL. We did not support RelayState previously so up until this change we were ignoring the parameter.
If you are encountering this issue, you may be sending us an invalid RelayState. RelayState can be configuredin the SAML configuration page in the xMatters Web UI or is configured by your Identity Provider (IDP)
If possible, please try the following if you are seeing issues:
Remove the RelayState parameter from the SAML configuration
Try setting the relay state to app.do (for example - change RelayState=12345 in the example above to RelayState=app.do
If your IDP requires a RelayState:
See if it is possible to not send the RelayState when connecting to xMatters (this configuration should be possible on the IDP side). Since this may differ between IDP providers please check your documentation on steps to control the RelayState.
Set ReadyState toRelayState=app.do
If you still require help please contact
View ArticleIntroduction
This knowledge base article covers how you can log detailed, low-level messages about HTTP requests initiated by the Integration Agent.
The Integration Agent has well-documented capabilities for logging messages from high-level entities such as the Integration Service scripts, but some debugging tasks require information from lower-level entities, such as the HTTPClient components that handle network communication between the Integration Agent and other devices. For example, the Integration Agent might log error messages when sending REST or SOAP requests to external devices, but these messages won't provide enough detail to allow you to diagnose the problem.
Investigating this type of problem is often performed using external tools such as Wireshark, but you can get comparable levels of troubleshooting information from the Integration Agent itself, without installing external tools.
Configuring the Integration Agent to record wire-level log messages
To configure the Integration Agent to record detailed, low-level messages about HTTP requests initiated by the Integration Agent (and the corresponding responses):
Add the following lines to <IAHome>\conf\log4j.xml at a suitable spot (for example, immediately prior to "<!-- All Integration Services -->"):
<logger name="org.apache.http.wire">
<level value="DEBUG"/>
</logger>
Save the file and restart the Integration Agent.
Viewing the wire-level messages
The messages are written to the Integration Agent log file. They provide the same process and thread information as the Integration Service script that initiated the web request, but it should be fairly obvious in the log file that these messages are network-specific. For example:
2017/08/25 15:37:32.006 -0700 PDT [applications|sample-plan-1]
DEBUG - >> "POST /reapi/2015-04-01/forms/4069a692-d848-497f-a267-2f84655fd076/triggers HTTP/1.1[\r][\n]"
In this example, the ">>" prefix indicates that this message contains information about an outgoing transmission from the Integration Agent. Messages with a "<<" prefix contain information about a response coming back to the Integration Agent.
Information included in the wire-level messages
In addition to the familiar details such as those shown in the message above, the wire-level messages include information about header, cookies and cookie management, and so on. In many ways, the information is the same as you would get from sending the request using cURL or POSTMan, but the wire messages provide information that is specific to the Integration Agent.
For example, the Integration Agent does not configure its HTTPClient component to perform preemptive authentication. This means if you try to diagnose Integration Agent connection problems using cURL, you won't get accurate information unless you use exactly the right flags and headers. However, the Integration Agent's wire messages provide this information without you needing to know all of the low-level details of the Integration Agent's components and their configuration.
Disabling wire-level log messages
You can easily disable wire-level logging. Simply comment out the lines you previously added to <IAHome>\conf\log4j.xml. For example:
Replace:
<logger name="org.apache.http.wire">
<level value="DEBUG"/>
</logger>
with
<!-- Wire-level logging - causes logfile bloat if enabled while not needed
<logger name="org.apache.http.wire">
<level value="DEBUG"/>
</logger>
-->
Save the file and restart the Integration Agent.
xMatters internal reference:
DOC-7328 Originally by Jeremy Brown
View ArticleThis article describes the process of configuring Single Sign On (SSO) with Azure. While attempting to complete the setup in Azure, some users noticed a few inaccuracies in the Microsoft documentation.
Here's what you'll see in Azure, and what's required to configure SSO properly with xMatters:
In the xMatters OnDemand Domain and URLs section:
Identifier: Remove the final backslash. For example: https://contuoso.au1.xmatters.com.au
Reply: The reply URL is the SAML link from the metadata. For example: https://contoso.au1.xmatters.com.au/sp/SSO.saml2
In the Azure SAML Configuration window:
Single Sign On URL: Use the myapps link.For example: https://myapps.microsoft.com/signin/xMatters\%20OnDemand/4975267c-7215-454c-a714-9e04ad58e7cd
In the xMatters SAML Configuration window, ensure the Identity Provider ID uses https://sts.windows.net rather than the login.microsoft.com URL.
In the Azure configuration, make sure the Sign On URL is NOT populated. It should be left blank.
Note that if you do enter a value for Sign On URL it will result in a looping effect between the Azure login screen and the xmatters login screen.
View ArticleQuestion
What happened to my manually added group members after I ran a data sync with my BMC Remedy integration?
Answer
The BMC Remedy integration's data sync feature tries to make your xMatters On-Demand group look like your Group in BMC Remedy. If there are users in an xMatters group that are not in BMC Remedy, the BMC Remedy data sync tends to delete them.
To keep your group members from being deleted, you can rename your shifts so there is no Default Shift in the xMatters group. The BMC Remedy data sync attempts to update the Default Shift; if the Default Shift does not exist, the new users are added to the roster inBMC Remedy. Once in the roster, you'll need to manually add the new people to the shifts.
View ArticleProblem:
If we delete a group in xMatters, can we save the schedules or shifts so that we can load it back once the group is re-created by ServiceNow?
Environment:
All versions of xMatters On-Demand
Answer:
There is no way to export and re-import shift schedules, it is not recommended to delete groups and have ServiceNow re-create them.
You can have ServiceNow take ownership of an existing group.
Here are the steps to implement this:
Turn off Dynamic Sync from Integration - xMatters > xMatters Configuration > Data Sync
Assign the x_xma_xmatters.xmatters roles to all users and groups that need to owned by xMatters.
Run the group sync script attached. (SNowGroups.rtf)
Run the user sync script attached. (SNowUsers.rtf)
Turn on Dynamic sync from Integration - xMatters > xMatters Configuration > Data Sync
After the change, you should be able to add/remove users and the current shift schedule will not be affected.
Please be sure to consult with your ServiceNow administrator if you are unfamiliar working with scripts in your ServiceNow environment.
View ArticleI've run into this error a handful of times and it confuses me each time because I never write down why it happens and what to do about it. Except this time. The circle has broken, I'm writing about what this error means, and what you can do about it. Or more actually, what you can ask someone to do about it.
tl;dr:Open a support ticket to allow your IA to connect.
The What:
This error occurs (generally) on new Integration Agent installs when you finally have all your settings figured out and configured properly and you have a pile of handfuls of hair as proof you've been working on installing the IA for a good while. Everything else has been pretty obvious, missing file here, invalid password there, but REGISTRATION_ACL_FAILED! is rather not terribly informative but you realize it is preventing anything from going through your IA on to xMatters on Demand.
The Why:
Essentially, this message is saying your IA hasn't been whitelisted by xMatters, so access is being denied. Thisis an added level of security in the security onion at xMatters.
The How:
An xMatters support representative will need to add the Agent ID to the list ofallowed agents. To make this happen:
Get the Agent ID, then
Create an xMatters support request. Provide the Agent ID and ask for it to be whitelisted.
To get the Agent ID:
Open a command window and navigate to <IAHome>/bin
Start the IA using the start_console executable. Starting the agent should take only a few seconds.
Use the `./iadmin get-status` command. This will interrogate the IA, and the output will include the Agent ID. For example:
[user@my-ia-host bin]$ ./iadmin.sh get-status
Version: 5.1.5 r81524
Release date: Mar 20, 2015
Agent started: Aug 30, 2015 6:25:39 PM
Agent ID: my-ia-host/111.11.11.111:8081
Integration Services:
... (lots of content omitted)
[user@my-ia-host bin]$
View ArticleIssue Details
When thexMatters integration agent log includes an AUTHENTICATION_ERROR entry, it's typicallycaused by a webserver user/password mismatch. This article will help you troubleshoot the most common causes of this error.
Specific Error Messages
Here is an example of the error that will be written to the IA log if the username (in <IAHome>\conf\iaconfig.xml) or password is incorrect.
2015-03-05 16:38:25,996 [Heartbeat-1] ERROR - The xMatters Server returned the response AUTHENTICATION_ERROR, which resulted in an exception.
2015-03-05 16:38:26,004 [Heartbeat-1] ERROR - HEARTBEAT_UNEXPECTED_ERROR
This message will be written every time a heartbeat is attempted (every 10 seconds by default.)
If the password file cannot be located, then a message similar to thefollowingwill be written when the integrationfirst starts up:
2015-03-05 16:39:03,841 [WrapperSimpleAppMain] WARN - Unable to decrypt the Web Service user password file located at ./.wspasswds. An empty password will be used, but this will likely cause the Integration Agents heartbeats to be rejected.
com.alarmpoint.integrationagent.exceptions.FileNotFoundException: The path ./.wspasswds was resolved to C:\xMatters\integrationagent-5.1.4\conf\.wspasswds, but does not refer to an existing file.
Subsequently, the "HEARTBEAT_UNEXPECTED_ERROR" messages will be written as described above.
If the Company name does not match a company that exists in xMatters, the messages in the IA log will be similar to the following:
2015-03-05 16:46:18,329 [Heartbeat-1] ERROR - The xMatters Server returned the response COMPANY_DOES_NOT_EXIST, which resulted in an exception.
2015-03-05 16:46:18,330 [Heartbeat-1] ERROR - HEARTBEAT_UNEXPECTED_ERROR
com.alarmpoint.integrationagent.exceptions.HeartbeatRejectedException: The xMatters Server at http://10.2.0.121:8888/api/services/AlarmPointWebService rejected the heartbeat/registration with the following reason: COMPANY_DOES_NOT_EXIST
Note that the company name is not case-sensitive.
Solution
By default, new integration agent installs don't have the .wspasswd file (this is where the password is stored for the IA_User that is referenced in the IAConfig.xml file).
To begin troubleshooting this issue, firstconfirm that the password file exists. Open the IAConfig.xml file and locate the<web-services-auth> section:
<web-services-auth>
<!--
| A string that is trimmed of leading/trailing whitespace and may be left blank.
+-->
<user>IA_User</user>
<!--
| A path (relative to this file), that refers to a file containing the
| encrypted (via iapassword), WS user's password. If the path is blank
| or does not refer to an existing file, an empty string is used for the password.
+-->
<password>
<file>./.wspasswd</file>
</password>
<!--
| A string that is trimmed of leading/trailing whitespace and may be left blank
| to refer to the default company.
+-->
<company>Default Company</company>
</web-services-auth>
The <user> field
The <user> field is the webservice user that the integration agent uses to connect to the xMatters web server. To ensure this value exists and the spelling is exactly the same, log in to the xMatters user interface as an administrator and search for this value under the Users tab, in the "WEB SERVICE USERS" section. (The default username is IA_User.)
Also ensure that this web service user account has the 'Register Integration Agent', 'Submit APXML', and 'Receive APXML' permissions enabled in the xMatters user interface. In some cases we've encountered,security-conscious admins have been removing these rights from the IA_User account and thus caused errors.
The <file> field
The <file> field value isthepath to wheretheencrypted password is stored. Confirm that thereferenced file contains the password created using the iapassword utility (for details, see the Integration Agent Guide ). (Thedefault passwordis ia_complex).
The <company> field
Thelast section is the <company> field, which normally causes the UNKNOWN_COMPANY error to be logged when misconfigured. Ensure that the content of this field matches the company that is present in the web user interface. For example, sometimes the<company>field includes special characters (e.g., the registered trademark symbol); in these cases, copy the Company name from the UI and paste it into the <company> field in theIAConfig.xml file. (The default value is Default Company.)
If you'restill experiencing the AUTHENTICATION_ERROR
If confirming the proper values in the fields above does not resolve the issue, set the log level to DEBUG in the log4j.xml file by ensuringthe following lines are not commented out:
<!-- Heartbeats -->
<logger name="com.alarmpoint.integrationagent.heartbeat">
<level value="DEBUG"/>
</logger>
<!-- APXML Message Exchanger -->
<logger name="com.alarmpoint.integrationagent.messaging">
<level value="DEBUG"/>
</logger>
Start the IA and reproduce the problem, then use the support-zip tool to create an archive of the log and the config files. Create an xMatters support ticket and attach the support-zip archive to it.
xMatters reference: DTN-4082
View ArticleContents
Introduction
How it works
Features and updates
Install the xMatters application
ServiceNow Roles
ServiceNow API user
Configure xMatters
Download the workflow
Set up an integration user
Import the workflow
Configure endpoint
Configure inbound integrations
Determine identifiers
Configure ServiceNow
Complete configuration pages
Add Engage with xMatters records to a related list
Seed users, groups and incident properties
How to use the integration
Validate synchronization
Trigger a notification
Respond to a notification
Engage with xMatters
Troubleshooting
Extended functionality
How it works in detail
Upgrading your integration
Upgrade your ServiceNow integration
Upgrade a customized integration
Upgrade the xMatters app
Previous versions
Updates from previous version
Download resources
The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
Introduction
xMatters for ServiceNow is a direct, cloud-to-cloud integration, leveraging an xMatters workflow to become the voice and interface of an automation engine. When ServiceNow detects something that requires attention, xMatters places phone calls, send emails or notifies your mobile app.
Integration Version
Version 5.4 of the xMatters application within ServiceNow is certified with Kingston, London, Madrid, and New York. See the features and updates in the latest version.
Are you using the ServiceNow steps in Flow Designer?
Flow Designer - our drag-and-drop, visual workflow builder - includes steps to easily add ServiceNow to your incident management workflow. You don't need to do any of the setup below if you simply want to use these in your flows; all you need to get started is an endpoint that targets your ServiceNow instance.
How it works
The integration consists of a pre-configured workflow and the xMatters application within ServiceNow, which includes the following components:
Incident notification: Handles automatic notification about incidents and the sharing of incident information between ServiceNow to xMatters.
Engage with xMatters: enables ad hoc communications, including conference bridges, using xMatters.
Synchronization: synchronizes ServiceNow users, groups, and group members to xMatters.
Each component use different sets of business rules, script includes, and web services to accomplish their tasks. If you want a behind-the-scenes look at how the integration works and how the pieces fit together, see our detailed how it works section.
See The Resolve response and the "Incident Management Best Practice" plug-in for important information on using the Resolve response with new data policies in the plug-in.
Our integration boffins have been hard at work experimenting with ways to extend the functionality and features offered in this integration. For more information, see the Extended functionality section.
Incident Alerts
When an incident occurs in ServiceNow that matches the criteria you define when you configure the integration, it sends a request to xMatters with information on the incident. If the incident is assigned to a user or group that exists in xMatters, xMatters notifies the user or on-call members of the group on their preferred devices. The recipient can take ownership of the incident, resolve it, or immediately escalate it to the next on-call member in a group (if assigned to a group).
In turn, xMatters updates the incident record in ServiceNow with information about the event status, who responded and how, and any comments that were added.
Engage with xMatters
This integration includes the Engage with xMatters feature, which allows users with a particular role to leverage xMatters to call in additional members from other teams to assist with an existing incident.
ServiceNow users can also use the Engage with xMatters feature to get help from or notify users of xMatters that don't exist in ServiceNow. If the group or person you're looking for doesn't exist in ServiceNow, you can still find them using the Engage with xMatters feature so you can get them up to speed directly.
For example, imagine a support team member takes ownership of a ticket, but needs help from the database team to resolve part of the problem. Instead of assigning the ticket to someone else and losing the benefits of incident ownership, the support team member can use the Engage with xMatters feature to contact the on-call person in the database group. When the on-call person in the database group gets the alert from xMatters, they can see who sent them the request. In the meantime, xMatters continues to update the work notes of the ServiceNow incident so the support team member still receives real-time updates on whether someone else has started work on the incident and who that person is.
Alternately, the support team member could use the Engage with xMatters conference bridges to bring multiple groups together to work on a solution. You can even add people to an on-going conference bridge for a ServiceNow incident.
User and group data load
The Batch Load features let you seed xMatters with your ServiceNow users, groups, and configuration items.
Once you've seeded xMatters with your users and groups, you can use dynamic data sync to keep xMatters in tune with ServiceNow. See Sync users, groups and incident properties for more information on the data sync component.
Features and updates in this version
Along with all of the fixes and updates from previous versions, this release (5.4) includes the following enhancements:
Version 5.4.1 certified for the New York release of ServiceNow.
Data sync now properly removes synced groups from xMatters when the group is removed in ServiceNow.
Updated the Engage with xMatters behavior to make sure corresponding events in xMatters are terminated when closing an Engage with xMatters record, and to address an issue that was causing some requests involving Mid-Server to run for a long time.
To upgrade from a previous version of the integration, use our instructions to guide you through the process.
For a list of previous updates, see what we added in previous versions.
The Resolve response and the "Incident Management Best Practice" plug-in
The Kingston release of ServiceNow introduced a new data policy, installed with the "Incident Management Best Practice" plug-in, that requires both the Resolution code and Resolution notes fields to be completed before the incident can be resolved or closed.
Currently, xMatters does not support completing both these fields when you select the Resolve response option. We're working on addressing this issue, but in the meantime, you can use one of the following workarounds:
Remove the "Resolve" response from the Incident Notifications tab in the xMatters Configuration screen in Service Now,
OR
Deactivate the "Make close info mandatory when resolved or closed" data policy installed with the plug-in in ServiceNow (see the ServiceNow documentation for instructions).
Install the xMatters application
To begin, you need to install the free xMatters app in ServiceNow (if you're upgrading, check out our instructions ). The application includes all of the batch loading utilities, configuration pages, business rules, scripts, web services, and other components required for ServiceNow to communicate with xMatters.
To install the xMatters application:
Go to the ServiceNow store at store.servicenow.com, and search for "xMatters".
When you find the app, click Get. You'll be prompted to enter your system credentials to add the app to ServiceNow.
Log into ServiceNow as a system administrator, and navigate to System Applications > Applications.
Click Downloads.
Search for "xMatters", and then click Install.
ServiceNow roles
This integration installs roles into ServiceNow to control interactions between users and integration components.
x_xma_xmatters.xmatters_admin: Only users with this role can modify xMatters components within the ServiceNow interface.
x_xma_xmatters.xmatters_rest: This role must be assigned to a ServiceNow API user to successfully access the REST endpoint and execute the required integration requests. See ServiceNow API User.
x_xma_xmatters.xmatters_engage: Users with this role can use the Engage with xMatters feature. (You should also assign this role to the ServiceNow API user so they can update the Engage with xMatters records.)
x_xma_xmatters.xmatters: Assign this role to any users or groups you want synchronized into xMatters.
Assigning a role to a group automatically assigns it to all users within a group, or you can add the role to an existing role to automatically assign it to any users with that role.
ServiceNow API user
To configure the ServiceNow endpoint in xMatters, you need the username and password for a ServiceNow API user to handle REST requests from xMatters to ServiceNow. To successfully access the REST endpoint and execute the required integration requests, this user must have the x_xma_xmatters.xmatters_rest role (added with the xMatters application). For additional security, select the Web service access only check box on the user's details page.
You should also assign the x_xma_xmatters.xmatters_engage role to this user so they can update the Engage with xMatters records.
For information about adding users and assigning roles in ServiceNow, refer to the ServiceNow documentation.
Configure xMatters
Now that you've completed the first part of the configuration in ServiceNow, you can configure xMatters. After you import the workflow into xMatters and copy some specific IDs and URLs, you'll return to ServiceNow to finish the setup.
Download the workflow
The workflow (.zip file) attached to this article contains pre-configured integrations, forms, properties, and messages specifically designed for ServiceNow. Download the attached .zip file to a location on your local machine, but don't unzip it.
Set up an integration user
This integration requires a user who can authenticate REST requests from ServiceNow when working with events these permissions are provided by the "REST Web Service User" role in xMatters. See Create an integration user for more information on creating an integration user.
Note: Keep the user ID and password of this user handy. You'll need them when configuring other parts of this integration.
Import the workflow
Next, import the ServiceNow workflow you downloaded into xMatters.
In xMatters, click the Workflows tab, and then click Import.
Browse to the .zip file or drag it onto the Import Workflow dialog, and then click Import Workflow.
Once the import finishes, the workflow should be automatically enabled. If it isn't, click the Disabled toggle to enable it.
Click the workflow name to open the Forms tab.
Make sure the dropdown beside the form says Web Service as one of the options. If not, click it and select Enable for Web Service.
For the Incident Alerts form, click the dropdown and select Sender Permissions.
Add the integration user, and then click Save Changes.
Repeat steps 6-7 for the Engage with xMatters and Conference Bridge forms.
Configure the endpoint
The workflow uses an endpoint with the ServiceNow authentication type. After you import the workflow, you need to set the endpoint's URL and authentication credentials (the username and password for your ServiceNow API user ).
To configure the endpoint:
In the Integration Builder, click Edit Endpoints.
Click the ServiceNow endpoint to open its details.
Update the Base URL to that of your ServiceNow system.
Enter the credentials for the ServiceNow API user.
Click Test Authentication to send a request using the username and password to the ServiceNow instance you entered in the Base URL field.
If the page displays a message that the authentication failed, check the credentials and permissions of the ServiceNow API user in ServiceNow and the Base URL you entered.
Click Save Changes, and then close the Endpoints dialog box.
Changing the authentication type
ServiceNow steps in Flow Designer require the ServiceNow authentication type so we recommend leaving the authentication type as is if you want to use ServiceNow steps in your flows. If you want to change the authentication type, you need to create a new endpoint to replace the existing one. The easiest way to do this is to delete the existing endpoint and create a new endpoint using the same name before any steps in Flow Designer are configured to use the endpoint.
Configure inbound integrations
The next step is to update each of the inbound integrations to use basic authentication, and retrieve the URLs, which you can then copy and paste into the appropriate location in ServiceNow.
To configure an inbound integration and retrieve its URL:
Navigate to the Integration Builder tab and expand the list of inbound integrations.
Click the name of an integration to view its details.
In the Select authentication method step, select Basic from the list and click Save.
Scroll down to the bottom of the page, and click Copy beside the field:
Determine identifiers
Some configuration fields in ServiceNow require a UUID (Universal Unique Identifier) for a specific component or property in the workflow. To retrieve the identifier for an element, look for the API icon in the xMatters user interface:
Clicking this button displays the identifier for the component in a dialog box. You can then copy and paste the string to the appropriate location in ServiceNow (you can see which fields need UUIDs in Complete the xMatters Configuration pages.
As an example, the following instructions describe how to retrieve the UUID for a specific response choice; the process is similar for any identifier you want to determine.
To retrieve the identifier for a response choice:
In xMatters, open the ServiceNow workflow.
On the Incident Alerts form, click Edit > Responses.
Beside the Accept response, click the API icon:
Configure ServiceNow
To configure ServiceNow to integrate with xMatters, you need to specify the assignment, synchronization, web services, and other settings on the configuration forms that are installed with the xMatters application. You can also seed your users and groups into xMatters using the data sync feature.
Complete the xMatters Configuration pages
The xMatters application installs four configuration pages into ServiceNow:
Common Configuration: Configures the connection, credentials, and logging level for the communication between xMatters and ServiceNow.
Incident Notifications: Enables or disables the incident notification features, and configures when and how ServiceNow sends an incident to xMatters for notification.
Engage with xMatters: Enables or disables the Engage with xMatters action in the ServiceNow user interface, sets the connection parameters for the feature, and sets the maximum number of results to return when searching for people or groups to target.
Data Sync: Configures the user and group seeding, and the data synchronization between xMatters and ServiceNow.
You need to update most of the settings on these pages with information specific to your xMatters deployment. See the following sections for more information on the settings.
To access the configuration pages in ServiceNow:
Log into ServiceNow as a user with the "admin" role, and open the Navigator.
Only users with the ServiceNow x_xma_xmatters.xmatters_admin role (added by the xMatters application) can modify xMatters components within the ServiceNow interface.
Locate and expand the Integration xMatters entry in the All Applications list, and then click xMatters Configuration.
Click the tabs at the top of the window to switch between pages.
Each setting on the pages includes a description of the required content for that setting; move your mouse pointer over the blue help icon beside a field to see more information.
Common Configuration
Hostname: The hostname of your xMatters deployment (for example, https://your-company.na1.xmatters.com) without any trailing '/'.
xMatters REST API User: The username of the integration user, used to authenticate requests to xMatters.
xMatters REST API Password: The password fo the integration user.
ServiceNow API User: The ServiceNow API user you created to authenticate requests from xMatters.
ServiceNow API Password: The password of the ServiceNow API user.
Enable MID Server & MID Server: Sets whether the integration should use a MID server when making requests to xMatters and the name of the MID server to use. See the ServiceNow documentation for information on using MID servers.
Enable Debugging: Sets whether DEBUG messages should be included in the xMatters log in ServiceNow. See the ServiceNow documentation for more information on ServiceNow logs.
Note: After you save your settings, the password fields turn into buttons, indicating there is a password entered for the user. Click the password button to update the saved password (for example, if you changed the integration user's password in xMatters).
Incident Notifications
Enable Incident Notifications: Click the check box to enable incident notifications.
Incident Alerts Integration URL: The integration URL for the workflow's Incident Alerts integration.
Response Options for Users: The UUIDs of the response options you want to be available on notifications to individual users. For the default integration, add the UUIDs for Accept and Resolve.
Response Options for Groups: The UUIDs of the response options you want to be available on notifications to groups: for the default integration, add Assign to Me, Resolve, and Escalate.
Priority: The priority of incidents that you want to send to xMatters.
Language Suffix: The language of incident properties sent to xMatters. This value is appended to property names when the integration performs certain actions.
Engage with xMatters
Enable Engage with xMatters: Click this check box to enable Engage with xMatters (users need to be assigned the x_xma_xmatters.xmatters_engage role to be able to use the feature).
Maximum number of suggestions in recipients list: The maximum number of suggestions the Engage with xMatters dialog box should display when searching for recipients.
Engage with xMatters integration URL: The integration URL for the workflow's Engage with xMatters integration.
Conference Bridge Integration URL: The integration URL for the workflow's Conference Bridge integration.
External Conference Bridges: The names of external conference bridges available in the Engage with xMatters dialog box; leave blank to use xMatters-hosted conference bridges. See the xMatters online help for more information on conference bridges.
Data Sync
Before configuring the data sync settings, we recommend you review the information on syncing users, groups, and properties from ServiceNow to xMatters.
See Change the default shift name for information on how to change the shift name used when syncing groups to avoid duplicate shifts in xMatters.
Properties
Enable Dynamic Sync: Turns on dynamic syncing of users, groups, and properties from ServiceNow to xMatters.
Site: The xMatters site you want users and groups added via the data sync to be assigned to.
Supervisor: The user ID of the supervisor in xMatters you want users added via the data sync to be assigned to. If this is left blank, the user will not have a supervisor in xMatters until you manually add one.
Always Sync Group Managers: Sync group managers even if they aren't assigned the x_xma_xmatters.xmatters role. See How ServiceNow Supervisors and Group Managers are synced for more information.
Time Zone: The time zone you want to set for users and groups when they're added to xMatters.
Externally Own Data: When set, synchronized data can only be removed from xMatters by removing it from ServiceNow
Web Login Type: Whether the user logs into xMatters using their web login (native) or LDAP.
Language: The language to set for users when they're added to xMatters.
LDAP Domain: The LDAP domain to use if Web Login Type is set to LDAP.
People
Role: The xMatters role to assign to people added to xMatters via the data sync process. You can give the user additional roles in xMatters.
Phone Device: The xMatters device name to assign to users' voice device.
SMS Phone Device: The xMatters device name to assign to users' SMS device.
Email Device: The xMatters device name to assign to users' email device.
Phone Extension Identifier: The component of a phone number in ServiceNow that denotes an extension (for example, 'x' or 'ext'); this lets xMatters recognize phone extensions.
Groups
Allow Duplicates: Enables the Allow Duplicates setting for a group added via the data sync; this setting determines whether a member can be added to the escalation timeline more than once.
Group - Use Default Devices: Enables the Use Failsafe Devices setting for a group added via the data sync; this setting determines if xMatters should contact a group members failsafe (default) device if they don't have any devices with an active timeframe.
See the online help for more information on these group settings.
Incident Properties
Configuration Item Identifier: The UUID of configuration item list property in the ServiceNow workflow. To sync the values of this list from ServiceNow to xMatters, see Batch load incident properties.
Configuration Item Query: The ServiceNow query of cmdb_ci records to send for the configuration item list property. See the ServiceNow documentation for more information on querying for cmdb_ci records.
Change the default shift name
Since this version of the integration was released, we changed the name given to the default shift created for every new group in xMatters. This requires a minor code change to the configuration script to avoid duplicate shifts.
The shift name is used in the synchronization of users and groups with xMatters.
NOTE: If you are upgrading an existing ServiceNow installation, check out the knowledge base article here for more information and suggested workarounds.
If you're installing a new version of the integration:
In ServiceNow, navigate to Advanced System Properties under Integration xMatters.
Change Group - Shift Name from '24x7' to 'Default Shift'.
Save your changes.
Add Engage with xMatters records to a related list
You can use the Related Lists feature in ServiceNow to have any related Engage with xMatters records included on the incident details.
To set up a related list:
Open an existing incident record, click the menu button, and then select Configure > Related Lists.
If the current application is not global, you will see a read-only screen with several options for editing. Select the Edit this view in Global link.
In the Available list, select xMatters Engage Records, and move it to the Selected list.
Click Save.
The xMatters Engage Records tab at the bottom of the incident details screen now displays the related Engage with xMatters records for that incident.
Sync user, group and incident property information with xMatters
After you've done the basic configuration for the integration, you can use the Batch Load features to seed xMatters with your ServiceNow users, groups, and configuration items.
Once you've seeded xMatters with your users and groups, you can use dynamic data sync to keep xMatters in tune with ServiceNow. If you add supervisors to a group or user in xMatters, or give a user additional roles, these are preserved in future syncs.
How it works
Here's an overview of the user, group and property sync from ServiceNow to xMatters:
Assign thex_xma_xmatters.xmatters role to any users or groups you want to synchronize into xMatters. Assigning this role to a group automatically assigns it to all users within the group, or you can add this role to an existing role to automatically assign it to all users with that existing role.
Batch load users then groups (and incident properties, if needed).
Enable Dynamic Sync on the Data Sync tab of the xMatters configuration pages to have any future changes synced to xMatters (either updates to synced objects or new users or groups assigned the x_xma_xmatters.xmatters role).
If you'd like a visual of how it works, we have diagrams of the batch load of users and groups, and for the dynamic data sync process.
Batch load vs. dynamic sync
There are some fundamental differences between batch data load and dynamic synchronization that you should consider when designing your integration between xMatters and ServiceNow.
Batch loads are generally only recommended when initially setting up your integration because they tend to be resouce-intense processes, while dynamic sync is generally recommended for on-going synchronization of users and groups from ServiceNow to xMatters.
Dynamic sync
If enabled, dynamic sync processes are run when a user or group is assigned the x_xma_xmatters.xmatters role, or when a user or group with that role is updated. The dynamic sync process is initiated for each object by a business rule in the foreground then runs in background worker processes.
What does this mean? If you add the role to 300 users, the integration fires the business rule 300 times once for each affected user and performs separate processing and REST requests to xMatters for each user. Since these business rules run independently from each other, no economies of scale are possible. If the integration needs to look up whether a ServiceNow user exists in xMatters, or if they're already set as a group supervisor, all of the logic, look-ups, and updates are performed each time the business rule fires. The same is true for determining group memberships: even if all members are part of the same group, the integration needs to query the group membership from xMatters for each user.
Also, the business rule itself must be run in the foreground, even though the REST calls and actual synchronization of data occurs in the background processes during a dynamic sync. This allows current values to be compared against previous values to determine if the sync is necessary.
Batch load
Batch loads are manual processes that run entirely in the background but deal with collections of objects, such as ALL users assigned the x_xma_xmatters.xmatters role.
Batch loads perform fewer data lookup queries than the dynamic sync, and cache the results for re-use. By minimizing redundant network traffic (REST requests) and database look-ups, a batch of user updates can be run much more efficiently, and entirely in background processes.
However, running a batch load script is a resource-intense process that can significantly affect server performance. This feature processes every user and group in your ServiceNow instance to determine whether they should be synchronized to xMatters. It is recommended that you only run the batch load process when the integration is initially installed, and only during non-peak server times to minimize the server strain and processing time required to complete the synchronization.
How ServiceNow Supervisors and Group Managers are synced
ServiceNow supervisors are synced to xMatters with the Person Supervisor role, if they don't already have it.
For a user's supervisors, the process is additive when the data sync happens, if a user has a supervisor in xMatters that they don't have in ServiceNow, the xMatters supervisor remains in place (in addition to any supervisor included in the sync from ServiceNow).
ServiceNow group managers are synced to xMatters with the Group Supervisor role, if they don't already have it.
You can choose to sync group managers from ServiceNow even if they don't have the xMatters sync role (x_xma_xmatter.xmatters). To use this feature, select the Always Sync Group Managers option on the xMatters Configuration Data Sync page.
On the xMatters Configuration > Data Sync configuration page, the Supervisor field displays the user that will be set as the supervisor of any users added to xMatters by the data sync. The selected user MUST exist in ServiceNow and must have the "x_xma_xmatters.xmatters" role.
If the selected supervisor does not exist as a user in ServiceNow, or does not have the "x_xma_xmatters.xmatters" role, the user and group data sync will not complete properly.
Batch load users and groups
The batch data load processes all users, groups, and group members in ServiceNow and attempts to update them in xMatters if they've been assigned to x_xma_xmatters.xmatters role. If the user or group does not exist in xMatters, it is added; if the user or group already exists in xMatters, it is updated. A user or group that exists in xMatters but not in ServiceNow is not modified by the batch process; updates to these items are handled by the user and group synchronization process based on individual changes only after the initial data load is complete.
NOTE: To prevent groups from attempting to include users that do not yet exist in xMatters, always run Batch Load Users before Batch Load Groups.
To initiate a batch data load:
In ServiceNow, under the Integration xMatters heading, click Batch Load Users.
On the Batch Load Users page, click Load Users.
When the process is complete, click Batch Load Groups.
On the Batch Load Groups page, click Load Groups.
Batch load incident properties
You can populate the configuration_item_list property in the xMatters integration with the configuration items available in ServiceNow. This allows users to subscribe to incidents that occur on a specific Configuration item.
By default, this property is set to include only the value NONE; to use this property, you must seed it with values using the Batch Load Incident Properties script. This retrieves the configuration items as defined by the "Configuration Item Property Query" in the Data Sync configuration page, and sends them to the configuration_item_list property in xMatters.
NOTE: The code to populate a value in the configuration_item_list property is disabled when the xMatters integration is first installed because, once enabled, the integration validates the entries in all of the list properties, and rejects the event if a configuration item is sent that is not in the list.
To enable this feature:
In ServiceNow, navigate to Advanced System Properties under Integration xMatters.
Select the box under Incident - Send Configuration Item List.
Save your changes.
To initiate a batch load of configuration items:
In the ServiceNow Navigator, under the Integration xMatters heading, click Batch Load Incident Properties.
On the Batch Load Incident Properties page, click Load Properties.
Notifying non-synchronized users
By default, the integration is configured so that ServiceNow only sends incidents to users who are synchronized with xMatters. However, you can set it up to send incidents to xMatters users who are not part of the data sync.
To enable this:
In ServiceNow, navigate to Advanced System Properties under Integration xMatters.
Select the box under Incident Send to Synced Recipients Only.
Save your changes.
How to use the integration
After you complete the configuration steps and perform the initial batch data load, you can test notification delivery and response, and the synchronization between xMatters and ServiceNow.
Validate synchronization
The user and group synchronizations work in an identical way, but you should still validate each synchronization independently.
Synchronize a user
When you create, modify, or remove a user in ServiceNow, the system activates a business rule and adds, updates, or deletes the corresponding user in xMatters, along with their device details.
To test the user synchronization:
In the ServiceNow Navigator, open User Administration > Users.
On the Users page, click New.
Enter the details for a new user on the User page, including first and last names, a user ID, and an email address that you can access (so you can use this user to test the notification portion of the integration in the following sections):
Click Submit.
In the Users list, locate the user you just added and click their name.
On the User page, in the Roles area, click Edit.
On the Edit Members page, in the Collection list, select x_xma_xmatters.xmatters and then click Add.
As explained above, this role is required for all ServiceNow users that you want to be synchronized with or receive notifications from xMatters.
Click Save.
Now log in to your xMatters instance and navigate to the Users page. If the synchronization was correctly configured, you'll see the user added to xMatters using the settings on the Data Sync configuration page, and the email address as their email device.
Synchronize a group
When you create, modify, or remove a group in ServiceNow, the system activates a business rule and adds, updates, or deletes the corresponding group in xMatters.
To test the group synchronization:
In the ServiceNow Navigator, open User Administration > Groups.
On the Groups page, click New.
Enter the details for a new group on the groups page, and then click Submit.
In the Groups list, locate the group you just added and click its name.
On the Group page, in the Roles area, click Edit.
On the Edit Members page, in the Collection list, select x_xma_xmatters.xmatters and then click Add.
As explained above, this role is required for all ServiceNow groups that you want to be synchronized with xMatters.
Click Save.
Now log in to your xMatters instance and navigate to the Groups page. If the synchronization was correctly configured, you'll see a group added in xMatters, with a default schedule and an empty roster. To further test the synchronization, add the test user you created for the user synchronization to the new group in ServiceNow, and confirm that the matching user is added to the group in xMatters.
Trigger a notification
To test the incident notification portion of the integration, you can create an incident in ServiceNow and assign it to the user or group you created for the synchronization test. If you assign it to the user, only that individual is notified; if you assign it to a group, xMatters works its way through the group escalation until someone responds with "Accept". When that happens, the Assigned to field in ServiceNow is updated to the user that responded.
To trigger a notification:
In the ServiceNow Navigator, open Incident > Create New.
On the Incident Details page, enter the details for a test incident with a Priority setting of "High" or "Critical".
In the Assigned To field, enter the name of your test user.
Click Submit to trigger the notification.
Respond to a notification
When the notification for the test user arrives on your device, open the message to view its details. For example, the following illustrates how a ServiceNow notification via xMatters might appear in the xMatters mobile app:
You can respond to the message simply by tapping one of the response choices. Other devices use similar methods.
View response results
After you respond to the notification, you can see how the integration automatically updates the Activity section in the incident Notes:
Engage with xMatters
You can test the Engage with xMatters feature using an existing incident.
To engage with xMatters:
In the ServiceNow navigator, open Incident > Open.
In the list of incidents, click an incident's number to open it.
On the incident's details page, click Engage with xMatters.
In the Engage with xMatters dialog box, enter additional details about the incident in the provided fields, and use the Recipients area to add the people or groups you want to notify.
For groups, this also searches the group description. Multiple search terms are treated as an AND (for example, searching for "database" and "admins" will return the group Database Admins but not Network Admins).
Click Submit.
To add a new user or group, simply add them to the recipients and click Submit. This adds them to the on-going Conference Bridge associated with the incident ID.
View Engage with xMatters responses
To view responses to Engage with xMatters notifications, use the ServiceNow Navigator to open Integration - xMatters > Miscellaneous > Engage with xMatters Records. Click the number of the record you want to view.
Troubleshooting
If you are encountering difficulties with the integration and want to view more detailed information, select the Enable Debugging check box on the xMatters Common Configuration page in ServiceNow. This increases the level of detail contained in the log messages for the integration.
To access the xMatters logs in ServiceNow, look in the Integration - xMatters menu, under Miscellaneous > xMatters Logs.
You can also find more information in the Activity Stream for each inbound and outbound integration: while on the Integration Builder tab, expand the inbound or outbound section, click the gear icon beside an integration service, and then click Activity Stream. The Activity Stream contains the incoming (and for outbound integrations, the outgoing) request, any logging statements, and the final event creation messages.
Data sync troubleshooting
One possible cause of data sync errors is a custom field in xMatters that's been set to mandatory. To work around this, you need to either:
populate that field with something in ServiceNow so it is included in the data sync, or
update the custom field in xMatters so it isn't mandatory.
Extended functionality
The unofficial/official Integration Research and Development wing of xMatters (those clever elves over at the xMatters Labs ) devised a couple of cool ways to extend and enhance this integration:
ServiceNow Major Incident - Add ServiceNow Major Incident Management into the xMatters notification process.
ServiceNow Inbound via IntegrationHub - Set up a trigger from any table and table entry in ServiceNow to create an event in xMatters.
Engage with xMatters for Problem Management - Extend the Engage with xMatters functionality to the Problem Management module in Service Now.
Post to Chat - Create a room in a chat application from the Engage with xMatters form.
ServiceNow Event Log - Log all xMatters event data (such as event and delivery statuses) to a table in ServiceNow
Inform with xMatters - An "add-on" for ad-hoc FYI type notifications from ServiceNow.
Engage CI Support Groups - An "add-on" for including the upstream and downstream CI groups to the Engage with xMatters form.
ServiceNow Service Portal On-Call Widget - Get on-call schedules in a widget in ServiceNow's Service Portal.
Check out each of these enhancements at the links provided, and keep an eye out for future improvements, too! (Remember that these features are in-progress or experimental additions to this integration, and as such we can provide only limited support should you choose to implement them.)
How it works: the nitty gritty
Each of the components is based on a trigger on a business rule; these triggers are set for Insert, Update, and Delete. After the change has been committed, initial checking within the business rules determines what type of operation has occurred. A call is then made to the script include, which processes the request and runs various rules based on the Incident Notification, Engage with xMatters, and Data Sync Configuration pages to determine what type of action to take.
If the script include determines that an action is to be sent to xMatters, it creates a ServiceNow REST message for incident events or an xMatters API REST message for synchronization requests. ServiceNow REST messages are received by the inbound integrations on the workflow, where the appropriate notification layout is determined based on the target device. xMatters API messages are received by the xMatters REST operations, and devices, users, or groups are added, updated, or deleted appropriately.
If you thrive on detail, we've put together a few diagrams to show you how the various pieces of the integration work together with the nuts and bolts of ServiceNow.
Incident flow
Engage with xMatters flow
Batch data sync users
Batch data sync groups
Dynamic data sync flow
Upgrading your integration
If you already have a working xMatters ServiceNow integration, use the following steps to upgrade your instance. (If you are installing the integration for the first time, or creating a new integration, head to the Install the xMatters application section).
Upgrading a standard (out-of-box) integration
Notes before you upgrade:
If you're upgrading from integration version 3.4 (or older), you may find it easier to disable your current integration and install the latest version as a fresh installation.
If you're upgrading from integration version 3.5, disable the data sync feature before beginning the upgrade process described below.
If you're upgrading an integration that was originally installed using an update set, there are some extra steps you need to do after you upgrade to continue to insert Engage with xMatters records.
To upgrade your integration:
Download the latest workflow ( attached to this article), and import it into your xMatters instance.
Make sure your integration user has editor permission to the workflow, and sender permissions on the forms.
Open the Integration Builder tab in the imported workflow, and set each of the inbound integrations to use Basic Authentication.
Copy the URL for each inbound integration to a safe place, such as a text file.
Copy the UUID for each of the response choices on the form to the same text file.
Log in to ServiceNow and disable the Incident and Engage with xMatters features for your existing integration.
Update the xMatters application to the latest version from the Applications listing screen in ServiceNow.
If you installed your integration using an update set, make sure you reach out to the Customer Support team.
Update the configuration pages with the inbound integration URLs and response UUIDs from your text file.
Re-enable the Incident and Engage with xMatters features.
After you've upgraded your integration, you can make any changes to the configuration of your xMatters app in ServiceNow.
Upgrading a customized integration
If you have any customizations (from working with an xMatters consultant or ones you've made yourself), we strongly recommend attempting the upgrade in a non-production or staging environment first. That way you can run the necessary tests to ensure that integration functionality won't be affected.
If you encounter any problems or want to get some help before attempting your upgrade, please contact xMatters Customer Support and we can help come up with a migration plan.
Upgrading a customized integration prior to version 5.3 to the latest version
In version 5.3, we moved values from the xMattersConfig Script Include into system properties. To upgrade from a version earlier than 5.3 to the latest, you need to update the system properties to match what is in your customized script include file.
Backup your xMattersConfig Script Include. You can find this under the Integration xMatters entry in the All Applications list, in the Script Includes section.
Update your integration, accepting the changes in the latest version.
Compare the backup of your original xMattersConfig file with the values in Advanced System Properties under Integration xMatters, using the table below, and update the system properties, if needed, to match the values your customized integration expects.
Advanced System Property
xMattersConfig Variable
Incident - Inactive States
INCIDENT.INACTIVE_STATES
Incident - Critical Priority
INCIDENT.CRITICAL_PRIORITY
Incident - Event Queue Suffix
INCIDENT.EVENT_QUEUE_BASE
Incident - Assignment Trigger
INCIDENT.TRIGGER_RULES.ASSIGNMENT
Incident - Priority Upgrade Trigger
INCIDENT.TRIGGER_RULES.PRIORITY_UPGRADE
Incident - SLA Alert Trigger
INCIDENT.TRIGGER_RULES.SLA_ALERT
Incident - Reopened Trigger
INCIDENT.TRIGGER_RULES.REOPENED
Incident - Delete Trigger
INCIDENT.TRIGGER_RULES.DELETE
Incident - Send Configuration Item List
INCIDENT.SEND_CONFIGURATION_ITEM_LIST
Incident - Suppress Events when Assigning to Self
INCIDENT.SUPPRESS_ASSIGN_SELF
Incident - Suppress Events with Empty Recipients
INCIDENT.SUPPRESS_NO_RECIPIENT
Incident - Send to Synced Recipients Only
INCIDENT.SEND_TO_SYNCED_ONLY
Event - High Priority
PRIORITY.HIGH
Event - Medium Priority
PRIORITY.MEDIUM
Event - Low Priority
PRIORITY.LOW
Event - Maximum Attempts
EVENTS.MAX_ATTEMPTS
Event - Maximum Field Length
EVENTS.MAX_FIELD_LENGTH
Group - Observed By All
OBSERVEDBYALL
Group - Shift Name
SHIFTNAME
User - Username Field
USERNAMEFIELD
Data Sync - Max User Processes (LEGACY - DO NOT USE)
MAX_USER_PROCESSES
Data Sync - Max Calls per Process
MAX_CALLS_PER_PROCESS
Data Sync - Calls per User Synced
CALLS_PER_USER
Data Sync - Calls per User Deleted
CALLS_PER_USER_DELETE
Data Sync - Calls per Group Synced
CALLS_PER_GROUP
Data Sync - Calls per Group Member Deleted
CALLS_PER_MEMBER_DELETE
Data Sync - Use Progress Workers (LEGACY - DO NOT USE)
USE_PROGRESS_WORKERS
Data Sync - Number of Queues
NUM_DATA_SYNC_QUEUES
REST Log Level - Normal Mode
REST_LOGLEVEL
REST Log Level - Debug Mode
REST_LOGLEVEL
If there are values in your customized xMattersConfig Script Include that are not in the Advanced System Properties or the new xMattersConfig Script Include, we recommend:
moving those values into another Script Include (you'll need to update any references to them in other Script Includes).
not adding them back into the xMattersConfig Script so this file can be updated in future without you having to worry about merging your customizations.
Previous updates
See our current features and updates list for additions in the latest update.
Updates in previous versions of v5 of the integration
The 5.3 version of the integration included the following updates:
Refined the data synchronization scripts to improve performance.
Moved values from xMattersConfig script include into system properties. If you've customized this script include in the past, there are a few extra steps you'll need to follow as part of the upgrade process.
The 5.2 version of the integration included the following updates:
Version 5.2.4 certified for the Madrid release of ServiceNow.
Updated the search behavior to use AND instead of OR when looking for people or groups to target in the Engage with xMatters dialog box.
Added a new setting in the Engage with xMatters configuration screen that lets you set the maximum number of results to return when searching for recipients to target.
Addressed issues with dynamic sync functionality when upgrading from a customized 4.x version, incident custom queue load balancing, and errors when syncing groups from ServiceNow for the first time when the group already exists in xMatters.
The 5.1 version of the integration included the following updates:
Certified for the London release of ServiceNow.
The group data sync now preserves (instead of overwriting) any supervisors you've set up for a group in xMatters.
Improvements to the xMatters configuration page in ServiceNow, including showing you if you've already entered a password and accepting xMatters hostnames with an extra backslash at the end. We also cleaned up some unused configuration settings.
The 5.0 version of the integration included the following updates and enhancements:
Updated Data Sync components to avoid hitting Progress Worker Quota limits introduced in ServiceNow Jakarta.
Added Initiator to the “Engage with xMatters” form.
Added targeted recipient and ServiceNow initiator information to the event notification (requires the 5.0 version of the workflow).
This version also includes all the features and updated added in previous versions of the xMatters integration.
Updates introduced in v4
The 4.2 version of this integration includes the following fixes and enhancements:
Implemented the new Event Comments trigger, which allows the integration to update the ServiceNow ticket with comments added to the event in xMatters (requires the 4.2 version of the workflow).
Added support for the ServiceNow Compact User Interface.
Updated the group data sync so any supervisors you've set up for a group in xMatters
View ArticleThey say it's hard to find good help these days.
Well, they must not be using xMatters, where great documentation is always just a click away!
Did you know? In xMatters you can...
See all videos...
...click the 'help' icon (the question mark) at the top right corner of any screen to open an online help topic that's specific to that screen.
...use the Tutorials & Videos widget on the Communication Center dashboard or homepage to open an extensive list of tutorial videos and other help resources.
You don't have to be signed into xMatters to view the documentation. Scroll down this document to explore the great online help resources that are available for xMatters and its related services:
Online Documentation
xMatters Online Help
The official documentation source for xMatters.
Read more...
xMatters REST API Documentation
Technical API documentation with dynamic code samples in 3 languages.
Read more...
Online Integration Guides
Detailed instructions on how to configure any of our available integrations.
Read more...
Video Resources
xMatters Learning Center
A repository of tutorial videos about how to configure and use xMatters.
View ArticleSome users, particularly in India, have reported that they are unable to acknowledge or respond to an event when xMatters calls them on a voice phone. Even though the call is connected normally, and the person answers the call, they are not given any chance to respond. Instead, xMatters simply plays the notification and hangs up.
The issue also seems intermittent and unpredictable; it does not happen on every call, and does not always happen to specific users.
Solution
The cause of this issue can usually be determined by examining the Event logs. In the instances where the response options are not presented, the report indicates that the call went straight to voicemail, even though the recipient answered the phone. When xMatters thinks that a machine has answered, it delivers the message and then says "Good bye", without presenting any response options (because a machine can't submit a response choice).
This is caused by xMatters picking up background noise, usually when the call is answeredin a noisy environment, and incorrectly determining that it has reached a machine or voicemail service.
Sometimes, this issue can be mitigated or eliminated by adjusting the machine detection settings, but this is not always possible in certain situations or environments. Machine detection is subtle and inexact at the best of times, and if your users often work in loud environments, it may be exceptionally difficult to fine tune your deploymentproperly.
In the meantime, you can help reduce the chances of this happening by advising any users working in a noisy environment that, when answering a call from xMatters, they should press the 'Mute' button on their phones immediately after saying "Hello". This will help prevent xMatters from inadvertently picking up any background noise and incorrectly thinking that a machine has answered the call.
xMatters internal reference: DTN5029
View ArticleContents
Introduction
How it works
Features and updates
Before you begin
Download the integration package
CA Service Desk Manager API user
Configure xMatters
Set up an integration user
Import the communication plan
Configure inbound integrations
Create a web service user
Determine identifiers
Configure the Integration Agent
Install the integration files
Set the password files
Configure deduplication
Configure CA Service Desk Manager
Install the integration script
Configure a notification method
Configure CA SDM to use managed login
How to usethe integration
Trigger a notification
Respond to a notification
View response results
Extend and optimize your integration
Purge temporary files
Turn off delivery annotation
Configure data load from CA Service Desk Manager
Download resources
The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
Introduction
This article provides you with the installation and configuration details you need to integrate the xMatters On-Demand service with CA Service Desk Manager.
This integration uses the communication plan technology within xMatters On-Demand to become thevoice and interface of an automation engine. When CA Service Desk Manager detects something that requires attention, xMatters places phone calls, send emails, or notifies your mobile app.
Integration Version
This version of the integration (3.0) is compatible with CA Service Desk Manager 14.1 (and later), and is designed specifically for xMatters On-Demand.
How it works
The integration adds functionality to the CA Service Desk Manager notification process, allowing users to own, acknowledge, escalate, and clear events remotely from their device.
Whenever CA SDM detects an event, a custom Notification Method runs a wrapper shell script that sends a notification file to the xMatters Integration Agent.The Integration Agent extracts the NX_NTX fields from the notification file and sends them to xMatters as event properties.
A REST-based call to the CA Service Desk Manager web services API updates the original event with response information.
You may need to customize some aspects of this integration to suit your specific requirements. For example, by default the integration filters out notifications that are not addressed to the ticket's assignee or group, and terminates previous xMatters events related to the same ticket. You can also customize the response options according to your business needs.
Features and updates in this version
This version of the integration includes several updates and enhancements:
The integration now targets the inbound integrations included in the communication plan rather than the form itself. This provides access to all of the enhancement and options available in the Integration Builder, including the Activity Stream, transformation scripts, and authentication options.
Before you begin
Before you get started configuring the integration, there are a couple of things you can do ahead of time to make it easier.
Download the integration package
The .zip file attached to this article contains all of the components required to integrate xMatters and CA Service Desk Manager. Download the attached .zip file to a location on your local machine, and extract the contents.
You may also notice that there is another .zip file named CAServiceDeskManager within the extracted integration package. This is the communication plan, which contains pre-configured integrations, forms, properties, and messages specifically designed for CA Service Desk Manager. Do NOT extract the contents of the communication plan .zip file! You'll import it directly into xMatters.
(This version of the integration uses a new version of the Integration Agent Utilities package that is included in the integration archive.)
CA Service Desk Manager API user
You'll need the username and password for a user who can access and make REST calls to the CA Service Desk Manager Web Services API.
For information about adding users and the Web Services API, refer to the CA Service Desk Manager documentation.
Configure xMatters On-Demand
The first step in configuring this integration is to get your On-Demand environment ready, and set up the components on the xMatters side.
Set up an integration user
This integration requires a user who can authenticate REST web service calls when working with events these permissions are provided by the "REST Web Service User" role in xMatters. See Create an integration user for more information.
Note:Make sure you keep the user ID and password of this user handy. You'll need them when configuring other parts of this integration.
Import the communication plan
The next step is to import the communication plan.
To import the communication plan:
InxMatters, click the Developer tab, and then click Import Plan.
Click Choose File, and then locate the plan you downloaded from this article (the .zip file).
Click Import Plan.
Once the import is finished, the plan should be automatically enabled. If it isn't, click Plan Disabled to enable the plan.
Click the Edit drop-down list for the plan, and select Access Permissions.
Addthe integration user, and then clickSave Changes.
In theEditdrop-down list for the plan, selectForms.
For thefirst form in the list (Change Order Alerts), clickthe Web Service Only drop-down list, and then selectSender Permissions.
Addthe integration user, and then click Save Changes.
Repeat steps 7-8 for each of the other forms in the plan.
Configure inbound integrations
You need to configure the authentication and retrieve the URLs for each of the inbound integrations to configure CA Service Desk Manager.
To configure an inbound integration:
In the Integration Builder, expand the list of inbound integrations.
Click the name of anintegration to view its details.
Under the Select authentication method step, select Basic from the drop-down list.
Click Update Integration.
Scroll down to the bottom of the page, and click Copy URLbeside the field (you'll need this later to configure the Integration Agent):
Create a web service user
This integration requires a web service user with specific permissions in xMatters to receive user responses and notifications about event status changes for the Integration Agent. ("Web service users" are a specific type of user in xMatters that can work with the SOAP web services used by the Integration Agent; this means you can't just use a regular user and assign them a specific role or permission you have to create a dedicated user.)
To set up a web service user:
In xMatters, click the Users tab, and then, in the menu on the left side of the page, click Add Web Service User.
On the Add Web Service User page, enter a User ID.
In the Denied Web Services list, locate and select the following web services, and then click Add:
Query User
Receive APXML
Register Integration Agent
Submit APXML
Enter aPassword for the new web service user.
Click Save.
Determine identifiers
Some configuration fields require a UUID (Universal Unique Identifier) for a specific component or property in a communication plan - usually the responses. To retrieve the identifier for an element, look for the API icon in the xMatters user interface:
Clicking this button displays the identifier for the component in a dialog box, which you can copy and paste into the appropriate location in the configuration file.
As an example, the following instructions describe how to retrieve the UUID for a specific response choice; the process is similarfor any identifier you want to determine.
To retrieve the identifier for a response choice:
In xMatters, open the CA Service Desk Manager communication plan.
On the Service Desk ManagerIncidents form, click Edit > Responses.
Besidethe Acknowledged response, click the API icon.
Configure the xMatters Integration Agent
Now that you've configured xMatters On-Demand, it's time to configurethe Integration Agent.
Theinstallation instructions below assume you already have a working xMatters Integration Agent. If this is a new installation and you have not yet deployed the Integration Agent, follow this link to download, deploy and configure it: Integration Agent for xMatters 5.x & xMatters On-Demand.
Install the integration files
The installation package contains all that you need to configure the integration.
Note: As usual for our Integration Agent documentation, <IAHOME> refers to the installation folder of the Integration Agent on your system.
To install the integration files:
Within the extracted integration archive,navigate tocomponents/integration-agent/integrationservices, and copy the entire contents of the folder to the <IAHOME>/integrationservicesdirectory.
Open the<IAHOME>/conf/IAConfig.xmlfile and add the following line to the "service-configs" section:
<path>applications/caservicedesk-3-0-0/caservicedesk.xml</path>
Save and close the file.
Open the <IAHOME>/integrationservices/applications/caservicedesk-3-0-0/configuration.js fileand add or set the values for thefollowing variables:
INTEGRATION_URLS
The URLs for each of the CA Service Desk Manager inbound integrations.
ERROR_URL
The URL of the inbound integration to use when response updates fail in CA Service Desk Manager.
RESPONSES
The identifiers for the response options for each xMatters form or CA Service Desk Manager ticket type.
INITIATOR
The User ID of the integration (REST) user you created in xMatters.
INITIATOR_PASSWORD_FILE
Path and filename of the password file containingthe encrypted password of the xMatters initiator user.
INITIATOR_USER_ID
The User ID of the user that authenticates REST API callsto xMatters.
SERVICE_DESK_URL
The URLof the CA Service Desk Manager web service.
SERVICE_DESK_USER
The user name of the CA Service Desk Manager web service user used to access the web services.
SERVICE_DESK_PASSWORD_FILE
Path and filename of the password file containingthe encrypted password of the web service user.
DEDUPLICATOR_FILTER_NAME
Name of the filter used to suppress duplicate notifications for this integration. For more information, see Configure deduplication.
DELETE_EXISTING_EVENTS
When set to true, sends a 'delete' prior to creating the new event to clear any existing events for this Incident ID.
ERROR_MESSAGE_BODY
ERROR_MESSAGE_SUBJECT
ERROR_MESSAGE_RESPONSE
Used to create an xMatters event when there is an exception while processing a user response.
ERROR_MESSAGE_PRIORITY
Priority of the xMatters event created if there's an exception while processing a user response.
INTEGRATED_PROPERTIES
Specifies the mapping of CA Service Desk Manager web service properties to the integrated properties in xMatters.
TICKET_CONTACTS
CA Service Desk Manager incident properties to be treated as Contact UUIDs and resolved via look-up call to CA SDM.
ADD_IGNORE_RESPONSE
If set to true (default), adds "Ignore" to the list of response choices. Set this tofalse to remove the option for a user to ignore a notification.
Save and close the file.
Restart the Integration Agent.
Set the password files
This integrationincludes encrypted files, located in the <IAHOME>\conf folder, that store the passwords for the integration user and the CA SDM web services user. You need to update (or create) the files with the correct passwords for these users.
Password files:
"initiatorpasswd" stores the password for the integration user
"caservicedesk" stores the password for the CA Service Desk Web Services user
Note: If you store the passwords with a different file name or in a different location, you must update the <IAHOME>/integrationservices/applications/caservicedesk-3-0-0/configuration.jsfile with the correct name and path.
To configure the REST API user password:
Open a command prompt and navigate to <IAHOME>\bin.
Run the following command, where <new_password> is the password for the INITIATOR (the integration user) specified in the<IAHOME>/integrationservices/applications/caservicedesk-3-0-0/configuration.jsfile, and <old_password> is the existing password (the default password for a newly installed integration is "password"):
iapassword.bat --new <new_password> --old <old_password> --file conf/.initiatorpasswd
Next, run the following command, where <new_password> is the password for the SERVICE_DESK_USER (the CA Service Desk Web service user) specified in the<IAHOME>/integrationservices/appllcations/caservicedesk-3-0-0/configuration.jsfile, and <old_password> is the existing password (the default password for a newly installed integration is "password"):
iapassword.bat --new <new_password> --old <old_password> --file conf/caservicedesk.pwd
For more information about working with the iapassword command, and creating password files,refer to the Integration Agent section in the help.
Configure deduplication
The integration package includes a filter that engages the Integration Agent's filtering and suppression module to suppress duplicate events (also know as deduplication). This can help avoid disruption of traffic due to event floods (a potential issue with improperly configured management systems).
The deduplicator is installed into the <IAHOME>\conf folder, and is configured by default to suppress up to 100 duplicate events for two minutes. You can configure the filter to extend the time period in which an event is considered to be a duplicate, the number of events within that period, and the tokens or properties that xMatters uses to determine what makes the event unique.
To enable deduplication:
Navigate to the <IAHOME>/conf directory.
If you have an existingdeduplicator-filter.xml file in this folder, create a backup version in a separate location.
Copy the deduplicator-filter.xml file fromtheintegration-agent/conffolder in the extracted integration archive into the <IAHOME>/conf directory.
If you backed up an existing deduplicator file, merge the contents of your backup with the newly installed <IAHOME>/conf/deduplicator-filter.xml file:open both files in a text editor, and then copy the <filter> node from the backup file into the new deduplicator file after the last </filter> node. Save and close the file.
Restart the Integration Agent.
For more information about the deduplication filter and the available settings, refer to the "Filtering and Suppression" section in the Integration Agent section of the help.
Configure CA Service Desk Manager
To configure CA Service Desk Manager tointegrate with xMatters, you need to:
Install the xMatters integration script on the CA SDM server.
Configure a new notification method to notify Users via xMatters.
Configure CA SDM to use managed login and allow impersonations (optional but recommended).
Note: You may also want to perform periodic maintenance on the CA SDM temporary files, as described in Purging temporary files.
Install the xMatters integration script
The integration has a pair of operating-system-specific integration scripts (xmattersIntegration for Linux andxmattersIntegration.bat for Windows) that are used as a bridge between CA SDM and the event integration service. You need to configure the file for your OS then copy it to your CA SDM server.
Configure the integration script
These scripts assume that the Integration Agent is installed in the default <IAHOME> folder. If you installed the Integration Agent in a different location, update the script files to reflect the installation path of your Integration Agent.
Note: CA SDM also requires that any paths in these script do not contain spaces. For example, the default installation path inWindows systems includes the "Program Files (x86)" folder. In this case, you needto replace all paths in the xmattersintegration.bat file with their "8dot3" formatted equivalents.
To determine the 8dot3 version of a folder name, run the dir /x command in its parent folder.This command produces a directory listing of the folder's contents in 8dot3 format (for example, "Program Files (x86)" often becomes PROGRA~2). Use these folder names when configuring your integration script.
To configure the integration script:
For Windows systems, determine the "8dot3" representation of the path to your <IAHOME> folder.
For example, the 8dot3 representation of the default Integration Agent installation folder shouldresemble the following:c:\progra~2\xmatters\integr~1
Open the <IAHOME>/integrationservices/applications/caservicedesk-3-0-0/xmattersintegration.bat (xmattersintegrationfor Linux) file in a texteditor.
Edit the following lines to replace the paths with the location of your Integration Agent's bin folder.
IF EXIST "C:\Program Files (x86)\xmatters\integrationagent\bin\" (
SET IAHOME=C:\Program Files (x86)\xmatters\integrationagent
For example, to use the path shown earlier, you would editthe lines as follows:
IF EXIST "c:\progra~2\xmatters\integr~1\bin\" (
SET IAHOME=c:\progra~2\xmatters\integr~1
Save and close the file.
Some versions of the integration script contains two sets of similar lines. We recommended you replace both sets of the lines with a single set as identified in the steps above.
Copy the integration script to the CA SDM Server
To simplify the command path for the xMatters Notification Method that you'll create in CA SDM, we recommend you copy the integration script files to the CA SDM server.
Copy the scripts to one of the following locations:
Windows: Copy the <IAHOME>\integrationservices\applications\caservicedesk-3-0-0\xmattersIntegration.bat file from the extracted integration archive to the <CA SDM Install Folder>\Service Desk Manager\bin folder. (On Windows systems, the default CA SDM installation location isC:\Program Files\CA\Service Desk Manager.)
Linux: Copy the <IAHOME>/integrationservices/applications/caservicedesk-3-0-0/xmattersIntegration file from the extracted integration archive to the <CA SDMInstall Folder>/Service Desk Manager/bin folder.
Configure a notification method
To configure a notification method for this integration, create a new notification for xMatters, and then set CA Service Desk Manager to use the new notification method.
To configure a notification method:
Click the Administration tab in the CA Service Desk Manager Web Client interface, expand Notifications, and then clickNotification Methods.
Click Create New.
In the Create New Notification Method dialog box, enter the following information:
Symbol: xM
Write to File: Yes
Supports SMTP: No
Record Status: Active
Description: Notify via xMatters
In the Notification method field, type the command appropriate for your operating system:
Windows: launchit.exe -b xmattersIntegration.bat
Linux: launchit -b xmattersIntegration
Set the xM notification method for contacts. CA SDM will now use xMatters On-Demand to notify users.
Configure the notification rules in CA SDM.
Configure CA SDM to use managed login and impersonation
It is recommended that you configure CA Service Desk Manager to support managed login and impersonation. This allows interactions from xMatters to CA SDM to be executed by impersonating the user responding to the notification.
Note: this requires each CA SDM user who you want to receive notifications from xMatters to have the Impersonate check box selected.
The Managed Login functionality in CA SDM performs the user authentication by: locating the defined security policy through the plain text policy code; finding the policyholder's public key associated with the policy; decrypting the encrypted policy code; matching decrypted content with the policy code; and, finally, opening a session with a back-end server.
To configure Managed Login on CA SDM, please refer to the CA Service Desk Manager documentation on how to create the Manager Certificate for use in this process.
Note: Upgrading CA Service Desk Manager overwrites the Manager Certificate. Ensure that you create a copy of the Manager Certificate file and store it in a location outside the CA SDM subdirectory.
To configure Managed Login with xMatters:
After the Manager Certificate has been created:
Open the <IAHOME>/integrationservices/appllcations/caservicedesk-3-0-0/configuration.jsfile.
Ensure that:
MANAGED_LOGIN and ENABLE_IMPERSONATE are set totrue,
ACCESS_POLICY_FILE_PATH matches the location of the Manager Certificate (ending in a forward slash), and
ACCESS_POLICY matches the name of the Manager Certificate (with no file extension).
The values in the configuration.js file are the default values that are applicable if you are following the CA Service Desk Manager documentation.
If you don't want the integration to use managed login, set the MANAGED_LOGIN and ENABLE_IMPERSONATE variables to false. In this case, the SERVICE_DESK_USER is used to update the tickets with xMatters responses. The managed login functionality applies only to the notification integration; the data load service does not use managed login.
Configure a contact for system notifications
If you configure the integration to use managed login, it's a good idea to configure a separate CA SDM contact to use for xMatters system notifications (such as delivery notifications). You can use any Access Type that has Functional Access for issue_mgr set to "Modify" in the CA SDM Security and Role Management details. For more information, refer to your CA SDM documentation.
To configure the contact:
In the CA SDM interface, click the Administration tab, expand Security and Role Management, then click Contacts.
Click Create New.
In the Create New Contact dialog box, enter the following information:
Last Name: xMatters
User ID: xMatters
Access Type: Administration
Status: Active
Click Save.
Note: If you enter a different User ID, you must also update the value of the ANNOTATION_USER variable in the configuration.js file.
And that's it! Your integration should be ready to go.
How to usethe integration
When the integration is configured, CA Service Desk Manager automatically sends information about any incidents it detects to xMatters via the Integration Agent.
Trigger a notification
To test the incident notification portion of the integration, create a new ticket or incident in CA Service Desk Manager that targets a user with a device that you can access (so you can respond to the notification when it arrives) and who exists in both CA SDM and xMatters.
Respond to a notification
On a device with the xMatters mobile app, you can respond to the message simply by tapping Respond, and then tapping one of the response choices.Other devices use similar methods.
View response results
After you respond to the notification, you can see how the integration automatically updates the event information with the response details in CA SDM.
In the following example, the "Assignee" field has been updated to the name of the responding User, the "Status" has been updated to "Acknowledged", and the "Incident Activity Log List" has been updated with additional entries to show the changes resulting from the User's response:
Extend and optimize your integration
You can use the following tips to customize your integration to better suit your deployment.
Purge temporary files
This integration relies on temporary files created by CA SDM to store notification data that will be sent to the integration agent. Files generated for the xMatters integration are not needed after the incident has been injected into the integration agent, but the temporary files are not automatically deleted.
If these files are not purged occasionally, injections into xMatters may be delayed while CA SDM searches for the next available file name (for example, after restarting CA SDM).
By default, these files are stored in the temp folder on the CA SDM server, and are named sequentially, for example, c:\temp\1, c:\temp\2, etc.
The integration creates a folder into which it moves the notification files after they have been processed. This folder is named "xmatters", and is located within the original file location. For example, if CA SDM creates a notification file called C:\temp\1, the integration renames the file and moves it to c:\temp\xmatters\1.<date>, where <date>is the current date and time in a "yyMMddHHmmss" format. (The directory and filename scheme can be customized by editing the caservicedesk.js integration script. This process is intended to reduce the delay that may be encountered after CA SDM is restarted, before notifications are sent to the integration agent.
It's recommended that you occasionally purge the files from the temp\xmatters folder to reduce congestion of the CA SDM server's file system.
Customize the handling of temporary files
The temporary file handling feature can be disabled or configured by modifying the following sections within the caservicedesk-event.js integration script.
// markProcessed() will attempt to move the parsed notification file to a subdirectory called xmatters.
// This is bound by file system access so if you don't want this feature just comment out fileParser.markProcessed().
fileParser.setCopyAfterProcessing(false);
fileParser.markProcessed();
//fileParser.markProcessed('<custom directory>');
// If you want a different directory, use full qualified path.
//fileParser.markProcessed('<custom directory>', '<custom filename>');
// If you want to control directory and file name.
To copy (instead of move) the CA SDM temporary file to the xmatters folder after processing, setsetCopyAfterProcessingto true.
To move the file to a folder other than \%temp\%\xmatters, edit the script to include the desired location using the format "fileParser.markProcessed('<custom directory>');". For example:
fileParser.markProcessed('c:\deleteme');
To move the file to a custom folder AND rename it, edit the script to include the desired location and name in the format "fileParser.markProcessed('<custom directory>', '<custom filename>');".For example:
fileParser.markProcessed('c:\deleteme', 'latest-notification-file.txt');
This causes the previous notification files to be overwritten, effectively preventing the folder from filling up with temporary files (since each file replaces the previous one when it is moved and renamed.)
Turn off delivery annotation
By default, this integrationannotates the originating CA SDM ticket for each device to which a notification is delivered. If this is not desirable in your environment, you can prevent delivery annotation of an incident by changing the ANNOTATE_DELIVERY variable to false in the configuration.js file.
Configure data load from CA Service Desk Manager to xMatters
This integration supports one-way batch loading of groups, users, and devices from CA SDM intoxMatters. To configure the data load according to your business needs, see Configuring data load for CA Service Desk Manager.
Download resources
View ArticleOne of the easiest ways to receive and respond to notifications is with the xMatters mobile app, available for iOS or Android devices.
This article provides helpful troubleshooting tips for installing and using the xMatters app.
Download & install the mobile app
If you're just getting started and are not quite sure about how to download and install the app on your device, be sure to check out the following tutorial videos that walk you through the process:
Installing the xMatters iOS app
Installing the xMatters Android app
You can also refer to the online help topic, Get the xMatters mobile app, for step-by-step instructions to download the app, and to log in using your host name or by scanning a QR code.
What's my host name?
The trickiest part of getting the app up and running is logging in with the host name of your xMatters deployment. The host name is the URL that you use to access the xMatters web user interface, without the "http://" or "https://".
For example, the hostname for https://mycompany.xmatters.com is mycompany.xmatters.com.
Using the mobile app
If you experience problems when you're using the xMatters mobile app, there are a couple of things you can try before you resort to filing a support ticket. These tasks may seem simple, but you'd be surprised how many times they do the trick:
1. Reboot your device
In many cases, the solution to an issue with an app or your device is to restart your device. (Yeah, the old "have you tried turning it off and on again?" There's a reason we ask that first - it often really does work!)
Most devices can be turned off by pressing and holding the power button until a message appears on the screen, confirming you'd like to turn the power off. Once powered down, press the power button on your device to restart it.
2. Reinstall the mobile app
If that didn't work, the next step is to make sure you have the latest version of the xMatters app, including any recent bug fixes, installed on your device. The best way to do this is to delete the app and reinstall it (see the videos above for help downloading and installing the xMatters mobile app on iOS and Android).
iOS: An easy way to delete the app on iOS is to find the xMatters icon, then push and hold it down until it starts jiggling. Tapthe X in the top left corner of the icon to delete it.
Android: Go into your device settings (usually a cog, wrench, or screwdriver icon), and find your Apps menu. Find the xMatters app, tap to open it, and tapthe uninstall button at the top of the page.
Contact technical support
If you've tried to reboot your device and reinstalled the mobile app, but you're still experiencing the issue, then it's time to contact technical support.
To help us help you more quickly, and reduce the amount of back and forth emails, try and include as much of the following information in your support ticket as possible (don't worry if you don't know all the answers).
Information to tell support:
What's your xMatters mobile device ID ?
What is the problem and how do you reproduce it?
Did this ever work for you?
When did the issue start? (On a certain date? After you upgraded to a new version of the app, or a new phone, etc.?)
What version of iOS or Android are you running?
Are you running the latest version of the xMatters app?
What model is your phone or device?
Can you provide a screenshot of the error or problem?
For iOS users, having diagnostic sharing turned on can also help reduce the time to resolution if your app is crashing. See the Enabling Diagnostics Sharing tutorial video for more information on this iOS feature.
Mobile Device ID
If the xMatters app ever crashes unexpectedly, you can quote your mobile device ID when you contact xMatters Support. Our support engineers can use it to look up relevant crash logs and other information to get greater insight into what went wrong and get your app and running smoothly again as quickly as possible.
To locate your mobile device ID, go to settings in the mobile app and tap About xMatters:
xMatters reference
DTN-4628 Originally by Daniel Thomas
View ArticleThe information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
The Historical On Call report lets supervisors know how much time their team members have been spending on call, and divvies up each person's history into manageable segments.
We're working on adding more features and functionality to the report to make it even more useful and to allow supervisors to fine-tune their exported data, but there's already one sure-fire way to make the data easily accessible and understandable: pivot tables!A pivot table is a great way to break down and summarize large data sets much like the one in your newly generated Historical On Call report so you can analyze and extract the data that's most important to you.
Of course, there are thousands of different tutorials, walkthroughs, videos, and other assorted resources online on how to use pivot tables, but we felt it would be helpful to show you how to create one tailored specifically for the Historical On Call report.
In this example, we'll come up with a way to easily see the total number of on-call minutes that we need to pay each user for, and be able to validate the data by drilling down into the totals and schedules.
Preparing and selecting your data
To begin, you'll need to generate a Historical On Call report from your xMatters instance, as explained in the online help. (If you'd rather just follow along with a completed pivot table, I've attached the finished example I made using this tutorial to the bottom of the article.)
Once you've generated your report, open it up in Microsoft Excel and let's have a look at what we've got.
Generate your own Historical On Call report
That's ... kind of a lot, I know. Let's break it down a bit.
For starters, each row of the report represents onesegment - a block of uninterrupted on-call time for a single user at the same escalation level (primary vs. secondary vs. tertiary, etc.).The data is sorted this way because shifts can span multiple days or weeks or months or just never really even end at all. By breaking shifts into segments, you don't have to wait for a shift to end before you can start calculating on-call time. This also makes it easier to account for temporary absences, rotations that move people out of primary on-call responder positions, and other changes in each person's schedule.
Also, remember that 24x7 shifts without escalations are not included in the report these are considered broadcast groups, not on-call schedules.
Creating your pivot table
To create a pivot table (or, as it's known in Microsoft Excel, a PivotTable), click the Insert tab, and then click PivotTable.
In the dialog box, you'll notice the entire worksheet is selected as the range, and so is the option to create the PivotTable in a new worksheet.
Both of these options are exactly what we want, so go ahead and click OK.Excel adds a new worksheet and opens the PivotTable Fields window so we can start sifting through our data.
Building your pivot table
Here's where we want to take a moment and think about what data we want to use to build our PivotTable. We know that, obviously, we want to get the total number of minutes each user spends on-call. But we also want to be able to limit that total to only the times each user held a primary or secondary on-call position. Oh, and of course, to make sure we're billing the right department, we need to know which groups the user was on call for!
Let's start with our most important value: the total number of on-call minutes. We'll make that the basis of our entire report by dragging the On-Call Time (mins) from the list of available fields and dropping it into the Values area.
That's pretty cool, but we know we don't want to pay out almost 25,000 minutes of on-call time, so let's add a few more details to help us tease out some more useful figures.
Drag and drop these fields to the Rowslist in the PivotTable Fields area:
User ID: We need this so we can tell who gets paid.
Group Name: This tells us which group the user was on call for.
On-Call Shift Name: This tells us which shift within each group the user was part of.
Shift Occurrence Start: This helps us identify the date and time of day for each shift where the user accumulated on-call minutes.
Already we're seeing a much more understandable breakdown each user's total, followed by the total for each group, and the total for each shift within the group.
Some of those still seem a little high, so let's try limiting the report to only those minutes where the user was either primary or secondary on-call.
Drag and drop the Escalation Level field into the Filters area, and notice how Excel adds it to the top row of the worksheet. Click the All drop-down field, and then select the Select Multiple Items check box. Adjust the checked items so only 1 and 2 are selected.
Click OK and review the updated PivotTable.
That immediately looks a lot different! Our first user has disappeared entirely because his on-call minutes were all escalation level 5 (management, amiright?) and our totals have reached much more manageable levels.
Customizing it your way
This is really just a quick and simple example about how to use pivot tables to analyze, organize, and summarize your exported Historical On Call report. There are lots of ways to use the available fields to quickly find and total up the information that matters to you.
If your on-call schedules are based on response windows (for example, only those people scheduled to respond within the first 15 minutes of an incident are considered to be on-call), you can use Escalation Time as your filter instead of Escalation Level.
If your shifts tend to be broken up a lot, or if you're looking for potential coverage gaps, use Segment Time Start instead of Shift Occurrence Start.
Use the Absence and Replacement Time fields to curate lists that let you see who's replacing who on a regular basis.
You can create multiple, independent PivotTables on different worksheets and have them all showing you different information and applying different filters while using the same underlying data.
Try it for yourself! and run it through the PivotTables feature a few times, or download the attached file and play around with some simple sample data.
View ArticleQuestion
Is there a way to get early access to new features?
Answer
Yes!
Releases to production environments occur on a quarterly cadence. Customers who want to previewnew features prior to quarterly releases can opt their non-production environments into theEarly Access Program (EAP). Customers enrolled in the EAP will see new features before aquarterly release.
If you're considering signing up for EAP, there are a few important things to know:
You can enroll in the early access program atany time by submitting a support request.
Opting out ofthe early access program can only be done at the start or end of a quarter
For more information, see xMatters Deployment & Early Access.
View Article
Working at xMatters
xMatters's Competitors
Recently Asked Questions
- What do you wish you knew before starting at Parallon Business Solutions?
- What kind of interview questions do they ask when hiring someone in the Sales department at DXC Technology?
- What kind of people does Carnegie Learning like to hire?
- What kind of people does Rippling like to hire?
- What Sales department questions did DXC Technology ask during your job interview?