Sumo Logic FAQs

If your logs are delivered in a multi-line format you may want to parse up until a line break in the message. In order to do so, use the following regular expressions as a stop anchor on the line break:

Linux Logs- "\n"

Windows Logs- "\r"

For example, if we have the following message in our logs:

12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: To: [email protected] Subject: New line Breaks in Message

To get [email protected] the ToAddress column., you can use the following queries to get the address:

... | parse "To:*\n" as ToAddress nodrop

Or

... | parse "To:*\r" as ToAddress nodrop

On December 31, 2019 the Node community will stop support for Node.js 8.x and will provide no further bug fixes, improvements or security updates after this date. In order to ensure you continue to run on a supported and secure runtime environment, AWS recommends updating to a more recent runtime version and will be deprecating the Node.js 8.x runtime in the following two stages.

Starting on January 6, 2019 AWS customers will no longer be able to create new functions using the Node.js 8.x runtime.

Starting on February 3, 2020 AWS customer will no longer be able to update functions, which use the Node.js 8.x runtime, however, functions will continue to process invocation events.

Sumo Logic has validated that there are no compatibility issues with the running of existing functions under the Node.js 10.x runtime.

Sumo Logic has already updated all SAM applications and existing CloudFormation templates with the Node.js 10 runtime and recommend re-deploying the new functions as they fix all deprecation warnings.

We expect the sumologic-process-logs AWS Lambda blueprint, to be updated by December 15, 2019, ahead of the deprecation date.

Problem:

You have created or updated a query and while trying to add/update it to a dashboard panel you get the following pop-up error:

https://help.sumologic.com/Visualizations-and-Alerts/Dashboards/Use-Time-Ranges-and-Filters/03Set-the-Time-Range-in-Dashboards

Possible causes and solutions:

This might be due to the length of the query you are trying to use to create a new panel. Queries built for dashboards/panels have a limitation of 10,240 characters, if you go above this number the panel will fail to be added to a dashboard with the error "There was a problem adding panel to the dashboard." You have to optimize your query to use less than10,240 characters.You can use a tool likewww.lettercount.comto count the characters in your query.

Check the time range selected, dashboard panels are limited to a 32-day maximum time range. A time range longer than 32-days will result in this error. For more information about setting the time range in dashboards, please go to .

If none of the above solutions solve your issue, then please reach out to the Sumo Logic Technical Support Team.

All data sent to Sumo Logic is indexed together within a defined partition or within a general partition. To help maintain compliance the indexes created for these partitions arethen stored in a Write Once Read Many (WORM) storage, which cannot be modified after creation.

You may request data be deleted from your account, however, this deletion will cover ALL data delivered into the account within a specified partition and for a specified time range. As of now, it is not possible to delete only specific messages sentinto Sumo Logic.

Any request for data deletion will need to be sent to Sumo Logic Support via your account administrator. In order to ensure the fastest processing of your request please include the following information when making a request for data deletion.

The name of the partition containing the data to be removed.

The time range, along with timezone, that returns this data in a search. This time range should be based on the receipt time of the message. To confirm the time range you may use the "Use Receipt Time" option found under the time range selection of your search.

Please CC an additional administrator on your support request as a secondary verification contact.

Analternative for deleting the data from the accountwould be tocreate Role filters that will hide the unwanted data so it is not searchable by users within the account. To do this one can apply a role filter matching this data to the user roles for whom the unwanted data should not be visible.

If you have an existing Collector and Syslog Source running on your host you can use the following to forward messages from journald to that Source.1.) In /etc/systemd/system directory create a new "unit" file called sumocollector.service with the following contents.

[Unit]Description=Send Journalctl to Sumo [Service]TimeoutStartSec=0ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat --udp localhost 514'Restart=alwaysRestartSec=5s[Install]WantedBy=multi-user.target

2.) Update the "ExecStart" command to point the output to your Sumo Logic Syslog Source. Assuming the Collector is on the local host and your Syslog source is configured to use UDP on port 514 the above template should work as is, however here is another example that would forward on TCP to a remote host on port 1514

ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat 192.168.1.2 1514'

3.) Run the following commands to enable and start the service

$ sudo systemctl enable /etc/systemd/system/sumocollector.service $ sudo systemctl start sumocollector.service

(Note: Restarts of this service will be handled by systemd)Once these commands are runyour data should start to be forwarded appropriately to Sumo Logic from the collector via your Syslog Source. You can run the following command to view the local logs and ensure the new unit has started.

$ journalctl -f -u sumocollector.service

The process for canceling your account with Sumo Logic will depend on your current account type. There are different processes depending on whether your account is currently under the Free/Trial tier or is under one of Sumo Logics paid Professional or Enterprise subscriptions.

Free/Trial Accounts

The account owner of a Sumo Logic Free or Trial account may delete his or her organization from Sumo Logic at any time, which will close the account permanently and delete all users and data from Sumo Logic.

Warning: The following actions cannot be undone once completed.

To delete a Sumo Logic Free/Trial account

Go toAdministration >Account.

At the bottom of the page, under the sectionDelete this Organization from Sumo Logic?, clickDelete Org.

In the dialogDelete This Organization from Sumo Logic?, enterDELETEto confirm.

ClickDelete Org.

The Sumo Logic Organization, all users and data are immediately deleted from Sumo Logic.

Paid Professional and Enterprise Accounts

If you would like to cancel your organization's Sumo Logic account and subscription you will need to contact your Sumo Logic account representative, or you may open a ticket with Sumo Logic Support requesting the cancelation of your service. The Sumo Logic accounts team will further assist you with the cancelation process.

You can POST logs (text files or .gzip files) directly to Sumo Logic HTTP Source (endpoint).

Some best practices/tips to consider are:Logging Appenders:

Use our logging SDK's (.Net, Javascript, Log4J, Log4J2, etc.)

Best practices:

It's best to build mobile logging asynchronously so that the app 'fires and forgets' about the log and doesn't freeze/interrupt the app

Hardcoding the Sumo endpoint into the logging framework is not advised (someone could delete it).

Performance-wise it's advised to send the logs to a queue so that data can be buffered, packaged, compressed and the data forwarding from there to Sumo can be decoupled from the logging code

Another consideration is whether or not the device has data/internet, or if the mobile app has offline capabilities

Symptoms:

When using Local File Management mode and updating the source.jsonfile to remove a Source from a Collector the Source is not getting removed and continues to ingest data.

Possible Cause:

The JSON supplied to the Collector cannot be validated by the Collector and as a result, is not applied. This could be due to multiple filters supplied with the same "Name" key within a Source.

The Collectors log files should contain more information on any JSON validation issues that might be occurring. Within the collector.log file, you will see messages with the following content.

com.sumologic.scala.collector.auth.CollectorRegistrationManager - [main] Response code is 200 with warnings:

Within this message, there should be more information on any validation issues encountered.Full Example:

2017-06-09 13:46:10,474 +0200 [JsonSync Manager] INFO com.sumologic.scala.collector.auth.CollectorRegistrationManager - [main] Response code is 200 with warnings: List(Key: validation.fields.invalid, Message: All filters must have different names.), errors: List()

Resolution Steps:

If you correct validation issues (such as removing duplicate filter name TEST123 from the affected sources), you should be able to save the file after deleting the sources, and see the changes taking effect in Sumo UI (no collector restart required).

After performing these steps, and if this is resolved, in collector.log you should see the following message with an empty list of warnings and errors:

2017-06-15 11:34:11,513 -0700 [JsonSync Manager] INFO com.sumologic.scala.collector.auth.CollectorRegistrationManager - [main] Response code is 200 with warnings: List(), errors: List()

2017-06-15 11:34:11,513 -0700 [JsonSync Manager] INFO com.sumologic.scala.collector.JsonFileSyncManager - Sources successfully synchronized

A "503 error in heartbeat manager" or "Service returned a 503, marking the collector as offline" is a response message from the Sumo Logic service that lets the Collector know that the service is currently not available to process the message request. When this occurs, the log messages that were attempting to be sent, are stored in the Collector cache. They are then resent as part of the next request to the service.

This is typically a temporary condition. If this error continues to occur for an extended period of time, please check the http://status.sumologic.com/ status page for documentation on any service-related issues.

Problem:

Based on the messages ingested against the source category used in my scheduled search "A B C", I should have received an alert.

Cause:

This can be caused by timestamp, timezone issues, latency of ingestion, or count incorrectly configured.

Resolution:

1.) Email sent bounced

Email sent by Sumo Logic but not received by the recipients of the email address for reasons such as the email address was meant for internal distribution only or email address was on the AWS Suppression list and therefore could not be delivered.

Run a query against the Audit Index if an alert was triggered for the given schedule search

_index=sumologic_audit _sourcecategory=scheduled_search triggered "<My Alert Name>"

If the above query against the Audit Index shows that an alert was triggered for the given schedule search and the email was not received, then please investigate potential causes for email blockage

2.) Timestamp parsing or timezone issue:

Run the query for the scheduled search in the search tab with "Use Receipt time" checked.

LinkedIn

Observe the message time and the receipt time for the messages that should have triggered the alert and also the timestamp inside the message.If you find a gap of hours between the message time and receipt time, you would suspect a default timezone configuration setting issue for the collector and/or source.

Following is an example: You can see the message is ingested or received by Sumo Logic at 11:41 CDT (Central time or UTC-05:00) but the message time assigned is 06:41 CDT. This implies the default collector or source time zone was assigned to the UTC timezone. This can be either corrected by editing the collector and/or source timezone to Central time using DST (America/Chicago for example) or by specifying a custom timestamp to the source which requires the timezone to be read from the message.

3.) Latency in ingestion of message:

If the gap between the message time and receipt time is of the order of say 5-15 minutes as could be the case of ingestion of S3 objects or cloudtrail data. In the following example, a 15 minute scheduled search executed for time range 7:30 to 07:45 did not see any data because the data for the time range ingested after the scheduled search executed.

One solution is to specify an offset to the time range of the query to account for the ingestion delay. If, for example, the ingestion delay was 15 minutes and the time range specified was -15m, the solution would update the time range it to -30m -15m to allow the scheduled search process 15-minute older data.

A second solution is to specify "Use Receipt time" in scheduled search configuration so that the query in the scheduled search will evaluate messages received or ingested in the specified time range

4.) Count incorrectly configured

I have an alert configured to trigger when the count of a condition reaches a certain value, however, I am not receiving any alerts. My query is:

_sourceCategory=aws/prod| json "message","logStream","logGroup"| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status| timeslice 1m| where action="REJECT"| count as drops by _timeslice

The threshold set for my Scheduled Search alert is:

Greater than > 1000

This query returns results where the "drops" count is more than 1000, so why am I not receiving my alerts?

The answer is: thresholds set within a Scheduled Search are based on the number of result rows returned with a query and do not consider any values that may be present within a column of those rows. If your query does not perform any aggregations the Scheduled Search threshold will apply to the number of raw messages returned with a query, as seen under the Messages tab of the search. If a query contains an aggregate operation, for example, count, sum, min, max, etc... the Scheduled Search threshold will be applied to the number of aggregate rows returned by the query, as seen within the Aggregate tab of the results.When performing an aggregation as part of a query, and wanting to alert when a specific aggregate value meets a threshold, the threshold for that field value will need to be included as part of the query itself. This can typically be done by providing a Where condition after the aggregation within the query. For example:

_sourceCategory=aws/prod| json "message","logStream","logGroup"| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status| timeslice 1m| where action="REJECT"| count as drops by _timeslice| where drops > 1000

This will cause there to only be a results row returned where the field value meets the threshold provided within the query. The threshold set within the Scheduled Search would then be set to alert based on the resulting number of rows that met the threshold set within the query. For example:

Greater than > 0

Facebook

Twitter

When running a script calling the Search Job API I am sporadically receiving the following error message with some of the requests. I have verified the job ID is valid per the initial request, so why do we receive this message?

{ "status" : 404, "id" : "X7X3X-EDY9M-FORQQ", "code" : "searchjob.jobid.invalid", "message" : "Job ID is invalid." }

Answer:

When using the Search Job API you will need to ensure you store the cookie from the original search job request and then pass this cookie back with any follow-up requests to get the results of that job. The cookie is needed to tell the Sumo Logic load balancers which service node is managing the generated search. If you do not pass back the cookies with the request its possible for your request to hit a node that is unaware of the search and a 404 error will be returned.

Note: The following examples will need to be performed against the APIendpoint for your account deployment. For more information on determining the endpoint needed for your account please see the following documentation.

Examples using cURL: Start the job:

curl -b cookies.txt -c cookies.txt -H 'Content-type: application/json' -H 'Accept: application/json' -X POST -T createSearchJob.json --user ACCESSID:ACCESSKEY https://api.sumologic.com/api/v1/search/jobs

Request the status:

curl -v --trace-ascii - -b cookies.txt -c cookies.txt -H 'Accept: application/json' --user ACCESSID:ACCESSKEY https://api.sumologic.com/api/v1/search/jobs/12345678999999

Request message results:

curl -v --trace-ascii - -b cookies.txt -c cookies.txt -H 'Accept: application/json' --user ACCESSID:ACCESSKEY 'https://api.sumologic.com/api/v1/search/jobs/12345678999999/messages?offset=0&limit=10

Problem:

When attempting to restart offline Collectors the restart fails and the following messages are found within the /<sumoHome>/logs/collector.log file.

2019-07-05 14:47:45,941 +00:00 [WrapperSimpleAppMain] WARN com.sumologic.scala.collector.auth.CollectorRegistrationManager - [main] Credentials no longer accepted because collector is deactivated.2019-07-05 14:47:45,941 +00:00 [WrapperSimpleAppMain] INFO com.sumologic.scala.collector.Collector - done.2019-07-05 14:47:45,942 +00:00 [WrapperSimpleAppMain] INFO com.sumologic.scala.collector.Collector - * Collector in deactivated state.2019-07-05 14:47:45,942 +00:00 [WrapperSimpleAppMain] INFO com.sumologic.util.state.ScalaTransition - Received event 'deactivate', transitioning from state 'Stopped' to state 'Deactivate'2019-07-05 14:47:45,942 +00:00 [WrapperSimpleAppMain] INFO com.sumologic.scala.collector.Collector - Deactivate collector...2019-07-05 14:47:45,945 +00:00 [Thread-2] INFO com.sumologic.scala.collector.Collector$ - Executing collector shutdown hook...2019-07-05 14:47:45,945 +00:00 [Thread-2] INFO com.sumologic.scala.collector.Collector - COLLECTOR STOPPING

Resolution:

These messages may be seen when a Sumo Logic Free account has gone over the ingest or storage usage limits and Collection has been temporarily deactivated. Once in this state Collectors will be unable to be restarted until ingest or storage quota has fallen back below the allowed limits for this account level.

Sumo Logic free accounts have the following ingest and storage limits.

Ingest: Average of500MBper day, up to 4 GB total over a 7 day period.

Storage: 4 GB Total

The current usage volume for ingest and storage can be viewed within the Administration > Account page.

Deactivation due to Ingest Limits

If a deactivation was due to high ingest volume Collection will only be re-enabled once ingest quota becomes available. Depending on the amount of data recently uploaded it could take between 1 to 7 days for the quota to become available.

Deactivation due to Storage Limits

If a deactivation was due to high storage volume Collection will only be re-enabled once storage becomes available. The time it may take for storage to become available will depend on the retention period set on the partitions or general index within the account. Retention periods may be adjusted (ie. lowered) by:

Going to the Manage Data > Settings page of your account.

Open the Partitionstab of the Settings page.

Select the hamburger icon to the far right of a selected Partition and then select Edit Retention Period.

Enter a new retention period for the index.

Once the retention period is updated any data received into the index prior to the new retention period will be purged from Sumo Logic and storage quota will start to become available within the account.

Problem:

At times I see below error messages in the collector.log.

2018-02-27 11:54:33,818 +1100 [Thread-112] WARN com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender - Unable to forward message to 10.65.85.21-514java.util.concurrent.RejectedExecutionException: Task com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender$$anon$2@4892d719 rejected from java.util.concurrent.ThreadPoolExecutor@8110635[Running, pool size = 1, active threads = 1, queued tasks = 512, completed tasks = 144926]at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)at com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender.process(SyslogForwardingPipeline.scala:152)

Resolution:

This error means the current volume of messages matching the Syslog forwarding rule is too high for the Collector to keep up with.

There is a single thread that polls from a queue (default size of 512 items). Upon dequeue, it will use a single-threaded TCP/UDP connection to send the event. The issue is that the 512-capacity queue is full, meaning that the Collector is not sending fast enough.

If the data flow varies over time (high throughput to low throughput), you may benefit by increasing the queue size to give more room for backpressure.

1. Stop the collector service

2. Add the following line to collector.properties (Found within the Collector Installation directory under the /config folder) (default is 512):

forwarding.syslog.maxCapacity = 1024

3. Start the collector service

Problem:

I create a rule which has the correct results in the sample windows. Then I select "count messages" and select one field as a dimension but the result of this log-to-metric rule is always shown as "1"

Solution:

By default, Sumo Logic will quantize using "avg" (average) so unless the query explicitly quantizes using sum or count, the count of metrics from logs-to-metrics will always yield a value of "1" even if the Logs2Metrics(L2M) rule is configured correctly.

In order to get the correct result count, you have to append "quantize using count" or "quantize using sum" to your queries on these data points to determine the count of the log lines.

For example:

_contentType=MetricFromLog metric=logins | quantize using count

or

_contentType=MetricFromLog metric=logins | quantize using sum

Problem:

We have a publicly accessible dashboard.It contains a geo location map widget that no longer displays correctly and insteaddisplays "Loading" forever. I'm able to see the map widget display correctly when logged into Sumo Logic UI.

This started after the query was updated in the panel supporting the geo location map widget.

Resolution:

From the dashboard, you can click the toggle to go intoLive Mode.

Cause:

A public or whitelist dashboard with a shareable URL is always view-only and has to be in live mode to receive continuous updates. The issue arose after the edit of the dashboard disabled Live mode on the dashboard and Live mode has to be re-enabled on the dashboard after the edit.

Problem:

When trying to share a dashboard to whitelist or to the world, the user receives a "Permission Denied" error when attempting to save.

Create a New Role

Resolution:

Permission denied error when attempting to share a public or whitelisted dashboard may occur if one of the following steps has not been performed.

Step 1. Enable the Sumo Logic Policy

In this step you enable sharing of dashboards outside your organization. You must have Admin privileges to enable the policy.

Go toAdministration > Security > Policies.

Select theEnablecheck box forShare Dashboards Outside of the Organization.

The policy is changed. It can take up to 5 minutes for the policy to go into effect.

Step 2. Verify your role allows external sharing

You must have specific role capabilities to share a dashboard with external users:

TheShare Dashboards outside of Organizationcapability is required to share dashboards with users who do not have a Sumo account.

TheShare Dashboards with the Whitelistcapability is required to share dashboards with users on the whitelist.

TheShare Dashboards with the Worldcapability is required to share dashboards withexternal user users.

If you do not already have a role that grants you the required capability, an Admin user can assign you one, as described in Assign Roles to Users, or create a new one as necessary, as described in.

Problem:

We have noticed that on one of our Local File Sources that a few of the log lines are not appearing to be sent to Sumo Logic. My log file has a rotation policy applied that will rotate the file every 500KB and will keep up to 10 rotated files. Is there a way to ensure the Collector gets the last bit of log messages that occurred just prior to the rotation of the file?

The logs look something like the following.

/var/log/mylog.log/var/log/mylog.log.1/var/log/mylog.log.2

And the path expression in our Source is configured as:

/var/log/mylog.log

Cause:

The Sumo Logic Collector will periodically scan the provided path expression looking for any new files that match that path and name. When a file is found it is opened for reading for a short period of time before the Collector will close the connection and then rescan the path expression. The Sumo Logic Collector keeps track of where it left off within a file by using a fingerprint that matches the first 2KB of a log file, as well as the last known byte position, read from the file. When the Collector scans the provided path and locates any files that match the expression it will then open and compare the first 2KB of those files to known fingerprints it has on record and if it finds a matching fingerprint it will then continue to read the file from the last known byte position. If a file rotation happens to occur at a time where the Collector does not have an open connection to the file it is possible the Collector may not get the last few messages, since that file has now been rotated to a different name which is not found given the path expression supplied.

Solution:

To address this problem it is recommended that within your Source you configure a path expression that will also match with the rotated versions of the file. This will ensure that the Collector opens the rotated file to get the last bit of data it was unable to read prior to the rotation. So for the above sample logs you would provide a path expression set like the following.

/var/log/mylog.log*

The wildcard added to the end of the file name will ensure that the Collector will find the rotated version of the file during the scanning process. The Collector will then open that rotated file, compare that to the fingerprint it had on record for the original version of the file, and will then read any new data from the last known byte position. This will ensure the Collector captures any messages that were added to the file just prior to the rotation

Problem:

When running a curl request to a Sumo Logic endpoint I am receiving the following errormessage.

About to connect() to api.sumologic.com port 443* Trying 184.72.240.213... connected* Connected to api.sumologic.com (184.72.240.213) port 443* successfully set certificate verify locations:* CAfile: /etc/pki/tls/certs/ca-bundle.crtCApath: none* SSLv2, Client hello (1):Unknown SSL protocol error in connection to api.sumologic.com:443* Closing connection #0 curl: (35) Unknown SSL protocol error in connection to api.sumologic.com:443

Cause:

This error is due to an unsupported version of TLS being used with the connection. On June 20th, 2018 Sumo Logic removed support for TLS 1.0 and TLS 1.1 from all public endpoints and now requires all connections to occur via TLS 1.2.

Resolution:To address the reported error you can:

Upgrade to the latest version of curl. To verify the curl version type curl -V. TLS1.2 is the default used by Curl versions 7.34 has defaultsupport for tlsv1.2

Use the --tlsv1.2 option. For example

curl --tlsv1.2 -u 'accessid:accesskey' -X POST -H "Content-Type: application/json" -T hosted_collector.json https://api.sumologic.com/api/v1/collectors/collectorid/sources

The If operator can be used to construct new fields for calculating things like the ratio of400 or 500 error codes to total requests. This example will show how to calculate this error ratio, then apply the Outlier operator to alert on sudden spikes in this ratio.I'll use Nginx in this example, but this use case can easily be applied to IIS, Apache, Load Balancer, and other logs with success and error status codes, or any example where you need to calculate and monitor the ratio of an occurrence of an event over time.Step 1: Parse Status Codes

_sourceCategory=prod/nginx| parse "HTTP/1.1\" * " assc

Step 2:Create 400s and 500s Counter Fields

_sourceCategory=prod/nginx| parse "HTTP/1.1\" * " as sc| if(sc matches "4*", 1, 0) as sc_400_counter| if(sc matches "5*", 1, 0) as sc_500_counter

Step 3:Createthe Denominator and Numerator for your Ratio over Time

Here, we calculate the ration of 500s to total requests, 400s to total requests, and all errors (500s + 400s) to total requests:

_sourceCategory=prod/nginx| parse "HTTP/1.1\" * " as sc nodrop| if(sc matches "4*", 1, 0) as sc_400_counter| if(sc matches "5*", 1, 0) as sc_500_counter| timeslice 15m| count as total_logs, sum(sc_400_counter) as sc_400s, sum(sc_500_counter) as sc_500s by _timeslice

Step 4:Createthe Ratio Fields and Hide Intermediary Fields

Here, we use some simple division on new query lines to create and name the ratios we want. We also use the Fields operator to explicitly show only the fields we want to display, hiding the unnecessary ones:

_sourceCategory=prod/nginx| parse "HTTP/1.1\" * " as sc nodrop| if(sc matches "4*", 1, 0) as sc_400_counter| if(sc matches "5*", 1, 0) as sc_500_counter| timeslice 15m| count as total_logs, sum(sc_400_counter) as sc_400s, sum(sc_500_counter) as sc_500s by _timeslice| (sc_400s/total_logs) as sc_400_ratio| (sc_500s/total_logs) as sc_500_ratio| ((sc_400s+sc_500s)/total_logs) as sc_errors_ratio| fields sc_400_ratio, sc_500_ratio, sc_errors_ratio, _timeslice

Step 5:Apply Outlier to the Ratio Over Time

Next, we use Outlier and apply it to one of our ratios. I've selected the 'sc_errors_ratio' field(all 400s and 500s) in this example:

_sourceCategory=prod/nginx| parse "HTTP/1.1\" * " as sc nodrop| if(sc matches "4*", 1, 0) as sc_400_counter| if(sc matches "5*", 1, 0) as sc_500_counter| timeslice 15m| count as total_logs, sum(sc_400_counter) as sc_400s, sum(sc_500_counter) as sc_500s by _timeslice| (sc_400s/total_logs) as sc_400_ratio| (sc_500s/total_logs) as sc_500_ratio| ((sc_400s+sc_500s)/total_logs) as sc_errors_ratio| fields sc_400_ratio, sc_500_ratio, sc_errors_ratio, _timeslice| outlier sc_errors_ratio

Step 6:Alert on Spikes in the Ratio of Errors to Total Requests

Finally, we'll filter down the data to just the spikes (pink triangles) so that we can alert on these. Just click 'Save As' then Schedule this Search' to set up an Alert!

_sourceCategory=graham/travel/nginx| parse "HTTP/1.1\" * " as sc nodrop| if(sc matches "4*", 1, 0) as sc_400_counter| if(sc matches "5*", 1, 0) as sc_500_counter| timeslice 15m| count as total_logs, sum(sc_400_counter) as sc_400s, sum(sc_500_counter) as sc_500s by _timeslice| (sc_400s/total_logs) as sc_400_ratio| (sc_500s/total_logs) as sc_500_ratio| ((sc_400s+sc_500s)/total_logs) as sc_errors_ratio| fields sc_400_ratio, sc_500_ratio, sc_errors_ratio, _timeslice| outlier sc_errors_ratio| where sc_errors_ratio_violation > 0

Question:

I am unable to whitelist outbound access by DNS names. How do find the list of IP addresses used by Sumo Logic so I can add these to my firewall rules?

Answer:

Sumo Logic runs within AWS and utilizes dynamic load balancing to manage incoming requests to the service. Due to the dynamic nature of the service, Sumo Logic cannot guarantee a static set of IPs will always be used so you will need to whitelist the range of possible IPs based on where your account is located.

The first thing to check is the Sumo Logic deployment your account is currently registered on, which was selected at the time of account sign up.The easiest way to see which deployment your account uses is to look at the Sumo Logic URL after you have logged into your account. For example, if you see "us2" that means you're running on the US2 pod. If you see "eu" or "au" you're on one of those pods.

AWS IP Address Ranges

Once your deployment has been determined you will need to determine the AWS Region where your account is managed in order to determine the list of IPs that may be used.Once you know which AWS region your account is hosted under you can get the list of IPs for that region by downloadingthe AWSIP Address Ranges JSON file from thedocumentation. AWS supplies some tools for filtering through the list of IPs found within this JSON file. For the purpose of whitelisting outbound access to Sumo Logic you will need to use the IPs for the "Amazon" and "EC2" services for your selected region.

Problem:

When attempting to install the Collector on Linux OS flavor machine, if it failed with any of the below errors:

1. "Could not display the GUI. This application needs access to an X Server"

*******************************************************************

You can also run this application in console mode without

access to an X server by passing the argument -c

******************************************************************

OR

2. "java.lang.ArrayIndexOutOfBoundsException: 0"

Exception:

java.lang.ArrayIndexOutOfBoundsException: 0at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:79)at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:97)at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)

Solution:

You can try the "-c" option to run the application in console mode.

$ sudo ./SumoCollector.sh -c -q -Vsumo.accessid=<myaccessid> -Vsumo.accesskey=<myaccessid>

Or, if it continues to fail, you can try installing this fonts package for your Linux distribution:

This Google link talks about how to update centos to fix the error:

https://communities.ca.com/thread/241790600-rehl-74-release-automation-installation-error

The root cause of this issue is due to the missing the font library on the server.

Installing the required font package issue will address the issue.

Run the below command on the required servers:

yum install dejavu-serif-fonts

Depending on the Centos flavor, you should install dejavu-serif-fonts package to overcome this issue.

You can't delete Field from the field management page until you remove the reference from the Field Extraction Rule (FER).

Deleting a field does not delete historical data assigned to that field. If you delete a field by mistake and one or more of those dependencies break, you can re-add the field to get things working properly again.

You should always disable a field and ensure things are behaving as expected before deleting a field. Please follow this link to get more information on deleting a field from the new field management page.

For each machine running Varnish:

Add the following line to /etc/rc.local:

$ varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid

Configure log rotation by adding the following line to

$ /etc/logrotate.d/varnish

(create it if it doesn't exist):

/var/log/varnish/*log {

create 640 http log

compress

postrotate

/bin/kill -USR1 `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true

endscript

}

Add a local file source to your collector pointing to the path(/var/log/varnish/*.log) defined above

If the search queries, scheduled search, and dashboard are identical, you can use export and import content feature to export content from one instance and import it to another instance.

In the Library, you can export content as JSON, including whole folders with subfolders, saved searches, and Dashboards. Then you can import the content as JSON into the Personal folder in the same Sumo Logic organization. All content names must be unique. You can also export and download your content as a JSON file to import it into another Sumo Logic organization.

Export Content in the Library

1. In the Library, do either:- To export a saved search or dashboard, navigate to it, click the details icon for that item, and select Export.- To export a whole folder (with subfolders), click the details icon for that folder select Export.

2. In the Export Item dialog, do either:- Click Copy to send it to the clipboard.- Click Download to download the content as a JSON file.

3. Click DONE to close the window.

Import Content in the Library

To import content into the same Sumo Logic organization that you exported it from, you must change the name of the content item. Or you can import it into a different organization using the JSON file you downloaded in the previous step.

1. From your Personal folder in the Library, click the details icon.2. Select Import.3. In the Content Import dialog, for Name, enter a unique name for the content. (If you are importing this content into a different Sumo Logic organization that does not already use this name, you can keep it the same.) For this saved search, we changed the name to append it with IMPORT.4. Paste the JSON you copied from the JSON dialog to the clipboard, or from the JSON file you downloaded.5. Click Import. The Import button is only available if you have pasted in a valid JSON.6. Your imported content is listed in the Personal folder. 7. From the root in the Personal folder, you can move content to organize it in another folder in the Library.

Limitations:

1. Recent Searches, Favorites, Scheduled Views, Partitions, Field Extraction Rules (FERs), and lookups are not supported.2. For content that is Shared, only the content is imported, not the Shared state. You can share it again as necessary. 3. Content may only be imported into the root of the Personal folder. From there you can move it into another folder to organize your content.4. All content names must be unique within a Sumo Logic organization. 5. Only error-free JSON will import successfully. If you modify your JSON manually after export, we recommend using any JSON validator to confirm that you are importing valid JSON.6. Import is limited to 1000 objects at a time.

Sumo Logic has two limitations on the Lookup files:

The size limit for the CSV file is 8MB. If your file is larger than 8MB, you will see this error

If using an HTTPS resource, the file must be downloaded within 10 seconds. If the file cannot be downloaded in 10 seconds, it is probably too large.

Sumo Logic does not provide the webhook template for the google hangout, but you can create a generic webhook connection in Sumo Logic for the same. Please see here to know how to setup webhook connections

You need to follow the steps in the below article from google to generate a Google Hangout webhook URL, which you can then use within Sumo Logic's generic webhook. You will need to check acceptable JSON payload from google hangout from the below link and you can use the same payload while creating generic webhook connection.

https://developers.google.com/hangouts/chat/how-tos/webhooks

The field value passed to the formatDate operator needs to be in milliseconds and should have type "long".

There could be below possibilities that can lead to this error

1. The field that you are trying to pass to a "formatdate" is in milliseconds but it is not of "Long" datatype and hence as a part of the parsing it results in this error. The solution is to convert those milliseconds into "Long", by using "toLong" as shown below

| formatDate(toLong(<field>), <date_format>) as <new_field>

2. The field that you are trying to pass to a "formatdate" is a timestamp in non milliseconds format

For example: You are doing below, where my_time has value "11/01/2019 12:52:27.881"

| formatDate(my_time, "MM/dd/yyyy HH:mm:ss.SSS") as my_time1

This will results in "Multiple definitions found for function formatDate(String, String)."

To solve this, we need to perform below

| parseDate(my_time, "MM/dd/yyyy HH:mm:ss.SSS") as my_time_milliseconds //convert date into milliseconds| formatDate(my_time_milliseconds, "MM/dd/yyyy HH:mm:ss.SSS") as my_time1 //apply formatDate

For more detailed information see here

When Collectors first startup they attempt to make an initial connection to the service in US1. If the customers Organizationsu is within another deployment, US2, Dub, Syd, JP or Germany etc the service will properly redirect the Collector to the Collector service URL for that deployment. This is not typically an issue, unless the customer has implemented firewall rules to allow outbound traffic to only the IP's associated with the deployment they have selected.

When this occurs the Collectors initial check to the US1 deployment will fail and the Collector will be unable to start.

To get passed this issue we need to tell the Collector which service URL to use for connecting to the selected deployment so it bypasses the initial request to US1. This can be done within user.properties file by adding the following parameter.

url= <deployment URL>

The deployment URLs for the current deployments are:

Deployment

URL

US2

https://collectors.us2.sumologic.com

Dublin

https://collectors.eu.sumologic.com

Sydney

https://collectors.au.sumologic.com

Japan

https://collectors.jp.sumologic.com

Germany

https://collectors.de.sumologic.com

Canada

https://collectors.ca.sumologic.com

FE

https://collectors.fed.sumologic.com

For more info and complete list of deployment URL see : Sumo Logic Endpoints and URL

Problem:

Why Excel rounds of the 18 digit numeric value to 15 places in a CSV file generated by scheduled search?

Resolution:

This is a limitation in MS Excel. If you open the .csv file in text editor then you would see complete 18 digit numeric value.

Excel follows the IEEE 754 specification on how to store and calculate floating-point numbers. Excel therefore stores only 15 significant digits in a number, and changes digits after the fifteenth place to zeroes.

Same is mentioned in MS article

https://support.microsoft.com/en-us/help/269370/last-digits-are-changed-to-zeroes-when-you-type-long-numbers-in-cells

For example "testId" is the column having 18 digits values. Say 123456789123456789, if you open the scheduled search generated CSV in Excel, you will see it as 123456789123456000.

There is a work around for that, you can use the CONCAT operator to add a non numeric character to testId, in below i am enclosing testId with double quotes

| concat("\"", testId, "\"") as testId

This way Excel will not detect it as Numeric and will not round up. And you will see testId as "123456789123456789"

Instead if double quotes, you can choose other non numeric characters as per your choice.

Problem:

Unable to install a new collector or the collector installation fails and goes offline.

Possible Cause:

Collector JVM has run into memory issues.

Log Message:

INFO | jvm 1 | 2017/02/17 08:18:03 | WrapperSimpleApp Error: java.lang.OutOfMemoryError: GC overhead limit exceededSTATUS | wrapper | 2017/02/17 08:18:03 | The JVM has run out of memory. Restarting JVM.

Solution:

The above messages from the collector.out.log file indicate that the collector is running out of memory.

In order to remediate that you need to go ahead and increase the collector memory to atleast a new value from what it is being set up currently as given below in the user.properties configuration file:

wrapper.java.maxmemory= <value>

Increase the value by doubling up the current value which should help.

After performing the above changes restart the Collector using the command:

[install_directory]/collector restart.

For more info: Increase memory in a Collector

Problem:

The Collector is up and running but has stopped pushing any new logs to Sumo Logic. Within the collector.log file we are seeing the following errors being reported:

Caused by: java.io.IOException: No space left on device

Cause:

During normal operation the Collector will temporarily cache log messages and store information regarding what files it has read to disk. The above error tells us that the host does not have the available disk space for the Collector to perform these actions.

Solution:

On your Collectors host server you will need to check the drive hosting your Collector, as well as your /tmp directory, has available disk space. After clearing up or adding more disk space on the host perform a restart of the Collector process on the host.

Additional Articles: Recommended free disk space requirement

Processing rules take effect immediately as soon as the configuration is saved. The collector restart is not required for adding/deleting or updating processing rules. However, you may see delay of 2-3 minutes to see the data in search because of the way the data is indexed by Sumo Logic.

Tokens are encrypted and stored in Sumo Logic using AES256-CBC with HMAC-SHA256 authentication and expire after a few hours and subscriptions are refreshed yearly.

Tokens are encrypted as soon as Sumo Logic gets them from O365 and are only decrypted when retrieving events keeping those secured.

Sumo Logic automatically updates them before they expire to prevent data loss.Successful and failedtoken-update events arelogged in the Audit Index. If the request fails Sumo Logic will continue to try to update a token for about a week. After several failures,we recommend recreating the Office 365 Audit Source.

Normally there is no additional charge but under some use cases, you could be charged extra as explained below.

Partition:If a data is duplicated in partitions then there will be a charge for the additionalcopy of the duplicated messages. If the partitions are created without an overlapping data set, then there is no charge for the ingestion.

For example, in the following example, the data in sourceCategory prod/Apache is targeted by two partitions one using a wild card and other using a specific sourceCategoryPartition1:_sourceCategory=prod/ApachePartition2:_sourceCategory=*/Apache

Refer the Best Practices for creating partitions

Schedule Views:

There are 2 use casesin schedule view where the customer will be charged:

If you are running a non-aggregate query(without any group by operator) to create a scheduled view then for those views that include raw data there will be a charge.

If you are running an aggregate query but if you count the log messages by the _raw field then it will consider that column result as raw data and in this scenario, there is a charge for this ingest.

To avoid a charge while creating scheduled views, please ensure that the queries are aggregated and avoid aggregating using the _raw field. Please refer following Doc link for more information.

The two most common approaches are Sumo Logics .net appender (log straight from the application to our HTTPS endpoints ) or using an installed collector with a local file source to scan logs written to files.

Yes, you can do that, using functionality called "Audit Index". Using the audit index you can see the record of the user's different activity in Sumo Logic.

For more info please use below link: User Activity Events Below is the query which you can use to get some user activity:

_index=sumologic_audit _sourceCategory=user_activity

Note: Only the users who have "Administrator" access can access the Audit index.

You can find this information in the "main.properties" file that can be found under "<Sumo_home>/config/creds directory.

For example:$ cat /Sumo\ Logic\ Collector/config/creds/main.properties#Auto generated.#Mon Nov 26 16:42:17 PST 2018obfuscated=trueapiId=151413262 . ---------> this apiId is the unique collector idkey=*******

In your deployment, you've probably created Dashboards with Panels that you use to keep an eye on specific issues or help you manage resources. These custom Dashboards are usually full of important data that you rely on.

In Sumo Logic App Dashboards, each Panel uses a very targeted search. In instances where no search results are returned, these Panels will not display any information. Instead, you'll see Panels that show the message "There is no data to display"

This does not mean that the app isn't working. There could be two reasons for the same:

1.) It means that no activity has occurred according to the search running behind the Panel.

2.) It means these panels use parsing operations in their underlying search query that expect a specific format. If the log data submitted does not fit the format of the parsing expressions the logs will not match the query and no data will be displayed.

Problem:

We have a Live Mode Dashboard that we have shared to a public URL, but when we access the Dashboard via the public URL, the Panels are not populating and we also see a message stating "Switching to Live Mode".

Resolution:

With the Dashboard in Live Mode, un-share and then re-share the Dashboard.

Cause:

The most common cause of this issue is if the Dashboard is shared before it is set to Live Mode, setting it to Live Mode after sharing will not cause the public Dashboard to be in Live Mode. So always make sure to set the dashboard in Live mode and then share it with public

Problem:

When trying to upgrade the collector from (19-182.25) to the latest version (19.245-4) it failed with the following java related errors.

INFO | jvm 1 | 2019/03/30 23:32:25 | WrapperManager: ERROR - The version of the Wrapper which launched this JVM is "3.5.13-st" while the version of the native library is "3.5.33-st".

Log File: Collector.out.log

Resolution:

The error message above is indicative of a known issue when upgrading a collector from version 19.209 or lower to version 19.216 or higher because of a wrapper update required to address a bug found with earlier wrapper versions.

The wrapper jar file currently in use is version "3.5.33" while the prior version of the wrapper, which launched the existing JVM is '3.5.13'.

This is documented in the Collector release notes for 19.216-33

Known Issues:

Downgrading from version 19.216 to 19.209 is not supported due to 19.216 having a new Tanuki wrapper.

When upgrading from version 19.209 to 19.216 on Ubuntu 16 and above using a shell executable file (.sh), Debian, or the Web UI you will need to manually restart the collector service after upgrading.

Upgrading Collectors without administrator privileges on Windows to 19.216 or higher will require a manual upgrade installation.

If you encounter this issue a manual start of the collector service should allow for completion of the upgrade.

The "name=name" property in user.properties is the name for the Collector. This parameter has a character length limit of 128 characters. Same information is mentioned here

Problem:

We have been losing log messages at the Sumologic collection receivers using hosted HTTP collectors that are serving as the endpoint for log messages (via log4j2.xml config file)

https://github.com/SumoLogic/sumologic-log4j2-appender

Cause:

Absence of configuration offlushAllBeforeStopping can cause current data in the buffer not to be ingested into Sumo Logic if the appender terminates abnormally.

Absence of configuration for maxQueueSizeByteswould utilize the default 1million bytes for the buffer size and if the size of the data set to be ingested exceeds that, then we have seen data not be ingested into Sumo Logic. Appender logs may or may not (depending on the version of the Appender) display the following example log message

2018-11-21 11:12:21,005 Log4j2-TF-1-AsyncLoggerConfig--1 WARN Evicted 1 messages from buffer

Resolution:

Ensure the following settings are configured correctly

flushAllBeforeStopping - is an optional setting that should be set to true sinceit will flush all messages before stopping regardless of flushingAccuracyMs and avoid potential loss of data in ingestion.

maxQueueSizeBytes is another optional setting set to 1000000 by default for maximum capacity (in bytes) of the message queue. If your message queue is bigger, it is recommended to increase this setting or else risk loss of data in ingestion

The relevant section in log4j2.xml would look like this

<SumoLogicAppender name="SumoAppender" url="${sumologic.httpsource.url}" flushAllBeforeStopping="true" maxQueueSizeBytes="<suitable_value>"> <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS Z} [%t] level: %-5p category: %c - message: %m%n" /> </SumoLogicAppender>

This is happening because of your SSO (Single sign-on)/ SAML account is not set up with the last name. We recommend checking with your SSO/SAML Administrator on your end to get your last name added. Once the "last name" is added, you should be able to take certification courses.

Yes, if the user has Content Administrator permission to the Library can able to manage/edit other user's content, even if the content was not shared with that user.

Yes, the collector does compress the logs. The installed collectors use the resources from the installed machine are used to compress, zip, encrypt the files/log messages and send it to the Sumo Cloud, in real-time. The compression ratios is 10x.

You can read more about this here

Question:

Can I create a Scheduled Search in which multiple alerts can be sent based on ranges of results?

For example:

If log_count <1 Send "No logs received" message

OR

If log_count > 10 Send "Runaway Process" message

Answer:

This is not possible - for the above situation, two separate Scheduled Searches would be needed to be configured.

There is a setting under the "My Preferences" section that can be used to set the "autocomplete" feature which further auto-populates options for you in the Search bar.

Preferences Page This can be toggled with a keyboard shortcut so you may have accidentally turned it off. Use <Esc> or <Alt> <Space> to toggle it.Please navigate to your 'Account' > 'Preferences' page and check if the autocomplete feature is enabled.

Our documentation for reference on this page is here:

Sumo Logic uses a fingerprint mechanism to track the position of the last message ingested from a file.Sumo Logic identifies new files based on a fingerprint of a file which is actually based on the first 2K bytes of a file. The fingerprint needs to be unique with the file names.

This fingerprint gets stored under the path: <Sumo_home>/config/blades directory.

You will see the hexadecimal blade.json file that represents the unique source under the configured collector on the server that has this information stored.

For example:

Sample Blade.json:

{ "blacklist": "", "blade.id": 137202106, "blade.name": "IIS_Logs", "blade.sourceCategory": "", "blade.sourceHostName": "", "blade.type": "LocalFile", "encoding": "UTF-8", "files": [{ "completed": false, "fingerPrint": "0484047c4969c961a53c04347362aabd", "fpBytesUsed": 2048, "isInTraining": true, "name": "/Users/rahul/Downloads/44802/iis_logs.txt", "pointer": 30140, "trainedPrefix": "" }

You can use Exclude Rules or Include Rules to specify what kind of data is sent toSumo Logic.

Exclude Rule: If you specifically exclude a message, it functions as a blacklist filter, and the data will never be sent toSumo Logic.

Include Rule: It does the opposite and act as a whitelist and will only allows you to ingest logs matching a specific regex filter.

More information about Include and Exclude rules can be found here

A 13 digit number after the collector name is the epoch timestamp added when the collector was first installed and registered. Collector names within an account must be unique so this number may be applied to a Collector name if another Collector with the same given name already exists within your Sumo Logic account.

If you wish to automatically delete a previous Collector with the same given name and have a new Collector installed in its place, you can use the clobber flag available withthe installation parameters of the Collector. This parameter should be used with extreme caution. Once a Collector is deleted from the service it cannot be recovered and any Source details will be lost.