DataStax FAQs

Summary

Opscenter tablessettingsandbackup_reportsare still repaired using the incremental repair method. This can generate I/O overhead because of the additional anti-compaction phase included necessary for the validation in the incremental repair. Incremental repair is not the recommended way to repair tables in Cassandra, because subrange repair is more performant and has less impact on the main operations of production clusters.

Scenario

For historical reasons, thesettingsandbackup_reportstables have always been configured for incremental repair. This is hardcoded in the Opscenter python files. It is not possible to change this settings without recompling the code, however it is possible to disable incremental repair for those tables using the incremental repair properties incluster_name.conf.

Procedure

In order to disablesettingsandbackup_reportsfor incremental repair, the user will have to explicitly set the property

[repair_service]

incremental_repair_tables=

in eithercluster_name.conffile oropscenterd.conf. By leaving the property blank, no tables will be repaired incrementally and all the tables will be repaired using subrange repair. One important thing is to unset theRepairedAtflag in each sstable of the tablessettingsandbackup_reportsby running:

nodetool mark_unrepaired "OpsCenter" backup_reports --force

nodetool mark_unrepaired "OpsCenter" settings --force

In this way the tables are marked unrepaired and the subrange repair can start repairing them.

After the changes are effected, proceed to - restart opscenter - disable repair service - enable repair service again.

Additional steps

In order to limit the amount of logging that incremental repair does, even if there are no tables to repair incrementally, users should also include this block in the `cluster_name.conf` or `opscenterd.conf`.

# Stop incremental repairs from spamming the event log

# Sleep 1 day after each cycle

incremental_sleep = 86400

This article is applicable to all 6.x versions of Opscenter. A jira is tracking the request for changing this behaviour in Opscenter: OPSC-15858 - change repair service propertyincremental_repair_tablesdefault to be empty

Backing up and restoring data in DataStax Enterprise

Managing backups and testing the restoration of them is an essential practice for data storage operations teams, and DSE is no exception to this rule. DSE is equipped for backing up and restoring part, or all, of the body of data stored in a cluster, to a fixed point in time, down to a transactional level of restoration. This article is intended as a useful reference for general knowledge, and steps required, to take backups, and to restore data to a cluster. Datastax Worldwide Support recommends testing these steps to familiarize teams with them prior to implementing them on a production database.

There are 3 layers of backup granularity available in DSE:

Snapshots - a hardlink to the sstable files in the data directory for a schema table, at the moment the snapshot is executed.

Incremental backups - a separate hardlink to an sstable, that is created at the same time the original sstable is written to disk in the data directory.

Commitlog archives - a copy of, or link to, an individual commitlog. The commitlog holds the transactional log of the node it is from, and is required for transactional-level restoration.

Using the 3 layers summarized above, the idea is to take regular snapshots, take incremental backups in between snapshots, and take commitlog archives that capture the running application of mutations (writes) to the node.

The layers are additive, are inclusive, and are redundant. This is to say that the snapshots include any data in the incremental backups, and the incremental backups will be persisted records of the commits in the commitlog. So, conceptually, when an incremental backup is taken, you could discard any earlier archived commitlogs, and when a snapshot is taken, you could discard any earlier incremental backups. Or, you could only save sets of incremental backups, and commitlogs, and get the same net result as with using snapshots as a base. But - the aim here is solely to illustrate the overlap of the backups, and not to recommend retention policy guidelines or data handling policy.

To get a good grasp on this, we can consider how Cassandra writes and stores data, in the context of the backups:

Once a commitlog flushes, it makes an incremental backup, at the same time it makes the live sstable, so the original log segment is not needed any more - the state of the data is persisted in the sstables, which you have a live copy of, and a backup copy of, that are actually hard-links of the same data on-disk. The commitlog archive will be where the transactional application of data, the mutation, is stored, while the live commitlogs will be rolled over per configuration, as they would normally, regardless of backups being taken or not.

Once the snapshot is taken, all data that is in the incremental backups will also be in the snapshot, so the incremental backups can be cleaned up. The sstables going into the live data directory will be the same files as the backups, and so, will be located in the subfolder holding the snapshot, while the live directory will continue to compact through sstable generations as they would normally.

Then, at restore time, you would:

load the most recent snapshot you have prior to the desired restore time;

also load any incremental backups from after the snapshot, into the live data directories for the keyspace or table thats being restored;

finally, load the commitlog archives from after the latest point in time covered by the sstables, into the live commitlog location, where they will be replayed on node startup. This will fill the gap in time from the persistence layer, the sstables, to get the data on that node back to the same state it was in, at the point in time requested.

Important note: A point-in-time restore requires a cluster restart for the commitlog replay to run.

In addition to the note above, there are some things its helpful to be clear on about these operations:

Snapshots are your primary backup vehicle. A snapshot will contain all the data stored on disk at the time its taken. Once the snapshot is taken it can be moved to another location and DSEwon'tcare.

Its important that snapshots from different nodes do not get stored in the same folder. The sstablesaren'twritten with unique filenames per node, meaning that they will overwrite one another if two or more files of the same filename are saved in the same location.

Incremental backups will take copies of sstables as they are created, and can also be moved to other locations once they exist. The same rules about unique filenames applies to them.

Incremental backups are useful to add to the collection of files included in a snapshot, to get you closer to the point in time of the restore - they make up the gap between the last snapshot you took and the very last sstable that was flushed for that keyspace.

The commitlog will hold a replay of any transactions that were in memory when the node stopped. The commitlog replay is the most expensive part of restoring, but is how you get to point-in-time restores with millisecond precision.

Commitlogs from one nodecan'tbe played on a different node or cluster. They are transactions specific to the node they came from.

Commitlogs from different nodes should also be stored separately from one another. They will overwrite in the same way sstables will, and transactions would be lost.

Taking snapshots

Using the nodetool snapshot command on a node will result in a new folder being created in the ../snapshots/ folder in any live data directories on that node for any tables in the snapshot. The new folder will be named with Unix timestamp by default, but can be tagged as an option to the command. The tag is useful for organizing the snapshots.

Inside that folder, hardlinks to the sstable files will be created. So, this snapshot subfolder will contain a complete image of the persisted state of the data, at the point in time of the snapshot.

If you want to get the complete state of the data persisted before the snapshot is taken, you can run nodetool flush to force a flush on one or more tables, emptying the commitlogs and making any data in them into sstables. These folders or the sstables in them can then be copied to remote storage easily using your favorite copy tool such as rsync or scp.

As mentioned previously, make sure that snapshots from different nodes are stored separately. A structure in remote storage with a parent folder for each node, with branches for keyspaces and tables going downward from there, in each nodes folder in the remote directory, is a good way to keep them separate.

Taking incremental backups

Incremental backups can be turned on in Cassandra. Enabling it means whenever a commitlog is flushed, through normal operations or through a command, which results in a new sstable in the live data directory for a table, a duplicate of that sstable will be created in the ../<keyspace>/backups/ folder for each keyspace. You must maintain and clean up these backups, since the database doesnt know about them, just like snapshots.

The incremental backups and the snapshots are both hardlinks to the live data sstables, so, once you have a snapshot of a keyspace, you can clear the contents of the incremental backups folder for that keyspace - all the files in it will be in the keyspace, and it might also have older files if youdon'tclean it out regularly.

You really only should retain the incrementals until your next snapshot is taken, in other words. Again, this is not a retention guideline recommendation, only a practical note about the redundancy of the backup layers.

Commitlog archives

This is managed by the commitlog_archiving.properties file. When you set the file up for archiving, you should set the archive_command - this can either run a command from the OS directly, or invoke a shell and run a script.. In place of the %path variable, it will echo out the absolute path of the commitlog file its archiving, so, if its about to archive /var/lib/cassandra/commitlog/CommitLog-4- 1471286005846.log then thats what the archive operation will echo out. In place of the %name variable, it will just echo the file name, so, CommitLog-4- 1471286005846.log for example.

There is an example in the commitlog_archiving.properties file of the archive_command being set to /bin/ln %path /backup/%name. Continuing with the same file from above, this would result in the literal command /bin/ln/ /var/lib/cassandra/commitlog/CommitLog-4- 1471286005846.log /backup/CommitLog-4- 1471286005846.log being run when the archive_command executes. The commitlog is archived at node startup, when a log is written to disk by the system, and when you force a flush.

The command set here should be considered carefully. The ln commandcan'twork across storage devices, for example. What OpsCenter does on your behalf when you use OpsCenter to enable commitlog archiving, is to instruct all agents to edit the commitlog_archiving.properties file for you, and sets the archive_command to /usr/share/datastax-agent/bin/archive_commitlog.sh %path %name, and archive_commitlog.sh contains cp $1 /var/lib/cassandra/commitlog/backups/$2.

Continuing with the same example, this makes the node call the archive_commitlog.sh with the arguments /var/lib/cassandra/commitlog/CommitLog-4- 1471286005846.log and CommitLog-4- 1471286005846.log, so then the system runs the cp command with them, which is means it runs cp /var/lib/cassandra/commitlog/CommitLog-4- 1471286005846.log /var/lib/cassandra/commitlog/backups/CommitLog-4- 1471286005846.log in our example. Once a new commitlog is written, the command is executed against it, and so on, indefinitely.

Thisisn'ta good location for the backups to stay, since its right alongside the live directory. If you need to update it, and youd like to use the same file, you can, by just pointing to it in the archive_command parameter in the commitlog_archiving.properties file. To change the location it makes the copy to, just change the filepath in cp $1 /var/lib/cassandra/commitlog/backups/$2 from /var/lib/cassandra/commitlog/backups/ to whatever you want.

Once you make changes to the commitlog_archiving.properties you need to restart the node. Youll see OpsCenter run a drain and restart when you turn the commitlog archiving on in the UI. This is because it made edits to this file and needs to restart the nodes for the changes to be applied.

So, restoring to a point in time, and filtering to a keyspace or table, can be fairly easy, or not practical at all, depending on your execution of archiving, and understanding the stages of the restore process.

Restoring snapshots

Restoring from a snapshot is pretty straightforward. When a node starts, it reads all the sstables in its data directories, and begins serving requests based on the token ranges it owns. Itwon'tcare much about the sstables, beyond making sure it is an sstable that matches the data directory subfolder its in, so it knows its for that table. If the name matches the table, the files will become the initial loading point for the data set. For responses to be returned, though, the tokenized keys in the sstable must in a range the node owns.

To restore snapshots, there are 3 ways. They all assume the data directory has been emptied, either with the truncate command, or by manually deleting the files when the node is down.

You can:

load any sstables from a snapshot into the data directory, and start/restart the node. The sstables will be validated and loaded on startup, regardless of the source. The node loading them has no way of knowing anything about the source of the sstables.

with an empty data directory you can load a snapshot into it, and run nodetool refresh to force the sstables to be read into the node.

you can run the sstableloader to stream sstables in a folder to a target cluster. The sstableloader will read any directory containing sstables that its passed as an argument, then contact the destination ring, determine the node token ranges, and stream sstables to the correct nodes. Its a way to bypass complexity of managing tokens when loading a data set.

Snapshots are a view of the data set on that node at that point in time. So, a snapshot taken at 10 AM can be used to restore the node to the state it was in at 10 AM that day.

If you take a snapshot every day at 10 AM, you know you can restore to 10 AM without a problem. But what if you need to restore to 10:30 AM? You need the changes that were applied to the data set in the time that has passed. There are two ways to acquire those changes and have the state of the data changed: incremental backups, and commitlog archives.

Restoring incremental backups

Actually, restoring incremental backups is no different from restoring snapshots. Theyre both just sstables, and whatever is in the data directory will be validated and read when the node restarts, when you run nodetool refresh <keyspace> <table>, or the sstables can be streamed with the sstableloader.

Since an incremental backup is a copy of an sstable that is created as they are flushed, and a snapshot is a collection of the sstables making up a data set at the point in time the snapshot was taken. you can apply them together to get the data set rolled forward closer to the point in time you want to restore to.

This means you can take a snapshot at 10 AM, have incremental backups generated every time a table flushes and generates an sstable, and then combine the two sets of file to move closer to a point you want to recover to. The rate at which the incremental backups are created depend on several factors but in a busy database, you will probably see them generated every couple of seconds at least. This gets reasonably fine granularity for restoring. If you need transactional levels of restoration, you need to replay the transactions for the nodes, on a per-node basis. To do this, you use the commitlog archives.

Restoring commitlogs

Restoring commitlogs is the process of clearing the live commitlog directory, loading one or more commitlog files from the backup location you put it in with the archive_command: or taking any other copy of the commitlogs you might have made, and restarting the node.

During node startup, the normal behavior of a node with regards to commitlogs is to check the commitlog_archiving.properties for instructions, load the appropriate tooling for archiving or apply any restore instructions, and then replay whatever is in the commitlog as transactions against the data set that is in the sstables in the live data directory.

If you have filled out the restore section of the commitlog_archiving.properties and start the node, the node will follow those instructions before moving on to the commitlog replay.

These are very similar to the archive_command:, in that the command passes variables, %from and %to, echoing them into whatever command or script is pointed to in restore_command:.

The %from variable will be whatever you put in the restore_directories: in the properties file. Note that the restore directory does not have to be related to the archive directory at all. The %to variable will be set to the live commitlog directory automatically and cannot be changed.

So, by executing a command to copy everything from the/a backup location to the live commitlog location, you can have various archived commitlogs replayed.

Using this, along with the snapshots and incremental sstables, you can restore with millisecond precision, at a transaction level. You do this by loading the sstables from your snapshot and any incremental backups between the snapshot and the restore window, then setting the point in time you want the commitlogs replayed up until. The mutations in the commitlog will be played against the data, bringing the node up to the timestamp indicated in the restore_point_in_time: parameter, which should be set as a string in YYYY:MM:DD HH:MM:SS format, and in GMT.

So, with a snapshot at 10 AM, incremental backups being created every ~30 seconds due to natural flushing in the write path, and commitlog archives from 10 AM to 10:30 AM, we can get to 10:30 AM with millisecond precision. The process would be, ideally, to load the snapshot and the incremental backups up until the last incremental before 10:30, so, 10:29:30 for example, and then the commitlogs covering the last 30 seconds at least, but maybe a couple of minutes on either side would be good, and restart the node. The node will load the data from the sstables when it starts, then scan the commitlog_archiving.properties, and would end up with any commitlogs that are in the restore_directories: being placed in the live commitlog directory, and replayed, until the restore_point_in_time: in epoch time is reached, or until the commitlogs are all replayed if it is not set.

Restoring Specific Schema Tables

If you want to filter the tables that the replay is applied to, you can, to achieve point-in-time recovery on a subset of tables. This is accomplished by setting the following line in the dse-env.sh or in the cassandra-env.sh file before the next start of the node: JVM_OPTS="$JVM_OPTS -Dcassandra.replayList=<keyspace>.<table>, <keyspace>.<table>, ..." for the list of tables you want the replay applied to. Be sure to remove it once the node is restored, or else only the tables specified will have their commitlogs replayed on the next startup.

If you are not running DSE as a service, you can pass the option in on the command line when you start DSE.

This is discussed on CASSANDRA-4809.

Some Helpful Notes for Planning

* Keep in mind that the node will replay all commitlogs in the directory, and that the commitlog replay is a fairly expensive process during the startup. The transactions are all played back by timestamp, and should be idempotent, so applying them twice should not be a problem. But, because it takes a while, and the nodewon'tbe UP until its done, you want to avoid playing back much more than you need to. For example,don'tload 2 days of commitlog archives when you only need to restore the last 20 minutes. You can place only the commitlogs you want in the restore_directories: if you only want a recent subset to be restored.

* Commitlogs will only replay for the node they are created on. They are transactions unique to the node andwon'twork on other nodes. So, the changes to the commitlog_archiving.properties would need to be made on all nodes for a restore to be successful.

* Depending on requirements, you may not need the precision that comes from the commitlog restore step. The incremental backups of sstables are created quite rapidly in most cases, so, depending on how close you need the restore point to be to the specified point in time, you may be able to reduce the complexity of restoring somewhat.

Summary

This article discusses how to lock a specific package version using yum.

Applies to

DSE and agent package installations on RHEL-based systems using Yum

Symptoms

When installing a specific version of DSE or Agent packages, yum automatically installs or upgrades to a later version than specified.

For example you might run the following command:

sudo yum install datastax-agent-5.2.5 --exclude dse-full

but the Agent package version 6.7 will be installed rather than version 5.2.5

Solution

As per RedHat Technote 98873 users can restrict yum to install or upgrade a package to a fixed specific version using theversionlockplugin of yum.

To do this:

Install the package namedyum-plugin-versionlock(calledyum-versionlockin RHEL 5).

# yum install yum-plugin-versionlock

The/etc/yum/pluginconf.d/versionlock.listwill be created on the system.

To install or lock the version of the DSE/Agent package, add that package name to the/etc/yum/pluginconf.d/versionlock.listfile. As per previous example:

# yum versionlock datastax-agent-5.2.5

(Alternatively, you can edit the filelist,/etc/yum/pluginconf.d/versionlock.list, directly.)

The above configuration will not allow to upgrade the Agentpackage to a version later than what the version that was installed at the time the locking was performed.

Yum will attempt to update all packages, while excluding the packages listed in the versionlock file.

See also

Upgrading to earlier versions of DSE with YUM installs latest version

Manually deploying DataStax Agents from RPM

Other Commands

To display the list, use:

# yum versionlock list

To discard the list, use:

# yum versionlock clear

Summary

This article covers an issue when a keyspace is not found.

Symptoms

I setup a role with login permissions, and the role correlates to a group in the LDAP directory. My roles permissions are:

cqlsh> list ALL permissions OF "DSE_Item";role | username | resource | permission | granted | restricted | grantable----------+----------+--------------------------------------+------------+---------+------------+-----------DSE_Item | DSE_Item | <keyspace item> | MODIFY | True | False | FalseDSE_Item | DSE_Item | <table item.item_country_by_item_id> | SELECT | True | False | False

However, when I try to perform any actions, I receive an error:

cqlsh> SELECT * FROM item.item_country_by_item_id LIMIT 10;Keyspace item not found.

Solution

Check whether the parameter system_keyspaces_filtering is set to true.

If so, then you have to also grant describe permissions on the keyspace in order for the user to have access to the keyspace:

GRANT DESCRIBE ON KEYSPACE item TO ROLE "DSE_Item";

Summary

In some cases, inadvertently altering one of the two DSE insights keyspaces, a known issue with the DSE insights plugin can prevent a node from starting up and causes issues with trying to add or replace nodes in a cluster.

Applies to

DSE 5.1.14 to 5.1.16

DSE 6.0.9

DSE 6.7.4

Symptoms

The node will fail to start with a large error stack. The important part of this error stack is here in thesystem.log:

1) Error in custom provider, java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement

at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)

at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)

while locating com.datastax.insights.client.InsightsClient

for parameter 0 at com.datastax.bdp.insights.InsightsClientRuntimeManager.<init>(InsightsClientRuntimeManager.java:37)

at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)

while locating com.datastax.bdp.insights.InsightsClientRuntimeManager

for parameter 0 at com.datastax.bdp.insights.InsightsPlugin.<init>(InsightsPlugin.java:66)

at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)

while locating com.datastax.bdp.insights.InsightsPlugin

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement

Subsequently the nodes will show as present in the cluster, but withDNin nodetool status.

Cause

When the node starts up, the DSE insights plugin checks for some user-defined types (UDTs) and tables that are declared in one of the two DSE insights keyspaces dse_insights and dse_insights_local. If these are missing (for example, you may have dropped them by mistake) or the RF has been changed (for example, using NetworkTopologyStrategy but not replicated to the DC where the nodes are being added), then the node will fail to start.

Workaround

Until the cluster is upgraded to DSE 5.1.17 or later, the only way to resolve this issue is to fix the keyspace replication for the dse_insights and dse_insights_local keyspaces.

Run the following commands in cqlshon a node that is up and functional in the cluster.

Note: You must use the UPDATE command to change the dse_insights_local keyspace because cqlsh restricts assignment of LocalStrategy to keyspaces.

alter KEYSPACE dse_insights WITH replication = {'class': 'org.apache.cassandra.locator.EverywhereStrategy'};

UPDATE system_schema.keyspaces SET replication = {'class': 'org.apache.cassandra.locator.LocalStrategy'} WHERE keyspace_name = 'dse_insights_local'

Solution

Upgrade to DSE 5.1.17 or later for the fix to prevent the DSE insights plugin from stopping the node startup. Upgrading prevents the issue from happening, but you must perform the workaround steps to resolve any existing missing or incorrect keyspace replication problems.

This problem is tracked by internal Jira DSP-19521.

In rare occasions, DSE will fail to start with the following exception on system.log:

INFO [main] 2018-06-07 05:32:27,547 LeaseProtocol.java:92 - Registered LeaseProtocol INFO [main] 2018-06-07 05:32:27,551 InternalQueryRouterProtocol.java:53 - Registered InternalQueryRouterProtocol INFO [main] 2018-06-07 05:32:28,647 RemoteMessageServer.java:76 - Starting internode-messaging internal message server on [/10.200.43.10]:8609 ERROR [main] 2018-06-07 05:32:28,719 CassandraDaemon.java:709 - Exception encountered during startup java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_144] at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_144] at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_144] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_144] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_144] at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1094) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:428) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:414) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:897) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) ~[netty-all-4.0.34.Final.jar:4.0.34.Final at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:349) ~[netty-all-4.0.34.Final.jar:4.0.34.Final at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.34.Final.jar:4.0.34.Final] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]

The problem stems from DSE Search not being able to bind to port 8609 for inter-node communications.

Execute the following commands to free the socket

sudo service dse stop

sudo kill -9 $(sudo lsof -i :8609 | awk '!/PID/ {print $2}' | sort -u)

sudo service dse start

Try restarting DSE

Should that still not work, try rebooting the node

If DSE still refuses to start with the same error to bind to port 8609, follow these recommendations from redhat for port 8906

Symptoms

Some gaps are visible in the OpsCenter graphs for some given metrics such as "Total Disk Used" or "Load" and some data points appear to be missing. As an example:

DataStax Agent configuration

Applies to

DataStax Enterprise 6.7 and prior versions

DataStax OpsCenter 6.7 and prior versions

Cause

The issue can occur if the metrics collector thread is unable to cope with the gathering of data operated by the agents.The agent log reflects it as follow:WARN [jmx-general-metrics] 2019-09-07 10:49:04,906 Too many outstanding requests 270 pausing collectors

Solution

Warning: setting a higher value can lead to some overhead and extra load on the nodes and performance testing should be considered. As a general recommendation, DataStax also recommends using a dedicated cluster for OpsCenter storage, which can be done at no additional cost:See the Datastax FAQ

The rollup_rate number depends on the number of tables and other metrics collected. For further information, refer todocumentationFor example, based on the formula in the documentation above, the default rollup_ratein the agent configuration file address.yaml allows to monitor for just under 300 tables per node (200 metrics per second * 60 = 12 000 metrics per minute, divided by 40 metrics collected per table -> 300 tables).

So for a cluster containing 600 tables, the rollup_rate in the address.yamlshould be doubled to 400.After applying the change and restarting the agents, confirm the warning mentioned in the cause section doesnt appear anymore in the log and that new data doesnt show gaps in the graphs.

Summary

A warning schema version mismatch detected occurs on restarting some DataStax Enterprise (DSE) nodes.

Applies to

DSE 5.0

DSE 6.0

DSE 6.7

Symptoms

If some nodes are unavailable while restarting another node, the following warning occurs:

Warning: schema version mismatch detected; check the schema versions of your nodes in system.local and system.peers.

Cause

This error can be a false positive if some nodes are unavailable. This confusing error message when nodes are down is known behavior as tracked in CASSANDRA-11142.

Determine cause

Run the nodetool describecluster command:

nodetool describecluster

If the nodes are unreachable, the command output looks like this:

Cluster Information: Name: Cluster12345abcde Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch

Partitioner: org.apache.cassandra.dht.Murmur3Partitioner

Schema versions:

UNREACHABLE: 1176b7ac-8993-395d-85fd-41b89ef49fbb: [10.202.205.203]

9b861925-1a19-057c-ff70-779273e95aa6: [10.80.207.102]

8613985e-c49e-b8f7-57ae-6439e879bb2a: [10.116.138.23]

Solution

Restart unreachable nodes.

Repeat step 1 until the output of the nodetool describecluster command shows that all nodes have the same schema version number and only one schema version appears in the output.

Related DataStax documentation

Fixing a table schema collision

nodetool describecluster

Summary

This article discussesa scenario when CQL Solr queries can return null values unexpectedly.

Applies to

DataStax Enterprise 5.1

DataStax Enterprise 6.0

DataStax Enterprise 6.7

Symptoms

When running a Solr CQL query, a non-indexed field returns 'null' even though a value exists for the field. Consider the following table:

CREATE TABLE customer.orders (item_id UUID, item text, lastname text, firstname text, solr_query text, order_type text, PRIMARY KEY (item_id, lastname));

For this example, we'll create a search index (Solr core) for the table using the following custom schema.xml file:

$ cat customer_orders_schema.xml<?xml version="1.0" encoding="UTF-8" standalone="no"?><schema name="autoSolrSchema" version="1.5"><types><fieldType class="org.apache.solr.schema.TrieDateField" name="TrieDateField"/><fieldType class="org.apache.solr.schema.UUIDField" name="UUIDField"/><fieldType class="org.apache.solr.schema.StrField" name="StrField"/><fieldType class="org.apache.solr.schema.BoolField" name="BoolField"/><fieldType class="org.apache.solr.schema.TextField" name="TextField"> <analyzer> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer></fieldType></types><fields><field indexed="true" multiValued="false" docValues="true" name="item_id" stored="true" type="UUIDField"/><field indexed="true" multiValued="false" docValues="true" name="lastname" stored="true" type="StrField"/><field indexed="true" multiValued="false" docValues="true" name="firstname" stored="true" type="StrField"/></fields><uniqueKey>(item_id,lastname)</uniqueKey></schema>

Next, we'll use dsetool to create the Solr core:

$ dsetool create_core customer.orders reindex=true schema=./customer_orders_schema.xml solrconfig=./customer_orders_config.xml

Now we'll insert data into the table:

insert into customer.orders (item_id, item, lastname, firstname, order_type) values (a6f94417-e27b-444f-9d0c-dccb588e421f, 'grill', 'Smith', 'Joe', 'app');

Running the following CQL Solr query returns 'null' for the 'order_type' field instead of the inserted value of 'app':

cqlsh> SELECT item_id, lastname, firstname, order_type FROM customer.orders WHERE item_id = a6f94417-e27b-444f-9d0c-dccb588e421f and solr_query='{"q": "*:*","fq":"(firstname:Joe)"}';item_id | lastname | firstname | order_type--------------------------------------+----------+-----------+------------ a6f94417-e27b-444f-9d0c-dccb588e421f | Smith | Joe | null(1 rows)

Cause

Solr single-pass CQL queries and HTTP queries use the custom Lucene stored fields codec which use the Solr schema to discover which fields to return.

There are a number of ways to generate a single-pass query instead of the standard two-pass query, without realizing it is happening:

Specify the distrib.singlePass Boolean parameter in the query expression:

SELECT * FROM keyspace.table WHERE solr_query = '{"q" : "*:*", "distrib.singlePass" : true}'

Use a token() or partition key restriction in the WHERE clause:

SELECT * FROM keyspace.table WHERE token(id) >= 3074457345618258604 AND solr_query = '{"q" : "*:*"}'

Execute a COUNT query:

SELECT count(*) FROM keyspace.table WHERE solr_query = '{"q" : "*:*"}'

Specify only the primary key elements in the SELECT clause, using the table from the example above, we could have:

SELECT item_id FROM customer.orders WHERE solr_query='{"q": "*:*","fq":"(item_id:a6f94417-e27b-444f-9d0c-dccb588e421f)"}';

Solution

If you end up with a Solr single-pass CQL query through any of the above methods, the custom Lucene stored fields codec will be used.That is, they use our custom Lucene stored fields codec, rather than just reading entire CQL rows as non-single pass CQL Solr queries would because the codec only knows about fields, indexed or not, that exist in the schema.xml.

Using the example above, we could add the following line to the 'customer_orders_schema.xml' file:

<field indexed="false" multiValued="false" docValues="false" name="order_type" stored="true" type="StrField"/>

After changing the file, we would need to reload the core:

dsetool reload_core customer.orders reindex=false schema=./customer_orders_schema.xml solrconfig=./customer_orders_config.xml

You'll notice that the same query returning 'null' for the 'order_type' field above, will now return the stored value of 'app' correctly:

cqlsh> SELECT item_id, lastname, firstname, order_type FROM customer.orders WHERE item_id = a6f94417-e27b-444f-9d0c-dccb588e421f and solr_query='{"q": "*:*","fq":"(firstname:Joe)"}'; item_id | lastname | firstname | order_type--------------------------------------+----------+-----------+------------ a6f94417-e27b-444f-9d0c-dccb588e421f | Smith | Joe | app(1 rows)

We could also re-write the query to avoid generating a single-pass query to achieve the same results:

cqlsh> SELECT item_id, lastname, firstname, order_type FROM customer.orders WHERE solr_query='{"q": "*:*","fq":"(item_id:a6f94417-e27b-444f-9d0c-dccb588e421f)"}';item_id | lastname | firstname | order_type--------------------------------------+----------+-----------+------------ a6f94417-e27b-444f-9d0c-dccb588e421f | Smith | Joe | app(1 rows)

*Note that if you use the generateResources=true option when creating the Solr core, it will include all table fields in the schema.xml file by default, for example:

dsetool create_core keyspace.table generateResources=true

In DSE 5.1 and earlier, the stored="true" property is required in the schema.xml file for the field to be returnable in a query.

In DSE 6.0 and later, the field must bepresent but with indexed=falsein theschema.xml.

DataStax documentation

JSON single-pass distributed query

Summary

This article relates to this warning message which is spamming the logs on DSE Search nodes:

LukeRequestHandler logs WARN "Error getting file length for [segments_NNN]" for Inconsequential NoSuchFileException Situations

Applies to

Datastax Enterprise 6.7

Datastax Enterprise 6.0

Datastax Enterprise 5.1

Symptoms

The warning will likely be incredibly frequent, spamming the logs on DSE Search nodes. The index will be functional and no discernable application impact is experienced. In the following example, the index is on the demo.table1 table. Note that the segment in the warning will vary:

WARN [http-bio-10.170.96.15-8983-exec-7855] 2019-09-09 10:18:03,056 LukeRequestHandler.java:601 - Error getting file length for [segments_1r]

java.nio.file.NoSuchFileException: /var/lib/cassandra/data/solr.data/demo.table1/index/segments_1r

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)

at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)

at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)

at java.nio.file.Files.readAttributes(Files.java:1737)

at java.nio.file.Files.size(Files.java:2332)

...

Cause

OSS Solr is part of DSE Search. This issue is from OSS Solr bug SOLR-9120 :

"Beginning with Solr 5.5, the LukeRequestHandler started attempting to report the name and file size of the segments file for the current Searcher+IndexReader in use by Solr however the filesize information is not always available from the Directory in cases where "on disk" commits have caused that file to be removed, for example...

you perform index updates & commits w/o "newSearcher" being opened

you "concurrently" make requests to the LukeRequestHandler or the CoreAdminHandler requesting "STATUS" (ie: after the commit, before any newSearcher)

these requests can come from the Admin UI passively if it's open in a browser

In situations like this, a decision was made in SOLR-8587 to log a WARNing in the event that the segments file size could not be determined but these WARNing messages look scary and have lead (many) users to assume something is wrong with their solr index."

The SOLR issue description confirms that this bug does not have any direct impact, other than spamming your logs.

This bug affects OSS Solr 5.5 and 6.0.

DSE 6.7, DSE 6.0, and DSE 5.1 include ApacheSolr 6.0.1 and do not include the OSS Solr fix available in Solr 7.2 and 8.0.

Workaround

To prevent this bug from spamming the logs, the simplest workaround is to change the logging level for only the LukeRequestHandler class.

To change the logging level online without a restart:

nodetool setlogginglevel -- org.apache.solr.handler.admin.LukeRequestHandler ERROR

To persist the change across node restarts, add the following line to your logback.xml file for DSE:

<logger name="org.apache.solr.handler.admin.LukeRequestHandler" level="ERROR"/>

The location of the logback.xml file depends on the type of installation:

Package installations: /etc/dse/cassandra/logback.xml

Tarball installations:installation_location/resources/cassandra/conf/logback.xml

See also

Changing DSE logging locations

Summary

This article relates to an issue where Spark Worker entries from a previous version of DSE become orphaned in the Spark Master recovery table and create error messages in the system.log.

Applies to

DataStax Enterprise 6.7.x

DataStax Enterprise 6.0.x

Symptoms

After upgrading to DSE 6.X, error messages like the following entry might occur in the system.log:

ERROR [dispatcher-event-loop-1] 2019-07-22 19:18:59,832 Logging.scala:91 - DseSparkMaster error

org.apache.spark.deploy.CassandraPersistenceEngine$CassandraPersistenceEngineException: Failed to deserialize worker_ with id=worker_worker-20181010025210-10.101.26.109-35648 in dc=nc-us-1-rxms_devqe. Consider cleaning up the recovery data and restarting the workers - see 'dsetool sparkmaster cleanup' and 'dsetool sparkworker restart' commands in DSE help.

...

Caused by: java.lang.IllegalArgumentException: DSE version < 6.0.0 is not supported

This happened even after these steps were taken to verify that all SSTables were correctly upgraded to the correct format and version for the upgraded version of DSE. For example, in DSE 6.7.X and 6.0.X, the correct SSTable format and version is bti-aa.

Example:

1. We ran the following command on every node in the cluster, where /var/lib/cassandra/data is the data_file_directories value configured in cassandra.yaml:

$ sudo find /var/lib/cassandra/data -type f -exec ls -lart {} \; > ${HOSTNAME}_dfd.txt

2. We analyzed all of the results to ensure that the files are the correct bit-aa format and version:

$ sudo find /var/lib/cassandra/data -type f -exec ls -lart {} \; > ${HOSTNAME}_dfd.txt$ find . -type f -name "*dfd.txt" -exec ls -l {} \;-rw-r--r--@ 1 cassdba staff 36214342 Jul 22 14:45 ./cassnode01_dfd.txt-rw-r--r--@ 1 cassdba staff 32398027 Jul 22 14:57 ./cassnode03_dfd.txt-rw-r--r--@ 1 cassdba staff 39276510 Jul 22 14:47 ./cassnode02_dfd.txt-rw-r--r--@ 1 cassdba staff 38100173 Jul 22 14:57 ./cassnode06_dfd.txt-rw-r--r--@ 1 cassdba staff 37044999 Jul 22 14:57 ./cassnode04_dfd.txt-rw-r--r--@ 1 cassdba staff 30484459 Jul 22 14:57 ./cassnode05_dfd.txt$ find . -type f -name "*dfd.txt" -exec cat {} \; | egrep -v 'solr.data|snapshots|.txt|aa-.*-bti|.log'$ # no records returned by above command indicating only bti-aa sstables are present

3. Finally, even running sparkmaster cleanup did not resolve the issue and the error still appears in system.log:

# Drops and recreates the Spark Master recovery table$ dsetool sparkmaster cleanup

# run on each node

$ dsetool sparkworker restart

Cause

This issue can be caused by orphaned entries in the Spark Master recovery table,dse_analytics.rm_shared_data. These entries represent running workers that a node is supposed to contact upon reboot. These entries should be cleaned out by DSE after a node restart. However, if an entire cluster is brought down suddenly, or after the cluster is upgraded, the entries can become orphaned. Work on this issue is tracked in an internal JiraDSP-19468.

Workaround

1. Query thedse_analytics.rm_shared_data table to verify and identify orphaned entries:

select dc, id, version from dse_analytics.rm_shared_data;

In the following example, the cluster is DSE 6.7.3 but there are entries from DSE 6.0.3, DSE 6.0.4, DSE 5.0, and a DC that had been previously removed from the cluster (dc1-mycluster).

Example:

cqlsh> select dc, id, version from dse_analytics.rm_shared_data;dc | id | version --------------+--------------------------------------------------+-----------------------dc1-mycluster | worker_worker-20181010011337-10.101.26.113-39262 | 6.0.3 dc1-mycluster | worker_worker-20181010013924-10.101.26.114-40991 | 6.0.3 dc1-mycluster | worker_worker-20181010025210-10.101.26.109-35648 | 5.0.0-Unknown dc1-mycluster | worker_worker-20181010025551-10.101.26.110-35403 | 5.0.0-Unknown dc2-mycluster | worker_worker-20181214143708-10.101.37.4-36468 | 6.0.4 dc2-mycluster | worker_worker-20181214144133-10.101.37.7-45010 | 6.0.4 dc2-mycluster | worker_worker-20181214144607-10.101.37.8-43466 | 6.0.4 dc2-mycluster | worker_worker-20181214144900-10.101.37.9-41278 | 6.0.4 dc2-mycluster | worker_worker-20181214145229-10.101.37.10-32782 | 6.0.4 dc2-mycluster | worker_worker-20181214145651-10.101.37.11-40608 | 6.0.4

2. Delete the orphaned entries

delete from dse_analytics where dc_id=[dc of orphaned row] and id=[id of orphaned row];

Example:

# deleting the first row in the above output that was for when the cluster was on 6.03cqlsh> delete from dse_analytics where dc='dc1-mycluster' and id='worker_worker-20181010011337-10.101.26.113-39262';

Solution

As always, DataStax recommends reading the DSE release notes and upgrading to the latest version. You can watch for a resolution for the associated JiraDSP-19468.

Summary

'Uncaught error' when starting DataStax Studio on Linux.

Applies to

DataStax Studio 6.0 on Linux

Symptoms

When starting DataStax Studio with ./server.sh, the following error occurs

[user@hostname:~/datastax-studio-6.0.2/bin]$./server.sh

Starting Studio. This may take a few minutes. You will be notified here when Studio is ready.Uncaught error from thread [Studio-eventuate.log.dispatchers.write-dispatcher-7] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Studio] java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-9150286051047390226.8: /tmp/libleveldbjni-64-1-9150286051047390226.8: failed to map segment from shared object: Operation not permitted] at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182) at org.fusesource.hawtjni.runtime.Library.load(Library.java:140) at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48) at com.rbmhtechnology.eventuate.log.leveldb.LeveldbEventLog.<init>(LeveldbEventLog.scala:97) at com.rbmhtechnology.eventuate.log.leveldb.LeveldbEventLog$$anonfun$7.apply(LeveldbEventLog.scala:362) at com.rbmhtechnology.eventuate.log.leveldb.LeveldbEventLog$$anonfun$7.apply(LeveldbEventLog.scala:362)<truncated for readability>

Cause

This issue occurs when the user running the Java process does not having the necessary rights and cannot generate temporary files under the /tmp directory.

Solution

Solution 1- Allow the user running DataStax studio read/write/execute access on /tmp - Remove the noexec parameter when mounting /tmp - Change the execution rights on /tmp. ie: sudo chmod 777 /tmp

Solution 2- Change the temp directory used by DataStax Studio - As per the cause, create/use a folder on which the user has execution rights. In the example below, we insert the STUDIO_JVM_ARGS with temp directory pointing to /home/user/tmp in /home/user/datastax-studio-6.0.2/bin/setenv.sh:

echo STUDIO_JVM_ARGS='-Djava.io.tmpdir=/home/user/tmp' | tee -a /home/user/datastax-studio-6.0.2/bin/setenv.sh

Restart DataStax Studio

Related documentation

Specifying JVM settings for DataStax Studio

Overview

DC/OS is a distributed operating system based on the Apache Mesos distributed systems kernel. On Mesosphere DC/OS (rebranded as D2 IQ), the DataStax Enterprise (DSE) Service is an automated service to deploy and manage DataStax Enterprise clusters.

Applies to

DSE 6.7 and previous versions on DC/OS from D2iQ. ie: DC/OS "Datastax Enterprise 3.0.0-6.7.2"

Status

Transparent Data Encryption (TDE) is not supported in DataStax Enterprise deployments on D2iQ DC/OS.

See also

Security - D2iQ Docs

Limitations of DSE - D2iQ Docs

DataStax docs on Transparent Data Encryption

Setting Up LDAP Authentication and Authorization, DSE 5.x, 6.x

Setting Up Authentication

1. Enable DSE Unified Authentication. Set the following in cassandra.yaml:

authenticator: com.datastax.bdp.cassandra.auth.DseAuthenticatorauthorizer: com.datastax.bdp.cassandra.auth.DseAuthorizerrole_manager: com.datastax.bdp.cassandra.auth.DseRoleManager

2. Indse.yaml, enable authentication and configure it to use the desired authentication method(s). Note that, as part of the steps needed to get LDAP auth working with DSE, you'll need to log into cqlsh using the built-in cassandra superuser, so you should go ahead and configure dse.yaml to also use "internal" authentication along with LDAP authentication. The below example (settings in dse.yaml) allows both LDAP and Cassandra internal authentication to be used:

#########################authentication_options: ## Turn authentication on/off enabled: true

default_scheme: ldap allow_digest_with_kerberos: true plain_text_without_ssl: warn transitional_mode: disabled other_schemes: - internal

## Set scheme_permissions to true if multiple schemes will ## be used for authenticating to DSE. Doing so prevents ## unintentional role assignment that might occur if user or ## group names overlap in the authentication service. When ## scheme_permissions is set to true, every role requires ## permissions to an authentication scheme ## (GRANT EXECUTE on <AUTHENTICATION_METHOD> SCHEME to <ROLE_NAME>) ## in order to be assigned to a login user. scheme_permissions: true

role_management_options: ## You can set the role management mode to EITHER ldap OR internal here (more ## on this in the separate explanation for setting up Authorization for ldap in DSE) ## You cannot set the role management mode to both internal AND ldap ## For now, we set it to ldap mode: ldap

authorization_options: ## Leave authorization disabled until you can get authentication working first enabled: false transitional_mode: disabled allow_row_level_security: false

## example ldap_options for MS Active Directoryldap_options:server_host: 10.200.255.28server_port: 389search_dn: cn=thatran1a,dc=qawin2012,dc=datastax,dc=lansearch_password: datastaxuse_ssl: falseuse_tls: falsetruststore_path:truststore_password:truststore_type: jksuser_search_base: cn=users,dc=qawin2012,dc=datastax,dc=lanuser_search_filter: (sAMAccountName={0})user_memberof_attribute: memberOf#group_search_type: directory_searchgroup_search_type: memberof_search#group_search_base: cn=users,dc=qawin201,dc=lan#group_search_filter: (member=cn={0},cn=users,dc=qawin2008,dc=datastax,dc=lan)group_name_attribute: cncredentials_validity_in_ms: 0connection_pool:max_active: 8max_idle: 8#########################

Restart dse for the dse.yaml changes to take effect.

3. When an ldap user logs in to dse/cassandra, dse will look up the ldap user's ldap group memberships and look for ROLES in dse/cassandra whose name matches any of the names of the ldap groups to which the ldap user belongs. Note that DSE does NOT require that an ldap user belong to just a single ldap group.

Log into cqlsh as the built-in cassandra superuser and then, for one or more of the ldap groups to which the ldap user belongs, create a ROLE in dse/cassandra whose name matches the name of the ldap group:

cassandra@cqlsh> create role usergroup1 with login=true;

NOTE that you do NOT have to create a dse/cassandra role for the ldap user itself (you just need to create a role for the ldap group) AND you do NOT have to do any role assignment (DSE will determine the user's dse/cassandra role based on the user's ldap group memberships).

4. Confirm that you can log in to cqlsh as an ldap userthat belongs to an ldap group for which you've already created a corresponding dse/cassandra role:

$ cqlsh -u lduser1 -p lduser1passwordConnected to TT_511_ldap at 127.0.0.1:9042.[cqlsh 5.0.1 | Cassandra 3.10.0.1695 | DSE 5.1.1 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.ldapuser1@cqlsh>

Setting Up Authorization / Role Management

Once you have authentication working, you can configure and test Authorization.

IMPORTANT NOTE:

Unlike DSE Authentication (which now (via DSE "Unified Authentication") allows different authentication methods to be used for a given DSE cluster), DSE Authorization is still limited/tied to using just a single mechanism. That's why the "role_management_options" "mode" setting in dse.yaml can only be set to EITHERinternal OR ldap but NOT both at the same time. So, what does it really mean that only internal role management can be used or that only ldap role management can be used? Read on to find out.

Using "ldap" for role management:

If/when you set the role management mode to "ldap", (1) role management is expected to be done on the ldap server by the ldap administrator (by creating different ldap groups, where each ldap group is mapped to a dse/cassandra ROLE (you need to create the dse/cassandra ROLE in cqlsh) and then assigning the group membership accordingly for different ldap users), and (2) DSE will automatically determine a user's dse/cassandra role by checking the user's ldap group memberships (so, you don't have to, in dse/cassandra, assign any roles to any of your ldap users).

In fact, when the role management mode is set to ldap, you will actually get an error if you try to

cqlsh> GRANT <ROLE_A> to <ROLE_B>;

or

cqlsh> GRANT <ROLE_A> to <USER_ROLE_C>

where USER_ROLE_C is a Cassandra internal user since, starting with Cassandra, users are roles in Cassandra (so, roles in cassandra can be "roles" in the traditional sense or they can be users).

cassandra@cqlsh> grant DBADMIN to internaluser1;

InvalidRequest: Error from server: code=2200 [Invalid query] message="Granting roles is not supported"

In short, you cannot assign any roles to any users in dse/cassandra if you use ldap role management, because the assignment is expected to be done on the ldap server.

However, you can still grant permissions to individual users/user roles, example:

cassandra@cqlsh> grant all on keyspace ks1 to internaluser1;

In summary, with ldap role management, you cannot assign roles to users in dse/cassandra (cqlsh), but you can still assign permissions to individual users/roles...which makes role management really only effective for "ldap roles" not for "internal roles".

For the exact steps needed to configure ldap role management, see Step 2 from the "Steps Needed to Configure and Enable Authorization" section below.

Using "internal" for role management:

If you set the role management mode to "internal", then you are expected to create roles in dse/cassandra and then assign those roles to other roles/user roles. In this case, in order to get Authorization to work for BOTH Cassandra internal users AND ldap users, you would have to:

Create a dse/cassandra role for each ldap user(not just each internal user) :(a) the dse/cassandra role name has to match the ldap username, where username=UID if you are using Open LDAP OR username=sAMAccountName if using Active Directory OR username=<something_else> in the uncommon case where customer is not using a typical ldap schema and(b) the role should be created with login=true and without any password set for it

GRANT EXECUTE on LDAP SCHEME to the role you created in Step 1 above. See https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/security/secRolesInternal.html

GRANT <INTERNAL_ROLE_NAME> to <ROLE_YOU_CREATED_FOR_LDAP_USER> where INTERNAL_ROLE_NAME is one of the dse/cassandra roles that you created in cqlsh to manage/organize the Cassandra internal users, and ROLE_YOU_CREATED_FOR_LDAP_USER is the role you created in Step 1 above.

GRANT EXECUTE on INTERNAL SCHEME to <INTERNAL_ROLE_NAME>

Finally, you can GRANT <PERMISSION> on <RESOURCE> to<INTERNAL_ROLE_NAME> to give access to specific resources

As can be seen from the steps above, when using "internal" for role management, you have to create a cassandra role for each ldap user VERSUS creating a cassandra role for just the ldap groups to which an ldap user belongs. This is because, when using "internal" role management, DSE will not do ldap group lookup to see what ldap groups an ldap user belongs to; it will only look up internal roles to see which internal role your user belongs to.

Steps Needed to Configure and Enable Authorization

1. Enable authorization in dse.yaml (and restart dse for it to take effect) :

#########################authorization_options: ## Leave authorization disabled until you can get authentication working first enabled: true transitional_mode: disabled allow_row_level_security: false#########################

2. Create the necessary dse/cassandra roles and do the necessary GRANTs. The specific roles and GRANTs needed in this step depends on whether you are using ldap for role management or internal for role management.

For internal role management, see above (in the section "Using internal for role management").

For ldap role management:

(a) Create a dse/cassandra role whose name matches the ldap user's group name:

cassandra@cqlsh> create role testgrp1 with login = true;

(b) Grant EXECUTE on LDAP SCHEME to each of the dse/cassandra roles that you created for your ldap users in the step above:

cassandra@cqlsh> grant EXECUTE on LDAP SCHEME to testgrp1;

3. Grant permissions on different resources to the roles you created for your ldap groups:

cassandra@cqlsh:ks2> grant all on keyspace ks1 to 'testgrp1';

cassandra@cqlsh> list permissions of 'testgrp1';

role | username | resource | permission----------+----------+----------------+------------ testgrp1 | testgrp1 | <scheme LDAP> | EXECUTE testgrp1 | testgrp1 | <keyspace ks1> | CREATE testgrp1 | testgrp1 | <keyspace ks1> | ALTER testgrp1 | testgrp1 | <keyspace ks1> | DROP testgrp1 | testgrp1 | <keyspace ks1> | SELECT testgrp1 | testgrp1 | <keyspace ks1> | MODIFY testgrp1 | testgrp1 | <keyspace ks1> | AUTHORIZE

(7 rows)

4. Log in as an ldap user and test authorization/permissions:

#########################$ cqlsh -u cathytran -p datastaxConnected to TT_511_ldap at 127.0.0.1:9042.[cqlsh 5.0.1 | Cassandra 3.10.0.1695 | DSE 5.1.1 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.cathytran@cqlsh> select * from ks1.users1;

id | uname----+--------1 | thanht

(1 rows)cathytran@cqlsh> select * from ks2.tab1;Unauthorized: Error from server: code=2100 [Unauthorized] message="User cathytran has no SELECT permission on <table ks2.tab1> or any of its parents"cathytran@cqlsh>#########################

5. If you will also be using Cassandra internal authentication to authenticate Cassandra users, then you'll need to also:

(a)cassandra@cqlsh> grant execute on INTERNAL SCHEME to <your_user_role_for_username>;

and

(b)Grant permissions on different resources to your internal user(s) like what you did in step 3 above for your ldap groups.

6. If you have any trouble with any of the above, please doublecheck ALL steps to make sure no mistakes were made. In DSE versions that are pre-v5.1.6, there is little to no debugging facility for troubleshooting problematic dse ldap configurations. In DSE 5.1.6 and later (which have the enhancement from DSP-14418 (Add TRACE logging to LdapUtils to help in debugging LDAP issues)), you can debug your auth config by adding

<logger name="com.datastax.bdp.cassandra.auth" level="TRACE"/>

to your cassandra logback.xml, this will give you trace level logging in the cassandra debug.log that you can use to troubleshoot your auth config, see alsohttps://academy.datastax.com/support-blog/troubleshooting-ldap-authentication-issues-dse-past-and-present.

Summary

This article provides instructions on using the lsof command to check the amount of file descriptors that the Apache Cassandra or DataStax Enterprise (DSE) database is using.

Symptoms

One reason to monitor the output of the lsofcommand is when a Java error indicates there are too many open files. Other scenarios can also lead you to investigate the number of open files.

Cause

You might be using a different version of the lsofcommand when the output of the command lsof | grep cassandra_pid shows an alarming number of file descriptors. For example, this command lists a large number of references to the stress.jar (cassandra-stress tool):

$ cat lsof.out |grep stress.jar |wc -l

242

To determine if your version of the lsof Linux command includes a thread id when the TID column header shows in the output. With TID in the results. When the thread id is included in the lsof results, duplicate entries are possible since there is one entry for each thread.

Another, much less likely, possible root cause of a large number of file descriptors is a file descriptor leak. Run lsof -n | grep java to check that the number of file descriptors opened by Java is reasonable and report the error to DataStax Support if the number is greater than a few thousand.

Solution

DataStax recommends using lsof with the -p option to list only open files for the Cassandra process:

lsof -p cassandra_pid

For example:

$ sudo lsof -p $(ps -ef | awk '( !/awk/ && /cassandra/ ){print $2}') | grep "stress"

java 1647 datastax mem REG 253,0 395156 785358 /home/datastax/dse-5.1.17/resources/cassandra/tools/lib/stress.jarjava 1647 datastax 48r REG 253,0 395156 785358 /home/datastax/dse-5.1.17/resources/cassandra/tools/lib/stress.jar

In cases where users want unlimited facets, they have been setting facet.limit to -1. This can be a potential issue when counting millions of distinct values. And at times, users may have found that the results were far larger than expected, bogging down the cluster, or even causing outages.

An alternative to unlimited facets is using limited with the following:

query1:

facet.sort=indexshard.shuffling.strategy=SEEDshard.shuffling.seed=10.1.1.250facet.offset=0facet.limit=15000

query2:

facet.sort=indexshard.shuffling.strategy=SEEDshard.shuffling.seed=10.1.1.250facet.offset=15000facet.limit=15000

Increasing the facet.offset during each iteration and using the same seed node for shard.shuffling.seed will ensure consistent results for that query.

NOTE: If working on several different facet counts, then using a different seed node is optimal. But, for the same facet query group (query1, query2), please use the same seed node during each query.

NOTE: This type of facet counting is not available in cql. You must use HTTP.

You will know when you've hit the end when the facet count is less than the facet.limit, or when you receive no further results.

Summary

The application is adding more Statements to the cache than can be stored.

Symptoms

The logs contain many warnings similar to:

INFO [ScheduledTasks:1] 2017-01-26 12:30:32,554 QueryProcessor.java:139 - 1532 prepared statements discarded in the last minute because cache limit reached (28648824 bytes)

Cause

Cassandra has a cache that stores a parsed version of the cql statement that have been prepared. Using this cached version allows future uses of the statement to skip the parsing stage and thus speeds up the application performance.

This process is described in more detail here:

http://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/

The message is logged because either there are too many prepared statements, or the statements being cached are too large.

Solution

Review the application logic to see how queries are prepared. By default Cassandra should be able to cache many thousands of statements - it is very rare that an application would actually need that many individual statements.

A common scenario is where the application mistakenly adds a text string to the cache - rather than using place holders and then binding the specific variables that are needed.

So for example if the following statements are prepared:

select * from table where col1 = "1" and col2 = "1" and col3 = "1" ;select * from table where col1 = "2" and col2 = "2" and col3 = "2" ;select * from table where col1 = "3" and col2 = "3" and col3 = "3" ;

Cassandra will store 3 separate statements. Whereas:

select * from table where col1 = ? and col2 = ? and col3 = ? ;

With bound values for each of the 3 queries in turn would create only one entry in the cache.

You can review entries in the cache by selecting from the system tables as follows:

cqlsh> select * from system.prepared_statements ;

prepared_id | logged_keyspace | query_string------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0x74916e9e8d493d4d087ce0a69e1ed8e9 | null | select * from table where col1 = "1" and col2 = "1" and col3 = "1" 0xad116a7c4b8d91406bb343437b78003e | null | select * from table where col1 = "2" and col2 = "2" and col3 = "2" 0xa7c749465c514c7a06649f4cbf847721 | null | select * from table where col1 = "3" and col2 = "3" and col3 = "3" 0xad35a16565cd45f0077c9df5e7f120f9 | null | select * from table where col1 = ? and col2 =? and col3 = ?

Summary

This article describes the expected authentication transitional mode in DSE 5.1.

Applies to

DSE 5.1

Scenario

When using authentication transitional mode in DSE 5.1.x, you must supply user credentialsregardless of the transitional mode you use, so long as authentication is enabled,

If transitional mode is set to 'permissive', the credentials don't have to be valid credentials, but the credentials still have to be provided by the client.

If you want to enable access control without any downtime, you must use one of the following procedures:

Update your apps to submit dummy credentials (they would have to bedummy credentialssince you don't have any users/user credentials yet since you can't create users without first enabling authentication)

Enable authentication, at the same time setting transitional_mode to permissive

Create your dse/cassandra users

Update the apps again so that, now, the user is prompted for their username and password

Disable transitional mode

OR :

Update your apps to prompt users for username and password (users to give their username and password, thoughthe username and password aren't enforced yet at this point)

Enable authentication, at the same time setting transitional_mode to permissive

Create your dse/cassandra users

Disable transitional mode (username and password should now be enforced so the credentials now must be valid in order to connect)

Additionally, NOTE: Anonymous login for strictmode transitional authentication is not supported by cqlsh. This work is tracked in internal Datastax jira DSP-11914.

Summary

The ring tab of a DC appears greyed out

Errors related to the /var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db appear in the agent log

Applies to

Problem only occurs with tarball agents: issue seen on 6.5.3 and 6.1.11 (from investigation, the issue was introduced in 6.1.7)

DataStax OpsCenter 6.5

DataStax OpsCenter 6.1

Symptoms

The content of the ring tab in OpsCenter shows the nodes greyed out with an unknown status

Datastax-agent output some error as per the excerpt below

WARN [C3P0PooledConnectionPoolManager[identityToken->xyz]-HelperThread-#0]:07:02,397 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7c47eac7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: path to '/var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db': '/var/lib/datastax-agent' does not exist

Cause

This error was potentially introduced with OpsCenter 6.1.7 and the Distributed Subrange Repair (DSR) feature. The tarball agents appear to be looking for a directory that only exists by default with the packaged installers /var/lib/datastax-agent/tmp. The problem was only noted after OpsCenter is restarted but agents kept running.

Solution

This issue was escalated to development under OPSC-15533. The solution is as follow:

Go to the cluster configuration file in OpsCenter. ie:/etc/opscenter/clusters/<clustername>.conf

Add the following section redirecting the agents to a valid temp directory. In the example below, make sure the user running the agent can read/write/execute to the folder/path/to/agent/tmp:

[agent_config]

tmp_dir = /path/to/agent/tmp

Background

Apache Cassandra exposes many metrics and commands over JMX. Both nodetool and OpsCenter use JMX to get metrics from cassandra and invoke commands. However, some commands are not exposed via OpsCenter or nodetool, so the use of a general-purpose JMX client is necessary. Here are some recommended JMX clients.

Java Console (JConsole)

JConsole is a graphical JMX client that is shipped with every JDK. Since it is easy to use and readily available, it may seem like the natural choice. However, it may be difficult to use with a remote server since it requires a graphical interface and because JMX requires a large range of ports to be open in your firewall in order to connect remotely. On the other hand, if you are monitoring a development server on your desktop, it may be the easiest choice. See Getting statistics and metrics in the DataStax documentation.

MX4J

MX4J provides a web-based JMX interface that can be accessed remotely from a web browser and only requires one HTTP port open in your firewall. Cassandra has built-in support for MX4J, but you must download it separately and put the JAR file in your classpath.

For DataStax Enterprise installations, a message that MX4J will not load and thatmx4j-tools.jaris not in the classpath might appear.For DataStax Enterprise, you can ignore this message. MX4j is not necessary to run DataStax Enterprise.

Instructions for installing and using MX4J on Cassandra available here:http://wiki.apache.org/cassandra/Operations#Monitoring_with_MX4J.

Once you have configured MX4J as directed, access the web interface, and you will see a list of available MBeans. Select the desired bean and you will see a list of attributes and operations. From here you can view the read-only attributes, set writeable attributes, and invoke operations.

Note that allowing unrestricted access to JMX via MX4J may be a security risk. We recommend using a firewall to limit access to trusted IP addresses.

jmxsh

jmxsh provides a command-line interface to JMX. It allows scripting in the TCL language, but you don't need to know TCL to use it. You only need to learn a few simple commands to use it, and it also provides a menu-driven interface to browse the available MBeans, operations, and attributes. Download the latest jxmsh from here:http://code.google.com/p/jmxsh/downloads/list, and review the documentation here:http://code.google.com/p/jmxsh/wiki/Summary.

To connect jmxsh to cassandra, type the following command (assuming you are running on the node you want to connect to and using the default JMX port):

java -jar jmxsh-R5.jar -h localhost -p 7199

Once connected, you can press enter at the prompt to enter the menu-driven MBean browser. It will allow you to select the desired MBean, attribute, or operation by pressing the number associated with the entry. You can also enter commands you wish to run directly.

For example, to take a heap dump, type the following command:

jmx_invoke -m com.sun.management:type=HotSpotDiagnostic dumpHeap /path/to/heapdump.hprof false

To bulk load some SSTables (seehttp://www.datastax.com/dev/blog/bulk-loading for more details), enter the following command:

jmx_invoke -m org.apache.cassandra.db:type=StorageService bulkLoad /path/to/SSTables

You can use DataStax Bulk Loader (dsbulk) to load and unload CSV or JSON data in and out of DataStax Enterprise (DSE) or DataStax Distribution of Apache Cassandra (DDAC) databases.

Summary

On a working DSE Search cluster, changes to application logic or data content can produce inconsistent results due to indexing errors. This article discusses an indexing error that resulted from a string that was unexpectedly too long after an external application change.

Applies to

DSE 5.1.0 to 5.1.16

DSE 6.0.0 to 6.0.7

DSE 6.7.0 to 6.7.2

Symptoms

A previously working query or application logic starts to produce results that might be missing some content. A review of the logs showed the following error:

ERROR [wiki.immense Index WorkPool work thread-1] 2019-08-06 16:31:47,306 Cql3SolrSecondaryIndex.java:772 - [wiki.immense]: Exception writing document id 31 to the index; possible analysis error: Document contains at least one immense term in field="field1" (whose UTF8 encoding is longer than

the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[51, 50, 55, 54, 55, 32, 49, 32, 50, 32, 51, 32, 52, 32, 53, 32, 54, 32, 55, 32, 56, 32, 57, 32, 49, 48, 32, 49, 49, 32]...', original message: bytes can be at most 32766 in length; got 185501. Perhaps the document has an indexed string field (solr.StrField) which is too large

org.apache.solr.common.SolrException: Exception writing document id 31 to the index; possible analysis error: Document contains at least one immense term in field="field1" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[51, 50, 55, 54, 55, 32, 49, 32, 50, 32, 51, 32, 52, 32, 53, 32, 54, 32, 55, 32, 56, 32, 57, 32, 49, 48, 32, 49, 49, 32]...', original message: bytes can be at most 32766 in length; got 185501. Perhaps the document has an indexed string field (solr.StrField) which is too large

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.indexDoc(CassandraDirectUpdateHandler.java:709)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.addDoc(CassandraDirectUpdateHandler.java:150)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doIndex(AbstractSolrSecondaryIndex.java:1285)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doUpdate(AbstractSolrSecondaryIndex.java:1007)

at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex$2.run(Cql3SolrSecondaryIndex.java:761)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex$2.run(AbstractSolrSecondaryIndex.java:943)

at com.datastax.bdp.concurrent.Worker.run(Worker.java:86)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Cause

A string field does not typically tokenize.For example, when the string 1 2 3 4 5 32767 is not tokenized into separate terms, it is treated as a single long string. When the maximum length allowed for this field type is exceeded, the "Document contains at least one immense term" error occurs. Typical examples include application changes, underlying data changes, or both. For example, to attach XML or JSON files. These changes can affect the field length of the document causes the length of the string being indexed for this field to be too long.

In our example, the Solr schema looked like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<schema name="autoSolrSchema" version="1.5">

<types>

<fieldType class="org.apache.solr.schema.TrieIntField" name="TrieIntField"/>

<fieldType class="org.apache.solr.schema.StrField" name="StrField"/>

<fieldType class="org.apache.solr.schema.TextField" name="TextField">

<analyzer>

<tokenizer class="solr.StandardTokenizerFactory"/>

<filter class="solr.LowerCaseFilterFactory"/>

</analyzer>

</fieldType>

</types>

<fields>

<field docValues="true" indexed="true" multiValued="false" name="key" stored="true" type="TrieIntField"/>

<field indexed="true" multiValued="false" name="field1" stored="true" type="StrField"/>

<field indexed="true" multiValued="false" name="field2" stored="true" type="TextField"/>

</fields>

<uniqueKey>key</uniqueKey>

</schema>

A long string that contained the string of all numbers between 1 and 32767 was generated and inserted into DSE. While field2 was able to accept and build such a long string (text field), field1 was not.

In Cassandra, we see keys 1, 21, and 31:

$ cqlsh -e "select key from wiki.immense"

key

-----

1

21

31

(3 rows)

In Solr we see only docs 1 and 21 (remember doc 31 failed to index)

http://10.101.34.15:8983/solr/wiki.immense/select?q=key%3A*&fl=key&wt=json&indent=true

{

"responseHeader":{

"status":0,

"QTime":8},

"response":{"numFound":2,"start":0,"maxScore":1.0,"docs":[

{

"key":1},

{

"key":21}]

}}

Solution

The correct solution is to identify the potential problem row or rows in the database and verify the data integrity.

The document id should be displayed in the error:

Exception writing document id 31 to the index; possible analysis error:

The document id is theuniqueKeyin the Solr schema that points to the partition key in the database. You can then query the database to verify the row or rows to identify the required corrective action:

Change the field type. Perhaps the new application logic requires strings of this length.

The row or rows contain invalid data and require manual correction.

Implement data validation checks to enforce controls on this field.

See also

Understanding field types DataStax documentation

JSON and DSE Search DataStax Developer Blog

Introduction

OpsCenter is capable of collecting metrics, managing repairs, performing backups, and performing other management operations on DataStax Enterprise (DSE) clusters even when Lifecycle Manager (LCM) is not used to deploy and configure DSE. As such, clusters can be added and removed to OpsCenter for purposes of management independent of whether they are managed in LCM. However, all LCM clusters are registered with OpsCenter for management.

When an LCM-managed cluster is deleted from OpsCenter management, LCM jobs will fail with the error: "Cluster connection settings could not be updated: Cannot find cluster configuration in opscenterd. Please update your cluster connection settings manually." LCM does not attempt to automatically recover from this situation.

Workaround

LCM is capable of re-registering the cluster with OpsCenter, but first it's necessary to manually clear the old and currently invalid cluster id:

Get a list of LCM clusters and make note of the ID of the affected cluster: curl http://127.0.0.1:8888/api/v1/lcm/clusters/ | json_pp

Clear the opsc-cluster-id, signalling to LCM that it should re-register the cluster:curl -X PUT http://127.0.0.1:8888/api/v1/lcm/clusters/LCM-CLUSTER-UUID -H "Content-Type: application/json" -d '{"opsc-cluster-id": null}' | json_pp

Run an install-job in LCM. It's necessary to run an install job even if DSE is already installed in order to cause LCM to re-register the cluster for management in OpsCenter. It is safe and efficient to run install jobs multiple times on the same nodes, LCM will detect that DSE is already installed with the expected version and efficiently skip software installation steps.

Notes:

LCM-CLUSTER-UUID must be replaced with the id found in step 1.

Piping to json_pp is optional, but sending output to json_pp, another json formatter, or using a rest-client like Postman makes output much easier to read.

If OpsCenter api/ui authentication is enabled, it's necessary to acquire a session token and include it in the requests in steps 1 and 2. This is documented athttps://docs.datastax.com/en/opscenter/6.5/api/docs/authentication.html

Technical Details

In order to reduce the frequency with which this issue is encountered in the field,OPSC-15026 has been filed as a request for OpsCenter deletion of clusters to automatically clear the opsc-cluster-id from LCM clusters. Inquire with DataStax support for an update on the status of this feature request.

Summary

A direct buffer out of memory condition can potentially occur under certain circumstances due to the way the JVM uses temporary buffer cache of the java nio package

Applies to

DSE 5.0

DSE 5.1.0-5.1.9

DSE 6.0

Symptoms

The DSE process will unexpectedly stop with the following error in the/var/log/cassandra/system.log

ERROR [CompactionExecutor:2] 2018-04-30 09:31:36,251 CassandraDaemon.java:207 - Exception in thread Thread[CompactionExecutor:2,1,main]

java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_171]

at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_171]

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_171]

at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:108) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:45) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:387) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:314) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:120) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:92) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:87) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.compress.CompressedRandomAccessReader.access$100(CompressedRandomAccessReader.java:38) ~[cassandra-all-3.0.13.1735.jar:5.0.9]

at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.createBuffer(CompressedRandomAccessReader.java:275) ~[cassandra-all-3.0.13.1735.jar:5.0.9]

at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:74) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:59) ~[cassandra-all-3.0.13.1735.jar:5.0.9]

at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.build(CompressedRandomAccessReader.java:283) ~[cassandra-all-3.0.13.1735.jar:5.0.9]

at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:145) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:133) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1739) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:94) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:51) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:41) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:69) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:584) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:497) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:359) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:395) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:93) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:153) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1478) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]

at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.13.1735.jar:3.0.13.1735]

at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]

Cause

The java nio package uses a temporary cache for off heap byte buffers. In Java versions earlier than 1.8.102 and 1.9.104, there was no limit on this cache. The JDK-8147468 bug outlines the issue for Java 9 (see backport bugs for earlier releases).

Workaround

Upgrade Java to 1.8.102 or later, and then apply the following setting into yourcassandra-env.shor yourjvm.options file:

-Djdk.nio.maxCachedBufferSize=1048576

Internal DataStax Jira DB-2028 covers this issue.

Solution

Upgrade to DSE 5.1.10, DSE 6.0.1, or later where this max cached direct buffer limit on NIO is correctly set to 1 MB by default. See Installing supporting software.

Summary

DSE has a FailureDetector mechanism that enables a node to perform regular checks on itself to detect if it has paused too long. This mechanism helps to prevent the node from creating false positives and marking other nodes down in gossip. If you see these messages at any time other than node startup, then this message could be a sign of high load or the underlying OS being paused as can occur on a virtual machine.

Applies to

All versions of DSE

Symptoms

The following error may be observed in thesystem.log:

WARN [GossipTasks:1] 2019-09-06 12:02:16,779 FailureDetector.java:288 -

Not marking nodes down due to local pause of 7994062116 > 5000000000

Cause

The failure detector checks the time every second as part of the GossipTasks thread. If a time discrepancy of more than 5 seconds occurs since the last check and the current system time, the local node does not mark other nodes down in gossip to avoid false positives.

You will always see one of these messages during a node startup as the JVM allocates resources to DSE. If there are subsequent messages then they are usually accompanied by other signs of load. For example, typically large GC pauses:

INFO [Service Thread] 2019-09-19 15:59:36,756 GCInspector.java:245 - G1 Young Generation GC in 270ms. G1 Eden Space: 90177536 -> 0; G1 Old Gen: 1931083912 -> 1946812552;

INFO [Service Thread] 2019-09-19 15:59:38,985 GCInspector.java:245 - G1 Young Generation GC in 417ms. G1 Eden Space: 90177536 -> 0; G1 Old Gen: 1962541208 -> 2075263120; G1 Survivor Space: 13631488 -> 8388608;

WARN [GossipTasks:1] 2019-09-19 15:59:47,198 FailureDetector.java:288 - Not marking nodes down due to local pause of 8211495979 > 5000000000

INFO [Service Thread] 2019-09-19 15:59:47,199 GCInspector.java:245 - G1 Young Generation GC in 281ms. G1 Eden Space: 1048576 -> 0; G1 Old Gen: 2075263120 -> 2084700304; G1 Survivor Space: 8388608 -> 0;

WARN [Service Thread] 2019-09-19 15:59:47,201 GCInspector.java:243 - G1 Old Generation GC in 7920ms. G1 Old Gen: 2084700304 -> 1773677016;

However if these messages are observed without signs of overload, then the DSE node is typically being starved of resources. For example, a VM host swapping out CPU cycles to other VM instances using the same hardware.

Solution

If you observe these messages on virtualized instances, ensure that DSE instances are not competing with other applications for hardware resources.

Further Reading

Capacity planning and hardware selection for DataStax Enterprise implementations.

VMware affinity rules

Overview

This article provides instructions on how to generate an OpsCenter Diagnostics Report for a specific node or group of nodes.

Applies to

OpsCenter 5.2.x or newer

Background

In some instances such as in a cluster with 50 or more nodes, it may not always be practical to generate a full Diagnostics Report from OpsCenter.

This article provides the steps for generating a Diagnostics for a specific node in the cluster, or a small number of nodes.

Prerequisites

This procedure only works on nodes where the DataStax OpsCenter agent is installed.

The simplest way of installing the agent is via the tarball installation since it does not require rootaccess.

Procedure

Follow these steps to generate a Diagnostics for a node.

Step 1 - Logon to the node and switch to the user running the DataStax OpsCenter agent, most commonly as the cassandra user:

$ sudo su -<agent_user_ID>

NOTE - Ensure that the user has the right permissions to /tmp since the agent will create files in that location.

Step 2 - Startup the agent if it is not running and confirm operation as per the followingexamples:

$ ps -ef | grep datastax-agent

106 7053 1 0 Nov06 ? 01:50:15 /usr/lib/jvm/jdk1.7.0_45/bin/java -Xmx128M -Djclouds.mpu.parts.magnitude=100000 -Djclouds.mpu.parts.size=16777216 -Dopscenter.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStorePassword=opscenter -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid -Dlog4j.configuration=file:/etc/datastax-agent/log4j.properties -Djava.security.auth.login.config=/etc/datastax-agent/kerberos.config -jar datastax-agent-5.2.0-standalone.jar /var/lib/datastax-agent/conf/address.yaml

$ sudo lsof -i -n | grep LISTEN | grep 61621

java 7053 cassandra 17u IPv4 20189 0t0 TCP *:61621 (LISTEN)

Step 3a - For OpsCenter 6.1 or newer, generate the Diagnostics as follows:

$ curl -L http://<node_IP>:61621/v1/diagnostics > diagnostics-`hostname -i`.tar.gz

This command will generate a zipped tarball in the directory that it is run.

For OpsCenter 6.1 nodes with SSL encryption enabled, supply the SSL certificate and password as follows:

$ curl -k --cert /var/lib/opscenter/ssl/opscenter.pem --key /var/lib/opscenter/ssl/opscenter.key \

-L https://<node_IP>:61621/v1/diagnostics > diagnostics-`hostname -i`.tar.gz

Step 3b - For older versions of OpsCenter 6.1, generate the Diagnostics as follows:

$ curl -L http://<node_IP>:61621/diagnostics > diagnostics-`hostname -i`.tar.gz

For older OpsCenter versions where SSL encryption is enabled, supply the SSL certificate and password as follows:

$ curl -k --cert /var/lib/opscenter/ssl/opscenter.pem --key /var/lib/opscenter/ssl/opscenter.key \

-L https://<node_IP>:61621/diagnostics > diagnostics-`hostname -i`.tar.gz

CLEANUP - The agent will have written files in the /tmp/opsc-* directory. Once complete, delete this directory since it will not be cleaned up automatically.

Repeat the steps above on other nodes as appropriate.

Issue:

When I issue the command:

dsetool stop_core_reindex <keyspace>.<core>

I receive an error stating connection refused.

Some dsetool commands use the same interface/port that the solr admin ui utilizes, which is 8983 by default. You can verify this with:

automaton@ip-10-101-32-78:~$ netstat -tulpn | grep -i listen(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)tcp 0 0 0.0.0.0:7080 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:7081 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:8778 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:7437 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:43505 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:8081 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:9042 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:7447 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:8983 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:7000 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:5598 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:5599 0.0.0.0:* LISTEN -tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN -tcp 0 0 10.101.32.78:8609 0.0.0.0:* LISTEN -tcp6 0 0 :::61621 :::* LISTEN -tcp6 0 0 :::22 :::* LISTEN -tcp6 0 0 :::18080 :::* LISTEN 15244/java

In this case, the default is:

tcp 0 0 0.0.0.0:8983 0.0.0.0:* LISTEN -

However, in your environment you may have locked this down a bit, so your output could read:

tcp 0 0 172.28.219.42:8983 0.0.0.0:* LISTEN 152025/java

Solution:

If your netstat output looks similar to the above, specifying ip, then you should run the command as:

dsetool -h172.28.219.42 stop_core_reindex <keyspace>.<core>

Summary

OpsCenter 6.0+ Only

When creating a new cluster with OpsCenter Lifecycle Manager, enabling client-to-node encryption withclient_encryption_options.require_client_auth=true will cause the install job to fail. If a user attempts to do this they will see one of two errors:

If password authentication is enabled on the DSE cluster the user will see a NoHostAvailable exception shown when LCM attempts to change the default password.

If authentication is not enabled the user will see an error when LCM attempts to register the cluster for OpsCenter monitoring. Note: In this case the cluster will typically be created successfully but will not be monitored by OpsCenter.

Workaround

The workaround for this issue is to create the cluster with client-to-node encryption enabled, but require_client_auth=false. Once the cluster is successfully created, it is possible enable require_client_auth through the following manual steps:

Create a certificate for each client. While LCM can create certificates for DSE servers, it cannot create them for clients and you must do so manually. For details on how to create client certificates, refer to the user guide for your driver,whichtypically hasan "SSL" or "Security" section. For example, the Java driver has an SSL section which details the procedure for creating client certificates. It is highly recommended to use a Certificate Authority (either a private self-signed CA or a commercial CA) to sign your client certificates rather than making each client-certificate self-signed. Using a CA makes trust-store administration much simpler as new client nodes are added and removed.

Add the client's CA (or certificates if you're not using a CA) to the trust-store for each DSE server. If you previously allowed LCM to create truststores for your DSE nodes (which happens by default when client-to-node or node-to-node encryption is enabled), this involves SSH'ing to each DSE node and running the appropriate keytool import command as described in the DSE documentation. If you have deployed your own DSE truststores, you must ensure that they trusts your client certificates.

Add the DSE server CA to the trust store for each client. If you previously allowed LCM to create certificates for your DSE servers (which happens by default when client-to-node or node-to-node encryption are enabled), you may manually download the LCM CA cert and add it to the truststore used by your driver as documented in the appropriate driver user-guide. If you prepared your own certificates, ensure that the client-truststores trusts the DSE server certificates.

Edit the Config Profile and set the following properties in cassandra.yaml#client_encryption_options

require_client_auth = true

truststore = path to the truststore deployed in Step #1

truststore_password = password to the truststore, if one exists

Run a Configure job on the cluster through LCM.

Edit the OpsCenter Connection Settings for the cluster via the Settings menu, supplying valid values for the client keystore.

Details

Client to node encryption enables encryption of CQL queries issued from Cassandra clients (and the corresponding responses from Cassandra servers), and instructs Cassandra clients to verify that the identity of any Cassandra server they talk to matches its certificate. LCM automatically creates and installs certificates for Cassandra servers if necessary using a self-signed private Certificate Authority. In order for Cassandra clients to be able to verify identity of the Cassandra server, they must be manually configured to "trust" the private certificate authority.

If require_client_auth=true, the DSE servers must additionally be configured to trust the client CA or certificates. There is no mechanism to do this automatically, and it must be done manually which prevents LCM fromconnecting tonew clusters with require_client_auth enabled during the initial install job. Once the necessarymodifications are madeto the DSE server truststores, LCM will not modify them further and subsequent jobs will complete as expected.

Summary

This article provides instructions to create a local repository for DSE installation packages that leverage apt and yum installation methods to ease upgrade and installation pain on servers that do not have external access to enable package installation downloads.

Scenario

In many cases DSE nodes, especially the servers they reside on, don't have external access to the internet. This scenario makes installing and upgrading DSE a challenge when using package installs.

Other Methods

This article does not cover methods that don't require setting up a local repository. For example, other methods use apt get to download, not install, packages from a machine that has access to the internet and then download or copy those packages to each node/server manually.

For example, these commands get the packages, but then require that you copy all these files to each respective node/server which can be laborious and time consuming:

sudo apt-get -d install Downloads the given package and all missing dependencies to the /var/cache/apt/archives system packages directory

sudo apt-get install <package_name> --download-only

Creating a local repository

A time-saving alternative is to create a local repository within your organization that holds these install packages and then leverages the apt or yum installation mechanisms.

The following steps relate productity to:

Set up an APT repository

Set up a YUM repository

Set up an APT repository

1. Begin by installing the required packages.

$ sudo apt install -y apt-mirror

2. Create the directory structure to store the repos packages.

$ sudo mkdir -p /var/www/apt-mirror

3. Configure /etc/apt/mirror.list. This example synced only what I needed, which is from Ubuntus main and universe repos and Singapores local mirror (for example, sg.archive.ubuntu.com). For your setup, add other repos to taste. As an example, Ive included the repo for PostgreSQL.

############# config ###################set base_path /var/www/apt-mirrorset nthreads 20set _tilde 0############## end config ##############deb https://debian.datastax.com/enterprise/ stable main

4. Download the Datastax repository key (gpg key):

$ curl -L https://debian.datastax.com/debian/repo_key | sudo apt-key add -

5. (Optional) If a proxy separates the repo from the Internet, add the following lines to the /etc/wgetrc.apt-mirror configuration file. The apt-mirror tool relies on wget to download packages, which in turn reads from this config file.

https_proxy = http://<your_proxy_server>:80/http_proxy = http://<your_proxy_server>:80/use_proxy = on

Set up a YUM repository

1. To begin, install the required packages:

$ sudo apt install -y epel-release createrepo yum-utils

2. Create the directory structure to serve the mirrored packages.

$ cd /var/www/yum-mirror$ sudo mkdir -p datastax/x86_64/

3. (Optional)If a proxy separates the repo from the Internet, add this configuration entry to/etc/yum/yum.conf. Otherwise, the yum.conf config file usually works out of the box.

proxy=http://<your_proxy_server>:80/

4. Create the configuration file/etc/yum/repos.d/dse.repo. This config file tellsreposyncto sync the mentioned repos. Modifybaseurlto sync from a repo close to you.

[datastax]

name=DataStax Repo for DataStax Enterprise

baseurl=https://rpm.datastax.com/enterprise/

enabled=1

gpgcheck=0

5. If signature verification (gpgcheck=1) is enabled, import the DataStax Enterprise repository key:

sudo rpm --import https://rpm.datastax.com/rpm/repo_key

6. (Optional)To add more repos, follow the instructions from the previous step, but change the filename of the config files and the config statements. This example also syncs the EPEL repo, since the config file was added to/etc/yum/repos.d/epel.repo.gpgcheck=0because EPEL has many packages with missing GPG keys. Enablinggpgcheckdownloads and deletes the offending packages immediately at each sync. Its ok to download these files to the repo. Whats important is that the clients enablegpgcheck.

[epel]name=Extra Packages for Enterprise Linux 7baseurl=https://dl.fedoraproject.org/pub/epel/7/x86_64/failovermethod=priorityenabled=1gpgcheck=0gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

Summary

This article provides information and troubleshooting for multiple errors in OpsCenter that are related toKeyError: 'x-datastax-agent-version'.

Applies to

DataStax Enterprise (DSE) 5.1

DataStax Enterprise 6.0

DataStax Enterprise 6.7

OpsCenter 6.5

OpsCenter 6.7

Symptoms

A failing query is referenced in opscenterd.log withKeyError: 'x-datastax-agent-version'.

The WARN message repeatsignoreta=xyza large amount of times.For readability, the URL WARN line is truncated in this example.

2019-04-24 02:26:13,228 [ClusterName] ERROR: Error processing HTTP response: Traceback (most recent call last):

File "/usr/share/opscenter/Lib/site-packages/opscenterd/Agents.py", line 103, in onSuccess

KeyError: 'x-datastax-agent-version'

(http-client-1)

2019-04-24 02:26:13,232 [ClusterName] WARN: HTTP request <https://1.2.3.4:61621/v1/tokenranges?partitions=1048576&ignoreks=solr_admin&ignoreks=dse_insights_local&ignoreks=dse_system&ignoreks=system_virtual_schema&ignoreks=system_views&ignoreks=system&ignoreks=dse_system_local&ignoreks=system_schema&ignoreks=dse_perf&ignoreks=dse_insights&ignoreta=prefix.objectName&ignoreta=prefix.objectName&ignoreta=prefix.objectName&ignoreta=prefix.objectName&ignoreta=prefix.objectName&ignoreta=prefix.objectName&<shorten for readability>

File "/usr/share/opscenter/Lib/site-packages/opscenterd/Agents.py", line 154, in onSuccess

KeyError: 'x-datastax-agent-version'

The agent.log response is based on the timestamp and IP address provided in the log:

WARN [qtp78782356-2274] 2019-04-24 02:26:13,225 URI is too large >8192

WARN [qtp78782356-2274] 2019-04-24 02:26:13,226 badMessage: 414 for HttpChannelOverHttp@47f3a964{r=479,c=false,a=IDLE,uri=-}

WARN [qtp78782356-2274] 2019-04-24 02:26:13,228 badMessage: java.lang.IllegalStateException: too much data after closed for HttpChannelOverHttp@47f3a964{r=479,c=true,a=COMPLETED,uri=null

Cause

This issue occurs when Repair Service is enabled in OpsCenter and in these scenarios:

HTTP query refused by the agent for the call to/v1/tokenranges due to a larger than allowed HTTP URI (more than 8 KB) as shown the example error message URI is too large > 8192. In this case, the cluster contains a lot of Materialised Views (MV) and some tables use NodeSync (DSE 6.0 and later) which are automatically added to the list ofignoretafor repair as indicated in the opscenter.log. This scenario generates an HTTP query larger than allowed by the HTTP endpoint on the agent.

When too many tables are added to theignore_tableslist in the [repair_service] section of the cluster configuration file. See the Basic Repair Service configuration reference documentation.

TroubleshootingHTTP call/v1/tokenranges

The KeyError:'x-datastax-agent-version'indicates a problem processing the query on the agent side and is not necessarily an issue of compatibility betweendatastax-agentandOpsCenter.

To troubleshoot, run a status query against the API of the agent directly using cURL. Caution: unless you are aware of the impact of running the URI provided in the warning against the agent, do not directly run the URI provided in the warning/error message.

See the Support Knowledge Base article extracting diagnostics via the API for examples and instructions on running safe queries against the agent. See also the datastax-agent API example curl commandsdocumentation.

A simple example command to request the status of the node without SSL: curl -v -L http://<agentIP>:61621/v1/connection-status?

A simple example command to request the status of the node with SSL: curl -v -k --cert /var/lib/opscenter/ssl/opscenter.pem --key /var/lib/opscenter/ssl/opscenter.key -L https://<agentIP>:61621/v1/connection-status?

Workaround

If update is not possible, DataStax recommends stopping the service in OpsCenter and regularly running manual repair on all the nodes.

On the first run:

$ nodetool repair

Perform maintenance every week or so, depending on the gc_grace_seconds:

$ nodetool repair -pr

Solution

The Repair Service in OpsCenter is unable to proceed when this issue occurs. This issue has been investigated by development under referenceOPSC-13245 and is fixed in OpsCenter 6.7.5 as per the Release Notes Repair Service section - note that both the agents and OpsCenter need to be updated to leverage the feature change

Troubleshooting on a different HTTP call

If the issue occurs on a different HTTP call, consider raising a ticket for Support and be sure toprovide both the agent and opscenter logs.

Summary

This article discusses a known issuewhere DSE Search queries do not return map collection data with warnings in the logs withCannot find dynamic field.

Symptoms

DSE Search queries return incomplete results, specifically missing fields with map collection data. Warning messages are repeatedly reported in system.log. Here are sample entries from a DSE 4.7.3 cluster:

WARN [http-8983-exec-1] 2016-07-28 16:45:43,172 Cql3CassandraRowReader.java:156 - Cannot find dynamic field : [home], please make sure [addresses] is being prepended/appended accordingly to build a correct dynamic field name. Did you mean addresseshome? homeaddresses?

WARN [http-8983-exec-1] 2016-07-28 16:45:43,172 Cql3CassandraRowReader.java:156 - Cannot find dynamic field : [work], please make sure [addresses] is being prepended/appended accordingly to build a correct dynamic field name. Did you mean addresseswork? workaddresses?

Cause

This issue is due to data inserted into a table which contains a map collection with incorrect map keys.

Consider the following table of users with a collection of addresses:

cqlsh> CREATE TABLE users (name text, addresses map<text,text>, PRIMARY KEY (name));

The addresses collection would have the following equivalent dynamic field in schema.xml:

<dynamicField indexed="true" multiValued="false" name="addresses*" stored="true" type="TextField"/>

NOTE - In this instance,name="addresses*" indicates that the field must be prefixed with "addresses*". For example:

cqlsh> INSERT INTO users (name, addresses) VALUES ( 'John', {'addresses_home' : '1 Smith St', 'addresses_work' : '10 Doe Blvd' } );

A Search query of this data returns the following results:

"response": {

"numFound": 1,

"start": 0,

"docs": [

{

"name": "John",

"addresses_home": "1 Smith St",

"addresses_work": "10 Main Blvd"

}

]

}

When data with incorrect map keys are inserted, a Search query has incomplete results with map data missing and generates warning messages in the logs. Here is a sample INSERTwith incorrect map keys:

cqlsh> INSERT INTO users (name, addresses) VALUES ( 'Jane', {'home' : '3 Doe St', 'work' : '30 Capitol Dr' } );

A Search query fails to return the map data for "Jane":

"response": {

"numFound": 2,

"start": 0,

"docs": [

{

"name": "John",

"addresses_home": "1 Smith St",

"addresses_work": "10 Doe Blvd"

},

{

"name": "Jane"

}

]

}

The warning messages above are also logged as a result of the Search query.

Solution

Check the data in the map collections and update the map keys to include either a prefix or suffix depending on the dynamic field schema definition.

In the example above, the record for "Jane" is fixed as follows:

cqlsh> INSERT INTO users (name, addresses) VALUES ( 'Jane', {'addresses_home' : '3 Doe St', 'addresses_work' : '30 Capitol Dr' } );

A Search query then returns expected results with no warnings in the logs:

"response": {

"numFound": 2,

"start": 0,

"docs": [

{

"name": "John",

"addresses_home": "1 Smith St",

"addresses_work": "10 Doe Blvd"

},

{

"name": "Jane",

"addresses_home": "3 Doe St",

"addresses_work": "30 Capitol Dr"

}

]

}

See also

DataStax doc- Using dynamic fields in DSE Search

DataStax doc- Advanced DSE Search tutorial

Overview

This article answers frequently asked questions about the use of thedisk_access_modeincassandra.yamland why this setting is important to DSE.

Applies to

DSE 5.1 and earlier

What is thedisk_access_modesetting incassandra.yaml?

What is thecassandra.yamlsettingdisk_access_mode, which settings can I use, and how can I determine which setting is appropriate for my environment?

In DSE 5.0 and earlier, compressed data (the default) was always effectively read usingdisk_access_mode: mmap_index_only, even though the option was set tommap. This read behavior occurred because the data from disk was decompressed when DSE read it using temp buffers.

With CASSANDRA-8464, this behavior changed to directly compress data to enable mmap to work correctly. The problem is when DSE has a lot of random reads (the hot dataset > memory) it's more efficient to setdisk_access_mode: mmap_index_onlyto prevent page faults.

DataStax recommends conducting normal tuning and running system activity reports. As part of the tuning process, be sure to monitor page faults and system performance on operator machines. For example, use a performance monitoring tool like sar (System Activity Report).

Switching tommap_index_onlyhas improved performance in these use cases:

Linux OOM killer terminates DSE due to rapid off-heap memory growth (mmapallocating too much too fast)

High end percentile latencies (for example, 99% and Max)

Sporadic read timeouts occur, usually in conjunction with latencies as above

If OOM errors, high latencies, and read timeouts are constantly observed, then consider settingdisk_access_mode: mmap_index_only.

NoteThese read errors are not observed in DSE 6.x and later. The standard mode is default and standard data is not mapped into memory.

What are some example use cases?

When checking page faults, you can use sar -B to observe higher page faults as well as page scans and steals (using a sar man page reference):

03:28:24 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff

...

03:49:18 PM 1704.00 0.00 39492.00 107.00 703.00 0.00 0.00 0.00 0.00

03:49:19 PM 1828.00 0.00 21955.00 89.00 1005.00 0.00 0.00 0.00 0.00

03:49:20 PM 1944.00 16.00 43172.00 79.00 981.00 0.00 0.00 0.00 0.00

03:49:21 PM 1868.00 264.00 223040.00 76.00 184417.00 10670.00 57090.00 19921.00 29.40

03:49:22 PM 2500.00 8.00 145066.00 111.00 97732.00 0.00 240918.00 91635.00 38.04

03:49:23 PM 2376.00 12.00 70774.00 116.00 234907.00 0.00 341864.00 223022.00 65.24

When checking pmap, we can see this process is using a large amount of RSS for anon which can again point to a large amount of off-heap mapped memory. Here is an example pmap output:

sudo pmap -x $(cat /run/dse/dse.pid) | awk 'NR==2;/anon/' | more Address Kbytes RSS Dirty Mode Mapping

0000000000400000 4 4 0 r-x-- java

0000000000600000 4 4 4 rw--- java

0000000000bde000 19676 19548 19548 rw--- [ anon ]

00000004c0000000 12588416 12588416 12588416 rw--- [ anon ]

00000007c0560000 1043072 0 0 ----- [ anon ]

00000032c8c00000 128 116 0 r-x-- ld-2.12.so

00000032c8e20000 4 4 4 r---- ld-2.12.so

00000032c8e21000 4 4 4 rw--- ld-2.12.so

00000032c8e22000 4 4 4 rw--- [ anon ]

00000032c9000000 1576 680 0 r-x-- libc-2.12.so

00000032c918a000 2048 0 0 ----- libc-2.12.so

When theres lots of mmap-ing going on, you may see a lot of[ anon ]in the pmap output. This should be verifiable against rapid RSS growth of the DSE process.

Which disk_access_mode mode am I using?

If disk_access_mode is not explicitly set incassandra.yaml, then the default isauto. In your DSEoutput.logorsystem.log,look Node configuration (bolded for emphasis in this example).

For example, we can see the node configuration line and subsequent message:

INFO [main] 2018-12-17 19:35:53,236 Config.java:509 - Node configuration:[aggregated_request_timeout_in_ms=120000;... **disk_access_mode=auto**;...

...

INFO [main] 2018-12-17 19:35:53,237 DatabaseDescriptor.java:350 - **DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap**

The following settings are valid fordisk_access_mode

auto disk access mode is determined automatically. On a 64-bit OS, mmap is used for index files and SSTable files. On a 32-bit OS, mmap is used only for the smaller index files. mmap use mmap for SSTable files and index files mmap_index_only use mmap only for the smaller index files, not the larger SSTable files standard use standard i/o, not mmap, for SSTable files and index files

See also

KB articles:

Increased memory utilisation on nodes after upgrading to DSE 5.0 or 5.1

Nodes become unresponsive during high app traffic periods when mmap is enabled on DSE 6

Credits Erick Ramirez, Thanh Tran, Mike Keeney,Stefania Alborghetti, Greg Smith

Summary

When reading from a cluster with the java driver the user will see a ReadTimeoutException. This can be caused by tombstones.

Symptoms

The customer was trying to read some data from some rather large rows in a table with many partitions. They were seeing the following exception:

ERROR - my_keyspace - Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded)com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded)at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:69)at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258)

Cause

The customer had a large number of tombstones present in the column family and this was causing the timeout. The following error was seen in the cassandra system.log file

ERROR [ReadStage:93968] 2014-12-02 14:16:14,283 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones in my_keyspace.my_table ; query aborted (see tombstone_failure_threshold)

Workaround

Increasing the following setting in the cassandra.yaml file to allow the read query to work

tombstone_failure_threshold:

Solution

Clean up tombstones by ensuring gc_grace_seconds is set to run at a more frequent time to suit your application or use TTLs for certain data. For example the default gc_grace_seconds is 864000 (10 days). If your TTL data is set to 6 days then you might want to change gc_grace_seconds to 604800 (7 days) to remove tombstones sooner.

Note: if you do reduce the gc_grace_seconds, be aware that repairs have to complete within this window too. For more info please see the following documentation link: http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html

Example to change gc_grace_seconds on a table

cqlsh:results> desc table example; CREATE TABLE example ( col1 text, col2 text, PRIMARY KEY ((col1))) WITH bloom_filter_fp_chance=0.010000 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.100000 AND gc_grace_seconds=864000 AND index_interval=128 AND read_repair_chance=0.000000 AND replicate_on_write='true' AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND speculative_retry='99.0PERCENTILE' AND memtable_flush_period_in_ms=0 AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={'sstable_compression': 'LZ4Compressor'};

cqlsh:results> alter table example with gc_grace_seconds = 10000;cqlsh:results> desc table example;

CREATE TABLE example ( col1 text, col2 text, PRIMARY KEY ((col1))) WITH bloom_filter_fp_chance=0.010000 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.100000 AND gc_grace_seconds=10000 AND index_interval=128 AND read_repair_chance=0.000000 AND replicate_on_write='true' AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND speculative_retry='99.0PERCENTILE' AND memtable_flush_period_in_ms=0 AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={'sstable_compression': 'LZ4Compressor'};

Summary

When authentication and authorization are turned on, stdoutand stderrlogs from Spark Worker UI (http://hostname:7081/) cannot be viewed for Finished Drivers/applications. This issue persists even with correct Spark authentication for the current user.

Applies to

DataStax Enterprise 6.0.4, 6.0.5,6.0.6

DataStax Enterprise 6.7.1, 6.7.2

Symptoms

Using the Spark Web UI or the Spark History Server web UI to view executor logs (stdout and/or stderr) of completed applications and drivers, you are unable to view the log contents and this error occurs:

HTTP ERROR 403Problem accessing /logPage/. Reason: you are not allowed to access this resource

Cause

This is a known issue in DSE 6.0.4, DSE 6.0.5, and DSE 6.0.6 (Internal DSP-17854 - Unable to access stderr/stdout from Spark Worker UI for Finished Drivers jobs).

Even when a user has the required permissions, the reference to the driver or executor in the table is removed when the job is completed and causes all requests to view information on the finished jobs to fail.

Workaround

View the logs on the file system of the Spark Worker nodes instead using the Spark Web UI. The UI reads the logs from:

$SPARK_WORKER_DIR/<appID or driverID>/<executor id>/<stdout or stderr>

By default, the SPARK_WORKER_DIR is /var/lib/spark/worker.

Solution

Upgrade to DSE 6.0.7 or later or DSE 6.7.3 or later.

And be sure the user has at least one of the following permissions:

GRANT DESCRIBE ON ANY SUBMISSION TO role_name;

Or

GRANT DESCRIBE ON ANY SUBMISSION IN WORKPOOL 'datacenter_name.*' TO role_name;

Or

GRANT DESCRIBE ON SUBMISSION id IN WORKPOOL 'datacenter_name.*' TO role_name;

Summary

Following a node failure, DSE is unable to start and the system log shows the following error:

ERROR [main] 2019-09-10 06:53:54,116 CassandraDaemon.java:854 - Exception encountered during startup

java.lang.RuntimeException: java.nio.charset.MalformedInputException: Input length = 1

Applies to

DataStax Enterprise (DSE) 6.7

DataStax Enterprise 6.0

DataStax Enterprise 5.1

Symptoms

After restarting a node which previously failed, an error is thrown just after the list of tokens from other nodes is received, without further information as to the cause of the issue. The full stack found in bothsystem.loganddebug.logis as follows:

ERROR [main] 2019-09-10 06:53:54,116 CassandraDaemon.java:854 - Exception encountered during startup

java.lang.RuntimeException: java.nio.charset.MalformedInputException: Input length = 1

at org.apache.cassandra.io.util.FileUtils.readLines(FileUtils.java:638)

at org.apache.cassandra.db.lifecycle.LogReplica.readLines(LogReplica.java:74)

at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)

at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)

at java.util.Iterator.forEachRemaining(Iterator.java:116)

at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)

at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)

at org.apache.cassandra.db.lifecycle.LogReplicaSet.readRecords(LogReplicaSet.java:105)

at org.apache.cassandra.db.lifecycle.LogFile.verify(LogFile.java:160)

at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:479)

at java.util.stream.ReferencePipeline$3$accept(ReferencePipeline.java:193)

at java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1720)

at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)

at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)

at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)

at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:521)

at org.apache.cassandra.db.lifecycle.LogTransaction$LogFilesByName.removeUnfinishedLeftovers(LogTransaction.java:472)

at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:439)

at org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:431)

at org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:553)

at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:676)

at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:282)

at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:524)

at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:694)

at com.datastax.bdp.DseModule.main(DseModule.java:96)

Caused by: java.nio.charset.MalformedInputException: Input length = 1

at java.nio.charset.CoderResult.throwException(CoderResult.java:281)

at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)

at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)

at java.io.InputStreamReader.read(InputStreamReader.java:184)

at java.io.BufferedReader.fill(BufferedReader.java:161)

at java.io.BufferedReader.readLine(BufferedReader.java:324)

at java.io.BufferedReader.readLine(BufferedReader.java:389)

at java.nio.file.Files.readAllLines(Files.java:3205)

at org.apache.cassandra.io.util.FileUtils.readLines(FileUtils.java:631)

... 32 common frames omitted

Cause

As part of the normal compaction process or memtable flush to sstables, temporary files will be created inside the data file directory. Normally these files are short lived and never remain in place, and once a compaction or flush is completed, these are cleaned up.However, if an issue occur during the operation of writing these files or during compaction (e.g.: hardware failure), it can lead to corrupted files remaining in the data directory and causing DSE to fail on next startup.

Solution

WARNINGDepending on how longthe node was down and the smallestgc_grace_periodof the tables, it might be recommended to rebuild the node to avoid resurrecting deleted data rather than using the following solution. Consider this carefully before bringing the node back in the cluster.

To resolve the issue, it is necessary to first identify in which keyspace/table the problem occur.

Add the following logger tologback.xml(e.g. for a package install:/etc/dse/cassandra/logback.xml:

<logger name="org.apache.cassandra.db.ColumnFamilyStore" level="TRACE"/>

Start DSE:$ sudo service dse start. As per previous occurrences, it should fail to start.

Go to thedebug.logand identify the table that is causing the problem:

TRACE [main] 2019-09-10 08:47:06,063 ColumnFamilyStore.java:675 - Removing temporary or obsoleted files from unfinished operations for table <TABLE_NAME>

ERROR [main] 2019-09-10 08:47:06,217 CassandraDaemon.java:854 - Exception encountered during startup

java.lang.RuntimeException: java.nio.charset.MalformedInputException: Input length = 1

Go to the data directory of DSE on that node, as defined in thecassandra.yamlparameterdata_file_directories. Example for a package install:

$ grep -A 5 data_file_directories /etc/dse/cassandra/cassandra.yaml $ ls -ltr <data_file_directories>/<keyspace>/<table>-<id>

The latest .db files and atxn_compact-<id>.logortxn_flush-<id>.logshould show at the bottom of the list as per the example below (note the prefix `aa` is shown as an example for DSE 6.x and might change on earlier versions).Move these files outside of the<data_files_directories>directory:

-rw-r--r-- 1 cassandra cassandra 296 May 14 23:22 aa-txn_flush_<id>.log

-rw-r--r-- 1 cassandra cassandra 296 May 14 23:22 aa-5-bti-Rows.db

-rw-r--r-- 1 cassandra cassandra 60 May 14 23:22 aa-5-bti-Partitions.db

-rw-r--r-- 1 cassandra cassandra 16 May 14 23:22 aa-5-bti-Filter.db

-rw-r--r-- 1 cassandra cassandra 10 May 14 23:22 aa-5-bti-Digest.crc32

-rw-r--r-- 1 cassandra cassandra 33191 May 14 23:22 aa-5-bti-Data.db

-rw-r--r-- 1 cassandra cassandra 55 May 14 23:22 aa-5-bti-CompressionInfo.db

-rw-r--r-- 1 cassandra cassandra 5238 May 14 23:22 aa-5-bti-Statistics.db

-rw-r--r-- 1 cassandra cassandra 94 May 14 23:22 aa-5-bti-TOC.txt

Start DSE:$ sudo service dse start

When DSE is up and running: a. If using NodeSync, it will take care of the keyspaces/tables under its supervision without further interaction. Use nodetool repairfor the other keyspaces/tables. b. If using OpsCenter repair service, disable repair if its ongoing in OpsCenter and run a full manual repair

Summary

When using in-memory tables in DSE. The setting ofdisk_access_modein thecassandra.yamlmust bemmap

Applies to

DSE 5.1 and earlier

Symptoms

If a a table with the compaction strategyorg.apache.cassandra.db.compaction.MemoryOnlyStrategyis created and the node hasdisk_access_mode: standardset in thecassandra.yaml, DSE can fail to start with the following error:

ERROR [main] 2019-01-22 12:15:38,368 MemoryOnlyStrategy.java:401 - There was a problem mlocking a segment for file BigTableReader(path='/var/lib/cassandra/data/mykeyspace/mytable-81718281fa3511e7b2f67198b740403f/.mms_hier_lfd/mc-13-big-Data.db')

ERROR [main] 2018-06-22 12:15:38,368 MemoryOnlyStrategy.java:402 - Exception: java.lang.IllegalArgumentException: SSTable Index is not mmapped, is compression turned off? BufferedSegmentedFile(path='/var/lib/cassandra/data/mykeyspace/mytable -81718281fa3511e7b2f67198b740403f/.mms_hier_lfd/mc-13-big-Index.db', length=355)

Cause

Memory only SSTables have to be mapped into off-heap memory. Therefore, thedisk_access_modesetting must use a mode that supports this.

Solution

Ensure thedisk_access_modesetting incassandra.yamlis set tommap.

See also

DataStax Enterprise documentation cassandra.yaml

Support KB article FAQ - Use of disk_access_mode in DSE 5.1 and earlier

Support KB article Increased memory use on nodes after upgrading to DSE 5.0 or DSE 5.1

Summary

This article discusses an issue that causes DSE Search nodes to be unresponsive to client or sharding requests.

Applies to

DSE 5.1.0 to 5.1.13

DSE 6.0.0 to 6.0.7

DSE 6.7.0 to 6.7.2

Symptoms

When the Apache Solr channel lock is interrupted when an internode handshake fails, a recursive error condition can occur that causes the DSE Search node to be unresponsive to any client or sharding requests and requests from peer DSE search nodes. Queries may timeout and applications receive subsequent errors as a result.

Cause

When an index writer is operating on the Lucene index, the index writer requires a lock on that directory (usually by means of a lock file). Originally reported in LUCENE-8262, the problems described above can occur if this thread is interrupted.

The following error stack may also be observed in the logs:

partition 'pkey1' on 'myks.mytable':

org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed

at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:836)

at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:850)

at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1663)

at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:286)

at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:218)

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:173)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.indexDoc(CassandraDirectUpdateHandler.java:545)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.addDoc(CassandraDirectUpdateHandler.java:138)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doIndex(AbstractSolrSecondaryIndex.java:1272)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doUpdate(AbstractSolrSecondaryIndex.java:1028)

at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex.lambda$indexDocument$7(Cql3SolrSecondaryIndex.java:815)

at io.reactivex.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onSuccess(SingleFlatMapCompletable.java:83)

at io.reactivex.internal.operators.single.SingleObserveOn$ObserveOnSingleObserver.run(SingleObserveOn.java:81)

at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:68)

at org.apache.cassandra.concurrent.StagedScheduler$TPCAwareDisposeTask.run(StagedScheduler.java:203)

at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:61)

at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:52)

at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/disk2/dse/cassandra/data/solr.data/myks.mytable/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],ctime=2019-01-08T11:12:40Z)

at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:166)

at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)

at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)

at org.apache.lucene.codecs.lucene53.Lucene53NormsConsumer.<init>(Lucene53NormsConsumer.java:43)

at org.apache.lucene.codecs.lucene53.Lucene53NormsFormat.normsConsumer(Lucene53NormsFormat.java:77)

at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:366)

at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:193)

at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:725)

at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:571)

at org.apache.lucene.index.DocumentsWriter.lambda$flushInParallel$0(DocumentsWriter.java:744)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

... 1 common frames omitted

Note:In some systems, DataStax has observed that incorrect disk readahead settings cause this problem to occur, although incorrect disk readahead settings do not cause the actual problem.

Internal jira DSP-18211 was raised to cover this scenario.

Workaround

Reload the Solr core with reindex and distributed set to false. Example using dsetool:

dsetool reload_core myks.mytable reindex=false distributed=false

Note:In some cases the node may incur a growth of commitlogs. If you observe this happening, drain and stop / start DSE as soon as convenient after the reload.

Solution

Upgrade to a version with the fix for this issue:

DSE 5.1 - Upgrade to DSE 5.1.14 or later

DSE 6.0 - Upgrade to DSE 6.0.7 or later

DSE 6.7 - Upgrade to DSE 6.7.3 or later

Summary

After upgrading DSE Search from DSE 5.0 or earlier to DSE 5.1 or later, indexing errors might occur if you have previously usedanalyzed text fields with docValues. These indexing errors can cause invalid indexes on nodes, thereby causing incorrect query results. In some cases, unwanted performance issues can occur, particularly if there is a large number of indexing errors.

Important: docValues on analyzed text fields will always produce unreliable results even in releases earlier than DSE 5.1. DataStax does not support using theTextFieldSolr field type withsolr.KeywordTokenizerandsolr.LowerCaseFilterFactoryto achieve single-token, case-insensitive indexing on a CQL text field if you are using functions like grouping, sorting or faceting.

Applies to

DSE 5.1.0 to 5.1.13

DSE 6.0.0 to 6.0.7

DSE 6.7.0 to 6.7.2

Symptoms

You may observe that a previously working query in DSE 5.0 like this example:

$ curl "http://10.101.33.246:8983/solr/wiki.solr/select/?q=id:999888000&facet=true&facet.field=title&facet.limit=10&wt=json&indent=true" {

"responseHeader":{

"status":0,

"QTime":21},

"response":{"numFound":1,"start":0,"maxScore":8.079184,"docs":[

{

"id":"999888000",

"date":"01-JUN-2010 16:48:42.000",

"title":"MC testing solr"}]

},

"facet_counts":{

"facet_queries":{},

"facet_fields":{

"title":[

"MC testing solr",1,

"(You're Puttin') A Rush on Me",0,

"1,2-Dithietane",0,

"1,3-Dithietane",0,

"1,4,2-Dithiazole",0,

"1700s in Scotland",0,

"1906 (film)",0,

"1921 (film)",0,

"1944 Sun Bowl",0,

"1950 Orange Bowl",0]},

"facet_dates":{},

"facet_ranges":{},

"facet_intervals":{}}}

After upgrading to DSE 5.1, this same query returns an empty result for faceting. For example :

$ curl "http://10.101.34.15:8983/solr/wiki.solr/select/?q=id:999888000&facet=true&facet.field=title&wt=json&indent=true" {

"responseHeader":{

"status":0,

"QTime":379},

"response":{"numFound":1,"start":0,"maxScore":8.079604,"docs":[

{

"id":"999888000",

"date":"01-JUN-2010 16:48:42.000",

"title":"MC testing solr"}]

},

"facet_counts":{

"facet_queries":{},

"facet_fields":{

"title":[]},

"facet_ranges":{},

"facet_intervals":{},

"facet_heatmaps":{}}}

You may also observe nodes using higher than normal CPU and errors in the/var/log/cassandra/system.log- these errors might also start to occur on a reindex:

ERROR [wiki.solr Index WorkPool work thread-0] 2019-07-29 15:18:51,241 Cql3SolrSecondaryIndex.java:772 - [wiki.solr]: Exception writing document id 23743719 to the index; possible analysis error: cannot change DocValues type from SORTED to SORTED_SET for field "title"

org.apache.solr.common.SolrException: Exception writing document id 23743719 to the index; possible analysis error: cannot change DocValues type from SORTED to SORTED_SET for field "title"

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.indexDoc(CassandraDirectUpdateHandler.java:709)

at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.addDoc(CassandraDirectUpdateHandler.java:150)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doIndex(AbstractSolrSecondaryIndex.java:1285)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doUpdate(AbstractSolrSecondaryIndex.java:1007)

at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex$2.run(Cql3SolrSecondaryIndex.java:761)

at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex$2.run(AbstractSolrSecondaryIndex.java:943)

at com.datastax.bdp.concurrent.Worker.run(Worker.java:86)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.IllegalArgumentException: cannot change DocValues type from SORTED to SORTED_SET for field "title"

at org.apache.lucene.index.FieldInfos$FieldNumbers.verifyConsistent(FieldInfos.java:291)

at org.apache.lucene.index.FieldInfos$FieldNumbers.setDocValuesType(FieldInfos.java:335)

at org.apache.lucene.index.DefaultIndexingChain.indexDocValueRT(DefaultIndexingChain.java:661)

at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:604)

at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:525)

at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:337)

at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)

at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1722)

at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:314)

at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:246)

at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:201)

... 12 common frames omitted

Cause

Using TextField types with docValues is not recommended and will not reliably return correct facets if there is case sensitive matching involved. The values stored in docValues will not be lower-cased and provide incorrect matching for sorting, aggregation, and so on.

Tip: DataStax recommends using the newLowerCaseStrFieldcustom type introduced in DSE 5.1. The custom field type provides the following features:

Converts the data into lowercase and correctly stores the lowercase data indocValues. (Note docValues will default to false for all field types)

Converts the query values to lowercase.

Workaround

There are a couple of ways to eliminate the error here:

Change the schema to setdocValues=falsefor the field and re-index (withoutdeleteAll). This change will break any faceting etc type queries, but then you can switchdocValuesback and reindex.

Or

Reindex in place (don't change any schema) but usedeleteAll=true.

Solution

The correct solution is to change the schema to use the LowerCaseStrField type for any previous text fields you use with docValues as described in the DataStax documentation Using LowerCaseStrField with search indexes.

Changing a field type in the schema will of course require a full reindex.

Summary:

When enabling commitlog archiving through OpsCenter, a defect has been identified that prevents the designated retention policy from being respected, resulting in unplanned disk usage. This has been identified and raised on internal JIRA OPSC-5439, and manually or programmatically removing the files is an acceptable workaround.

Note:It should be understood that once commitlog archives are removed, point-in-time restoration requiring the removed files is no longer possible.

Symptom:

When commitlog archiving is turned on, files older than the retention period are left in the specified commitlog archive location.

Cause:

An internal marker in the commitlog archival process is not called, resulting in the deletion step failing to be taken.

Workaround:

Manual or automated removal of these files will not interrupt the operation of DSE/Cassandra or OpsCenter in any way. A task in cron to remove any files older than the retention period will usually function as an acceptable workaround.

Overview

This article discusses the various scripts and configuration files associated with a Spark Worker memory configuration.

Applies to

NOTE - This article was specifically written for the following software versions. Althoughit may apply to earlier or future versions, no guarantee is given since it may change at any time.

DataStax Enterprise (DSE) 4.7.x

Apache Spark 1.2.x

Background

Spark configuration can be a minefield for both new and experienced users.

For seasoned Spark users, the additional configuration items introduced by DSE can sometimes be confusing. For new users, the DSE configuration file appears to be yet another place to configure Spark.

In this article, we will try to explain a common Spark configuration item to make sense of the various scripts and configuration files.

Scenario

John has a DSE Analytics cluster running where each worker has 8 GB of memory:

URL:spark://10.1.2.3:7077

Workers: 2

Cores: 16 Total, 16 Used

Memory: 16.2 GB Total, 1024.0 MB Used

Applications: 1 Running, 0 Completed

Drivers: 0 Running, 0 Completed

Status: ALIVE

The problem is thatthe Spark UI shows that each of the 2 executors is only using 512 MB of memory (2 x 512 = 1024.0 MB used).

John believes he has configured the memory correctly but is baffled by the different locations where it can be done.

Files explained

spark-env.sh

This is the generic Spark startup script where all environment variables are configured. This is from open-source Apache Spark.

In the example scenario above, each Spark worker is configured in spark-env.sh with:

export SPARK_WORKER_MEMORY=8192m

export SPARK_WORKER_CORES=8

NOTE - For new Spark users, it is not necessary to specify these since DSE automatically configures them in dse.yaml (see below).

spark-defaults.conf

This is a generic Spark configuration file where Spark properties (as opposed to environment variables in the startup script above) are configured. This is also from open-source Spark.

Configure the available Spark properties in this configuration file for system-wide defaults.

dse.yaml

This is the primary file used specifically by DataStax Enterprise for configuring enhanced functionalitybrought by DSE to Cassandra and other components such as Spark, Solr and Hadoop. The key point is that this is used to configure the enhanced parts of the software components, but does not replace the configuration files of those components.

Put simply, dse.yaml does not replace the configuration files for Spark. It is merely a place where the DSE-enhanced parts of Spark (i.e. enhancements not in open-source Spark) is configured.

In relation to Spark memory and core management, dse.yaml contains the following default property:

initial_spark_worker_resources: 0.7

This property is used for calculating the default memory and cores available to the Spark worker (unless explicitly set in spark-env.sh above). This means that the values for SPARK_WORKER_MEMORY and SPARK_WORKER_CORES get automatically set when DSE is started in Analytics mode.

Resolution

The8 GB of memory available is the total amount of memory that a Spark Worker can allocate to executors on a Spark node(remember thatthere could be1 or more executors running per node).

The 512 MB of memory used is what the executor on the node used out of the available 8 GB. This is because the default memory for eachexecutor is 512 MB.

If for example John wanted each executor to use 4 GB for his application, John should add the following line to spark-defaults.conf:

spark.executor.memory 4g

NOTE - This requires a DSE restart for the change to take effect.

See also

DataStax doc - Configuring Spark nodes

Apache Spark doc - Spark 1.2.1 Configuration

Apache Spark doc - Spark 1.2.1 Available Properties

Overview

Often users may wish to drill down on a query to find out where the most time is spent in different phases. It is also useful to be able to query the percentile latency of such aquery too. This note outlines how to set this up and how to viewthe results.

The examples here will be outlined using "jconsole", however there are many other jmx querying tools available. In many cases where a user is accessing a production system it might not be possible to open JMX ports to remote connections. It is possible to use SSH tunneling and then connect the JMX client that way. However the methods of achieving this are not discussed in this note.

Setup

The setup of the query is actually already covered on the Datastax documentation site, see the following example:

https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/srch/srchCmtQryMbeans.html

In particular for query metrics:

https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/srch/srchQryMbean.html

1. Prepare the query

The first thing that needs to be setup is the query, as outlined in the docs it takes the following form;

http://<ip>:<port>/solr/<core>/select/?<query you need to measure>&query.name=<a distinct name for your query>

For example:

curl "http://192.168.56.20:8983/solr/wiki.solr/select?q=body:The&wt=json&indent=true&query.name=body1"

2. Runthe query

The query needs to run enough times to start to produce some meaningful results, the more queries, the more accurate the results will be. Generally it's a good idea to run at least 1000 iterations. The solr stress tool might also be a good way of generating these.

3. Readthe results

Using a JMX client you can query the QueryMetrics mbean to pull back the required percentile value. This can be done on each node as required. Put the phase in "p0", the query name in "p1" and then percentile in "p2"

For example:

https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding

See also

Some further information and reading

https://support.datastax.com/hc/en-us/articles/204226179

Use the attached multidump.sh script to take thread dumps and generate top -H output at regular intervals while your nodes are experiencing high CPU utilization. The multidump.sh script must be run as the same user that owns the DSE java process; otherwise, the jstack command will not work. The usage is:

$ multidump.sh pid interval count

NOTE: I would like to reiterate that you must run this as the same user that owns the dse process. For example, if the process owner is cassandra, you can run the script using the following syntax:

$ sudo -u cassandra multidump.sh interval count

If DSE is running as a service, use the pid from /var/run/dse.pid. If DSE is started from the command line, use 'ps -ef | grep java' to determine the pid.The recommended parameters are an interval of 5 seconds and a count of 60, which would run for a total of 5 minutes. This will generate two files, jstack.out and top.out, which you should send to us.

If you have difficulties using the multidump script, you can use the multidump-alt script instead. The usage is the same, but it uses kill -3 to take the thread dumps instead ofjstack. This will not kill the process; the JVM interprets the -3 signal as a request to print a thread dump on stdout. When DSE is run as a service, stdout is redirected to /var/log/cassandra/output.log file, so send us that as well as top.out. If DSE is run as a standalone process, find out where its output is redirected and capture the thread dumps from there.

Summary

This article discussed a situation where users who try to access the Spark Master web interface on an upgraded cluster are prompted to enter credentials to login.

Applies to

DataStax Enterprise 6.7

DataStax Enterprise 6.0

DataStax Enterprise 5.1

Symptom

After upgrading an Analytics node, users who try to access the Spark Master web interface (for example http://10.101.32.168:7080/ ) are prompted with a login box. Versions earlier than DSE 5.1 did not require authentication. Below is an example of a sign in dialog box:

Monitoring Spark with the web interface

If users do not provide credentials, a HTTP 401 unauthorized error occurs with a message like:

Problem accessing /. Reason:

Authentication is required to access this resource

Providing incorrect credentials will return the following error:

Problem accessing /. Reason:

Authentication failed

Cause

A new security feature was added in DSE 5.1.0 which leverages DSE Advanced Security to secure cluster components and services. If DSE Unified Authentication is enabled with LDAP or internal authentication, users will be prompted for login credentials when accessing the Spark Master web interface. See DSP-11311 in the 5.1.0 DSE Analytics changes and enhancements section of the release notes.

Note that Kerberos authentication is not supported in the Spark web interface.

Workaround

Spark web interface basic HTTP authentication supports only LDAP and internal authentication. For clusters which use only Kerberos, disable the Spark web UI authentication by following the steps below.

STEP 1 - On the first DSE Analytics node, modify the Spark daemon configuration file. For DSE package installations:

$ vi/etc/dse/spark/spark-daemon-defaults.conf

For DSE tarball installations:

$ vi <path_to_installation>/resources/spark/conf/spark-daemon-defaults.conf

STEP 2 - Disable theSparkUIAuthFilter by commenting out (add a # at the start) or removing the following line:

spark.ui.filters com.datastax.bdp.auth.SparkUIAuthFilter

STEP 3 - Restart DSE on the node for the change to take effect.

If the authentication filter has been disabled correctly, there should not be an entry like this in the logs:

INFO [dispatcher-event-loop-1] 2019-07-28 16:20:18,438 Logging.scala:54 - \

Adding filter: com.datastax.bdp.auth.SparkUIAuthFilter

STEP 4 - Repeat the steps above on the next node until all DSE Analytics node have been done.

See also

Docs -

Summary

This article discusses a workaround for servers running OpsCenter with limited space in /tmp.

Applies to

Opscenter 6.1.x

Symptoms

While attempting to generate a Diagnostics Report in OpsCenter, an error message is displayed on the user's browser indicating that there is insufficient space to build the diagnostic tarball. Here are 2 example errors depending on the version of OpsCenter installed:

Not enough space to store all data for diagnostic tarball

Can't write out diagnostic file for OpsCenter machine, because it will fill up a disk

Similarlyone of the following messages is reported in opscenterd.log :

Not enough space to store all data for diagnostic tarball

Not enough space to write the diagnostic file out on OpsCenter machine

In some cases, it can also report the following error in the logs:

2018-10-12 12:34:56-0600 [] ERROR: Failed to build diagnostic tarball:

A process has ended with a probable error condition: process ended by signal 13.

stderr:

gzip: stdout: No space left on device

exit code: None Traceback (most recent call last):

File "build/lib/python2.7/site-packages/opscenterd/WebServer.py", line 3603, in DiagnosticDataController

ProcessTerminated: A process has ended with a probable error condition: process ended by signal 13.

Cause

When generating a Diagnostics Report, OpsCenter requests each agent to run diagnostic commands, collect logs and configuration files. OpsCenter then receives individual diagnostic data from agents and temporarily places them in the /tmp filesystem to build a single zipped tarball that eventually gets sent back to a user's browser.

Depending on how the OpsCenter server is built, the /tmp filesystem might not have enough capacity to hold all the data collected from the agents and therefore fails to build the Diagnostic Report.

Workaround

Follow these steps to provide an alternate working directory for OpsCenter to use.

STEP 1 - Create a new directory on a filesystem other than /tmp.

STEP 2 - Grant the OpsCenter userfull read and write permissions to the directory. In some cases, it may be necessary to grant permissions on the parent directory as well.

STEP 3 - Locate the OpsCenter startup script:

for packaged installations - /etc/init.d/opscenterd

for tarball installations - <install_location>/bin/opscenter

STEP 4 - Add the following to the script before the lineTWISTD="./bin/twistd":

export TMPDIR="/path/to/alternate/temp"

STEP 5 - Restart OpsCenter for the change to take effect.

STEP 6 - Generate a Diagnostics Report.

Appendix

For versions 6.5.x and 6.7.x of Opscenter, refer to the online docs: Creating an alternate directory for diagnostic information - 6.7.x and Creating an alternate directory for diagnostic information 6.5.x

Changing the IP address on your nodes is as easy as it sounds, Change the IP in the cassandra.yaml on all nodes then do a restart of the service.

After all nodes that have had their IP's changed have been restarted, and you verify that Cassandra is communicating as normal, verify that the old IP's are not in the system.peers table in CQLSH.

to check this run on all nodes:

cqlsh> SELECT * FROM system.peers;

If the old IP's are showing up in the system.peers then you can remove them from each node with:

delete from system.peers where peer = '<IP address>';

Issue

Customer was trying to connect using scala, but was complaining of kerberos issues. He initially provided the following error:

scala> val session=cluster.connectcom.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: sqaz002udw11.russellreynolds.com/172.27.2.73:9042 (com.datastax.driver.core.exceptions.TransportException: [sqaz002udw11.russellreynolds.com/172.27.2.73:9042] Connection has been closed))at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:237)at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:84)at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1622)at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1540)at com.datastax.driver.core.Cluster.init(Cluster.java:151)at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:333)at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:308)at com.datastax.driver.core.Cluster.connect(Cluster.java:250)... 61 elided

Since the error is NoHostAvailableException, it's probably not kerberos related.

Solution

Instead, we found the customer was using ssl for client_encryption_options, so we had to get scala working with ssl, and here's the final code base:

import io.netty.handler.ssl.SslContextBuilderimport javax.net.ssl.TrustManagerFactoryimport java.security.KeyStoreimport com.datastax.driver.core.RemoteEndpointAwareJdkSSLOptionsval ks = KeyStore.getInstance("JKS")val trustStore = new java.io.FileInputStream("/path/to/truststore.jks")ks.load(trustStore, "XXXXXXXX".toCharArray())val tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())tmf.init(ks)val sslContext = javax.net.ssl.SSLContext.getInstance("TLS")sslContext.init(null,tmf.getTrustManagers,null)val sslOptions = RemoteEndpointAwareJdkSSLOptions.builder().withSSLContext(sslContext).build()import javax.security.auth.login.LoginContextval lc = new LoginContext("DseClient")lc.loginval subject = lc.getSubjectval hostAndPort = Seq("localhost","9042")import com.datastax.driver.core.Clusterimport com.datastax.driver.dse.auth._val cluster = Cluster.builder.addContactPointsWithPorts(new java.net.InetSocketAddress(hostAndPort(0),hostAndPort(1).toInt)).withAuthProvider(DseGSSAPIAuthProvider.builder().withSubject(subject).build()).withSSL(sslOptions).buildval session = cluster.connectval rs = session.execute("DROP TABLE IF EXISTS datascience.persons")session.close

The key was moving away fromRemoteEndpointAwareNettySSLOptions and towardRemoteEndpointAwareJdkSSLOptions. If using Netty, it seems there's a lot of dependencies that are required.

Scenario and Symptoms

After an upgrade to DSE 5.1 or later, an existing job that was working in a DSE 5.0 environment can fail with the following error:

java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: scala.None$ is not a valid external type for schema of date staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, clientId), StringType), true) AS clientId#53 +- staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, clientId), StringType), true) +- validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, clientId), StringType) +- getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, clientId) +- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object) +- input[0, org.apache.spark.sql.Row, true]

Cause

As part of the upgrade from DSE 5.0 to 5.1there is also a change to the Apache Spark component from version 1.6.x.x to Apache Spark 2.0.x.x

The error reported above occurs because Apache Spark 2.0 does not accept optional types with Row encoder.

SPARK-19056 (Row encoder should accept optional types) https://issues.apache.org/jira/browse/SPARK-19056

import org.apache.spark.sql.types._

import org.apache.spark.sql.Row

spark.createDataFrame(

sc.parallelize(Seq(Row(None))),

StructType(Seq(StructField("v", StringType, true)))

).first

// Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: scala.None$ is not a valid external type for schema of string

This error is expected behavior. Even though this error did not occur in earlier versions of DSE that included Spark 1.6.x, using optional types with Row encoder is not documented or supported.

Solution

For Spark 2.0 and later, use Dataset for typed operation and custom objects.

For example:

val ds = Seq(1 -> None, 2 -> Some("str")).toDS

ds.toDF // schema: <_1: int, _2: string>

Summary

This article relates to an issue where revoking permissions on a Spark submission fails.

Applies to

DataStax Enterprise 6.7.x

DataStax Enterprise 6.0.x

DataStax Enterprise 5.1.x

Symptom

Example permissions for a completed job:

admin@cqlsh> LIST ALL PERMISSIONS OF sparkrole;

role | username | resource | permission | granted | restricted | grantable

-----------+-----------+------------------------------------------------------------------------+------------+---------+------------+----------- ...

sparkrole | sparkrole | <submission app-20190519161729-0004 in work pool default in Analytics> | MODIFY | True | False | False

sparkrole | sparkrole | <submission app-20190519161729-0004 in work pool default in Analytics> | AUTHORIZE | True | False | False

sparkrole | sparkrole | <submission app-20190519161729-0004 in work pool default in Analytics> | DESCRIBE | True | False | False

...

Where a role has stale permissions, attempts to revoke all the permissions on the application ID fails with "Resource ... doesn't exist".

cqlsh> REVOKE ALL ON SUBMISSION 'app-20190519161729-0004' IN WORKPOOL 'Analytics.default' FROM sparkrole ;

InvalidRequest: Error from server: code=2200 [Invalid query] \

message="Resource <submission app-20190519161729-0004 in work pool default in Analytics> doesn't exist"

Cause

A Spark permission is considered valid when an equivalent entry exists in the shared DSE resource manager table dse_analytics.rm_shared_data.

In the example above, the permissions on the resource app-20190210100644-0850 cannot be revoked because it does not exist in thedse_analytics.rm_shared_datatable. The respective entries in the system_auth.role_permissions table are orphaned since they do not have matching entries in the shared Analytics table.

Workaround

To revoke the stale permissions, manually delete the entries inthe system_auth.role_permissions table:

cqlsh> DELETE FROM system_auth.role_permissions WHERE role='sparkrole' AND resource = 'any_submission_in_any_workpool_in_any_dc/Analytics/default/app-20190519161729-0004' ;

Overview

This article discusses how to check if TieredCompactionStrategy is being used by a table in response to the DataStax Support Alert for DSE Tiered Storage sent on July 11, 2019.

Procedure

$ cqlsh -e "DESCRIBE FULL SCHEMA;" \ | grep -F "org.apache.cassandra.db.compaction.TieredCompactionStrategy"

Add all necessary connection options to cqlsh. For instance, to connect with authentication enabled using SSL:

$ cqlsh -u <user> --ssl -e "DESCRIBE FULL SCHEMA;" \ | grep -F "org.apache.cassandra.db.compaction.TieredCompactionStrategy"

If nothing is returned by the grep, then TieredCompactionStrategy is not being used.

The following example shows a table that is using TieredCompactionStrategy:

$ cqlsh -e "DESCRIBE FULL SCHEMA;" \ | grep -F "org.apache.cassandra.db.compaction.TieredCompactionStrategy" AND compaction = {'class': 'org.apache.cassandra.db.compaction.TieredCompactionStrategy', 'config': 'strategy1', 'max_tier_ages': '3600,7200', 'tiering_strategy': 'TimeWindowStorageStrategy'}

Applies to

DataStax Enterprise 6.7

DataStax Enterprise 6.0

Symptom

This example error can appear in the system.log when running Spark jobs and every executor process dies before it even starts:

ERROR [ExecutorRunner for app-20190724064306-0000/5407] 2019-07-24 07:04:55,032 Logging.scala:91 - Error running executor java.lang.IllegalStateException: Cannot find any build directories. at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248) at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240) at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194) at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117) at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39) at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45) at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63) at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51) at org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:150) at org.apache.spark.deploy.worker.DseExecutorRunner$$anon$2.run(DseExecutorRunner.scala:88)

Cause

We have observed this problem following an upgrade process where the dse-spark-env.sh file had been incorrectly merged by the user.

Solution

Verify that the below environment is set in the dse-spark-env.sh(default location/etc/dse/spark) for all nodes in the cluster:

export SPARK_SCALA_VERSION="2.11"

We want to reiterate why running a major compaction is not advisable. The main downside is that we will end up with one very large sstable that will almost never have a compaction partner in the future. For example if a table on a node has a total size of 500GB, running a major compaction on it means that we end up with a single 500GB sstable. For SizeTieredCompactionStrategy, We would recommend temporarily reconfiguring the following table sub-compaction properties using the ALTER TABLE command : - min_threshold - set to 2 so that only a minimum of 2 similar-sized sstables are required to trigger a minor compaction instead of the default 4 - tombstone_threshold - set to 0.1 so that if at least 10% of an sstable are tombstones, Cassandra will compact the table alone instead of waiting for the higher default ratio of 0.2 - unchecked_tombstone_compaction - set to true to allow Cassandra to run tombstone compaction without having to check if an sstable is eligible for compaction We can make these settings more aggressive by temporarily lowering gc_grace_seconds to below the default 10 days (e.g. 259200 for 3 days). Here is an example command on how to use ALTER TABLE: cqlsh> ALTER TABLE music.album_by_tracks WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'min_threshold': 2}; NOTE - Create a backup of the schema before making any changes using the following command: $ cqlsh -e 'DESCRIBE FULL SCHEMA' > schema.cql You should save a copy of the output schema.cql for future reference. Please be aware that this is just a temporary measure and you should revert the configuration once the desired outcome has been achieved. After monitoring the compactions and you are satisfied that the tombstones have been compacted out, you should reset the sub-compaction properties for the relevant tables. For more information, see Compaction subproperties.