Neo4j FAQs

The Out Of Memory Killer or OOM Killer is a process that the linux kernel employs when the system is critically low on memory. This

situation occurs because the linux kernel has over allocated memory to its processes.

When a process starts it requests a block of memory from the kernel. This initial request is usually a large request that the process

will not immediately or indeed ever use all of. The kernel, aware of this tendency for processes to request redundant memory, over

allocates the system memory. This means that when the system has, for example, 2GB of RAM the kernel may allocate 2.5GB to processes.

Normally, this situation does not cause a problem. However, if enough processes begin to use all of their requested memory blocks

then there will not be enough physical memory to support them all. This means that the running processes require more memory than

is physically available. This situation is critical and must be resolved immediately.

The solution that the linux kernel employs is to invoke the OOM Killer to review all running processes and kill one or more of them

in order to free up system memory and keep the system running.

Note

The OOM Killer will only get invoked when the system is critically low on memory.

Due to its nature, Neo4j will always have a high memory footprint and its always likely to be a candidate to be killed when the OOM

Killer sweeps the running processes. While what gets killed often seems random or simply the highest memory consumer, the OOM Killer

doesnt operate like that. Instead, it chooses which process to kill based on its oom_score. This is a value controled by the

operation system itself based on a number of criteria.

You can check the oom_score of a process by looking at /proc/$PID/oom_score.

While wecan'ttruly prevent Neo4j from being killed if needed, we can adjust the oom_score to make its process less likely to be

terminated by the OOM Killer. To do this, you can edit the following file:

/proc/$PID/oom_score_adj

Note

Files in /proc are not actual files,they'rean interface to lower-levels of the operating system. Therefore you cannot edit them using nano or vi. Rather, you can echo the value into the file as such:

echo -100 > /proc/$PID/oom_score_adj

You can adjust the value on this file, valid values are integers in the range of -1000 to 1000. The lower the value, the lower the

chance that its going to be killed.

Again this will not prevent Neo4j from being killed, but it will influence the likelihood of it happening.

Be aware that if the process restarts, you will have to set the oom_score_adj again. If you want to implement this as a more

permanent solution its recommended that you automate this adjustment whenever Neo4j starts, for its new PID.

Last Modified: 2019-11-27 11:49:11 UTC by Jos Rocha.

Relevant for Neo4j Versions: 3.0, 3.1, 3.2, 3.3, 3.4, 3.5.

Relevant keywords performance, memory, java, oom, out of memory,linux.

Starting with version 3.4, we changed in the behaviour of how instances sync with the Leader. This change can potentially affect the

behaviour of your cluster when using strategy plugins,

depending on your configuration.

Pre-3.4 - on the context of multi datacenter architectures - strategy plugins were sets of rules that defined how Read Replicas contacted servers in the cluster in order to synchronize transaction logs. Post-3.4 this was expanded to Core instances as well, meaning that you can actively select the desired strategy from where your Core instances pull updates from.

This is particularly important if you have a user-defined strategy

that restricts the instances from where your Read Replicas pull updates from.

Imagine the following scenario:

here

Figure 1: North & South DCs

Lets say you want to prevent the Read Replica in the South DC to pull updates from the North DC. You could set

causal_clustering.upstream_selection_strategy=user-defined and have the following strategy configured

causal_clustering.user_defined_upstream_strategy=groups(south); halt()

Note

Please note that the upstream strategy defined above only works when you define server groups on your instances. In the picture, all instances on the South DC belong to server group south whereas the ones on the North DC belong to server group north. You can read more about

server groups .

This will effectively prevent the Read Replica to connect to the North DC. However, what weve seen in several implementations was that

- for ease or coherence - customers would set causal_clustering.upstream_selection_strategy to be the same on ALL instances, knowing

that setting would only apply to Read Replicas. Post-3.5, if you do this, the Core instances in the South DC in the example above will

not be able to pull updates from Leader over at the North DC.

It is therefore recommended that, for Core instances, you configure a less restrictive upstream strategy. You can use the ones available

out-of-the-box:

connect-to-random-core-server - Connect to any Core Server selecting at random from those currently available.

typically-connect-to-random-read-replica - Connect to any available Read Replica, but around 10% of the time connect to any

random Core Server.

connect-randomly-to-server-group - Connect at random to any available Read Replica in any of the server groups specified in

the comma-separated list causal_clustering.connect-randomly-to-server-group.

leader-only - Connect only to the current Raft leader of the Core Servers.

Setting the Core instances upstream strategy to leader-only will make Neo4j behave like pre-3.4 but you can choose another one.

The important is to make sure you select a strategy that will allow your Core instances to pull from the Leader regardless of where

its located.

Also, you can select multiple upstream strategies by separating them with a comma (","). It is perfectly acceptable to have the following

configuration:

causal_clustering.upstream_selection_strategy=user-defined, leader-only

This will first try the user-defined strategy and then the leader-only strategy.

Last Modified: 2019-11-27 11:50:42 UTC by Jos Rocha.

Relevant for Neo4j Versions: 3.2, 3.3, 3.4, 3.5.

Relevant keywords network, multi-dc.