CloudBees FAQs

Issue

The build console output shows a similar message:

Invoking ambiguous Pipeline Step triggerRemoteJob (org.jenkinsci.plugins.ParameterizedRemoteTrigger.pipeline.RemoteBuildPipelineStep). triggerRemoteJob could refer to any of the following steps: [org.jenkinsci.plugins.ParameterizedRemoteTrigger.pipeline.RemoteBuildPipelineStep, com.cloudbees.opscenter.triggers.RemoteTriggerBuilderStep].

Followed by:

You can invoke steps by class name instead to avoid ambiguity. For example: steps.'org.jenkinsci.plugins.ParameterizedRemoteTrigger.pipeline.RemoteBuildPipelineStep'(...)

The warning log you are seeing was added in JENKINS-53333. Which was released as part of Pipeline: Groovy plugin version 2.55.

The issue is that there are 2 plugins defining the Pipeline step triggerRemoteJob.

The plugins are:

operations-center-context

parameterized-remote-trigger

This means that when calling the short form (triggerRemoteJob) Jenkins Pipeline engine is unable to determine which step you are referring to.

Prior to Pipeline: Groovy plugin version 2.55, Pipeline would just choose whatever step it found first.

Related

https://issues.jenkins-ci.org/browse/JENKINS-53333

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Parameterized-Remote-Trigger Plugin

Pipeline: Groovy >= 2.55

Resolution

Instead of using triggerRemoteJob(...), use:

steps.'org.jenkinsci.plugins.ParameterizedRemoteTrigger.pipeline.RemoteBuildPipelineStep'(...) - if you want to use the parameterized-remote-trigger implementation.

steps.'com.cloudbees.opscenter.triggers.RemoteTriggerBuilderStep'(...) - if you want to use the Operations Center implementation.

OR

Uninstall the parameterized-remote-trigger plugin so that the step is no longer ambiguous.

Issue

I would like to use Kubernetes Pod Security Policies with CloudBees Core on Modern Cloud Platforms.

Environment

CloudBees Core on Modern Cloud Platforms

Kubernetes Plugin

Resolution

Pod Security Polices are an optional Kubernetes feature (and still beta but very stable and available for all major cloud providers) so they are not enabled by default on most Kubernetes distributions - to include GCP GKE, and Azure AKS. PSPs can be created and applied to a ClusterRole or a Role resource definition without enabling the PodSecurityPolicy admission controller. This is important, because once you enable the PodSecurityPolicy admission controller any pod that does not have a PSP applied to it will not get scheduled.

It is recommended that you define at least two Pod Security Policies for your Core Modern Kubernetes cluster:

A restrictive Pod Security Policy used for all CloudBees components, additional Kubernetes services being leveraged with Core Moder and the majority of dynamic ephemeral Kubernetes based agents used by you Core Modern cluster

The second Pod Security Policy will be almost identical except for RunAsUser will be set to RunAsAny to allow running as root - this is specifically to run Kaniko containers (Please refer to Using Kaniko with CloudBees Core ), but you may have other uses cases that require containers to run as root

Restrictive Pod Security Policy

Create the following restrictive Pod Security Policy (PSP) (or one like it):

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

name: cb-restricted

annotations:

seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'

apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'

seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'

apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

spec:

# prevents container from manipulating the network stack, accessing devices on the host and prevents ability to run DinD

privileged: false

fsGroup:

rule: 'MustRunAs'

ranges:

# Forbid adding the root group.

- min: 1

max: 65535

runAsUser:

rule: 'MustRunAs'

ranges:

# Don't allow containers to run as ROOT

- min: 1

max: 65535

seLinux:

rule: RunAsAny

supplementalGroups:

rule: RunAsAny

# Allow core volume types. But more specifically, don't allow mounting host volumes to include the Docker socket - '/var/run/docker.sock'

volumes:

- 'emptyDir'

- 'secret'

- 'downwardAPI'

- 'configMap'

# persistentVolumes are required for CJOC and Managed Master StatefulSets

- 'persistentVolumeClaim'

- 'projected'

# nfs (EFS) may be used for OC and Managed Master persistent volumes

- 'nfs'

hostPID: false

hostIPC: false

hostNetwork: false

# Ensures that no child process of a container can gain more privileges than its parent

allowPrivilegeEscalation: false

Create a ClusterRole that uses the cb-restricted PSP (this can be applied to as many ServiceAccounts as necessary):

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

name: psp-restricted-clusterrole

rules:

- apiGroups:

- extensions

resources:

- podsecuritypolicies

resourceNames:

- cb-restricted

verbs:

- use

Bind the restricted ClusterRole to all the ServiceAccounts in the cloudbees-core Namespace (or whatever Namespace you deployed CloudBees Core). The following RoleBinding will apply to both of the CloudBees Core Modern defined Roles - the cjoc ServiceAccount for provisioning Managed/Team Masters StatefulSets from CJOC and the jenkins ServiceAccount for scheduling dynamic ephemeral agent pods from Managed/Team Masters:

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: cb-core-psp-restricted

namespace: cloudbees-core

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: psp-restricted-clusterrole

subjects:

# All service accounts in ingress-nginx namespace

- apiGroup: rbac.authorization.k8s.io

kind: Group

name: system:serviceaccounts

Kaniko Pod Security Policy (RunAsRoot)

Create the following PSP for running Kaniko jobs and other Pods that must run as root:

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

name: kaniko-psp

annotations:

seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'

apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'

seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'

apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

spec:

# prevents container from manipulating the network stack, accessing devices on the host and prevents ability to run DinD

privileged: false

fsGroup:

rule: 'RunAsAny'

runAsUser:

rule: 'RunAsAny'

seLinux:

rule: RunAsAny

supplementalGroups:

rule: RunAsAny

# Allow core volume types. But more specifically, don't allow mounting host volumes to include the Docker socket - '/var/run/docker.sock'

volumes:

- 'emptyDir'

- 'secret'

- 'downwardAPI'

- 'configMap'

# persistentVolumes are required for CJOC and Managed Master StatefulSets

- 'persistentVolumeClaim'

- 'projected'

hostPID: false

hostIPC: false

hostNetwork: false

# Ensures that no child process of a container can gain more privileges than its parent

allowPrivilegeEscalation: false

Create a ServiceAccount, Role and RoleBindings for use with Kaniko Pods:

---

apiVersion: v1

kind: ServiceAccount

metadata:

name: kaniko

---

kind: Role

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

name: kaniko

rules:

- apiGroups: ['extensions']

resources: ['podsecuritypolicies']

verbs: ['use']

resourceNames:

- kaniko-psp

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: RoleBinding

metadata:

name: kaniko

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: kaniko

subjects:

- kind: ServiceAccount

name: kaniko

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: cjoc-kaniko-role-binding

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: cjoc-agents

subjects:

- kind: ServiceAccount

name: kaniko

Update all Kaniko related Pod Templates and/or Pod Template yaml to use the kaniko ServiceAccount instead of the default jenkins ServiceAccount. Here is an example yaml based Jenkins Kubernetes Pod Template configuration:

kind: Pod

metadata:

name: kaniko

spec:

serviceAccountName: kaniko

containers:

- name: kaniko

image: gcr.io/kaniko-project/executor:debug-v0.10.0

imagePullPolicy: Always

command:

- /busybox/cat

tty: true

volumeMounts:

- name: kaniko-secret

mountPath: /secret

env:

- name: GOOGLE_APPLICATION_CREDENTIALS

value: /secret/kaniko-secret.json

volumes:

- name: kaniko-secret

secret:

secretName: kaniko-secret

securityContext:

runAsUser: 0

Bind Restrictive PSP Role for Ingress Nginx

CloudBees recommends the ingress-nginx Ingress controller to manage external access to Core Modern. The NGINX Ingress Controller is a top-level Kubernetes project and provides an example for using Pod Security Policies with the ingress-nginx Deployment. Basically, all you have to do is run the following command before installing the NGINX Ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/psp/psp.yaml

The above command will create the following PSP, Role and RoleBinding:

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

annotations:

# Assumes apparmor available

apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'

apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'

seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'

name: ingress-nginx

spec:

allowedCapabilities:

- NET_BIND_SERVICE

allowPrivilegeEscalation: true

fsGroup:

rule: 'MustRunAs'

ranges:

- min: 1

max: 65535

hostIPC: false

hostNetwork: false

hostPID: false

hostPorts:

- min: 80

max: 65535

privileged: false

readOnlyRootFilesystem: false

runAsUser:

rule: 'MustRunAsNonRoot'

ranges:

- min: 33

max: 65535

seLinux:

rule: 'RunAsAny'

supplementalGroups:

rule: 'MustRunAs'

ranges:

# Forbid adding the root group.

- min: 1

max: 65535

volumes:

- 'configMap'

- 'downwardAPI'

- 'emptyDir'

- 'projected'

- 'secret'

---

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

name: ingress-nginx-psp

namespace: ingress-nginx

rules:

- apiGroups:

- policy

resourceNames:

- ingress-nginx

resources:

- podsecuritypolicies

verbs:

- use

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: ingress-nginx-psp

namespace: ingress-nginx

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: ingress-nginx-psp

subjects:

- kind: ServiceAccount

name: default

- kind: ServiceAccount

name: nginx-ingress-serviceaccount

NOTE: You can also run that command after you have already installed the NGINX Ingress controller but PSP will only be applied after restarting or recreating the ingress-nginx Deployment.

Enable the Pod Security Policy Admission Controller

Once PSPs have been applied to all the ServiceAccounts in your Kubernetes cluster you can enable the Pod Security Policy Admission Controller:

External documentation on enabling and using Pod Security Policies:

AWS EKS (enabled by default for version 1.13 or later): https://docs.aws.amazon.com/eks/latest/userguide/pod-security-policy.html

GCP GKE (beta feature): https://cloud.google.com/kubernetes-engine/docs/how-to/pod-security-policies

Azure AKS (preview feature): https://docs.microsoft.com/en-us/azure/aks/use-pod-security-policies

PKS: https://docs.pivotal.io/pks/1-5/pod-security-policy.html

Restrictive Pod Security Policies and Jenkins Kubernetes Pod Template Agents

The Jenkins Kubernetes plugin (for ephemeral K8s agents) defaults to using a K8s emptyDir volume type for the Jenkins agent workspace. This causes issues when using a restrictive PSP such at the cb-restricted PSP above. Kubernetes defaults to mounting emptyDir volumes as root:root with permissions set to 750 - as detailed by this GitHub issue. In order to run containers in a Pod Template as a non-root user you must specify a securityContext at the container or pod level. There are at least two ways to do this (in both cases at the pod level):

In the Kubernetes cloud configuration UI via the Raw yaml for Pod field:

In the raw yaml of a pod spec that you embed or load into your Jenkins job from a file:

kind: Pod

metadata:

name: nodejs-app

spec:

containers:

- name: nodejs

image: node:10.10.0-alpine

command:

- cat

tty: true

- name: testcafe

image: gcr.io/technologists/testcafe:0.0.2

command:

- cat

tty: true

securityContext:

runAsUser: 1000

Description

It is possible, before the completion of the step, for the Flowserver to declare a timeout error (AGENT_TIMEOUT) even though the step as completed. You can see in the step log that the steps completed normally and the step can even show start and end timestamps in the log file.

Typically, this occurs when you have a heavily loaded agent, where the agent machine is so bogged down processing one or more commands and possibly paging, that the Flowagent service cannot process messages from the Flowserver. One way to alleviate the problem is to configure the server to be more tolerant of busy agents.

By default, every thirty seconds, the server sends a ping message to the agent to verify that the agent is still healthy. If several of these ping messages go unanswered, the server concludes that it has a dead agent and aborts any running steps (in the server's record of job step status). Because the Flowserver cannot communicate with the Flowagent, it does not send an abort message to the agent, and the commands currently executing on the agent continue to run to completion.

Solution

Increase the Flowserver's setting for the agent-ping timeout. This is one setting at the server that affects all agents. If it takes one minute to respond, set a 1.5 minute timeout instead of the default 0.5 minutes.

To set the wait for agent response time, which is in milliseconds, to 1.5 minutes use:

ectool --server setProperty /server/settings/agentSocketTimeout 90000

The Flowserver does not need to be restarted for the new value to take effect. Rerun the step to see if the agentSocketTimeout had the desired effect.

Bonus Solution

Overburdened agents may not respond to a request for a new connection quickly either.

The agentConnectionTimeout property is the time the Flowserver waits for a request for a connection to time out. The does not happen as often as it is usually the agent step activity initiated over the connection causing the slow response.

To set the wait for socket creation, which in in milliseconds, to 1.5 minutes use:

ectool --server setProperty /server/settings/agentConnectionTimeout 90000

The Flowserver does not need to be restarted for the new value to take effect. Rerun the step to see if the agentConnectionTimeout had the desired effect.

Issue

When I am trying to update the Jenkins configuration and hit save, I am getting a StackOverflow exception as shown below:

WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving http://jenkinsurl:8080/configSubmit

java.lang.StackOverflowError

at java.lang.System.getProperty(System.java:714)

at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:84)

at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:49)

at java.security.AccessController.doPrivileged(Native Method)

at java.io.BufferedWriter.<init>(BufferedWriter.java:109)

at java.io.BufferedWriter.<init>(BufferedWriter.java:88)

at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:76)

at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:57)

at hudson.XmlFile.write(XmlFile.java:168)

at hudson.model.Descriptor.save(Descriptor.java:872)

at hudson.model.Saveable$save.callCurrent(Unknown Source)

Environment

CloudBees Jenkins Enterprise (CJE)

CloudBees Jenkins Enterprise - Managed Master (CJE-MM)

CloudBees Jenkins Team (CJT)

CloudBees Jenkins Platform - Client Master (CJP-CM)

Jira Trigger Plugin versions below 0.2.4

Resolution

This is a known Jenkins issue Jenkins-39096 which is fixed in versions of the Jira Trigger plugin equal or higher than 0.2.4.

You can check the plugin changelog to get additional information on the changes included in the different versions.

Issue

When you try to access the instance, you get a Bad Message 413 (Request Entity Too Large) error. This is a known Jenkins issue JENKINS-25046, specifically when using Jetty as the web server. As each time that Jenkins restarts its session cookie changes. The problem is that after some time, your browser can have a bunch of session cookies each one with a different name. This causes the cookie request header to be very long, this will cause the server to return a 413 response and a blank page.

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

In Jenkins versions higher than 2.66, you will have a Jetty version included that will allow you to customize the above described behavior via the following System properties:* executableWar.jetty.disableCustomSessionIdCookieName the default value for this property is false. You could set this to true to prevent the problem described above.* executableWar.jetty.sessionIdCookieName which defaults to null. But sets a custom session ID cookie name when the previous parameter is set to false.

References

Jetty Session IDs

Issue

Jenkins jobs will sit in the build queue, and not get started, even when there are build agents available for the chosen label, and you see stack traces in the logs similar to the one shown below:

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXXX failed

How do we know what is causing this queue freeze?

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

The Queue.MaintainTask is a periodic task which is run in the instance and it is responsible for maintenance operations such as adding elements to the queue, assigning elements in the queue to nodes or executors, etc. If for some reason, this task fails, this causes the queue to become unresponsive and the jobs eventually stop being run as they stay stuck in the queue.

In order to determine what is the cause for this problem, we need to pay special attention to the full stack trace of the error which shows up in the logs.

You will be able to see some potential causes below, the intent of the list below is to allow you understand the pattern that you might follow to determine the root cause of the failure, thus helping you recover the instance as fast as possible.

Nodes Plus Plugin

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXX failed

Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from XXXX/XXX:XXX

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)

at hudson.remoting.Channel.call(Channel.java:957)

at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1059)

at hudson.Launcher$ProcStarter.start(Launcher.java:455)

at com.cloudbees.jenkins.plugins.nodesplus.CustomNodeProbeBuildFilterProperty.getProbeResult(CustomNodeProbeBuildFilterProperty.java:180)

In the stacktrace we can clearly see that there is a correlation between the getProbeResult method and the task failure.

Microfocus Plugin

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXX failed

java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.core.impl.Log4jLogEvent

at org.apache.logging.log4j.core.impl.DefaultLogEventFactory.createEvent(DefaultLogEventFactory.java:54)

at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:401)

at org.apache.logging.log4j.core.config.DefaultReliabilityStrategy.log(DefaultReliabilityStrategy.java:49)

at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)

at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2116)

at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2100)

at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1994)

at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1966)

at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:739)

at com.microfocus.application.automation.tools.octane.events.WorkflowListenerOctaneImpl.onNewHead(WorkflowListenerOctaneImpl.java:79)

In this case, the exception thrown is different but the effect is the same, the periodic task starts failing.

Build Blocker Plugin

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXX failed

java.util.regex.PatternSyntaxException: Dangling meta character '*' near index 0

*.XXX.*XXX

^

at java.util.regex.Pattern.error(Pattern.java:1957)

at java.util.regex.Pattern.sequence(Pattern.java:2125)

at java.util.regex.Pattern.expr(Pattern.java:1998)

at java.util.regex.Pattern.compile(Pattern.java:1698)

at java.util.regex.Pattern.<init>(Pattern.java:1351)

at java.util.regex.Pattern.compile(Pattern.java:1028)

at java.util.regex.Pattern.matches(Pattern.java:1133)

at java.lang.String.matches(String.java:2121)

at hudson.plugins.buildblocker.BlockingJobsMonitor.checkForPlannedBuilds(BlockingJobsMonitor.java:162)

at hudson.plugins.buildblocker.BlockingJobsMonitor.checkForQueueEntries(BlockingJobsMonitor.java:86)

at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:151)

Again, the stacktrace will allow us to determine the source of the problem affecting the queue.

Pipeline Graph Analysis Plugin

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXX failed

java.lang.IndexOutOfBoundsException: Index: 0

at java.util.Collections$EmptyList.get(Collections.java:4454)

at org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView.bruteForceScanForEnclosingBlock(StandardGraphLookupView.java:150)

at org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView.findEnclosingBlockStart(StandardGraphLookupView.java:197)

at org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView.findAllEnclosingBlockStarts(StandardGraphLookupView.java:217)

Block Queued Job Plugin

SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.model.Queue$MaintainTask@XXX failed

java.lang.NullPointerException

at org.jenkinsci.plugins.blockqueuedjob.condition.JobResultBlockQueueCondition.isBlocked(JobResultBlockQueueCondition.java:70)

at org.jenkinsci.plugins.blockqueuedjob.BlockItemQueueTaskDispatcher.canRun(BlockItemQueueTaskDispatcher.java:35)

at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)

Workaround

Depending on the situation that you are experiencing, you might need to upgrade or disable (if possible) the corresponding plugin showing up as responsible for the failure.

Issue

After upgrading CloudBees Core to release 2.190.2.2, Kubernetes planned agents are stuck in Pending and builds are hanging forever, waiting for the planned agents to come online with Waiting for next available executor.

In such cases, the agent pod are not even scheduled

Environment

CloudBees Core 2.190.2.2

CloudBees Core on Modern Cloud Platforms - Managed Master 2.190.2.2

CloudBees Core on Modern Cloud Platforms - Operations Center 2.190.2.2

CloudBees Core on Traditional Platforms - Client Master 2.190.2.2

CloudBees Core on Traditional Platforms - Operations Center 2.190.2.2

CloudBees Jenkins Platform - Client Master 2.190.2.2

CloudBees Jenkins Platform - Operations Center 2.190.2.2

CloudBees Jenkins Distribution 2.190.2.2

Jenkins LTS

Kubernetes Plugin from version 1.19.1 to 1.21.2 (excluded)

Related Issue(s)

JENKINS-56307 (regression)

JENKINS-60055 (fix)

Explanation

This is a bug in the kubernetes plugin introduced by JENKINS-56307 in version 1.19.1 of the kubernetes plugin.

This version introduces a new Node Provisioner strategy NoDelayProvisionerStrategy that is enabled by default. The strategy provisions a node as soon as the Node Provisioner detects a need for more agents. As opposed to the default strategy that makes his decision based on load estimates.

There is a bug in the implementation that causes agents to never be provisioned and builds to hang.

Resolution

This issue has been fixed in version 1.21.2 of the kubernetes plugin.

Solution

The solution is to upgrade the Kubernetes plugin to version 1.21.2 that contains the fix

Note At the moment of writing this article, this version is not yet available under the CloudBees Assurance Program

Workaround

The workaround is to disable the NoDelayProvisionerStrategy. This can be done by adding the system property -Dio.jenkins.plugins.kubernetes.disableNoDelayProvisioning=true to the Masters startup. This requires to restart the Master in order to take effect. See How to add Java arguments to Jenkins to under stand how to do this.

Summary

This article outlines the timeout settings in Cloubees Flow.

Timeouts that can be configured in with UI settings (Administration > Settings):

Agent abort timeout -Time the server waits for responses to the abort agent command. After this timeout has elapsed, the step is force aborted.Defaults to 90 seconds.

Agent connection timeout -Time to wait for connections to agents to open.Defaults to 60 seconds.

Idle connection timeout -The number of milliseconds to keep idle connections open. Defaults to 5 minutes.

Agent socket timeout -The number of milliseconds the server waits for responses to agent commands. Defaults to 60 seconds.

Idle login session timeout -Number of minutes before an idle user session is terminated. Defaults to 3 days.

Webhook session timeout -Number of minutes before a webhook session is terminated. Defaults to 180 days.

Timeouts that can be configured in with ecconfigure command:

Agent Configuration Options:

--agentIdleConnectionTimeout=milliseconds -Idle connection timeout, in milliseconds.

--agentIdleOutboundConnectionTimeout=seconds -Idle time after which an outbound connection is closed, in seconds.

--agentIdlePostRunnerTimeout=seconds -Idle time after which a PostRunner thread is terminated, in seconds.

--agentIdleServerRequestWorkerTimeout=seconds -Idle time after which a ServerRequestWorker thread is terminated, in seconds.

--agentIdleWorkerTimeout=seconds -Idle time after which a Worker thread is terminated, in seconds.

--agentOutboundConnectTimeout=milliseconds -Timeout for the agent establishing outbound connections, in milliseconds.

--agentOutboundRequestTimeout=hours -Timeout after which the agent gives up trying to send a request to a server, in hours.

--agentServerConnectTimeout=seconds -Socket connection timeout for outbound requests to a server, in seconds.

--agentServerReadTimeout=seconds -Socket read timeout for responses from a server, in seconds.

--agentWrapperPingTimeout=seconds -Java virtual machine-wrapper heartbeat timeout, in seconds.

--agentWrapperShutdownTimeout=seconds -The wrapper's shutdown timeout, in seconds.

--agentWrapperStartupTimeout=seconds -The wrapper's startup timeout, in seconds.

Server Configuration Options:

--serverWrapperPingTimeout=seconds -Java virtual machine-wrapper heartbeat timeout, in seconds.

--serverWrapperStartupTimeout=seconds -The wrapper's startup timeout, in seconds.

--serverWrapperShutdownTimeout=seconds -The wrapper's shutdown timeout, in seconds.

Repository Server Configuration Options:

--repositoryIdleConnectionTimeout=milliseconds -The idle connection timeout, in milliseconds.

--repositoryWrapperPingTimeout=seconds -Java virtual machine-wrapper heartbeat timeout, in seconds.

--repositoryWrapperStartupTimeout=seconds -The wrapper's startup timeout, in seconds.

--repositoryWrapperShutdownTimeout=seconds -The wrapper's shutdown timeout, in seconds.

Issue

I would like to use the CloudBees Flow Perl API to look up the history for a procedure of a job.

Environment

CloudBees Flow

Resolution

You can create a script that will query for all runs of a project (calledtest2in the below script), for a specific procedure of that project (calledtestProcedurein the below script):

use ElectricCommander;

my $ec = ElectricCommander->new({

server => "localhost",

});

# TODO update username and password

$ec->login('username','password');

# TODO update "test2" and "testProcedure"

my $result = $ec->findObjects({

objectType => 'job',

filter => {

operator => "and",

filter => [{

propertyName => 'projectName',

operator => "equals",

operand1 => "test2"

},{

propertyName => 'procedureName',

operator => "equals",

operand1 => "testProcedure"

},],

},

});

print $result->findnodes_as_string("/"). "\n";

This script is using the API command calledfindObjects, as documented here: http://docs.electric-cloud.com/eflow_doc/9_1/API/HTML/FlowAPI_Guide_9_1.htm#EFlow_api/APImisc.htm#findObjects

When you save the above script as a file, such asjobProcedureHistory.pland then run this perl script using the command:

ec-perl jobProcedureHistory.pl

As documented at: Using Perl (ec-perl)

You will see output similar to below, showing the history:

<responses version="2.3" dispatchId="2612" nodeId="10.0.2.15">

<response requestId="2" nodeId="10.0.2.15">

<objectId>job-e718c0c6-f5dc-11e9-8728-080027bd4d5b</objectId>

<objectId>job-ee1f0064-f5dc-11e9-8728-080027bd4d5b</objectId>

<objectId>job-f09e3193-f5dc-11e9-bb4c-080027bd4d5b</objectId>

<object>

<objectId>job-e718c0c6-f5dc-11e9-8728-080027bd4d5b</objectId>

<job>

<jobId>e718c0c6-f5dc-11e9-8728-080027bd4d5b</jobId>

<jobName>job_28_20191023173433</jobName>

<archived>0</archived>

<combinedStatus>

<message>Error</message>

<status>completed_error</status>

</combinedStatus>

<createTime>2019-10-23T21:34:33.010Z</createTime>

<directoryName>job_28_20191023173433</directoryName>

<elapsedTime>1017</elapsedTime>

<environmentWaitTime>0</environmentWaitTime>

<external>0</external>

<finish>2019-10-23T21:34:34.027Z</finish>

<lastModifiedBy>admin</lastModifiedBy>

<launchedByUser>admin</launchedByUser>

<licenseReshareWaitTime>0</licenseReshareWaitTime>

<licenseWaitTime>0</licenseWaitTime>

<liveProcedure>testProcedure</liveProcedure>

<modifyTime>2019-10-23T21:34:34.210Z</modifyTime>

<outcome>error</outcome>

<owner>admin</owner>

<priority>normal</priority>

<procedureName>testProcedure</procedureName>

<projectName>test2</projectName>

<propertySheetId>e718c0ca-f5dc-11e9-8728-080027bd4d5b</propertySheetId>

<resourceWaitTime>0</resourceWaitTime>

<start>2019-10-23T21:34:33.010Z</start>

<status>completed</status>

<totalWaitTime>0</totalWaitTime>

<workspaceWaitTime>0</workspaceWaitTime>

</job>

</object>

<object>

<objectId>job-ee1f0064-f5dc-11e9-8728-080027bd4d5b</objectId>

<job>

<jobId>ee1f0064-f5dc-11e9-8728-080027bd4d5b</jobId>

<jobName>job_29_20191023173444</jobName>

<archived>0</archived>

<combinedStatus>

<message>Error</message>

<status>completed_error</status>

</combinedStatus>

<createTime>2019-10-23T21:34:44.799Z</createTime>

<directoryName>job_29_20191023173444</directoryName>

<elapsedTime>1102</elapsedTime>

<environmentWaitTime>0</environmentWaitTime>

<external>0</external>

<finish>2019-10-23T21:34:45.901Z</finish>

<lastModifiedBy>admin</lastModifiedBy>

<launchedByUser>admin</launchedByUser>

<licenseReshareWaitTime>0</licenseReshareWaitTime>

<licenseWaitTime>0</licenseWaitTime>

<liveProcedure>testProcedure</liveProcedure>

<modifyTime>2019-10-23T21:34:46.016Z</modifyTime>

<outcome>error</outcome>

<owner>admin</owner>

<priority>normal</priority>

<procedureName>testProcedure</procedureName>

<projectName>test2</projectName>

<propertySheetId>ee1f2778-f5dc-11e9-8728-080027bd4d5b</propertySheetId>

<resourceWaitTime>0</resourceWaitTime>

<start>2019-10-23T21:34:44.799Z</start>

<status>completed</status>

<totalWaitTime>0</totalWaitTime>

<workspaceWaitTime>0</workspaceWaitTime>

</job>

</object>

...

Tested product version

CloudBees Flow Version 9.1

Issue

I want to install a new plugin on my Jenkins master, but it does not appear in the Update Center of my instance.

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Resolution

CloudBees versions of Jenkins include access to the CloudBees Assurance Program (CAP). The CloudBees Assurance Program specifies the set of plugins, plugin versions, and plugin dependencies that are either verified or trusted, depending on how much they have been tested. This provides greater stability and security for Jenkins environments. Not only are these plugins independently stable, but they are tested as a whole (in aggregate) to ensure compatibility with each other and the Jenkins instance. Due to this, any plugins or plugin versions that have not undergone this additional testing are not listed in the Update Center.

If there is a plugin you would like to install that is not a part of CAP, please follow the steps below:

1. Navigate to Manage Jenkins > Beekeeper Upgrade Assistant, then uncheck Enroll this instance in the CloudBees Assurance Program and Save it. This action will switch the Update Center to access the full list of available Jenkins plugins.

2. Next navigate to Manage Jenkins > Manage Plugins > Advanced tab and click on Check Now to download the latest Plugin Manager metadata, after which you should find the specific plugin or plugins you would like to install on the Available tab.

3. Lastly, after installing the desired plugins, it is highly recommended that you re-enable enrollment to the CloudBees Assurance Program (CAP) to prevent the unintentional installation of plugins not verified by CAP.

Issue

A security scan of your Jenkins installation is warning that the HTTP TRACE/TRACK methods are enabled, and that this is a possible security issue. You want to disable these methods to silence the security warning.

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

Jenkins bundles the Jetty HTTP server and uses this by default as the web application server. Prior to version 2.205, the Jetty configuration file that was bundled with Jenkins did not disable the HTTP TRACE method, and this could cause security scanners to display a warning. Wedon'tbelieve that the TRACE method presents any significant security risk, but there is also no practical reason to have it enabled. If you are using a Jenkins version older than 2.205 and you wish to disable HTTP TRACE, you will need to do this manually.

This document is not relevant if you are running Jenkins in Tomcat.

Workaround

1. Make a backup of JENKINS_HOME/war/WEB-INF/web.xml.

2. Edit JENKINS_HOME/war/WEB-INF/web.xml and search for the existing security-constraint sections - there should be two like this already:

<security-constraint>

<web-resource-collection>

<web-resource-name>Hudson</web-resource-name>

<url-pattern>/loginEntry</url-pattern>

<!--http-method>GET</http-method-->

</web-resource-collection>

<auth-constraint>

<role-name>**</role-name>

</auth-constraint>

</security-constraint>

<security-constraint>

<web-resource-collection>

<web-resource-name>other</web-resource-name>

<url-pattern>/*</url-pattern>

</web-resource-collection>

<!-- no security constraint -->

</security-constraint>

3. Below this, add the following third entry:

<!-- ==================================================================== -->

<!-- Disable TRACE method with security constraint (copied from jetty) -->

<!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->

<security-constraint>

<web-resource-collection>

<web-resource-name>Disable TRACE</web-resource-name>

<url-pattern>/*</url-pattern>

<http-method>TRACE</http-method>

</web-resource-collection>

<auth-constraint/>

</security-constraint>

4. Restart Jenkins.

After this change, you should get a 403 if you try to use the TRACE method, rather than a 200. Please also note that if you upgrade Jenkins, this change will be lost and you will need to perform these steps again.

We wanted to let you know that there is a new way to share files with CloudBees Support. Many of you have been using Sharefile to share large files with Support.. Going forward, we ask that you use the CloudBees Uploads Service when you need to share such files relating to a support case.

This service makes it easy to share these larger files since it is natively integrated with Zendesk. When you login to https://uploads.cloudbees.com, you will be presented with a list of cases. Just choose the case in question and then upload the file. The support engineer will be automatically notified of your file, saving you that step. You can learn more about this service in this article about uploading large files.

Don't worry! If you have any problems with this service, you can fall back to using Sharefile for now if needed. We don't expect any problems, but sometimes corporate network rules must be adjusted for a service like this, so Sharefile will be available until those changes can be made. Also, in the rare case that you need to upload a file larger than 10 GB, you can follow this KB article on how to split these files.

Future of Sharefile

This change in the process for uploading large files is part of a larger effort we are making to phase out the use of Sharefile for CloudBees customers. We will also be changing the way we deliver your licenses. Instead of storing these in Sharefile, they will be attached to your delivery statements, or to the support case if you are requesting a temporary license. Such deliveries can be referred to at any time to retrieve the license in the future. We're also available to help you retrieve any licenses you may lose track of and aren't able to find in Zendesk - just ask!

Later in 2020, we also plan to make the installers for Flow and Accelerator available via our public website so that Sharefile access is not required to install the product. We also plan to make hotfixes available in a similar manner.

As always, thanks for being a CloudBees customer. If you have any concerns about these changes, please feel free to email [email protected] or open a support case and we'll be happy to work through any issues. We're here to help!

Issue

This article covers the support of the Jenkins Configuration as Code plugin (JCasC) in CloudBees products.

Can I use the Jenkins Configuration as Code plugin in CloudBees Core?

Can I use the Jenkins Configuration as Code plugin in CloudBees Jenkins Distribution?

What can I configure using the Jenkins Configuration as Code plugin in CloudBees Core?

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

CloudBees Jenkins Support

State of Support

As of August 2019, the Jenkins Configuration as Code Plugin is supported as a Technical Preview within certain CloudBees products.

What is a Technical Preview and what does that mean for support?

A plugin that is made available as a Technical Preview has the following characteristics:

The plugin has not undergone end-to-end testing with CloudBees products.

The plugin is not bundled into the products, but it can be installed from the CloudBees update center.

CloudBees will process support requests related to the Jenkins Configuration as Code plugin.

CloudBees will process support requests related to plugin compatibility with JCasC plugin.

Resolution of reported issues is subject to CloudBees internal priorities.

Current state of support in CloudBees Jenkins Distribution

The Jenkins Configuration as Code Plugin is provided as a Technical Preview within CloudBees Jenkins Distribution.Please refer to this page for more information about the current status and limitations of Jenkins Configuration as Code.The referenced page also provides quick start guidelines and public demos which can be used.

The provided materials apply to CloudBees Jenkins Support customers and to freemium users of CloudBees Jenkins Distribution.

Current state of support in CloudBees Jenkins Support

The Jenkins Configuration as Code Plugin is provided as a Technical Preview within CloudBees Jenkins Support.Starting from the September releases of CloudBees Products and from Jenkins LTS 2.176.3, CloudBees plans to certify the plugin as Tier 2 (Compatible) so that common issue and request for enhancement processes apply to users of the plugin.See the CloudBees plugin support policies for more information.

If you use CloudBees Jenkins Distribution, please refer to the section above.If you use Jenkins LTS, please see the Jenkins Configuration as Code community documentation to get more information about the plugin.

Also see this documentation page for more information.

Current state of support in CloudBees Core

As of August 2019, the Jenkins Configuration as Code Plugin is not fully supported in the CloudBees Core, CloudBees Jenkins Enterprise or CloudBees Jenkins Platform products.

Specific issues with the Jenkins Configuration as Code Plugin within CloudBees Core include:

Enabling the Jenkins Configuration as Code Plugin on Core instances may lead to system instability, because CloudBees products have not been fully tested against the Configuration as Code Plugin.

Although it is technically possible to use the Jenkins Configuration as Code Plugin with Team, Client or Managed Masters in the CloudBees Core environment, you will not be able to configure:

CloudBees proprietary plugins, including those delivered through CAP. The plugin manager will override configuration properties set by the Jenkins Configuration as Code plugin.

CloudBees-specific configurations in masters or CloudBees Operation Center.

Any community/open source plugins that do not support or are not compatible with the Jenkins Configuration as Code Plugin.

Any plugins with limited or poorly-implemented support for Jenkins Configuration as Code.

Some configuration changes made by the Jenkins Configuration as Code Plugin on a master may be overwritten by values from CloudBees Operations Center.

For more information about the current state, please refer to the following documentation pages:

Configuration as Code (CasC) support in CloudBees Core on Modern Platforms

Configuration as Code (CasC) support in CloudBees Core on Traditional Platforms

Current state of support in other Jenkins-based CloudBees products

Please refer to the following materials:

Configuration as Code is not supported in CloudBees Jenkins Enterprise, version 1.

CloudBees Jenkins X Distribution

CloudBees Jenkins X Support

Additional information

Configuration as Code in CloudBees Jenkins Distribution

CloudBees Blog: Jenkins Configuration as Code: Plugins

CloudBees Blog: Jenkins Configuration as Code: Look Ma, No Hands

CloudBees Blog: Jenkins Configuration as Code: Sensitive Data

CloudBees Blog: Use the Configuration-as-Code Plugin to Declaratively Configure Your Jenkins Instance

CloudBees Blog: Jenkins Configuration as Code Technical Preview for CloudBees Jenkins Support and CloudBees Jenkins Distribution

Demonstration examples of Jenkins Configuration as Code on GitHub

Jenkins Configuration as Code plugin page

Jenkins Configuration as Code plugin repository on GitHub

Issue

Attempting to run jstack or jmap on the Jenkins process results in the following exception:

Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for my_pid: Operation not permitted

sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for my_pid: Operation not permitted

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)

at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)

at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)

at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)

at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)

at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)

at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)

at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)

at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)

at sun.tools.jstack.JStack.main(JStack.java:106)

Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for my_pid: Operation not permitted

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)

at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Resolution

You can try using jcmd as an alternative:

For threads:

jcmd [PID] Thread.print > [FILE]

For heap:

jcmd [PID] GC.heap_dump [PATH]

If you are facing problems to access a CloudBees Core Operations Center running on modern platforms, we recommend to review

Troubleshooting CloudBees Core on Modern: Part 1 - Operations Center is not accessible

Issues

CloudBees managed master running on Kubernetes reports HTTP ERROR 404 in the browser.

Unable to access Jenkins master from the browser.

Jenkins master does not come up after upgrade on Kubernetes.

The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

The HTTP 404 Not Found Error means that the web page you were trying to reach could not be found on the server. It is a client-side error which means that the server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

Description

This routing diagnostic guide helps to troubleshoot network issues relating to one of the members of the CloudBees Core on Modern Platforms, which means that you are running CloudBees Core on a Kubernetes cluster. We assume the reader has already followed the CloudBees installation guide, and that every component of the cluster was configured as recommended in the following guides:

CloudBees Core on modern cloud platforms administration guide

CloudBees Core Reference Architecture - Kubernetes

For whatever reason, the questioned managed master, say mm1, does not work as expected. Most likely, the issue is related to a Kubernetes network misconfiguration that prevents the master from being provisioned, respond to requests, and being accessed in a browser. The following chart schematically shows the CloudBees Core running on a Kubernetes cluster, an administrative kubectl agent and a user browser that renders the problem.

https://support.cloudbees.com/hc/en-us/requests/new

General health-check tests

Answer the questions below to narrow down the root cause for the outage; it will save you time in fixing it. For the sake of illustration, we use mm1 managed master in the diagnostic steps.

1.] Can you access the Operations Center (CJOC) from a browser?

No. Use kubectl to further troubleshoot the Kubernetes cluster. Validate the status of Operations Center (cjoc-0) pod. Most likely, the outage is more general and goes beyond a single master.

2.] Can you access the questioned master from a browser; have you tried the incognito mode?

The master is not accessible from any browser.

3.] Is there any other CloudBees managed master that you can access from a browser?

Yes. Try to access the questioned master pod with kubectl. It is possible that the JENKINS_HOME folder is accessible. In this case, having an auto-generated support bundles from the JENKINS_HOME/support folder will help. Continue with this guide to get the explicit steps.

4.] Can you successfully provision a brand new managed master?

Yes. The solution is therefore to identify the broken piece of configuration for the questioned master.

HTTP diagnostics with curl

You are going to run curl commands to send HTTP requests to the questioned master

curl -I https://example.com/mm1/login

| | |

| | |__ managed master name, i.e. --prefix value

| |

| |__ external domain name, the one you use in a browser to access CloudBees Core

|

|__ http or https

When you send curl requests to a healthy CloudBees Core master it responds with the HTTP/1.1 200 OK status.

HTTP/1.1 200 OK

1.] Check the questioned master with curl

A healthy CloudBees Core master should be accessible on the network. Open a Linux terminal on any appropriate desktop and run

curl -I http://example.com/mm1/login

The full output should be similar to

HTTP/1.1 200 OK

Server: openresty/1.15.8.1

Date: Mon, 23 Sep 2019 06:44:39 GMT

Content-Type: text/html;charset=utf-8

Content-Length: 1966

Connection: keep-alive

Vary: Accept-Encoding

X-Content-Type-Options: nosniff

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Cache-Control: no-cache,no-store,must-revalidate

X-Hudson: 1.395

X-Jenkins: 2.176.3.2

X-Jenkins-Session: 6a97d870

X-Hudson-CLI-Port: 50001

X-Jenkins-CLI-Port: 50001

X-Jenkins-CLI2-Port: 50001

X-Frame-Options: sameorigin

X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AM...

Set-Cookie: JSESSIONID.6284910f=node0ljc203mrj7m9199phkqz58czk1.node0;Path=/mm1;HttpOnly

HTTP/1.1 404

If the master responds with HTTP ERROR 404 in the browser, most likely, you will see a similar response to curl requests

curl -I http://example.com/mm1/login

output from a non-responding master

HTTP/1.1 404 Not Found

Server: openresty/1.15.8.1

Date: Mon, 23 Sep 2019 04:56:50 GMT

Content-Type: text/html

Content-Length: 159

Connection: keep-alive

HTTP/1.1 503 Service Unavailable

The 503 Service Unavailable error is an HTTP status code that means the web server is not available right now. Most likely, the Jenkins instance is restarting, too busy, or is not ready to handle the request.

HTTP/1.1 503 Service Unavailable

Server: openresty/1.15.8.1

Date: Mon, 23 Sep 2019 05:03:46 GMT

Content-Type: text/html;charset=utf-8

Content-Length: 1056

Connection: keep-alive

X-Content-Type-Options: nosniff

Expires: 0

Cache-Control: no-cache,no-store,must-revalidate

Cluster level diagnostics with kubectl

1.] Check the status of the questioned master pod. Is the pod running?

kubectl get pod mm1-0 -o wide

expected output

NAME READY STATUS RESTARTS AGE IP NODE

mm1-0 1/1 Running 0 1m 10.52.25.3 gke-cluster-example-core-masters-72f5634a-txbz

2.] Pod events key (last section in the output)

Describe the master pod and review the Events key (last section in the output)

kubectl describe pod mm1-0

3.] Check the location of JENKINS_HOME folder

The default (CloudBees Core 2.190.x.x) location of the JENKINS_HOME folder is /var/jenkins_home. Run

kubectl describe pod mm1-0 | grep jenkins_home

expected output

/var/jenkins_home from jenkins-home (rw)

4.] Check whether the pod responds to shell commands, and validate that the volumes are mounted

kubectl exec -ti mm1-0 -- df -h

expected output reads as follows

Filesystem Size Used Avail Use% Mounted on

overlay 95G 4.6G 90G 5% /

tmpfs 64M 0 64M 0% /dev

tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup

/dev/sda1 95G 4.6G 90G 5% /etc/hosts

/dev/sdc 9.8G 639M 8.7G 7% /var/jenkins_home

shm 64M 0 64M 0% /dev/shm

tmpfs 7.4G 12K 7.4G 1% /run/secrets/kubernetes.io/serviceaccount

tmpfs 7.4G 0 7.4G 0% /proc/acpi

tmpfs 7.4G 0 7.4G 0% /proc/scsi

tmpfs 7.4G 0 7.4G 0% /sys/firmware

5.] Verify that the JENKINS_HOME partition has available disk space

Yes. Jenkins instance has decent amount of available disk space. Access JENKINS_HOME folder inside the container. List the content of JENKINS_HOME folder.

kubectl exec mm1-0 -- ls /var/jenkins_home

6.] Check read/write access to the JENKINS_HOME folder

Validate that the persistent volume resource accepts read/write operations

kubectl exec -ti mm1-0 -- bash -c 'echo "OK" > /var/jenkins_home/~writeTest.log \

&& cat /var/jenkins_home/~writeTest.log \

&& rm /var/jenkins_home/~writeTest.log'

The expected output is OK.

7.] Can you get the Jenkins log?

kubectl logs -f --tail 100 mm1-0

Analyze the output. Does it provide additional clues? Use Ctrl + c to quit the log mode.

8.] What is the last trace/error in the log?

The log files could vary significantly between instances as different levels of verbosity are allowed. In addition, you may see traces related to master configurations, installed plugins, and bootstrap scripts. The following traces are the most critical to assess healthiness of the provisioned managed master.

[Mon Sep 23 06:40:36 GMT 2019] Requested provisioning successfully.

[Mon Sep 23 06:40:38 GMT 2019] Requested start successfully

[Mon Sep 23 07:41:38 GMT 2019][Normal][Pod][mm1-0][Pulled] Successfully pulled image "cloudbees/cloudbees-core-mm:2.176.3.2"

[Mon Sep 23 07:41:38 GMT 2019][Normal][Pod][mm1-0][Started] Started container jenkins

[Mon Sep 23 06:43:02 GMT 2019] Accepting initial connection from http://example.com/mm1/ on 10.52.25.3/10.52.25.3:39202 with identity f1:8d:d6:f6:5e:ed:fe:25:17:38:12:c0:cb:ce:a1:d6 (STORED)

[Mon Sep 23 06:43:14 GMT 2019] Connected

[Mon Sep 23 09:01:55 GMT 2019] Checking license validity...

[Mon Sep 23 09:01:55 GMT 2019] License will expire in 2 days 22 hr (not before next check)

[Mon Sep 23 09:01:55 GMT 2019] Current license is valid

9.] Does Jenkins log update in real time?

Yes. Analyze the output.

10.] Access (get a shell to a running container) the questioned master pod.

kubectl exec -ti mm1-0 -- bash

To check whether Jenkins instance is started, i.e. java process is running, run

ps -eaf | grep java

expected output

jenkins 7 1 23 06:41 ? 00:03:05 java -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Xbootclasspath/p:/usr/share/jenkins/alpn-boot.jar -Duser.home=/var/jenkins_home -Xmx2150m -Xms2150m -Djenkins.model.Jenkins.slaveAgentPort=50001 -Djenkins.install.runSetupWizard=true -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle -Duser.timezone=PST -DMASTER_NAME=mm1 -Dcb.BeekeeperProp.autoInstallIncremental=true -Djenkins.model.Jenkins.slaveAgentPortEnforce=true -DMASTER_GRANT_ID=0db2670e-c797-48b3-9042-e997207bf6be -Dcb.IMProp.warProfiles.cje=kubernetes.json -DMASTER_INDEX=0 -DMASTER_OPERATIONSCENTER_ENDPOINT=http://cjoc.example.svc.cluster.local/cjoc/ -Dcb.BeekeeperProp.noFullUpgrade=true -Dhudson.DNSMultiCast.disabled=true -DMASTER_ENDPOINT=http://example.com/mm1/ -XX:NativeMemoryTracking=summary -jar -Dcb.distributable.name=Docker Common CJE -Dcb.distributable.commit_sha=49e35b48176fc789078f52e12f8fb09382da938a /usr/share/jenkins/jenkins.war --webroot=/tmp/jenkins/war --pluginroot=/tmp/jenkins/plugins --prefix=/mm1/

jenkins 1334 1311 0 06:54 pts/0 00:00:00 grep java

Network diagnostics

1.] Start by identifying the internal IP address and ports of the questioned pod

kubectl describe pod mm1-0 | grep -E "IP|Ports"

expected output

IP: 10.52.25.3

Ports: 8080/TCP, 50001/TCP

Host Ports: 0/TCP, 0/TCP

2.] Check network statistics for the questioned pod

kubectl exec -ti mm1-0 -- netstat -tupe

expected output shows established connections

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name

tcp 0 0 mm1-0.mm1.cje-sup:39202 cjoc.cje-support-:50000 ESTABLISHED jenkins 97862 7/java

tcp 0 0 mm1-0.mm1.cje-sup:45014 kubernetes.default.:443 ESTABLISHED jenkins 97824 7/java

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35292 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35248 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-sup:45018 kubernetes.default.:443 ESTABLISHED jenkins 99765 7/java

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49890 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49876 ESTABLISHED jenkins 117321 7/java

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35272 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35264 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35256 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49892 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49880 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49874 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.25.1:35280 TIME_WAIT root 0 -

tcp 0 0 mm1-0.mm1.cje-sup:44990 kubernetes.default.:443 ESTABLISHED jenkins 99734 7/java

tcp 0 0 mm1-0.mm1.cje-supp:8080 10.52.21.8:49888 TIME_WAIT root 0 -

Pod open ports

1.] Validate the configured ports are open on the questioned master.

Run netcat tool on cjoc-0 pod to probe open ports of the questioned master pod

kubectl exec -ti cjoc-0 -- nc -zv 10.52.25.3 8080

kubectl exec -ti cjoc-0 -- nc -zv 10.52.25.3 50001

expected outputs

10.52.25.3 (10.52.25.3:8080) open

10.52.25.3 (10.52.25.3:50001) open

Service diagnostics and routing

1.] Check the Service that defines access to the questioned master pod. You should see Service of type ClusterIP

kubectl get services mm1 -o wide

the output

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

mm1 ClusterIP 10.55.249.188 <none> 80/TCP,50001/TCP 41m com.example.cje.tenant=mm1

2.] Describe the service and validate that the TargetPorts are set correctly

kubectl describe service mm1 | grep -E "IP|Port"

the output

Type: ClusterIP

IP: 10.55.249.188

Port: http 80/TCP

TargetPort: 8080/TCP

Port: agent 50001/TCP

TargetPort: 50001/TCP

3.] Validate the service object for the questioned master

Validate that the TargetPort are open

kubectl exec -ti cjoc-0 -- nc -zv mm1 80

kubectl exec -ti cjoc-0 -- nc -zv mm1 50001

expected outputs

mm1 (10.55.249.188:80) open

mm1 (10.55.249.188:50001) open

4.] Describe the ingress object for the questioned master

Describe the ingress object and review Address and Host values

kubectl describe ing mm1

expected output

Name: mm1

Namespace: cje-example

Address: 35.190.219.203

Default backend: default-http-backend:80 (<none>)

Rules:

Host Path Backends

---- ---- --------

example.com

/mm1/ mm1:80 (10.52.31.29:8080)

Annotations:

ingress.kubernetes.io/proxy-body-size: 50m

ingress.kubernetes.io/proxy-request-buffering: off

ingress.kubernetes.io/ssl-redirect: true

kubernetes.io/ingress.class: nginx

nginx.ingress.kubernetes.io/proxy-body-size: 50m

nginx.ingress.kubernetes.io/proxy-request-buffering: off

nginx.ingress.kubernetes.io/ssl-redirect: true

Events: <none>

and validate that the master pod can be reached from the CJOC pod

kubectl exec -ti cjoc-0 -- curl -I 10.52.31.29:8080/mm1/login | head -n1

kubectl exec -ti cjoc-0 -- curl -I example.com/mm1/login | head -n1

The output should be HTTP/1.1 200 OK.

Internet access

If a master is expected to have the Internet access, then validate it can reach any external URL

kubectl exec -ti mm1-0 -- nc -zv www.google.com 443

expected output is similar to

www.google.com (108.177.15.106:443) open

kube-proxy logs

Follow the troubleshooting page

Is the kube-proxy working

Advanced Network Troubleshooting

1.] Utility pod

To further troubleshoot the routing issues, we need tools such as ping, curl, netcat, telnet, dnsutils. You are going to create a diagnostic pod with the listed packages installed. On a Debian based test pod, run

apt-get update && apt-get install iputils-ping iproute2 curl telnet netcat net-tools -y

Create a new utility-pod pod

kubectl create deployment utility-pod --image=nginx

kubectl get pods | grep utility-pod

output

utility-pod-7b45c4f7dd-dj8gv 1/1 Running 0 85s

Open an interactive session on a utility-pod and install the tools.

kubectl exec -ti utility-pod-7b45c4f7dd-dj8gv -- bash

apt-get update && apt-get install iputils-ping iproute2 curl telnet netcat net-tools -y

Good troubleshooting guides include

Debug Kubernetes Services

Troubleshoot Clusters

Troubleshooting OpenShift

Note: Once the issue is resolved, remove the utility-pod deployment

kubectl delete deploy utility-pod

2.] Involve k8s support team

Contact your corporate k8s support team for further troubleshooting.

Submit a CloudBees Support request and a CloudBees engineer will schedule a call with you.

Issue

After creating a copy of the local Jenkins home /lastSuccessful fails with java.nio.file.DirectoryNotEmptyException

You are getting a stacktrace similar to the one below:

/opt/jenkins/jobs/PACE/jobs/Scheduled_Production_Copy_LPAR6/lastSuccessful failed

java.nio.file.DirectoryNotEmptyException: /opt/jenkins/jobs/PACE/jobs/Scheduled_Production_Copy_LPAR6/lastSuccessful

at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)

at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)

at java.nio.file.Files.deleteIfExists(Files.java:1118)

at sun.reflect.GeneratedMethodAccessor4202.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at hudson.Util.createSymlinkJava7(Util.java:1194)

at hudson.Util.createSymlink(Util.java:1112)

at hudson.model.Run.createSymlink(Run.java:1851)

at hudson.model.Run.updateSymlinks(Run.java:1832)

at hudson.model.Run.execute(Run.java:1743)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)

at hudson.model.ResourceController.execute(ResourceController.java:89)

at hudson.model.Executor.run(Executor.java:240)

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

The issue is related to the method used to copy the $JENKINS_HOME from one location to another one, which in this particular case broke the symlinks.

The recommended option, it is to rollback to a security copy of your previous $JENKINS_HOME if possible and copy it in the new location using a method which preserve the symlinks.

If this is not possible, to fix the master where you have these erroneous directories you should follow the steps shown below:

Important Notice: You should have a current backup of your $JENKINS_HOME or you should create one before trying the method described below.

# Logon on the master where you have the issue

ssh machine

# Shutdown the jenkins master (take care that nothing is running)

sudo service jenkins stop

# Find all erroneous directories

find /opt/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \)

# Review the list of erroneous directories

# Rename them

find /opt/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \) -exec mv {} {}.err \;

# Restart jenkins

sudo service jenkins start

Once that you have verified that everything is working correctly, you should delete the faulty directories following the steps shown below:

# Delete them

find /opt/jenkins/jobs -type d -name "*.err" -exec rm -rf {} \;

Tested product/plugin versions

Jenkins LTS

If you feel like your support case needs extra attention, just add the string #escalate to your comment.Please include a description of why you think more attention is needed and what impact the issue is having on your business.

Example:

#escalate

I would like to bring extra attention to this request open since xx days.

We are not able to provide the information requested due to xyz.

This is causing our production environment to crash on a regular basis.

What To Expect

After you escalate a case, a support manager will follow up with you on an action plan to resolve your concern.

References

CloudBees support and maintenance terms & conditions CloudBees supported platforms CloudBees plugin support policies

Issue

You observe that your logs include lots of traces like the one shown below:

org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFile

WARNING: Avoid calling getLogFile on JobXXX #BuildYYY

java.lang.UnsupportedOperationException

at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1082)

at hudson.plugins.timestamper.io.LogFileReader.nextLine(LogFileReader.java:157)

at hudson.plugins.timestamper.action.TimestampsActionOutput$1.readNextLine(TimestampsActionOutput.java:173)

at hudson.plugins.timestamper.action.TimestampsActionOutput$1.read(TimestampsActionOutput.java:132)

at java.io.BufferedReader.fill(BufferedReader.java:161)

This can fill the logs and cause different problems in your instance.

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on traditional platforms - Client Master

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

This is a known issue, JENKINS-54128 related to the timestamper plugin and its interaction with workflow-job plugin and groovy-postbuild plugin.

In order to correct the problem:1. Take a backup as per the Backup and restore guide 2. Upgrade the above mentioned plugins (or at least the one that you are using), to the versions listed below:

Pipeline: Job 2.29 or higher

Groovy Postbuild 2.5

Workaround

If you can get rid of the warnings in the logs, one easy way to do it is to set the log level for org.jenkinsci.plugins.workflow.job.WorkflowRun to SEVERE. That will remove the warning messages from your logs, but our recommendations is that you move to a more long term solution by planning the plugin updates suggested in the previous section.

Tested product/plugin versions

Jenkins LTS 2.138.41.0.1

Pipeline: Job 2.29

Groovy Postbuild 2.3.1

References

Configure Loggers for Jenkins

Issue

When creating a new node, the option to Launch agent by connecting it to the master is not available in the Launch method

When creating a new node, the option to Launch agent via Java Web Start is not available in the Launch method field:

TcpSlaveAgentListener

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Distribution

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

Jenkins LTS

Resolution

Note that before 2.176.1 the Launch agent by connecting it to the master used to be known as Launch agent via Java Web Start and TCP port for inbound agents as TCP port for JNLP agents.

The option Launch agent by connecting it to the master is only available if the inbound agents port is enabled (i.e. the is running).

The inbound agents port can be enabled under Manage Jenkins > Configure Global Security > Agents > TCP port for inbound agents. Once enabled, the option Launch agent by connecting it to the master will be available in the configuration of the Jenkins Nodes.

Note: It is recommended to use a Fixed agent port for better manageability.

Issue

Any kind of communication issue between Jenkins master and a dedicated JNLP agent.

Required Data JNLP dedicated agents

This article describes how to collect the minimum required information for communication issues between a Jenkins master and an agent so that it can be efficiently troubleshooted.

If the required data is bigger than 20 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

CloudBees Jenkins Platform (CJP)

Operations Center (OC)

Client Master (CM)

CloudBees Jenkins Enterprise (CJE)

Required Data check list

[ ] Support bundle[ ] Agent name[ ] Agent config.xml[ ] Agent logs[ ] Master logs[ ] Process lists in the JNLP agent machine[ ] Connectivities checks

Support bundle

A support bundle from the Jenkins instance after the issue is exposed. Please, follow the KB below in case youdon'tknow how to generate a support bundle.

What is a support-bundle and how to create one

Agent name

The name of the agent under Manage Jenkins -> Manage Nodes

Agent config.xml

It is possible to access to the agent config.xml from the UI. You just need to add at the end of the item URL config.xml. i.e JENKINS_URL>/computer/<AGENT_NAME>/config.xml

<slave>

<name>new-node</name>

<description/>

<remoteFS>/Users/fbelzunc/jnlp-agent</remoteFS>

<numExecutors>1</numExecutors>

<mode>NORMAL</mode>

<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>

<launcher class="hudson.slaves.JNLPLauncher"/>

<label/>

<nodeProperties/>

</slave>

ELB/HaProxy/Firewall ?

Please, specify if there is any HAProxy/ELB/../Firewall between the Jenkins master and the JNLP agent.

Agent logs

In case the JNLP agent is running as a service, stop the service and run the agent from the command line. If the issue happens randomly, then we suggest to redirect the logs to a text file so you can grab the error messages when the issue is exposed.

java -jar slave.jar -jnlpUrl http://support-cluster.cloudbees.com/master-1/computer/new-node/slave-agent.jnlp -secret 575ff66ea96bf140d3dae4fa8d7e96ba1221096678dxxxxxxd1bf1a116

Check that the JNLP port is accesible to the agent

Attach the full logs to the support tickets

Remoting work directory

Remoting work directory is available starting from Remoting 3.8,which is available in Jenkins 2.68). Before this version there was no working directory concept in the library itself; all operations were managed by library users (e.g. Jenkins agent workspaces).

Master logs

Attach the master logs around the time the issue is exposed that you can directly find in the UI under Manage Jenkins-> System Log -> All Jenkins Logs.

Notice that the idea is to correlate the master logs with the agent logs at the time the issue is exposed.

Process lists in the JNLP agent machine

In a Unix environment with ps -aux and in Windows with Get-Process from Powershell console.

Connectivities checks

Use jenkins-cli to check the connection

In the agent box, download the CLI and run a help command in your favorite mode. For example, using http

java -jar jenkins-cli.jar [-s JENKINS_URL] -auth <user>:<token> help > cli.txt

Check that the agent is able to see the JENKINS headers

# curl -IvL <JENKINS_URL>

curl -IvL https://jenkins:8443 > curl.txt

Curl command can be available on a Windows box using for example curl Download Wizard

# telnet <JENKINS_HOST> <JNLP_PORT>

telnet jenkins.host.example.com 50234 > telnet.txt

Issue

Is there a way to uninstall a plugin via a Cluster Operation?

I see:* Disable plugin* Enable plugin* Install plugin

But no Uninstall plugin step.

Environment

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Operations Center

Cluster Operations

Resolution

First, you will need to find the plugin id on the master. You can use the following script in the Script Console to do that:

Jenkins.instance.pluginManager.plugins.each{

plugin ->

println (" Display Name: ${plugin.getDisplayName()}, Plugin Id: ${plugin.getShortName()}")

}

To uninstall the plugin, you can use the following script in a Cluster Operation:

String pluginId = "mock-security-realm"

def plugin = Jenkins.getInstance().getPluginManager().getPlugin(pluginId)

plugin.doDoUninstall()

In the above example, the plugin mock-security-realm is to be uninstalled.

This will result in an Uninstallation pending status for the plugin. The plugin will be removed completely after the master is next restarted.

Issue

I want to use some Jenkins features that are not available at REST API, Could I execute they as I do in the console script but through REST API?

Environment

CloudBees Jenkins Enterprise > 1.596.1.1

Jenkins LTS > 1.596.1

Jenkins > 1.600

Resolution

Jenkins supports scripting with the Groovy language. You can get a scripting console by opening in a browser the URL /script of your Jenkins instance. (i.e: http://localhost:8080/script )

The advantage of the Groovy language is that those Groovy scripts are executed within Jenkins and have access to everything (config, plugins, jobs, etc).

To automate the execution of this script, you can save it to a file (lets say /tmp/script.groovy) and run the following curl command:

curl -d "script=$(cat /tmp/script.groovy)" -v --user username:ApiToken http://localhost:8080/scriptText

curl -d "script=println 'this script works'" -v --user username:ApiToken http://localhost:8080/scriptText

Issue

Move/Copy/Promote fails for Multibranch Pipeline jobs when copying between different masters.

Moving a newly created Multibranch Pipeline job fails.

Move/Copy/Promote process for Multibranch Pipeline jobs fail throwing:

ERROR: Caught an exception

hudson.os.PosixException: Unable to get file permissions null

at hudson.util.IOUtils.mode(IOUtils.java:133)

at com.cloudbees.opscenter.replication.CustomTarArchiver.visit(CustomTarArchiver.java:75)

at hudson.util.DirScanner.scanSingle(DirScanner.java:49)

at hudson.FilePath$ExplicitlySpecifiedDirScanner.scan(FilePath.java:3122)

at com.cloudbees.opscenter.dumbwaiter.Compression.tarGz(Compression.java:73)

at com.cloudbees.opscenter.replication.ComputedFolderReplicationFileSet.createArchive(ComputedFolderReplicationFileSet.java:104)

at com.cloudbees.opscenter.replication.ItemReplicationFileSet.createItemReplicationArchive(ItemReplicationFileSet.java:210)

at com.cloudbees.opscenter.replication.FolderReplicationFileSet.processChild(FolderReplicationFileSet.java:86)

at com.cloudbees.opscenter.replication.FolderReplicationFileSet.createArchive(FolderReplicationFileSet.java:72)

at com.cloudbees.opscenter.replication.ItemReplicationFileSet.createItemReplicationArchive(ItemReplicationFileSet.java:184)

at com.cloudbees.opscenter.replication.stages.SendFiles.apply(SendFiles.java:114)

at com.cloudbees.opscenter.replication.ItemReplicationTask.runStages(ItemReplicationTask.java:529)

at com.cloudbees.opscenter.replication.ItemReplicationTask._perform(ItemReplicationTask.java:475)

at com.cloudbees.opscenter.replication.ItemReplicationRecord.perform(ItemReplicationRecord.java:195)

at com.cloudbees.opscenter.replication.ItemReplicationRecord.run(ItemReplicationRecord.java:167)

at hudson.model.ResourceController.execute(ResourceController.java:97)

at hudson.model.Executor.run(Executor.java:429)

Checking Move/Copy/Promote history (Manage Jenkins > Move/Copy/Promote History) I can see the following stacktrace reported:

ERROR: Caught an exception

hudson.os.PosixException: native error calling stat: No such file or directory $JENKINS_HOME/jobs/multibranch-pipeline/indexing/indexing.log ENOENT

at hudson.os.PosixAPI$1.error(PosixAPI.java:36)

at jnr.posix.BaseNativePOSIX.stat(BaseNativePOSIX.java:309)

at jnr.posix.CheckedPOSIX.stat(CheckedPOSIX.java:265)

at jnr.posix.LazyPOSIX.stat(LazyPOSIX.java:267)

at hudson.util.IOUtils.mode(IOUtils.java:128)

at com.cloudbees.opscenter.replication.CustomTarArchiver.visit(CustomTarArchiver.java:76)

at hudson.util.DirScanner.scanSingle(DirScanner.java:49)

at hudson.FilePath$ExplicitlySpecifiedDirScanner.scan(FilePath.java:2845)

at com.cloudbees.opscenter.dumbwaiter.Compression.tarGz(Compression.java:74)

at com.cloudbees.opscenter.replication.ComputedFolderReplicationFileSet.createArchive(ComputedFolderReplicationFileSet.java:107)

at com.cloudbees.opscenter.replication.ItemReplicationFileSet.createItemReplicationArchive(ItemReplicationFileSet.java:140)

at com.cloudbees.opscenter.replication.stages.SendFiles.apply(SendFiles.java:107)

at com.cloudbees.opscenter.replication.ItemReplicationTask.runStages(ItemReplicationTask.java:515)

at com.cloudbees.opscenter.replication.ItemReplicationTask._perform(ItemReplicationTask.java:473)

at com.cloudbees.opscenter.replication.ItemReplicationRecord.perform(ItemReplicationRecord.java:191)

at com.cloudbees.opscenter.replication.ItemReplicationRecord.run(ItemReplicationRecord.java:163)

at hudson.model.ResourceController.execute(ResourceController.java:98)

at hudson.model.Executor.run(Executor.java:410)

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

Explanation

This is a known issue (filed as CTR-167 in our internal Jira) where the copy operation is trying to copy a file that does not exist. Specifically, it is indexing/indexing.log which corresponds to a Scan Multibranch Pipeline Log on the UI. The mentioned log is not there because there are no Branch Sources in your Multibranch Pipeline, therefore no scans are done.

Resolution/Workaround

You have two options:

If the Multibranch Pipeline job is new, emtpy and unused, you can delete it to get rid of this error.

If you prefer to not delete it, you can just configure the Multibranch Pipeline job to add a Branch Source. After doing this, the Move/Copy/Promote process should work again:

How to troubleshoot Move Copy Promote issues?

Tested product/plugin versions

CloudBees Jenkins Platform - Client Master 2.164.3.2

Pipeline: Multibranch 2.20

Operations Center Context 2.150.2.8

References

Issue

We want to restrict the user access to specific jobs under sub-folder level.

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

One of the most important things to keep in mind in this type of scenarios is to create roles as granular as possible. As an example for restricting user access to specific jobs at folders and sub-folders level We have created a main folder Folder1 that contains two different subfolders Folder1/FolderA and Folder1/FolderB along with two Freestyle projects called job1 and job2 at Folder1/FolderA/job1 and Foldera/FolderB/job2.

Main folder Name: Folder1

Sub-folder Names: Folder1/FolderA and Folder1/FolderB

Ideally after completing the configuration steps we would like to make sure that one of our users has access Folder1/FolderA/joba the other one to Folder1/FolderB/job2

The following table summarizes the role settings used in this example

| User | Roles | Patterns |

| --- |--- |--- |

| admin | admin | |

| Folder1View_user | Folder1View | Folder1 |

| Folder1_FolderA_user | Folder1FolderA | Folder1/FolderA.* |

| Folder1_FolderB_user | Folder1FolderB | Folder1/FolderB.* |

1. Manage Roles

First of all, we must create new roles, to do so, navigate to Jenkins->Manage Jenkins->Manage and Assign Roles->Manage Roles from the root Jenkins dashboard. Create an admin role with the Overall/Administer permission. Click Save.

Furthermore, create three new Project Roles, namely: Folder1View, Folder1FolderA, Folder1FolderB. The first role, Folder1View grants read access only to Folder1, while Folder1FolderA and Folder1FolderB gives both Read and Build permissions to Folder1/FolderA and Folder1/FolderB respectively. For project and agent roles you can set a regular expression pattern for matching items. The regular expression aimed at matching the full item name. In this particular case, since we want our users to have access to all jobs within a specific sub-folder Folder1/FolderA, we use the following regular expression as pattern Folder1/FolderA.*. Click Save.

Role Strategy Plugin

2. Assign Roles

To assign roles please navigate to Jenkins->Manage Jenkins->Manage and Assign Roles->Assign Roles

During the assign role process, keep in mind that:

User groups represent authorities provided by the Security Realm (e.g. LDAP plugin can provide groups)

There are also two built-in groups: authenticated (users who logged in) and anonymous (any users, including ones who have not logged in)

First of all, we must make sure all users have OverallRead selected as global role. This will allow every user to see the contents of the top folder when they login in Jenkins.

Finally, define the item roles as shown in the picture. As it was mentioned before the key is to create roles as granularly as possible, this is actually reflected in the picture below, under item roles. As you can see, Folder1FolderA_user needs not only Folder1FolderA item role but also Folder1View, without the later that user would not be able to navigate into FolderA.

Your assign roles screen should now look like:

References

Issue

Im trying to copy jobs from one folder to another. Ican'tfind a way to do that.

Environment

CloudBees Jenkins Enterprise

Folders

Folders Plus

Resolution

To move folders you need either Folders or Folders Plus installed.

Lets say I have two folders:

my-folder and

my-folder-2 and have a Jenkins job under

my-folder/my-job. If would like to move this job from one location to another one, just need to click in the dropdown menu that you can see below and click on move option.

Issue

String arg1 = "arg1"

String arg2 = "arg2"

String cm = "curl -H \\'Authorization: ${arg1}\\' -H \\'X-CUSTOM-HEADER:${arg2}\\' www.google.com"

node {

sh cm

}

The expected output is:

[Pipeline] {

[Pipeline] sh

+ curl -H 'Authorization: arg1' -H 'X-CUSTOM-HEADER:arg2' www.google.com

Actual output is:

Output:

[Pipeline] sh

+ curl -H ''\''Authorization:' 'arg1'\''' -H ''\''X-CUSTOM-HEADER:arg2'\''' www.google.com

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on traditional platforms - Client Master

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Distribution

Jenkins LTS

Pipeline plugin

Resolution

No action is required.

The weird escaping is just for the build console output of the command that was run, which escapes things one more level than what was actually received (to prevent things like ANSI escape codes from applying to that output).

Issue

I want to configure Client Master with HTTPS and connect it to a CJOC

Environment

CloudBees Core on modern cloud platforms - Managed Master (CORE-MM)

CloudBees Core on modern cloud platforms - Operations Center (CORE-CLOUD-OC)

CloudBees Core on traditional platforms - Client Master (CORE-CM)

CloudBees Core on traditional platforms - Operations Center (CORE-OC)

CloudBees Jenkins Enterprise - Managed Master (CJE-MM)

CloudBees Jenkins Enterprise - Operations Center (CJE-OC)

CloudBees Jenkins Platform - Client Master (CJP-CM)

CloudBees Jenkins Platform - Operations Center (CJP-OC)

Resolution

CJOC is not a HTTPS/SSL/TLS endpoint

CJOC does not initiate the connection to the master, so it doesnt care if the master is on HTTPS or HTTP.

CJOC is a HTTPS/SSL/TLS endpoint

The Client Master initiates the connection to CJOC. In order for the Client Master to discover the agent port that CJOC is listening on you need to give the master the TLS cert of CJOC.

You can manually add the certificates to the masters trust store following the documentation.

CloudBees Core on modern cloud platforms or CloudBees Jenkins Enterprise 2.x

Deploy Self Signed Certificates in Masters and Agents Deploy Self Signed Certificates in Masters and Agents (Custom Location)

CloudBees Core on traditional platforms or CloudBees Jenkins Platform:* Client Master - Running on a TLS Endpoint * How to install a new SSL certificate

CloudBees Jenkins Enterprise 1.x

How to import certificates to a CJE cluster

Issue

I would like to learn how to create a CloudBees Flow plugin.

Environment

CloudBees Flow

Resolution

There are a few options available when starting to create a plugin, we recommend you start with:

Flow Plugin Development Framework (flowpdf)

The Flow Plugin Development Framework (flowpdf in short) provides a combination of Tools, APIs and libraries which can be used to create a flow plugin in 4 steps. Refer to the Development Process section below.

That page links to the: Flow Plugin Developer Guide

That guide has a Tutorial section that walks you through the process: Plugin Tutorial using the Framework

An alternative option for creating a plugin is the EC-Admin plugin which has a procedure namedprojectAsCodethat will create a plugin from a project.

Depending on your use case, you may be able to use the API instead of writing a plugin, which is documented here: CloudBees Flow APIs (version 9.1)

The documentation for other versions of the API can be found at: http://docs.electric-cloud.com/eflow_doc/FlowIndex.html

Issue

Agent provisioning fail and master logs show an issue similar to the following:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://127.0.0.1/api/v1/namespaces/cloudbees-core/pods. Message: Pod "default-jnlp-dfptn" is invalid: metadata.labels: Invalid value: "MY%20MASTER": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'). Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels, message=Invalid value: "MY%MASTER": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'), reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, name=default-jnlp-dfptn, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "default-jnlp-dfptn" is invalid: metadata.labels: Invalid value: "MY%MASTER": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?'), metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:503)

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:442)

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:406)

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365)

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:234)

at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:796)

at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:326)

at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:322)

at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:124)

at com.cloudbees.jenkins.plugins.kube.ListenableKubernetesLauncher.launch(ListenableKubernetesLauncher.java:30)

at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)

at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)

at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Environment

CloudBees Core on Modern Cloud Platforms

Kube Agent Management Plugin >= 1.1.14

Related Issue

CPLT2-5649: Tag each pod created by a master with a backlink to the master

CPLT2-5956: Master label doesnt work when master has a space in its name

Explanation

This is caused by a feature introduced in Kube Agent Management plugin version 1.1.14 that automatically adds a label with the masters name to agent pods. If the master contains a space in its name, the label cannot be applied.

Resolution

The issue has been resolved in the Kube Agent Management plugin version 1.1.17 and is available in the October release of CloudBees Core 2.190.2.2.

Solution

Upgrade CloudBees Core to version 2.190.2.2 or later.

Workaround

The workaround for a live instance is to execute the following groovy script under Manage Jenkins > Script Console:

import hudson.*

def podTemplateExtensionList = org.csanchez.jenkins.plugins.kubernetes.PodTemplateFilter.all()

println "Current ExtensionList:" + podTemplateExtensionList

def masterLinkExtension = podTemplateExtensionList.get(com.cloudbees.jenkins.plugins.kube.MasterLinkFilter.class)

println "Removing " + masterLinkExtension

podTemplateExtensionList.remove(masterLinkExtension)

println "Resulting ExtensionList: " + org.csanchez.jenkins.plugins.kubernetes.PodTemplateFilter.all()

This will however not survive a restart.

To make this workaround around permanent, create a groovy script named CPLT2-5956.groovy under /var/jenkins_home/init.groovy.d/ in the $JENKINS_HOME of the masters impacted.

Issue

How to make GitHub (and/or GitHub Enterprise) trigger jobs from GitHub Organization Folders in Jenkins.

Environment

Jenkins

CloudBees Jenkins Enterprise - Managed Master (CJEMM)

CloudBees Jenkins Team (CJT)

CloudBees Jenkins Platform - Client Master (CJPCM)

Jenkins LTS

Plugins

GitHub plugin

GitHub Branch Source

GitHub

GitHub

GitHub Enterprise

Resolution

On Jenkins

A. Plugin Configuration

GitHub plugin as explained in GitHub Webhook configuration.

For GitHub Branch Source in case you wish to integrate with a GitHub Enterprise, it needs to be added in Manage Jenkins > Configure System > GitHub Enterprise Servers in order to fetch organizations from that endpoint.

GitHub API v3

B. Job Configuration

Steps:

Branch Sources > Add source > GitHub Select the API endpoint

For Owner use the organization

For Credential use Username and Password, typing as password the personal API Token.

Since SCM API 2.2.4 Behaviors section > Make sure Filter by name is removed unless you only want to discover a subset of branches. If it is configured make sure to include PR-* to Discover Pull Requests.

Notes: Automatic branch project triggering > Branch names to build automatically Allows you to control the SCM commit trigger coming from branch indexing.

After saving the job, there is a view with contains all Repositories.

Each of the repositories contains Branches and Pull Request views as explained in [GitHub Webhook Pipeline Multibranch]

On GitHub

C. Validate GitHub WebHook

Make sure post-receive hooks has been created on GitHub at the Organization as follows:

<JENKINS_URL>/github-webhook/ for Pull request, Push and Repository events.

Troubleshooting

If the post-receive hooks are not there, please review Troubleshooting

References

Demo of GitHub Organization, multibranch projects, and pull requests

Tested products/plugins version

The latest update of this article has been tested with:

CJP-CM 2.73.3.1

GitHub plugin 1.27.0

GitHub Branch Source Plugin 2.3.1

Issue

I have an issue when installing / upgrading CloudBees Core from the cloudbees/cloudbees-core helm chart:

Error: render error in "cloudbees-core/templates/cjoc-statefulset.yaml": template: cloudbees-core/templates/cjoc-statefulset.yaml:119:24: executing "cloudbees-core/templates/cjoc-statefulset.yaml" at <eq .Values.OperationsCenter.RunAsUser 0.0>: error calling eq: incompatible types for comparison

Environment

CloudBees Core on Modern Cloud Platforms

Helm 2.15.0

Related Issue(s)

helm/helm issue #6708

helm/helm PR #6010 (Regression)

helm/helm PR #6749 (Fix)

Resolution

There is a known issue helm issue #6708 introduced in Helm / Tiller 2.15.0 that causes template generation to fail when comparing numeric value types.This issue has been fixed in Helm / Tiller 2.15.1.

The solution is to:

Upgrade helm (client) to version 2.15.1 or later (the method depends on how it is installed )

Upgrade tiller (server) with helm init --upgrade

Symptoms

Trying to perform a Move/Copy/Promote you get the error:

java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class ...

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

Diagnosis/Treatment

In a Support Bundle from the master where this issue is occurring, open the file-descriptors.txt file and look for the library where the class indicated in the error is included. Two different versions of this library will appear loaded in the core or a same plugin.

Look for the right library version for the core or plugin version where it is being loaded and remove the other one.

Issue

Kubernetes plugin is throwing errors while provisioning the agents pointing to io.fabric8.kubernetes.client.KubernetesClientException with a similar stacktrace like the following:

Error in provisioning; agent=KubernetesSlave name: python-pod-0ppnb-qnvc7, template=PodTemplate{, name='python-pod-0ppnb', namespace='jenkins', label='python-pod', nodeUsageMode=EXCLUSIVE, containers=[ContainerTemplate{name='python-agent', image='quay.io/dghubble/python-agent:4462978780179489a5608c480d9c64f8cb61bc7f', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}

io.fabric8.kubernetes.client.KubernetesClientException:

at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)

at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)

at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)

at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)

at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Environment

CloudBees Core

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Kubernetes plugin < 1.18.2

Known Issue

JENKINS-59000.

The error is inherited from the kubernetes client as you can tell from the fix in the plugin

Resolution

Upgrade Kubernetes Plugin to version 1.18.2 or later.

Summary

Cloudbees Accelerator v11.2 LTS version is anticipated to be released by the end of 2019. Under that version and for all future versions of Accelerator, support for the following will be dropped:

32-bit platforms

All Windows

All Linux

64-bit platforms:

Windows 7

Windows Server 2008

RHEL 5.4

RHEL 7.1

SLES Linux 11

Ubuntu 14.10, 15.04, 15.10, 16.10, 17.04, 17.10

Solaris

SPARC

i686

In addition, we will be dropping support for ClusterManager installations on Windows OSs.

Please recognize that all these OS flavors have been tested against the most recent 11.1.x versions, which will remain supported through 2020, so you can continue to use Cloudbees Accelerator with these operating systems on a supported platform through next year.

Also, while the above listed OS versions are no longer officially supported, some of them may continue to work against future versions, but they will not be formally tested by CloudBees, and any issues observed will need to be duplicated against a supported platform for Engineering to explore the issue.

Also, for Windows environments, with the up-coming release of the v6.0 Visual Studio Plugin (VSP), we will be dropping support for:

VS 2005

VS 2008

Again, the 5.2.x version of the VSP remains available for anyone who may still be using these versions of Visual Studio.

Support

Please contact Customer Support with any questions related to this news at [email protected].

When using ec-groovy to interact with Flow, you may receive the following error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This is a known issue that may occur when using ec-groovy. The following steps should be followed to resolve this problem:

Obtain the ElectricFlow server's certificate using the following command:

COMMANDER_HOME=/opt/electriccloud/electriccommander && LD_LIBRARY_PATH=$COMMANDER_HOME/lib && OPENSSL_CONF="$COMMANDER_HOME/conf/agentssl.cnf" && echo -e "quit\n" | $COMMANDER_HOME/bin/openssl s_client -connect replace-electricflow-server-name-here:8443

Copy the server certificate contents after "Server certificate" into a file, say ef-server.crt

Verify the certificate contents:

COMMANDER_HOME=/opt/electriccloud/electriccommander && LD_LIBRARY_PATH=$COMMANDER_HOME/lib && OPENSSL_CONF="$COMMANDER_HOME/conf/agentssl.cnf" && echo -e "quit\n" | $COMMANDER_HOME/bin/openssl x509 -in ef-server.crt -text

Import the certificate into the Java keystore:

COMMANDER_HOME=/opt/electriccloud/electriccommander && $COMMANDER_HOME/jre/bin/keytool -import -trustcacerts -keystore $COMMANDER_HOME/jre/lib/security/cacerts -storepass changeit -noprompt -alias myEFcert -file ef-server.crt

ec-groovy should now run successfully from the command-line.

Issue

Using Firefox, some actions are triggering twice.For instance:

clicking on build with parameter for a job using a file parameter can trigger two builds.

uploading a plugin in the plugin manager results in it being uploaded twice (and the second upload failing).

Environment

CloudBees Core on modern cloud platforms - Managed Master > 2.176.x

CloudBees Core on modern cloud platforms - Operations Center > 2.176.x

CloudBees Core on traditional platforms - Client Master > 2.176.x

CloudBees Core on traditional platforms - Operations Center > 2.176.x

CloudBees Jenkins Enterprise - Managed Master > 2.176.x

CloudBees Jenkins Enterprise - Operations Center > 2.176.x

CloudBees Jenkins Platform - Client Master > 2.176.x

CloudBees Jenkins Platform - Operations Center > 2.176.x

CloudBees Jenkins Distribution > 2.176.x

CloudBees Jenkins X Distribution > 2.176.x

Jenkins LTS > 2.176.x

Using Firefox as a browser.

Resolution

This issue appeared in Jenkins 2.176.x and was fixed in 2.202 (weekly).The fix was back-ported in the upcoming 2.190.3 LTS.

Workaround

In case you are running an impacted version, the only known workaround is to use another browser.

References

Issue

Some Jenkins plugins are based on the Ruby Runtime, should I use them?

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on traditional platforms - Client Master

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

Jenkins is written in Java and most of the plugins are written in Java too.Some plugins are written in Ruby and are depending on the Ruby Runtime Plugin to run.This plugin is providing the Ruby runtime as well as the necessary bindings for plugins written in Ruby to integrate with Jenkins.

There are several issues with this:* the Ruby Runtime Plugin is up for adoption and did not receive any update since 2016.* mixing up Ruby Runtime and Java Runtime is adding complexity and potential bugs.

The recommendation is therefore not to install any plugin depending on Ruby.

How to recognize such a plugin?

Look into the dependencies of the plugin if it depends on the Ruby Runtime Plugin.Note that no Tier 1 or Tier 2 plugin is depending on the Ruby Runtime Plugin.

Issue

How to create an agent in Linux from console

Environment

CloudBees Jenkins Enterprise - Managed Master (CJE-MM)

CloudBees Jenkins Team (CJT)

CloudBees Jenkins Platform - Client Master (CJP-CM)

Jenkins LTS

Resolution

For adding a node as an agent to Jenkins master using the console, follow following steps.

Step 1: On Jenkins master go to Manage Jenkins > Manage Nodes.

Remoting Best Practices

Step 2: Select New Node and enter node name.

Step 3: select Permanent Agent -> Press OK.

Step 4: On the next page, fill in the following fields:

Set a number of executors (one or more) as needed. Default is 1.

Set a Remote FS Root, a home directory for the master on the agent machine, e.g., /opt/jenkins/

Set Usage to Use this node as much as possible.

Choose Launch method as Launch agent by connecting it to the master (known as Launch agent via Java Web Start before 2.176.1).

Set Availability, > Keep this slave online as much as possible.

Press Save.

Step 5: Now for connecting to agent to the master by

Open a browser on the agent machine and go to the Jenkins master server url ( http://YourJenkinsMaster:8080 ).

Go to Manage Jenkins > Manage Nodes, click on the newly created agent machine.

Run from agent command line. Login to agent node -> download the slave.jar file from Jenkins Master UI to agent machine then while executing the command, please specify download path of slave.jar file.

On your console:

nohup java -jar /opt/jenkins/slave.jar -jnlpUrl http://YourJenkinsMaster:8080/computer/NameOfYourSlave/slave-agent.jnlp -jnlpCredentials userJenkinsMaster:passwordJenkinsMaster &

Now agent node will be in Connected state.

We recommend you choose connecting agent with ssh in linux servers How to connect to remote SSH Slaves because of this advantage:* If you restart your jenkins instance with any agent connected by JNLP you''ll lose the connection with your agents but if you use ssh Jenkins manages the connection with the agents so if you restart Jenkins all the agents should remain in place.

References

How to config slaves on windows and linux

Issue

This article explains how to inject secrets into Jenkins Build Jobs:* How to configure secrets in Jenkins?* How to inject them in a Pipeline Job?* How to inject them in a other Jobs?

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

Install the Credentials Plugins

If you havent already, install the latest versions of these plugins into your Jenkins instance.

Credentials plugin - provides a centralized way to define credentials that can be used by your Jenkins instance, plugins and build jobs.

Credentials Binding plugin - allows you to configure your build jobs to inject credentials as environment variables.

Plain Credentials plugin - a plugin dependency required by the Credentials Binding plugin.

Note: If you do not see these plugins in your list of available plugins, try refreshing the list of available plugins using the [Check now] button on the Advanced tab of the plugin management page. Also, be sure to restart Jenkins after installing the plugins.

Defining Credentials and Secrets

After installing the credentials plugins, your Jenkins sidebar will contain a new Credentials page that can be used to define credentials and secrets. The easiest way to define secrets for use in your build jobs is to:

Click the Credentials link in the sidebar

Click on the Global credentials domain

Click [Add Credential]

Select a credential kind that can be used from your build jobs (later, using a credentials binding). The following types of credentials are most useful to use from your build jobs.

Username with password - a username/password pair

References

Secret file - an uploaded file

Secret text - a raw secret string

*Note: be sure keep the default Global scope for credentials that need to be accessible to build jobs. *

*Note: the complete list of secret types can vary from one instance to another as other plugins can contribute to secret types.

*Note: be sure to specify the ID for the credential because that is how it is referenced in the job when using theparameter expression option. *

Using Credentials and Secrets from Build Jobs

In Pipelines

The credentials you created in Jenkins are injected in a Pipeline using the withCredentials step.Depending on the credential type, the syntax can vary slightly.The best way for you to find the exact syntax you need for a given credential type is to use the Pipeline Snippet Generator.You can access it browsing <jenkins_url>/pipeline-syntax/ or <jenkins_url>/<pipeline_jo>/pipeline-syntax (this can be important if you intend to generate steps for scoped variables).Once in the Pipeline Snippet Generator, you can select the withCrendentials step, input what you need in the UI and generate the step:

You can then use the generated snippet in your Pipeline.The way it works is that the variable you defined in the withCredentials step will then be available in the withCredentials scope.For instance, in the example of the screenshot:

withCredentials([usernamePassword(credentialsId: 'hello-kb', passwordVariable: 'pass', usernameVariable: 'user')]) {

// the code in here can access $pass and $user

}

In Non Pipeline Jobs

After installing the Credentials Bindings plugin, the Build Environment section of your build job Configuration page will include a new option to Use secret text(s) or file(s).Enabling this option allows you add credential bindings where the Variable value will be used as the name of the environment variable that your build can use to access the value of the credential.

Normally you will start your script with

set +x

so that commands you run are not echoed to the log, in case you mention the values of secrets in those commands.

The credential value will vary based on the kind of credential:

Secret text bindings are injected as a raw secret string

Shell example: Echo a secret text binding

# MY_SECRET is injected from a secret text binding as defined in Credentials.

echo "My secret is $MY_SECRET"

Username with password bindings are injected as a username:password formatted string

Shell example: Split a username and password binding into variables named MY_USERNAME and MY_PASSWORD.

# MY_SECRET_USER injected from username and password binding

# The username and password can be conjoined or separated

MY_USERNAME=`echo $MY_SECRET_USER` #separated example

MY_PASSWORD=`echo $MY_SECRET_USER_PASSWORD` #separated example

MY_USERNAME and MY_PASSWORD=`echo $MY_CREDENTIALS` #conjoined example

Secret file bindings are injected as filesystem path to the uploaded secret file

Shell example: Read the data from a secret file binding into a local variable

# MY_SECRET_FILE path injected from secret file binding

MY_FILE_DATA=`cat $MY_SECRET_FILE`

echo "The secret file data is: $MY_FILE_DATA"

Using Folders to control Credential usage

In larger build environments, it may be preferable to avoid putting all credentials into the Global domain (where they can be used by any build) and instead make it so that only some build jobs have access to certain credentials. This can be accomplished using the Folders plugin. The Credentials plugin is folder-aware and will enhance the Configure page for Folders with a Credentials section. Credentials defined on a folder can only be used by builds within that folder.

You can further secure access to credentials to only specific users in your organization by combining the Credentials and Folder plugins with the Role Based Access Control plugin.

Alternative ways to inject build variables

NOTE: Those are here for informational purpose but shouldnt be used anymore.

Build Secrets plugin - deprecated by Credentials plugin

EnvInject plugin - not recommended for secure data

Summary

We are pleased to announce the general availability of the CloudBees Flow 9.2 FR. This release is of the Feature Release type. For more information about types of releases, please go to CloudBees Flow Release Strategy.

This release modernizes how Flow is installed with deployment to Kubernetes, and improves visibility, management and administration of release automation with enhancements across the product.

Some of the highlights are:

Simplify server administration: Deploy Flow to Kubernetes

CloudBees Flow can now be installed in containers on Kubernetes. Use Helm charts to configure and deploy all components of the Flow architecture in Kubernetes, greatly simplifying configuration and installation of Flow. Flow server clusters can easily be scaled to meet enterprise workloads, making set up of clustered environments painless. Leverage your Kubernetes clusters and simplify server management and upgrades with Flow deployments in containers. This capability is available as a technical preview in 9.2, see the installation guide for instructions to get started.

Easily manage your automation as code in DSL with new import and export enhancements

Any object in CloudBees Flow can be authored and stored in the Groovy based DSL. DSL provides greater control and versioning of your pipelines, allowing management of ARO automation as code. In 9.2, we introduce enhancements to the DSL import and export functions, making it even easier to treat your pipelines as a product and manage them as code:

Two new Self Service Catalog items to import and export your DSL with the click of a button

Export Flow objects in multiple files in a directory structure that is easy to understand, edit and maintain, eliminating any need to review and manually edit the DSL.

A new overwrite mode for importing DSL replaces any existing fields on import, ensuring that the DSL code is the true representation of the automation.

New options for controlling which fields are exported to DSL, like suppressing null values and default fields and only exporting top level objects, for simpler and more manageable code.

Enterprise release management and visibility at scale

Enterprise releases can be complex with many moving parts and dependencies. In many cases enterprise releases are comprised of numerous sub-releases and application pipelines, where each application team determines when the application can join the release. In 9.2, we introduce new features for managing and monitoring enterprise releases at scale:

The new Portfolio List builds on the portfolio view, with a scalable list view showing real time visibility into the stage level status of each pipeline and its relationships in the release hierarchy. The new list format shows more pipelines in one view, scaling to meet the size of enterprise releases. Advanced search options allow you to quickly filter the view to focus on relevant pipelines.

New options in releases now enable application pipelines to be attached to a running release. Application owners can determine whether to attach their pipeline to the release when they see fit. Attached pipelines are added in the portfolio view, providing centralized visibility across the entire release train. In case an issue with the application arises and itcan'tbe released, the pipeline can be detached from that release. This provides flexibility to adjust the release payload even after the release has begun.

New options to customize the label for releases and pipelines in the Portfolio views, making the view easy to understand at a glance.

Maintain system size with data retention rules for purging

Expanding on data archiving introduced in 9.1, users can now define data retention rules with multiple strategies for maintaining data in the Cloudbees Flow system. The new purge option for data management will allow administrators to remove old runtime objects, like pipelines, deployments and jobs, from the database, as well as associated workspace files from the filesystem. This capability empowers administrators to easily manage and maintain system size and performance.

Get started with your custom dashboards with the DevOps Insight User Guide

DevOps Insight usage and administration information is now available in its own volume. This guide has a wealth of new content for understanding and extending the data model, creating custom dashboards and reports, server configuration and more. Use the guide for a deep understanding of DevOps Insight and to get started creating your own custom dashboards.

Release Command Center: Copy configurations for quick setup

Continuing work in the last two releases that improved setup of the Release Command Center, now when configuring the dashboard configurations, these can be copied from existing dashboards. Copying configuration details simplifies setting up a new release by leveraging previous data.

Administration pages migration for a unified experience

Various administration pages have been migrated from the Platform UI to Flows Deploy UI, with a new configurations page for managing system configurations. This provides central accessibility to administration pages with a consistent look and feel.

Plugins

The Flow Plugin Development Framework (FlowPDF) introduced in 9.1 simplifies creation of plugins in a few simple steps, empowering users to extend CloudBees Flow to work with a variety of tool chains. The guide is available here . We continue to improve FlowPDF in 9.2 with new enhancements:

Generate code for collecting reporting data from third party tools. This enables users to extend DevOps Insight and build dashboards with data from any tool.

Boilerplate code generation that simplifies creation of REST based plugins, along with support for different authentication mechanisms.

New libraries for error and exception handling.

A descriptive plugin developer guide that has step by step tutorials for building plugins along with sample code to get started.

We continuously develop and improve plugins used across DevOps processes:

Introducing two new plugins, EC-Bamboo and EC-OctopusDeploy, which have been built using FlowPDF.

Enhancements to EC-Jenkins, EC-EC2, EC-ServiceNow, EC-Git, EC-WebSphere, and more.

Download Instructions

Download Cloudbees Flow 9.2 FR from the CloudBees Sharefile site.

Login using your personalized account to the CloudBees Sharefile at https://cloudbees.sharefile.com

Navigate to: /products/flow/release_9.2/9.2.0.139827or click here: 9.2.0.139827

Server and agent installers are located in the directory corresponding to the operating system (for example,windows)

Release notes and product documentation are available online: CloudBees Flow Documentation

Supported Configurations

See the System Requirements and Supported Platforms section in theCloudBees Flow Installation Guidefor a complete list of supported platforms and configurations.

Upgrade

Upgrades to CloudBees Flow 9.2 FR are supported from ElectricFlow 5.x, 6.x, 7.x 8.x, 9.1 and 9.0.x versions. For complete upgrade requirements and instructions, please see the CloudBees Flow Installation Guide

Support

Contact Customer Support with questions related to this release at [email protected].

Issue

How do I email a html report generated in a job build (located in workspace) in an email notification?

I also want to ensure that the html report is embedded in the email body.

Environment

CloudBees Core

CloudBees Jenkins Enterprise

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Email-ext Plugin

Resolution

In Post-build actions - Editable Email Notification

Ensure Content Type is set to html.

In Default Content add ${FILE,path="my.html"}. Replace my.html with your html file. Path is workspace relative.

Alternatively, if you are interested in embedding an html file in an emails body using a pipeline, you can use as a reference the contents of stage('Email') listed below. The stage('Writing html file') has been included in the example for completeness. It is used to generate a simple html file and give you the opportunity of testing the code snippet as is.

node {

stage('Writing html file')

{

sh 'echo "<html>" >> myfile.html'

sh 'echo "<header><title> This is the title</title></header>" >> myfile.html'

sh 'echo "<body> how do you do? </body>" >> myfile.html'

sh 'echo "</html>" >> myfile.html'

sh 'ls -al myfile.html'

sh 'head -1 myfile.html'

}

stage('Email')

{

env.ForEmailPlugin = env.WORKSPACE

emailext mimeType: 'text/html',

body: '${FILE, path="myfile.html"}',

subject: currentBuild.currentResult + " : " + env.JOB_NAME,

to: '[email protected]'

}

}

Issue

How can I print all of the current environment variables in a Pipeline build?

Environment

Jenkins

CloudBees Jenkins Enterprise

Pipeline plugin

Resolution

There is a RFE for this issue as it can be seen on JENKINS-36528.

As a workaround: there are different resolution depending on the OS type of the agent/node. Add the following snippets in the scope where you wish to know the environment variables available

For linux env

sh 'env > env.txt'

for (String i : readFile('env.txt').split("\r?\n")) {

println i

}

A similar output that this one would be expected:

Started by user UserX

[Pipeline] Allocate node : Start

Running on principal in /jenkins_home/jobs/pipeline_example/workspace

[Pipeline] node {

[Pipeline] sh

[workspace] Running shell script

+ env

[Pipeline] readFile

[Pipeline] echo

LESSOPEN=| /usr/bin/lesspipe %s

[Pipeline] echo

JENKINS_HOME=/jenkins_home

[Pipeline] echo

...

SSH_AGENT_PID=1149

...

HUDSON_COOKIE=127483ad-7a1b-4669-bfe2-68ad2f49d733

...

MAVEN_HOME=/home/carlosrodlop/opt/maven/apache-maven-3.3.9

...

HTTP_ADDRESS=0.0.0.0

...

HTTP_PORT=8183

...

PATH=/usr/local/sbin:/usr/local/bin:...

...

EXAMPLE_JENKINS_VARIABLE=Test Variable

...

BUILD_TAG=jenkins-pipeline_example-1

...

SSH_AUTH_SOCK=/tmp/ssh-gAI1d7HUdJ4a/agent.1038

...

BUILD_NUMBER=1

...

HUDSON_SERVER_COOKIE=c76ea747cf0360cc

...

JOB_NAME=pipeline_example

...

PWD=/jenkins_home/jobs/pipeline_example/workspace

...

DEBUG_PORT=8193

...

[Pipeline] } //node

[Pipeline] Allocate node : End

[Pipeline] End of Pipeline

Finished: SUCCESS

For windows env

bat 'set > env.txt'

for (String i : readFile('env.txt').split("\r?\n")) {

println i

}

A similar output that this one would be expected:

Started by user UserX

[Pipeline] node

Running on master in C:\.jenkins\jobs\pipeline_example\workspace

[Pipeline] {

[Pipeline] bat

[workspace] Running batch script

C:\.jenkins\jobs\pipeline_example\workspace>set 1>env.txt

[Pipeline] readFile

[Pipeline] echo

...

BUILD_DISPLAY_NAME=#29

...

BUILD_ID=29

...

BUILD_NUMBER=29

...

BUILD_TAG=jenkins-example_job-29

...

BUILD_URL=http://jenkins.example/job/pipeline_example/29/

...

HUDSON_HOME=C:\.jenkins

...

HUDSON_SERVER_COOKIE=5d726cd684fad3f3

...

HUDSON_URL=http://jenkins.example:8080/

...

JENKINS_HOME=C:\.jenkins

...

JENKINS_SERVER_COOKIE=durable-5fe1ef76e59e42e0d503e3d9c9f7613c

...

JENKINS_URL=http://jenkins.example:8080:8080/

...

JOB_NAME=pipeline_example

...

JOB_URL=http://localhost:8080/job/pipeline_example/

...

NODE_LABELS=master

...

NODE_NAME=master

...

NUMBER_OF_PROCESSORS=1

...

OS=Windows_NT

...

Path=C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\

...

WORKSPACE=C:\Users\userx\.jenkins\jobs\pipeline_example\workspace

[Pipeline] }

[Pipeline] // node

[Pipeline] End of Pipeline

Finished: SUCCESS

References

Pipeline as Code in Jenkins

Issue

Suddenly all the Pipeline builds that use Maven Integration fail with errors that look like:

java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V

at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:83)

at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1108)

at org.jenkinsci.plugins.pipeline.maven.fix.jenkins49337.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:142)

Caused: javax.xml.parsers.FactoryConfigurationError: Provider for class javax.xml.parsers.DocumentBuilderFactory cannot be created

at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:311)

at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)

at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)

at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:74)

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

Locate and uninstall the plugin that loads the outdated implementation of the Java API for XML Processing (JAXP), e.g., Xerces 2.6.

Known plugins that cause the issue:

Discard Old Build. No fix available. The plugin has not been updated for six years.

Fortify 360. This plugin is not maintained and not supported by the original maintainer. Please use Fortify plugin instead.

If neither of the plugins is installed you can either:

Search through the open file descriptors to locate all the xercesImpl-*.jar files. If you see any with version 2.6.x, they are most likely the cause. The plugin id is the part of the path right after the plugins directory.

/tmp/jenkins/plugins/fortify360/WEB-INF/lib/xercesImpl-2.6.2.jar

/tmp/jenkins/plugins/build-pipeline-plugin/WEB-INF/lib/xercesImpl-2.9.1.jar

/tmp/jenkins/plugins/pipeline-aws/WEB-INF/lib/xercesImpl-2.12.0.jar

Use What Plugin is providing this class? to locate plugins loading the org.apache.xerces.jaxp.DocumentBuilderFactoryImpl class.

Looking for: org/apache/xerces/jaxp/DocumentBuilderFactoryImpl.class

Found in Plugin:pipeline-aws

jar:file:/tmp/jenkins/plugins/pipeline-aws/WEB-INF/lib/xercesImpl-2.12.0.jar!/org/apache/xerces/jaxp/DocumentBuilderFactoryImpl.class

Found in Plugin:discard-old-build

jar:file:/tmp/jenkins/plugins/discard-old-build/WEB-INF/lib/xercesImpl-2.6.2.jar!/org/apache/xerces/jaxp/DocumentBuilderFactoryImpl.class

If you are a CloudBees customer and having doubts or issues with the above, please generate a support bundle and open a support ticket.

References

[JENKINS-47486] FactoryConfigurationError: Provider for class javax.xml.parsers.DocumentBuilderFactory cannot be created (was Exceptions after running goals) [JENKINS-57833] All builds with maven pipeline plugin fail with java.lang.AbstractMethodError while parsing event spy log

Issue

Jenkins suddenly crashed and /var/log/kern.log or /var/log/dmesg.log shows:

[XXXXX] Out of memory: Kill process <JENKINS_PID> (java) score <SCORE> or sacrifice child

[XXXXX] Killed process <JENKINS_PID> (java) total-vm:XXXkB, anon-rss:XXXkB, file-rss:XXXkB, shmem-rss:XXXkB

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on modern cloud platforms - Operations Center

CloudBees Core on traditional platforms - Client Master

CloudBees Core on traditional platforms - Operations Center

CloudBees Jenkins Distribution

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Enterprise - Operations Center

CloudBees Jenkins Team

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Platform - Operations Center

Jenkins LTS

Linux

Background

A Java process is made up of:

Java heap space (set via -Xms and -Xmx)

the Metaspace (previously PermGen in Java 7)

the Native Memory area

Each one of these areas will use RAM. The memory footprint of Jenkins (a Java application) is the sum of the maximum Java heap size, the Metaspace size and the native memory area.

It is important to understand that the Operating System itself and any other processes running on the machine have their own requirements regarding RAM and CPU. The Operating System uses a certain amount of RAM which leaves the remaining RAM to be split among Jenkins and any other processes on the machine.

Resolution

(This does not indicate a problem with Jenkins. It indicates that the Operating System is unable to provide enough resources for all the programs it has been asked to run.)

The OOM Killer is a function of the linux kernel that kill rogue processes that are requesting more memory that the OS can allocate so that the system can survive. The function applies some heuristics (it gives each process a score) to decide which process to kill when the system is in such state. The process monopolizing the most amount of memory and not releasing enough of it is more likely to be killed.

If you are affected by this error, there could be different causes:

Too much memory is allocated to Jenkins

Other processes are running on the same machine as Jenkins

Following are recommendations for each case.

1) Too much memory allocated to Jenkins

You cannot (shouldnt) allocate as much memory as there is available on the machine. That is because the Operating System needs resources to manage the system.

We recommend keeping a ratio of Total Memory Available / Maximum Memory allocated to Jenkins JVM to 70%.

2) Other processes are impacting Jenkins

In this scenario, Jenkins is not the only process running on the machine but it is killed because it is the process consuming the most memory on the OS.

We strongly recommend that Jenkins be the only non-process running on the machine hosting it. Should you run other processes, like for example monitoring agents, ensure that they are not overloading the system or otherwise that enough resources are available to handle the load on the machine.

How to find the culprit

It is possible to check the processes consuming the most memory at any time on the machine with commands like:

$ top -o mem

or:

$ ps aux --sort -pmem

You can also dig into the /var/log/kern.log or the /var/log/dmesg.log. In these logs, locate the Out of memory: Kill process <JENKINS_PID> message. Just above that message, the kernel dumps the stats of the processes that were running. For example:

[...]

[XXXXX] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name

[XXXXX] [ 480] 0 480 13863 113 26 0 -1000 auditd

[XXXXX] [12345] 123 12345 4704977 3306330 6732 0 0 java

[XXXXX] [11939] 0 11939 46699 328 48 0 0 crond

[XXXXX] [11942] 0 11942 28282 45 12 0 0 sh

[XXXXX] [16789] 456 16789 1695936 38643 165 0 0 java

[...]

[XXXXX] Out of memory: Kill process 12345 (java) score 869 or sacrifice child

[XXXXX] Killed process 12345 (java) total-vm:18819908kB, anon-rss:13225320kB, file-rss:0kB, shmem-rss:0kB

[...]

In this example, the Jenkins PID was 12345 and it was killed. We can see in the summary (Killed process line) that Jenkins was reserving ~18 GiB of memory (see total-vm) of which ~12.6 GiB resided in RAM (anon-rss). However, in the table there is also another process with PID 16789 that is reserving ~6.4 GiB of memory (note that table memory values are in 4 KiB pages). You can then investigate more about this other process and see what it does by running the following command:

$ ps -f <pid>

Resources

For more details about the OOM Killer and this particular issue, have a look at the following links:

RHeL - Performance Tuning Guide

How to Configure the Linux Out-of-Memory Killer

Out of memory: Kill process or sacrifice child

Taming the OOM killer

Issue

Your web browser is not displaying certain page as expected

Your web browser is showing a blank page when navigating in the web interface

Environment

CloudBees Jenkins Enterprise - Managed Master (CJEMM)

CloudBees Jenkins Enterprise - Operations Center (CJEOC)

CloudBees Jenkins Team (CJT)

CloudBees Jenkins Platform - Client Master (CJPCM)

CloudBees Jenkins Platform - Operations Center (CJPOC)

Jenkins LTS

CloudBees Flow

Required Data for Rendering Problems or Loading Issues

This article describes how to collect the minimum required information for Rendering Problems or Loading Issues.

If the required data is bigger than 20 MB you will not be able to use ZenDesk to upload all the information. In this case we would like to encourage you to use our upload service in order to attach all the required information.

Theres no straightforward resolution for this kind of problems. First step is to try to use a more recent web browser. If it doesnt help, you should open a support ticket.

In order to help CloudBees to understand whats happening on your instance, you should collect some data. Please use either Chrome or Firefox to run this procedure, these browsers include the required tools to help us.

Take a screen shot of the web browser window, including its URL field by pressing Alt-Print screen

Record the HTTP session and Javascript console content of your Web browser

Chrome

Open the Developer Tools by pressing F12

Click on the Network tab

Ensure recording is enabled (round button at the top left of the Network tab should be red)

Activate Preserve log checkbox

Click the Clean button (second button at the top left of the Network tab)

Access the failing page or reload it by pressing F5

Save the capture by right-clicking on the grid and choosing Save as HAR with Content

Click on the Console tab

Take a screen shot of the browser window by pressing Alt-Print screen

Firefox

Ensure you have Firebug and NetExport extension installed (restart Firefox if you just installed them)

Open Firebug by pressing F12

Click on the Console tab and enable Console

Click on the Net tab and enable Net capture

Click the Clear and Persist button

Access the failing page

Save the capture by clicking Export button and Save as

Click on the Console tab

Take a screen shot of the browser window by pressing Alt-Print screen

Internet Explorer

In the case Chrome or Firefox is not an option.

###IE 9 ###

Follow the steps as covered in IE9 Developer Tools: Network Tab to save an XML file and provide it to us.

IE 11

Follow the steps as covered in Analyzing your webpages network traffic, and use Export captured traffic.

Appendix: HAR file

HAR file stands for HTTP ARchive and is produced by HTTP tracking tools. It contains a log of HTTP client/server conversation and is very useful for an additional / off-line analysis.

Problem

Job step exit code is very important as it is used to decide the result of the job step (which in many cases decides the procedure result and often used as a condition by following steps). In some cases, the job step exit code (thus the job step result) is not natural.

For example, a job run from the procedure in the following screenshot (please check the commands of the steps in the screenshot)

online document

will get the result in this picture:

Explanation

The result may be different to the result you expect. In order to understand why we get this result, we should first understand how these command are run by the flow agent.

By default (in Windows) the flow agent uses "cmd.exe" as the shell to run the step commands. Please check the for more details. This can also be confirmed from the agent log:

2019-11-08T16:56:58.291 | INFO | 0xcb4120 | Parsed request: cmdId: run-b7a1883c-0205-11ea-b60c-6cb920524153-b7c2cb3b-0205-11ea-b513-6cb920524153 cwd: shell: command: c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe /versionREM THIS IS A COMMENT LINE stdout: //flowserver/commander-workspace/job_1931_20191108165656/test_and_result_in_s.b7c2cb3b-0205-11ea-b513-6cb920524153.log postpShell: postpCommand: userName: 2019-11-08T16:56:58.291 | DEBUG | 0xcb4120 | Updated shell: cmd /q /c "{0}.cmd"2019-11-08T16:56:58.291 | DEBUG | 0xcb4120 | Converting command-block to CRLF line terminations.2019-11-08T16:56:58.291 | DEBUG | 0xcb4120 | Final cmd to send to run: C:/Program Files/Electric Cloud/ElectricCommander/bin/ecwrapper 0 "" 4 cmd /q /c "C:\Windows\TEMP\ecmdrAgent/agent.HHFESABH7FJTGIU8.run-b7a1883c-0205-11ea-b60c-6cb920524153-b7c2cb3b-0205-11ea-b513-6cb920524153.cmd" "//flowserver/commander-workspace/job_1931_20191108165656/test_and_result_in_s.b7c2cb3b-0205-11ea-b513-6cb920524153.log" ""2019-11-08T16:56:58.291 | DEBUG | 0xcb4120 | About to run command file: C:\Windows\TEMP\ecmdrAgent/agent.HHFESABH7FJTGIU8.run-b7a1883c-0205-11ea-b60c-6cb920524153-b7c2cb3b-0205-11ea-b513-6cb920524153.cmd

Please note that the commands are stored to an "cmd" file and then run by "cmd /q /c".

So we can run the commands in the same way and check the results. Here's the result when running them in command window (which is actually cmd.exe):

c:\test>type test_and_result_in_failure.cmdc:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe /versionc:\test>cmd.exe /q /c test_and_result_in_failure.cmd'c:\Program' is not recognized as an internal or external command,operable program or batch file.c:\test>echo %errorlevel%1c:\test>

c:\test>type test_and_result_in_success_1.cmdc:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe /versionecho abcc:\test>cmd.exe /q /c test_and_result_in_success_1.cmd'c:\Program' is not recognized as an internal or external command,operable program or batch file.abcc:\test>echo %errorlevel%0c:\test>

c:\test>type test_and_result_in_success_2.cmdc:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe /versionREM THIS IS A COMMENT LINEc:\test>cmd.exe /q /c test_and_result_in_success_2.cmd'c:\Program' is not recognized as an internal or external command,operable program or batch file.c:\test>echo %errorlevel%0c:\test>

With this test, we can conclude that flow agent strictly follow the result from the shell (in this case, cmd.exe).

Issue

We will occasionally encounter the following error in our BitBucket Multibranch builds:

hudson.plugins.git.GitException: Command "git checkout -f XXXXXXXX" returned status code 128:

stdout:

stderr: fatal: reference is not a tree: XXXXXXXX

Environment

CloudBees Core on modern cloud platforms - Managed Master

CloudBees Core on traditional platforms - Client Master

CloudBees Jenkins Enterprise - Managed Master

CloudBees Jenkins Platform - Client Master

CloudBees Jenkins Distribution

Jenkins LTS

Resolution

This is the expected behavior of the plugin under certain circumstances. As documented in JENKINS-45997 :

When new commits are pushed to a branch which has an open pull request, the pull request hashes are eagerly updated in the database but the refs on disk are not. This behavior has existed unchanged since pull requests were introduced in Stash 1.3.0. If someone views the pull request, then the refs are updated as part of performing a new merge to show an up-to-date diff. Any integration which attempts to use refs/pull-requests/*/from is going to be racy and unreliable because that ref is not eagerly updated.

Workaround

If you want to get past this issue, the recommended workaround is to enable Call Can Merge option in the BitBucket Endpoint configuration section which can be found in Manage Jenkins -> Global Configuration.

Tested product/plugin versions

CloudBees Core on traditional platforms - Client Master 2.164.3.2

BitBucket Branch Source Plugin 2.4.4 and 2.2.12

References

JENKINS-45997 BitBucket PRs failing to checkout on initial run after PR updated with new commits

Issue

How do I enable CSRF protection?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Resolution

GOTO: Jenkins > Manage Jenkins > Configure Global Security and enable Prevent Cross Site Request Forgery exploits.

Select Default Crumb Issuer from Crumb Algorithm and save to apply changes and enable.

See the CSRF Protection Wiki page for more.

Issue

How do I generate a crumb?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

wget

cURL

Resolution

Using wget

wget --user=admin --password=admin --auth-no-challenge -q --output-document - 'http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'

The output is the crumb.

Parameters:

--user=<USERNAME>

--password=<PASSWORD_OR_API_TOKEN>

Replace http://localhost:8080 with your Jenkins URL.

Using cURL

curl -u "admin:admin" 'http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'

The output is the crumb.

Parameters:

-u "<USERNAME>:<PASSWORD_OR_API_TOKEN>"

Replace http://localhost:8080 with your Jenkins URL.

Debugging Issues

Im seeing the following response:

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>

<title>Error 404 Not Found</title>

</head>

<body><h2>HTTP ERROR 404</h2>

<p>Problem accessing /crumbIssuer/api/xml. Reason:

<pre> Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>

...

...

CSRF Protection is not enabled. GOTO: Jenkins > Manage Jenkins > Configure Global Security and enable Prevent Cross Site Request Forgery exploits.

Issue

How do I use an issued crumb in an API call.

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

cURL

wget

Resolution

Using wget

The following example retrieves a crumb and uses it to build a job called someJob.

# Replace with your Jenkins URL and admin credentials

SERVER="http://localhost:8080"

CRUMB=$(wget --user=admin --password=admin --auth-no-challenge -q --output-document - "$SERVER"/crumbIssuer/api/xml?xpath=concat\(//crumbRequestField,%22:%22,//crumb\))

wget --user=admin --password=admin --auth-no-challenge --header="$CRUMB" --post-data="" -q "$SERVER"/job/someJob/build

Parameters:

--user=<USERNAME>

--password=<PASSWORD_OR_API_TOKEN>

--header="<ISSUED_CRUMB_FOR_USER>"

Using cURL

The following example retrieves a crumb and uses it to build a job called someJob.

# Replace with your Jenkins URL and admin credentials

SERVER="http://localhost:8080"

CRUMB=$(curl -u "admin:admin" "$SERVER"/crumbIssuer/api/xml?xpath=concat\(//crumbRequestField,%22:%22,//crumb\))

curl -X POST -u "admin:admin" -H "$CRUMB" "$SERVER"/job/someJob/build

Parameters:

-u "<USERNAME>:<PASSWORD_OR_API_TOKEN>"

-H "<ISSUED_CRUMB_FOR_USER>"

Debugging Issues

Im seeing the following response:

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>

<title>Error 403 No valid crumb was included in the request</title>

</head>

<body><h2>HTTP ERROR 403</h2>

<p>Problem accessing /job/someJob/build. Reason:

<pre> No valid crumb was included in the request</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>

...

...

This is caused by an invalid crumb being used.

Issue

Does the crumb expire?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Resolution

Once an user has received a crumb, it can re-used for all subsequent requests as long as this user stays in the same session. At code level, this dependency can be seen here.

Note :

Session may expire on its own depending on Jenkins settings.

Session might be aborted due to unexpected reasons, such as Failover of HA in CJP or SSO expiration

So generally keeping the same crumb is safe for short command sequences, but not for scripts running for several hours

Issue

How is the crumb generated?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Resolution

Using MD5. The crumb is a combination of user authentication (if authentication is applied), originating IP address and the crumb salt.

Issue

Is the crumb salt configurable?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Resolution

Currently no. It is hard-coded.

Issue

How do I use Jenkins CSRF protection behind a reverse proxy?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Reverse Proxy (applies to all)

Resolution

Issue#

Why is CURL working even if Idon'tinclude a crumb in the header?

Environment

CloudBees Jenkins Enterprise

CloudBees Operation Center

Resolution

If you authenticate with a username and password, a valid crumb is required.

If you authenticate with a username and a user API token then a crumb is not needed from Jenkins 2.96 weekly/2.107 LTS. For more information please refer to CSRF crumb no longer required when authenticating using API token or JENKINS-22474.

See CSRF Protection, Gotchas section.

Issue

I have an issue with cloudbees-core chart: It is not listed, It cannot be downloaded, etc.

Environment

CloudBees Core on Modern Cloud Platforms

Resolution

At the moment of writting this article, the ChartMuseum API allows you to:

Helm Chart Repository: index, download

Chart Manipulation: list, describe, upload, delete

Server Info: health checks

CloudBees Chartmuseum public URL: https://charts.cloudbees.com/public/cloudbees/

Example

One cloudbees-core chart version is listed under helm search but it is not available by helm fetch

$> helm search cloudbees/cloudbees-core -l

NAME CHART VERSION APP VERSION DESCRIPTION

cloudbees/cloudbees-core 2.164.1.2 2.164.1.2 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 3.6.0+4d2e34de1e86 2.190.2.2 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 3.5.0 2.176.4.3 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 3.4.1 2.176.3.3 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 2.176.203 2.176.2.3 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 2.176.104 2.176.1.4 The Continuous Delivery Solution for Enterprises

cloudbees/cloudbees-core 2.164.302 2.164.3.2 The Continuous Delivery Solution for Enterprises

$> helm fetch cloudbees/cloudbees-core --version 2.164.1.2

Error: chart "cloudbees-core" matching version "2.164.1.2" not found in cloudbees index. (try 'helm repo update'). improper constraint: 2.164.1.2

Looking at Chart Manipulation methods, we find describe a chart version GET /api/charts/<name>/<version>.

curl -X GET https://charts.cloudbees.com/public/cloudbees/api/charts/cloudbees-core/2.164.1.2

{"error":"improper constraint: 2.164.1.2"}

There is a problem with the Chart museum version, it seems that it does not support four coordinates.

Tested product/plugin version

The latest update of this article has been tested with

chartmuseum v0.9.0

Issue

I want to prevent my node to be allocated if Disk Usage is over 80%

Environment

CloudBees Jenkins Enterprise

CloudBees Jenkins Operations Center

Resolution

First, Node Monitoring can be configured at Manage Jenkins/Manage Nodes/Configure. The default Free Space Threshold is set to 1GB which means that nodes are not allocated if the amount of Free Space is under 1GB. This is a setup for all slaves.

A way to monitor a specific thread is to use the Nodes Plus Plugin. A probe command can be specified to decide whether or not a node is able to accept tasks. If the command returns an exit code other than 0, the slave cannot accept tasks. The best way to use this feature is to create a script on the slave machine and launch it as a probe command.

Example

I want to prevent my unix node from being allocated if the Disk Usage is over 80%.

The following command gives me the file system usage on the machine:

my-slave:~$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/disk1 233G 138G 96G 60% /

My $SLAVE_HOME is located on /dev/disk1 so I can workout a command like the following to grab the usage value (Use% is the fifth value in the line) of this disk:

my-slave:~$ df -h | grep /dev/disk1 | awk '{print $5}' | sed 's/%//g'

60

I can then create a probe.sh located under $SLAVE_HOME to exit with error code -1 when my disk usage is over 80%:

#!/bin/bash

CURRENT=$(df -h | grep /dev/disk1 | awk '{print $5}' | sed 's/%//g')

THRESHOLD=80

if [ $CURRENT -gt $THRESHOLD ]; then

exit -1

fi

I configure my node with the probe as following: