CloudBolt Software FAQs

Version

General Availability

Deprecation Date

End of General Support

7.0

3/14/2017

3/14/2018

6/14/2018

7.1

4/27/2017

4/27/2018

7/27/2018

7.2

6/2/2017

6/2/2018

9/2/2018

7.2.1

6/21/2017

6/21/2018

9/21/2018

7.2.2

7/10/2017

7/10/2018

10/10/2018

7.3

8/11/2017

8/11/2018

11/11/2018

7.4

9/26/2017

9/26/2018

12/26/2018

7.5

10/31/2017

10/31/2018

1/31/2019

7.6

12/7/2017

12/7/2018

3/7/2019

7.7

1/17/2018

1/17/2019

4/17/2019

7.7.1

2/9/2018

2/9/2019

5/9/2019

7.7.2

3/2/2018

3/2/2019

6/2/2019

7.7.3

3/8/2018

3/8/2019

6/8/2019

8.0

4/20/2018

4/20/2019

7/20/2019

8.1.0.1

6/1/2018

6/1/2019

9/1/2019

8.1.1

6/29/2018

6/29/2019

9/29/2019

8.2

7/24/2018

7/24/2019

10/24/2019

8.3

9/18/2018

9/18/2019

12/18/2019

8.4

11/8/2018

11/8/2019

2/8/2020

8.4.1

12/5/2018

12/5/2019

3/5/2020

8.5

1/4/2019

1/4/2020

4/4/2020

8.6

3/7/2019

3/7/2020

6/7/2020

8.7

5/29/2019

5/29/2020

8/29/2020

8.8

7/26/2019

7/26/2020

10/26/2020

9.0

10/16/2019

10/16/2020

1/16/2021

9.1

11/25/2019

11/25/2020

2/25/2021

Deprecation:Upgrades from this version to the latest version will stop being tested on this date. Upgrades are still supported, but may take multiple steps. This date is typically 12 months from General Availability.

End of General Support: CloudBolt Professional Services can still be engaged to help after this date, but CloudBolt support will require upgrading to a newer version before troubleshooting/advising. This date is typically 3 months from Deprecation.

Requirements:

CloudBolt version 8.6 or newer

Understand and keep a note of the valid job types. Valid job types are:

provision,decom,healthcheck,expire,servermodification,deploy_blueprint,install_pod,installapplications,uninstallapplications,syncvms,sync_svrs_from_pe,functionaltest,runautomations,runautomationactions,sync_users_from_ldap,run_flow,install_apps_with_connector,manage_nics,orchestration_hook,delete_snapshots

Configuration steps:

Set an environment variable called CBJE_INCLUDE_JOB_TYPES to either a single job type or a comma-separated list of job types from the list of valid job types.

Run the job engine

On other job engines that you want to run all other job types, set the environment variable CBJE_EXCLUDE_JOB_TYPES to the same value as in step one.

Run the job engine

Test Steps:

Run jobs of several types to confirm that:

they are all running properly

the expected JE is running them

Warning: do not set both of those environmentvariables at the same time. The CB JE will respect both; so if you set them to the same value, the JE will run no jobs at all.

There are cases where it makes sense to provide separate sync schedules for specific resource handlers. To create such a schedule, go to the specific Resource Handler instance you'd like to schedule on its own, click the "Servers" tab, then click the button labeled "Sync VMs from resource handler". This action will kick off a new CloudBolt job -- note the Job ID for this job.

To create a new "Recurring Job" for this sync, go to "Admin > Recurring Jobs", then select "Add from job" which is accessible from the "Add a recurring job" button drop-down. In the "Add a Recurring Job" dialog, enter a name, cron expression, and the Job ID noted above. For example, to sync every hour on the hour, enter the expression:0 0 * ? * *

You will now have a new Recurring Job that will sync the specified resource handler according to the cron schedule specified.

Note: The main "Sync all Resource Handlers" recurring job will continue to sync ALL resource handlers. If this is not desired, then the above process should be repeated for each resource handler.

Problem

I have multiple datastoresnamed DS1, DS2, & DS3available to my vCenter instance.

When provisioning new servers, I want CloudBolt to automatically use the data store with the most available capacity.

The user should not have to select or interact with the availabledatastores.

Solution

For a given vCenter-backed environment, go to the VMware tab and click "import parameters from <resource handler name>". This will import all available datastore and resource pools from the associated vCenter instance.

Note the datastores from which you want CloudBolt to choose.

Click the "Edit" pencil and click "remove all" to remove all datastores from the list.

Click "Add Another" and enter each datastore as a comma delimited list, e.g. DS1, DS2, DS3. There should only be ONE value consisting of a comma-delimited string of datastore names in the list of datastores.

Since there's only one option, the datastore parameter will be hidden from the user and the single option will be automatically selected.

Click Save.

For servers in this environment, CloudBolt will now select the datastore with the most free space from the list DS1, DS2, and DS3

Troubleshooting a provisioning issue? Here are a couple of VMware guest customization problemsthat can happen when provisioning a Windows VM. These issues aren'tspecific to CloudBolt, but this guide will tellyou how to recognize them when they pop up during CloudBolt provisioning.

Issue #1: "Invalid credentials" as Administrator

Symptom: Scripts fail when run as Administrator after guest configuration (from the Post-Network Configuration step onward). The error message varies between vCenter version, but one variation looks like "msg = 'A general system error occurred: vix error codes = (1, 0).".

Cause: Starting with Windows 7, the sysprep command run as part of guest customization will disable the Administrator account by default. This causes an "invalid credentials" error when CloudBolt tries to run scripts as Administrator.

Solution: Instruct your template to re-enable the Administrator account after sysprep with the following steps.

1. Boot up your template and create a %WINDIR%\Setup\Scripts\ directory. 2. Inside that directory, create a SetupComplete.cmd file.3. Add the following contents to that file: net user administrator /active:yes

See https://kb.vmware.com/kb/2034622 for more details on this issue.

Issue #2: CloudBolt hangs waiting for hostname to be reported

Symptom: CloudBolt hangs while waiting for the hostname to be set by guest customization. On the server being provisioned, the sysprep error log (C:\Windows\System32\sysprep\Panther\setuperr.log) contains a line like:

<DATE> <TIME>, Error SYSPRP Package Microsoft.DesktopAppInstaller_1.0.1471.0_x64__8wekyb3d8bbwe was installed for a user, but not provisioned for all users. This package will not function properly in the sysprep image.

Cause: The sysprep command, which happens as part of guest customization, is failing to clean Appx packages and generalize the image.

Solution: Remove the offending package on the template. Follow the steps in the Resolution section of Microsoft article below.

See https://support.microsoft.com/en-us/help/2769827/sysprep-fails-after-you-remove-or-update-windows-store-apps-that-include-built-in-windows-images for more details.

While CloudBolt is typically installed as a single appliance, the individual pieces of CloudBolt can be split onto different systems. The easiest way to do this is to install 2 separate virtual appliances, then use the following steps to point the Apache front-end from one system to the MySQL database from the second system.

Configuring the db server: 1) Get to the mysql shell (usually mysql -u root --password=Vi-gn3tt3 cloudbolt from the commandline)

2) Type:

GRANT ALL PRIVILEGES ON cloudbolt.* TO root@'cloudbolt_ip' IDENTIFIED BY 'Vi-gn3tt3'; (Replacing cloudbolt_ip with the ip address of the server you want to be able to access this database from)

3) Close and exit from the shell

4) Edit /etc/my.cnf and make sure the bind-address is set to the ip of the database server

5) Restart mysqld with service mysqld restart

6) Open the iptables to give access to the mysql port: /sbin/iptables -A INPUT -i eth0 -p tcp --destination-port 3306 -j ACCEPT Change eth0 to the appropriate value (use ifconfig to see which eth is being used)

7) Save changes to iptables with service iptables save

Configuring the CloudBolt front-end:

1) Set up the CloudBolt server you want to be able to access the remote database

2) From /opt/cloudbolt/customer_settings.py in the DATABASES section, add values for HOST and PORT where host is the ip of the database server and port is 3306

3) Restart apache with service httpd restart

The CloudBolt appliance ships with LVM installed, therefore it's relatively easy to expand the root disk partition If you know the right commands. The following steps allow you to expand the rood disk partition online without rebooting the CloudBolt appliance beyond it's default size of 40GB.

Add a new disk to the CloudBolt VM.

Format this new disk with(in this case /dev/sdb) with a linux partition:

Run this command: `cfdisk /dev/sdb`

In cfdisk, select the free space, and then select "New" at bottom of terminal.

Make the new partition a `Logical` partition.

Enter the amount of disk space you want to add to the CloudBolt appliance.

Select `Write` to save changes. You may have to make the partition `Bootable` to allow it to write.

Add physical disk to LVM: `pvcreate /dev/sdb1`

Add physical volume to volume group: `vgextend vg_c2 /dev/sdb1`. If that command does not work, try `vgdisplay` to check the name of your volume group and use that in place of `vg_c2`

Extend the logical volume: `lvextend -l +100%FREE /dev/mapper/vg_c2-lv_root`

Resize the file system: `resize2fs /dev/mapper/vg_c2-lv_root. For these last two steps, do the same as step #4, change `vg_c2` to the name of your volume group.

That is all you need to do to extend the primary partition on your CloudBolt instance.

Dependencies

Python Library Dependencies (Pre-CB v.7.4)

You'll have to first install a library and its dependencies on the CloudBolt appliance with pip. When prompted, type "y" to continuing during the uninstall process.

$ yum install xmlsec1 xmlsec1-openssl$ pip install djangosaml2==0.16.11

CloudBolt Application Setup

Update customer_settings.py

All configuration changes should be made here:

/var/opt/cloudbolt/proserv/customer_settings.py

All the changes you see in this section should be made to this file. If the file doesn't exist on your CloudBolt server, you can create it yourself.

INSTALLED_APPS

The INSTALLED_APPS setting tells CloudBolt which Django applications should be loaded at startup. Since djangosaml2 is a Django app, we need to append it to the list of applications to be loaded:

from settings import INSTALLED_APPSINSTALLED_APPS += ('djangosaml2',)

AUTHENTICATION_BACKENDS

This setting configures which authentication plugins are available to CloudBolt. In this case we're adding the SAML2 authentication backend.

from settings import AUTHENTICATION_BACKENDSAUTHENTICATION_BACKENDS += ('djangosaml2.backends.Saml2Backend',)

LOGIN_REQUIRED_URLS_EXCEPTIONS

We need to instruct CloudBolt not to require login to the urls used for authenticating SAML users.

from settings import LOGIN_REQUIRED_URLS_EXCEPTIONSLOGIN_REQUIRED_URLS_EXCEPTIONS += (r'^/saml2/',)

LOGIN_URL

This step is optional, and forces all user logins to use SAML2.

LOGIN_URL = '/saml2/login/'

Update customer_urls.py

The '/var/opt/cloudbolt/proserv/customer_urls.py' file is used to provide url mapping to application views. While the djangosaml2 application provides its own routing, we need to include this routing with the parent application routes by adding the following to '/var/opt/cloudbolt/proserv/customer_urls.py'.

Again, if this file doesn't already exist, simply create and edit. If you want to set the URL for the SAML path to something other than saml2/ you can do that by changing saml2/ to whatever you'd like; just keep you'll have to update the LOGIN_URL and LOGIN_REQUIRED_URLS_EXCEPTION settings in customer_settings.py.

Note: If the first two lines are already in the file, you do not need to add them again.

from django.conf.urls import include, urlfrom urls import urlpatternsurlpatterns += [ url(r'^saml2/', include('djangosaml2.urls')),]

Service Provider (SP) Configuration (On CloudBolt Server)

Create saml2 directory

If the '/var/opt/cloudbolt/proserv/saml2/' and '/var/opt/cloudbolt/proserv/saml2/attribute-maps/' directories don't exist in '/var/opt/cloudbolt/proserv', please create them as this is the directory where all SAML certs, and IDP metadata will be stored.

Import SAML Attribute Maps

You can download the pysaml2 attribute maps from: https://github.com/rohe/pysaml2/tree/master/src/saml2/attributemaps.

All the .py files in this directory should be downloaded to '/var/opt/cloudbolt/proserv/saml2/attribute-maps/'.

If its easier, heres a small script you can run from the cloudbolt server.

mkdir -p /var/opt/cloudbolt/proserv/saml2/attribute-maps/cd /var/opt/cloudbolt/proserv/saml2/attribute-maps/wget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/__init__.pywget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/adfs_v20.pywget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/adfs_v1x.pywget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/basic.pywget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/saml_uri.py wget https://raw.githubusercontent.com/IdentityPython/pysaml2/master/src/saml2/attributemaps/shibboleth_uri.py

Generate Self-Signed Key-Pair

These attribute maps are used to map standard identifiers for attributes provided by your IdP to local attributes that are mapped to the local system in the SAML_ATTRIBUTE_MAPPING variable. No changes should be required to the attribute map files.

Run the following commands and copy saml.key and saml.crt to/var/opt/cloudbolt/proserv/saml2:

$ openssl genrsa -out saml.key 2048$ openssl req -new -key saml.key -out certreq.csr$ openssl x509 -req -in certreq.csr -signkey saml.key -out saml.crt

If names other than saml.key and saml.crt are used, please update their references in the SAML_CONFIG settings in customer_settings.py.

Fetch IdP Metadata

SAML2 identity providers will often allow you to download their IdP metadata for use on your CloudBolt instance. In the configuration below, our metadata was downloaded from Google's SAML2 configuration control panel and stored at /var/opt/cloudbolt/proserv/saml2/remote_metadata.xml. You'll see this file is referenced in the example configuration file below.

Setup SAML_CONFIG in customer_settings.py

The following configuration is used to connect CloudBolt to Google's SAML2 IDP. You'll want to refer to pysaml2 documentation at https://pythonhosted.org/pysaml2 for more information on the available directives and configuration options.

import osimport saml2SAML2_DIR = os.path.join('/var/opt/cloudbolt/proserv/saml2')SAML_CONFIG = { 'xmlsec_binary': '/usr/bin/xmlsec1', 'entityid': 'https://samldemo2.company.com/saml2/metadata/', 'service': { 'sp': { 'want_assertions_signed': False, 'want_response_signed': False, 'allow_unsolicited': True, 'name': 'CloudBolt SP', 'endpoints': { 'assertion_consumer_service': [ ('https://samldemo2.company.com/saml2/acs/', saml2.BINDING_HTTP_POST), ], 'single_logout_service': [ ('https://samldemo2.company.com/saml2/ls/', saml2.BINDING_HTTP_REDIRECT), ], }, 'required_attributes': ['email'], }, }, 'debug': 1, 'key_file': os.path.join(SAML2_DIR, 'saml.key'), # private part 'cert_file': os.path.join(SAML2_DIR, 'saml.crt'), # public part 'allow_unknown_attributes': True, 'attribute_map_dir': os.path.join(SAML2_DIR, 'attribute-maps'), 'metadata': { 'local': [os.path.join(SAML2_DIR, 'GoogleIDPMetadata-company.com.xml')], }, 'contact_person': [{ 'given_name': 'First', 'sur_name': 'Last', 'company': 'Company', 'email_address': '[email protected]', 'contact_type': 'technical' }], 'organization': { 'name': 'Company', 'display_name': 'Company', 'url': 'http://www.company.com', }, 'valid_for': 24, # how long is our metadata valid}SAML_DJANGO_USER_MAIN_ATTRIBUTE = 'username'SAML_CREATE_UNKNOWN_USER = TrueSAML_ATTRIBUTE_MAPPING = { 'email': ('email', ), 'givenName': ('first_name', ), 'sn': ('last_name', ), 'uid': ('username', ),}

In the above example, the key parameters are as follows:

Entity ID: https://samldemo2.company.com/saml2/metadata/

ACS URL: https://samldemo2.company.com/saml2/acs/

SLS/Logout URL: https://samldemo2.company.com/saml2/ls/

IDP Metadata file: GoogleIDPMetadata-company.com.xml

Integrating LDAP Group Mappings for SAML2 SSO Auto-Created Users

When a user logs in via SSO and theyve never logged in before, Django will auto create a user in the local user space in CloudBolt without an LDAPUtility assigned.

Once logged in, the user might expect certain permissions assigned to their user account because they logged in with their LDAP/AD credentials, however, because they logged in via SAML SSO initially, they were not authenticated with an LDAP domain when logging on, so CloudBolt does not know that they should be assigned to any LDAP groups.

The following code modifies the default behavior of the django user create method and passes in the SAML attributes sent via SSO to the External Users Sync Orchestration Action.

Save the following code as:/usr/local/lib/python3.6/site-packages/djangosaml2/backends2.py

import osimport saml2SAML2_DIR = os.path.join('/var/opt/cloudbolt/proserv/saml2')SAML_CONFIG = { class CustomSaml2Backend(Saml2Backend): def update_user(self, user, attributes, attribute_mapping, force_save=False): """ Custom SAML2 update_user method to handle group attributes that will map to CloudBolt groups """ logger.info(f'LDAP attributes = {attributes}') logger.info(f'LDAP attribute_mapping: {attribute_mapping}') user = super(CustomSaml2Backend, self).update_user( user, attributes, attribute_mapping, force_save=False ) profile = user.userprofile run_hooks("external_users_sync", job=None, users=[profile], attributes=attributes) return user

Also verify the permissions on this file are similar to ./backends.py for ./backends2.py

Change the /var/opt/cloudbolt/proserv/customer_settings.py file for the following property (replace this entry: djangosaml2.backends.Saml2Backend) so that it looks like this:

AUTHENTICATION_BACKENDS += ('djangosaml2.backends2.CustomSaml2Backend',)

Contact your account representative for an updated external users sync script, which calls theexternal users sync while possibly passing the SAML Attributes that will allow for any LDAP domain configured in CloudBolt to match the users login LDAP domain name.

Here's an excerpt from said plugin:

def run(job, mapping=None, dry_run=False, **kwargs): logger.debug("Running hook {}".format(__name__)) users = kwargs.get('users', None) attributes = kwargs.get('attributes', None) if attributes and not dry_run: setup_ldap_for_saml_logins(users=users, attributes=attributes)

NOTE: It is necessary to configure your SAML2 provider to send an extra attribute in the SAML Assertion data in order to correctly identify the users LDAP domain. The plugin mentioned above assumes a Microsoft ADFS SAML2 provider, and has the configuration set for the appropriate property name in Microsofts SAML2 assertion attributes data. You will need to modify this property based on the property name that your SAML2 provider is sending. UPN translates to UserPrincipalName.

UserPrincipalName follows this format: sAMAccountName@LDAPDomainName

i.e. [email protected]

UPN_ATTR = 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn'

Okta, OnePoint and other SAML2 providers should be able to pass an extra attribute with that data, but the name is arbitrary so make sure to modify the UPN_ATTR to the name of the attribute being passed from the appropriate SAML2 provider.

Afteryou'vefinished all this, we should now be able to restart the httpd service.

service httpd restart

You should now be able to test SAML2 auto-user creation and have the LDAP Domain configured appropriately on the CloudBolt user sent via SAML2, as well as the appropriate permissions based on your LDAP Group membership (as defined by the LDAPGroupMapping object).

Test the configuration, and look at the /var/log/cloudbolt/application.log if you have issues or if the website doesnt properly load.

Encrypted SAML Responses

As a part of hardening the SAML integration from a security perspective, we can encrypt the SAML response object (all the data that the we use to create users in Django/CloudBolt)

This data includes:

LDAP username,

Email address,

GivenName/FirstName,

SurName/LastName,

LDAP groups the user is a member of (potentially),

SAML Response encryption requires updates on both the IdP and the customer_settings.py on the CloudBolt server in the form of a certificate to encrypt/decrpyt the SAML response data. For the purposes of testing, you may use the SSL certificate that your dev CloudBolt servers Apache service is running as.

The IdP will need the public key, and the CloudBolt server will need the public and private keys.

Without the configuration to encrypt saml responses, you will see a log entry in the cloudbolt application.log when a user logs in via a saml assertion:

2019-01-22 13:57:02,079 [DEBUG] saml2.response: ***Unencrypted assertion***

Response Encryption can be attained by adding this information to the /var/opt/cloudbolt/proserv/customer_settings.py in the SAML_CONFIG

'encryption_keypairs': [{ 'key_file': '/etc/pki/tls/private/cloudbolt.name.net.key', 'cert_file': '/etc/pki/tls/certs/cloudbolt.name.net.crt' }],

You can use a different key than apaches SSL certificate.

Note: You will need to restart the httpd service in order for this configuration change to become active.

Make sure that you configure the IdP to use the public key certificate that is referenced in the SAML_CONFIG[encryption_keypairs][cert_file] section in customer_settings.py, so that the SAML response can be sent encrypted with a certificate that SAML knows how to decrypt.

Once the configuration is updated in both the IDP and the CloudBolt customer_settings.py (and httpd is restarted), on a new SSO login, you should see that the response is encrypted based on this log entry:

2019-01-22 14:14:12,250 [DEBUG] saml2.response: ***Encrypted assertion/-s***

...and SAML2 authentication should process correctly.

If you get an error in the browser similar to:

'NoneType' object has no attribute 'authn_statement'

...you at least know that the IdP is sending an encrypted response but the SAML_CONFIG in CloudBolt doesnt know how to process it.

This typically means one or more of the following conditions have happened:

The configuration change was not made to the customer_settings.py or it has typographical errors (correct file names?)

The httpd service was not restarted after making the configuration changes in the customer_settings.py

Example customer_settings.py (excerpt):

https://pysaml2.readthedocs.io/en/latest/howto/config.html

Common Identity Providers (IdPs)

No matter what SAML Identity Provider in use, all that is needed is the same information from any of them to correctly create a user in CloudBolt and attach that user to the correct LDAP Domain (for automatic placement of authenticated users into CloudBolt Groups).

Username

FirstName

LastName

Email

UserPrincipalName

User Group List [Distinguished Names] (OPTIONAL, but suggested for future enhancements to the external users sync script. LDAP-based lookups from CloudBolt work just fine without this)

Okta

Okta is a pretty straight-forward IdP and lots of customers have successfully implemented Okta to CloudBolt integration. About the only issue we tend to see is that you can customize the Attribute names and we need to know what the attribute names Okta is sending for each attribute:

From this example we would need the FirstName, LastName, Email and eventually we would need to define UserName and or UserPrincipalName as well, in order to set the customer_settings.py SAML_ATTRIBUTE_MAPPINGS settings correctly for CloudBolt.

This particular string could be used as a template to pass an appropriate value for the UserPrincipalName based on what is configured for user.login. In the example below, the user had something similar to this in user.login [email protected] (which didnt match the LDAP domain name required).

Placing this string in the value of the ATTRIBUTE STATEMENTS in Okta for an Attribute named UPN allowed us to send the needed data in the format required to override the emaildomainname that was present in user.login and pass the valid LDAP domain name as part of this Attribute.

String.substringBefore(user.login, "@")+"@somedomain.com"

ADFS (Microsoft)

Microsofts IdP (Active Directory Federation Services) is a little harder to work with, because they customized it a lot and the attribute names are URLs not just names like username.

Additionally, differences in ADFS v2 (Windows 2012) and subsequent changes/improvements in ADFS v3 (Windows 2016) make implementations slightly different. Once a Relying Party Trust has been created, verify the following screenshots look similar to this. Tabs that are not shown here, have nothing configured in them. See examples of the Attributes we can make use of in ADFS (screenshot from ADFS v3):

After configuring the Relying Party Trust, the customer will need to Edit Claim Issuance Policy for this Relying Party Trust with the following settings.

NOTE: ADFS can pass the Group list that a user is a member of so that instead of querying LDAP, the SAML Response data can include the group membership list (as Group DistinguishedNames) from Active Directory. In addition the only LDAP Attribute mapping missing fromthe image above is the User-Principal-Name with an Outgoing Claim Type of UPN.

In ADFS, we can click on the View Rule Language button to see how these Attributes look with their actual attribute names (while noting that some of the URLs are prefixed with http://schemas.microsoft.com and others are prefixed with http://schemas.xmlsoap.com:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"] => issue(store = "Active Directory", types = ("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname", "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", "http://schemas.microsoft.com/ws/2008/06/identity/claims/role", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"), query = ";mail,sn,givenName,sAMAccountName,memberOf,userPrincipalName;{0}", param = c.Value);

NOTE: These URL attribute names are going to be the names we need to map in the customer_settings.py for CloudBolt in SAML_ATTRIBUTE_MAPPINGS.

APPENDIX:

Documentation for SAML2 configuration entries

Troubleshooting

Time Skew

Time Skew is a major factor when configuring SAML because SAML will refuse to authenticate if the system clocks are off by a particular amount. Heres an example from the CloudBolt logs where the CloudBolt server clock and the ADFS server clock are off by too much time. Notice that we are 1.5 minutes ~ outside of the acceptable window in order to use the SAML Response object. (the CloudBolt server is 4 hours off of Z aka UTC time.

Please consider configuring the CloudBolt server as an NTP client to correct clock skew.

2019-04-08 12:23:31,528 [DEBUG] saml2.response: conditions: <saml:Conditions xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2019-04-08T16:24:59.893Z" NotOnOrAfter="2019-04-08T17:24:59.893Z"><saml:AudienceRestriction><saml:Audience>https://cloudbolt.corp.tmnas.com/saml2/acs/</saml:Audience></saml:AudienceRestriction></saml:Conditions>2019-04-08 12:23:31,530 [ERROR] saml2.response: Exception on conditions: Can't use response yet: (now=2019-23-08T16:23:31Z + slack=0) <= notbefore=2019-04-08T16:24:59.893Z

Example ADFS configured CloudBolt customer_settings.py

If the Test URL button from ADFS doesn't work, but you can browse the metadata url from Internet Explorer, Chrome, et al, Check the following Registry Keys on the ADFS server to verify they exist and are set appropriately (in order to enable TLS 1.2 in 64-bit and 32-bit .Net Frameworks for v2 and v4).

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727]"SchUseStrongCrypto"=dword:00000001[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]"SchUseStrongCrypto"=dword:00000001[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319]"SchUseStrongCrypto"=dword:00000001[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v2.0.50727]"SchUseStrongCrypto"=dword:00000001

NOTE: A Reboot of your AD FS Server is required after changing the SchUseStrongCrypto Values

Key points:

want_response_signed: Response Signing ensures that the RP (CloudBolt) knows that the IdP sent the data because its signed with a certificate that CloudBolt can verify locally (enabled in this example)

encryption_keypairs: Response Encryption encrypts the data to ensure an extra layer of security - in addition to the Response being signed, We can further encrypt the data with another certificate (at IdP) and decode it with the private key in CloudBolt (enabled in this example) - not necessary for basic communication (only provides added security)

allow_unknown_attributes: Allows extra attributes to be sent in the SAML Response to the SP beyond what is expecting in the SAML_ATTRIBUTE_MAPPING

DEBUG = True (allow debug messages to get sent to the application.log - in conjunction with setting the SAML logging to debug via SAML_CONFIG[debug])

AUTHENTICATION_BACKENDS (need to set this to backends2 in order to allow for SAML integrated Django user creation via SAML - in conjunction with external_users_sync modifications)

import osimport saml2from settings import INSTALLED_APPSfrom settings import AUTHENTICATION_BACKENDSfrom settings import LOGIN_REQUIRED_URLS_EXCEPTIONSDEBUG = TrueADMINS = ( ("admin", "[email protected]"),)MANAGERS = ADMINSBASE_EMAIL_URL = 'http://cloudbolt.hhgttg.com'TIME_ZONE = 'America/New_York'#ENQUEUE_JOBS = True#SAMLINSTALLED_APPS += ('djangosaml2',)AUTHENTICATION_BACKENDS += ('djangosaml2.backends2.CustomSaml2Backend',)LOGIN_REQUIRED_URLS_EXCEPTIONS += (r'^/saml2/',)#OPTIONAL: ALL LOGINS FROM AD REQUIRE SAMLLOGIN_URL = '/saml2/login/'SAML2_DIR = os.path.join('/var/opt/cloudbolt/proserv/saml2')SAML_CONFIG = { 'xmlsec_binary': '/usr/bin/xmlsec1', 'entityid': 'https://cloudbolt.hhgttg.com/saml2/metadata/', 'service': { 'sp': { 'allow_unsolicited': True, #must be true 'name': 'CloudBolt SP', 'want_assertions_signed': True, #assertion signing (default=True) 'want_response_signed': True, #is response signing required 'name_id_format': None, 'endpoints': { 'assertion_consumer_service': [ ('https://cloudbolt.hhgttg.com/saml2/acs/', saml2.BINDING_HTTP_POST), ], #url to where SSO needs to POST the authentication data 'single_logout_service': [ ('https://cloudbolt.hhgttg.com/saml2/ls/', saml2.BINDING_HTTP_REDIRECT), ], #url to where sso needs to redirect on logout }, }, }, 'debug': 1, #send saml logs to application log in debug mode 'encryption_keypairs': [{ 'key_file': '/etc/pki/tls/private/cloudbolt.hhgttg.com.key', 'cert_file': '/etc/pki/tls/certs/cloudbolt.hhgttg.com.crt' }], #this controls response encryption (if required by your customer or IdP) 'key_file': os.path.join(SAML2_DIR, 'saml.key'), # private part 'cert_file': os.path.join(SAML2_DIR, 'saml.crt'), # public part 'allow_unknown_attributes': True, #allow extra attributes in response 'attribute_map_dir': os.path.join(SAML2_DIR, 'attribute-maps'), 'metadata': { 'local': [os.path.join(SAML2_DIR, 'remote_metadata.xml')], }, #this is the metadata file from the IdP 'contact_person': [{ 'given_name': 'Douglas', 'sur_name': 'Adams', 'company': 'HHGTTG', 'email_address': '[email protected]', 'contact_type': 'technical' }], #contact data - not integral to authentication 'organization': { 'name': 'hhgttg.com', 'display_name': 'The HitchHikers Guide To The Galaxy', 'url': 'http://www.somewhere.com', }, #contact data - not integral to authentication 'valid_for': 42, #how long is this configuration information valid for (hours)}#what attribute should we create the username with: username is typically usedSAML_DJANGO_USER_MAIN_ATTRIBUTE = 'username' SAML_CREATE_UNKNOWN_USER = True #create users thatdon'texist: True | FalseMS_CLAIMS = 'http://schemas.microsoft.com/ws/2008/06/identity/claims/'ORG_WS_CLAIMS = 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/'SAML_ATTRIBUTE_MAPPING = { ORG_WS_CLAIMS + 'emailaddress': ('email', ), MS_CLAIMS + 'windowsaccountname': ('username', ), ORG_WS_CLAIMS + 'surname': ('last_name', ), ORG_WS_CLAIMS + 'givenname': ('first_name', ),}#upnisn'tin this list and doesnt need to be

Abbreviations

IdP

Identity Provider: Server that provides authentication data to CloudBolt

SP

Service Provider (CloudBolt)

LDAP

Lightweight Directory Access Protocol (Enterprise Directory for User data, commonly implemented as Microsoft Active Directory)

SAML2

Security Assertion Markup Language version 2 : the open-source backend protocol that makes Single Sign-on possible between an IdP and an SP

UPN

User Principal Name: The username for a particular LDAP domain in the following format username@ldapdomain

DESCRIPTION

The Puppet Enterprise action "Puppet_ent_2015.3_auto_install_agent" fails with the following error when a puppet agent run is already active.

>>

Command returned stdout: [Notice: Run of Puppet configuration client already in progress; skipping (C:/ProgramData/PuppetLabs/puppet/cache/state/agent_catalog_run.lock exists)

>>

Reference TT# 7046 - https://cloudboltsoftware.zendesk.com/agent/tickets/7046

OBSERVATION

The action "Puppet_ent_2015.3_auto_install_agent" was failing intermittently because an existing puppet agent run (puppet agent -t) was in progress.

RESOLUTION

The action "Puppet_ent_2015.3_auto_install_agent" was updated to not trigger a puppet agent run (puppet agent -t) during the agent installation process as a workaround.

Specifically,

Lines 130-143 in Puppet_ent_2015.3_auto_install were commented out. This omission essentially removes a triple check.

In other words, a typical installation would go as follows:

Agent gets installed

Certificate signed

Agent run forced (commented out too, but is scheduled to run on the server within 30 mins]

Additional modifications

Puppet_Puppet_ent_2015.3_auto_install_agent action was updated with the latest code in master that will be part of CB 8.8

2. The auto install action was changed to use Polycoms internal agent download from the Puppet Master

3.Timeout was increased to 600 seconds on all server.execute script calls

At times it is necessary to store encrypted credentials, which might be used by scripts and hooks within CloudBolt.

These objects can be defined in CloudBolt under Admin > Connection Info (under Resources & Networks). By clicking the 'New Connection Info' button you can create your own instances that are encrypted and stored in thedatabase.

Once you've created a Connection Info object, you can get it in your script with something like:

```

from utilities.models import ConnectionInfo

conn = ConnectionInfo.objects.get(name="THE NAME OF YOUR INSTANCE HERE")

```

Connection Info consists of IP/hostname, port, protocol, username, password and SSH Key (if it resides in CB) as details for connections to systems outside of CloudBolt. It might be used to securely store credentials for custom actions. Also used internally by CloudBolt for a number of connections, such as load balancers and accessing remote action source code.

Problem

When building a server from CloudBolt, VM customizations such as hostname, network, etc. don't seem to be applied to my new VM.

Solution

First, make sure you have the latest VMware Tools installed on your template. You'll need the official VMware version -- not the open source version.

If you're running the right version of VMware Tools in your template, take a look at the customization support matrix from VMware to make sure your version of vCenter and update supports the OS you're trying to install:

http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

In some environments, it may be necessary to provide CloudBolt with a vCenter user account following the principle of least privilege. In those cases, the vSphere admin will need to provision an account for CloudBolt using a role with a specific set of privileges.

Below is a listing of the minimum required privileges for CloudBolt to function in vCenter:

Datastore.AllocateSpace

Datastore.Browse

Datastore.Config

Datastore.DeleteFile

Datastore.FileManagement

Datastore.UpdateVirtualMachineFiles

Datastore.UpdateVirtualMachineMetadata

Folder.Create

Folder.Delete

Folder.Move

Folder.Rename

Network.Assign

Resource.ApplyRecommendation

Resource.AssignVMToPool

Resource.QueryVMotion

StoragePod.Config

System.Anonymous

System.Read

System.View

Task.Create

Task.Update

VApp.ApplicationConfig

VApp.AssignResourcePool

VApp.AssignVApp

VApp.AssignVM

VApp.Clone

VApp.Create

VApp.Delete

VApp.Export

VApp.ExtractOvfEnvironment

VApp.Import

VApp.InstanceConfig

VApp.ManagedByConfig

VApp.Move

VApp.PowerOff

VApp.PowerOn

VApp.Rename

VApp.ResourceConfig

VApp.Suspend

VApp.Unregister

VirtualMachine.Config.AddExistingDisk

VirtualMachine.Config.AddNewDisk

VirtualMachine.Config.AddRemoveDevice

VirtualMachine.Config.AdvancedConfig

VirtualMachine.Config.Annotation

VirtualMachine.Config.ChangeTracking

VirtualMachine.Config.CPUCount

VirtualMachine.Config.DiskExtend

VirtualMachine.Config.DiskLease

VirtualMachine.Config.EditDevice

VirtualMachine.Config.HostUSBDevice

VirtualMachine.Config.ManagedBy

VirtualMachine.Config.Memory

VirtualMachine.Config.MksControl

VirtualMachine.Config.QueryFTCompatibility

VirtualMachine.Config.QueryUnownedFiles

VirtualMachine.Config.RawDevice

VirtualMachine.Config.ReloadFromPath

VirtualMachine.Config.RemoveDisk

VirtualMachine.Config.Rename

VirtualMachine.Config.ResetGuestInfo

VirtualMachine.Config.Resource

VirtualMachine.Config.Settings

VirtualMachine.Config.SwapPlacement

VirtualMachine.Config.Unlock

VirtualMachine.Config.UpgradeVirtualHardware

VirtualMachine.GuestOperations.Execute

VirtualMachine.GuestOperations.Modify

VirtualMachine.GuestOperations.Query

VirtualMachine.Interact.DeviceConnection

VirtualMachine.Interact.PowerOff

VirtualMachine.Interact.PowerOn

VirtualMachine.Interact.Reset

VirtualMachine.Interact.ConsoleInteract

VirtualMachine.Interact.SetCDMedia

VirtualMachine.Interact.SetFloppyMedia

VirtualMachine.Inventory.Create

VirtualMachine.Inventory.CreateFromExisting

VirtualMachine.Inventory.Delete

VirtualMachine.Inventory.Move

VirtualMachine.Inventory.Register

VirtualMachine.Inventory.Unregister

VirtualMachine.Provisioning.Clone

VirtualMachine.Provisioning.CloneTemplate

VirtualMachine.Provisioning.CreateTemplateFromVM

VirtualMachine.Provisioning.Customize

VirtualMachine.Provisioning.DeployTemplate

VirtualMachine.Provisioning.DiskRandomAccess

VirtualMachine.Provisioning.DiskRandomRead

VirtualMachine.Provisioning.GetVmFiles

VirtualMachine.Provisioning.MarkAsTemplate

VirtualMachine.Provisioning.MarkAsVM

VirtualMachine.Provisioning.ModifyCustSpecs

VirtualMachine.Provisioning.PromoteDisks

VirtualMachine.Provisioning.PutVmFiles

VirtualMachine.Provisioning.ReadCustSpecs

VirtualMachine.State.CreateSnapshot

VirtualMachine.State.RemoveSnapshot

VirtualMachine.State.RenameSnapshot

VirtualMachine.State.RevertToSnapshot

AWS EC2 and Hosted Chef (in-progress)

Requirements

A hosted chef account from http://chef.io

An AWS account

A RHEL/CentOS AMI

Setup

Create an AWS resource handler connection in CloudBolt.

This includes an Internet-accessible VPC.

Import an AMI for the region used

On your CB server, create the directory/var/opt/cloudbolt/resourcehandlers/aws/<instance_id>/; where <instance_id> is take from the URL for the Resource Handler configuration page for this resource handler, e.g. forhttps://your_cloudbolt_server/admin/resourcehandlers/1/, the <instance_id> is "1"

Upload your private .pem file to the directory created in the previous script. This is case sensitive, so if the key name is MyKeY, then the .pem file should be MyKeY.pem.

Create a Configuration Management connection to Hosted Chef in CloudBolt.

Create an organization in Hosted Chef and be sure to save the validator key as <organization_name>_validator.pem

Create an Configuration Manager Chef instance in CloudBolt under Admin > Configuration Managers > "Add a configuration manager".

Create /var/opt/cloudbolt/connectors/chef/<provider_id>/ where <provider_id> is taken from the URL after clicking your Chef Configuration Manager instance in CloudBolt. For instance in this url:https://your_cloudbolt_instance/providers/1; the provider_id = 1.

Be sure to download the private key for your Hosted Chef user account and save it as <username>.pem to /var/opt/cloudbolt/connectors/chef/<provider_id>/<username>.pem

From Hosted Chef, select the organization in the list of organizations and click "Generate Knife Config" to download the knife config to your local computer. Copy this file (knife.rb) to/var/opt/cloudbolt/connectors/chef/<provider_id>/.

Copy <organization_name>_validator.pem to/var/opt/cloudbolt/connectors/chef/<provider_id>/

Complete the install of knife per instructions provided by Chef. There's a link to these instructions in the CloudBolt documentation for Chef setup.

Import any cookbooks or roles being used from Hosted Chef.

Create an environment for the above resource handler.

Set the configuration manager to the chef instance created above

Set the AWS Region and select the appropriate VPC

Add an OS Build (AMI) that was imported via the Resource Handler configuration

Be sure the availability zone is set the zone containing your VPC

Upload sleep orchestration hook attached to this article. By default this script sleeps for 3 minutes.

Link to this file is just below this text.

Add to the Pre-Network hook, give it any name,and bind it to the CloudBolt Environment defined in step 3.

If CloudBolt needs to wait longer for the EC2 instance to initialize before bootstrapping the Chef client, edit line 6 of sleep.py.

Provision a server -- you should see an Applications select list at the bottom of your server configuration.

Questions or comments? Leave them here!

Problem

The following error message is displayed after attempting to open a console to a server from CloudBolt C2:

Attempt to find a free TCP port from CloudBolt to ESX server <ESX HOST> took too long. There is likely a firewall between CloudBolt and ESX. Tried ports 7004-7009.

Solution

Log into the vCenter web application that your VM resides.

Navigate to the summary pane for your VM in vCenter and note the ESXI Host listed in the host category.

Navigate to the configuration for the ESXi Host name found from step 2.

Select the "Manage" tab, select "Settings",and then select the "Security Profile"listed under the System dropdown.

Open the Firewall properties with the "Edit"button.

Check to enable "VM serial port connected over network".

If the Error Still Occurs

ssh to the CloudBolt server

telnet <ESX host IP> 7000

Repeat step 2 for each of your ESX hosts.

If you get a connection timeout when trying to do that, then this is the problem that CB is seeing and there's either a FW on ESX or between CB and ESX.

If instead you get a connection refused, then (in all likelyhood) CB can actually reach the ESX server and ESX is responding that that port is not open, which is okay. If this is the case, CB should not be returning that error message.

VITAL INFORMATION

The following is a collection of content hosted in the CloudBolt Knowledgebase or Docs dealing with common requests and issues.

CloudBolt Release Notes (latest GA version)

CloudBolt Docs (latest GA version)

Knowledgebase Articles

Submit a Support Ticket

List of My Support Tickets

EMERGENT ISSUES

Please see below for the most up to date list of emergent issues that are actively under investigation by CloudBolt. When resolved, they will be mentioned in Recently Resolved Issues (below) and will appear in our published Release Notes.

Upgrade to 8.2 fails if using a custom database name

Ordering nested BluePrint from API does not include sub-blueprint items. Bug#158613750

Interactive Servers Report table does not update correctly as above chart is filtered

Sub-Blueprints are not being passed in the context for blueprint actions

Radius authentication dictionary has issues with Unicode keys

Potential memory leak with DEBUG

Issues with Workarounds

Celery Processes fail to start jobs:We are aware of an issue in CloudBolt 7.7 versions where the Job engine runs out of memory and its workers hang. To resolve this restart the celeryd service and workers completely.

Global Search Fails to Find Servers: We are aware of an issue in CloudBolt 7.4 where searches (using the global search tool in the toolbar) fails to find servers when a partial hostname is entered, but that searches for a full hostname work. Instead, use the Servers list page and search for your servers there. #151526469

RECENTLY RESOLVED ISSUES

These issues have been resolved in the listed version of the product. They are also mentioned in our published Release Notes.

Issues Resolved in 7.7:

7.7 Release Notes

Issues Resolved in 7.6.0.1:

7.6.0.1 Release Notes

Appliance redundancy

Because CloudBolt is easily installed and upgraded in a matter of minutes, the need for appliance redundancy is solely driven by the high availability requirements at the company where it's deployed. While it's possible to run CloudBolt's database outside of the appliance in a DB Cluster and have multiple instances of the framework server accessed through a load balancer, the level of complexity to create and maintain such setup makes it less than desirable for the majority of the customers. For customers looking for simple strategies for appliance redundancy, the suggestions are:

Hot Stand-by

Install a secondary CloudBolt appliance.

Turn off cron-based jobs.

Setup rsync to replicate:

the data in the database from the primary appliance.

the contents of /var/opt/cloudbolt

the contents of /var/www/html/cloudbolt/static/uploads

At the end of every synchronization run collectstatic on the secondary appliance

Every time the primary appliance is upgraded, also upgrade the backup.

Cold Stand-by

Install a secondary CloudBolt appliance.

Leave it powered-off, might have it configured with same IP as the primary

Save a copy of the latest upgrader package run against the primary appliance

*A variation of the cold stand-by strategy is to have no stand-by at all, and instead re-deploy from the OFV if needed.

Data Backup (redundant if using a hot stand-by)

While its trivial to create the primary appliance from any of the methods described in the previous section, one should make sure than can also restore to the new appliance all their data, extensions, hook modules, license file and any other configurations necessary. Data and configuration restoration is also very simple if you make sure to back-up all three main areas of CloudBolt. The suggested process is to run a daily script in the appliance that creates a tarball and rsyncs it to a remote location.

The tarball should contain:

A database dump - mysqldump

#mysqldump --user=root --password=[your_db_password] cloudbolt > ~/dump-[date/name].sql

A copy of the contents of /var/opt/cloudbolt/

A copy of the contents of /var/www/html/cloudbolt/static/uploads

Restore Instructions

Power on the cold stand-by mentioned on the first section, or re-deploy from OVA

(If necessary) Download/run upgrader to bring the c2 version current

Reset db and load from backup of mysqldbdump

#/opt/cloudbolt/manage.py reset_db

#mysql --user=root --password=[your_db_password] cloudbolt < [your_backup_file]

Overwrite /var/opt/cloudbolt with contents from backup tarball

Overwrite /var/www/html/cloudbolt/static/uploads contents from backup tarbal

Run /opt/cloudbolt/manage.py collectstatic

Restart httpd

Sometimes, when upgrading to a newer release of CloudBolt, This error occurs:

Restarting rabbitmq-server: RabbitMQ is not running

This usually means one of two things, either the RabbitMQ port is blocked by the firewall, or the DNS server can't resolve the hostname 'cloudbolt'. The latter is the more common of the two.

To check the RabbitMQ port run this command:

epmd -debug

Make sure the port number is 4369.

To fix the DNS server not resolving the hostname edit the '/etc/hosts' file and add 'cloudbolt' to the list of hostnames so that the server sees it as 127.0.0.1

The upgrader should run without any errors now.

CloudBolt supports running scripts on remote servers, including Windows PowerShell scripts.

Symptoms:

During the process of running the remote script, the job fails with an error similar to:

Process 1048 completed on guest with error:

(vim.vm.guest.ProcessManager.ProcessInfo) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

name = 'powershell.exe',

pid = 1048L,

owner = 'Administrator',

cmdLine = '"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" C:\\Windows\\Temp\\03cc3978aafa493881a6c75ec1fef19c.ps1 10.50.22.230 > C:\\Windows\\Temp\\c2_stdout_tmp 2> C:\\Windows\\Temp\\c2_stderr_tmp',

startTime = 2015-03-26T18:53:36Z,

endTime = 2015-03-26T18:53:45Z,

exitCode = 1

}

Solutions:

There are a myriad of causes for this error. The best first step in troubleshooting is to get console on your Windows server and run the command exactly as seen in the error output where you see "cmdLine = ". In the above case that would mean opening a PowerShell prompt and running this (notice that the surrounding single quotes have been removed, but the double quotes stay):

"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" C:\\Windows\\Temp\\03cc3978aafa493881a6c75ec1fef19c.ps1 10.50.22.230 > C:\\Windows\\Temp\\c2_stdout_tmp 2> C:\\Windows\\Temp\\c2_stderr_tmp

One solution to consider is setting the Execution Policy in the Windows template to "Unrestricted". This can be done in PowerShell with `Set-ExecutionPolicy Unrestricted`.

The following roles should be attached to a group to which user accounts representingCB service accounts are added. A key and secret can then be generated for each user account.

Predefined Roles required:

AmazonEC2FullAccessBilling

Inline Roles

{

"Version": "2012-10-17",

"Statement": [{

"Effect": "Allow",

"Action": [

"ce:*"

],

"Resource": "*"

}]

}

Note: CloudBolt 7.3 introduces a number of features that allowscustomizing the overall branding of CloudBolt through the UI using the Branded Portals.Images and colors set on the Portal will alter how the site looks to end users connecting to that Portal, and the name used for the Portal will be used in place of CloudBolt throughout the product. We recommend upgrading to the latest version of CloudBolt to get these features.

Customizing CloudBolt via the Branded Portals is available in theAdmin under Users & Settings ->Branding and Multi-Tenant Portals.

Manual Customizations

The look and feel of CloudBolt can becustomizedto varying degrees, ranging from changing the menu colors to match yourcompany's branding, toadding a banner tothe header, towhite-labeling the entire product and removing all references to 'CloudBolt'. The following guide will explain how to makethesechanges, as well as any number of furthercustomizations. For manyof these tasks, you will need SFTP and SSH access to your CloudBoltserver. In general, changes should always be made in the /var/opt/cloudbolt/proserv/ directory to avoid getting overwritten when CloudBolt is upgraded.

CloudBolt Logos

Several logos used throughout CloudBolt can be customized per Portal via the UI. Edit the portal to beused, and fields are available to set images for the logo shown in the topmenu, the footer logo, the loading icon, and a custom banner to be displayed above the topmenu.

Replacing other logos can be done with two steps: first, copy a new logo to your CloudBolt server and place itin the appropriate subdirectory within /var/opt/cloudbolt/proserv/static/, and then collect those new files to be served by CloudBolt by logging in via SSH and executing the command:

/opt/cloudbolt/manage.py collectstatic --noinput

Login Page & About PageLogo

The login page and the window that opens to show version and build info both include a CloudBolt logo. To replace this logo, use an SVG-formatted version of your logo that is dark-colored and displays well at a height of 48px. Copy your logo to this location on the CloudBolt server:

/var/opt/cloudbolt/proserv/static/logos/cb-horizontal.svg, and execute the command:

/opt/cloudbolt/manage.py collectstatic --noinput

Favicon

Many browsers display an icon next to the web address and when bookmarked. To replace the icon, use an ICO-formattedfile at least 16px squared. Copy your logo to this location on the CloudBolt server:

/var/opt/cloudbolt/proserv/static/img/favicon.ico, and execute the command:

/opt/cloudbolt/manage.py collectstatic --noinput

Note: on CloudBolt 7.7.1 and older, the file name needs to beCloudBolt.ico, but we recommend adding both to maintain future compatibility.

Branding and Colors

Most of thecolors used throughout CloudBolt can be customized through the UI. Those settings are available in the Admin under Users & Settings ->Branding. Edit the portal being used, and fields are available to set colors for text, background, menus, and tooltips.

Manual Customizations

For more stylechanges that can't be made via the Portal Branding or by overwriting images, a customer css file can be created that allows for any customizations.

mkdir -p /var/opt/cloudbolt/proserv/static/css

vim /var/opt/cloudbolt/proserv/static/css/customer.css

Add whatever css customizations are necessary, then run:

/opt/cloudbolt/manage.py collectstatic --noinput

For example, if you want to replace one of the SVG logos like the header imagementionedabove with a different image type, you can add the following to customer.css:

.nav-header-tabs li[data-topnav="dashboard"] a { background-image: url(/static/logos/my-custom-header-logo.png); }

Or another example, if you want to completely hide the logo in the footer instead of replace it, you could include the following:

html body #footer img { // Hide the original logo which is in an img tag that we cannot control with CSS display:none !important; }

Completely hide the footer:

html body #footer { display: none !important; }

Hide a link in the Docs menu:

/* Hide the 'Create a Support Ticket' menu item */ li[data-topnav=help] .dropdown-menu li:last-child { display: none; }

Completely hide the Docs menu:

/* Hide the entire 'Docs' menu */ li[data-topnav=help] { display: none; }

Change the loading icon, aka spinner:

html body .spinner { background: url('../logos/your-own.png') no-repeat 0 0 !important; }

Templates and Email

Full templates can also be overwritten. Any template can be copied from its location in /opt/cloudbolt/templates/ and put in a matching path within /var/opt/cloudbolt/proserv/templates/. For example, to make changes to the login template:

mkdir /var/opt/cloudbolt/proserv/templates/registration/

cp /opt/cloudbolt/templates/registration/login.html /var/opt/cloudbolt/proserv/templates/registration/

vim /var/opt/cloudbolt/proserv/templates/registration/login.html

(edit as desired)

service httpd restart

Another example, to customize the copyright statement on the About page (accessed by clicking the logo in the footer),execute the following commands:

cp /opt/cloudbolt/templates/about.html /var/opt/cloudbolt/proserv/templates/

vim /var/opt/cloudbolt/proserv/templates/about.html

(edit as desired)

service httpd restart

To customize the content ofemails that CloudBolt sends, copy any file in /opt/cloudbolt/templates/email/ to /var/opt/cloudbolt/proserv/templates/email/ (making the directory if necessary). Then, make your desired changes to the new locationand call service httpd restart.

Undoing Customizations

Colors customized using the Portal settings can be undone via the link at the bottom of the Portal's edit page.

To undo customizations made by placing files in the `/var/opt/cloudbolt/proserv/` directory, remove the files and execute the following commands on your CloudBolt server:

/opt/cloudbolt/manage.py collectstatic --noinput

service httpd restart

Inquiring minds here at CloudBolt want to know:

How does your company/organization monitor its deployed compute resources?

What tools do you use?

Is there a single monitoring system or multiple?

How do users access monitoring data for their deployed servers/resources?

Any information you can provide would be very helpful as we continue to craft the roadmap for CloudBolt 8.0 and beyond.

Join Windows Host to AD Domain on Provision

Since CloudBolt utilizes the guest customization wizard provided by VMware, Windows guests on vCenter can be automatically joined to an AD domain when provisioned. In addition to creating the machine account, the guest will also be registered with the AD DNS host.

What You'll Need

AD service account credentials. This account should be able to register machine accounts.

A Windows vCenter template. This template MUST include theinstalledVMware Tools.

Joining the Domain

Under Admin > LDAP Authentication Settings, click "New LDAP Utility" to create an instance connecting CloudBolt to your AD server.

For any vCenter-backed environment that will provided Windows servers, add the parameter titled "Domain to Join (LDAPUtility)".

After the parameter is added, click the "pencil" icon next to the text "Unconstrained" to open the parameter options dialog.

At the bottom of the dialog, select the AD domain you linked to in step one and click "Add option".

Now, when provisioning Windows servers in the selected environment, the vCenter will attempt to join the Windows server to the domain.

Be sure to let the VMware Tools report that they are complete. This usually includes a couple automatic reboots. To test that the procedure worked, simply lookup your guest hostname in DNS.

NOTE: This document refers to the Job Engine included in CloudBolt versions before 7.7. To troubleshoot newer versions of the Job Engine, refer to the documentation here: http://docs.cloudbolt.io/job-engine.html#how-do-i-troubleshoot-my-job-engine

Failures in the CloudBolt Job Engine are rare, and it is designed to be resilient against most types of provider-specific failures. However, user-written actions have the potential to cause the job engine to have issuesand you may need to troubleshoot these issues.

What is the normal progression of job statuses?

PENDING

QUEUED

RUNNING

If canceled:

TO_CANCEL

CANCELED

If job completes: (for actions, these get determined by what is returned by the action)

SUCCESS -or- WARNING -or- FAILURE

How does the job enginerun and determine jobs to run?

Aroot cron job that runs once a minute which...

Note: It runs every minute to prevent memory leaks fromcustom actions from breaking the job engine, and so that it re-runs on a frequent enough basis be barely noticeable by the end-user.

Runs runjobs.sh, which checks for locks, then...

Runs runjobs.py

That picks jobs up from PENDING status and runs them (see next section for details)

When there are no more jobs to run and it is within five seconds of the next minute, runjobs.py exits, then runjobs.sh exits

How does each job get processed by the job engine?

The UI creates a PENDINGJob

runjobs.py finds it, QUEUEDsit

runjobs.py spawns new thread, changes job toRUNNING

job.run() executes, writes progress back to CloudBolt

(If the 'Cancel Job' button is clicked in the UI at this point, the job is set to TO_CANCEL, an Exception is raised inside the job's thread, and then the job is set to CANCELED.)

run() completes, returns results to runjobs.py

Job status gets set to SUCCESS, WARNING, orFAILUREdepending on the return from run()

runjobs.py writes the results to CloudBolt

What can go wrong with myjob engine?

It could be failing to start up (jobs never go from pending queued)

e.g. If you have disabled or modified the cron job on your CloudBolt server

Possibly if there are problems with the runjobs.sh, runjobs.py scripts (and you can run them interactively after disabling the cron job to test this)

There could be problems in the underlying database (and running runjobs.sh interactively would also help troubleshoot this)

A job can run infinitely, then the job engine will never exit

This can be caused by infinite loops or deadlocking issues within your scripts or jobs

The job engine could get in a bad state where it changes jobs pending queued, but never runs them

This can also be a byproduct of deadlock or other hung jobs

A job could be spawning more threads, and never cleaning them up (causing the job engine to queue, but not run new jobs)

How do I troubleshoot myjob engine?

Answer the following questions to understand the scope of your issues:

Which job or order cause you to notice theproblem? Note its job number, status, and URL.

Which related jobs are having problems? Note their job numbers, status, and URLs.

Which job(s) started running most recently? Note their job numbers, status, and URLs.

Have other jobs started running after the jobs you noted above?

Are some jobs in a different status from others? Note this and review the above descriptions of the sequence of how the job engine runs to understand whichjob was the last job to run.

Are your jobs in a PENDING state? This indicates that runjobs.py has not restarted recently. Wait two minutesfor the job engine to restart and pick up the job. If it does not begin running, refresh the page. If the job is still not running, check whether the job engine is still running (see next section) other jobs.

Are your jobs in a QUEUED state? This indicates the job engine has identified it should run a job but has not started running it yet. Check whether the job engine is still running (see next section).

Are your jobs in a RUNNING state? This indicates the job engine has started running the job, and is either still running the job correctly or the job engine has crashed. Check whether the job engine is still running (see next section). If the job is still running, review the job's log to see what its last step was and whether there are any error messages.

Are your jobs in a TO_CANCEL state? This indicates the job has been marked for cancelation and the job engine will recognize this soon and cancel the job. If the job never enters the CANCELED state, check whether the job engine is still running (see next section).

Are your jobs in a CANCELED state? This indicates the job engine has canceled those jobs successfully, and should be available to run future jobs. Try running a new jobif it runscorrectly, there is no problem: your job engine is behaving as designed.

Are your jobs in a FAILURE, WARNING, or SUCCESS state? Your jobs completed running and returned this status, so the job engine is working properly. Troubleshoot the underlying job for any failures you are seeing.

If the above steps do not help, create a support ticket and describe which of the above steps you took as well as the specific job numbers (with yourexpected and their actual status for each), and attach your application.log, jobengine.log, and each job's individual /var/log/cloudbolt/jobs/12345.log file. Provide screenshots of how the jobs appear in the UI, as well as the output from ps -ef | grep runjobs.

How do I check whether the job engine is running?

Run ps -ef | grep runjobs and look to see how long runjobs has been running.

If runjobs is running:

Look for QUEUED, TO_CANCEL, or RUNNING jobs in CloudBolt. These jobs should be run by the current runjobs instance.

If runjobs has been running for a long time, look at the log output from the jobs from the previous step to see if one of them has hung.

Review the /var/log/cloudbolt/jobengine.log to see which jobs the job engine has picked up and whether any error messages are present.

If runjobs is not running:

tail -f /var/log/cloudbolt/jobengine.log and wait twominutes to see if the cron job runs and outputs messages to this log file.

No runjobs process means the cron job has not kicked off yet, the cron job failed to run, or the cron job is disabled/missing.

Check your cron jobs, try running runjobs.sh manually, and check the jobengine.log file for relevant errors.

CloudBolt is aiming to supportPython 3 by version 8.0, and any custom Plug-ins or Actions must also be updated before upgrading. Here is a brief guide to updating your Python code to be compatible with both Python 2 and 3.

One of the easiest tools to automatically check your plugin compatibility is 2to3. Run it on the source files of your Plug-ins, and a diff against the original source file is printed.

MostPlug-ins andActions won't be usingmethods or syntax that have significant changes. But there are a few common differences that you should review.

Print statements need to be a method:

http://python-future.org/compatible_idioms.html#strings-and-bytes

Raising exceptions should also use a method to set a specific error message:

Catchingexceptions needto use 'as' instead of a comma:

Unicode changes will impact how data is read from external sources like the 'requests' library:

Python 2

Python 3

r = requests.get( https://google.com )

content = r.content()

r = requests.get( https://google.com )

content = r.content.decode()

String formatting has changed, and the old % formatting is no longer allowed:

Python 2

Python 3

%d %s % (i, s)

{} {}.format(i, s)

%d/%d=%f % (355, 113, 355/113)

{:d}/{:d}={:f}.format(355, 113, 355/113)

Here are a fewexcellent resources on the topic that can provide more detailed information:

https://docs.djangoproject.com/en/1.11/topics/python3/

https://docs.python.org/3/howto/pyporting.html#pyporting-howto

If you have any trouble updating your code, please review those resource and then contact CloudBolt support at [email protected].

Unlike earlier versions of Ubuntu, VMware does not fully support customization of Ubuntu version 14.x. One symptom you may see is a failure during provisioning such as:

Hostname not set to expected value after 240 seconds.Expected: bonds-web-serverActual: ubuntu

CloudBolt however, can help you solve this problem. In order for the OS to recognize that its hostname has changed, the virtual machine must be rebooted. CloudBolt ships with an orchestration hook that allows you to do this as part of the provisioning process. To enable it in your CloudBolt environment:

Navigate in the CloudBolt UI from the C2 Admin page to the Orchestration Hooks list page.

Click on the hook titled "Reboot ubuntu servers" to go to the detail page for this hook.

Click "Edit" then check the "Enable" box.

To learn more about writing CloudBolt Orchestration Hooks, see the Job Hook Development section of our Admin Manual or contact our support team directly.

A customer recently asked: How can I control the size and number of backups of my /var/log/cloudbolt/application.log file?

By default the logging configuration is set in settings.py. Since this file gets overwritten with each update, we want to override this configuration in /var/opt/cloudbolt/proserv/customer_settings.py -- a file that is preserved across CloudBolt updates.

To override the default logging settings for the application.log file and jobengine.log you can add the following to your customer_settings.py file:

from settings import LOGGING

LOGGING['handlers']['logfile']['maxBytes'] = 2^20 * 10 # 10 Megabytes

LOGGING['handlers']['logfile']['backupCount'] = 10 # Maintain 10 backups

LOGGING['handlers']['jobengine_logfile']['maxBytes'] = 2^20 * 10 # 10 Megabytes

LOGGING['handlers']['jobengine_logfile']['backupCount'] = 10 # Maintain 10 backups

This bumps the max size for each log from 5MB to 10MB and retains 10 backups. You can adjust the size and number of backups to suit your environment.

When provisioning vanilla Windows marketplace templates via CloudBolt, WinRM and the firewall policyprevent remote execution by default. This presents a 'chicken and the egg' scenario where in order to enable remote scripting against the VM, it must first be accessed via RDP and configured. When deploying VMs at scale, this solution becomes untenable.

In order to workaround this issue, a Cloudbolt plugin can install a VM Extension, which runs a Powershell script on Azure VMs using the Azure VM Agent.Azure supports anumber of VM Extensionswhich can be launched at provision time or after the VM is running, enabling anything from launching a simple Powershell script on the VM toinstalling antivirus software. More info about VM extensions can be found here.

In order to enable WinRM on a newly provisioned VM in CloudBolt, we can use a Post Provision hook to install a VM Extension. Although a different Powershell script might be required for any specific use case, the commands we'll use are stored in github here and are as follows:

winrm quickconfig -q

winrm set winrm/config/service '@{AllowUnencrypted="true"}'

winrm set winrm/config/service/auth '@{Basic="true"}'

Start-Service WinRM set-service WinRM -StartupType Automatic

Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled falseSimply put, the above powershell commands enable WinRM, ensures it is started automatically and completely disables the Windows firewall. However, you may use any Powershell script you wish as long asit enablesthe same configuration. Additionally, you may decide to store your script in Azure storage. For more information about uploading your script to an Azure storage account, as opposed to github, see here.

During Step 3 (Settings: Configure optional features) ofcreating a new VM in the Azure portal, the 'Extensions' feature allows for the addition of an extension to the VM. See below:

For this example, we'll leverage the 'Custom Script Extension' extension published by Microsoft. When selected in the portal it will prompt for a file to upload, but we'll be calling a script stored in github and installing via a CloudBolt plugin (attached to this KB article).

The attached python scriptshould be uploaded as a CloudBolt plugin under Server Actions and tested (as a Server Action) before including it as a Post Provisioning (trigger point 10) Orchestration Action. If deployed as a Post Provisioning Orchestration action, it should occur first in the list if any subsequent scripts require remote execution.

Once imported into your environment, ensure that allAction Input types are set to 'String', except the 'Settings' Action Input which should be set to type: 'Code'. Also ensure that its Resource Technology is set specifically to Azure ARM and that is it Shared:

Now set some defaults on the Action Inputs (Admin -> Server Actions, filter down to your Action:

Extension Name: CustomScriptExtensionProtected Settings: (Leave this blank)Publisher: Microsoft.ComputeSettings: {"fileUris": ["https://raw.githubusercontent.com/CloudBoltSoftware/cloudbolt-forge/master/actions/remote_scripts/enable_winRM.ps1"],"commandToExecute": "powershell.exe -ExecutionPolicy Unrestricted -File enable_winRM.ps1"}Version: 1.9

See below:

Now test this Server Action against an applicable server under CloudBolt management:

To verify that the plugin properly installs the VM extension, check the 'Extensions' section in the Azure portal for that particular VM. The Status column should say 'Provisioning succeeded':

You can get more details by double-clicking the Extension:

Note that the installation process is asynchronous. The plugin includes a four minute (240 second) sleep timer in order to allow the installation to complete. This is important to remember when deploying this plugin as an Orchestration Action which might include remote scripts after it that depend on remote execution. The sleep timer can be increased or reduced depending on your use case.

If the VM extension is installed successfully via the plugin, Ad-Hoc scripts as well as Remote Execution via CloudBolt will be enabled on the VM.

Note: In CloudBolt 7.5, emails are managed via the Email Templates available at Admin -> Email Settings. To customize an existing email template, click the pencil icon to the right of the template being used, and edit the Body field.

It is possible in CloudBolt to customize the contents of a particular email that CloudBolt sends out. To do so, ssh to the CB server as root, copy the version of the template shipped with CB to a customer-specific directory (which will not be rewritten when CB is upgraded), and modify the file.

Here are the specific commands to run to override the email that CB sends when an order is complete:

mkdir /var/opt/cloudbolt/proserv/templates/email/cp /opt/cloudbolt/templates/email/order_complete-body.template /var/opt/cloudbolt/proserv/templates/email/vi /var/opt/cloudbolt/proserv/templates/email/order_complete-body.template

In the contents of the file, if you want to remove the rate for example, you would change this block:

{% for server in order.prov_server_list %} * Hostname: {{ server.get_vm_name }}{% if server.ip %} Primary IP: {{ server.ip }} {% endif %}{% if server.rate %} Rate: {{ server.rate_display }}{% endif %}{% endfor%}

To this:

{% for server in order.prov_server_list %} * Hostname: {{ server.get_vm_name }}{% if server.ip %} Primary IP: {{ server.ip }} {% endif %}{% endfor%}

Issue: When you try to execute Chef Bootstrap on 2008 R2, sometimes it takes a bit longer to finish the execution.

Cause: This could be due to the amount of memory allocated for WinRM shells.

Solution:Apply patch that Microsoft suggests from this link https://support.microsoft.com/en-us/help/2842230/-out-of-memory-error-on-a-computer-that-has-a-customized-maxmemorypershellmb-quota-set-and-has-wmf-3.0-installed

CB Admin can remove an environment any time without damaging or disrupting the existing servers, which locate in the environment. This could be done by associating the servers with an unassigned environment.

CB error while attempting to remove an environment with active servers looks like this:

Error: Specific error - Environment "US East (Northern Virginia) [us-east-1] ENV-A-1" cannot be deleted, because it has 19 active servers. They need to be decommissioned or moved to a different environment first.

Steps to be taken:

Click on CB Admin -> Environment

Select the Environment, which you want to remove and has active servers.

Click on server Tab

Select all servers

Click on pencil (Change Owner, Group and Environment)

In Environment selection select unassigned

7. Confirm

Now you can delete your Environment safely.

The following is a run-down of all the CloudBolt-related directory on the CloudBolt virtual appliance:

/opt/cloudbolt Base CloudBolt Install Upgrade Safe? NO Python Path? YES Should Replicate? NO

/var/log/cloudbolt CB application and job Logs Upgrade Safe? YES Python Path? NO Should Replicate? YES

/var/run/cloudbolt Lockfiles for VM sync and Job Engine Upgrade Safe? NO Python Path? NO Should Replicate? NO

/var/opt/cloudbolt/proserv Customer scripts, settings, overrides, graphics, url mappings and xui extensions. SAML2 configuration Upgrade Safe? YES Python Path? YES Should Replicate? YES

/var/opt/cloudbolt Application settings that are persisted across upgrades Upgrade Safe? YES Python Path? NO Should Replicate? YES

/var/www/html/cloudbolt Static web assets Upgrade Safe? NO Python Path? NO Should Replicate? NO

/var/www/html/cloudbolt/uploads Content uploaded to CloudBolt Server Upgrade Safe? YES Python Path? NO Should Replicate? YES

In order to bootstrap chef on a Windows server via WinRM,you'll need toinstall the knife-windows plugin on your CloudBolt instance so that knife willbe able to connect to a remote Windows server via the WinRM transport. More info about this plugin is availableat: https://github.com/chef/knife-windows.

You can install the pluginby using the the chefdk gem command:

$ /opt/chefdk/embedded/bin/gem install knife-windows

You can verify it is installed by running:

$ knife winrm --help

CloudBolt uses the native Logging facility for python. The documentation for this module is located at: https://docs.python.org/2/library/logging.html.

You'll see from this documentation that it's a hierarchical module-based logging system that allows you to not only control what classes are logged to what handlers, but also their log level.

To start overriding the default Logging configuration for CloudBolt, copy the LOGGING definition from /opt/cloudbolt/settings.py (approx lines 513 to 588) to /var/opt/cloudbolt/proserv/customer_settings.py. Here's an example that you could copy into customer_settings.py that also removes a few variables:

LOGGING = {

'version': 1,

'disable_existing_loggers': False,

'formatters': {

'standard': {

'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'

},

},

'handlers': {

'mail_admins': {

'level': 'ERROR',

'class': 'django.utils.log.AdminEmailHandler'

},

'logfile': {

'level': 'INFO',

'class': 'logging.handlers.RotatingFileHandler',

'filename': logfile,

'maxBytes': 1024 * 1024 * 5, # 5 MB

'backupCount': 5,

'formatter': 'standard',

},

'jobengine_logfile': {

'level': 'INFO',

'class': 'logging.handlers.RotatingFileHandler',

'filename': jobengine_logfile,

'maxBytes': 1024 * 1024 * 5, # 5 MB

'backupCount': 5,

'formatter': 'standard',

},

'console': {

'level': 'INFO',

'class': 'logging.StreamHandler',

'formatter': 'standard'

}

},

'loggers': {

'': {

'handlers': default_handlers,

'level': 'INFO',

'propagate': True

},

'django.request': {

'handlers': ['mail_admins'],

'level': 'ERROR',

'propagate': True,

},

'standard': {

'handlers': ['mail_admins'],

'level': 'ERROR',

'propagate': True,

},

# We set the DB log messages to level ERROR, otherwise the logs

# get a ton of DEBUG SQL dumped into them

'django.db.backends': {

'handlers': [],

'level': 'ERROR',

'propagate': True,

},

'jobengine': {

'handlers': jobengine_handlers,

'level': 'INFO',

'propagate': False,

},

# Logger for individual jobs

'jobengine.job': {

'handlers': [], # Must be populated uniquely, per job

'level': 'INFO',

'propagate': False,

},

'ase_sa_common': {

'handlers': non_console_handlers,

'level': 'INFO',

'propagate': False,

}

}

}

For a real-world example, the following customer_settings.py file redirects all logging to syslogd in addition to the default locations::

from settings import LOGGING

# Logging overrides

LOGGING['handlers']['syslogger'] = {

'class': 'logging.handlers.SysLogHandler',

'level': 'DEBUG',

'formatter': 'standard',

}

LOGGING['loggers']['']['handlers'].append('syslogger')

utilities_models_logger = {

'handlers': LOGGING['loggers']['']['handlers'],

'level': 'ERROR',

'propagate': True,

}

LOGGING['loggers']['utilities.models'] = utilities_models_logger

Problem

I'm installing my own SSL certificate on my CloudBolt server. I have a CER certificate file, but when I copy it to he CloudBolt server and restart httpd, I get a start-up error.

Solution

Apache expects a .CRT to be a X.509 certificate in base64 encoded format. To covert the binary CER file, copy the CER file to the CloudBolt server and run the following command:

$ openssl x509 -inform DER -in ssl_certificate.cer -out ssl_certificate.crt

The short answer: no.

When you create any Resource Handler (whether AWS, OpenStack, vCenter, etc.), all you're essentially doing is making a one-way connection. No changes are made in that external system.

Our philosophy around this is that CloudBolt can unify your various clouds/systems without assuming management of the underlying content (templates, keys, floating IPs, security groups, etc).

Let's say you want to see how CloudBolt works with Amazon's AWS. Creating an AWS resource handler simply makes CloudBolt aware of your instances and metadata such as AMIs, key names, security groups, and VPC subnets. Deleting this metadata inCloudBolt does not remove them from your EC2 account.

Destructive changes generally only involve removal of instances when you delete servers.

Whenever we provide features to manage external content, such as deleting a server's NIC, we try to make sure it's clear that there will be impact outside of CloudBolt.

Certificate based authentication has been available since version 2.0 of the product. If you want to get certificate based authentication a try in a CloudBolt follow these simple steps:

Configuring PKI in CloudBolt's Web Interface

Log into CloudBolt as an admin user

From the DB browser page, replace "/admin/db/" with "/alladmin/"

Find the Utilities section and click on '+ Add' in the PKIUtilities row

Enter regex expressions that should evaluate to username, first, last and email fields based on the Subject (or DN) field in the user certificates your company uses

Optionally, you can determine what groups the new users should have requestor, approver, etc roles in. IMPORTANT: Do not assign the inventory manager role as it's been deprecated

Save the form

Configure PKI in CloudBolt's server settings

Edit /var/opt/cloudbolt/proserv/customer_settings.py, and append to the flie this following section:

# Enable PKI section

LOGIN_URL = "/pki/login"

from settings import INSTALLED_APPS

INSTALLED_APPS += ('pki',)

Configure apache to require client certificates for the/pki/loginurl

scpa copy of the root ca certificate to the c2 server. For this article we'll use/var/opt/cloudbolt/proserv/ca.crtas the path for the certificate

Edit /etc/httpd/conf/httpd.conf, and add the following section:

<Location /pki/login>SSLCACertificateFile /var/opt/cloudbolt/proserv/ca.crt

SSLVerifyClient require

SSLVerifyDepth 5

SSLOptions +StdEnvVars

</Location>

* If you are using a intermediate CA cert, instead ofSSLCACertificateFile use the directiveSSLCertificateChainFile

3. Restart httpd

CloudBolt's engineering team constantly provides new features and fixes bugs in the product. Our agile philosophy leads to rapid releases and rapid delivery of small, incremental features. We ship an Alpha nearly every week (which is tightly coupled to our 1-week sprint schedules) until we've hit the features we want in a release, usually 13 months. Then we begin shipping RCs (Release Candidates) for that release, usually for about a week. Finally, we ship a GA (General Availability) that is ready for production use.

A question we often get is: Can I run pre-release versions of CloudBolt in production? The answer isn't straightforward. We can't promise a pre-release version will work 100% correctly since it contains features that are still in development and bugs that haven't been found or fixed yet. However, we love feedback from our customers, which is why we provide pre-release versions of CloudBolt to our customers.

What follows are some details about our development process that might help you decide whether to run a pre-release version of the product.

CloudBolt's Development and Testing Process

Behind-the-Scenes Releases

As our engineers develop fixes and features for CloudBolt, they check them into our source code repositoryinstantly, a suite of automated tests begins testing the change, and within minutes we have a base understanding of whether the change broke anything obvious. These tests automatically build a Frequent pre-release version of CloudBolt, upgrade a server with that version, and run a suite of unit tests that cover about 33% of our codebase. (A number that we are always pushing higher.) We also use the upgraded CloudBolt server to deploy and tear down a server in Amazon, just to make sure CloudBolt can perform its most basic task.

Later that night, after our engineers have gone home, a Nightly pre-release version of CloudBolt is built. Everything in the Frequent testing repeats, integrating all of the day's changes. We also go many steps further and use the Continuous Infrastructure Testing (CIT) feature built into CloudBolt to perform a wide range of functions with CloudBolt, running about five hours of tests. We prove that provisioning on VMware, AWS, Azure, GCE, Softlayer, and others works correctly. We test remote scripts and actions, configuration managers like Chef and Puppet, verify the UI responds quickly, run Rules and Sync VMs, and finally test upgrades from the supported versions of CloudBolt (see our Deprecation Schedule for more details.). Our team reviews the results of these tests daily and usually resolves any issues discovered the previous night the next day.

Every weekend, we build a Weekly pre-release version of CloudBolt that repeats the above steps but includes another ten hours of automated CIT testing and which builds around 100 servers using CloudBolt with various combinations of technologies, crawls our API for errors, and runs through a large suite of edge cases. The following Monday, we review the results of the weekend and begin working to resolve any issues that were found through testing. If we discover and fix issues, we'll re-run the individual tests that failed until they are back to green.

Publicly-Available Releases

Most Tuesdays (or typically once during each week), we manually build an Alpha from the latest green point in our codebase. Since it was automatically tested, the only test we perform is to make sure the upgrader runs successfully on our individual development servers. If that works, we publish it to our Downloads page and send an email to Alpha testers. (Want to get on this list? Submit a support ticket to [email protected] requesting to be added to the list of Alpha testers.) Your feedback from using these Alphas results in us filing additional bugs and feature requests which we often work to include in subsequent sprints, but otherwise go into our backlog for future consideration with similar stories in a sprint. We likely won't provide a fix right away, since these releases aren't intended to be run in production, but will still accept support tickets so you can report any issues you find. Our advice will usually be to upgrade to the latest Alpha version if you haven't yet, and we may not begin researching the ticket until you have done so.

Near the end of a release cycle, we will cut our first Release Candidate (RC). Functionally, these are the same as an Alpha, but the cutting of a Release Candidate means that our engineering team has transitioned into a new phase of the development cycle: now we're regression testing the product. Throughout the development cycle, as developers worked on different areas of the product, they were noting areas of the product that they touched as an area that needs regression testing. In a larger release, we may go through upwards of 200400 manual regression tests in a week's QA cycle and may file upwards of 20 bugs that weren't caught through automated testing. These defects are fixed that same week, and we'll re-test the bugs that were discovered. Since these early Release Candidates may still have untested bugs, we likely won't provide fixes right away (as with Alphas), and don't recommend running RCs in production, but you are still welcome to submit support tickets for anything you find. We will likely require you to upgrade to the latest RC if you find an issue in an older RC.

At the end of the QA cycle, we'll cut a Final Release Candidate (RC). This final RC should look and feel identical to a GA release, and often contains the exact same bits, but occasionally a few additional release stoppers will be discovered by customers or our developers before we cut the GA, so a handful of additional bugfixes might make it into the product. We call the ~1-week period after QA before we cut the GA "Cooldown Week" and usually work on smaller improvements. Later Release Candidates are usually fairly stable. Depending on your tolerance for risk, you may want to run an RC in production, but be aware that any last-minute bugs that are discovered may get fixed before a GA.

With the end of Cooldown Week and following one final suite of (green) weekly tests, we cut the General Availability (GA) release and celebrate publishing it by kicking off the next sprint of awesome features.

So can I run pre-release versions of CloudBolt in production?

Alphas may contain features that are in the process of being developed and are likely to contain untested features and undiscovered bugs. If you are comfortable using CloudBolt under such a scenario, if you are comfortable performing frequent upgrades on your CloudBolt appliance, and if you are comfortable running with such bugs in production, running an Alpha in production might be right for you.

Release Candidates, though more stable, fall into the same category as Alphas.

In both categories of pre-releases, CloudBolt will not provide patches to resolve your issue,and will provide fixes in the form of a future upgrader.

GA releases will be supported according to our Deprecation Schedule, but youmay be required to upgrade to a later version of CloudBolt in order to receive support on a particular issue, especially if it has already been fixed in a later version.

Sometimes, you might want to prevent a Server from being deleted through CloudBolt. You can add the "Server Lock" parameter to any Server in CloudBolt to block users from deleting a server. As a parameter, you can also bake this into new servers in an Environment or Group.

To automatically import users from an LDAP or AD connection do the following. Note: this assumes there is one LDAP/AD connection setup on your instance of CloudBolt. If there are multiple, you may have to adjust the array index in line the "job_params.ldap =LDAPUtility.objects.all()[0]" to [1] or [2].

On the C2 server, run:

$ /opt/cloudbolt/manage.py shell_plus

Once you get a prompt, enter the following commands line-by-line:

job_params = SyncUsersFromLdapParameters.objects.create()job_params.ldap = LDAPUtility.objects.all()[0]job_params.failure_email_address = GlobalPreferences.get().cbadmin_emailjob_params.save()job = Job(type="sync_users_from_ldap", job_parameters=job_params )job.save()

This will create a job that will connect to your LDAP server and import the users matching the search filter you provided.

Also keep in mind, by default AD limits the number of results returned, so if you try to import 1000+ users, AD will issue an error. To fix this error, you'll need to adjust this setting in your AD server.

To convert the CloudBolt OVA to an image suitable for deployment we first need to install qemu-img to perform the conversion. You can use brew to install qemu-img on macOS, or your favorite Linux package manager. Next, extract the vmdk file contained within the OVA file:

$ tar xvf cloudbolt-latest.ova

One of the files from the above extraction is calledCloudBolt-6.2-disk1.vmdk where the version is dependent on the version of CloudBolt you've downloaded.

At this point, we can convert the vmdk file to a qcow2 image for use in OpenStack:

$qemu-img convert -O qcow2CloudBolt-*.vmdk cloudbolt.qcow2

The end result is a new image file called cloudbolt.qcow2 that is ready for import into OpenStack. This will not include CPU, Disk, and Memory metadata. For a production instance of CloudBolt we recommend you'd allocate at a minimum of 2 CPUs, 8 GB or RAM and 50 GB of storage.

CloudBolt support customization of Windows VMware VMs' timezones during provisioning. By default, CB presents users with a limited list of timezone choices, but that can be customized to include others. To do this, ssh to the CB server as root, edit `/var/opt/cloudbolt/proserv/customer_settings.py` and paste this in there:

TZ_CHOICES = (

(1, 'US/Samoa'), (2, 'US/Hawaii'), (2, 'US/Aleutian'), (3, 'US/Alaska'), (4, 'US/Pacific'), (10, 'US/Mountain'), (20, 'US/Central'), (35, 'US/Eastern'), (85, 'GMT'),)

You can remove lines and add more lines with other options, just make sure the codes match appropriately. Here is a reference of the available zones:

MSDN values for first column:http://msdn.microsoft.com/en-us/library/ms912391(v=winembedded.11).aspx

VMware Names for second column:https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/timezone.html

Need to create a list of users that are currently authenticated with CloudBolt? First and foremost understand that unless youhave the setting in "Admin > Inactivity timeout minutes" set to a value, a users will remain authenticated until they explicitly logout. If this isn't set, every user that has authenticated will be return by the script that follows.

After running /opt/cloudbolt/manage.py shell_plus, enter the script:

for u in [u for u in User.objects.order_by('-last_login') if u.is_authenticated()]:

print u.last_login, u.username

This will return a list of authenticated users sorted in descending order of their last login date/time.

The Problem

Amazon Linux AMIs don't ship with scp installed, therefore CloudBolt is unable to execute remote scripts as part of the provisioning process.

The Solution

Add an "AWS User Data" parameter to your AWS-backed environment, that tells AWS EC2 to install the openssh-clients package as part of provisioning. Here's an example of how that script looks:

#cloud-config

repo_update: true

repo_upgrade: all

packages:

- openssh-clients

Add this as the only option to this parameter and CloudBolt will pass this to AWS when provisioning your VM. For more information on user-data and cloud-init scripts in AWS, see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

Unable to Connect Amazon AWS Resource Handler

Sometimes users run into the situation where they're unable to connect their CloudBolt instance to AWS.

This could be related to a proxy issue, which is addressed by entering the appropriate proxy info in ADMIN > MISCELLANEOUS SETTINGS.

The other frequent issue we see causing this is related to the time on the CloudBolt server. If the time on the server is off by a minute or more, the CB server will be unable to connect to AWS due to time differences in the signed API request coming from CloudBolt.

To fix the date on the CloudBolt appliance, either manually set your time with:

$ date -s "2 OCT 2006 18:00:00"

or run:

$ ntpdate -s time.nist.gov

To fetch the latest date/time from a public ntp server.

Some of our customers are running the CB appliance behind a load balancer. While the specifics of setting up external DBs and load balancing varies from customer to customer, there are a few CB appliance aspects everyone should be aware of when setting up a distributed architecture. This KB article is meant to address those CB specifics.

Front-end replication

There are several files that you want to keep in sync between your front-end instances. We recomend rsync'ing the following directories:

/opt/cloudbolt/cbhooks/hookmodules/

/var/opt/cloudbolt/proserv/

/var/log/cloudbolt/jobs/

/var/www/html/cloudbolt/static/

External DBs

If you are setting an external DB make sure to set the correct DB connections in /var/opt/cloudbolt/proserv/customer_settings.py. You'll need a section that looks like this:

DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'cloudbolt', 'USER': 'YOUR_DB_CB_ACCOUNT', 'PASSWORD': 'YOUR_DB_CB_PASSWORD', 'HOST': 'YOUR_EXTERNAL_DB_IP', 'PORT': 'YOUR_EXTERNAL_DB_PORT', 'OPTIONS': {"init_command": 'set storage_engine=INNODB, \ SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED', }, } }

Already Have CloudBolt Installed?

CloudBolt now shipswith all of the product documentation self-contained. Look at the top of a CloudBolt page, and you'll see a link to the docs -- just click the life preserver!

http://docs.cloudboltsoftware.com

Don't Yet Have CloudBolt Installed?

Go to .

Azure's new Resource Manager provides a new way to deploy and manage your solutions. If you used the earlier Service Managementmodel and want to learn about the changes, see Understanding Resource Manager deployment and classic deployment.

If you'd like to learn more about the benefits of the new Resource Manager, read the Azure Resource Manager overview.

Resource Handlers are available for both management systemswithin CloudBolt. Theycan co-exist, howevervirtual machines and networks managed under one model will not be visible or available under the other.

Many resource types will transition easily between the two management systems. The resource providers that differentiate between the two models are:

Compute - For more details, read the MSDN article on Azure VM Resizing and SKU Change in ASM and ARM.

Storage

Network

For these resource types, you must be aware of which version you are using because the supported operations will differ. If you are ready to migrate your resources to Resource Manager deployment, see Platform supported migration of IaaS resources from Classic to Azure Resource Manager.

Migrations can be performed a number of ways:

Via the instructions provided by Microsoft via Migrate IaaS resources from classic to Azure Resource Manager by using Azure PowerShell

Via the ASM2ARM PowerShell script module. More details are available here: https://phidiax.com/blog/post/azure-vm-migration-from-asm-to-arm-using-the-asm2arm-tool-multiple-vms-and-virtual-network

Via AzCopy, which can be run from Powershell or CMD shell. More details are available here:https://www.phidiax.com/blog/post/azure-vm-migration-from-asm-to-arm-using-azcopy-and-powershell

We will update the documentation as support evolves for bothmanagement systems.

Console Encryption

Enabling console encryption will cause all console-related traffic to be served over HTTPS instead of downgrading to HTTP. To enable console encryption follow these steps.

Ensure that the certs installed on C2 are trusted by a certificate authority installed on your client's web browsers. A self-signed certificate with user-added security exception is not sufficient for the console feature to work in browsers other than Google Chrome.

Edit /var/opt/cloudbolt/proserv/customer_settings.py, adding the following lines and changing the certificate and key paths to match the paths that are set in /etc/httpd/conf.d/ssl.conf:

# The PEM-formatted SSL certs used by # the webserver to provide HTTPS.ENCRYPT_CONSOLE = TrueSSL_CERT_PATH = "/etc/pki/tls/certs/localhost.crt"SSL_KEY_PATH = "/etc/pki/tls/private/localhost.key"

Ensure the permissions of your certificate and key are set correctly by running these commands, replacing SSL_CERT_PATH and SSL_KEY_PATH with the values you set above:

$ chown apache:apache SSL_CERT_PATH SSL_KEY_PATH$ chmod 400 SSL_CERT_PATH SSL_KEY_PATH$ # Be sure that the ancestor directories of your certs have the executable bit$ chmod +x <a list of your certificate's parent directories>

Restart CloudBolt by running:

$ service httpd restart

Problem

Youhave a script that you want to run on some remote server, but the script shouldn't be run concurrently. If the script is running and it's triggered again, the second execution should wait until the first is finished.

Solution

We're going to use the fasteners python library to create a file lock that can be used to make sure our script runs one at a time on our target server. This will be implemented as a CloudBolt plugin, so the remote script will be embedded in our python script.

Let's start by installing the fasteners library on the CloudBolt server with:

$ pip install fasteners

The following example shows howto utilize the fasteners library to enforce interprocess synchronization using a file lock. This example assumes the job object has a server set, so this would be run on every server associated with a CloudBolt job. This could be easily modified by looking up a server instance to run this on a dedicated remote script host.

import fasteners

from settings import VARDIR

REMOTE_SCRIPT = '''

echo "Hello World"

'''

@fasteners.interprocess_locked('{}/run/cb_job_lock'.format(VARDIR))

def run(job, logger=None, **kwargs):

for svr in job.server_set.all():

svr.execute_script(

script_contents=REMOTE_SCRIPT

)

return '', '', ''

ssh as root to the C2 server

Edit /var/opt/cloudbolt/proserv/customer_settings.py

add a line to the file:

SESSION_COOKIE_AGE = 60 * 30 # 30 minute inactivity timeout

That will set the user timeout in seconds. in this example the users will be logged out after 30 minutes of inactivity

Restart apache:service httpd restart

TIP: To test that this is working, you could set the timeout to 2 seconds, ensure that you are auto-logged out very quickly after logging in, and then adjust the timeout to something appropriately higher.