Quorum FAQs

Issue:

While trying to connect via XenCenter to a 5.5 XenServer, the connection failed with Unknown error occurred. Windows 10 enhanced security is the culprit and will not allow for connecting to systems with outdated weak SSL certs. The connection works fine with XenServer 6.02 and 6.5.

Reason:

To fix this a regeneration of the xapi SSL cert on the hypervisor with 1024-bit encryption is needed, but not before saving off the old cert just in case. Since this is a certificate specific to xapissl, it did not appear to effect anything. Both xe and XenCenter behaved normally and no VMs were affected.

Resolution:

Here is the process to update the certificate on the hypervisor - the IP should be the IP of the hypervisor.

Stop xapissl service and backup the xapi-ssl.pem

#cd /etc/xensource

#cp -p xapi-ssl.pem xapi-ssl.pem.bak

#service xapissl stop

Edit the /opt/xensource/libexec/generate_ssl_cert script to set the 1024-bit encryption (default is 512).

#vi /opt/xensource/libexec/generate_ssl_cert

Find and change this line:

[To insert a text in VI, click Insert. When done click Esc, then type : and wq! to save]

openssl genrsa > privkey.rsa

to

openssl genrsa 1024 > privkey.rsa

Generate the cert but remove the current cert first:

#rm f xapi-ssl.pem

#/opt/xensource/libexec/generate_ssl_cert "/etc/xensource/xapi-ssl.pem" 'HV IP Address'

Replace the HV IP Address in the command above with the hypervisor IP you are updating.

Restart the XAPI services

#service xapissl start

#xe-toolstack-restart

A new xapi-ssl.pem file will be generated with 1024 bit encryption

------------------------------------------------------------------------------------

Here is a link to download XenCenter 6.5

https://docs.quorum.net/file_share/TOOLS/XenCenter-65.exe

------------------------------------------------------------------------------------

Escalation:

Standard Escalation Procedure

How to access ZenDesk cases

1. go to https://quorum.zendesk.com/hc/en-us and click "Sign in"; upper right corner

2. select name drop down and select "My activities"

Issue:

Remove a PN

Reason:

Remove a PN

Resolution:

Highlight it and click - in Protection Config. Leave the box unchecked. That leaves the snapshots but removes the server from config and backups. When you're ready to remove snapshots you would click "Find Orphans" on both HA and DR to remove the snapshots and reclaim that space.

Escalation:

Standard Escalation Procedure

Issue:

Restart a 4.x Appliance

Reason:

Restart a 4.x Appliance

Resolution:

4.x appliances MUST be restarted from the Provisioner drop down only.

Rebooting from the HV or XC will cause a hang to occur due to how the new Filer communicates.

Escalation:

Standard Escalation Procedure

Issue:

POD Installation

Reason:

POD Installation

Resolution:

Installation Guide

OnQ POD Installation

Version 1.0

http://www.dell.com/learn/us/en/12/shared-content~data-sheets~en/documents~dell-poweredge-fc630-spec-sheet.pdf

Parts

The onQ Pod Infrastructure is composed of several parts. The onQ-Blade provides the compute and memory for the onQ systems and recovery nodes. The onQ-Chassis holds up to 4 onQ-Blades. The onQ-Head Unit powers the onQ-Storage options. The onQ-Storage is a storage JBOD with 30x6TB drives. And finally the onQ-StorageExp is an additional 30 Drives for expansion of the onQ-Storage.

Rack Installation

Installing in your rack comes with some tips and tricks to save you time and potentially some back pain.

Tip 1 If you plan for later expansion of your onQ POD, leave space above and below.

Tip 2 Install the Storage unit in the rack without the drives in it. It is have equipment and should be a 2 man lift

Tip 3 Install in the order shown below, keeping the onQ-Head unit between the onQ-Chassis and the onQ-Storage.

Tip 4 Install the onQ-Pod towards the bottom of your rack. This again is just to save the lifting of the equipment on your back.

Tip 5 Install the Drives as labeled. Your onQ-Pod is all configured and ready, so the Drive order matters.

Cabling

The below is for the networking and SAS connection cables. Cabling reflects that this is a highly expandable and redundant architecture.

Pod Power

onQ-Chassis will require (2) 220v C19 PDU Outlets. (2) PSUs will be rated at 1755W and pair will work in a 1+1 redundant configuration. onQ-Head Unit will require (2) standard 110v 5-15P or 220v C13 PDU outlets. Each PSU will be 750W and work in a 1+1 redundant configuration. onQ-Storage will require (2) 220v C19 PDU Outlets. Each PSU will be rated at 2000W and each pair will work in a 1+1 redundant configuration.

Installing Drives

onQ-Storage Drawers

onQ-Storage Drawer

The first step is to ensure that drives are secure in the sled so they will be secure in the drawers. The drive slots in each drawer are numbered from front to back, left to right (as in the diagram). Load drawers from drawer 1 on the top and work your way down to the bottom. If you only have the first 30 drives, then you will end up with drawers 1 and 2 fully populated and drawer 3 half populated. The rest of the space is for the onQ-StorageExp of 30 more drives.

More information

onQ POD consists of these OEM Dell Components. Each component spec sheet can be found here:

Dell R730XD - https://www.dell.com/learn/us/en/04/shared-content~data-sheets~en/documents~dell_poweredge_r730xd_specsheet.pdf

Dell MD3060e - http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/DellStorage_MD3060e_Dense_JBOD_SS.pdf

Dell FX2 - http://www.dell.com/learn/us/en/12/business~smb~merchandizing~en/documents~poweredge_fx2_spec_sheet.pdf

Dell FC630 -

Escalation:

Contact Quorum

Issue:

Updated Filer to 4.1HF1.. Now HA shows stopped in XC with "This VM needs storage that cannot be seen from that server" andwon'tstart.

Reason:

Restart Needed

Resolution:

Restart the whole box from the Filer Dropdown and watch in XC to see if it starts going down.

Escalation:

Standard Escalation Procedure

onQ 3.9

http://www.quorum.net/help/3.9/onQ/index.html

onQ 3.8

http://www.quorum.net/help/3.8SP1/onQ/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Release%20Notes/Welcome.html

Issue:

Oracle Troubleshooting

Reason:

Oracle Troubleshooting

Resolution:

====================

What is new in 3.8SP1 (Windows oracle DB protection).

====================

How to do protect the oracle DB:

PN: To prepare your environment.

Make sure squirtcopy can take good snapshot.

http://www.quorum.net/help/3.8SP1/onQ/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Recovery/Restore_Database_Windows.html

Bug 6859 Oracle VSS writer error in xml too long during snapshot taken

Bug 6874 Squirt copy should reports oracle writer error

RN: To recover the Oracle database.

Make sure the Oracle writer is in stable state before restoring/recovering DB.

bug 6842 new restore_oracle.bat (has green color terminal)

Follow the restore procedure:

http://www.quorum.net/help/3.8SP1/onQ/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Recovery/Restore_Database_Windows.html

Online doc:

The 3.8SP1 online doc has a new updated link to download the batch scripts.

Install the Quorum Oracle database recovery scripts in the %programfiles%\Quorum\usr folder.

http://www.quorum.net/help/Oracle-tools.zip

====================

What to do if DB cannot be open after running the restore_oracle.bat.

====================

Start cmd.exe as administrator (or runas /user:administrator) and

use it for the incomplete recovery procedule. This will bring up database

to the most recent consistent state. Actually it will bring up database to

all available DB changes in RN (up to the last change in the oracle VSS

writer metadata xml file).

If wrong cammands are issues, start a new RN, run the the restore_oracle.bat (3.8SP1 version)

and continue following steps.

Set the ORACLE_SID to the DB SID (Assume the SID is ORCL in the example)

C:\> SET ORACLE_SID=ORCL

Start the sqlplus shell

C:\> SQLPLUS / AS SYSDBA

Issue the oracle DB imcomplete recovery procedure.

Use recover database until cancel method.

SQL> RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE;

key in "enter" until fail and then type "CANCEL"

If DB still cannot open procced to 2, otherwise go to step D.

Perform point-in-time recovery.

Find out the missing last change number.

Write down the last NEXT_CHANGE# value (assume it is 118009).

SQL> SELECT * FROM V$LOG_HISTORY

Try to recover to the last chage number in the last archived redo log in RN

It is the last NEXT_CHANGE# value minus 1 (118009 -1)

SQL> RECOVER DATABASE UNTIL CHANGE 118008 USING BACKUP CONTROLFILE;

Open the DB with resetlogs

SQL> alter database open resetlogs;

Verify DB is opened (It will have new change number in databse view)

SQL> SELECT CURRENT_SCN FROM V$DATABASE;

Back up and restore Oracle 10g+ database on Windows

The following procedures walks you through how to back up and restore an Oracle 10g+ database on a Windows Recovery Node. These procedures assume that you are knowledgeable and comfortable with Oracle databases.

Restoring and recovering an Oracle database, after you have booted the recovery node (RN), is a unique process from that of other databases. These procedures cover the prerequisites for hot backups with Volume Shadow Copy, and the required tools developed by Quorum to recover and open your Oracle Database on your recovery node.

These procedures outline the steps to recover an Oracle database with ARCHIVELOG mode enabled and the Oracle VSS Writer service installed. Hot backups are not possible with an Oracle database using NOARCHIVELOG mode. Databases using NOARCHIVELOG mode must be put in a consistent (shutdown) state in order to perform a backup.

Recovery, maintenance and troubleshooting of Oracle databases on the protected node (PN) will not be covered in these procedures.

These procedures might not work with your specific configuration; as such, discuss your configuration with Quorum Support before you perform this procedure.

(Step 1) To prepare your environment:

1.

Verify that your RN is running:

Windows 2003 or greater

Oracle Database 10g or greater

2.

Ensure that you have your archive logs stored on a separate volume from your database (DBF) files.

Oracle requires that you put archive logs on a separate volume when performing backups with the Oracle VSS Writer service.

Quorum cannot recommend an architecture that violates Oracle's specifications. However, it should be noted that Quorum did not observe any adverse effects when putting the database and archive logs on the same volume while performing VSS backups and restores.

3.

From the onQ Portals Protection Config tab, verify that you have the following volumes protected.

The volume that contains your database files.

The volume that contains the archive logs.

4.

For the protected node on which you want to perform hot backups and for every Oracle database instance, install the Oracle VSS Writer service (service.msc).

The Oracle VSS Writer service is installed by default with Oracle Database 11g. Oracle Database 10g releases require a manual installation of the Oracle VSS Writer service. The installation instructions for the Oracle VSS Writer service can be found here.

5.

Configure the Oracle services:

Make sure Oracle is in ARCHIVELOG mode.

Configure Oracle VSS Writer service with the SYSDBA user, mode Automatic; then, test that the user account has SYSDBA set correctly:

> sqlplus user_account as SYSDBA

For example:

Where sysadm_amy is the user_account.

> sqlplus sysadm_amy as SYSDBA

Configure the Volume Shadow Copy Service with Local Service account, mode Automatic.

Configure the Software Shadow Copy Provider service with Local Service account, mode Automatic.

(Step 2) To back up the Oracle database:

Perform the following procedure on the PN and for each database that you want to restore.

Verify that the Oracle VSS Writer service is running.

> oravssw /q /status

OracleVssWriterORCL - The service is running.

On the PN, verify that the VSS writers are Stable.

> vssadmin list writers

...

Writer name: 'Task Scheduler Writer'...

State: [1] Stable

Last error: No error

Writer name: 'VSS Metadata Store Writer'...

State: [1] Stable

Last error: No error

Writer name: 'Performance Counters Writer'...

State: [1] Stable

Last error: No error

Writer name: 'Oracle VSS Writer - ORCL'...

State: [1] Stable

Last error: No error

Writer name: 'System Writer'...

State: [1] Stable

Last error: No error

Writer name: 'ASR Writer'...

State: [1] Stable

Last error: No error

Writer name: 'Registry Writer'...

State: [1] Stable

Last error: No error

Writer name: 'COM+ REGDB Writer'...

State: [1] Stable

Last error: No error

Writer name: 'Shadow Copy Optimization Writer'...

State: [1] Stable

Last error: No error

Writer name: 'WMI Writer'...

State: [1] Stable

Last error: No error

On the PN, unzip the Quorum Oracle database recovery scripts in the C:\Program Files\Quorum\usr folder.

Modify the restore_oracle.bat batch file:

Make a backup copy of the restore_oracle.bat batch file.

Change the placeholder database names to match your Oracle database names. Each instance should be separated by a single space and encapsulated by parenthesis.

> set DB_INSTANCES=(database_instance_name)

For example:

Where there are three database instances: orcl, orcl1, and orcl2:

> set DB_INSTANCES=(orcl oracl1 oracl2)

Save your changes.

Initiate a backup of the PN, or wait for a scheduled backup to complete.

Repeat Step 2: Verify that the VSS writers are Stable.

Verify that the backup completed successfully. Go to Monitor backups.

Check the PNs Event Log for VSSrelated errors. If there are errors, resolve them. For troubleshooting information, see the appropriate return code in A0801.

(Step 3) To restore the Oracle database:

After the RN boots, you can now begin the Oracle Database recovery process. This process can be performed with an RN booted in test mode or production mode. Perform the following procedure on the RN and for each database instance that you want to restore. This procedure includes two ways to restore your database(s). Begin with the recommended solution.

(Recommended) Using restore_oracle.bat

This procedure uses Quorums restore_oracle.bat scripts. This solution is recommended.

With the RN, in test or production mode, navigate to the C:\Program Files\Quorum\usr folder, then run the restore_oracle.bat file completely.

You will be prompted for your database password twice for each database instance: once for a shutdown command and once for an open resetlogs command.

Verify that your database(s) opens without any errors.

If the database(s) opens without error,you'vesuccessfully restored your database(s). Youre done!

If the database(s) fails to open successfully, the cause is the result of known bugs in Oracles VSS Writer. Perform the troubleshooting procedure outlined in (Step 4, if necessary) To fix an incomplete database recovery:.

Using manual process

This procedure uses a manual process to restore your database(s). This solution is not automated as is the case with using Quorums restore_oracle.bat scripts, which executes the same commands in the following manual process.

Shut down the database that you want to restore.

> set ORACLE_SID=database_instance_name

> sqlplus SYS as SYSDBA

SQL> SHUTDOWN IMMEDIATE;

SQL> EXIT;

For example:

Where orcl is the database_instance_name or Oracle System ID (SID).

> set ORACLE_SID=orcl

> sqlplus SYS as SYSDBA

SQL> SHUTDOWN IMMEDIATE;

SQL> EXIT;

Restore the Oracle VSS Writers metadata and generate the required archive log, if any.

> qvss.exe restore

Open the database for service and reset the redo logs.

> sqlplus SYS as SYSDBA

SQL> ALTER DATABASE open resetlogs;

SQL> EXIT;

Compare the PNs and RNs row counts. They should match, if no changes were made after the backup.

> sqlplus system@database_instance_name

SQL> select count (*) from table_name;

For example:

Where testtable is the table_name and where orcl is the database_instance_name or Oracle System ID (SID).

> sqlplus system@orcl

SQL> select count (*) from testtable;

Check the PNs Event Log for VSSrelated errors. If there are errors, resolve them. For troubleshooting information, see the appropriate return code in A0801.

Verify that your database(s) opens without any errors.

If the database(s) opens without error,you'vesuccessfully restored your database(s). Youre done!

If the database(s) fails to open successfully, the cause is the result of known bugs in Oracles VSS Writer. Perform the troubleshooting procedure outlined in (Step 4, if necessary) To fix an incomplete database recovery:.

(Step 4, if necessary) To fix an incomplete database recovery:

If you attempted to restore an Oracle database either manually or using restore_oracle.bat and your database(s) fails to open, you have an incomplete database recovery. This procedure includes two ways to fix an incomplete database recovery. Begin with the recommended solution.

(Recommended) Using recover database until cancel method

This solution uses Oracles recover database until cancel recovery method to fix an incomplete database recovery. This procedure is the easiest way to fix such a problem, and is recommended before attempting any other incomplete database recovery method.

Restart a fresh RN, in test or production mode. If youre using restore_oracle.bat, execute that file now, but choose not to start the databases at the end of batch file.

Set the ORACLE_SID to the database SID. Where the SID is ORCL:

C:\> SET ORACLE_SID=ORCL

Start the sqlplus shell:

C:\> SQLPLUS / AS SYSDBA

Perform the following Oracle DB incomplete recovery procedure, which uses the recover database until cancel method:

Execute the recover database until cancel method.

Verify that the suggested log file is available or specify an available archived log file to be restored, then press Enter.

If the suggested log file is unavailable or there are no archive log files, type CANCEL.

SQL> RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE;

ORA-00279: change 5619127 generated at 11/18/2014 13:03:07 needed for thread 1

ORA-00289: suggestion : E:\LOGS\ARCH_863902564_1_81.ARC

ORA-00280: change 5619127 for thread 1 is in sequence #81

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

CANCEL

Media recovery cancelled.

SQL>

Open the database with resetlogs:

SQL> alter database open resetlogs;

Verify that the database opens successfully. The database has a new change number in the database view.

SQL> SELECT CURRENT_SCN FROM V$DATABASE;

If the database fails to open successfully, the cause is a known bug (race condition) in Oracles VSS Writer. Perform the procedure outlined in Using pointintime recovery method.

Using pointintime recovery method

This solution uses Oracles pointintime recovery method to restore the database to the target time (or SCN) for recovery. This procedure is guaranteed to succeed, but is more complicated; for this reason, use it as a last resort. Perform the following procedure for each database instance.

Restart a fresh RN, in test or production mode. If youre using restore_oracle.bat, execute that file now, but choose not to start the databases at the end of batch file.

Set the ORACLE_SID to the database SID. Where the SID is ORCL:

C:\> SET ORACLE_SID=ORCL

Start the sqlplus shell:

C:\> SQLPLUS / AS SYSDBA

Perform the following Oracle DB incomplete recovery procedure, which uses the pointintime recovery method.

Determine the missing last change number and record the last NEXT_CHANGE# value:

SQL> SELECT MAX(NEXT_CHANGE#) FROM V$LOG_HISTORY;

Lets assume the last change value is 118009.

Recover to the last change number in the last archived redo log on the RN. This value is the last NEXT_CHANGE# value (see Step 4) minus 1. Where the last change value is 118009, therefore, the value is 118008 (118009 -1).

SQL> RECOVER DATABASE UNTIL CHANGE 118008 USING BACKUP CONTROLFILE;

Open the database with resetlogs:

SQL> alter database open resetlogs;

Verify that the database opens successfully. The database has a new change number in database view.

SQL> SELECT CURRENT_SCN FROM V$DATABASE;

If your database still fails to open successfully, contact Quorum Support.

Oracle failed dB's in RN's

I have been spending a lot of time with a customer on their Oracle RN issues. Just about all of their oracle servers had failed DBs in the RNs. With assistance from Oracle Support we were able to get things working. Here is the needed process on the RN:

Shutdown the database

Run qvss.exe restore

Execute: recover database using backup controlfile until cancel;

Hit ENTER to accept the suggested log file to apply

Enter cancel at the next prompt

Execute: alter database open resetlogs;

The above is done in lieu of the reset logs script we normally run. I am not sure if any cases you have might involve oracle, but give the above solution a try before we deem it universal.

From < https://system.netsuite.com/app/center/center.nl/c.3466275/ctype.KB/it.I/id.810465/KB.466/.f?category=21 >

Escalation:

Contact Quorum

Issue:

Consistently failing ASR Writer that causes backups to crash or go into a scan loop.

Reason:

Consistently failing ASR Writer that causes backups to crash or go into a scan loop.

Resolution:

Cycle the VSS service, which puts the ASR Writer in a stable state.

Rerun backups

If they fail: Check the sq.log for what is being specifically reported as well as the Eventviewer to see if there are other VSS/volsnap events.

Make sure the space on the volumes is not tight

In Eventviewer also look at System Log for volsnap errors.

No volsnap, but an error with SID and permissions

Is a COM issue. Server is busticated and they need to reboot the PN.

Escalation:

Standard Escalation Procedure

Issue:

onQ Ports (Network & Firewall Requirements)

Reason:

onQ Ports (Network & Firewall Requirements)

Resolution:

Note:Traffic to onQ Appliances should not be open to any public access, except 443 and 123.

Browser Console

5900-59XX

May actually be 55900 - 55999

LAN/Network Communications

The following port(s) are required for administrative access to the portal on each onQ Appliance and for alerts, updates, and licensing. These ports should not be open to public access:

Direction

Port

Protocol

Purpose

LANonly Inbound

80 & 443

TCP

onQ Portal access. Needed internally on private network LAN only. No Internet access recommended.

Inbound

443

TCP

https from onQ IP needed to the Internet for licensing and alerts

Outbound

443

HTTPS

Access for alerts (httpsalerts.onqcentral.com), updates and licensing (updates.onqcentral.com).

Inbound & Outbound

123

UDP

Communication with NTP (ntp.org).

onQ AppliancetoPN Communications

Generally, onQtoPN communications are over a LAN that is not secured. In the event that there are any filtering devices between the PN and the onQ Appliance, the following ports need to be accessible on the PN:

Direction

Port

Protocol

Purpose

Outbound

3000

TCP

Allows squirtcopy communications

Inbound

5000

TCP

For FLR processing, allows WVhds for Windows and Netcat for Linux to receive data from onQ Manager

Inbound

5990

TCP

Allows DCRM to accept connections from onQ Manager

Inbound

5990

UDP

Allows LAMD to communicate with onQ Manager

Inbound

5991

UDP

Allows LAMD to monitor the PN

Inbound

5992

UDP

Allows LAMD to send PN heartbeat

HAtoDR Communications

The following ports must be accessible between the HA Appliance and the DR Appliance pairs across WAN or LAN:

Direction

Port

Protocol

Purpose

Inbound & Outbound

22

TCP

Allows secure TCP connections between onQ Appliances. May be remapped.

Inbound & Outbound

81

TCP

Allows interappliance link monitoring. May be remapped.

WANDR Site Communications

The following ports must be accessible/inaccessible between your WAN and DR site:

Direction

Port

Protocol

Purpose

Inbound & Outbound

22

TCP

<DR IP> to <HA IP>

Inbound & Outbound

81

TCP

<DR IP> to <HA IP>

Inbound

80

TCP

Deny ANY PUBLIC / External / Internet to <DR IP> (HTTP protocol)

Inbound

443

TCP

(https) Allow <DR IP> outbound to Public / External / Internet

Outbound

443

SSL, SMTP, & SSH, TCP

Access for alerts (alerts.onqcentral.com), updates (updates.onqcentral.com), and licensing (lic.onqcentral.com).

https://docs.quorum.net/help/4.0HF1/onQ/index.html#page/Getting%2520Started%2FSitePreparation.3.03.html%23wwconnect_header

Escalation:

Contact Quorum

Issue:

Kroll Resources

Reason:

Kroll Resources

Resolution:

Escalation:

Standard Escalation Procedure

Issue:

Linux RN Build

Reason:

Linux RN Build

Resolution:

Escalation:

Contact Quorum Support

Issue:

Filer Space Issues

Reason:

Filer Space Issues

Resolution:

Do not touch the Filer at the risk of decapitation: capture "df -h" output and send it over.

Escalation:

Standard Escalation Procedure

Issue:

Disaster Recovery Run Book

Reason:

The purpose of this disaster recovery run book is to prepare your company in the event of extended service outages caused by factors beyond its control (e.g., natural disasters, man-made events), and to restore services to the widest extent possible in a minimum time frame. The plan identifies the best practices for using the Quorum onQ Solution from the cloud location based in QDATACENTER.

Resolution:

Terminology

The figure below shows some of the terminology we use when discussing the onQ and its components:

About Hybrid Cloud

Hybrid Cloud is a DR Appliance in the cloud (DRaaS). The Deployment of a Hybrid Cloud-enabled HA Appliance is virtually identical to a non-cloud DR Appliance. The only difference is that you must also add information so that the HA Appliance can VPN to the DR Appliance. This secure connection enables the HA Appliance to send snapshots to the DR Appliance.

In the event of an HA site disaster, Quorum can ship you the cloud DR Appliance, replacing it with a new DR instance. This onsite DR Appliance assumes the HA role.

The Quorum Data Center that will be hosting your applications in Santa Clara CA are SAS-70 Type II certified. All data transferred from the HA Appliance to the DR Appliance goes through a 128-bit AES encrypted session behind a 256-bit AES VPN tunnel that is connected directly from the HA Appliance to the DR Appliance. There is a dedicated virtual firewall isolating each individual custom virtual network. All connections to the DR Appliance or the DR RNs are via VPN to that firewall.

Moreover, Hybrid Cloud is certified PCI compliant. As the first vendor in the industry to offer cloud recovery that meets the strict payment card industry's (PCI) security specifications, Quorum ensures all sensitive data like credit card data remains secure through the backup and recovery process.

Scope

The scope of this plan is limited to COMPANY systems protected by the Quorum onQ Appliances. This is a disaster recovery run book, not a daily problem resolution procedures document.

Plan Objectives

Serves as a guide for the COMPANY recovery teams.

Provides procedures and resources needed to assist in recovery when both the primary servers and the Quorum onQ high availability (HA) appliances (first line of defense) at the data center in YOUR CITY are not available.

Assists in avoiding confusion experienced during a crisis by documenting, testing and reviewing recovery procedures.

Documents storage, safeguarding and retrieval procedures for vital records.

Assumptions

Key people will be available following a disaster.

A national disaster such as nuclear war is beyond the scope of this plan.

This document and all vital records are stored in a secure off-site location and not only survived the disaster but are accessible immediately following the disaster.

Disaster definition

Loss of computer processing from the datacenter in YOUR CITY for any reason (power, water), connectivity (system sites), or catastrophic event (weather, natural disaster, vandalism) that causes an interruption in the service provided by COMPANY computer operations.

Team member responsibilities

Each team member will designate an alternate backup

All the members should keep an updated calling list of their work team members work, home, and cell phone numbers both at home and at work.

All team members should keep these records for reference at home in case the disaster happens during after normal work hours. All team members should familiarize themselves with the contents of this plan.

Invoking the plan

This plan becomes effective when a disaster occurs. Normal problem management procedures will initiate the plan, and remain in effect until operations are resumed at the original YOUR CITY location, or a replacement location and control is returned to the appropriate primary systems.

Disaster declaration

The COMPANY Emergency Management Team and Location Response Coordinator are responsible for declaring a disaster for Technical Services and activating the various recovery systems as outlined in this plan.

In a major disaster situation affecting multiple business units, the decision to declare a disaster will be determined by COMPANY. The Emergency Management Team/Location Response Coordinator will respond based on the directives specified by COMPANY IT.

Notification

The Emergency Management Team (EMT) can at its sole discretion activate the onQ disaster recovery (DR) appliances whenever it feels it is in the best interest of COMPANY to do so. If the pending disaster is something that can be tracked and expected like an impending severe weather system then the EMT should trigger an immediate backup of protected nodes regardless of when the next backup is scheduled. But of course disasters are often unplanned events. Regardless of the disaster circumstances, or the identity of the person(s) first made aware of the disaster, it is recommended the EMT activate the onQ DR immediately in the following cases:

Two (2) or more servers are down concurrently for two (2) or more hours.

Attempts to recover servers from the onQ high availability appliances at the same data center in YOUR CITY, USA are unsuccessful.

Any problem that would cause either of the above conditions to be present or there is certain indication that either of the conditions are about to occur.

The important point is that COMPANYs EMT is not dependent on Quorum or any third party to activate the onQ DR appliance. COMPANY has complete discretion on when to recover the protected nodes in the Quorum Data Center.

Business continuity phase (Failover to the Cloud)

Logon to the onQ DR Appliances User Interface

Step 1: Initiate a VPN connection to Hybrid Cloud ensuring Java Script is enabled on your browser and the onQ Appliance is in your browsers Trusted Sites list.

Step 2: Point the browser to the onQ Appliances LAN IP address. You assigned this IP address during the configuration process during the Quick Start implementation process and further outlined in the onQ Admin Guide

Step 3: Type in your Admin User Credentials

Establish a VPN connection to Hybrid Cloud

How you establish a VPN connection to Hybrid Cloud depends on your operating system.

To establish a VPN connection to Hybrid Cloud (Windows 7):

From the Network and Sharing Center, right-click on DR VPN Connection, then Connect.

Type the credentials that your Quorum Support Engineer provided you, then Connect.

Wait a few seconds for your client to establish a VPN Connection to the Quorum Data Center.

To establish a VPN connection to Hybrid Cloud (Windows XP):

From Network Connections, right-click on DR VPN Connection, then Connect.

Type the credentials that your Quorum Support Engineer provided you, then Connect.

Wait a few seconds for your client to establish a VPN Connection to the Quorum Data Center.

If authentication is successful, the Connection window minimizes to your taskbar as a dual computer icon. Run your mouse over it to see the status of the connection.

To establish a VPN connection to Hybrid Cloud (Mac OS X):

Do one of the following:

Choose the VPN configuration from the VPN menu.

From the Network page, select your VPN connection from the connection list, then click the Connect button.

Wait a few seconds for your client to establish a VPN Connection to the Quorum Data Center.

Unlock UI

The onQ Portal has pages that have icons that lock and unlock. This feature ensures that youdon'tmake an unintentional change to your protection configuration. Only users of role Administrator or Operator can unlock these icons; users of role Monitor cannot unlock these icons. After you unlock the icons they become command buttons, enabling you to perform certain actions. To unlock these icons, simply use the padlock icon to toggle between these two states:

For example, if the padlock icon shows a closed padlock and is labeled Unlock to enable actions, you can view the status information related to protected nodes, but you cannot perform any actions.

To unlock the UI for a given page:

Go to the page you want to unlock:

(DR) DASHBOARD tab > DR STATUS page

(DR) DASHBOARD tab > RECOVERY NODES page

Locate the closed padlock icon in the lower left corner.

Click on the icon to switch the icon to the open state, assuming the icon is in the closed state.

The icon changes to an open padlock and is enlabeled. Some of the fields in the table are revealed to be clickable buttons:

Create on-demand Recovery Nodes:

At the DASHBOARD tab > RECOVERY NODES page.

The amount of space required appears in the Space Required column. The RN Space Available is shown just above the Unlock button. Click on the recovery node name, then Build the Recovery Node.

Start Recovery Nodes on DR Appliance

Starting an individual Recovery Node (RN), a group of RNs, or all the RNs on the DR Appliance are done in a similar way.

Use this procedure in conjunction with the appropriate failover or failback procedures found in the onQ Admin Guide.

When you start an RN you are indirectly shutting down the PN and PN proxy (VM) respectively and disabling backups for that PN.

Note: So as to avoid network conflicts, if the corresponding PN is already running in production, the onQ Portal will not let you start the RN in production. In that case, the onQ Portal displays the following error message:

Its possible to allocate resources to recovery nodes beyond what is physically available on the onQ Appliance. If youdon'thave enough memory to start all the recovery nodes, you have two choices:

You can start your recovery nodes individually. The boot order must take into account any interdependencies.

.

To start all recovery nodes in production mode:

(Optional) Consider suspending RN tests to free up resources.

From the DR Appliance, go to DASHBOARD tab > DR STATUS page.

Unlock the page

Click on the START All RNs button.

Click Apply to start all the DR recovery nodes.

When you start in production mode, the startup order will take into account the order you earlier specified startup dependencies.

To start a group of recovery nodes in production mode or test mode:

From the DR Appliance, go to DASHBOARD tab > RECOVERY NODES page.

Unlock the page.

For the group, click the Power State (

) button.

Select either Power On: Test Network or Power On: Production Network.

The startup order will take into account the order you specified in Startup Dependencies in the PROTECTION CONFIGURATION/Advanced section of the portal.

To start an individual recovery node in production mode:

You might be prompted to install application-specific software licenses upon startup of recovery nodes.

Consider suspending RN tests to free up resources.

From the DR Appliance, go to DASHBOARD tab > RECOVERY NODES page.

Locate the Power State column.

When you click the button in the Power State column, a box appears that offers you a choice between the following two alternatives: Power On: Test Network or Power On: Production Network. Choose the Power On: Production Network power option.

Fail over HA Appliance to DR Appliance (Workflow)

If you expect the YOUR CITY site to be down for a significant period of time, or ifyou'veexperienced an onQ Appliance failure and are waiting for a replacement consider the following:

Quorum can ship you the cloud DR Appliance, replacing it with a new DR instance. This onsite DR Appliance assumes the HA role.

Only an HA Appliance (local) can back up recovery nodes. Because your HA Appliance is unavailable, the DR Appliance (remote) must take on this role; otherwise, your recovery nodes, which are now acting as the protected nodes, will not be backed up. Therefore, a failover involves the DR Appliance taking on the HA role and running the recovery nodes.

As soon as your local site is available, you must fail back.

To fail over to remote onQ Appliance:

Step 1: Ensure that the local onQ Appliance is down (powered off) or that protection is off.

When an onQ Appliance boots, protection is automatically off. Protection on an HA Appliance instructs the HA Appliance to perform backups. Given that the HA Appliance (local) still has its role set to HA, protection on that HA Appliance must be off until the failback completes.

Step 2: Log on to the DR Appliances onQ Portal.

Step 3: Start all the recovery nodes on the DR Appliance (remote onQ Appliance). The boot order must take into account any interdependencies.

If applicable, restore the Oracle database:

Step 4: Back up and restore Oracle.

Step 5: Add a host entry for each protected node, if you have not already done so.

Step 6: Change the DR Appliances role to HA

The recovery nodes on the remote onQ Appliance become the protected nodes; the remote onQ Appliance backs up these recovery nodes

Business recovery phase (Failback from the Cloud)

This section documents the steps necessary to activate business recovery plans to support full restoration of systems at the YOUR CITY site after the disaster has passed and operations return to normal. Coordinate resources to reconstruct business operations at the YOUR CITY, USA location, and to return the onQ HA and DR appliances to normal.

Restore PNs using Quark

The following disaster recovery workflows prompt you to use Quorum Ultimate Automated Recovery Kit (QUARK):

QUARK is an easy-to-use wizard that walks you through two types of restores:

Bare Metal Restore (BMR). You have a server failure or site failure and you want to restore a Recovery PN from scratch.

Incremental/Reversion Restore. You have a server failure and you want to restore to the latest snapshot, or you have a corruption problem and need to restore to a previous snapshot (aka reversion).

You can also restore PN data using two other methods: file level restores (FLR) and Windows Share Restore (WSR). These methods are commonly used in non-disaster recovery scenarios.

QUARK is stored in the DOWNLOADS under APPLIANCE section of the portal.

QUARK supports two boot options:

USB. This media is used for most installs. If for any reason QUARK cannot inject the necessary drivers, you can add drivers to this USB for use during the BMR without needing additional media.

ISO. Burn this ISO to a CD-ROM or, in the case of virtualization, save this ISO to your network or library and point to it from your virtualization infrastructure.

Log on to the onQ Portal.

Goto APPLIANCE CONFIG tab > ADVANCED button > SOFTWARE UPGRADES page > Check for Updates button. This step ensures that you have the most current version of QUARK for your onQ Appliance.

Go to APPLIANCE CONFIG tab > ADVANCED button > DOWNLOADS, then do one of the following.

USB method. Select the Quark_usb_BCV-<release>-<date>-<time>.zip, Download, then save the .zip file to a USB key.

ISO method. Select the Quark_BCV-<release>-<date>-<time>.iso, Download, then save the .iso file to your virtualization infrastructure, or burn it to a CD-ROM.

Now you are ready to perform the BMR

To restore data to your PN:

In this procedure, Recovery PN (also called target) refers to the server to which you are restoring. This Recovery PN can be a physical or virtual machine.

If your Recovery PN is a virtual machine, this procedure assumes that you have a snapshot of your virtual machine or a template from which to deploy a clone. In any case, this virtual machine is only a shell onto which QUARK can load its snapshot(s). This virtual machine need not look like your Recovery PN. Simply reconfigure the machines BIOS to boot QUARK, then QUARK will walk you through the BMR.

Note: Although QUARK attempts to inject the necessary drivers for your system, its a best practice to have a copy of these drivers as part of your business continuity plan.

Your Recovery PN must have enough resources to accommodate the new data. To be safe, ensure that your Recovery PN has at least the same amount of disk space and memory as the failed PN.

Verify that your platform is supported. Go to Support in onQ Release Notes.

Configure Hardware RAID on the Recovery PN.

If necessary, configure any hardware RAID controllers in the BIOS of the Recovery PN. For instructions, refer to the hardware manufacturers documentation.

Launch the QUARK wizard:

Configure the Recovery PNs BIOS to boot QUARK.

Boot the Recovery PN to load the QUARK image. The Start BMR page appears.

From the Start BMR page, allow the QUARK wizard to check for basic network and disk controller drivers on which QUARK depends, then Next.

If the QUARK wizard detects a missing driver, the Add Drivers page appears.

Note: To manually launch this Add Drivers utility, run the following command from the command prompt:

# QUARK_Wizard.exe AddDriver.

(If necessary) In the Add Drivers page, upload the missing network driver, then Next.

If the wizard detects an existing image, the Continue onQ Restore page appears.

Proceed to Step 6. If the wizard detects a raw disk, the Network Configuration page appears. Skip to Step 7.

From the Continue onQ Restore page, select the Continue with onQ Restore workflow, then Next.

If the wizard detects a DHCP server, the wizard displays the assigned default IP address and netmask in the Network Configuration page.

If DHCP is not enabled on the network, the wizard displays the assigned default Windows workgroup address.

(Optional) In the Network Configuration page, change the default network configuration, then Next:

Select a NIC from the Interface drop-down list.

Select the Set Static IP check box to set the default IP address to a static IP.

The Log on to onQ page appears.

In the Log on to onQ page, type the IP Address and credentials of the onQ Appliance that manages the Recovery PNs snapshot to which you want to restore, then Next.

The onQ Appliance analyses the snapshots that belong to the Recovery PN, and determines the restore types (Full BMR or Incremental) that are available for this recovery, then displays the Select PN page.

If this process fails, browse the list of error messages in (Step 4) To interpret QUARK Errors: to correct the problem.

In the Select PN page, select the restore type and the Recovery PN from the drop-down list of PNs that the onQ Appliance manages, then Next.

If the onQ Appliance doesnt have the incremental backups that are needed to perform the restore (that is, the PN is a new server and does not have any data on its disk), the wizard displays the Full BMR option only; otherwise, both options are available.

Use the Incremental/Reversion restore type to:

Restore to the latest version. Simply choose the latest snapshot.

or

Revert to a previous version (also known as a reversion). Simply select the snapshot that represents a point in time. This scenario is ideal when you know that the latest version is corrupt.

In the Select Snapshot page, select the snapshot to which you want to restore the Recovery PN, Next, then Yes to reset the disks. From the Begin PN Restore page, click Next to confirm your PN and snapshot selection. (Full BMR) The following dialog appears, and the QUARK Disk Configuration wizard launches.

Note: Resetting the disks ensures that you do not write PN data on top of corrupted data. Reformatted and clean disks are ideal. However, ifyou'vealready created the disk partitions outside of QUARK using a disk partition tool, you can skip this task.

Configure the Recovery PN with the same disk configuration as the original PN, then Close:

Note: If a USB/CD-ROM takes a PN drive letter, QUARK automatically reassigns the USB/CD-ROM a different drive letter. Your may also use the Advanced option to get direct access to DISKPART and manually adjust the volumes and partitions.

For every volume that appears in the PN Layout field, add a corresponding volume. The volume can be either a drive or a mount point. Optionally, you can change the Volume restore size of the drive or mount point; the default is the allocated size as shown in the onQ Portal.

If the drive is a boot drive, specify the partition style when the QUARK Disk Configuration prompts you:

If you want the PN to boot with BIOS, select MBR/BIOS.

If want the PN to boot with UEFI, select GPT/EFI.

Optionally, change the partition size.

Select the volumes that you want to restore. Typically youll want to select all the volumes; however, you might want to restore only the operating system so as to get the PN up and running quickly as demonstrated in the example below; you can always restore the remaining volumes later using QUARK, File Level Restore (FLR), Windows Share Restore (WSR), or a third party tool.

If you want to delete a volume or view volume information (for example, file system), right-click on the volume and Delete Partition and Volume Information respectively.

When the Begin PN Restore page appears, click Next.

If you receive the following error message, the volumes that you added do not match the PN layout. However, as outlined in Step c, you do not need to restore all the volumes.

Wait while the wizard formats the drives. This process can take several minutes

Note: If you accidently close the QUARK Disk Configuration wizard or you need to reconfigure your disks before you restore, you can launch the wizard from the command prompt:

# QUARK_Wizard.exe showDiskConfig

The PN Restore Progress page appears

12 Wait from the onQ Appliance to restore the Recovery PN to the snapshot you selected.

If the onQ Appliance detects a problem, the QUARK wizard aborts the BMR, then displays a status message. Browse the list of error messages in (Step 4) To interpret QUARK Errors: to correct the problem.

Do the following:

From the P2P Adjust OS Wizard, select the Recovery PNs operating system and, when prompted, choose Adjust the OS to the new hardware automatically, then Next.

If the Recovery PNs hardware is different from the image, the wizard needs to inject the necessary drivers into the image to adjust for this change.

Your snapshot might contain the necessary drivers in its cache; if so, this wizard will inject those drivers; otherwise, youll need to supply the drivers.

When prompted, click Yes apply the changes physically, then Next:

Select Reboot the system or Shutdown the system to complete the restore, then Finish.

Troubleshooting

If you reboot the PN without adjusting for new hardware, you can perform the adjustments after the restore completes. Simply launch the wizard again by repeating Step 3 and Step 4, then select the Perform OS Adjustment workflow from the Continue onQ Restore page.

If your PN will not boot, go to (Step 3) To manually adjust a PNs operating system for new hardware:.

If after performing a BMR your PN does not boot (BSOD), youre missing boot-critical drivers. In the rare case that the P2P tool did not properly adjust your PNs operating system for the new hardware, you need to manually adjust the operating system using the P2P Adjust OS Wizards Set parameters for the OS adjustment option. Before you begin, retrieve the necessary drivers.

Launch the QUARK wizard:

Configure the Recovery PNs BIOS to boot QUARK.

Boot the Recovery PN to load the QUARK image. The Start BMR page appears.

From the Start BMR page, allow the QUARK wizard to check for basic network and disk controller drivers on which QUARK depends, then Next.

From the Continue onQ Restore page, select the Perform OS Adjustment workflow.

From the P2P Adjust OS Wizard, select the Recovery PNs operating system, then Next.

Choose Set parameters for the OS adjustment, then Next.

Specify the path to the driver, select the Inject all necessary drivers from the specified repository check box and the Keep the latest driver version check box, then Next.

Select the driver, then Next.

Select the Yes, apply the changes physically radio button to make the necessary adjustments, then Next.

Click Finish. Now that your PN has the correct boot-critical drivers, your PN should now boot.

Select Reboot the system or Shutdown the system to complete the adjustments, then Finish.

Plan review and maintenance

This plan is intended to be a living document and as such must be reviewed on a regular basis. The plan will be reviewed semi-annually and exercised on an annual basis. Automatic testing is a feature of the Quorum onQ but running a recovery node is a good way to train new COMPANY employees on the DR recovery capabilities.

Test Mode

When you start a recovery node, you have an opportunity to run the recovery node in test mode. You can only access a recovery node in test mode through the integrated console.

Running a recovery node in test mode means that it runs in a limited private network environment isolated from your production facilities.

The reason for running in test mode is to ensure, without risk to ongoing operations, that the recovery node is intact and will run in the event of a disaster. Keep in mind that this capability is in addition to automatic testing.

The onQ Appliance itself occasionally and briefly runs a recovery node in test mode as part of an internal quality-assurance process. However, best practice is to check your recovery nodes periodically, just as you would ordinary backups.

While an RN is running in test mode the corresponding PN also continues to run although the backup process is suspended. Backups (and subsequent rebuilds of the RN) resume according to current settings when the test mode has finished.

Frequency of plan update:

Quarterly or when there is a change in personnel

Provide hard copy of plan to all team members. Team members must store copy at home, in a personal car, or electronically via a hand-held device or laptop computer.

Regularly review and update information in the disaster recovery plan. Communicate with the Emergency Management Coordinator to get up-to-date information periodically.

Hold initial team meeting to get team members acquainted with the plan and hold annual/semi-annual meetings to review the plan on an ongoing basis.

Issue:

Dell Live Image

Reason:

Dell Live Image

Resolution:

Links for DSP live ISO and instructions for creating the thumb drive for SLI.

DSP LiveUser Guide:

http://downloads.dell.com/FOLDER01960556M/1/SLI_20_UserGuide.pdf

DSP Live ISO image

http://downloads.dell.com/FOLDER01960516M/1/SLI20_A01.iso

Boot to CentOS, setup the NICs using the Network Connections

After setting up NICs, run a DSET from the DSP live to pull additional information we can review for NIC or other issues.

To run the DSET, they can select the DSET (fourth icon from the bottom in the first column)

Terminal window will close when complete

Select File System

Scroll down to the tmp folder

The DSET report zip file will be saved in the data folder

It is recommend to have a second Thumb drive to save the zip file on

Escalation:

Standard Escalation Procedure

Issue:

Linux Out of Memory

Reason:

Linux Out of Memory

Resolution:

Debugout-of-memorywith/var/log/messages

The kernel will have logged a bunch of stuff before this happened, but most of it will probably not be in/var/log/messages, depending on how your(r)syslogdis configured. Try:

grep oom /var/log/*grep total_vm /var/log/*

The former should show up a bunch of times and the latter in only one or two places. That is the file you want to look at.

Find the original "Outofmemory" line in one of the files that also containstotal_vm. Thirty second to a minute (could be more, could be less) before that line you'll find something like:

kernel: foobar invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

You should also find a table somewhere between that line and the "Outofmemory" line with headers like this:

[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name

This may not tell you much more than you already know, but the fields are:

pid The process ID.

uid User ID.

tgid Thread group ID.

total_vm Virtualmemoryuse (in 4 kB pages)

rss Residentmemoryuse (in 4 kB pages)

nr_ptes Page table entries

swapents Swap entries

oom_score_adj Usually 0; a lower number indicates the process will be less likely to die when the OOM killer is invoked.

You can mostly ignorenr_ptesandswapentsalthough I believe these are factors in determining who gets killed. This is not necessarily the process using the mostmemory, but it very likely is. For more about the selection process, see here. Basically, the process that ends up with the highest oom score is killed -- that's the "score" reported on the "Outofmemory" line; unfortunately the other scores aren't reported but that table provides some clues in terms of factors.

Again, this probably won't do much more than illuminate the obvious: the system ran out ofmemoryandmysqldwas choosen to diebecause killing it would release the most resources. This does not necessary meanmysqldis doing anything wrong. You can look at the table to see if anything else went way out of line at the time, but there may not be any clear culprit: the system can run out ofmemorysimply because you misjudged or misconfigured the running processes.

Escalation:

Standard Escalation Procedure

Issue:

XenCenter Console Not Showing

Reason:

XenCenter Console Not Showing

Resolution:

Restart XenCenter and the xapi service (Check with team before doing this)

XAPI has been restarted.

Next: Check the onQ's on that blade.

Reboot blade if necessary.

Escalation:

Standard Escalation Procedure

Issue:

XenServer License Expiration

Reason:

XenServer License Expiration

Resolution:

XenCenter may show that the XenServer license has expired. If you go toToolsmenu and click onLicense Manageryou may see something similar to the below:

Cause:

In most cases this is related to the free version of XenServer and it just means that there is no access to support from Citrix. In this case, if all other components and VMs work as expected, this error can be ignored

Escalation:

Standard Escalation Procedure

This release includes the following new features:

Centralized enrollment for agentbased PNs. If you have a large number of agentbased PNs, consider using the onQ centralized installation manager to enroll them in one large batch. With the onQ centralized installation manager, you do not need to log on to each PN separately to install the onQ Service. For platform support, go to Centralized Enrollment Support. For instructions on enrollment, go to (Agentbased Centralized PNs) Enroll protected nodes.

Agentless enrollment support for vCenter. In addition to enrolling specific ESXi servers, you can now enroll vCenter so as to protect all the virtual machines that vCenter manages across multiple ESXi servers, providing you the quickest deployment possible and vMotion support. For more information, go to (Agentless Linux/Windows PNs) Enroll protected nodes. Keyboard Support

onQ Flex enhancements. In this release, RNs can now have an RN type. Also, onQ tracks all RN changes, including resources. These improvements to RN management provide you more finetuning of your RTO (Recovery Time Objective) and costs for that objective in an effort to conserve Hybrid Cloud and minimize upfront deployment costs. If you want an onQ Flex configuration, contact Quorum Support. For more information about an onQ Flex configuration and the related onQ Portal changes, go to (onQ Flex) Modify RN type and/or RN build policy and (Start Here) Upgrade Support and Requirements.

Relaxed restrictions on agentless enrollment and concurrent backups. Agentless enrollment now supports the ability for onQ to run three concurrent PN backups as these PNs can now be on different ESX/ESXi hosts.

onQ policy prevents 100% repository disk space utilization. If an HA Appliances repository disk space utilization exceeds 85%, onQ disables backups globally and resumes backups when disk space utilization improves (less than 85%); however, during this time, you can perform immediate backups. This policy attempts to prevent your HA Appliance from using 100% of its repositorys disk space.

Restart Protection option. onQ Portal streamlined the capability to restart protection from the dropdown menu. Restart is now only one step! You no longer need to stop protection, then start protection (two steps).

onQ Portal performance improvements. Performance improvements include faster status queries including tooltips, DR transfer status, and Dashboard status updates.

AZERTY keyboard support. The RN console is now compatible with an AZERTY keyboard. See for a list of supported keyboards.

Dell G13 Hardware. This release supports Dells next generation hardware, providing you higher performance.