Patching operations can fail for various reasons. Typically, an operation fails because a database node is down, there is insufficient space on the file system, or the virtual machine cannot access the object store.

• Determining the Problem
  In the Console, you can identify a failed patching operation by viewing the patch history of an Oracle Exadata Database Service on Cloud@Customer system or an individual database.
• Troubleshooting and Diagnosis
  Diagnose the most common issues that can occur during the patching process of any of the Oracle Exadata Database Service on Cloud@Customer components.

If you were unable to resolve the problem using the information in this topic, follow the procedures below to collect relevant database and diagnostic information. After you have collected this information, contact Oracle Support.

• Collecting Cloud Tooling Logs
  Use the relevant log files that could assist Oracle Support for further investigation and resolution of a given issue.
• Collecting Oracle Diagnostics
  

Description: This is an intermittent issue. During virtual machine operating system update with 19c Grid Infrastructure and running databases, dbnodeupdate.sh waits for RHPhelper to drain the connections, which will not progress because of a known bug "DBNODEUPDATE.SH HANGS IN RHPHELPER TO DRAIN SESSIONS AND SHUTDOWN INSTANCE".

/var/log/cellos/dbnodeupdate.trc will show this as the last line:

(ACTION:) Executing RHPhelper to drain sessions and shutdown instances. 
(trace:/u01/app/grid/crsdata/scaqak04dv0201/rhp//executeRHPDrain.150721125206.trc)

[FATAL] [INS-32156] Installer has detected that there are non-readable files in oracle home.
CAUSE: Following files are non-readable, due to insufficient permission oracle.ahf/data/scaqak03dv0104/diag/tfa/tfactl/user_root/tfa_client.trc
ACTION: Ensure the above files are readable by grid.

Cause: Installer has detected a non-readable trace file, oracle.ahf/data/scaqak03dv0104/diag/tfa/tfactl/user_root/tfa_client.trc created by Autonomous Health Framework (AHF) in Oracle home that causes adding a cluster VM to fail.

AHF ran as root created a trc file with root ownership, which the grid user is not able to read.

chown grid:oinstall /u01/app/19.0.0.0/grid/srvm/admin/logging.properties

chown -R grid:oinstall /u01/app/19.0.0.0/grid/oracle.ahf*

chown -R grid:oinstall /u01/app/grid/oracle.ahf*

Description: Adding a VM to a VM cluster completes successfully, however, for Data Guard-enabled databases, the new VM is not added to the nodelist in the /var/opt/oracle/creg/<db>.ini file.

Cause: Data Guard-enabled databases will not be extended to the newly added VM. And therefore, the <db>.ini file will also not be updated because the database instance is not configured in the new VM.

Action: To add an instance to primary and standby databases and to the new VMs (Non-Data Guard), and to remove an instance from a Data Guard environment, see My Oracle Support note 2811352.1.

** CPU Scale Update **An error occurred during module execution. Please refer to the log file for more information

Cause: After provisioning a VM cluster, the /var/opt/oracle/cprops/cprops.ini file, which is automatically generated by the database as a service (DBaaS) is not updated with the common_dcs_agent_bindHost and common_dcs_agent_port parameters and this causes CPU offline scaling to fail.

common_dcs_agent_bindHost=<IP_Address>
common_dcs_agent_port=7070

netstat -tunlp | grep 7070

netstat -tunlp | grep 7070
tcp 0 0 <IP address 1>:7070 0.0.0.0:* LISTEN 42092/java
tcp 0 0 <IP address 2>:7070 0.0.0.0:* LISTEN 42092/java

You can specify either of the two IP addresses, <IP address 1> or <IP address 2> for the common_dcs_agent_bindHost parameter.

Description: After performing the switchover, the new standby (old primary) database remains shut down and fails to restart.

Action: After performing switchover, do the following:

1. Restart the standby database using the srvctl start database -db <standby dbname> command.
2. Reload the listener as grid user on all primary and standby cluster nodes.
   • To reload the listener using high availability, download and apply patch 25075940 to the Grid home, and then run lsnrctl reload -with_ha.
   • To reload the listener, run lsrnctl reload.

After reloading the listener, verify that the <dbname>_DGMGRL services are loaded into the listener using the lsnrctl status command.

Description: If the SCAN listener port for the client network and disaster recovery network (DR network) are different, then Data Guard (DG) configuration fails during verification phase of create data guard association.

Action: Use the same SCAN listener ports (or default port) on all networks. To fix the listener port after the cluster has been configured, run the GI home/bin/srvctl modify listener -listener listener_name -endpoints endpoints command. For more information, see srvctl modify listener in the Oracle Real Application Clusters Administration and Deployment Guide.

[FATAL] [DBAAS-60022] Command '/u02/app/oracle/product/23.0.0.0/dbhome_3/bin/srvctl 'start' 'service' '-db' 'db23ano' '-service' 'db23ano_PDBJULY242.paas.oracle.com'' has failed on nodes [localnode].

Action: If the Grid Infrastructure version is 23.4.0.24.05, upgrade to version 23.5.0.24.07 to resolve this issue.

Description: PDB creation might fail for subset of PDBs when multiple PDBs are getting created in parallel.

ORA-03113: end-of-file on communication channel

Action: Retry the failed PDB creation.