Oracle Cloud Infrastructure Documentation

Post Node Replacement Checks

Utility Nodes

  1. After replacing an active Ambari node (for example, un0), and you want to move Ambari back to the original node (for example, un0), complete the following:
    1. Sign in to the mn0 node as root user, and then run: 
      sudo python /home/opc/cloud/flask-microservice/bigdataservice/devops/run_ambari_failover.py
    2. Restart all services on the node un0, un1, and any other flagged services from Ambari.
  2. Ensure MySQL replication is working on the standby node (un1): 
    sudo su

    Launch MySQL prompt:

    mysql

    Run:

    "SHOW SLAVE STATUS \G"

    There should be no errors in the output.

  3. Access Apache Ambari., and ensure the correct master MySQL node is added for the MySQL host properties for Hive, Ranger, Oozie, and Hue.
    Config Key
    hive-site javax.jdo.option.ConnectionURL
    oozie-site oozie.service.JPAService.jdbc.url
    hue-desktop-site db_host
    ranger-admin-site ranger.jpa.jdbc.url
    ranger admin-properties db_host
    ranger-env ranger_privelege_user_jdbc_url
  4. Restart all required services in Ambari.
  5. Continue to monitor MySQL replication status for at least a couple of days using step 2.
  6. Create a service request if issues are observed.

Master Node

  1. Ensure active and standby Namenode status is reflected in Ambari and both are reported healthy.
  2. Ensure active and standby Resource Manager status is reflected in Ambari and both are reported healthy.
  3. Restart all required services in Ambari.
  4. Create a service request if issues are observed.