Oracle Cloud Infrastructure Documentation

Python Based Bootstrap Helper Functions

Python script based bootstrap helper functions exist to update Ambari configurations.

These helper functions help you:
  • Update the configuration of an Hadoop component in the cluster
  • Update the configuration that's applicable to a particular host or group of hosts
  • Obtain the configuration value and take actions based on the value
  • Export configuration from a config type (for example, HDFS core-site.xml)
  • Add components such as Zookeeper, Namenode, and Resource Manager to the master nodes in an HA cluster.
  • Remove components such as DataNode and NodeManager.
Note

  1. You can run these helper functions only from predefined method in the Python script.
  2. Python 2.7 is the supported python version in the Python script.
  3. You must manage the Python script at your own risk because it can lead to microservice shut down.

To run a bootstrap script, see Running the Bootstrap Script.

For more information on Python bootstrap script helper functions, see:

Bootstrap Python Script Helper Functions

For more information on Python bootstrap script, see Python Based Bootstrap Helper Functions.

For Python based bootstrap helper function examples, see Python Script Examples.

To run a bootstrap script, see Running the Bootstrap Script.

Category Helper Function Functionality
Config Groups createConfigGroup(service_name, hosts, config_group="Default") This helper function creates a config group for a service with a list of hosts. You must provide the service name, list of hosts, and config group name as input.
removeConfigGroup(service_name, config_group="Default") This helper function removes the config group in a service. You must provide the service name and config group name as input.
addHostsInConfigGroup(service_name, hosts, config_group="Default" This helper function adds hosts to the config group in a service with a list of hosts. You must provide service name, list of hosts, and config group name as input.
removeHostsInConfigGroup(service_name, hosts, config_group="Default" This helper function removes hosts to the config group in a service with a list of hosts. You must provide service name, list of hosts, and config group name as input.
listConfigGroups(service_name) This helper function displays all the config groups in a service. You must provide service name.
getHostsInConfigGroup(service_name, config_group="Default" This helper function displays all the hosts in the config group in a service. You must provide config group and service name.
importConfigGroup(service_name, from_config_group, to_config_group) This helper function clones the configs from one config group to another config group in a service. You must provide from config group, to config group, and service name.
Config updateConfig(service_name, config_type, config_properties, config_group="Default", config_meta={})

This helper function updates the config properties to the config_type file of config_group in a service. You must provide service name, config type, config properties, config group, and config meta.

Note : config_properties is a map of config key and value pairs.

config_meta is map of config key and config metadata. Possible type values in config meta are PASSWORD and TEXT with default value as TEXT.

For example: helper.updateConfig(config_group="test", service_name="HDFS", config_type="core-site", config_properties={"fs.trash.interval": "400", "test.password":"XXXX"}, config_meta={"test.password": {"type":"PASSWORD"}})

removeConfig(service_name, config_type, config_name, config_group="Default") This helper function removes the config from the config type file of config group in a service. You must provide config group, config type, config name, and service name.

For example: helper.removeConfig(service_name="HDFS", config_type="core-site", config_name="fs.trash.interval")
getConfig(service_name, config_type, config_name, config_group="Default") This helper function displays the config property value of config name in the config type file of a config group in a service. You must provide config group, config type, config name, and service name.
restartStaleConfig() This helper function restarts the services where configs are stale.
exportConfig(service_name, config_type, config_group="Default") This helper function displays all the configs in config type file of config group in a service. You must provide config group, config type, and service name.
Shell execution runShellCommandOnAllNodes(command) This helper function executes shell command on all the cluster nodes. You must provide the command.
runShellCommandOnNode(command, host) This helper function executes shell command on the requested host. You must provide the command and host.
Utility getClusterName() Displays cluster name.
getMasterNodesIps() Displays master nodes lps.
getWorkerNodesIps() Displays worker nodes lps.
getUtilityNodesIps() Displays utility node lps.
getQueryServerNodesIps() Displays query server nodes lps.
getComputeOnlyWorkerNodesIps() Displays compute only worker node lps.
getEdgeNodesIps() Displays Edge node lps.
getAllNodesIps() Displays all nodes lps.
getEventType() Displays the event type. Possible event type values are CreateBdsInstance, AddWorkerNodes, StartBdsInstance, ChangeShape, on-demand.
getLogger()

This helper function returns the logger instance which you can use to log info and error logs. For example, getLogger().info("log info msg")
get_last_added_host_names()

Returns Python list of hostnames added in last "Add Node" operation.

Returns None, if no new hosts are added to the cluster.
get_last_added_host_ips()

Returns Python list of private IPs added in last "Add Node" operations.

Returns None, if no new hosts are added to the cluster.
executeAmbariFunc(method, path, payload=None, params=None, headers=None)

This helper function runs Ambari REST API based method type, path, and payload. You must provide method, path, payload(If Any), params(If Any) and headers.

Parameters:

  • method: Possible values: GET, POST, PUT, and DELETE.
  • path: Relative path
  • payload: Request payload If any
  • params: Request params If any
  • headers: Request headers If any
Remote JMX updateRemoteJmx(service_names=None, component_names_mapping=None, enable=True) This helper function enables or disables the remoteJMX metric.
  • service_names: Optional list of services to be enabled with remoteJMX , service names mentioned in <remote-jmx-section>. If no value is passed, the operation is applied to all services
  • service_to_components: Optional Service to components mapping. If no value is passed, the operation is applied to all components of the selected services.
  • enable: True for enabling remoteJMX, False for disabling remoteJMX.
getRemoteJmxInfo() This helper function returns:
  • Map of service → component → remoteJMX info
  • This function only returns this mapping for services available in the Ambari UI.
Add components add_component_to_host(request_dict)

This function takes a Python dictionary and adds components to the hosts mentioned in the dictionary. Supported components are Namenode(NN), ResourceM,anager(RM) and Zookeeper_Server.

Example:

request_dict = {"NAMENODE": {
                "hosts": ["h1", "h2"]
                },
                "RESOURCEMANAGER": {
                "hosts": ["h1", "h2"]
                }}

Caution: Support for NN and RM are only for HA clusters and to the hosts having default config groups.
Remove components remove_component_from_host(request_dict)

This function takes a Python dictionary and removes components from the hosts mentioned in the dictionary with timeouts. Supported components are Datanode and NodeManager.

Example:

request_dict = {"DATANODE": {
                "hosts": ["h1", "h2"],
                "timeout_minutes": 30(default 30)
                },
                "NODEMANAGER": {
                "hosts": ["h1", "h2"]
                }}

list of supported service names = [HDFS, YARN, MAPREDUCE2, TEZ, HIVE, OOZIE, ZOOKEEPER, AMBARI_METRICS, RANGER, HUE, KAFKA, KERBEROS, ODHUTILS, SPARK3, HBASE, TRINO, REGISTRY, FLINK, JUPYTERHUB]

list of supported event types = ["CreateBdsInstance", "AddWorkerNodes", "StartBdsInstance", "ChangeShape"] type = ["PASSWORD", "TEXT"]

Python Script Examples

For more information on the Python helper functions, see Bootstrap Python Script Helper Functions.