Private Endpoints Use Cases

This is the most common use case for Data Flow and private endpoints.

Integrate Data Flow with on-premises data sources through private endpoints ensures secure and efficient data processing across hybrid environments.

By using private network connectivity options such as Site-to-Site VPN or FastConnect, you can seamlessly connect Data Flow applications to data repositories hosted in the on-premises infrastructure. This gives robust, low-latency communication while maintaining strict security boundaries. Use it for use cases that require secure data access and processing across cloud and on-premises environments.

A key element of this setup is the DNS resolution, which now takes place within the Private Access Gateway. The configuration provides a DNS name (fully qualified domain name, or FQDN) for the private endpoints, not the private IP address itself. If you've configured your network setup for DNS, your hosts can access the private endpoint using the FQDN. Data Flow supports network security groups (NSGs) with its resources. You can request that the Data Flow service set up the private endpoint in an NSG within your VCN. NSGs let you write security rules to control access to the private endpoint without knowing the private IP address assigned to the private endpoint.

When connectivity between the OCI customer VCN and the on-premises network has been established, for the Data Flow application to operate correctly, it's necessary to associate the private IPs in the on-premises network with a private Domain resolver in the OCI customer VCN.

To illustrate this part, we're using two networks connected using the OCI Network Visualizer. The hdi-dataflow-VCN private subnet is connected to a disdemo-DRG dynamic routing gateway attachment.

From this point onwards, the relevant part is to decide how the DNS resolution occurs. In this example, we created two DNS resolvers:

• One attached to hdi-dataflow-vcn standard view (industrial.com)
• Another in a customer-private-view with the resolution for the domain oraclevcn.com, indicating this is a private subnet in an OCI VCN.

In the "Private resolver details" of the selected VCN, select Manage private views. Two options exist:

• Protected private view, in our case, creating the hdi-dataflow-vcn private view (Order 1).
• Using Manage private views again, create the customer-private-view (Order 2).

Navigating to the first private view hdi-dataflow-vcn, under the Private Zones, we created the industrial.com zone with Primary Zone type.

Similarly, navigate to the customer-private-view and verify that a Private zone is already created within the Private subnet of the referred VCN.

Under the industrial.com zone in the Private view, select Manage records to create a new record for the IP of an external MySQLServer running in the customer on-premises network. For example:

Name: MySQL_Customer_OnPremises(.industrial.com)

Type: A - IPv4 address

TTL in seconds: 3600

RDATA/Answers

Address: 10.xx.xx.xx (the IP address of the on-premises resource).

Similarly, do the same for private access within another OCI private VCN that has been peered with the current VCN, such as the one in the previous image, through the local peering gateway (LPG). So, in the internal zone privatesubnet.dfarchitecture.oraclevcn.com add record to the IP of another MySQL instance running in the private subnet of the peered VCN. For example:

Domain: mysql-private.privatesubnet.dfarchitecture.oraclevcn.com

Type: A - IPv4 address

TTL in seconds: 3600

RDATA/Answers

Address: 12.xx.xx.xx (the IP address of the resource in the peered VCN).

Again, the DNS resolution is the key factor to be addressed when using Data Flow private endpoints. Resolve them using record types appropriate to the downstream network. For these types and configurations, follow the Managing DNS zones documentation.

Data Flow supports private endpoint integration for seamless cross-regional data access using remote VCN peering through an upgraded Dynamic Routing Gateway (DRG).

This configuration lets Data Flow applications in one region to securely connect to data sources hosted in another region's VCN, as depicted in the following image:

You can ensure high-performance, low-latency data transfers while maintaining a robust security posture by using the upgraded DRG's advanced capabilities, including transitive routing and centralized connectivity.

Also, with OCI’s multicloud networking capabilities, you can extend this setup to connect with, for example, Microsoft Azure. By using OCI-Azure Interconnect or a similar multicloud connectivity framework, you can enable Data Flow to process data stored in Azure resources such as Azure Blob Storage or Azure SQL Database, as shown in the following image:

This architecture supports centralized connectivity, transitive routing, and low-latency data transfer between OCI and Azure while maintaining stringent security and compliance standards.

For the Data Flow private endpoint, it matters how the DNS names are resolved before the DRG, using private resolvers, as shown in Connect to On-Premises Resources. A tangible benefit is that the instances for private connectivity in the service, VCN can access a consumer-specified workload without traversing the Internet. Beyond that, the Data Flow private endpoint can extend private connectivity from instances in service VCN to the consumer's on-premises network and other networks accessible through the consumer VCN. From the usability perspective, you can continue interacting with only the service Console (or API) and don't need another interface to enable private access. Despite the flexibility of operation using the Data Flow private endpoint, some limitations exist:

• The default limit for a Data Flow private endpoint is no more than five per tenancy per region.
• If Internet connectivity is required for a Spark Application run with private endpoints enabled, the corresponding DNS zone (for example, Google's APIs/google.zone) needs to be mentioned in the parameter (zones) section for private endpoints under the Application. So, if the zone is allowlisted, the traffic is routed to the customer VCN for resolution. The customer network is responsible for internet connectivity after the packet reaches the consumer gateway VCN. The network traffic is dropped for all other zones not mentioned in the parameter (DNS Zones).

Data Flow can connect to the RAC (real application cluster) or Exadata machine as a client application using SCAN (Single Client Access Name).

The SCAN is a virtual name similar to those used for virtual IP addresses. However, unlike a virtual IP, the SCAN virtual name is associated with the entire cluster rather than an individual node and several IP addresses, not one address only.

When the SCAN proxy feature is enabled, a reverse connection entry point (RCE) is established to handle IP-based redirects. As shown in the following image, a private endpoint VNIC (private endpoint virtual network interface card) is created in the Customer VCN. The RCE private endpoint VNIC is unique for each Data Flow private endpoint setup. One important consideration about the TLS connection to a database cluster is that the database SCAN listeners redirect the network traffic to an FQDN, not the IP address directly. Only the FQDN redirects from a SCAN listener enable TLS. Therefore, configure the database cluster to redirect to an FQDN if TLS is a requirement.

The configuration steps that happen behind the scenes to create the SCAN proxy feature:

• The user configures the SCAN proxy in the Data Flow private endpoint configuration
• The Data Flow updates RCE to include SCAN configuration (the SCAN listener DNS name and port), which provides a new IP (SCAN proxy IP) in the service VCN binding to the same SCAN port
• Data Flow then uses the SCAN proxy IP to create a DNS mapping within the service network, using the original SCAN listener DNS name and the SCAN proxy IP

The previous image shows an example of a RAC Oracle Database system within a private subnet in the customer VCN. The flow of the picture states the sequence used to identify the connectivity:

1. Data Flow starts a connection to the SCAN Proxy endpoint within the service VCN using the DNS name. Data Flow defines a customer RAC connection through the SCAN Proxy by selecting a specific port. RCE SCAN proxy forwards the request to the underlying private endpoint VNIC listener in the customer network. It then inspects the SCAN listener response for an IP of the underlying database cluster instance, creates a Class E NAT IP, and replaces the cluster instance IP with NAT IP in the SCAN proxy response.
2. The Private Access Gateway receives the redirect request from the SCAN listener and automatically translates the local listener IP in the customer's VCN into a mapped IP address, then returns this information to the Data Flow components and makes a connection request to one of the local listeners.

In the Data Flow private endpoint configuration, enter the DNS name of the scan host in the SCAN details section and its associated port number. For example:

DNS name: oracleDB-scan-sub0911000090.dfarchitecture.oraclevcn.com

Port: 16001

When running a Spark serverless execution, it's critical to account for network traffic patterns and isolation to ensure best performance, security, and compliance.

Serverless Spark jobs run within a managed environment where network traffic flows between your application, data sources, and external services. To minimize latency and control traffic, ensure data sources are colocated within the same region and Virtual Cloud Network (VCN) where possible. Here are some more considerations and information about the Data Flow private endpoint setup:

• After an Application run with Data Flow private endpoint resource is attached, the network traffic to the Internet is routed to the your VCN subnet through private endpoint Infrastructure as long as the DNS zone is allowlisted during private endpoint creation. It fails if your VCN doesn't have an Internet Gateway attached. If the DNS zone isn't allowlisted, network traffic is dropped. Network traffic to OCI services, for example, Object Storage in the Oracle Services Network is still routed through Data Flow's VCN.
• When the Data Flow run is running, the Spark Data Flow Application running from any nodes assigned to the tenant start a network connection to the your private resource with the DNS name, for example, customer1.instance1.subnet.oraclevcn.com). This involves a DNS lookup of the DNS Proxy IP assigned to your private resource, created during private endpoint/reverse connection endpoint setup. In a reverse connection, a server starts the connection to a client, letting Data Flow access the resource privately by connecting to a specified endpoint within the your network.
• Your DNS zones in the private views create a proxy that returns a Class E IP address (240.0.0.0-255.255.255.255) for the customer.instance.subnet.oraclevcn.com allocating from a specific CIDR range, for example, 255.33.36.2, as portrayed in the test script in Test the Data Flow Private Endpoint. In this example, Data Flow nodes running your jobs can establish a network connection to 255.33.36.0/24, and a Stateful Egress Rule is created with that destination CIDR range. This means that when a Data Flow instance starts traffic to another host and that traffic is allowed by egress security rules, any traffic that the instance later receives from that host for a period is considered response traffic(ingress) and is allowed. A route table rule is also added to route the Data Flow private endpoint appropriately for the destination CIDR range as 255.33.36.0/24, allocated to that resource.