Skip to content

Latest commit

 

History

History
105 lines (63 loc) · 5.19 KB

create-hadoop-cluster-run-job.md

File metadata and controls

105 lines (63 loc) · 5.19 KB
Raw

Demo: Creating a Hadoop Cluster with Cloud Dataproc

Cloud Dataproc is a fast, easy-to-use, fully-managed cloud service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way. Operations that used to take hours or days take seconds or minutes instead. Create Cloud Dataproc clusters quickly and resize them at any time, so you don't have to worry about your data pipelines outgrowing your clusters.

This demo shows you how to use the Google Cloud Platform (GCP) Console to create a Google Cloud Dataproc cluster, run a simple Apache Spark job in the cluster, then modify the number of workers in the cluster.

Confirm Cloud Dataproc API is enabled

To create a Dataproc cluster in GCP, the Cloud Dataproc API must be enabled. To confirm the API is enabled:

Click Navigation menu > APIs & Services > Library:

Type Cloud Dataproc in the Search for APIs & Services dialog. The console will display the Cloud Dataproc API in the search results.

Click on Cloud Dataproc API to display the status of the API. If the API is not already enabled, click the Enable button.

If the API's enabled, you're good to go.

Create a cluster

In the Cloud Platform Console, select Navigation menu > Dataproc > Clusters, then click Create cluster.

Set the following fields for your cluster. Accept the default values for all other fields.

Field Value
Name example-cluster
Region global
Zone us-central1-a
Note: A *Zone* is a special multi-region namespace that is capable of deploying instances into all Google Compute zones globally. You can also specify distinct regions, such as `us-east1` or `europe-west1`, to isolate resources (including VM instances and Google Cloud Storage) and metadata storage locations utilized by Cloud Dataproc within the user-specified region.

Click Create to create the cluster.

Your new cluster will appear in the Clusters list. It may take a few minutes to create, the cluster Status shows as "Provisioning" until the cluster is ready to use, then changes to "Running."

Submit a job

To run a sample Spark job:

Click Jobs in the left pane to switch to Dataproc's jobs view, then click Submit job:

Set the following fields to update Job. Accept the default values for all other fields.

Field Value
Cluster example-cluster
Job type Spark
Main class or jar org.apache.spark.examples.SparkPi
Arguments 1000 (This sets the number of tasks.)
Jar file file:///usr/lib/spark/examples/jars/spark-examples.jar

Click Submit.

How the job calculates Pi: The Spark job estimates a value of Pi using the Monte Carlo method. It generates x,y points on a coordinate plane that models a circle enclosed by a unit square. The input argument (1000) determines the number of x,y pairs to generate; the more pairs generated, the greater the accuracy of the estimation. This estimation leverages Cloud Dataproc worker nodes to parallelize the computation. For more information, see Estimating Pi using the Monte Carlo Method and see JavaSparkPi.java on GitHub.

Your job should appear in the Jobs list, which shows your project's jobs with its cluster, type, and current status. Job status displays as Running, and then Succeeded after it completes.

View the job output

To see your completed job's output:

Click the job ID in the Jobs list.

Check Line wrapping or scroll all the way to the right to see the calculated value of Pi. Your output, with Line wrapping checked should show the value of Pi.

Your job has successfully calculated a rough value for pi!

Optional Demo: Update a cluster

To change the number of worker instances in your cluster:

  1. Select Clusters in the left navigation pane to return to the Dataproc Clusters view.

  2. Click example-cluster in the Clusters list. By default, the page displays an overview of your cluster's CPU usage.

  3. Click Configuration to display your cluster's current settings.

  4. Click Edit. The number of worker nodes is now editable.

  5. Enter 4 in the Worker nodes field.

  6. Click Save.

Your cluster is now updated. Check out the number of VM instances in the cluster

To rerun the job with the updated cluster, you would click Jobs in the left pane, then click SUBMIT JOB.

Set the same fields you set in the Submit a job section:

Field Value
Cluster example-cluster
Job type Spark
Main class or jar org.apache.spark.examples.SparkPi
Arguments 1000 (This sets the number of tasks.)
Jar file file:///usr/lib/spark/examples/jars/spark-examples.jar

Click Submit.

Congratulations!

Now you know how to use the Google Cloud Platform Console to create and update a Dataproc cluster and then submit a job in that cluster.