Skip to content

Commit bcf6e21

Browse files
jimrightjimright
authored
Add GCP datalake example (#130)
Signed-off-by: Jim Enright <jenright@cloudera.com>
1 parent 6d83599 commit bcf6e21

File tree

7 files changed

+291
-0
lines changed

7 files changed

+291
-0
lines changed

‎public-cloud/gcp/datalake/.gitignore‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
ansible-navigator.log
16+
runs
17+
context

‎public-cloud/gcp/datalake/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# CDP Public Cloud - Environment and Datalake on GCP Base Example
2+
3+
> Constructs a CDP Public Cloud Environment and Datalake. Generates via Ansible the GCP infrastructure and CDP artifacts, including SSH key, cross-account Service Account, GCS buckets, etc.
4+
5+
## Requirements
6+
7+
To run, you need:
8+
9+
* Docker (or a Docker alternative)
10+
* GCP Service Account provisioning credentials (set via `GCP_SERVICE_ACCOUNT_FILE`)
11+
* CDP credentials (set via `CDP_PROFILE`)
12+
13+
## Set Up
14+
15+
First, set up your `ansible-navigator` aka `cdp-navigator` environment -- follow the instructions in the [NAVIGATOR document](https://github.com/cloudera-labs/cldr-runner/blob/main/NAVIGATOR.md) in `cloudera-labs/cldr-runner`.
16+
17+
Then, clone this project and change your working directory.
18+
19+
```bash
20+
git clone https://github.com/cloudera-labs/cloudera-deploy.git; cd cloudera-deploy/public-cloud/gcp/datalake
21+
```
22+
23+
## Configure
24+
25+
Set the required environment variables:
26+
27+
```bash
28+
export GCP_SERVICE_ACCOUNT_FILE=absolute-path-to-service-account-file
29+
export CDP_PROFILE=your-cdp-profile
30+
```
31+
32+
Tweak the `definition.yml` parameters to your liking. Notably, you should add and/or change:
33+
34+
```yaml
35+
name_prefix: ex01 # Keep this short (4-7 characters)
36+
admin_password: "BadPass@1" # 1 upper, 1 special, 1 number, 8-64 chars.
37+
infra_region: us-east1
38+
gcp_project_id: gcp-project-id # GCP Project ID
39+
```
40+
41+
> [!NOTE]
42+
> You can override these parameters with any typical Ansible _extra variables_ flags, i.e. `-e admin_password=my_password`. See the [cldr-runner FAQ](https://github.com/cloudera-labs/cldr-runner/blob/main/FAQ.md#how-do-i-add-extra-variables-and-tags-to-ansible-navigator) for details.
43+
44+
### SSH Keys
45+
46+
This definition will create a new SSH keypair on the host in your `~/.ssh` directory if you do not specify a SSH public key.
47+
48+
If you wish to use an existing SSH key, set `public_key_file` to the key's local path.
49+
50+
## Execute
51+
52+
Then set up the CDP Public Cloud by running the playbook:
53+
54+
```bash
55+
ansible-navigator run main.yml
56+
```
57+
58+
## Tear Down
59+
60+
Tear down the CDP Public Cloud by running the playbook:
61+
62+
```bash
63+
ansible-navigator run teardown.yml
64+
```

‎public-cloud/gcp/datalake/ansible-navigator.yml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
ansible-navigator:
18+
playbook-artifact:
19+
save-as: "runs/{playbook_name}-{time_stamp}.json"
20+
21+
ansible-runner:
22+
artifact-dir: runs
23+
rotate-artifacts-count: 3
24+
25+
logging:
26+
level: debug
27+
append: False
28+
29+
ansible:
30+
inventory:
31+
entries:
32+
- inventory.ini
33+
34+
execution-environment:
35+
container-engine: docker
36+
enabled: True
37+
environment-variables:
38+
pass:
39+
- GCP_SERVICE_ACCOUNT_FILE
40+
- CDP_PROFILE
41+
set:
42+
ANSIBLE_CALLBACK_WHITELIST: "ansible.posix.profile_tasks"
43+
ANSIBLE_GATHERING: "smart"
44+
ANSIBLE_DEPRECATION_WARNINGS: False
45+
ANSIBLE_HOST_KEY_CHECKING: False
46+
ANSIBLE_SSH_RETRIES: 10
47+
GCP_AUTH_KIND: "serviceaccount"
48+
image: ghcr.io/cloudera-labs/cldr-runner:gcp-latest
49+
pull:
50+
policy: missing
51+
volume-mounts:
52+
- src: "${GCP_SERVICE_ACCOUNT_FILE}"
53+
dest: "${GCP_SERVICE_ACCOUNT_FILE}"
54+
- src: "~/.cdp"
55+
dest: "/runner/.cdp"
56+
options: "Z"
57+
- src: "~/.ssh"
58+
dest: "/runner/.ssh"
59+
options: "Z"
60+
container-options:
61+
- "--network=host"

‎public-cloud/gcp/datalake/definition.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
################################################################################
18+
# Global variables
19+
################################################################################
20+
# Either define here or override using _extra variables_ in the CLI or AWX.
21+
# For example, '-e name_prefix=basex'
22+
# name_prefix: # You must specify a name prefix
23+
# admin_password: # You must specify an admin password
24+
infra_region: us-east1 # CSP region for infra
25+
infra_type: gcp # CSP
26+
27+
#gcp_project_id: # You must specify a GCP Project ID
28+
29+
# Limit to the caller/controller
30+
allowed_cidrs: "{{ lookup('ansible.builtin.url', 'https://api.ipify.org', wantlist=True) | product(['32']) | map('join', '/') | list }}"
31+
32+
################################################################################
33+
# CDP Environment and Datalake variables
34+
################################################################################
35+
env:
36+
tunnel: no
37+
public_endpoint_access: yes
38+
39+
infra:
40+
gcp:
41+
project: "{{ gcp_project_id }}"
42+
vpc:
43+
extra_cidr: "{{ allowed_cidrs }}"
44+
extra_ports: [22, 443]

‎public-cloud/gcp/datalake/inventory.ini‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
localhost ansible_connection=local ansible_python_interpreter="{{ ansible_playbook_python }}"

‎public-cloud/gcp/datalake/main.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
- name: Set up the cloudera-deploy variables
18+
hosts: localhost
19+
connection: local
20+
gather_facts: yes
21+
tasks:
22+
- name: Read definition variables
23+
ansible.builtin.include_role:
24+
name: cloudera.exe.init_deployment
25+
public: yes
26+
when: init__completed is undefined
27+
28+
- name: Initialization of GCP deployment
29+
block:
30+
- name: GCloud Auth using the Service Account
31+
command: >
32+
gcloud auth activate-service-account
33+
--key-file={{ lookup('env', 'GCP_SERVICE_ACCOUNT_FILE') }}
34+
35+
- name: Set the GCP project for GCloud
36+
command: >
37+
gcloud config set project {{ gcp_project_id }}
38+
tags:
39+
- always
40+
41+
- name: Set up CDP Public Cloud infrastructure (Ansible-based)
42+
ansible.builtin.import_playbook: cloudera.exe.pbc_infra_setup.yml
43+
44+
- name: Set up CDP Public Cloud (Env and DL example)
45+
ansible.builtin.import_playbook: cloudera.exe.pbc_setup.yml

‎public-cloud/gcp/datalake/teardown.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
- name: Set up the cloudera-deploy variables
18+
hosts: localhost
19+
connection: local
20+
gather_facts: yes
21+
tasks:
22+
- name: Read definition variables
23+
ansible.builtin.include_role:
24+
name: cloudera.exe.init_deployment
25+
public: yes
26+
when: init__completed is undefined
27+
28+
- name: Initialization of GCP deployment
29+
block:
30+
- name: GCloud Auth using the Service Account
31+
command: >
32+
gcloud auth activate-service-account
33+
--key-file={{ lookup('env', 'GCP_SERVICE_ACCOUNT_FILE') }}
34+
35+
- name: Set the GCP project for GCloud
36+
command: >
37+
gcloud config set project {{ gcp_project_id }}
38+
tags:
39+
- always
40+
41+
- name: Tear down CDP Public Cloud (Env and DL example)
42+
ansible.builtin.import_playbook: cloudera.exe.pbc_teardown.yml
43+
44+
- name: Tear down CDP Public Cloud infrastructure (Ansible-based)
45+
ansible.builtin.import_playbook: cloudera.exe.pbc_infra_teardown.yml

0 commit comments

Comments
 (0)