Nomad Enterprise HVD on GCP GCE

Terraform module aligned with HashiCorp Validated Designs (HVD) to deploy Nomad Enterprise on Google Cloud Platform (GCP) using Compute Engine instances.

Prerequisites

General

• Terraform CLI >= 1.9 installed on workstations.
• Git CLI and Visual Studio Code editor installed on workstations are strongly recommended.
• Google account that Nomad will be hosted in with permissions to provision these resources via Terraform CLI.
• (Optional) Google GCS for GCS Remote State backend that will solely be used to stand up the Boundary infrastructure via Terraform CLI (Community Edition).

Google

• GCP Project created
• Following APIs enabled
  • dns.googleapis.com
  • secretmanager.googleapis.com
  • compute.googleapis.com
  • servicenetworking.googleapis.com
  • networkservices.googleapis.com

Networking

• GCP VPC and the following subnets:
  • Subnets for the VM instances (compute)

📝 Note: Specify at least three subnets for high availability.

Firewall rules

• This module will create firewall rules for VM instances.
• Define CIDR ranges for Nomad access, managed via the input variable cidr_ingress_api_allow.

TLS certificates

TLS certificates for Nomad have very specific requirements for server and client nodes. Nomad Agent Certificates Let's Encrypt Certificates can not be used for Nomad due to the Nomad specific requirements. This module has the option to not enable TLS with nomad_tls_enabled however this should never be used outside a lab environment.

• TLS certificate (e.g. cert.pem) and private key (e.g. privkey.pem) for the Nomad web UI, in PEM format.
  • TLS private key must not be password-protected.
• TLS certificate authority (CA) bundle (e.g. ca_bundle.pem) in PEM format.
• These TLS files will be stored as secrets in GCP Secrets Manager.

📝 Store these TLS files as GCP Secrets Manager secrets to securely manage your certificates.

Gossip encryption

Gossip Encryption is required for this module. A gossip encryption Key can be generated by the Nomad CLI with nomad operator gossip keyring generate. Create this before deploying the module and store this in GCP Secrets Manager.

Secrets management

The following secrets must be stored in GCP Secrets Manager to bootstrap the Nomad server deployment:

• Nomad license - text string used for Nomad Enterprise licensing - This is only needed for Nomad server deployment
• Nomad gossip encryption Key - text string generated by nomad operator gossip keyring generate - This is only needed for Nomad server deployment
• Nomad TLS certificate, CA certificate and private key - stored as base64-encoded PEM secrets in GCP Secrets Manager

📝 See the Nomad documentation for more details on securing these secrets in GCP Secrets Manager.

Usage

1. Configure the prerequisites.

2. In the examples directory, you will find subdirectories with ready-made Terraform configurations for deploying this module. Select an example that matches your use case, and copy its contents to a new directory.

   📝 Example structure for managing multiple Nomad deployments:

   .
   └── environments
       ├── production
       │   ├── backend.tf
       │   ├── main.tf
       │   ├── outputs.tf
       │   ├── terraform.tfvars
       │   └── variables.tf
       └── sandbox
           ├── backend.tf
           ├── main.tf
           ├── outputs.tf
           ├── terraform.tfvars
           └── variables.tf
   

   📝 This example has two separate Nomad deployments: one for a sandbox environment and one for a production environment.

3. Update the terraform.tfvars file with your custom values, then run terraform init, terraform plan, and terraform apply.

4. After terraform apply completes successfully, connect to the Nomad compute instance shell using SSH to monitor the cloud-init logs:

   Viewing logs:

   tail -f /var/log/nomad-cloud-init.log
   

5. Once the installation finishes, verify the health of the Nomad cluster by checking the Nomad web UI or by running:

   nomad server members
   

6. If nomad_acl_enabled is true the Nomad cluster will need to be bootstrapped with the command nomad acl bootstrap. This will generate a bootstrap token that can be used to login to the CLI or UI.

Module support

This open source software is maintained by the HashiCorp Technical Field Organization, independently of our enterprise products. While our Support Engineering team provides dedicated support for our enterprise offerings, this open source software is not included.

• For help using this open source software, please engage your account team.
• To report bugs/issues with this open source software, please open them directly against this code repository using the GitHub issues feature.

Please note that there is no official Service Level Agreement (SLA) for support of this software as a HashiCorp customer. This software falls under the definition of Community Software/Versions in your Agreement. We appreciate your understanding and collaboration in improving our open source projects.

Requirements

Name	Version
terraform	~> 1.9
google	~> 5.39

Providers

Name	Version
google	~> 5.39

Resources

Name	Type
google_compute_firewall.allow_api	resource
google_compute_firewall.allow_iap	resource
google_compute_firewall.allow_nomad_health_checks	resource
google_compute_firewall.allow_outbound	resource
google_compute_firewall.allow_rpc	resource
google_compute_forwarding_rule.nomad_fr	resource
google_compute_health_check.nomad_auto_healing	resource
google_compute_instance_template.nomad	resource
google_compute_region_backend_service.nomad_bs	resource
google_compute_region_health_check.nomad_hc	resource
google_compute_region_instance_group_manager.nomad	resource
google_dns_record_set.nomad	resource
google_project_iam_member.nomad_iam	resource
google_service_account.nomad_sa	resource
google_storage_bucket_iam_binding.snapshots_creator	resource
google_storage_bucket_iam_binding.snapshots_viewer	resource
google_compute_network.network	data source
google_compute_subnetwork.subnetwork	data source
google_compute_zones.available	data source
google_dns_managed_zone.nomad	data source
google_netblock_ip_ranges.legacy	data source
google_netblock_ip_ranges.new	data source

Inputs

Name	Description	Type	Default	Required
nomad_client	Boolean to enable the Nomad client mode.	bool	n/a	yes
nomad_datacenter	Specifies the data center of the local agent. A datacenter is an abstract grouping of clients within a region. Clients are not required to be in the same datacenter as the servers they are joined with, but do need to be in the same region.	string	n/a	yes
nomad_fqdn	Fully qualified domain name to use for joining peer nodes and optionally DNS	string	n/a	yes
nomad_gossip_key_secret_name	Name of Secret Manager secret containing Nomad gossip encryption key.	string	n/a	yes
nomad_license_sm_secret_name	Name of Secret Manager secret containing Nomad license.	string	n/a	yes
nomad_server	Boolean to enable the Nomad server mode.	bool	n/a	yes
nomad_tls_ca_bundle_sm_secret_name	Name of Secret Manager containing Nomad TLS custom CA bundle.	string	n/a	yes
nomad_tls_cert_sm_secret_name	Name of Secret Manager containing Nomad TLS certificate.	string	n/a	yes
nomad_tls_privkey_sm_secret_name	Name of Secret Manager containing Nomad TLS private key.	string	n/a	yes
project_id	(required) The project ID to host the cluster in (required)	string	n/a	yes
additional_package_names	List of additional repository package names to install	set(string)	[]	no
application_prefix	(optional) The prefix to give to cloud entities	string	"nomad"	no
auto_join_tag	(optional) A list of a tag which will be used by Nomad to join other nodes to the cluster. If left blank, the module will use the first entry in tags	list(string)	null	no
autopilot_health_enabled	Whether autopilot upgrade migration validation is performed for server nodes at boot-time	bool	true	no
boot_disk_size	(optional) The disk size (GB) to use to create the boot disk	number	30	no
boot_disk_type	(optional) The disk type to use to create the boot disk	string	"pd-balanced"	no
cidr_ingress_api_allow	CIDR ranges to allow API traffic inbound to Nomad instance(s).	list(string)	[ "0.0.0.0/0" ]	no
cidr_ingress_rpc_allow	CIDR ranges to allow RPC traffic inbound to Nomad instance(s).	list(string)	[ "0.0.0.0/0" ]	no
cloud_dns_managed_zone	Zone name to create Cloud DNS record in if create_cloud_dns_record is set to true.	string	null	no
cni_version	Version of CNI plugin to install.	string	"1.6.0"	no
common_labels	(optional) Common labels to apply to GCP resources.	map(string)	{}	no
compute_image_family	(optional) The family name of the image, https://cloud.google.com/compute/docs/images/os-details,defaults to Ubuntu	string	"ubuntu-2204-lts"	no
compute_image_project	(optional) The project name of the image, https://cloud.google.com/compute/docs/images/os-details, defaults to Ubuntu	string	"ubuntu-os-cloud"	no
create_cloud_dns_record	Boolean to create Google Cloud DNS record for nomad_fqdn resolving to load balancer IP. cloud_dns_managed_zone is required when true.	bool	false	no
enable_auto_healing	(optional) Enable auto-healing on the Instance Group	bool	false	no
enable_iap	(Optional bool) Enable https://cloud.google.com/iap/docs/using-tcp-forwarding#console, defaults to true.	bool	true	no
google_service_account_iam_roles	(optional) List of IAM roles to give to the Nomad service account	list(string)	[ "roles/compute.viewer", "roles/secretmanager.secretAccessor", "roles/cloudkms.cryptoKeyEncrypterDecrypter" ]	no
health_check_interval	(optional) How often, in seconds, to send a health check	number	30	no
health_timeout	(optional) How long, in seconds, to wait before claiming failure	number	15	no
initial_auto_healing_delay	(optional) The time, in seconds, that the managed instance group waits before it applies autohealing policies	number	1200	no
load_balancing_scheme	(optional) Type of load balancer to use (INTERNAL, EXTERNAL, or NONE)	string	"INTERNAL"	no
machine_type	(optional) The machine type to use for the Nomad nodes	string	"n2-standard-4"	no
metadata	(optional) Metadata to add to the Compute Instance template	map(string)	null	no
network	(optional) The VPC network to host the cluster in	string	"default"	no
network_project_id	(optional) The project that the VPC network lives in. Can be left blank if network is in the same project as provider	string	null	no
network_region	(optional) The region that the VPC network lives in. Can be left blank if network is in the same region as provider	string	null	no
node_count	(optional) The number of nodes to create in the pool	number	6	no
nomad_acl_enabled	Boolean to enable ACLs for Nomad.	bool	true	no
nomad_architecture	Architecture of the Nomad binary to install.	string	"amd64"	no
nomad_audit_disk_size	(optional) The disk size (GB) to use to create the Nomad audit log disk	number	50	no
nomad_audit_disk_type	(optional) The disk type to use to create the Nomad audit log disk	string	"pd-balanced"	no
nomad_data_disk_size	(optional) The disk size (GB) to use to create the disk	number	500	no
nomad_data_disk_type	(optional) The disk type to use to create the Nomad data disk	string	"pd-ssd"	no
nomad_dir_bin	Path to install Nomad Enterprise binary	string	"/usr/bin"	no
nomad_dir_config	Path to install Nomad Enterprise binary	string	"/etc/nomad.d"	no
nomad_dir_home	Path to hold data, plugins and license directories	string	"/opt/nomad"	no
nomad_dir_logs	Path to hold Nomad file audit device logs	string	"/var/log/nomad"	no
nomad_enable_ui	(optional) Enable the Nomad UI	bool	true	no
nomad_group_name	Name of group to own Nomad files and processes	string	"nomad"	no
nomad_metadata_template	(optional) Alternative template file to provide for instance template metadata script. place the file in your local ./templates folder no path required	string	"nomad_custom_data.sh.tpl"	no
nomad_nodes	Number of Nomad nodes to deploy.	number	6	no
nomad_port_api	TCP port for Nomad API listener	number	4646	no
nomad_port_rpc	TCP port for Nomad cluster address	number	4647	no
nomad_port_serf	TCP port for Nomad cluster address	number	4648	no
nomad_region	Specifies the region of the local agent. A region is an abstract grouping of datacenters. Clients are not required to be in the same region as the servers they are joined with, but do need to be in the same datacenter.	string	null	no
nomad_snapshot_gcs_bucket_name	Name of Google Cloud Storage bucket to hold Nomad snapshots	string	null	no
nomad_tls_disable_client_certs	Disable client authentication for the Nomad listener. Must be enabled when tls auth method is used.	bool	true	no
nomad_tls_enabled	Boolean to enable TLS for Nomad.	bool	true	no
nomad_tls_require_and_verify_client_cert	(optional) Require a client to present a client certificate that validates against system CAs	bool	false	no
nomad_upstream_servers	List of Nomad server addresses to join the Nomad client with.	list(string)	null	no
nomad_user_name	Name of system user to own Nomad files and processes	string	"nomad"	no
nomad_version	(optional) The version of Nomad to use	string	"1.9.5+ent"	no
packer_image	(optional) The packer image to use	string	null	no
region	(optional) The region to host the cluster in	string	"us-central1"	no
subnetwork	(optional) The subnet in the VPC network to host the cluster in	string	"default"	no
systemd_dir	Path to systemd directory for unit files	string	"/lib/systemd/system"	no
tags	(optional) A list containing tags to assign to all resources	list(string)	[ "nomad" ]	no

Outputs

Name	Description
nomad_cli_config	Environment variables to configure the nomad CLI
nomad_url	URL to access Nomad application based on value of nomad_fqdn input.