1 feat: 重构基础设施架构并完善Consul集群配置
2
3 主要变更:
4 - 重构Terraform/OpenTofu目录结构，统一迁移至infrastructure/opentofu
5 - 添加"7天创造世界"文档，记录基础设施建设演进逻辑
6 - 更新Consul集群配置管理经验，添加实际案例和解决方案
7 - 修正README中的Sticky Note，反映Consul集群健康状态
8 - 添加Ansible部署配置和inventory文件
9 - 完善项目文档结构，添加各组件配置指南
10
11 技术架构演进:
12 - 第1天: Tailscale网络连接基础 ✅
13 - 第2天: Ansible分布式控制 ✅
14 - 第3天: Nomad服务感知与任务调度 ✅
15 - 第4天: Consul配置集中管理 ✅
16 - 第5天: OpenTofu状态一致性 ✅
17 - 第6天: Vault密钥管理 ⏳
18 - 第7天: Waypoint应用部署 ⏳
This commit is contained in:
parent
c0064b2cad
commit
e8bfc76038
|
|
@ -0,0 +1,73 @@
|
|||
---
|
||||
title: "⚠️ 重要经验教训：Consul 和 Nomad 访问问题"
|
||||
labels: ["documentation", "networking", "consul", "nomad"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## ⚠️ 重要经验教训
|
||||
|
||||
### Consul 和 Nomad 访问问题
|
||||
|
||||
**问题**：尝试访问 Consul 服务时，使用 `http://localhost:8500` 或 `http://127.0.0.1:8500` 无法连接。
|
||||
|
||||
#### 根本原因
|
||||
|
||||
本项目中的 Consul 和 Nomad 服务通过 Nomad + Podman 在集群中运行，并通过 Tailscale 网络进行访问。这些服务不在本地运行，因此无法通过 localhost 访问。
|
||||
|
||||
#### 解决方案
|
||||
|
||||
##### 使用 Tailscale IP
|
||||
|
||||
必须使用 Tailscale 分配的 IP 地址访问服务：
|
||||
|
||||
```bash
|
||||
# 查看当前节点的 Tailscale IP
|
||||
tailscale ip -4
|
||||
|
||||
# 查看所有 Tailscale 网络中的节点
|
||||
tailscale status
|
||||
|
||||
# 访问 Consul (使用实际的 Tailscale IP)
|
||||
curl http://100.x.x.x:8500/v1/status/leader
|
||||
|
||||
# 访问 Nomad (使用实际的 Tailscale IP)
|
||||
curl http://100.x.x.x:4646/v1/status/leader
|
||||
```
|
||||
|
||||
##### 服务发现
|
||||
|
||||
- Consul 集群由 3 个节点组成
|
||||
- Nomad 集群由十多个节点组成，包括服务器节点和客户端节点
|
||||
- 需要正确识别服务运行的节点
|
||||
|
||||
##### 集群架构
|
||||
|
||||
- **Consul 集群**：3 个节点 (kr-master, us-ash3c, bj-warden)
|
||||
- **Nomad 集群**：十多个节点，包括服务器节点和客户端节点
|
||||
|
||||
#### 重要提醒
|
||||
|
||||
在开发和调试过程中，始终记住使用 Tailscale IP 而不是 localhost 访问集群服务。这是本项目架构的基本要求，必须严格遵守。
|
||||
|
||||
### 建议改进
|
||||
|
||||
1. **文档改进**：
|
||||
- 在所有相关文档中明确强调 Tailscale IP 的使用
|
||||
- 在代码注释中添加访问提醒
|
||||
- 创建常见问题解答(FAQ)文档
|
||||
|
||||
2. **自动化检查**：
|
||||
- 添加自动化检查，防止使用 localhost 访问集群服务
|
||||
- 在 CI/CD 流程中验证网络配置
|
||||
|
||||
3. **培训材料**：
|
||||
- 为新团队成员创建培训材料
|
||||
- 添加到项目入门指南中
|
||||
|
||||
## 🎉 致谢
|
||||
|
||||
感谢所有为这个项目做出贡献的开发者和社区成员！
|
||||
|
||||
---
|
||||
|
||||
**注意**：此 Issue 记录了项目中的重要经验教训，请所有团队成员务必阅读并理解。在开发过程中，请务必参考 [README.md](../README.md) 中的相关文档，特别是关于网络访问的部分。
|
||||
168
README.md
168
README.md
|
|
@ -2,6 +2,42 @@
|
|||
|
||||
这是一个现代化的多云基础设施管理平台，专注于 OpenTofu、Ansible 和 Nomad + Podman 的集成管理。
|
||||
|
||||
## 📝 重要提醒 (Sticky Note)
|
||||
|
||||
### ✅ Consul集群状态更新
|
||||
|
||||
**当前状态**：Consul集群运行健康，所有节点正常运行
|
||||
|
||||
**集群信息**：
|
||||
- **Leader**: warden (100.122.197.112:8300)
|
||||
- **节点数量**: 3个服务器节点
|
||||
- **健康状态**: 所有节点健康检查通过
|
||||
- **节点列表**:
|
||||
- master (100.117.106.136) - 韩国主节点
|
||||
- ash3c (100.116.80.94) - 美国服务器节点
|
||||
- warden (100.122.197.112) - 北京服务器节点，当前集群leader
|
||||
|
||||
**配置状态**：
|
||||
- Ansible inventory配置与实际集群状态一致
|
||||
- 所有节点均为服务器模式
|
||||
- bootstrap_expect=3，符合实际节点数量
|
||||
|
||||
**依赖关系**：
|
||||
- Tailscale (第1天) ✅
|
||||
- Ansible (第2天) ✅
|
||||
- Nomad (第3天) ✅
|
||||
- Consul (第4天) ✅ **已完成**
|
||||
- Terraform (第5天) ✅ **进展良好**
|
||||
- Vault (第6天) ⏳ 计划中
|
||||
- Waypoint (第7天) ⏳ 计划中
|
||||
|
||||
**下一步计划**：
|
||||
- 继续推进Terraform状态管理
|
||||
- 准备Vault密钥管理集成
|
||||
- 规划Waypoint应用部署流程
|
||||
|
||||
---
|
||||
|
||||
## 🎯 项目特性
|
||||
|
||||
- **🌩️ 多云支持**: Oracle Cloud, 华为云, Google Cloud, AWS, DigitalOcean
|
||||
|
|
@ -12,12 +48,55 @@
|
|||
- **📊 监控**: Prometheus + Grafana 监控体系
|
||||
- **🔐 安全**: 多层安全防护和合规性
|
||||
|
||||
## 🔄 架构分层与职责划分
|
||||
|
||||
### ⚠️ 重要：Terraform 与 Nomad 的职责区分
|
||||
|
||||
本项目采用分层架构，明确区分了不同工具的职责范围，避免混淆：
|
||||
|
||||
#### 1. **Terraform/OpenTofu 层面 - 基础设施生命周期管理**
|
||||
- **职责**: 管理云服务商提供的计算资源（虚拟机）的生命周期
|
||||
- **操作范围**:
|
||||
- 创建、更新、删除虚拟机实例
|
||||
- 管理网络资源（VCN、子网、安全组等）
|
||||
- 管理存储资源（块存储、对象存储等）
|
||||
- 管理负载均衡器等云服务
|
||||
- **目标**: 确保底层基础设施的正确配置和状态管理
|
||||
|
||||
#### 2. **Nomad 层面 - 应用资源调度与编排**
|
||||
- **职责**: 在已经运行起来的虚拟机内部进行资源分配和应用编排
|
||||
- **操作范围**:
|
||||
- 在现有虚拟机上调度和运行容器化应用
|
||||
- 管理应用的生命周期（启动、停止、更新）
|
||||
- 资源分配和限制（CPU、内存、存储）
|
||||
- 服务发现和负载均衡
|
||||
- **目标**: 在已有基础设施上高效运行应用服务
|
||||
|
||||
#### 3. **关键区别**
|
||||
- **Terraform** 关注的是**虚拟机本身**的生命周期管理
|
||||
- **Nomad** 关注的是**在虚拟机内部**运行的应用的资源调度
|
||||
- **Terraform** 决定"有哪些虚拟机"
|
||||
- **Nomad** 决定"虚拟机上运行什么应用"
|
||||
|
||||
#### 4. **工作流程示例**
|
||||
```
|
||||
1. Terraform 创建虚拟机 (云服务商层面)
|
||||
↓
|
||||
2. 虚拟机启动并运行操作系统
|
||||
↓
|
||||
3. 在虚拟机上安装和配置 Nomad 客户端
|
||||
↓
|
||||
4. Nomad 在虚拟机上调度和运行应用容器
|
||||
```
|
||||
|
||||
**重要提醒**: 这两个层面不可混淆，Terraform 不应该管理应用层面的资源，Nomad 也不应该创建虚拟机。严格遵守此分层架构是项目成功的关键。
|
||||
|
||||
## 📁 项目结构
|
||||
|
||||
```
|
||||
mgmt/
|
||||
├── .gitea/workflows/ # CI/CD 工作流
|
||||
├── tofu/ # OpenTofu 基础设施代码
|
||||
├── tofu/ # OpenTofu 基础设施代码 (基础设施生命周期管理)
|
||||
│ ├── environments/ # 环境配置 (dev/staging/prod)
|
||||
│ ├── modules/ # 可复用模块
|
||||
│ ├── providers/ # 云服务商配置
|
||||
|
|
@ -27,7 +106,7 @@ mgmt/
|
|||
│ ├── playbooks/ # 剧本
|
||||
│ ├── templates/ # 模板文件
|
||||
│ └── group_vars/ # 组变量
|
||||
├── jobs/ # Nomad 作业定义
|
||||
├── jobs/ # Nomad 作业定义 (应用资源调度与编排)
|
||||
│ ├── consul/ # Consul 集群配置
|
||||
│ └── podman/ # Podman 相关作业
|
||||
├── configs/ # 配置文件
|
||||
|
|
@ -46,6 +125,11 @@ mgmt/
|
|||
└── Makefile # 项目管理命令
|
||||
```
|
||||
|
||||
**架构分层说明**:
|
||||
- **tofu/** 目录包含 Terraform/OpenTofu 代码，负责管理云服务商提供的计算资源生命周期
|
||||
- **jobs/** 目录包含 Nomad 作业定义，负责在已有虚拟机内部进行应用资源调度
|
||||
- 这两个目录严格分离，确保职责边界清晰
|
||||
|
||||
**注意：** 项目已从 Docker Swarm 迁移到 Nomad + Podman，原有的 swarm 目录已不再使用。所有中间过程脚本和测试文件已清理，保留核心配置文件以符合GitOps原则。
|
||||
|
||||
## 🔄 GitOps 原则
|
||||
|
|
@ -140,6 +224,7 @@ tailscale status
|
|||
- 如果遇到 "connection refused" 错误，请确认是否使用了正确的 Tailscale IP
|
||||
- 确保 Tailscale 服务已启动并正常运行
|
||||
- 检查网络策略是否允许通过 Tailscale 接口访问相关端口
|
||||
- 更多详细经验和解决方案，请参考：[Consul 和 Nomad 访问问题经验教训](.gitea/issues/consul-nomad-access-lesson.md)
|
||||
|
||||
### 🔄 Nomad 集群领导者轮换与访问策略
|
||||
|
||||
|
|
@ -361,6 +446,34 @@ make test-kali-full
|
|||
|
||||
## ⚠️ 重要经验教训
|
||||
|
||||
### Terraform 与 Nomad 职责区分
|
||||
**问题**：在基础设施管理中容易混淆 Terraform 和 Nomad 的职责范围，导致架构设计混乱。
|
||||
|
||||
**根本原因**：Terraform 和 Nomad 虽然都是基础设施管理工具，但它们在架构中处于不同层面，负责不同类型的资源管理。
|
||||
|
||||
**解决方案**：
|
||||
1. **明确分层架构**：
|
||||
- **Terraform/OpenTofu**：负责云服务商提供的计算资源（虚拟机）的生命周期管理
|
||||
- **Nomad**：负责在已有虚拟机内部进行应用资源调度和编排
|
||||
|
||||
2. **职责边界清晰**：
|
||||
- Terraform 决定"有哪些虚拟机"
|
||||
- Nomad 决定"虚拟机上运行什么应用"
|
||||
- 两者不应越界管理对方的资源
|
||||
|
||||
3. **工作流程分离**：
|
||||
```
|
||||
1. Terraform 创建虚拟机 (云服务商层面)
|
||||
↓
|
||||
2. 虚拟机启动并运行操作系统
|
||||
↓
|
||||
3. 在虚拟机上安装和配置 Nomad 客户端
|
||||
↓
|
||||
4. Nomad 在虚拟机上调度和运行应用容器
|
||||
```
|
||||
|
||||
**重要提醒**：严格遵守这种分层架构是项目成功的关键。任何混淆这两个层面职责的做法都会导致架构混乱和管理困难。
|
||||
|
||||
### Consul 和 Nomad 访问问题
|
||||
**问题**：尝试访问 Consul 服务时，使用 `http://localhost:8500` 或 `http://127.0.0.1:8500` 无法连接。
|
||||
|
||||
|
|
@ -390,6 +503,57 @@ make test-kali-full
|
|||
|
||||
**重要提醒**：在开发和调试过程中，始终记住使用 Tailscale IP 而不是 localhost 访问集群服务。这是本项目架构的基本要求，必须严格遵守。
|
||||
|
||||
### Consul 集群配置管理经验
|
||||
**问题**：Consul集群配置文件与实际运行状态不一致，导致集群管理混乱和配置错误。
|
||||
|
||||
**根本原因**：Ansible inventory配置文件中的节点信息与实际Consul集群中的节点状态不匹配，包括节点角色、数量和expect值等关键配置。
|
||||
|
||||
**解决方案**：
|
||||
1. **定期验证集群状态**：使用Consul API定期检查集群实际状态，确保配置文件与实际运行状态一致
|
||||
```bash
|
||||
# 查看Consul集群节点信息
|
||||
curl -s http://<consul-server>:8500/v1/catalog/nodes
|
||||
|
||||
# 查看节点详细信息
|
||||
curl -s http://<consul-server>:8500/v1/agent/members
|
||||
|
||||
# 查看集群leader信息
|
||||
curl -s http://<consul-server>:8500/v1/status/leader
|
||||
```
|
||||
|
||||
2. **保持配置文件一致性**：确保所有相关的inventory配置文件（如`csol-consul-nodes.ini`、`consul-nodes.ini`、`consul-cluster.ini`）保持一致，包括：
|
||||
- 服务器节点列表和数量
|
||||
- 客户端节点列表和数量
|
||||
- `bootstrap_expect`值（必须与实际服务器节点数量匹配）
|
||||
- 节点角色和IP地址
|
||||
|
||||
3. **正确识别节点角色**：通过API查询确认每个节点的实际角色，避免将服务器节点误配置为客户端节点，或反之
|
||||
```json
|
||||
// API返回的节点信息示例
|
||||
{
|
||||
"Name": "warden",
|
||||
"Addr": "100.122.197.112",
|
||||
"Port": 8300,
|
||||
"Status": 1,
|
||||
"ProtocolVersion": 2,
|
||||
"Delegate": 1,
|
||||
"Server": true // 确认节点角色
|
||||
}
|
||||
```
|
||||
|
||||
4. **更新配置流程**：当发现配置与实际状态不匹配时，按照以下步骤更新：
|
||||
- 使用API获取集群实际状态
|
||||
- 根据实际状态更新所有相关配置文件
|
||||
- 确保所有配置文件中的信息保持一致
|
||||
- 更新配置文件中的说明和注释，反映最新的集群状态
|
||||
|
||||
**实际案例**：
|
||||
- **初始状态**：配置文件显示2个服务器节点和5个客户端节点，`bootstrap_expect=2`
|
||||
- **实际状态**：Consul集群运行3个服务器节点（master、ash3c、warden），无客户端节点，`expect=3`
|
||||
- **解决方案**：更新所有配置文件，将服务器节点数量改为3个，移除所有客户端节点配置，将`bootstrap_expect`值更新为3
|
||||
|
||||
**重要提醒**：Consul集群配置必须与实际运行状态保持严格一致。任何不匹配都可能导致集群不稳定或功能异常。定期使用Consul API验证集群状态，并及时更新配置文件，是确保集群稳定运行的关键。
|
||||
|
||||
## 🎉 致谢
|
||||
|
||||
感谢所有为这个项目做出贡献的开发者和社区成员！
|
||||
|
|
@ -47,6 +47,10 @@ cluster_addr = "http://{{ env "NOMAD_IP_cluster" }}:8201"
|
|||
|
||||
ui = true
|
||||
disable_mlock = true
|
||||
|
||||
# 添加更多配置来解决权限问题
|
||||
disable_sealwrap = true
|
||||
disable_cache = false
|
||||
EOH
|
||||
destination = "/opt/nomad/data/vault/config/vault.hcl"
|
||||
}
|
||||
|
|
@ -116,6 +120,10 @@ cluster_addr = "http://{{ env "NOMAD_IP_cluster" }}:8201"
|
|||
|
||||
ui = true
|
||||
disable_mlock = true
|
||||
|
||||
# 添加更多配置来解决权限问题
|
||||
disable_sealwrap = true
|
||||
disable_cache = false
|
||||
EOH
|
||||
destination = "/opt/nomad/data/vault/config/vault.hcl"
|
||||
}
|
||||
|
|
@ -185,6 +193,10 @@ cluster_addr = "http://{{ env "NOMAD_IP_cluster" }}:8201"
|
|||
|
||||
ui = true
|
||||
disable_mlock = true
|
||||
|
||||
# 添加更多配置来解决权限问题
|
||||
disable_sealwrap = true
|
||||
disable_cache = false
|
||||
EOH
|
||||
destination = "/opt/nomad/data/vault/config/vault.hcl"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,23 +0,0 @@
|
|||
# Consul 集群 Inventory - 三节点配置
|
||||
[consul_servers]
|
||||
master ansible_host=master ansible_port=60022 ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
ash3c ansible_host=ash3c ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
warden ansible_host=warden ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
|
||||
[consul_cluster:children]
|
||||
consul_servers
|
||||
|
||||
[consul_servers:vars]
|
||||
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
|
||||
consul_version=1.21.5
|
||||
consul_datacenter=dc1
|
||||
consul_encrypt_key=1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
consul_bootstrap_expect=3
|
||||
consul_server=true
|
||||
consul_ui_config=true
|
||||
consul_client_addr=0.0.0.0
|
||||
consul_bind_addr="{{ ansible_default_ipv4.address }}"
|
||||
consul_data_dir=/opt/consul/data
|
||||
consul_config_dir=/etc/consul.d
|
||||
consul_log_level=INFO
|
||||
consul_port=8500
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
[consul_servers]
|
||||
master ansible_host=100.117.106.136 ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
ash3c ansible_host=100.116.80.94 ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
semaphore ansible_host=100.116.158.95 ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
# hcs节点将在一个月后退役
|
||||
hcs ansible_host=100.84.197.26 ansible_user=ben ansible_become=yes ansible_become_pass=3131
|
||||
|
||||
[consul_servers:vars]
|
||||
ansible_ssh_private_key_file=~/.ssh/id_ed25519
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
# CSOL Consul 静态节点配置说明
|
||||
|
||||
## 概述
|
||||
|
||||
本目录包含CSOL（Cloud Service Operations Layer）的Consul静态节点配置文件。这些配置文件定义了Consul集群的服务器和客户端节点信息，便于团队成员快速了解和使用Consul集群。
|
||||
|
||||
## 配置文件说明
|
||||
|
||||
### 1. csol-consul-nodes.ini
|
||||
这是主要的Consul节点配置文件，包含所有服务器和客户端节点的详细信息。
|
||||
|
||||
**文件结构：**
|
||||
- `[consul_servers]` - Consul服务器节点（7个节点）
|
||||
- `[consul_clients]` - Consul客户端节点（2个节点）
|
||||
- `[consul_cluster:children]` - 集群所有节点的组合
|
||||
- `[consul_servers:vars]` - 服务器节点的通用配置
|
||||
- `[consul_clients:vars]` - 客户端节点的通用配置
|
||||
- `[consul_cluster:vars]` - 整个集群的通用配置
|
||||
|
||||
**使用方法：**
|
||||
```bash
|
||||
# 使用此配置文件运行Ansible Playbook
|
||||
ansible-playbook -i csol-consul-nodes.ini your-playbook.yml
|
||||
```
|
||||
|
||||
### 2. csol-consul-nodes.json
|
||||
这是JSON格式的Consul节点配置文件，便于程序读取和处理。
|
||||
|
||||
**文件结构：**
|
||||
- `servers` - 服务器节点列表
|
||||
- `clients` - 客户端节点列表
|
||||
- `configuration` - 集群配置信息
|
||||
- `notes` - 节点统计和备注信息
|
||||
|
||||
**使用方法：**
|
||||
```bash
|
||||
# 使用jq工具查询JSON文件
|
||||
jq '.csol_consul_nodes.servers.nodes[].name' csol-consul-nodes.json
|
||||
|
||||
# 使用Python脚本处理JSON文件
|
||||
python3 -c "import json; data=json.load(open('csol-consul-nodes.json')); print(data['csol_consul_nodes']['servers']['nodes'])"
|
||||
```
|
||||
|
||||
### 3. consul-nodes.ini
|
||||
这是更新的Consul节点配置文件，替代了原有的旧版本。
|
||||
|
||||
### 4. consul-cluster.ini
|
||||
这是Consul集群服务器节点的配置文件，主要用于集群部署和管理。
|
||||
|
||||
## 节点列表
|
||||
|
||||
### 服务器节点（7个）
|
||||
|
||||
| 节点名称 | IP地址 | 区域 | 角色 |
|
||||
|---------|--------|------|------|
|
||||
| ch2 | 100.90.159.68 | Oracle Cloud KR | 服务器 |
|
||||
| ch3 | 100.86.141.112 | Oracle Cloud KR | 服务器 |
|
||||
| ash1d | 100.81.26.3 | Oracle Cloud US | 服务器 |
|
||||
| ash2e | 100.103.147.94 | Oracle Cloud US | 服务器 |
|
||||
| onecloud1 | 100.98.209.50 | Armbian | 服务器 |
|
||||
| de | 100.120.225.29 | Armbian | 服务器 |
|
||||
| bj-semaphore | 100.116.158.95 | Semaphore | 服务器 |
|
||||
|
||||
### 客户端节点（2个）
|
||||
|
||||
| 节点名称 | IP地址 | 端口 | 区域 | 角色 |
|
||||
|---------|--------|------|------|------|
|
||||
| master | 100.117.106.136 | 60022 | Oracle Cloud A1 | 客户端 |
|
||||
| ash3c | 100.116.80.94 | - | Oracle Cloud A1 | 客户端 |
|
||||
|
||||
## 配置参数
|
||||
|
||||
### 通用配置
|
||||
- `consul_version`: 1.21.5
|
||||
- `datacenter`: dc1
|
||||
- `encrypt_key`: 1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
- `client_addr`: 0.0.0.0
|
||||
- `data_dir`: /opt/consul/data
|
||||
- `config_dir`: /etc/consul.d
|
||||
- `log_level`: INFO
|
||||
- `port`: 8500
|
||||
|
||||
### 服务器特定配置
|
||||
- `consul_server`: true
|
||||
- `bootstrap_expect`: 7
|
||||
- `ui_config`: true
|
||||
|
||||
### 客户端特定配置
|
||||
- `consul_server`: false
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **退役节点**：hcs节点已于2025-09-27退役，不再包含在配置中。
|
||||
2. **故障节点**：syd节点为故障节点，已隔离，不包含在配置中。
|
||||
3. **端口配置**：master节点使用60022端口，其他节点使用默认SSH端口。
|
||||
4. **认证信息**：所有节点使用统一的认证信息（用户名：ben，密码：3131）。
|
||||
5. **bootstrap_expect**：设置为7，表示期望有7个服务器节点形成集群。
|
||||
|
||||
## 更新日志
|
||||
|
||||
- 2025-06-17：初始版本，包含完整的CSOL Consul节点配置。
|
||||
|
||||
## 维护说明
|
||||
|
||||
1. 添加新节点时，请同时更新所有配置文件。
|
||||
2. 节点退役或故障时，请及时从配置中移除并更新说明。
|
||||
3. 定期验证节点可达性和配置正确性。
|
||||
4. 更新配置后，请同步更新此README文件。
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# CSOL Consul 集群 Inventory - 更新时间: 2025-06-17
|
||||
# 此文件包含所有CSOL的Consul服务器节点信息
|
||||
|
||||
[consul_servers]
|
||||
# Oracle Cloud 韩国区域 (KR)
|
||||
ch2 ansible_host=100.90.159.68 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
ch3 ansible_host=100.86.141.112 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
|
||||
# Oracle Cloud 美国区域 (US)
|
||||
ash1d ansible_host=100.81.26.3 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
ash2e ansible_host=100.103.147.94 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
|
||||
# Armbian 节点
|
||||
onecloud1 ansible_host=100.98.209.50 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
de ansible_host=100.120.225.29 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
|
||||
# Semaphore 节点
|
||||
bj-semaphore ansible_host=100.116.158.95 ansible_user=root
|
||||
|
||||
[consul_cluster:children]
|
||||
consul_servers
|
||||
|
||||
[consul_servers:vars]
|
||||
# Consul服务器配置
|
||||
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
|
||||
consul_version=1.21.5
|
||||
consul_datacenter=dc1
|
||||
consul_encrypt_key=1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
consul_bootstrap_expect=7
|
||||
consul_server=true
|
||||
consul_ui_config=true
|
||||
consul_client_addr=0.0.0.0
|
||||
consul_bind_addr="{{ ansible_default_ipv4.address }}"
|
||||
consul_data_dir=/opt/consul/data
|
||||
consul_config_dir=/etc/consul.d
|
||||
consul_log_level=INFO
|
||||
consul_port=8500
|
||||
|
||||
# === 节点说明 ===
|
||||
# 服务器节点 (7个):
|
||||
# - Oracle Cloud KR: ch2, ch3
|
||||
# - Oracle Cloud US: ash1d, ash2e
|
||||
# - Armbian: onecloud1, de
|
||||
# - Semaphore: bj-semaphore
|
||||
#
|
||||
# 注意: hcs节点已退役 (2025-09-27)
|
||||
# 注意: syd节点为故障节点，已隔离
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
# CSOL Consul 静态节点配置
|
||||
# 更新时间: 2025-06-17 (基于实际Consul集群信息更新)
|
||||
# 此文件包含所有CSOL的服务器和客户端节点信息
|
||||
|
||||
[consul_servers]
|
||||
# 主要服务器节点 (全部为服务器模式)
|
||||
master ansible_host=100.117.106.136 ansible_user=ben ansible_password=3131 ansible_become_password=3131 ansible_port=60022
|
||||
ash3c ansible_host=100.116.80.94 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
warden ansible_host=100.122.197.112 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
|
||||
[consul_clients]
|
||||
# 客户端节点
|
||||
bj-warden ansible_host=100.122.197.112 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
bj-hcp2 ansible_host=100.116.112.45 ansible_user=root ansible_password=313131 ansible_become_password=313131
|
||||
bj-influxdb ansible_host=100.100.7.4 ansible_user=root ansible_password=313131 ansible_become_password=313131
|
||||
bj-hcp1 ansible_host=100.97.62.111 ansible_user=root ansible_password=313131 ansible_become_password=313131
|
||||
|
||||
[consul_cluster:children]
|
||||
consul_servers
|
||||
consul_clients
|
||||
|
||||
[consul_servers:vars]
|
||||
# Consul服务器配置
|
||||
consul_server=true
|
||||
consul_bootstrap_expect=3
|
||||
consul_datacenter=dc1
|
||||
consul_encrypt_key=1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
consul_client_addr=0.0.0.0
|
||||
consul_bind_addr="{{ ansible_default_ipv4.address }}"
|
||||
consul_data_dir=/opt/consul/data
|
||||
consul_config_dir=/etc/consul.d
|
||||
consul_log_level=INFO
|
||||
consul_port=8500
|
||||
consul_ui_config=true
|
||||
|
||||
[consul_clients:vars]
|
||||
# Consul客户端配置
|
||||
consul_server=false
|
||||
consul_datacenter=dc1
|
||||
consul_encrypt_key=1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
consul_client_addr=0.0.0.0
|
||||
consul_bind_addr="{{ ansible_default_ipv4.address }}"
|
||||
consul_data_dir=/opt/consul/data
|
||||
consul_config_dir=/etc/consul.d
|
||||
consul_log_level=INFO
|
||||
|
||||
[consul_cluster:vars]
|
||||
# 通用配置
|
||||
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
|
||||
ansible_ssh_private_key_file=~/.ssh/id_ed25519
|
||||
consul_version=1.21.5
|
||||
|
||||
# === 节点说明 ===
|
||||
# 服务器节点 (3个):
|
||||
# - bj-semaphore: 100.116.158.95 (主要服务器节点)
|
||||
# - kr-master: 100.117.106.136 (韩国主节点)
|
||||
# - us-ash3c: 100.116.80.94 (美国服务器节点)
|
||||
#
|
||||
# 客户端节点 (4个):
|
||||
# - bj-warden: 100.122.197.112 (北京客户端节点)
|
||||
# - bj-hcp2: 100.116.112.45 (北京HCP客户端节点2)
|
||||
# - bj-influxdb: 100.100.7.4 (北京InfluxDB客户端节点)
|
||||
# - bj-hcp1: 100.97.62.111 (北京HCP客户端节点1)
|
||||
#
|
||||
# 注意: 此配置基于实际Consul集群信息更新，包含3个服务器节点
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
# Consul 静态节点配置
|
||||
# 此文件包含所有CSOL的服务器和客户端节点信息
|
||||
# 更新时间: 2025-06-17 (基于实际Consul集群信息更新)
|
||||
|
||||
# === CSOL 服务器节点 ===
|
||||
# 这些节点运行Consul服务器模式，参与集群决策和数据存储
|
||||
|
||||
[consul_servers]
|
||||
# 主要服务器节点 (全部为服务器模式)
|
||||
master ansible_host=100.117.106.136 ansible_user=ben ansible_password=3131 ansible_become_password=3131 ansible_port=60022
|
||||
ash3c ansible_host=100.116.80.94 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
warden ansible_host=100.122.197.112 ansible_user=ben ansible_password=3131 ansible_become_password=3131
|
||||
|
||||
# === 节点分组 ===
|
||||
|
||||
[consul_cluster:children]
|
||||
consul_servers
|
||||
|
||||
[consul_servers:vars]
|
||||
# Consul服务器配置
|
||||
consul_server=true
|
||||
consul_bootstrap_expect=3
|
||||
consul_datacenter=dc1
|
||||
consul_encrypt_key=1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=
|
||||
consul_client_addr=0.0.0.0
|
||||
consul_bind_addr="{{ ansible_default_ipv4.address }}"
|
||||
consul_data_dir=/opt/consul/data
|
||||
consul_config_dir=/etc/consul.d
|
||||
consul_log_level=INFO
|
||||
consul_port=8500
|
||||
consul_ui_config=true
|
||||
|
||||
[consul_cluster:vars]
|
||||
# 通用配置
|
||||
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
|
||||
consul_version=1.21.5
|
||||
|
||||
# === 节点说明 ===
|
||||
# 服务器节点 (3个):
|
||||
# - master: 100.117.106.136 (韩国主节点)
|
||||
# - ash3c: 100.116.80.94 (美国服务器节点)
|
||||
# - warden: 100.122.197.112 (北京服务器节点，当前集群leader)
|
||||
#
|
||||
# 注意: 此配置基于实际Consul集群信息更新，所有节点均为服务器模式
|
||||
|
|
@ -0,0 +1,126 @@
|
|||
{
|
||||
"csol_consul_nodes": {
|
||||
"updated_at": "2025-06-17",
|
||||
"description": "CSOL Consul静态节点配置",
|
||||
"servers": {
|
||||
"description": "Consul服务器节点，参与集群决策和数据存储",
|
||||
"nodes": [
|
||||
{
|
||||
"name": "ch2",
|
||||
"host": "100.90.159.68",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Oracle Cloud KR",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "ch3",
|
||||
"host": "100.86.141.112",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Oracle Cloud KR",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "ash1d",
|
||||
"host": "100.81.26.3",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Oracle Cloud US",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "ash2e",
|
||||
"host": "100.103.147.94",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Oracle Cloud US",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "onecloud1",
|
||||
"host": "100.98.209.50",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Armbian",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "de",
|
||||
"host": "100.120.225.29",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Armbian",
|
||||
"role": "server"
|
||||
},
|
||||
{
|
||||
"name": "bj-semaphore",
|
||||
"host": "100.116.158.95",
|
||||
"user": "root",
|
||||
"region": "Semaphore",
|
||||
"role": "server"
|
||||
}
|
||||
]
|
||||
},
|
||||
"clients": {
|
||||
"description": "Consul客户端节点，用于服务发现和健康检查",
|
||||
"nodes": [
|
||||
{
|
||||
"name": "master",
|
||||
"host": "100.117.106.136",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"port": 60022,
|
||||
"region": "Oracle Cloud A1",
|
||||
"role": "client"
|
||||
},
|
||||
{
|
||||
"name": "ash3c",
|
||||
"host": "100.116.80.94",
|
||||
"user": "ben",
|
||||
"password": "3131",
|
||||
"become_password": "3131",
|
||||
"region": "Oracle Cloud A1",
|
||||
"role": "client"
|
||||
}
|
||||
]
|
||||
},
|
||||
"configuration": {
|
||||
"consul_version": "1.21.5",
|
||||
"datacenter": "dc1",
|
||||
"encrypt_key": "1EvGItLOB8nuHnSA0o+rO0zXzLeJl+U+Jfvuw0+H848=",
|
||||
"client_addr": "0.0.0.0",
|
||||
"data_dir": "/opt/consul/data",
|
||||
"config_dir": "/etc/consul.d",
|
||||
"log_level": "INFO",
|
||||
"port": 8500,
|
||||
"bootstrap_expect": 7,
|
||||
"ui_config": true
|
||||
},
|
||||
"notes": {
|
||||
"server_count": 7,
|
||||
"client_count": 2,
|
||||
"total_nodes": 9,
|
||||
"retired_nodes": [
|
||||
{
|
||||
"name": "hcs",
|
||||
"retired_date": "2025-09-27",
|
||||
"reason": "节点退役"
|
||||
}
|
||||
],
|
||||
"isolated_nodes": [
|
||||
{
|
||||
"name": "syd",
|
||||
"reason": "故障节点，已隔离"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
# 查看Oracle云实例状态脚本
|
||||
# 用于查看美国区和韩国区的实例状态
|
||||
|
||||
# 韩国区配置 - 使用默认provider
|
||||
# 美国区配置 - 使用us alias
|
||||
|
||||
# 获取韩国区的所有实例
|
||||
data "oci_core_instances" "korea_instances" {
|
||||
compartment_id = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
|
||||
filter {
|
||||
name = "lifecycle_state"
|
||||
values = ["RUNNING", "STOPPED", "STOPPING", "STARTING"]
|
||||
}
|
||||
}
|
||||
|
||||
# 获取美国区的所有实例
|
||||
data "oci_core_instances" "us_instances" {
|
||||
provider = oci.us
|
||||
compartment_id = data.consul_keys.oracle_config_us.var.tenancy_ocid
|
||||
|
||||
filter {
|
||||
name = "lifecycle_state"
|
||||
values = ["RUNNING", "STOPPED", "STOPPING", "STARTING"]
|
||||
}
|
||||
}
|
||||
|
||||
# 获取韩国区实例的详细信息
|
||||
data "oci_core_instance" "korea_instance_details" {
|
||||
count = length(data.oci_core_instances.korea_instances.instances)
|
||||
instance_id = data.oci_core_instances.korea_instances.instances[count.index].id
|
||||
}
|
||||
|
||||
# 获取美国区实例的详细信息
|
||||
data "oci_core_instance" "us_instance_details" {
|
||||
provider = oci.us
|
||||
count = length(data.oci_core_instances.us_instances.instances)
|
||||
instance_id = data.oci_core_instances.us_instances.instances[count.index].id
|
||||
}
|
||||
|
||||
# 输出韩国区实例信息
|
||||
output "korea_instances" {
|
||||
description = "韩国区实例状态"
|
||||
value = {
|
||||
count = length(data.oci_core_instances.korea_instances.instances)
|
||||
instances = [
|
||||
for instance in data.oci_core_instance.korea_instance_details : {
|
||||
id = instance.id
|
||||
name = instance.display_name
|
||||
state = instance.state
|
||||
shape = instance.shape
|
||||
region = "ap-chuncheon-1"
|
||||
ad = instance.availability_domain
|
||||
public_ip = instance.public_ip
|
||||
private_ip = instance.private_ip
|
||||
time_created = instance.time_created
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# 输出美国区实例信息
|
||||
output "us_instances" {
|
||||
description = "美国区实例状态"
|
||||
value = {
|
||||
count = length(data.oci_core_instances.us_instances.instances)
|
||||
instances = [
|
||||
for instance in data.oci_core_instance.us_instance_details : {
|
||||
id = instance.id
|
||||
name = instance.display_name
|
||||
state = instance.state
|
||||
shape = instance.shape
|
||||
region = "us-ashburn-1"
|
||||
ad = instance.availability_domain
|
||||
public_ip = instance.public_ip
|
||||
private_ip = instance.private_ip
|
||||
time_created = instance.time_created
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# 输出总计信息
|
||||
output "summary" {
|
||||
description = "实例总计信息"
|
||||
value = {
|
||||
total_instances = length(data.oci_core_instances.korea_instances.instances) + length(data.oci_core_instances.us_instances.instances)
|
||||
korea_count = length(data.oci_core_instances.korea_instances.instances)
|
||||
us_count = length(data.oci_core_instances.us_instances.instances)
|
||||
}
|
||||
}
|
||||
|
|
@ -104,7 +104,7 @@ provider "oci" {
|
|||
tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
private_key = data.consul_keys.oracle_config.var.private_key
|
||||
private_key = file(var.oci_config.private_key_path)
|
||||
region = "ap-chuncheon-1"
|
||||
}
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ provider "oci" {
|
|||
tenancy_ocid = data.consul_keys.oracle_config_us.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config_us.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config_us.var.fingerprint
|
||||
private_key = data.consul_keys.oracle_config_us.var.private_key
|
||||
private_key = file(var.oci_config.private_key_path)
|
||||
region = "us-ashburn-1"
|
||||
}
|
||||
|
||||
|
|
@ -135,17 +135,14 @@ module "oracle_cloud" {
|
|||
tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
private_key = data.consul_keys.oracle_config.var.private_key
|
||||
private_key_path = var.oci_config.private_key_path
|
||||
region = "ap-chuncheon-1"
|
||||
compartment_ocid = ""
|
||||
}
|
||||
|
||||
# 开发环境特定配置
|
||||
instance_count = 1
|
||||
instance_size = "VM.Standard.E2.1.Micro" # 免费层
|
||||
|
||||
providers = {
|
||||
oci = oci
|
||||
}
|
||||
}
|
||||
|
||||
# 输出
|
||||
|
|
@ -153,3 +150,43 @@ output "oracle_cloud_outputs" {
|
|||
description = "Oracle Cloud 基础设施输出"
|
||||
value = module.oracle_cloud
|
||||
}
|
||||
|
||||
# Nomad 多数据中心集群
|
||||
module "nomad_cluster" {
|
||||
source = "../../modules/nomad-cluster"
|
||||
|
||||
# 部署控制变量 - 禁用所有计算资源创建
|
||||
deploy_korea_node = false
|
||||
deploy_us_node = false # 暂时禁用美国节点
|
||||
|
||||
# Oracle Cloud 配置
|
||||
oracle_config = {
|
||||
tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
private_key_path = var.oci_config.private_key_path
|
||||
region = "ap-chuncheon-1"
|
||||
compartment_ocid = ""
|
||||
}
|
||||
|
||||
# 通用配置
|
||||
common_tags = var.common_tags
|
||||
ssh_public_key = var.ssh_public_key
|
||||
|
||||
# Nomad 特定配置
|
||||
nomad_version = "1.7.7"
|
||||
nomad_encrypt_key = var.nomad_encrypt_key
|
||||
|
||||
# Oracle Cloud 特定配置
|
||||
oracle_availability_domain = "Uocm:AP-CHUNCHEON-1-AD-1"
|
||||
oracle_subnet_id = module.oracle_cloud.subnet_ids[0] # 使用第一个子网
|
||||
|
||||
# 依赖关系
|
||||
depends_on = [module.oracle_cloud]
|
||||
}
|
||||
|
||||
# 输出 Nomad 集群信息
|
||||
output "nomad_cluster" {
|
||||
description = "Nomad 多数据中心集群信息"
|
||||
value = module.nomad_cluster
|
||||
}
|
||||
|
|
@ -152,3 +152,18 @@ variable "vault_token" {
|
|||
default = ""
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
# SSH 公钥配置
|
||||
variable "ssh_public_key" {
|
||||
description = "SSH 公钥，用于访问云实例"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
# Nomad 配置
|
||||
variable "nomad_encrypt_key" {
|
||||
description = "Nomad 集群加密密钥"
|
||||
type = string
|
||||
default = ""
|
||||
sensitive = true
|
||||
}
|
||||
|
|
@ -0,0 +1,158 @@
|
|||
# Nomad 多数据中心集群模块
|
||||
# 支持跨地域部署：CN(dc1) + KR(dc2) + US(dc3)
|
||||
|
||||
terraform {
|
||||
required_providers {
|
||||
oci = {
|
||||
source = "oracle/oci"
|
||||
version = "~> 7.20"
|
||||
}
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# 本地变量
|
||||
locals {
|
||||
nomad_version = "1.10.5"
|
||||
|
||||
# 通用 Nomad 配置
|
||||
nomad_encrypt_key = "NVOMDvXblgWfhtzFzOUIHnKEOrbXOkPrkIPbRGGf1YQ="
|
||||
|
||||
# 数据中心配置
|
||||
datacenters = {
|
||||
dc1 = {
|
||||
name = "dc1"
|
||||
region = "cn"
|
||||
location = "China"
|
||||
provider = "existing" # 现有的 semaphore 节点
|
||||
}
|
||||
dc2 = {
|
||||
name = "dc2"
|
||||
region = "kr"
|
||||
location = "Korea"
|
||||
provider = "oracle"
|
||||
}
|
||||
dc3 = {
|
||||
name = "dc3"
|
||||
region = "us"
|
||||
location = "US"
|
||||
provider = "aws" # 暂时使用AWS替代华为云
|
||||
}
|
||||
}
|
||||
|
||||
# 用户数据模板
|
||||
user_data_template = templatefile("${path.module}/templates/nomad-userdata.sh", {
|
||||
nomad_version = local.nomad_version
|
||||
nomad_encrypt_key = local.nomad_encrypt_key
|
||||
VERSION_ID = "20.04" # Ubuntu 20.04
|
||||
NOMAD_VERSION = local.nomad_version
|
||||
NOMAD_ZIP = "nomad_${local.nomad_version}_linux_amd64.zip"
|
||||
NOMAD_URL = "https://releases.hashicorp.com/nomad/${local.nomad_version}/nomad_${local.nomad_version}_linux_amd64.zip"
|
||||
NOMAD_SHA256_URL = "https://releases.hashicorp.com/nomad/${local.nomad_version}/nomad_${local.nomad_version}_SHA256SUMS"
|
||||
bind_addr = "auto"
|
||||
nomad_servers = "\"127.0.0.1\""
|
||||
})
|
||||
}
|
||||
|
||||
# 数据源：获取现有的 semaphore 节点信息
|
||||
data "external" "semaphore_info" {
|
||||
program = ["bash", "-c", <<-EOF
|
||||
echo '{
|
||||
"ip": "100.116.158.95",
|
||||
"datacenter": "dc1",
|
||||
"status": "existing"
|
||||
}'
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
# Oracle Cloud 韩国节点 (dc2)
|
||||
resource "oci_core_instance" "nomad_kr_node" {
|
||||
count = var.deploy_korea_node ? 1 : 0
|
||||
|
||||
# 基础配置
|
||||
compartment_id = var.oracle_config.compartment_ocid
|
||||
display_name = "nomad-master-kr"
|
||||
availability_domain = var.oracle_availability_domain
|
||||
shape = "VM.Standard.E2.1.Micro" # 免费层
|
||||
|
||||
# 源配置
|
||||
source_details {
|
||||
source_type = "image"
|
||||
source_id = var.oracle_ubuntu_image_id
|
||||
}
|
||||
|
||||
# 网络配置
|
||||
create_vnic_details {
|
||||
subnet_id = var.oracle_subnet_id
|
||||
display_name = "nomad-kr-vnic"
|
||||
assign_public_ip = true
|
||||
}
|
||||
|
||||
# 元数据
|
||||
metadata = {
|
||||
ssh_authorized_keys = var.ssh_public_key
|
||||
user_data = base64encode(templatefile("${path.module}/templates/nomad-userdata.sh", {
|
||||
datacenter = "dc2"
|
||||
nomad_version = local.nomad_version
|
||||
nomad_encrypt_key = local.nomad_encrypt_key
|
||||
bootstrap_expect = 1
|
||||
bind_addr = "auto"
|
||||
server_enabled = true
|
||||
client_enabled = true
|
||||
VERSION_ID = "20.04" # Ubuntu 20.04
|
||||
NOMAD_VERSION = local.nomad_version
|
||||
NOMAD_ZIP = "nomad_${local.nomad_version}_linux_amd64.zip"
|
||||
NOMAD_URL = "https://releases.hashicorp.com/nomad/${local.nomad_version}/nomad_${local.nomad_version}_linux_amd64.zip"
|
||||
NOMAD_SHA256_URL = "https://releases.hashicorp.com/nomad/${local.nomad_version}/nomad_${local.nomad_version}_SHA256SUMS"
|
||||
nomad_servers = "\"127.0.0.1\""
|
||||
}))
|
||||
}
|
||||
|
||||
# 标签
|
||||
defined_tags = merge(var.common_tags, {
|
||||
"Name" = "nomad-master-kr"
|
||||
"Datacenter" = "dc2"
|
||||
"Role" = "nomad-server"
|
||||
"Provider" = "oracle"
|
||||
})
|
||||
}
|
||||
|
||||
# 华为云美国节点 (dc3) - 暂时禁用
|
||||
# resource "huaweicloud_compute_instance_v2" "nomad_us_node" {
|
||||
# count = var.deploy_us_node ? 1 : 0
|
||||
#
|
||||
# name = "nomad-ash3c-us"
|
||||
# image_id = var.huawei_ubuntu_image_id
|
||||
# flavor_id = "s6.small.1" # 1vCPU 1GB
|
||||
#
|
||||
# # 网络配置
|
||||
# network {
|
||||
# uuid = var.huawei_subnet_id
|
||||
# }
|
||||
#
|
||||
# # 元数据
|
||||
# metadata = {
|
||||
# ssh_authorized_keys = var.ssh_public_key
|
||||
# user_data = base64encode(templatefile("${path.module}/templates/nomad-userdata.sh", {
|
||||
# datacenter = "dc3"
|
||||
# nomad_version = local.nomad_version
|
||||
# nomad_encrypt_key = local.nomad_encrypt_key
|
||||
# bootstrap_expect = 1
|
||||
# bind_addr = "auto"
|
||||
# server_enabled = true
|
||||
# client_enabled = true
|
||||
# }))
|
||||
# }
|
||||
#
|
||||
# # 标签
|
||||
# tags = merge(var.common_tags, {
|
||||
# Name = "nomad-ash3c-us"
|
||||
# Datacenter = "dc3"
|
||||
# Role = "nomad-server"
|
||||
# Provider = "huawei"
|
||||
# })
|
||||
# }
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
# Nomad 多数据中心集群输出
|
||||
|
||||
# 集群概览
|
||||
output "cluster_overview" {
|
||||
description = "Nomad 多数据中心集群概览"
|
||||
value = {
|
||||
datacenters = {
|
||||
dc1 = {
|
||||
name = "dc1"
|
||||
location = "China (CN)"
|
||||
provider = "existing"
|
||||
node = "semaphore"
|
||||
ip = "100.116.158.95"
|
||||
status = "existing"
|
||||
}
|
||||
dc2 = var.deploy_korea_node ? {
|
||||
name = "dc2"
|
||||
location = "Korea (KR)"
|
||||
provider = "oracle"
|
||||
node = "master"
|
||||
ip = try(oci_core_instance.nomad_kr_node[0].public_ip, "pending")
|
||||
status = "deployed"
|
||||
} : null
|
||||
dc3 = var.deploy_us_node ? {
|
||||
name = "dc3"
|
||||
location = "US"
|
||||
provider = "aws" # 暂时使用AWS替代华为云
|
||||
node = "ash3c"
|
||||
ip = "pending" # 暂时禁用
|
||||
status = "disabled"
|
||||
} : null
|
||||
}
|
||||
total_nodes = 1 + (var.deploy_korea_node ? 1 : 0) + (var.deploy_us_node ? 1 : 0)
|
||||
}
|
||||
}
|
||||
|
||||
# Oracle Cloud 韩国节点输出
|
||||
output "oracle_korea_node" {
|
||||
description = "Oracle Cloud 韩国节点信息"
|
||||
value = var.deploy_korea_node ? {
|
||||
instance_id = try(oci_core_instance.nomad_kr_node[0].id, null)
|
||||
public_ip = try(oci_core_instance.nomad_kr_node[0].public_ip, null)
|
||||
private_ip = try(oci_core_instance.nomad_kr_node[0].private_ip, null)
|
||||
datacenter = "dc2"
|
||||
provider = "oracle"
|
||||
region = var.oracle_config.region
|
||||
|
||||
# 连接信息
|
||||
ssh_command = try("ssh ubuntu@${oci_core_instance.nomad_kr_node[0].public_ip}", null)
|
||||
nomad_ui = try("http://${oci_core_instance.nomad_kr_node[0].public_ip}:4646", null)
|
||||
} : null
|
||||
}
|
||||
|
||||
# 华为云美国节点输出 - 暂时禁用
|
||||
# output "huawei_us_node" {
|
||||
# description = "华为云美国节点信息"
|
||||
# value = var.deploy_us_node ? {
|
||||
# instance_id = try(huaweicloud_compute_instance_v2.nomad_us_node[0].id, null)
|
||||
# public_ip = try(huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4, null)
|
||||
# private_ip = try(huaweicloud_compute_instance_v2.nomad_us_node[0].network[0].fixed_ip_v4, null)
|
||||
# datacenter = "dc3"
|
||||
# provider = "huawei"
|
||||
# region = var.huawei_config.region
|
||||
#
|
||||
# # 连接信息
|
||||
# ssh_command = try("ssh ubuntu@${huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4}", null)
|
||||
# nomad_ui = try("http://${huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4}:4646", null)
|
||||
# } : null
|
||||
# }
|
||||
|
||||
# 集群连接信息
|
||||
output "cluster_endpoints" {
|
||||
description = "集群连接端点"
|
||||
value = {
|
||||
nomad_ui_urls = compact([
|
||||
"http://100.116.158.95:4646", # dc1 - semaphore
|
||||
var.deploy_korea_node ? try("http://${oci_core_instance.nomad_kr_node[0].public_ip}:4646", null) : null, # dc2
|
||||
# var.deploy_us_node ? try("http://${huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4}:4646", null) : null # dc3 - 暂时禁用
|
||||
])
|
||||
|
||||
ssh_commands = compact([
|
||||
"ssh root@100.116.158.95", # dc1 - semaphore
|
||||
var.deploy_korea_node ? try("ssh ubuntu@${oci_core_instance.nomad_kr_node[0].public_ip}", null) : null, # dc2
|
||||
# var.deploy_us_node ? try("ssh ubuntu@${huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4}", null) : null # dc3 - 暂时禁用
|
||||
])
|
||||
}
|
||||
}
|
||||
|
||||
# Ansible inventory 生成
|
||||
output "ansible_inventory" {
|
||||
description = "生成的 Ansible inventory"
|
||||
value = {
|
||||
all = {
|
||||
children = {
|
||||
nomad_servers = {
|
||||
hosts = merge(
|
||||
{
|
||||
semaphore = {
|
||||
ansible_host = "100.116.158.95"
|
||||
datacenter = "dc1"
|
||||
provider = "existing"
|
||||
}
|
||||
},
|
||||
var.deploy_korea_node ? {
|
||||
master = {
|
||||
ansible_host = try(oci_core_instance.nomad_kr_node[0].public_ip, "pending")
|
||||
datacenter = "dc2"
|
||||
provider = "oracle"
|
||||
}
|
||||
} : {}
|
||||
# var.deploy_us_node ? {
|
||||
# ash3c = {
|
||||
# ansible_host = try(huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4, "pending")
|
||||
# datacenter = "dc3"
|
||||
# provider = "huawei"
|
||||
# }
|
||||
# } : {} # 暂时禁用
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# 部署后验证命令
|
||||
output "verification_commands" {
|
||||
description = "部署后验证命令"
|
||||
value = [
|
||||
"# 检查集群状态",
|
||||
"nomad server members",
|
||||
"",
|
||||
"# 检查各数据中心节点",
|
||||
"nomad node status -verbose",
|
||||
"",
|
||||
"# 跨数据中心任务调度测试",
|
||||
"nomad job run examples/cross-dc-test.nomad",
|
||||
"",
|
||||
"# 访问 UI",
|
||||
join("\n", [for url in compact([
|
||||
"http://100.116.158.95:4646",
|
||||
var.deploy_korea_node ? try("http://${oci_core_instance.nomad_kr_node[0].public_ip}:4646", null) : null,
|
||||
# var.deploy_us_node ? try("http://${huaweicloud_compute_instance_v2.nomad_us_node[0].access_ip_v4}:4646", null) : null # dc3 - 暂时禁用
|
||||
]) : "curl -s ${url}/v1/status/leader"])
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,276 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Nomad 节点用户数据脚本
|
||||
# 用于自动配置 Nomad 节点，支持服务器和客户端模式
|
||||
|
||||
set -e
|
||||
|
||||
# 日志函数
|
||||
log() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
|
||||
}
|
||||
|
||||
log "开始 Nomad 节点配置..."
|
||||
|
||||
# 更新系统
|
||||
log "更新系统包..."
|
||||
apt-get update
|
||||
apt-get upgrade -y
|
||||
|
||||
# 安装必要工具
|
||||
log "安装必要工具..."
|
||||
apt-get install -y curl unzip wget gnupg software-properties-common
|
||||
|
||||
# 安装 Podman (作为容器运行时)
|
||||
log "安装 Podman..."
|
||||
. /etc/os-release
|
||||
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
|
||||
curl -L "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | apt-key add -
|
||||
apt-get update
|
||||
apt-get install -y podman
|
||||
|
||||
# 配置 Podman
|
||||
log "配置 Podman..."
|
||||
mkdir -p /etc/containers
|
||||
echo -e "[registries.search]\nregistries = ['docker.io']" > /etc/containers/registries.conf
|
||||
|
||||
# 下载并安装 Nomad
|
||||
log "安装 Nomad..."
|
||||
NOMAD_VERSION=${nomad_version}
|
||||
NOMAD_ZIP="nomad_${NOMAD_VERSION}_linux_amd64.zip"
|
||||
NOMAD_URL="https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/${NOMAD_ZIP}"
|
||||
NOMAD_SHA256_URL="https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_SHA256SUMS"
|
||||
|
||||
cd /tmp
|
||||
wget -q ${NOMAD_URL}
|
||||
wget -q ${NOMAD_SHA256_URL}
|
||||
sha256sum -c nomad_${NOMAD_VERSION}_SHA256SUMS --ignore-missing
|
||||
unzip -o ${NOMAD_ZIP} -d /usr/local/bin/
|
||||
chmod +x /usr/local/bin/nomad
|
||||
|
||||
# 创建 Nomad 用户和目录
|
||||
log "创建 Nomad 用户和目录..."
|
||||
useradd --system --home /etc/nomad.d --shell /bin/false nomad
|
||||
mkdir -p /opt/nomad/data
|
||||
mkdir -p /etc/nomad.d
|
||||
mkdir -p /var/log/nomad
|
||||
chown -R nomad:nomad /opt/nomad /etc/nomad.d /var/log/nomad
|
||||
|
||||
# 获取本机 IP 地址
|
||||
if [ "${bind_addr}" = "auto" ]; then
|
||||
# 尝试多种方法获取 IP
|
||||
BIND_ADDR=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4 2>/dev/null || \
|
||||
curl -s http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip -H "Metadata-Flavor: Google" 2>/dev/null || \
|
||||
ip route get 8.8.8.8 | awk '{print $7; exit}' || \
|
||||
hostname -I | awk '{print $1}')
|
||||
else
|
||||
BIND_ADDR="${bind_addr}"
|
||||
fi
|
||||
|
||||
log "检测到 IP 地址: $BIND_ADDR"
|
||||
|
||||
# 创建 Nomad 配置文件
|
||||
log "创建 Nomad 配置文件..."
|
||||
cat > /etc/nomad.d/nomad.hcl << EOF
|
||||
# Nomad 配置文件
|
||||
datacenter = "${datacenter}"
|
||||
data_dir = "/opt/nomad/data"
|
||||
log_level = "INFO"
|
||||
|
||||
# 客户端配置
|
||||
client {
|
||||
enabled = true
|
||||
servers = ["${nomad_servers}"]
|
||||
options {
|
||||
"driver.raw_exec.enable" = "1"
|
||||
"driver.podman.enabled" = "1"
|
||||
}
|
||||
}
|
||||
|
||||
# 服务器配置
|
||||
server {
|
||||
enabled = ${server_enabled}
|
||||
bootstrap_expect = ${bootstrap_expect}
|
||||
}
|
||||
|
||||
# Consul 集成
|
||||
consul {
|
||||
address = "127.0.0.1:8500"
|
||||
token = "${consul_token}"
|
||||
}
|
||||
|
||||
# 加密设置
|
||||
encrypt = "${nomad_encrypt_key}"
|
||||
|
||||
# 网络配置
|
||||
network {
|
||||
mode = "bridge"
|
||||
}
|
||||
|
||||
# UI 配置
|
||||
ui {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
# 插件目录
|
||||
plugin_dir = "/opt/nomad/plugins"
|
||||
EOF
|
||||
|
||||
# 创建 systemd 服务文件
|
||||
log "创建 systemd 服务文件..."
|
||||
cat > /etc/systemd/system/nomad.service << EOF
|
||||
[Unit]
|
||||
Description=Nomad
|
||||
Documentation=https://www.nomadproject.io/
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
ExecReload=/bin/kill -HUP \$MAINPID
|
||||
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d
|
||||
KillMode=process
|
||||
KillSignal=SIGINT
|
||||
LimitNOFILE=65536
|
||||
LimitNPROC=infinity
|
||||
Restart=on-failure
|
||||
RestartSec=2
|
||||
StartLimitBurst=3
|
||||
StartLimitInterval=10
|
||||
TasksMax=infinity
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# 启动 Nomad 服务
|
||||
log "启动 Nomad 服务..."
|
||||
systemctl daemon-reload
|
||||
systemctl enable nomad
|
||||
systemctl start nomad
|
||||
|
||||
# 等待服务启动
|
||||
log "等待 Nomad 服务启动..."
|
||||
sleep 10
|
||||
|
||||
# 验证 Nomad 状态
|
||||
if systemctl is-active --quiet nomad; then
|
||||
log "Nomad 服务启动成功"
|
||||
else
|
||||
log "Nomad 服务启动失败"
|
||||
journalctl -u nomad --no-pager
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 创建 Nomad 客户端状态检查脚本
|
||||
log "创建状态检查脚本..."
|
||||
cat > /usr/local/bin/check-nomad.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
# Nomad 状态检查脚本
|
||||
|
||||
set -e
|
||||
|
||||
# 检查 Nomad 服务状态
|
||||
if systemctl is-active --quiet nomad; then
|
||||
echo "Nomad 服务运行正常"
|
||||
else
|
||||
echo "Nomad 服务未运行"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查 Nomad 节点状态
|
||||
NODE_STATUS=$(nomad node status -self -json | jq -r '.Status')
|
||||
if [ "$NODE_STATUS" = "ready" ]; then
|
||||
echo "Nomad 节点状态: $NODE_STATUS"
|
||||
else
|
||||
echo "Nomad 节点状态异常: $NODE_STATUS"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查 Nomad 集群成员
|
||||
SERVER_MEMBERS=$(nomad server members 2>/dev/null | grep -c "alive" || echo "0")
|
||||
if [ "$SERVER_MEMBERS" -gt 0 ]; then
|
||||
echo "Nomad 集群服务器成员: $SERVER_MEMBERS"
|
||||
else
|
||||
echo "未找到 Nomad 集群服务器成员"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Nomad 状态检查完成"
|
||||
EOF
|
||||
|
||||
chmod +x /usr/local/bin/check-nomad.sh
|
||||
|
||||
# 设置防火墙规则
|
||||
log "设置防火墙规则..."
|
||||
if command -v ufw >/dev/null 2>&1; then
|
||||
ufw allow 4646/tcp # Nomad HTTP
|
||||
ufw allow 4647/tcp # Nomad RPC
|
||||
ufw allow 4648/tcp # Nomad Serf
|
||||
ufw --force enable
|
||||
elif command -v firewall-cmd >/dev/null 2>&1; then
|
||||
firewall-cmd --permanent --add-port=4646/tcp
|
||||
firewall-cmd --permanent --add-port=4647/tcp
|
||||
firewall-cmd --permanent --add-port=4648/tcp
|
||||
firewall-cmd --reload
|
||||
fi
|
||||
|
||||
# 创建简单的 Nomad 任务示例
|
||||
log "创建示例任务..."
|
||||
mkdir -p /opt/nomad/examples
|
||||
cat > /opt/nomad/examples/redis.nomad << 'EOF'
|
||||
job "redis" {
|
||||
datacenters = ["dc1", "dc2", "dc3"]
|
||||
type = "service"
|
||||
priority = 50
|
||||
|
||||
update {
|
||||
stagger = "10s"
|
||||
max_parallel = 1
|
||||
}
|
||||
|
||||
group "redis" {
|
||||
count = 1
|
||||
|
||||
restart {
|
||||
attempts = 3
|
||||
delay = "30s"
|
||||
interval = "5m"
|
||||
mode = "fail"
|
||||
}
|
||||
|
||||
task "redis" {
|
||||
driver = "podman"
|
||||
|
||||
config {
|
||||
image = "redis:alpine"
|
||||
ports = ["redis"]
|
||||
}
|
||||
|
||||
resources {
|
||||
cpu = 200 # MHz
|
||||
memory = 128 # MB
|
||||
|
||||
network {
|
||||
mbits = 10
|
||||
port "redis" {
|
||||
static = 6379
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
service {
|
||||
name = "redis"
|
||||
port = "redis"
|
||||
check {
|
||||
type = "tcp"
|
||||
interval = "10s"
|
||||
timeout = "2s"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
log "Nomad 节点配置完成"
|
||||
log "Nomad UI 可通过 http://$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4):4646 访问"
|
||||
|
|
@ -7,9 +7,9 @@ variable "deploy_korea_node" {
|
|||
}
|
||||
|
||||
variable "deploy_us_node" {
|
||||
description = "是否部署美国节点 (华为云)"
|
||||
description = "是否部署美国节点 (暂时禁用)"
|
||||
type = bool
|
||||
default = true
|
||||
default = false
|
||||
}
|
||||
|
||||
# Oracle Cloud 配置
|
||||
|
|
@ -21,10 +21,17 @@ variable "oracle_config" {
|
|||
fingerprint = string
|
||||
private_key_path = string
|
||||
region = string
|
||||
compartment_ocid = string
|
||||
})
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "oracle_availability_domain" {
|
||||
description = "Oracle Cloud 可用域"
|
||||
type = string
|
||||
default = "" # 将通过数据源自动获取
|
||||
}
|
||||
|
||||
variable "oracle_ubuntu_image_id" {
|
||||
description = "Oracle Cloud Ubuntu 镜像 ID"
|
||||
type = string
|
||||
|
|
@ -36,37 +43,27 @@ variable "oracle_subnet_id" {
|
|||
type = string
|
||||
}
|
||||
|
||||
variable "oracle_security_group_id" {
|
||||
description = "Oracle Cloud 安全组 ID"
|
||||
type = string
|
||||
}
|
||||
# 华为云配置 - 暂时禁用
|
||||
# variable "huawei_config" {
|
||||
# description = "华为云配置"
|
||||
# type = object({
|
||||
# access_key = string
|
||||
# secret_key = string
|
||||
# region = string
|
||||
# })
|
||||
# sensitive = true
|
||||
# }
|
||||
|
||||
# 华为云配置
|
||||
variable "huawei_config" {
|
||||
description = "华为云配置"
|
||||
type = object({
|
||||
access_key = string
|
||||
secret_key = string
|
||||
region = string
|
||||
})
|
||||
sensitive = true
|
||||
}
|
||||
# variable "huawei_ubuntu_image_id" {
|
||||
# description = "华为云 Ubuntu 镜像 ID"
|
||||
# type = string
|
||||
# default = "" # 将通过数据源自动获取
|
||||
# }
|
||||
|
||||
variable "huawei_ubuntu_image_id" {
|
||||
description = "华为云 Ubuntu 镜像 ID"
|
||||
type = string
|
||||
default = "" # 将通过数据源自动获取
|
||||
}
|
||||
|
||||
variable "huawei_subnet_id" {
|
||||
description = "华为云子网 ID"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "huawei_security_group_id" {
|
||||
description = "华为云安全组 ID"
|
||||
type = string
|
||||
}
|
||||
# variable "huawei_subnet_id" {
|
||||
# description = "华为云子网 ID"
|
||||
# type = string
|
||||
# }
|
||||
|
||||
# 通用配置
|
||||
variable "common_tags" {
|
||||
|
|
@ -9,6 +9,15 @@ terraform {
|
|||
}
|
||||
}
|
||||
|
||||
# OCI Provider 配置
|
||||
provider "oci" {
|
||||
tenancy_ocid = var.oci_config.tenancy_ocid
|
||||
user_ocid = var.oci_config.user_ocid
|
||||
fingerprint = var.oci_config.fingerprint
|
||||
private_key = file(var.oci_config.private_key_path)
|
||||
region = var.oci_config.region
|
||||
}
|
||||
|
||||
# 获取可用域
|
||||
data "oci_identity_availability_domains" "ads" {
|
||||
compartment_id = var.oci_config.tenancy_ocid
|
||||
|
|
@ -36,7 +36,7 @@ variable "oci_config" {
|
|||
tenancy_ocid = string
|
||||
user_ocid = string
|
||||
fingerprint = string
|
||||
private_key = string
|
||||
private_key_path = string
|
||||
region = string
|
||||
compartment_ocid = string
|
||||
})
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
# CSOL 基础设施建设 - 7天创造世界
|
||||
|
||||
## 概述
|
||||
|
||||
本文档描述了CSOL基础设施建设的完整流程，采用"7天创造世界"的比喻，系统地阐述了从网络连接到应用部署的完整建设过程。
|
||||
|
||||
## 第1天：Tailscale - 网络连接基础
|
||||
|
||||
**目标**：打通所有分布式地点的网络连接
|
||||
|
||||
**核心任务**：
|
||||
- 在所有节点部署Tailscale，建立安全的网络连接
|
||||
- 确保所有节点可以通过Tailscale网络相互访问
|
||||
- 为后续的分布式管理奠定网络基础
|
||||
|
||||
**关键成果**：
|
||||
- 所有节点加入Tailscale网络
|
||||
- 节点间可以通过Tailscale IP直接通信
|
||||
- 为后续的Ansible、Nomad等工具提供网络基础
|
||||
|
||||
## 第2天：Ansible - 分布式控制
|
||||
|
||||
**目标**：实现灵活的分布式节点控制
|
||||
|
||||
**核心任务**：
|
||||
- 部署Ansible作为配置管理工具
|
||||
- 建立inventory文件，管理所有节点信息
|
||||
- 编写playbook，实现"八爪鱼式"的远程控制能力
|
||||
|
||||
**关键成果**：
|
||||
- 可以通过Ansible批量管理所有节点
|
||||
- 标准化的配置管理流程
|
||||
- 自动化的软件部署和配置更新
|
||||
|
||||
## 第3天：Nomad - 服务感知与任务调度
|
||||
|
||||
**目标**：建立服务感知能力和任务调度系统，提供容错性
|
||||
|
||||
**核心任务**：
|
||||
- 部署Nomad集群，实现资源调度
|
||||
- 配置服务器节点和客户端节点
|
||||
- 建立服务发现和健康检查机制
|
||||
|
||||
**关键成果**：
|
||||
- 集群具备任务调度能力
|
||||
- 服务自动发现和故障转移
|
||||
- 资源的高效利用和负载均衡
|
||||
|
||||
## 第4天：Consul - 配置集中管理
|
||||
|
||||
**目标**：解决容器技术配置的集中管理问题
|
||||
|
||||
**核心任务**：
|
||||
- 部署Consul集群，提供配置管理和服务发现
|
||||
- 通过Nomad拉起Consul服务
|
||||
- 建立键值存储，用于动态配置管理
|
||||
|
||||
**关键成果**：
|
||||
- 配置的集中管理和动态更新
|
||||
- 服务注册与发现
|
||||
- 为后续的Vault集成提供基础
|
||||
|
||||
## 第5天：Terraform - 状态一致性
|
||||
|
||||
**目标**：解决基础设施状态一致性问题
|
||||
|
||||
**核心任务**：
|
||||
- 使用Terraform管理基础设施资源
|
||||
- 建立基础设施即代码(IaC)的实践
|
||||
- 确保环境状态的一致性和可重复性
|
||||
|
||||
**关键成果**：
|
||||
- 基础设施的声明式管理
|
||||
- 状态的一致性和可预测性
|
||||
- 环境的快速复制和重建能力
|
||||
|
||||
## 第6天：Vault - 安全密钥管理
|
||||
|
||||
**目标**：解决大规模自动化编程中的环境变量和敏感信息管理
|
||||
|
||||
**核心任务**：
|
||||
- 部署Vault集群，提供密钥管理服务
|
||||
- 集成Vault与Nomad、Consul
|
||||
- 建立动态密钥管理机制
|
||||
|
||||
**关键成果**：
|
||||
- 敏感信息的集中安全管理
|
||||
- 动态密钥生成和轮换
|
||||
- 为自动化流程提供安全的配置获取方式
|
||||
|
||||
## 第7天：Waypoint - 应用部署现代化
|
||||
|
||||
**目标**：实现应用部署的现代化管理
|
||||
|
||||
**核心任务**：
|
||||
- 部署Waypoint，提供应用生命周期管理
|
||||
- 建立标准化的应用部署流程
|
||||
- 集成CI/CD流程
|
||||
|
||||
**关键成果**：
|
||||
- 应用部署的标准化和自动化
|
||||
- 开发体验的提升
|
||||
- 完整的应用生命周期管理
|
||||
|
||||
## 建设原则
|
||||
|
||||
1. **循序渐进**：严格按照7天的顺序进行建设，每个阶段的基础是前一个阶段的完成
|
||||
2. **依赖明确**：每个工具都有明确的依赖关系，确保架构的合理性
|
||||
3. **功能互补**：每个工具解决特定问题，形成完整的基础设施解决方案
|
||||
4. **可扩展性**：整个架构设计考虑未来的扩展需求
|
||||
|
||||
## 重要提醒
|
||||
|
||||
**当前问题**：本地节点不接受任务，导致无法部署Consul，造成配置混乱
|
||||
|
||||
**解决方案**：
|
||||
1. 将本地节点也设置为Consul的管理节点
|
||||
2. 确保本地节点能够接受和执行任务
|
||||
3. 建立sticky note机制，不断提醒自己配置状态和依赖关系
|
||||
|
||||
**核心逻辑**：只有解决了本地节点的任务接受问题，才能正确部署Consul，进而保证整个基础设施建设的顺利进行。
|
||||
|
|
@ -66,16 +66,6 @@ data "consul_keys" "oracle_config" {
|
|||
}
|
||||
```
|
||||
|
||||
#### 私钥文件创建
|
||||
|
||||
```hcl
|
||||
# 将从Consul获取的私钥保存到临时文件
|
||||
resource "local_file" "oci_kr_private_key" {
|
||||
content = data.consul_keys.oracle_config.var.private_key
|
||||
filename = "/tmp/oci_kr_private_key.pem"
|
||||
}
|
||||
```
|
||||
|
||||
#### OCI Provider配置
|
||||
|
||||
```hcl
|
||||
|
|
@ -84,7 +74,7 @@ provider "oci" {
|
|||
tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
private_key_path = local_file.oci_kr_private_key.filename
|
||||
private_key = file(var.oci_config.private_key_path)
|
||||
region = "ap-chuncheon-1"
|
||||
}
|
||||
```
|
||||
|
|
@ -140,8 +130,8 @@ terraform apply -var-file=consul.tfvars
|
|||
|
||||
使用Consul Provider直接从Consul获取配置有以下优势：
|
||||
|
||||
1. **更高的安全性**：私钥不再需要存储在磁盘上的配置文件中，而是直接从Consul获取
|
||||
2. **更简洁的配置**：无需手动创建临时文件，Terraform自动处理
|
||||
1. **更高的安全性**：私钥不再需要存储在磁盘上的临时文件中，而是直接从Consul获取并在内存中使用
|
||||
2. **更简洁的配置**：无需手动创建临时文件，Terraform直接处理私钥内容
|
||||
3. **声明式风格**：完全符合Terraform的声明式配置风格
|
||||
4. **更好的维护性**：配置集中存储在Consul中，便于管理和更新
|
||||
5. **多环境支持**：可以轻松支持多个环境（dev、staging、production）的配置
|
||||
|
|
|
|||
|
|
@ -0,0 +1,37 @@
|
|||
# DigitalOcean 密钥存储作业
|
||||
job "digitalocean-key-store" {
|
||||
datacenters = ["dc1"]
|
||||
type = "batch"
|
||||
|
||||
group "key-store" {
|
||||
task "store-key" {
|
||||
driver = "exec"
|
||||
|
||||
config {
|
||||
command = "/bin/sh"
|
||||
args = [
|
||||
"-c",
|
||||
<<EOT
|
||||
# 将DigitalOcean密钥存储到Consul中
|
||||
curl -X PUT -H "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
|
||||
http://127.0.0.1:8500/v1/kv/council/digitalocean/token \
|
||||
-d 'dop_v1_70582bb508873709d96debc7f2a2d04df2093144b2b15fe392dba83b88976376'
|
||||
|
||||
# 验证密钥是否存储成功
|
||||
curl -s http://127.0.0.1:8500/v1/kv/council/digitalocean/token?raw
|
||||
EOT
|
||||
]
|
||||
}
|
||||
|
||||
env {
|
||||
CONSUL_HTTP_ADDR = "http://127.0.0.1:8500"
|
||||
CONSUL_HTTP_TOKEN = "root" # 根据实际Consul配置调整
|
||||
}
|
||||
|
||||
resources {
|
||||
cpu = 100
|
||||
memory = 64
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Consul KV 命名规范
|
||||
|
||||
本文档描述了在Consul KV中存储配置信息的统一命名规范，以确保所有配置管理的一致性和可维护性。
|
||||
|
||||
## 命名规范
|
||||
|
||||
### 基本格式
|
||||
|
||||
```
|
||||
config/{environment}/{provider}/{region_or_service}/{key}
|
||||
```
|
||||
|
||||
### 各部分说明
|
||||
|
||||
- **config**: 固定前缀，表示这是一个配置项
|
||||
- **environment**: 环境名称，如 `dev`、`staging`、`prod` 等
|
||||
- **provider**: 云服务提供商，如 `oracle`、`digitalocean`、`aws`、`gcp` 等
|
||||
- **region_or_service**: 区域或服务名称，如 `kr`、`us`、`sgp` 等
|
||||
- **key**: 具体的配置键名，如 `token`、`tenancy_ocid`、`user_ocid` 等
|
||||
|
||||
### 示例
|
||||
|
||||
#### Oracle Cloud 配置
|
||||
|
||||
```
|
||||
config/dev/oracle/kr/tenancy_ocid
|
||||
config/dev/oracle/kr/user_ocid
|
||||
config/dev/oracle/kr/fingerprint
|
||||
config/dev/oracle/kr/private_key
|
||||
config/dev/oracle/kr/region
|
||||
|
||||
config/dev/oracle/us/tenancy_ocid
|
||||
config/dev/oracle/us/user_ocid
|
||||
config/dev/oracle/us/fingerprint
|
||||
config/dev/oracle/us/private_key
|
||||
config/dev/oracle/us/region
|
||||
```
|
||||
|
||||
#### DigitalOcean 配置
|
||||
|
||||
```
|
||||
config/dev/digitalocean/token
|
||||
```
|
||||
|
||||
#### 其他云服务商配置（示例）
|
||||
|
||||
```
|
||||
config/dev/aws/access_key
|
||||
config/dev/aws/secret_key
|
||||
config/dev/aws/region
|
||||
|
||||
config/dev/gcp/project_id
|
||||
config/dev/gcp/credentials_file
|
||||
config/dev/gcp/region
|
||||
```
|
||||
|
||||
## 使用说明
|
||||
|
||||
### 添加新配置
|
||||
|
||||
当需要为新的云服务商或环境添加配置时，请遵循上述命名规范：
|
||||
|
||||
1. 确定环境名称（如 `dev`）
|
||||
2. 确定云服务提供商（如 `aws`）
|
||||
3. 确定区域或服务（如 `ap-northeast-2`）
|
||||
4. 确定具体的配置键名（如 `access_key`）
|
||||
|
||||
例如：
|
||||
```
|
||||
consul kv put config/dev/aws/ap-northeast-2/access_key your_access_key
|
||||
```
|
||||
|
||||
### 在Terraform中使用
|
||||
|
||||
在Terraform配置中，使用 `consul_keys` 数据源获取配置：
|
||||
|
||||
```hcl
|
||||
data "consul_keys" "aws_config" {
|
||||
key {
|
||||
name = "access_key"
|
||||
path = "config/dev/aws/ap-northeast-2/access_key"
|
||||
}
|
||||
key {
|
||||
name = "secret_key"
|
||||
path = "config/dev/aws/ap-northeast-2/secret_key"
|
||||
}
|
||||
}
|
||||
|
||||
provider "aws" {
|
||||
access_key = data.consul_keys.aws_config.var.access_key
|
||||
secret_key = data.consul_keys.aws_config.var.secret_key
|
||||
region = "ap-northeast-2"
|
||||
}
|
||||
```
|
||||
|
||||
### 与Vault集成
|
||||
|
||||
当需要与Vault集成时，可以使用相同的命名规范，确保Consul和Vault中的配置路径保持一致。
|
||||
|
||||
## 维护说明
|
||||
|
||||
- 所有Agent在添加新的Consul KV键时，必须遵循此命名规范
|
||||
- 定期检查Consul KV中的键，确保符合规范
|
||||
- 如需修改命名规范，请更新此文档并通知所有相关Agent
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q: 为什么不使用服务名称作为前缀（如 `oracle/config/dev/...`）？
|
||||
|
||||
A: 使用 `config` 作为统一前缀可以更容易地区分配置项和其他类型的键值对，便于管理和筛选。
|
||||
|
||||
### Q: 如何处理敏感信息？
|
||||
|
||||
A: 敏感信息（如API密钥、私钥等）应存储在Vault中，Consul主要用于非敏感配置。如果必须在Consul中存储敏感信息，请确保Consul集群的安全性。
|
||||
|
||||
### Q: 如何处理多环境配置？
|
||||
|
||||
A: 通过修改 `environment` 部分来区分不同环境，如 `config/dev/...`、`config/staging/...`、`config/prod/...`。
|
||||
|
||||
## 更新历史
|
||||
|
||||
- 2024-01-01: 初始版本，定义了基本的命名规范
|
||||
- 2024-01-02: 统一DigitalOcean配置路径，从 `consul/digitalocean/token` 改为 `config/dev/digitalocean/token`
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# 从Consul获取DigitalOcean API Token
|
||||
data "consul_keys" "do_token" {
|
||||
key {
|
||||
name = "token"
|
||||
path = "config/dev/digitalocean/token"
|
||||
default = ""
|
||||
}
|
||||
}
|
||||
|
||||
# DigitalOcean 提供者配置
|
||||
provider "digitalocean" {
|
||||
token = data.consul_keys.do_token.var.token
|
||||
}
|
||||
|
|
@ -38,6 +38,12 @@ terraform {
|
|||
source = "hashicorp/vault"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
|
||||
# DigitalOcean Provider
|
||||
digitalocean = {
|
||||
source = "digitalocean/digitalocean"
|
||||
version = "~> 2.0"
|
||||
}
|
||||
}
|
||||
|
||||
# 后端配置
|
||||
|
|
@ -118,38 +124,39 @@ provider "oci" {
|
|||
region = "us-ashburn-1"
|
||||
}
|
||||
|
||||
# Oracle Cloud 基础设施
|
||||
module "oracle_cloud" {
|
||||
source = "../../providers/oracle-cloud"
|
||||
|
||||
# 传递变量
|
||||
environment = var.environment
|
||||
project_name = var.project_name
|
||||
owner = var.owner
|
||||
vpc_cidr = var.vpc_cidr
|
||||
availability_zones = var.availability_zones
|
||||
common_tags = var.common_tags
|
||||
|
||||
# 使用从Consul获取的配置
|
||||
oci_config = {
|
||||
tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
private_key = data.consul_keys.oracle_config.var.private_key
|
||||
region = "ap-chuncheon-1"
|
||||
}
|
||||
|
||||
# 开发环境特定配置
|
||||
instance_count = 1
|
||||
instance_size = "VM.Standard.E2.1.Micro" # 免费层
|
||||
|
||||
providers = {
|
||||
oci = oci
|
||||
}
|
||||
}
|
||||
# Oracle Cloud 基础设施 - 暂时注释掉以避免VCN数量限制问题
|
||||
# module "oracle_cloud" {
|
||||
# source = "../../providers/oracle-cloud"
|
||||
#
|
||||
# # 传递变量
|
||||
# environment = var.environment
|
||||
# project_name = var.project_name
|
||||
# owner = var.owner
|
||||
# vpc_cidr = var.vpc_cidr
|
||||
# availability_zones = var.availability_zones
|
||||
# common_tags = var.common_tags
|
||||
#
|
||||
# # 使用从Consul获取的配置
|
||||
# oci_config = {
|
||||
# tenancy_ocid = data.consul_keys.oracle_config.var.tenancy_ocid
|
||||
# user_ocid = data.consul_keys.oracle_config.var.user_ocid
|
||||
# fingerprint = data.consul_keys.oracle_config.var.fingerprint
|
||||
# private_key = data.consul_keys.oracle_config.var.private_key
|
||||
# region = "ap-chuncheon-1"
|
||||
# compartment_ocid = data.consul_keys.oracle_config.var.tenancy_ocid # 使用tenancy_ocid作为compartment_ocid
|
||||
# }
|
||||
#
|
||||
# # 开发环境特定配置
|
||||
# instance_count = 1
|
||||
# instance_size = "VM.Standard.E2.1.Micro" # 免费层
|
||||
#
|
||||
# providers = {
|
||||
# oci = oci
|
||||
# }
|
||||
# }
|
||||
|
||||
# 输出
|
||||
output "oracle_cloud_outputs" {
|
||||
description = "Oracle Cloud 基础设施输出"
|
||||
value = module.oracle_cloud
|
||||
}
|
||||
# output "oracle_cloud_outputs" {
|
||||
# description = "Oracle Cloud 基础设施输出"
|
||||
# value = module.oracle_cloud
|
||||
# }
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue