调整
This commit is contained in:
+27
-20
@@ -6,6 +6,7 @@
|
|||||||
- `09:00` 推送结果到飞书群
|
- `09:00` 推送结果到飞书群
|
||||||
|
|
||||||
调度配置写在 [prefect.yaml](/Users/idss/YXT_Project/gmv-analytics/prefect.yaml) 中。
|
调度配置写在 [prefect.yaml](/Users/idss/YXT_Project/gmv-analytics/prefect.yaml) 中。
|
||||||
|
运行容器的职责是启动 Prefect worker,不再使用 `--serve` 作为主方案。
|
||||||
|
|
||||||
## 1. 创建 Docker 镜像
|
## 1. 创建 Docker 镜像
|
||||||
|
|
||||||
@@ -13,7 +14,7 @@
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /Users/idss/YXT_Project/gmv-analytics
|
cd /Users/idss/YXT_Project/gmv-analytics
|
||||||
docker build -t gmv-analytics-prefect:latest .
|
docker build -t gmv-analytics-prefect:0.1.0 .
|
||||||
```
|
```
|
||||||
|
|
||||||
镜像内容来自:
|
镜像内容来自:
|
||||||
@@ -25,10 +26,11 @@ docker build -t gmv-analytics-prefect:latest .
|
|||||||
|
|
||||||
- 加载 `.env`
|
- 加载 `.env`
|
||||||
- 自动把 `PROJECT_MAPPING_FILE` 修正为容器内的 `/app/店铺映射表.xlsx`
|
- 自动把 `PROJECT_MAPPING_FILE` 修正为容器内的 `/app/店铺映射表.xlsx`
|
||||||
- 默认执行:
|
- 默认执行 `supervisord`
|
||||||
|
- 再由 [supervisord.conf](/Users/idss/YXT_Project/gmv-analytics/supervisord.conf) 启动:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python prefect_gmv_scheduler.py --serve
|
prefect worker start --pool "gmv-analytics-pool" --type process
|
||||||
```
|
```
|
||||||
|
|
||||||
## 2. 启动 Docker 容器
|
## 2. 启动 Docker 容器
|
||||||
@@ -37,17 +39,17 @@ python prefect_gmv_scheduler.py --serve
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d \
|
docker run -d \
|
||||||
--name gmv-prefect-scheduler \
|
--name prefect-gmv-worker \
|
||||||
--env-file /Users/idss/YXT_Project/gmv-analytics/.env \
|
--env-file /Users/idss/YXT_Project/gmv-analytics/.env \
|
||||||
-e PROJECT_MAPPING_FILE=/app/店铺映射表.xlsx \
|
-e PROJECT_MAPPING_FILE=/app/店铺映射表.xlsx \
|
||||||
gmv-analytics-prefect:latest
|
gmv-analytics-prefect:0.1.0
|
||||||
```
|
```
|
||||||
|
|
||||||
停止容器:
|
停止容器:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker stop gmv-prefect-scheduler
|
docker stop prefect-gmv-worker
|
||||||
docker rm gmv-prefect-scheduler
|
docker rm prefect-gmv-worker
|
||||||
```
|
```
|
||||||
|
|
||||||
### 方式 B:用 `docker compose`
|
### 方式 B:用 `docker compose`
|
||||||
@@ -102,12 +104,12 @@ prefect deploy --all
|
|||||||
- Prefect API 服务可访问
|
- Prefect API 服务可访问
|
||||||
- 已经有可用的 work pool
|
- 已经有可用的 work pool
|
||||||
当前 [prefect.yaml](/Users/idss/YXT_Project/gmv-analytics/prefect.yaml) 默认使用:
|
当前 [prefect.yaml](/Users/idss/YXT_Project/gmv-analytics/prefect.yaml) 默认使用:
|
||||||
`default-agent-pool`
|
`gmv-analytics-pool`
|
||||||
|
|
||||||
如果还没有 worker,可以在宿主机启动一个:
|
如果还没有 worker,可以在宿主机启动一个:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
prefect worker start --pool default-agent-pool --type process
|
prefect worker start --pool gmv-analytics-pool --type process
|
||||||
```
|
```
|
||||||
|
|
||||||
## 4. 自动执行是怎么发生的
|
## 4. 自动执行是怎么发生的
|
||||||
@@ -121,16 +123,16 @@ prefect worker start --pool default-agent-pool --type process
|
|||||||
3. Prefect worker 拉起执行
|
3. Prefect worker 拉起执行
|
||||||
4. 代码入口在 [prefect_gmv_scheduler.py](/Users/idss/YXT_Project/gmv-analytics/prefect_gmv_scheduler.py)
|
4. 代码入口在 [prefect_gmv_scheduler.py](/Users/idss/YXT_Project/gmv-analytics/prefect_gmv_scheduler.py)
|
||||||
|
|
||||||
如果你走的是容器内 `--serve` 方案:
|
如果你走的是 Docker worker 方案:
|
||||||
|
|
||||||
1. 容器常驻运行 `python prefect_gmv_scheduler.py --serve`
|
1. 容器常驻运行 `prefect worker start --pool "gmv-analytics-pool" --type process`
|
||||||
2. 代码里会本地注册并维持两个定时调度
|
2. Prefect Server 到点触发 deployment
|
||||||
3. 到时间后容器内部直接执行 flow
|
3. worker 从 `gmv-analytics-pool` 拉起执行
|
||||||
|
|
||||||
建议:
|
建议:
|
||||||
|
|
||||||
- 正式环境优先用 `prefect.yaml + prefect deploy --all`
|
- 正式环境优先用 `prefect.yaml + prefect deploy --all`
|
||||||
- `--serve` 更适合本地或过渡期
|
- Docker 容器只负责提供 worker 执行环境
|
||||||
|
|
||||||
## 5. 单次手动执行
|
## 5. 单次手动执行
|
||||||
|
|
||||||
@@ -158,7 +160,7 @@ python /Users/idss/YXT_Project/gmv-analytics/prefect_gmv_scheduler.py --mode all
|
|||||||
docker run --rm \
|
docker run --rm \
|
||||||
--env-file /Users/idss/YXT_Project/gmv-analytics/.env \
|
--env-file /Users/idss/YXT_Project/gmv-analytics/.env \
|
||||||
-e PROJECT_MAPPING_FILE=/app/店铺映射表.xlsx \
|
-e PROJECT_MAPPING_FILE=/app/店铺映射表.xlsx \
|
||||||
gmv-analytics-prefect:latest \
|
gmv-analytics-prefect:0.1.0 \
|
||||||
python prefect_gmv_scheduler.py --mode sync
|
python prefect_gmv_scheduler.py --mode sync
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -173,7 +175,7 @@ docker run --rm \
|
|||||||
如果用 `docker run`:
|
如果用 `docker run`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker logs -f gmv-prefect-scheduler
|
docker logs -f prefect-gmv-worker
|
||||||
```
|
```
|
||||||
|
|
||||||
如果用 `docker compose`:
|
如果用 `docker compose`:
|
||||||
@@ -200,14 +202,14 @@ docker compose logs -f
|
|||||||
进入容器查看:
|
进入容器查看:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker exec -it gmv-prefect-scheduler sh
|
docker exec -it prefect-gmv-worker sh
|
||||||
tail -f /app/logs/gmv_sync.log
|
tail -f /app/logs/gmv_sync.log
|
||||||
```
|
```
|
||||||
|
|
||||||
如果是 compose:
|
如果是 compose:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker exec -it gmv-prefect-scheduler sh
|
docker exec -it prefect-gmv-worker sh
|
||||||
tail -f /app/logs/gmv_sync.log
|
tail -f /app/logs/gmv_sync.log
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -244,10 +246,15 @@ cd /Users/idss/YXT_Project/gmv-analytics
|
|||||||
prefect deploy --all
|
prefect deploy --all
|
||||||
```
|
```
|
||||||
|
|
||||||
如果你只是希望一个容器自己在里面定时跑,也可以直接:
|
如果你要的是“容器里运行 worker,由 Prefect Server 统一调度”,直接:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
这时它会直接跑 `--serve`,由容器自己维持调度。
|
然后在能连到 Prefect API 的环境里执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /Users/idss/YXT_Project/gmv-analytics
|
||||||
|
prefect deploy --all
|
||||||
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user