475 lines
14 KiB
Markdown
475 lines
14 KiB
Markdown
# 07 - 配置系统详细设计
|
||
|
||
## 1. 模块概述
|
||
|
||
配置系统（`config/`）是 COBOL 迁移验证平台 V3 的全局管理模块，提供两级配置体系：
|
||
|
||
1. **全局配置**（`Config`）：项目级参数，包括 LLM 模型、超时、容忍度、Spark 配置、质量门限等
|
||
2. **程序级配置**（`ProgramSchema`）：每个 COBOL 程序的 DB 表定义、子程序列表、运行场景
|
||
|
||
此外，`MappingConfig` 提供 COBOL 字段到 Java 字段的映射配置，支持多种转换策略。
|
||
|
||
配置来源包括 TOML 文件（全局）、YAML 文件（程序级 schema 和字段映射）。
|
||
|
||
## 2. 文件清单
|
||
|
||
| 文件 | 职责 | 依赖 |
|
||
|------|------|------|
|
||
| `__init__.py` | 包入口，公开 API 导出 | `mapping`, `dataclasses` |
|
||
| `mapping.py` | COBOL-Java 字段映射配置 | `yaml`, `dataclasses` |
|
||
| `program_schema.py` | 程序级 DB schema + 运行场景定义 | `yaml`, `dataclasses` |
|
||
| `programs/*.yaml` | 58 个电信程序的 YAML schema 文件 | 无 |
|
||
|
||
## 3. 两级配置体系
|
||
|
||
### 3.1 架构图
|
||
|
||
```
|
||
全局配置 (Config)
|
||
|-- from_toml("aurak.toml")
|
||
|-- LLM 模型/超时/缓存
|
||
|-- Spark 配置
|
||
|-- 质量门限
|
||
|-- gixsql 配置
|
||
|
|
||
+-- 程序级配置 (ProgramSchema)
|
||
|-- load_schema("KYU04CAL")
|
||
|-- db_tables: [TableDef, ...]
|
||
|-- subprograms: [str, ...]
|
||
|-- runs: [ScenarioDef, ...]
|
||
|
|
||
+-- 字段映射 (MappingConfig)
|
||
|-- from_yaml("KYU04CAL.yaml")
|
||
|-- field_mappings: [FieldMapping, ...]
|
||
|-- redefines_strategy: dict
|
||
```
|
||
|
||
### 3.2 配置优先级
|
||
|
||
1. 命令行参数（最高）
|
||
2. 环境变量
|
||
3. TOML 配置文件
|
||
4. YAML schema 文件
|
||
5. 代码默认值（最低）
|
||
|
||
## 4. 全局配置（Config）
|
||
|
||
### 4.1 数据类定义
|
||
|
||
```python
|
||
@dataclass
|
||
class Config:
|
||
project_name: str = ""
|
||
copybook_paths: list = ["./copybooks"]
|
||
dialect: str = "ibm"
|
||
llm_model: str = "deepseek-v4-flash"
|
||
llm_timeout: int = 120
|
||
llm_cache_dir: str = ".cache/llm"
|
||
coverage_default: str = "boundary"
|
||
rounding_mode: str = "TRUNCATE"
|
||
tolerance: float = 0.01
|
||
runner_mode: str = "native"
|
||
spark_master: str = "local[*]"
|
||
spark_input_format: str = "json"
|
||
num_records: int = 1000
|
||
branch_pass: float = 0.80
|
||
max_llm_cost: float = 0.50
|
||
quality_gate_mode: str = "warn"
|
||
quality_gate_decision_threshold: float = 0.90
|
||
quality_gate_paragraph_threshold: float = 1.0
|
||
gcov_enabled: bool = False
|
||
gcov_work_dir: str = ".gcov_output"
|
||
gcov_threshold: float = 0.5
|
||
max_quality_retries: int = 4
|
||
gixsql_path: str = "gixsql/bin/gixpp.exe"
|
||
gixsql_lib_path: str = "gixsql/lib"
|
||
gixsql_db_path: str = ".db/gixsql"
|
||
gixsql_compile_flags: str = "-fixed -ext cpy --coverage"
|
||
```
|
||
|
||
### 4.2 配置分组
|
||
|
||
| 分组 | 配置项 | 说明 |
|
||
|------|--------|------|
|
||
| 项目基础 | `project_name`, `copybook_paths`, `dialect` | 项目标识、COPYBOOK 路径、COBOL 方言 |
|
||
| LLM | `llm_model`, `llm_timeout`, `llm_cache_dir`, `max_llm_cost` | 模型选择、超时、缓存、成本控制 |
|
||
| 覆盖率 | `coverage_default`, `branch_pass`, `gcov_*` | 覆盖率目标、分支通过率、gcov 集成 |
|
||
| 比对 | `rounding_mode`, `tolerance` | 舍入模式、数值容忍度 |
|
||
| 运行器 | `runner_mode`, `spark_*`, `num_records` | native/spark 模式、Spark 参数 |
|
||
| 质量门限 | `quality_gate_*`, `max_quality_retries` | 决策阈值、段落阈值、重试次数 |
|
||
| gixsql | `gixsql_*` | DB 程序编译路径、库路径、编译标志 |
|
||
|
||
### 4.3 TOML 加载
|
||
|
||
```python
|
||
@classmethod
|
||
def from_toml(cls, path="aurak.toml") -> Config
|
||
```
|
||
|
||
TOML 文件结构：
|
||
|
||
```toml
|
||
[project]
|
||
name = "..."
|
||
copybook_paths = ["./copybooks"]
|
||
dialect = "ibm"
|
||
|
||
[llm]
|
||
model = "deepseek-v4-flash"
|
||
|
||
[coverage]
|
||
default_target = "boundary"
|
||
|
||
[comparison]
|
||
rounding_mode = "TRUNCATE"
|
||
default_tolerance = 0.01
|
||
|
||
[runner]
|
||
mode = "native"
|
||
|
||
[spark]
|
||
master = "local[*]"
|
||
num_records = 1000
|
||
|
||
[gixsql]
|
||
path = "gixsql/bin/gixpp.exe"
|
||
lib_path = "gixsql/lib"
|
||
db_path = ".db/gixsql"
|
||
compile_flags = "-fixed -ext cpy --coverage"
|
||
```
|
||
|
||
### 4.4 错误处理
|
||
|
||
- TOML 文件不存在或解析失败 -> 返回默认 `Config()` 实例
|
||
- 缺失字段使用默认值填充
|
||
|
||
## 5. 字段映射配置（mapping.py）
|
||
|
||
### 5.1 数据类定义
|
||
|
||
```python
|
||
@dataclass
|
||
class FieldMapping:
|
||
cobol_field: str # COBOL 字段名
|
||
java_field: str # Java 字段名
|
||
field_type: str = "string" # 字段类型
|
||
precision: int = 0 # 精度（小数位）
|
||
trim: bool = False # 是否去空格
|
||
format: str = "" # 格式化规则
|
||
init_strategy: str = "auto" # 初始化策略
|
||
|
||
@dataclass
|
||
class MappingConfig:
|
||
program: str = "" # 程序 ID
|
||
dialect: str = "ibm" # COBOL 方言
|
||
field_mappings: list[FieldMapping] # 字段映射列表
|
||
redefines_strategy: dict # REDEFINES 处理策略
|
||
```
|
||
|
||
### 5.2 YAML 加载
|
||
|
||
```python
|
||
@classmethod
|
||
def from_yaml(cls, path: str) -> MappingConfig
|
||
```
|
||
|
||
YAML 文件结构：
|
||
|
||
```yaml
|
||
program: KYU04CAL
|
||
dialect: ibm
|
||
field_mapping:
|
||
- cobol_field: "BR-AMT"
|
||
java_field: "billAmount"
|
||
field_type: "decimal"
|
||
precision: 2
|
||
- cobol_field: "CUST-ID"
|
||
java_field: "customerId"
|
||
field_type: "string"
|
||
trim: true
|
||
redefines_strategy:
|
||
WS-BLOCK: "overlay"
|
||
```
|
||
|
||
### 5.3 字段查询
|
||
|
||
```python
|
||
def get_java_field(self, cobol_name: str) -> str
|
||
```
|
||
|
||
按 COBOL 字段名查找对应的 Java 字段名，未找到时返回原始 COBOL 字段名。
|
||
|
||
### 5.4 模块级断言
|
||
|
||
```python
|
||
_m = FieldMapping(cobol_field="BR-AMT", java_field="billAmount",
|
||
field_type="decimal", precision=2)
|
||
assert _m.cobol_field == "BR-AMT"
|
||
```
|
||
|
||
导入时自动执行结构验证。
|
||
|
||
## 6. 程序级 Schema 配置（program_schema.py）
|
||
|
||
### 6.1 数据类定义
|
||
|
||
```python
|
||
@dataclass
|
||
class ColumnDef:
|
||
name: str # 列名
|
||
type: str # SQL 类型（CHAR(6), NUMERIC(4), VARCHAR(30)）
|
||
primary_key: bool = False # 是否主键
|
||
nullable: bool = False # 是否可空
|
||
default: Optional[str] = None # 默认值
|
||
cobol_field: Optional[str] = None # 对应 COBOL 字段名
|
||
|
||
@dataclass
|
||
class TableDef:
|
||
name: str # 表名
|
||
columns: list[ColumnDef] # 列定义列表
|
||
create_if_missing: bool = True # 缺失时自动创建
|
||
sql_name: Optional[str] = None # COBOL SQL 中的表名（可能与 YAML 名不同）
|
||
|
||
@dataclass
|
||
class SysinDef:
|
||
period: str | None = "202607" # 处理期间
|
||
include_invalid_period: bool = False # 是否包含无效期间
|
||
modes: list[str] = ["NORMAL"] # 运行模式列表
|
||
final_mode: str = "RESET" # 最终模式
|
||
|
||
@dataclass
|
||
class ScenarioDef:
|
||
id: str # 场景 ID
|
||
sysin: SysinDef # SYSIN 卡片配置
|
||
inject_duplicate_pk: bool = False # 是否注入重复主键
|
||
row_overrides: dict[str, dict[str, str]] # 行级覆盖
|
||
delete_all_rows: bool = False # 是否清空所有行
|
||
drop_tables: list[str] = [] # 需要删除的表列表
|
||
command_line: str | None = None # 自定义命令行
|
||
seed_extra_rows: dict[str, int] = {} # 额外种子行数
|
||
|
||
@dataclass
|
||
class ProgramSchema:
|
||
program_id: str # 程序 ID
|
||
db_tables: list[TableDef] # DB 表定义列表
|
||
subprograms: list[str] # 子程序列表
|
||
db_type: str = "SQLite" # 数据库类型
|
||
db_name: str = "OVERTIME.DB" # 数据库文件名
|
||
runs: list[ScenarioDef] # 运行场景列表
|
||
coverage_dates: dict[str, list[dict]] # 覆盖率日期配置
|
||
command_line: str | None = None # 全局命令行
|
||
```
|
||
|
||
### 6.2 YAML 加载
|
||
|
||
```python
|
||
@classmethod
|
||
def from_yaml(cls, path: str | Path) -> ProgramSchema
|
||
```
|
||
|
||
### 6.3 Schema 查找
|
||
|
||
```python
|
||
def load_schema(program_id: str, search_dirs: list[str | Path] | None = None) -> ProgramSchema
|
||
```
|
||
|
||
在 `config/programs/` 目录下按 `{program_id}.yaml` 查找，未找到时抛出 `FileNotFoundError`。
|
||
|
||
## 7. YAML Schema 配置
|
||
|
||
### 7.1 文件结构
|
||
|
||
每个程序对应一个 YAML 文件，位于 `config/programs/` 目录：
|
||
|
||
```
|
||
config/programs/
|
||
KYU04CAL.yaml
|
||
KYU05DED.yaml
|
||
KYU06UPD.yaml
|
||
KIN02UPD.yaml
|
||
KIN03EXP.yaml
|
||
KIN06CLD.yaml
|
||
KIN08DBU.yaml
|
||
KIN09CSV.yaml
|
||
SHA02MNC.yaml
|
||
SHA03MNP.yaml
|
||
SHA04TWO.yaml
|
||
SHA06TWM.yaml
|
||
SHA07KBR.yaml
|
||
ZAN06UPD.yaml
|
||
...
|
||
```
|
||
|
||
### 7.2 YAML Schema 模板
|
||
|
||
```yaml
|
||
program_id: {PROGRAM_ID}
|
||
db_type: SQLite
|
||
db_name: {DB_NAME}
|
||
|
||
db_tables:
|
||
- name: {COBOL_TABLE_NAME}
|
||
sql_name: {SQL_TABLE_NAME} # 可选，COBOL SQL 中的表名
|
||
columns:
|
||
- name: {COL_NAME}
|
||
type: {SQL_TYPE} # CHAR(N), NUMERIC(N,M), VARCHAR(N), DECIMAL(N,M), TIMESTAMP
|
||
primary_key: true/false
|
||
nullable: true/false
|
||
cobol_field: {COBOL_FIELD} # 可选，对应 COBOL 字段名
|
||
|
||
subprograms:
|
||
- {SUB_PROGRAM_ID}
|
||
|
||
runs:
|
||
- id: {SCENARIO_ID}
|
||
sysin:
|
||
period: "{YYYYMM}"
|
||
modes: ["NORMAL"]
|
||
inject_duplicate_pk: true/false
|
||
row_overrides:
|
||
{TABLE_NAME}:
|
||
{COL_NAME}: "{VALUE}"
|
||
delete_all_rows: true/false
|
||
drop_tables:
|
||
- {TABLE_NAME}
|
||
command_line: "{COMMAND}" # 可选
|
||
seed_extra_rows:
|
||
{TABLE_NAME}: {COUNT}
|
||
```
|
||
|
||
### 7.3 SQL 类型映射
|
||
|
||
| YAML 类型 | SQL 类型 | 说明 |
|
||
|-----------|----------|------|
|
||
| `CHAR(N)` | 定长字符 | COBOL DISPLAY 类型 |
|
||
| `VARCHAR(N)` | 变长字符 | COBOL X(n) 类型 |
|
||
| `NUMERIC(N)` | 整数 | 无小数位 |
|
||
| `NUMERIC(N,M)` | 定点数 | N 位总长，M 位小数 |
|
||
| `DECIMAL(N,M)` | 定点数 | 同 NUMERIC |
|
||
| `TIMESTAMP` | 时间戳 | 日期时间 |
|
||
|
||
### 7.4 运行场景（ScenarioDef）
|
||
|
||
每个程序可定义多个运行场景，用于覆盖不同测试路径：
|
||
|
||
| 场景类型 | 典型配置 | 说明 |
|
||
|----------|----------|------|
|
||
| `normal` | 默认 SYSIN | 正常路径测试 |
|
||
| `collision` | `inject_duplicate_pk: true` | 主键冲突测试 |
|
||
| `abnormal` | 自定义 row_overrides | 异常路径测试 |
|
||
| 自定义 | `command_line` | 特殊命令行参数 |
|
||
|
||
### 7.5 SYSIN 卡片配置
|
||
|
||
SYSIN 卡片是 COBOL 程序的运行时输入，通过 `SysinDef` 配置：
|
||
|
||
- `period`：处理期间（如 `"202607"`），用于时间相关逻辑
|
||
- `modes`：运行模式列表（如 `["NORMAL"]`），影响程序分支
|
||
- `include_invalid_period`：是否包含无效期间测试
|
||
- `final_mode`：最终模式（如 `"RESET"`）
|
||
|
||
## 8. 配置加载机制
|
||
|
||
### 8.1 加载流程
|
||
|
||
```
|
||
1. 读取 aurak.toml -> Config.from_toml()
|
||
2. 根据 program_id 查找 programs/{id}.yaml -> load_schema()
|
||
3. 可选: 读取字段映射 YAML -> MappingConfig.from_yaml()
|
||
4. 合并全局配置 + 程序级配置 -> 运行时参数
|
||
```
|
||
|
||
### 8.2 默认值策略
|
||
|
||
- 所有配置项均有代码级默认值
|
||
- TOML/YAML 仅覆盖显式指定的字段
|
||
- 缺失字段回退到 dataclass 默认值
|
||
|
||
### 8.3 已有程序 Schema 统计
|
||
|
||
截至当前，`config/programs/` 目录包含 14 个程序的 YAML schema：
|
||
|
||
| 程序 ID | DB 表数 | 子程序数 | 运行场景数 |
|
||
|---------|---------|----------|------------|
|
||
| KYU04CAL | - | - | - |
|
||
| KYU05DED | - | - | - |
|
||
| KYU06UPD | - | - | - |
|
||
| KIN02UPD | - | - | - |
|
||
| KIN03EXP | - | - | - |
|
||
| KIN06CLD | - | - | - |
|
||
| KIN08DBU | - | - | - |
|
||
| KIN09CSV | - | - | - |
|
||
| SHA02MNC | - | - | - |
|
||
| SHA03MNP | - | - | - |
|
||
| SHA04TWO | - | - | - |
|
||
| SHA06TWM | - | - | - |
|
||
| SHA07KBR | 2 | 2 | 0 |
|
||
| ZAN06UPD | 2 | 5 | 3 |
|
||
|
||
## 9. 接口定义
|
||
|
||
### 9.1 模块公开 API
|
||
|
||
```python
|
||
# config/__init__.py
|
||
Config # 全局配置 dataclass
|
||
MappingConfig # 字段映射配置
|
||
FieldMapping # 单个字段映射
|
||
|
||
# config/mapping.py
|
||
MappingConfig.from_yaml(path) -> MappingConfig
|
||
MappingConfig.get_java_field(cobol_name) -> str
|
||
|
||
# config/program_schema.py
|
||
ProgramSchema.from_yaml(path) -> ProgramSchema
|
||
load_schema(program_id, search_dirs) -> ProgramSchema
|
||
```
|
||
|
||
### 9.2 典型使用模式
|
||
|
||
```python
|
||
from config import Config
|
||
from config.program_schema import load_schema
|
||
from config.mapping import MappingConfig
|
||
|
||
# 加载全局配置
|
||
config = Config.from_toml("aurak.toml")
|
||
|
||
# 加载程序 schema
|
||
schema = load_schema("KYU04CAL")
|
||
|
||
# 加载字段映射（如果存在）
|
||
mapping = MappingConfig.from_yaml("mappings/KYU04CAL.yaml")
|
||
java_field = mapping.get_java_field("BR-AMT")
|
||
|
||
# 使用配置
|
||
for table in schema.db_tables:
|
||
print(f"Table: {table.name}, SQL: {table.sql_name}")
|
||
for col in table.columns:
|
||
print(f" {col.name}: {col.type} (PK={col.primary_key})")
|
||
|
||
for scenario in schema.runs:
|
||
print(f"Scenario: {scenario.id}, Period: {scenario.sysin.period}")
|
||
```
|
||
|
||
## 10. 错误处理
|
||
|
||
### 10.1 错误策略
|
||
|
||
| 异常场景 | 处理方式 | 影响 |
|
||
|----------|----------|------|
|
||
| TOML 文件不存在 | 返回默认 Config | 使用代码默认值 |
|
||
| TOML 解析失败 | 返回默认 Config | 使用代码默认值 |
|
||
| YAML schema 不存在 | 抛出 FileNotFoundError | 流水线中断 |
|
||
| YAML 字段缺失 | 使用默认值填充 | 可能不完整 |
|
||
| SQL 类型无法识别 | 原样传递 | 下游处理 |
|
||
|
||
### 10.2 已知局限
|
||
|
||
1. `Config.from_toml` 吞掉所有异常，配置错误静默降级
|
||
2. `load_schema` 搜索路径硬编码为 `config/programs/`
|
||
3. `MappingConfig` 的 `redefines_strategy` 仅存储字典，无验证逻辑
|
||
4. `ProgramSchema` 的 `coverage_dates` 类型定义为 `dict[str, list[dict[str, str]]]`，缺乏 schema 验证
|
||
5. `SysinDef` 的 `period` 字段无格式校验，允许非法期间值
|
||
6. 程序级 YAML 无统一 schema 验证，依赖调用方保证格式正确
|