826 lines
30 KiB
Markdown
826 lines
30 KiB
Markdown
# 数据流设计文档
|
||
|
||
> 版本: v1.0 | 日期: 2026-08-22
|
||
> 本文档描述 COBOL 迁移验证平台 V3 的完整数据流，覆盖非 DB（flat file）管道和 DB（SQL/SQLite）管道。
|
||
|
||
---
|
||
|
||
## 1. 数据流概述
|
||
|
||
### 1.1 设计目标
|
||
|
||
| 目标 | 说明 | 实现方式 |
|
||
|------|------|----------|
|
||
| **可追溯性** | 每条测试数据可追溯到源码决策点 | 路径约束 (field, op, value, want_true) 记录决策分支选择 |
|
||
| **一致性** | 非 DB 与 DB 管道共享同一套解析/生成引擎 | cobol_testgen/ 统一提供 extract_structure + generate_data |
|
||
| **容错性** | 单步失败不中断整体管道 | 每步返回 DbPipelineResult(success, data)，失败立即终止后续 |
|
||
| **可观测性** | 全流程可监控、可调试 | 覆盖率报告、gcov 数据、JSON 中间产物、运行日志 |
|
||
|
||
### 1.2 管道对比
|
||
|
||
| 维度 | 非 DB 管道 | DB 管道 |
|
||
|------|-----------|---------|
|
||
| 编排器 | orchestrator.py | orchestrator_db.py |
|
||
| 运行器 | runners/cobol_runner.py | runners/gixsql_runner.py |
|
||
| 输入格式 | 固定长度平面文件 | 平面文件 + SQLite DB 种子 |
|
||
| 输出格式 | 平面文件 | 平面文件 + SQLite DB |
|
||
| 验证方式 | comparator/ 逐字段比对 | Step 6 验证 COBOL/Java 输出一致性 |
|
||
| 步骤数 | 4 步（生成-运行-比对-报告） | 6 步（环境-生成-运行-提取-Java-验证） |
|
||
---
|
||
|
||
## 2. 非 DB 管道数据流
|
||
|
||
### 2.1 总体流程
|
||
|
||
COBOL 源码 (.cbl)
|
||
|
|
||
v [输入阶段]
|
||
read.py: preprocess() + parse_data_division()
|
||
|
|
||
v [生成阶段]
|
||
core.py: build_branch_tree() -> branch_tree + assignments
|
||
|
|
||
v
|
||
design.py: enum_paths() -> path_infos
|
||
|
|
||
v
|
||
design.py: make_base_record() -> base_record
|
||
|
|
||
v
|
||
design.py: apply_constraint() -> records (测试数据)
|
||
|
|
||
v [运行阶段]
|
||
output.py: output_json() -> 测试数据 JSON
|
||
output.py: output_input_files() -> 固定长度平面文件
|
||
|
|
||
v
|
||
runners/cobol_runner.py: compile() + run() -> 输出文件
|
||
|
|
||
v [比对阶段]
|
||
comparator/aligner.py: 按主键对齐 COBOL/Java 记录
|
||
comparator/field_compare.py: 逐字段比较
|
||
comparator/normalizer.py: EBCDIC/COMP-3 标准化
|
||
|
|
||
v
|
||
report/generator.py: 生成 HTML 报告
|
||
|
||
### 2.2 输入阶段
|
||
|
||
**COBOL 源码 -> read.py 预处理 -> DATA DIVISION 解析 -> fields 结构信息**
|
||
|
||
COBOL 源码 (fixed/free format)
|
||
|
|
||
v read.preprocess()
|
||
+-- resolve_copybooks(): 展开 COPY 语句
|
||
+-- _is_fixed_format(): 自动检测 fixed/free 格式
|
||
+-- 展开至每行 <=72 字符 (fixed) 或保持原样 (free)
|
||
+-- 去除 EXEC CICS/SQL 块、逗号、ALL 关键字
|
||
|
|
||
v 预处理后源码 (preprocessed)
|
||
|
|
||
v read.extract_data_division()
|
||
+-- 提取 DATA DIVISION 文本块
|
||
|
|
||
v read.parse_data_division()
|
||
+-- Lark grammar.lark 解析 (Earley parser, dynamic lexer)
|
||
+-- 逐行解析 FieldDef: level, name, PIC, USAGE, VALUE, REDEFINES, OCCURS
|
||
+-- PIC 子句解析 -> PicInfo (type, digits, decimal, length, signed)
|
||
+-- 返回 list[FieldDef]
|
||
|
|
||
v expand_occurs()
|
||
+-- OCCURS 展开为下标副本: WS-CELL(1), WS-CELL(2), ...
|
||
|
||
**输出数据结构:**
|
||
|
||
# fields: list[dict] -- 每个字段一个 dict
|
||
{
|
||
'name': 'R01EMP-ID', # 字段名 (大写)
|
||
'level': 1, # 层号 (01, 05, 77, 88)
|
||
'pic': 'X(8)', # PIC 子句原始文本
|
||
'pic_info': { # PIC 解析结果
|
||
'type': 'alphanumeric', # numeric | alphanumeric | alphabetic
|
||
'digits': 0, # 整数位数 (numeric)
|
||
'decimal': 0, # 小数位数 (numeric)
|
||
'length': 8, # 总长度 (alphanumeric)
|
||
'signed': False,
|
||
},
|
||
'section': 'FILE', # DATA DIVISION 节
|
||
'occurs': 0, # OCCURS 次数
|
||
'redefines': None, # REDEFINES 目标
|
||
'usage': None, # COMP | COMP-3 | BINARY | DISPLAY
|
||
'is_88': False, # 是否 88 级条件
|
||
'parent': None, # 88 级父字段名
|
||
'value': None, # VALUE 子句
|
||
'values': None, # 88 级多值列表
|
||
'is_filler': False,
|
||
}
|
||
### 2.3 生成阶段
|
||
|
||
**fields -> core.py build_branch_tree -> design.py enum_paths -> make_base_record -> apply_constraint -> 测试数据**
|
||
|
||
#### 2.3.1 分支树构建
|
||
|
||
PROCEDURE DIVISION 文本
|
||
|
|
||
v core.build_branch_tree_fallback()
|
||
+-- pipeline_bridge: 3秒超时桥接
|
||
| +-- 优先: procedure_parser.py (新解析器, 行级状态机)
|
||
| +-- 回退: core._BrParser (旧解析器, 正则驱动)
|
||
|
|
||
+-- 段落扫描: scan_paragraphs() -> {name: (start, end)}
|
||
+-- 逐段解析 IF / EVALUATE / PERFORM / READ / WRITE / MOVE / COMPUTE
|
||
+-- 返回 (branch_tree: BrSeq, assignments: dict)
|
||
|
||
**分支树节点类型:**
|
||
|
||
| 节点 | 类 | 属性 | 说明 |
|
||
|------|----|------|------|
|
||
| 序列 | BrSeq | children: list | 顺序执行的语句序列 |
|
||
| 条件 | BrIf | condition, cond_tree, true_seq, false_seq | IF-ELSE 分支 |
|
||
| 评估 | BrEval | subject, subjects, when_list, other_seq | EVALUATE 多分支 |
|
||
| 循环 | BrPerform | perf_type, condition, body_seq | PERFORM UNTIL/VARYING |
|
||
| 查找 | BrSearch | table_name, is_all, when_list | SEARCH/SEARCH ALL |
|
||
| 赋值 | Assign | target, source_info | MOVE/COMPUTE/ADD/SUBTRACT/MULTIPLY/DIVIDE |
|
||
| 调用 | CallNode | program_name, using_params | CALL 子程序 |
|
||
| 跳转 | GoTo | target, body_seq | GO TO |
|
||
| 退出 | ExitNode | exit_type | EXIT PARAGRAPH/PERFORM/PROGRAM |
|
||
|
||
#### 2.3.2 路径枚举
|
||
|
||
branch_tree + assignments
|
||
|
|
||
v design_mcdc.enum_paths() / design.enum_paths()
|
||
+-- 遍历分支树，收集每个决策点 (BrIf/BrEval/BrPerform)
|
||
+-- 对每个决策点生成 True/False 或 WHEN 分支路径
|
||
+-- MC/DC 约束集: cond.mcdc_sets() -> 每个叶条件的 T/F 独立约束
|
||
+-- 路径约束合并: prior_false 累积（EVALUATE WHEN 入口条件）
|
||
+-- PERFORM VARYING: 最后一次迭代约束 (last-iteration)
|
||
+-- SEARCH: 表索引约束 + AT END 分支
|
||
+-- 返回 path_infos: list[(constraints, path_assignments, term_type)]
|
||
|
||
**路径约束格式:**
|
||
|
||
# path_infos 的每个元素
|
||
(
|
||
# constraints: Path = list[Constraint]
|
||
[
|
||
('WRK-MONTH', '>', '0', True), # (field, op, value, want_true)
|
||
('WRK-MONTH', '<', '13', True),
|
||
('R01EMP-ID', '<>', '00000000', True),
|
||
],
|
||
# path_assignments: dict -- 赋值表
|
||
{
|
||
'WRK-PREV-EMP-ID': [{'type': 'move', 'source_vars': ['R01EMP-ID']}],
|
||
'DBV-EMP-ID': [{'type': 'move_literal', 'literal': 'A0000001'}],
|
||
},
|
||
# term_type: str
|
||
'normal' # 或 'abend'
|
||
)
|
||
|
||
#### 2.3.3 测试数据记录生成
|
||
|
||
path_infos + data_fields
|
||
|
|
||
v design.generate_records()
|
||
|
|
||
+-- 遍历每条路径 (seq=1,2,3...):
|
||
| |
|
||
| +-- make_base_record(seq, data_fields)
|
||
| | +-- 按 VALUE 子句设置初始值
|
||
| | +-- 按 PIC 类型生成默认值:
|
||
| | | +-- numeric: _make_numeric_value(idx, seq, total_digits)
|
||
| | | +-- alphanumeric: _make_alpha_value(idx, seq, length)
|
||
| | | +-- date: seq_date(record_num) -> YYYYMMDD
|
||
| | +-- 返回 base_record: dict {field_name: value}
|
||
| |
|
||
| +-- Pass A: propagate_assignments(rec, path_assign, data_fields)
|
||
| | +-- 模拟赋值传播: MOVE/COMPUTE/READ INTO 等
|
||
| |
|
||
| +-- Pass B: apply_constraint(rec, field, op, val, want, ...)
|
||
| | +-- trace_to_root(): 沿 MOVE 链追溯到根字段
|
||
| | +-- invert_through_chain(): 反向求解约束值
|
||
| | +-- satisfying_value(): 满足约束的值计算
|
||
| | | +-- numeric: 边界值 +/-1
|
||
| | | +-- alphanumeric: 字典序边界
|
||
| | | +-- date: YYYYMMDD 格式边界
|
||
| | +-- 递归写入 base_record[field] = 满足值
|
||
| |
|
||
| +-- Pass B.5: forward propagate (变量间 MOVE 一致性)
|
||
| +-- Pass B.75: COMPUTE 重算 (约束修改源字段后)
|
||
| +-- get_term_type(path_cons) -> (filtered_cons, term_type)
|
||
|
|
||
+-- 返回 (records, kept_path_cons, term_types)
|
||
|
||
**测试数据记录格式:**
|
||
|
||
# records: list[dict] -- 每条记录一个 dict
|
||
[
|
||
{
|
||
'R01EMP-ID': 'A0000001',
|
||
'R01DATE': '20250101',
|
||
'R01LINE': '0001',
|
||
'WRK-PREV-EMP-ID': '',
|
||
'WRK-MONTH': '06',
|
||
'DBV-EMP-ID': 'A0000001',
|
||
'_assigned_fields': {'R01EMP-ID', 'R01DATE'}, # 内部标记
|
||
'_w02_path': True, # PREV 连锁标记
|
||
},
|
||
# ... 更多记录
|
||
]
|
||
### 2.4 运行阶段
|
||
|
||
**测试数据 -> output.py -> cobol_runner compile/run -> 输出文件**
|
||
|
||
records + fd_fields + open_dir
|
||
|
|
||
v output.output_json()
|
||
+-- 按 FD 分组: field_to_fd 映射字段到 FD
|
||
+-- 按 open_dir 确定方向: INPUT/OUTPUT/I-O
|
||
+-- 按 roles 分类: input/inout -> input 块, output/inout -> expected_output 块
|
||
+-- 不属于任何 FD 的字段 -> working_storage 块
|
||
+-- 输出 JSON: {program, records: [{input, expected_output, working_storage, termination}]}
|
||
|
|
||
v output.output_input_files()
|
||
+-- 仅处理 INPUT/I-O 方向的 FD
|
||
+-- 按 FD 分组，每 FD 输出一个 JSON: {stem}_{fd_name}.json
|
||
+-- abend 记录单独输出: {stem}_abend_{fd_name}.json
|
||
+-- 二进制模式: 按 field offsets 打包为固定长度二进制文件
|
||
|
|
||
v flatfile.write_all_files()
|
||
+-- analyze_fd_layout(): 解析 FD 记录布局 (字段名/PIC/offset/length)
|
||
+-- 按 FD 布局序列化每条记录为固定长度字节
|
||
+-- write_flat_file(): 输出到 outdir/{assign_name}
|
||
|
|
||
v runners/cobol_runner.compile()
|
||
+-- cobc -std=ibm -free -x src.cbl -o exe
|
||
|
|
||
v runners/cobol_runner.run()
|
||
+-- 设置 CWD，复制输入文件到 input/
|
||
+-- subprocess.run(exe) 执行
|
||
+-- 捕获 stdout/stderr，收集输出文件
|
||
+-- 返回 RunResult(success, records, log)
|
||
|
||
### 2.5 比对阶段
|
||
|
||
**输出文件 -> aligner.py -> field_compare.py -> normalizer.py -> report/generator.py -> HTML 报告**
|
||
|
||
COBOL 输出文件 + Java 输出文件
|
||
|
|
||
v comparator/aligner.py: align_records()
|
||
+-- 按主键字段 (如 CUST-ID) 分组
|
||
+-- 取两侧键的并集，按字符串排序
|
||
+-- 逐键配对: MATCHED / MISSING_IN_SPARK / EXTRA_IN_SPARK
|
||
+-- 返回 list[(cobol_record, java_record, status)]
|
||
|
|
||
v comparator/normalizer.py: normalize_encoding()
|
||
+-- EBCDIC -> ASCII 解码 (EBCDIC_037 映射表)
|
||
+-- COMP-3 压缩十进制解码 (nibble -> decimal)
|
||
+-- 日期格式标准化 (YYYYMMDD -> YYYY-MM-DD)
|
||
|
|
||
v comparator/field_compare.py: compare_field()
|
||
+-- 按字段类型选择比较策略:
|
||
| +-- numeric: Decimal 精度比较 + 容差 (tolerance=0.01)
|
||
| +-- date: YYYYMMDD 格式归一化后比较
|
||
| +-- string: strip 后直接比较
|
||
+-- 返回 FieldResult(field_name, status, cobol_value, java_value)
|
||
+-- status: PASS / MISMATCH / TOLERATED / NOT_SET
|
||
|
|
||
v comparator/rounding_detect.py: detect_rounding()
|
||
+-- 判断数值差异是否由 COBOL ROUNDED 子句引起
|
||
+-- 计算舍入误差范围
|
||
|
|
||
v report/generator.py: ReportGenerator
|
||
+-- generate_json(): 输出 JSON 报告 (VerificationRun 数据)
|
||
+-- generate_html(): 输出 HTML 报告
|
||
| +-- 覆盖率卡片: 段落覆盖率、分支覆盖率、决策点覆盖率
|
||
| +-- HINA 卡片: 判定类型、确信度
|
||
| +-- 质量评分卡片: quality_score
|
||
| +-- 重试历史卡片: heal_retry, simple_retry
|
||
| +-- 字段比对表格: PASS/MISMATCH 高亮
|
||
+-- 返回报告文件路径
|
||
---
|
||
|
||
## 3. DB 管道数据流
|
||
|
||
### 3.1 总体 6 步流程
|
||
|
||
COBOL 源码 (含 EXEC SQL)
|
||
|
|
||
v [Step 1: 环境整备]
|
||
step1_setup_environment()
|
||
+-- 复制源码到 ASCII 工作目录
|
||
+-- gixpp 预处理 + CONNECT TO 路径修补
|
||
+-- cobc 编译 -> exe_path
|
||
|
|
||
v [Step 2: 输入数据生成]
|
||
step2_generate_inputs(scenario)
|
||
+-- extract_structure() -> 分支树 + 赋值表
|
||
+-- generate_all_data() -> 测试数据记录
|
||
+-- _init_database() + _populate_database() -> SQLite DB 种子
|
||
+-- flatfile.write_all_files() -> 平面文件
|
||
|
|
||
v [Step 3: COBOL 执行]
|
||
step3_run_cobol(scenario)
|
||
+-- runner.run() -> 输出文件 + gcov 数据
|
||
|
|
||
v [Step 4: 中间数据提取]
|
||
step4_extract_intermediate()
|
||
+-- SELECT * FROM each table -> W01 JSON
|
||
|
|
||
v [Step 5: Java 执行] (可选)
|
||
step5_run_java()
|
||
+-- java -jar migration.jar -i W01.json -o output/
|
||
|
|
||
v [Step 6: 验证]
|
||
step6_verify()
|
||
+-- 比较 Java 输出与 COBOL 期望值
|
||
+-- 返回 VerificationRun (PASS/MISMATCH)
|
||
|
||
### 3.2 Step 1: 环境整备 (step1_setup_environment)
|
||
|
||
**职责:** gixpp 预处理 + cobc 编译
|
||
|
||
cobol_src_dir/{program_id}.cbl
|
||
|
|
||
v _copy_sources_to_workdir()
|
||
+-- 复制主源码 {program_id}.cbl -> work_dir/src/
|
||
+-- 复制 COPYBOOK (*.cpy) -> work_dir/src/
|
||
+-- 复制子程序 (SUB*.cbl) -> work_dir/src/
|
||
(搜索: cobol_src_dir, sub/, production/sub/, cobol-tna-system/sub/)
|
||
|
|
||
v runner.preprocess(src, preprocessed/, copybook_dirs)
|
||
+-- gixpp 预处理 + CONNECT TO 路径修补
|
||
| gixpp 错误转换: 'data/kin.db' -> 'sqlite://localhost/kin'
|
||
| 修补为: 'sqlite:///{db_path}'
|
||
|
|
||
v runner.compile(pp, exe, copybook_dirs, extra_srcs)
|
||
+-- cobc 编译 -> work_dir/bin/{program_id}.exe
|
||
+-- 编译日志写入 runtime_dir/logs/compile/
|
||
|
|
||
v 返回 DbPipelineResult(step=1, success, data={exe_path, log})
|
||
|
||
**输入:** cobol_src_dir, copybook_dirs, schema.subprograms
|
||
**输出:** src_path, pp_path, exe_path
|
||
|
||
### 3.3 Step 2: 输入数据生成 (step2_generate_inputs)
|
||
|
||
**职责:** COBOL 解析 -> 测试数据生成 -> DB 初始化 -> 平面文件输出
|
||
|
||
src_text + schema + scenario
|
||
|
|
||
v COBOL 解析
|
||
+-- extract_structure(src_text) -> 分支树 + 赋值表
|
||
+-- generate_all_data() -> 测试数据记录 (白盒+机能+策略)
|
||
+-- 后处理: R02APPL-ID 链接 R01APPL-ID
|
||
|
|
||
v DB 初始化
|
||
+-- 确定 DB 路径（场景分离: {program_id}_{scenario_id}.db）
|
||
+-- 清理旧 DB -> _init_database(db_path)
|
||
| +-- _create_tables(): 按 YAML schema 创建表 + 主键
|
||
+-- _populate_database(): 注入种子行
|
||
| +-- 解析 COBOL -> 分支树 -> 路径枚举
|
||
| +-- build_db_input(): 生成 DB 输入行
|
||
| | +-- collect_sql_meta(): 提取 SQL 元数据 (SELECT/INSERT/UPDATE/DELETE)
|
||
| | +-- _hostvar_root(): WHERE 宿主变量 MOVE 链解析
|
||
| | +-- _resolve_where_hostvar(): 宿主变量追溯到输入记录根字段
|
||
| | +-- 用输入键建种子（与运行时一致）
|
||
| +-- 覆盖率驱动数据补充（日期、假期等）
|
||
| +-- 区间协调（INSURANCE-RATES / EMP-MASTER）
|
||
| +-- INSERT OR IGNORE 写入 DB
|
||
+-- _inject_extra_seed_rows(): 大结果集注入
|
||
+-- _inject_sql_error_rows(): PK 冲突行注入
|
||
|
|
||
v 记录修补
|
||
+-- records[0].R01EMP-ID = SPACE（触发空社员路径）
|
||
+-- 全零 EMP-ID -> SPACE 清洗
|
||
+-- 注入重复 EMP-ID（AGG UPDATE 路径）
|
||
+-- _deduplicate_r01_pk(): PK 去重
|
||
+-- _inject_aggregation_boundaries(): 聚合边界数据
|
||
|
|
||
v 平面文件输出
|
||
+-- write_all_files(): 全 FD 平面文件
|
||
+-- write_sysin_file(): SYSIN 配置
|
||
|
|
||
v 返回 DbPipelineResult(step=2, data={records, flat_files, db_path})
|
||
|
||
**输入:** src_path, pp_path, schema, scenario
|
||
**输出:** generated_records, generated_structure, db_path, 平面文件
|
||
|
||
### 3.4 Step 3: COBOL 执行 (step3_run_cobol)
|
||
|
||
**职责:** 调用编译后的 COBOL 程序并收集 gcov 覆盖率数据
|
||
|
||
exe_path + schema + scenario
|
||
|
|
||
v 环境准备
|
||
+-- 创建 runtime/run_{id}/main/{input,output}/ 目录
|
||
+-- 复制生成的平面文件 -> input/
|
||
+-- 复制 JSON -> json/
|
||
|
|
||
v 文件方向映射 (_scan_assign_to)
|
||
+-- 正则扫描 SELECT/ASSIGN-TO -> {文件名: 方向}
|
||
+-- OPEN 语句解析 -> INPUT/OUTPUT 方向确定
|
||
|
|
||
v DB 路径准备
|
||
+-- 场景 DB -> 复制到默认 DB 路径
|
||
+-- CWD/data/kin.db（CONNECT TO 路径）
|
||
|
|
||
v 执行
|
||
+-- 清理前次 .gcda 文件
|
||
+-- runner.run(exe, cwd, db_path, env_overrides, command_args)
|
||
+-- 日志写入 runtime_dir/logs/
|
||
|
|
||
v gcov 数据收集
|
||
+-- .gcda 从 CWD + exe_dir 复制到 gcov/run_{id}/
|
||
+-- .gcno 同步（共享 .gcno，COPY 不 MOVE）
|
||
|
|
||
v 返回 DbPipelineResult(step=3, data={returncode, log, ...})
|
||
|
||
### 3.5 Step 4: 中间数据提取 (step4_extract_intermediate)
|
||
|
||
**职责:** 从 SQLite DB 导出 Java 程序所需的 JSON 中介数据
|
||
|
||
_current_db_path + schema.db_tables
|
||
|
|
||
v
|
||
+-- 打开 DB
|
||
+-- 遍历 schema.db_tables，对每张表执行 SELECT * FROM [table]
|
||
+-- 构建 meta = {program_id, tables: {table_name: [rows]}}
|
||
+-- 写入 work_dir/intermediate/{program_id}_W01.json
|
||
+-- 返回 DbPipelineResult(step=4, data={tables, w01_path})
|
||
|
||
### 3.6 Step 5: Java 执行 (step5_run_java)
|
||
|
||
**职责:** 调用 Java 转换程序处理 COBOL 输出数据
|
||
|
||
java_input_path + java_jar
|
||
|
|
||
v
|
||
+-- 创建 java_output 目录
|
||
+-- 构建命令: java -jar {java_jar} -i {java_input_path} -o {java_out}
|
||
+-- subprocess.run(cmd, capture_output=True, timeout=60)
|
||
+-- 返回 DbPipelineResult(step=5, data={returncode, log})
|
||
|
||
### 3.7 Step 6: 结果验证 (step6_verify)
|
||
|
||
**职责:** 比较 Java 输出与 COBOL 期望值
|
||
|
||
java_output_path + _current_db_path
|
||
|
|
||
v
|
||
+-- 构建 VerificationRun 结果对象
|
||
+-- 读取 DB 各表行数（调试信息）
|
||
+-- 扫描 java_output_path 下的 .txt/.json 文件
|
||
+-- 设置 exit_code 和 status（PASS/MISMATCH）
|
||
+-- 返回 VerificationRun
|
||
---
|
||
|
||
## 4. 核心数据结构
|
||
|
||
### 4.1 fields 列表格式
|
||
|
||
fields 是整个数据流的核心数据结构，贯穿输入-生成-运行全流程。
|
||
|
||
# 类型: list[dict]
|
||
# 来源: read.parse_data_division() + expand_occurs()
|
||
# 用途: 分支树构建、路径枚举、约束应用、JSON 输出
|
||
|
||
[
|
||
{
|
||
'name': str, # 字段名 (大写, 如 'R01EMP-ID')
|
||
'level': int, # 层号 (01, 05, 77, 88)
|
||
'pic': str | None, # PIC 子句原始文本
|
||
'pic_info': { # PIC 解析结果
|
||
'type': str, # 'numeric' | 'alphanumeric' | 'alphabetic' | 'unknown'
|
||
'digits': int, # 整数位数 (numeric)
|
||
'decimal': int, # 小数位数 (numeric)
|
||
'length': int, # 总长度 (alphanumeric)
|
||
'signed': bool, # 是否有符号
|
||
},
|
||
'section': str, # 'FILE' | 'WORKING-STORAGE' | 'LINKAGE'
|
||
'occurs': int, # OCCURS 次数 (0=无)
|
||
'occurs_depending': str | None, # OCCURS DEPENDING ON 目标
|
||
'redefines': str | None, # REDEFINES 目标字段名
|
||
'usage': str | None, # 'COMP' | 'COMP-3' | 'BINARY' | 'PACKED-DECIMAL' | 'DISPLAY'
|
||
'is_88': bool, # 是否 88 级条件
|
||
'parent': str | None, # 88 级父字段名
|
||
'value': str | None, # VALUE 子句值
|
||
'values': list[str] | None, # 88 级多值列表
|
||
'is_filler': bool, # 是否 FILLER
|
||
},
|
||
# ... 更多字段
|
||
]
|
||
|
||
### 4.2 Constraint 约束格式
|
||
|
||
Constraint 是路径枚举和约束应用的基本单元。
|
||
|
||
# 类型: tuple (4 元组)
|
||
# 定义: models.py 中 Constraint = tuple
|
||
# 格式: (field, operator, value, want_true)
|
||
|
||
Constraint = (
|
||
str, # field: 字段名 (如 'WRK-MONTH', 'R01EMP-ID')
|
||
str, # operator: 比较运算符 ('=' | '<>' | '>' | '<' | '>=' | '<=' | 'not_in')
|
||
str, # value: 比较值 (字符串形式, 如 '12', '00000000', 'SPACE')
|
||
bool, # want_true: True=满足条件, False=不满足条件
|
||
)
|
||
|
||
# 示例:
|
||
('WRK-MONTH', '>', '0', True) # WRK-MONTH > 0 为真
|
||
('R01EMP-ID', '<>', '00000000', True) # R01EMP-ID 不等于 00000000 为真
|
||
('SQLCODE', '=', '100', False) # SQLCODE = 100 为假 (即 SQLCODE != 100)
|
||
|
||
### 4.3 Path 路径格式
|
||
|
||
Path 是一条完整执行路径的所有约束集合。
|
||
|
||
# 类型: list[Constraint]
|
||
# 定义: models.py 中 Path = list[Constraint]
|
||
# 含义: 所有约束同时满足时，程序沿该路径执行
|
||
|
||
Path = [
|
||
('WRK-MONTH', '>', '0', True),
|
||
('WRK-MONTH', '<', '13', True),
|
||
('R01EMP-ID', '<>', '00000000', True),
|
||
('__DP', '=', 'T', True), # 决策点标记 (设计内部使用)
|
||
]
|
||
|
||
# path_infos 格式 (生成记录的输入):
|
||
path_infos = [
|
||
# (constraints, path_assignments, term_type)
|
||
(
|
||
[Constraint, ...], # 路径约束列表
|
||
{str: list[dict]}, # 赋值表 (目标 -> 赋值操作列表)
|
||
'normal' | 'abend', # 终止类型
|
||
),
|
||
# ... 更多路径
|
||
]
|
||
|
||
# 赋值表 (path_assignments) 格式:
|
||
{
|
||
'WRK-PREV-EMP-ID': [
|
||
{'type': 'move', 'source_vars': ['R01EMP-ID']}
|
||
],
|
||
'DBV-EMP-ID': [
|
||
{'type': 'move_literal', 'literal': 'A0000001'}
|
||
],
|
||
'WS-COUNT': [
|
||
{'type': 'compute', 'op': 'add', 'left': 'WS-COUNT', 'right': '1'}
|
||
],
|
||
}
|
||
|
||
### 4.4 测试数据 JSON 格式
|
||
|
||
测试数据 JSON 是非 DB 管道和 DB 管道共享的输出格式。
|
||
|
||
{
|
||
'program': str, # 程序名 (如 'KIN04CHK')
|
||
'records': [
|
||
{
|
||
'input': { # 按 FD 分组的输入字段
|
||
'FD_NAME': {
|
||
'FIELD1': str,
|
||
'FIELD2': str,
|
||
...
|
||
}
|
||
},
|
||
'expected_output': { # 按 FD 分组的期望输出字段
|
||
'FD_NAME': {
|
||
'FIELD1': str,
|
||
...
|
||
}
|
||
},
|
||
'working_storage': { # 不属于任何 FD 的工作存储字段
|
||
'WRK-MONTH': str,
|
||
'WS-COUNT': str,
|
||
...
|
||
},
|
||
'termination': str, # 'normal' | 'abend'
|
||
},
|
||
# ... 更多记录
|
||
],
|
||
'db_input': { # DB 管道专用: DB 种子数据 (可选)
|
||
'table_name': [
|
||
{'col1': val1, 'col2': val2, ...},
|
||
...
|
||
]
|
||
}
|
||
}
|
||
---
|
||
|
||
## 5. 数据流图
|
||
|
||
### 5.1 非 DB 管道完整流程 (Mermaid)
|
||
|
||
`mermaid
|
||
flowchart TD
|
||
A[COBOL 源码 .cbl] --> B[read.py preprocess]
|
||
B --> C[read.py parse_data_division]
|
||
C --> D[expand_occurs]
|
||
D --> E[fields: list dict]
|
||
|
||
E --> F[core.py build_branch_tree]
|
||
F --> G[branch_tree + assignments]
|
||
|
||
G --> H[design.py enum_paths]
|
||
H --> I[path_infos]
|
||
|
||
I --> J[design.py generate_records]
|
||
E --> J
|
||
J --> K[records: list dict]
|
||
|
||
K --> L[output.py output_json]
|
||
L --> M[测试数据 JSON]
|
||
|
||
K --> N[output.py output_input_files]
|
||
K --> O[flatfile.write_all_files]
|
||
N --> P[固定长度平面文件]
|
||
O --> P
|
||
|
||
P --> Q[cobol_runner compile]
|
||
Q --> R[cobol_runner run]
|
||
R --> S[COBOL 输出文件]
|
||
|
||
S --> T[comparator/aligner align_records]
|
||
U[Java 输出文件] --> T
|
||
T --> V[记录对 pairs]
|
||
|
||
V --> W[comparator/normalizer]
|
||
W --> X[comparator/field_compare]
|
||
X --> Y[FieldResult list]
|
||
|
||
Y --> Z[report/generator generate_html]
|
||
Z --> AA[HTML 验证报告]
|
||
`
|
||
|
||
### 5.2 DB 管道完整流程 (Mermaid)
|
||
|
||
`mermaid
|
||
flowchart TD
|
||
A[COBOL 源码 含 EXEC SQL] --> B[Step 1: 环境整备]
|
||
B --> B1[gixpp 预处理]
|
||
B1 --> B2[cobc 编译]
|
||
B2 --> B3[exe_path]
|
||
|
||
A --> C[Step 2: 输入数据生成]
|
||
B3 --> C
|
||
C --> C1[extract_structure]
|
||
C1 --> C2[generate_all_data]
|
||
C2 --> C3[records]
|
||
|
||
C3 --> C4[flatfile.write_all_files]
|
||
C4 --> C5[平面文件 input/]
|
||
|
||
C3 --> C6[build_db_input]
|
||
C6 --> C7[INSERT INTO SQLite DB]
|
||
C7 --> C8[db_path]
|
||
|
||
C5 --> D[Step 3: COBOL 执行]
|
||
C8 --> D
|
||
D --> D1[runner.run]
|
||
D1 --> D2[COBOL 输出文件]
|
||
D1 --> D3[gcov 数据]
|
||
|
||
D2 --> E[Step 4: 中间数据提取]
|
||
C8 --> E
|
||
E --> E1[SELECT * FROM tables]
|
||
E1 --> E2[W01 JSON]
|
||
|
||
E2 --> F[Step 5: Java 执行 可选]
|
||
F --> F1[java -jar migration.jar]
|
||
F1 --> F2[Java 输出文件]
|
||
|
||
F2 --> G[Step 6: 验证]
|
||
D2 --> G
|
||
G --> G1[VerificationRun]
|
||
G1 --> G2[PASS / MISMATCH]
|
||
`
|
||
|
||
### 5.3 数据流关键节点汇总
|
||
|
||
`mermaid
|
||
flowchart LR
|
||
subgraph 输入层
|
||
A1[COBOL 源码]
|
||
A2[COPYBOOK]
|
||
A3[YAML schema]
|
||
end
|
||
|
||
subgraph 解析层
|
||
B1[preprocess]
|
||
B2[parse_data_division]
|
||
B3[build_branch_tree]
|
||
end
|
||
|
||
subgraph 生成层
|
||
C1[enum_paths]
|
||
C2[generate_records]
|
||
C3[build_db_input]
|
||
end
|
||
|
||
subgraph 输出层
|
||
D1[output_json]
|
||
D2[write_all_files]
|
||
D3[write_sysin_file]
|
||
end
|
||
|
||
subgraph 执行层
|
||
E1[cobol_runner]
|
||
E2[gixsql_runner]
|
||
E3[java -jar]
|
||
end
|
||
|
||
subgraph 验证层
|
||
F1[aligner]
|
||
F2[field_compare]
|
||
F3[coverage]
|
||
F4[report]
|
||
end
|
||
|
||
A1 --> B1 --> B2 --> B3
|
||
A2 --> B1
|
||
A3 --> C3
|
||
B3 --> C1 --> C2
|
||
C2 --> D1
|
||
C2 --> D2
|
||
C3 --> D2
|
||
C3 --> D3
|
||
D2 --> E1
|
||
D2 --> E2
|
||
D3 --> E2
|
||
E1 --> F1
|
||
E2 --> F1
|
||
E3 --> F1
|
||
F1 --> F2 --> F4
|
||
B3 --> F3 --> F4
|
||
`
|
||
|
||
---
|
||
|
||
## 6. 跨模块数据流转
|
||
|
||
### 6.1 模块间数据传递关系
|
||
|
||
| 源模块 | 目标模块 | 传递数据 | 数据格式 |
|
||
|--------|----------|----------|----------|
|
||
| read.py | core.py | preprocessed, data_fields | str, list[dict] |
|
||
| read.py | design.py | data_fields, open_dir, file_sec | list[dict], dict, dict |
|
||
| core.py | design.py | branch_tree, assignments | BrSeq, dict |
|
||
| cond.py | design.py | path constraints | list[Constraint] |
|
||
| design.py | output.py | records, kept_path_cons, term_types | list[dict], list, list[str] |
|
||
| design.py | to_sql.py | records, data_fields, assignments | list[dict], list[dict], dict |
|
||
| output.py | runner.py | records (JSON/binary) | dict, bytes |
|
||
| runner.py | comparator/ | output files | file paths |
|
||
| to_sql.py | orchestrator_db.py | db_input (DB seed rows) | dict[str, list[dict]] |
|
||
| flatfile.py | orchestrator_db.py | flat files (binary) | file paths |
|
||
| orchestrator_db.py | gixsql_runner.py | exe_path, db_path, env | Path, Path, dict |
|
||
| gixsql_runner.py | orchestrator_db.py | RunResult | dataclass |
|
||
| comparator/aligner.py | comparator/field_compare.py | aligned pairs | list[tuple] |
|
||
| comparator/field_compare.py | report/generator.py | FieldResults | list[FieldResult] |
|
||
| coverage.py | report/generator.py | DecisionPoints, coverage rates | list[DecisionPoint], float |
|
||
| data_merger.py | __init__.py | merged records | list[dict] |
|
||
| hina/strategy.py | data_merger.py | strategy records | list[dict] |
|
||
|
||
### 6.2 核心数据流路径
|
||
|
||
**路径 1: 非 DB 管道 (flat file)**
|
||
|
||
read.py (fields) -> core.py (branch_tree) -> design.py (records)
|
||
-> output.py (JSON + flat files) -> cobol_runner (output files)
|
||
-> comparator/ (verification results) -> report/ (HTML)
|
||
|
||
**路径 2: DB 管道 (SQL/SQLite)**
|
||
|
||
read.py (fields) -> core.py (branch_tree) -> design.py (records)
|
||
-> to_sql.py (DB seed rows) -> flatfile.py (flat files)
|
||
-> gixsql_runner (output files + DB state)
|
||
-> step4 (W01 JSON) -> step5 (Java output) -> step6 (verification)
|
||
|
||
**路径 3: 覆盖率收集**
|
||
|
||
core.py (branch_tree) -> coverage.py (decision_points)
|
||
-> mark_coverage (covered branches) -> gcov.py (dynamic coverage)
|
||
-> coverage.py (merged coverage) -> report/ (HTML report)
|
||
|
||
### 6.3 关键中间产物
|
||
|
||
| 产物 | 生成位置 | 消费位置 | 格式 |
|
||
|------|----------|----------|------|
|
||
| preprocessed | read.py | core.py, read.py | str (预处理后源码) |
|
||
| data_fields | read.py | core.py, design.py, to_sql.py | list[dict] |
|
||
| branch_tree | core.py | design.py, coverage.py | BrSeq |
|
||
| assignments | core.py | design.py, to_sql.py | dict |
|
||
| path_infos | design.py | design.py (generate_records) | list[tuple] |
|
||
| records | design.py | output.py, to_sql.py, flatfile.py | list[dict] |
|
||
| fd_fields | read.py | output.py | dict[str, set[str]] |
|
||
| open_dir | read.py | output.py, flatfile.py | dict[str, str] |
|
||
| file_sec | read.py | output.py, __init__.py | dict[str, list[str]] |
|
||
| db_input | to_sql.py | orchestrator_db.py | dict[str, list[dict]] |
|
||
| flat files | flatfile.py | runner.py | file paths (binary) |
|
||
| JSON | output.py | runner.py, coverage.py | file path |
|
||
| gcov data | gcov.py | coverage.py | dict[int, int] |
|
||
| DecisionPoints | coverage.py | report/generator.py | list[DecisionPoint] |
|
||
| VerificationRun | comparator/ | report/generator.py | dataclass |
|
||
| FieldResult | field_compare.py | report/generator.py | dataclass |
|