"""GixsqlOrchestrator — DB COBOL プログラムの全6Step実行""" from __future__ import annotations import json import logging import os import subprocess import sqlite3 from dataclasses import dataclass, field from pathlib import Path from typing import Optional from config import Config from config.program_schema import ProgramSchema, load_schema from cobol_testgen import extract_structure, generate_data from cobol_testgen.flatfile import write_all_files, write_sysin_file from cobol_testgen.file_io import read_output_file from cobol_testgen.read import preprocess, resolve_copybooks, resolve_sql_includes, parse_file_control, parse_file_section, parse_data_division, extract_data_division, scan_open_statements from cobol_testgen.read import strip_exec_sql_from_data_div from cobol_testgen.gcov import run_gcov from cobol_testgen.coverage import run_coverage, generate_coverage_index from cobol_testgen.design_mcdc import enum_paths as mcdc_enum_paths from cobol_testgen.to_sql import collect_sql_meta, build_db_input from cobol_testgen.core import extract_sql_assignments from cobol_testgen import expand_occurs import shutil from data.diff_result import VerificationRun, FieldResult from runners.gixsql_runner import GixsqlCobolRunner, GixsqlTableData logger = logging.getLogger(__name__) @dataclass class DbPipelineResult: """DB 管线単体実行結果""" program_id: str step: int | float # pipeline step number success: bool message: str = "" data: dict = field(default_factory=dict) class GixsqlOrchestrator: """6Step DB 管线オーケストレーター""" def __init__(self, config: Config, program_id: str, cobol_src_dir: str | Path, copybook_dirs: list[str | Path] | None = None, work_dir: str | Path | None = None, skip_jvm: bool = True): self.config = config self.program_id = program_id self.cobol_src_dir = Path(cobol_src_dir) self.copybook_dirs = copybook_dirs or [] self.skip_jvm = skip_jvm v3_root = Path(__file__).parent # cobol-java-v3/ # Build artifacts in temp (ASCII-only, gixpp can't handle Chinese paths) if work_dir is None: temp = Path(os.environ.get("TEMP", "C:\\Temp")) work_dir = temp / "gixsql_build" / program_id self.work_dir = Path(work_dir) # Runtime data under V3 (DB, flat files, CWD) self.runtime_dir = v3_root / "runtime" / program_id self.schema: ProgramSchema = load_schema(program_id) self.runner = GixsqlCobolRunner( gixpp_path=config.gixsql_path, lib_path=config.gixsql_lib_path, compile_flags=config.gixsql_compile_flags, ) # Derive DB path: C:\Temp\gix\.db (matches COBOL CONNECT TO, short enough for col 72) self.db_path = Path("C:/Temp/gix") / f"{self.program_id}.db" # Pipeline state self.src_path: Optional[Path] = None self.pp_path: Optional[Path] = None self.exe_path: Optional[Path] = None self.java_input_path: Optional[Path] = None self.java_output_path: Optional[Path] = None self.generated_records: list[dict] = [] self.generated_structure: dict | None = None # ── Step 1: 環境整備（gixpp + compile） ── def _copy_sources_to_workdir(self) -> tuple[Path, list[str]]: """Copy source + copybooks to ASCII-only workdir (gixpp can't handle Chinese paths).""" src_dir = self.work_dir / "src" src_dir.mkdir(parents=True, exist_ok=True) # Copy main source orig = self.cobol_src_dir / f"{self.program_id}.cbl" ascii_src = src_dir / f"{self.program_id}.cbl" if not ascii_src.exists(): ascii_src.write_bytes(orig.read_bytes()) self.src_path = ascii_src # Copy copybooks flat_cpy = [] for d in self.copybook_dirs: pd = Path(d) if pd.exists(): for f in pd.glob("*.cpy"): dst = src_dir / f.name if not dst.exists(): dst.write_bytes(f.read_bytes()) flat_cpy.append(str(dst)) # Copy SUB programs sub_dirs = [self.cobol_src_dir, self.cobol_src_dir.parent / "sub"] for sub in self.schema.subprograms: found = False for sd in sub_dirs: sp = sd / f"{sub}.cbl" if sp.exists(): dst = src_dir / f"{sub}.cbl" if not dst.exists(): dst.write_bytes(sp.read_bytes()) found = True break if not found: logger.warning(f" SUB {sub}.cbl not found in {sub_dirs}") return src_dir, flat_cpy def step1_setup_environment(self) -> DbPipelineResult: """gixpp 前処理 → cobc コンパイル""" try: ascii_dir, flat_cpy = self._copy_sources_to_workdir() src = ascii_dir / f"{self.program_id}.cbl" pp = self.runner.preprocess(src, self.work_dir / "preprocessed", copybook_dirs=[ascii_dir]) self.pp_path = Path(pp) exe = self.work_dir / "bin" / f"{self.program_id}.exe" extra_srcs = [] for sub in self.schema.subprograms: sp = ascii_dir / f"{sub}.cbl" if sp.exists(): extra_srcs.append(sp) result = self.runner.compile( pp, exe, copybook_dirs=[ascii_dir], extra_srcs=extra_srcs, ) if result.success: self.exe_path = Path(result.exe_path) return DbPipelineResult( self.program_id, 1, result.success, message=result.log[:200], data={"exe_path": str(exe), "log": result.log[:500]}, ) except Exception as e: return DbPipelineResult(self.program_id, 1, False, str(e)) # ── Step 2: 入力データ生成 ── def step2_generate_inputs(self) -> DbPipelineResult: """テストデータ生成 + フラットファイル出力 + DB初期化""" try: src_text = self.src_path.read_text(encoding="utf-8-sig") # Use the pre-gixpp source for Lark parsing (gixpp output contains SQLCA etc.) parse_text = self.pp_path.read_text(encoding="utf-8") if self.pp_path and self.pp_path.exists() else src_text # COBOL 解析 + テストデータ生成 cbd = [str(d) for d in self.copybook_dirs] st = extract_structure(src_text, copybook_dirs=cbd) self.generated_structure = st recs = generate_data(src_text, st, copybook_dirs=cbd) # Post-process: link R02 cancel APPL-IDs to matching R01 insert APPL-IDs for rec in recs: if 'R02APPL-ID' in rec and 'R01APPL-ID' in rec: rec['R02APPL-ID'] = rec['R01APPL-ID'] # DB 初期データ構築: single DB under V3 runtime/ dir self.db_path.parent.mkdir(parents=True, exist_ok=True) self._init_database(self.db_path) # DB 初期行投入（DELETE/UPDATE が作用する行、SELECT が返す行） self._populate_database(self.db_path, src_text, recs) # フラットファイル書き出し input_dir = self.work_dir / "input" input_dir.mkdir(parents=True, exist_ok=True) flats = write_all_files(recs, src_text, input_dir, copybook_dirs=[str(d) for d in self.copybook_dirs]) # SYSIN 設定ファイル生成（プログラム固有のカード形式） sysin_path = write_sysin_file(recs, src_text, input_dir, copybook_dirs=[str(d) for d in self.copybook_dirs]) if sysin_path: logger.info(f" SYSIN file written: {sysin_path}") flats.append(("SYSIN", sysin_path, 0)) self.generated_records = recs return DbPipelineResult( self.program_id, 2, True, data={"records": len(recs), "flat_files": len(flats), "db_path": str(self.db_path)}, ) except Exception as e: return DbPipelineResult(self.program_id, 2, False, str(e)) # ── Step 3: COBOL 実行 ── def step3_run_cobol(self) -> DbPipelineResult: """COBOL DB プログラム実行""" if not self.exe_path or not self.exe_path.exists(): return DbPipelineResult(self.program_id, 3, False, "exe not found (run step1 first)") # Subprogram DLLs are in cobol-tna-system/bin/ cobol_bin = Path(self.cobol_src_dir).parent / "bin" self.runtime_dir.mkdir(parents=True, exist_ok=True) result = self.runner.run( self.exe_path, self.runtime_dir, self.db_path, input_dir=self.work_dir / "input", cobol_lib_path=str(cobol_bin) if cobol_bin.exists() else None, ) return DbPipelineResult( self.program_id, 3, result.success, data={"returncode": result.returncode, "log": result.log[:500]}, ) # ── カバレッジレポート（パイプライン外、オプション） ── def generate_coverage_report(self, output_dir: str | Path | None = None) -> DbPipelineResult: """COBOL 実行後：gcov データ収集 + 静的パスとマージし HTML レポート""" try: if not self.exe_path or not self.exe_path.exists(): return DbPipelineResult(self.program_id, 0, False, "exe not found (run step3 first)") if output_dir is None: v3_root = Path(__file__).parent output_dir = v3_root / "reports" / self.program_id / "coverage" output_dir = Path(output_dir) # 1. Copy .gcno + .gcda from CWD (compile-time cwd) to runtime_dir # cobc generates .gcno in CWD; at runtime, program writes .gcda to same CWD gcno_gcda_count = 0 for ext in (".gcno", ".gcda"): for f in Path.cwd().glob(f"*{ext}"): if f.stat().st_size > 0: shutil.copy2(str(f), str(self.runtime_dir / f.name)) gcno_gcda_count += 1 if gcno_gcda_count == 0: return DbPipelineResult(self.program_id, 0, False, f"no .gcno/.gcda found in CWD (--coverage missing?)") # 3. Parse gcov data gcov_data = run_gcov(f"{self.program_id}_pp", str(self.runtime_dir)) if not gcov_data: gcov_data = run_gcov(self.program_id, str(self.runtime_dir)) for sub in self.schema.subprograms: sd = run_gcov(sub, str(self.runtime_dir)) if sd: gcov_data.update(sd) # 4. Static branch tree from step2 st = self.generated_structure branch_tree = st.get("branch_tree_obj") if st else None if not branch_tree: return DbPipelineResult(self.program_id, 0, True, data={"gcov_lines": len(gcov_data), "note": "no branch tree — gcov data only"}) # 5. Re-parse fields (same as generate_data) src_text = self.src_path.read_text(encoding="utf-8-sig") cbd = [str(d) for d in self.copybook_dirs] pp = preprocess(src_text, extra_search_paths=cbd) data_div = extract_data_division(pp) data_fields = parse_data_division(data_div) if data_div else [] fdict = [] for idx, f in enumerate(data_fields): entry = { 'name': f.name, 'level': f.level, 'pic': f.pic, 'pic_info': { 'type': f.pic_info.type if f.pic_info else 'unknown', 'digits': f.pic_info.digits if f.pic_info else 0, 'decimal': f.pic_info.decimal if f.pic_info else 0, 'length': f.pic_info.length if f.pic_info else 0, 'signed': f.pic_info.signed if f.pic_info else False, }, 'section': f.section, 'occurs': f.occurs_count, 'occurs_depending': f.occurs_depending, 'redefines': f.redefines, 'usage': f.usage, } if f.is_88: entry['is_88'] = True entry['parent'] = f.parent fdict.append(entry) fdict = expand_occurs(fdict) # 6. Enumerate paths branch_paths = mcdc_enum_paths(branch_tree, fdict) # 7. Read preprocessed source for gcov line number matching gcov_source = None if self.pp_path and self.pp_path.exists(): gcov_source = self.pp_path.read_text(encoding="utf-8") # 8. Generate merged HTML (use gcov_source for line numbers) output_dir.mkdir(parents=True, exist_ok=True) prefix = str(output_dir / self.program_id) cov_result = run_coverage( branch_tree, branch_paths, fdict, src_text, prefix, index_relpath="index.html", gcov_data=gcov_data or None, gcov_source=gcov_source, ) generate_coverage_index([cov_result], str(output_dir.parent)) # Clean up .gcno/.gcda from CWD (avoid accumulation) for ext in (".gcno", ".gcda"): for f in Path.cwd().glob(f"*{ext}"): try: f.unlink() except PermissionError: pass total = cov_result.get("total_branches", 0) covered = cov_result.get("covered_branches", 0) pct = covered / total * 100 if total else 0 self._last_coverage_dict = cov_result return DbPipelineResult( self.program_id, 0, True, data={ "gcov_lines": len(gcov_data), "coverage": f"{covered}/{total} ({pct:.1f}%)", "reports": str(output_dir), "_cov_dict": cov_result, }, ) except Exception as e: logger.exception("generate_coverage_report failed") return DbPipelineResult(self.program_id, 0, False, str(e)) # ── Step 4: DB → Java 中介データ ── def step4_extract_intermediate(self) -> DbPipelineResult: """SQLite → JSON 中介データ抽出（Step 4: DB→Java中介データ）""" if not self.db_path or not self.db_path.exists(): return DbPipelineResult(self.program_id, 4, False, "db not found (run step3 first)") try: conn = sqlite3.connect(str(self.db_path)) conn.row_factory = sqlite3.Row # Read from actual COBOL SQL tables (using sql_name or name) output_tables = {} for table in self.schema.db_tables: sql_name = table.sql_name or table.name try: rows = conn.execute(f"SELECT * FROM [{sql_name}]").fetchall() output_tables[table.name] = [dict(r) for r in rows] except sqlite3.OperationalError: output_tables[table.name] = [] conn.close() w01_path = self.work_dir / "intermediate" / f"{self.program_id}_W01.json" w01_path.parent.mkdir(parents=True, exist_ok=True) meta = { "program_id": self.program_id, "tables": output_tables, } w01_path.write_text(json.dumps(meta, ensure_ascii=False, indent=2)) self.java_input_path = w01_path return DbPipelineResult( self.program_id, 4, True, data={"tables": len(output_tables), "w01_path": str(w01_path)}, ) except Exception as e: return DbPipelineResult(self.program_id, 4, False, str(e)) # ── Step 5: Java 実行 ── def step5_run_java(self, java_cmd: str = "java", java_jar: str | Path | None = None) -> DbPipelineResult: """Java プログラム実行""" if not self.java_input_path or not self.java_input_path.exists(): return DbPipelineResult(self.program_id, 5, False, "intermediate data not found (run step4 first)") java_out = self.work_dir / "java_output" java_out.mkdir(parents=True, exist_ok=True) if java_jar: cmd = [java_cmd, "-jar", str(java_jar), "-i", str(self.java_input_path), "-o", str(java_out)] else: cmd = [java_cmd, "-version"] try: r = subprocess.run(cmd, capture_output=True, timeout=60) log = (r.stdout.decode("utf-8", "replace") + "\n" + r.stderr.decode("utf-8", "replace")) ok = r.returncode == 0 self.java_output_path = java_out return DbPipelineResult( self.program_id, 5, ok, data={"returncode": r.returncode, "log": log[:500]}, ) except subprocess.TimeoutExpired: return DbPipelineResult(self.program_id, 5, False, "Java timeout") # ── Step 6: 検証 ── def step6_verify(self) -> VerificationRun: """Java 出力と COBOL 期待値を比較""" vr = VerificationRun( program=self.program_id, runner="gixsql", gixsql_version="0.9.1", sqlite_path=str(self.db_path) if self.db_path else "", step_reached=6, ) if self.db_path and self.db_path.exists(): after_tables = self.runner.read_db_tables( self.db_path, [t.name for t in self.schema.db_tables], ) for table_data in after_tables: vr.debug[f"table_{table_data.table_name}_rows"] = len(table_data.rows) if self.java_output_path and self.java_output_path.exists(): java_files = list(self.java_output_path.glob("*.txt")) + \ list(self.java_output_path.glob("*.json")) vr.debug["java_output_files"] = [str(f) for f in java_files] vr.fields_matched = len(java_files) vr.exit_code = 0 if vr.fields_mismatched == 0 else 1 vr.status = "PASS" if vr.exit_code == 0 else "MISMATCH" return vr # ── 全Step一括実行 ── def run_all(self, skip_steps: set[int] | None = None, generate_coverage: bool = True) -> VerificationRun: """Step 1 → 6 を順次実行（skip_jvm=True で Step 5/6 をスキップ）""" skip = set(skip_steps or []) if self.skip_jvm: skip.update({5, 6}) steps = [ (1, self.step1_setup_environment), (2, self.step2_generate_inputs), (3, self.step3_run_cobol), (4, self.step4_extract_intermediate), ] if not self.skip_jvm: steps.extend([ (5, self.step5_run_java), (6, self.step6_verify), ]) results = [] last_step = max(s for s, _ in steps) for step_num, step_fn in steps: if step_num in skip: continue logger.info(f" Step {step_num}...") result = step_fn() results.append(result) if not result.success and step_num < last_step: vr = VerificationRun( program=self.program_id, runner="gixsql", status="BLOCKED", exit_code=2, step_reached=step_num, debug={"step_results": [r.__dict__ for r in results]}, ) return vr # Optional coverage report (non-blocking, not part of numbered pipeline) cv_flags = getattr(self.config, 'gixsql_compile_flags', '') if '--coverage' in cv_flags and generate_coverage: self.generate_coverage_report() if not self.skip_jvm: vr = results[-1] # step6_verify returned VerificationRun vr.debug["step_results"] = [r.__dict__ for r in results[:-1] if r] else: vr = VerificationRun( program=self.program_id, runner="gixsql", status="PASS", exit_code=0, step_reached=last_step, debug={"step_results": [r.__dict__ for r in results]}, ) return vr # ── Internal helpers ── def _init_database(self, db_path: Path): """Create tables from schema + COBOL EXEC SQL table definitions.""" self._create_tables(db_path) def _create_tables(self, db_path: Path): conn = sqlite3.connect(str(db_path)) for table in self.schema.db_tables: col_defs = [] pk_cols = [] for col in table.columns: col_defs.append(f"[{col.name}] {col.type}") if col.primary_key: pk_cols.append(f"[{col.name}]") if pk_cols: col_defs.append(f"PRIMARY KEY ({', '.join(pk_cols)})") ddl = f"CREATE TABLE IF NOT EXISTS [{table.name}] (\n " + \ ",\n ".join(col_defs) + "\n)" conn.execute(ddl) # If sql_name differs, also create the COBOL-visible SQL table name if table.sql_name and table.sql_name != table.name: conn.execute(ddl.replace(f"[{table.name}]", f"[{table.sql_name}]")) conn.commit() conn.close() logger.info(f" DB initialized: {db_path}") def _populate_database(self, db_path: Path, src_text: str, records: list[dict]): """テストデータから DB 初期行を生成し挿入する。""" from cobol_testgen.pipeline_bridge import build_branch_tree_fallback from cobol_testgen.read import extract_procedure_division cbd = [str(d) for d in self.copybook_dirs] src_resolved = resolve_copybooks(src_text, ".", extra_search_paths=cbd) src_resolved = resolve_sql_includes(src_resolved, ".") preprocessed = preprocess(src_resolved) data_div = extract_data_division(preprocessed) data_fields = parse_data_division(data_div) if data_div else [] fields_dict = [] for f in data_fields: fields_dict.append({ 'name': f.name, 'level': f.level, 'pic': f.pic, 'pic_info': { 'type': f.pic_info.type if f.pic_info else 'unknown', 'digits': f.pic_info.digits if f.pic_info else 0, 'decimal': f.pic_info.decimal if f.pic_info else 0, 'length': f.pic_info.length if f.pic_info else 0, 'signed': f.pic_info.signed if f.pic_info else False, }, 'section': f.section, 'occurs': f.occurs_count, 'occurs_depending': f.occurs_depending, 'value': f.value, 'values': f.values, 'redefines': f.redefines, 'usage': f.usage, }) fields_dict = expand_occurs(fields_dict) proc_div = extract_procedure_division(preprocessed) branch_tree, assignments = build_branch_tree_fallback(proc_div, fields_dict) # Merge SQL assignments from original source sql_assigns = extract_sql_assignments(src_text) for tgt, asgn_list in sql_assigns.items(): for asgn in asgn_list: assignments.setdefault(tgt, []).append(asgn) branch_paths = mcdc_enum_paths(branch_tree, fields_dict) data_div2, declared_columns = strip_exec_sql_from_data_div(data_div) sql_meta = collect_sql_meta(assignments, declared_columns) if not sql_meta: logger.info(" No SQL metadata found, skipping DB population") return db_input = build_db_input( branch_paths, fields_dict, assignments, sql_meta, declared_columns, records=records, ) if not db_input: logger.info(" No DB input rows generated") return conn = sqlite3.connect(str(db_path)) for table_name, rows in db_input.items(): if not rows: logger.info(f" Table {table_name}: 0 initial rows (will be created at runtime)") continue col_names = list(rows[0].keys()) placeholders = ", ".join("?" for _ in col_names) quoted_cols = ", ".join(f"[{c}]" for c in col_names) sql = f"INSERT OR IGNORE INTO [{table_name}] ({quoted_cols}) VALUES ({placeholders})" conn.executemany(sql, [tuple(r.get(c, "") for c in col_names) for r in rows]) logger.info(f" Table {table_name}: {len(rows)} initial rows inserted") conn.commit() conn.close() logger.info(f" DB populated: {db_path}")