feat(writer): 输出语言一致性保障 + 用户可选输出语言

- 步骤0: 新增中文镜像模板 scripts/make_zh_template.py 与 samples/概要设计书模板_中文.docx(7章锚点原样保留)
- 步骤1: config.WriterConfig.output_language→Settings.writer;GenerationContext.output_language + to_vars.language_instruction(zh/ja/auto);writer_agent【语言约束】改引变量;context_builder/orchestrator/run_trial 透传 --output-language
- 步骤A: 新建 src/genesis/writer/language.py(detect_script/resolve_expected_language/find_language_violations);WriterAgent.generate_chapter 按期望语言强制、违规重试、耗尽硬失败;max_retries 默认 1→2
- 步骤B: _format_impact 影响调查标签按 output_language 本地化(zh 新建/变更/删除/警告)
- 步骤C: eval scorer 第 11 维度 language_consistency(不可验证=满分,不拉低总分);ChapterArtifact.expected_language;QAValidator.validate_doc 透传;QALoop.run 透传 output_language
- 测试: test_zh_template/test_language_plumbing/test_writer_language/test_scorer_language/test_language_coverage,并更新 test_phase5_e2e
- 全量 pytest 424 passed / 99.15%(覆盖率门槛 99% 达标)
This commit is contained in:
lhl
2026-08-25 23:30:24 +08:00
parent 9d0d3409c2
commit d01e1b720f
20 changed files with 1049 additions and 31 deletions
+1
View File
@@ -115,3 +115,4 @@
| 2026-08-24 13:18 | Agent 实现 | 修复无子节章 LLM 自造标题:prompt【小节约束】禁止空结构时输出 heading 块(尽力约束)+ WriterAgent.generate_chapter 程序化强制(sub_headings 为空的章剔除生成结果中的 heading 块,模板结构为准)。真实试运行验证:自造同名 H2 清零、H2 总数 14→6(仅模板定义的子节)、无子节章正文全为段落 | src/genesis/writer/writer_agent.py, tests/test_phase5_writer_agent.py | x-preview-f-free (opencode) | | 2026-08-24 13:18 | Agent 实现 | 修复无子节章 LLM 自造标题:prompt【小节约束】禁止空结构时输出 heading 块(尽力约束)+ WriterAgent.generate_chapter 程序化强制(sub_headings 为空的章剔除生成结果中的 heading 块,模板结构为准)。真实试运行验证:自造同名 H2 清零、H2 总数 14→6(仅模板定义的子节)、无子节章正文全为段落 | src/genesis/writer/writer_agent.py, tests/test_phase5_writer_agent.py | x-preview-f-free (opencode) |
| 2026-08-24 14:02 | 架构设计 | 输出语言一致性保障方案落盘:用户可选输出语言(auto/zh/ja)+ 中文模板镜像 + 确定性脚本检测与重试强制(A)+ impact 标签本地化(B+ QA 第11维度 language_consistencyC);关键决策:表格照抄原文、双模板锚点不变、违规硬失败 | docs/superpowers/plans/2026-08-24-language-consistency.md | x-preview-f-free (opencode) | | 2026-08-24 14:02 | 架构设计 | 输出语言一致性保障方案落盘:用户可选输出语言(auto/zh/ja)+ 中文模板镜像 + 确定性脚本检测与重试强制(A)+ impact 标签本地化(B+ QA 第11维度 language_consistencyC);关键决策:表格照抄原文、双模板锚点不变、违规硬失败 | docs/superpowers/plans/2026-08-24-language-consistency.md | x-preview-f-free (opencode) |
| 2026-08-24 14:35 | 反馈迭代 | 修复 _AI_USAGE_LOG.md 表格渲染断裂(用户报告 2026-08-09 03:30 起不以表格显示):删除表内 4 处空行与 1 处误写入的工具输出残留行(End of file);L55 摘要内嵌错误码表的 6 个管道符转义为反斜杠管;L107 被拼接的两条 2026-08-23 记录拆分为独立两行;零日志行删除,修复后全表 116 行均为有效 5 列 | _AI_USAGE_LOG.md | x-preview-f-free (opencode) | | 2026-08-24 14:35 | 反馈迭代 | 修复 _AI_USAGE_LOG.md 表格渲染断裂(用户报告 2026-08-09 03:30 起不以表格显示):删除表内 4 处空行与 1 处误写入的工具输出残留行(End of file);L55 摘要内嵌错误码表的 6 个管道符转义为反斜杠管;L107 被拼接的两条 2026-08-23 记录拆分为独立两行;零日志行删除,修复后全表 116 行均为有效 5 列 | _AI_USAGE_LOG.md | x-preview-f-free (opencode) |
| 2026-08-25 10:20 | Agent 实现 | 输出语言一致性 + 可选输出语言(实施中,暂停于步骤 A 修正点):步骤0 新增 scripts/make_zh_template.py 与 samples/概要设计书模板_中文.docx7章锚点镜像)+ tests/test_zh_template.py;步骤1 config.WriterConfig.output_language→Settings.writer、GenerationContext.output_language+to_vars.language_instruction、writer_agent【语言约束】引变量、context_builder/orchestrator/run_trial 透传 --output-language + tests/test_language_plumbing.py;步骤A 新建 src/genesis/writer/language.pydetect_script/resolve_expected_language/find_language_violations)并接入 WriterAgent.generate_chapter 违规重试、max_retries 默认1→2 + tests/test_writer_language.py(待修正 resolve_expected_language 标题仅假名可信) | src/genesis/config.py; src/genesis/writer/models.py; src/genesis/writer/writer_agent.py; src/genesis/writer/context_builder.py; src/genesis/writer/orchestrator.py; src/genesis/writer/language.py; scripts/make_zh_template.py; scripts/run_trial.py; tests/test_zh_template.py; tests/test_language_plumbing.py; tests/test_writer_language.py; samples/概要设计书模板_中文.docx | x-preview-f-free (opencode) |
@@ -0,0 +1,155 @@
# 输出语言一致性保障 + 用户可选输出语言(2026-08-24
## 背景
真实 LLM 试运行暴露:生成正文曾出现中日混杂(早期第 1-2 章中文、第 3-7 章日文;prompt 渲染缺陷
修复后全日文,但属概率性改善)。盘点确认:
1. **唯一保障是 prompt 层【语言约束】**,无任何程序化检测(grep 全代码库证实)
2. **输入侧存在中文诱导源**`_format_impact()` 固定标签(`受影响:``[警告]`)进入每章 prompt
3. design.md §7.2 QA 十项清单**没有语言一致性校验项**
4. 日文章标题多为纯汉字(「5. DB設計」无假名),仅凭标题无法判定期望语言
用户需求升级:**让用户可选择输出中文还是日文**(参数化),并配套中文模板。
## 关键用户决策(已确认)
| # | 决策点 | 结论 |
|---|--------|------|
| 1 | Excel 表格数据是否翻译 | **照抄原文**——数据是源数据忠实呈现,翻译违反 §7.2 内容准确性/可追溯性 |
| 2 | 子节标题语言 | **双模板方案**——提供中文/日文两套模板,按用户选择使用;锚点 id 语言中立不变 |
| 3 | 违规处理 | 重试耗尽后**硬失败**WriterGenerationError),与既有失败语义一致 |
| 4 | 实施范围 | A(强制校验)+ B(impact 标签本地化)+ C(QA 第 11 维度)+ 参数贯通,全做 |
## 核心设计:模板 × 语言参数 双轨制
```
中文输出 → samples/概要设计书模板_中文.docx(新建镜像)+ --output-language zh
日文输出 → 既有日文模板 + --output-language ja(或不传,走 auto
```
| 层 | 日文模板(现状) | 中文模板(新增镜像) |
|---|---|---|
| H1/H2 标题 | 1. はじめに / 2.1 機能一覧表 | 1. 前言 / 2.1 功能一览表 |
| 锚点 id | `section:function_list` … | **完全相同(不变)** → 映射/注入器零改动 |
| 封面 | 概要設計書 / バージョン | 概要设计书 / 版本 |
## 实施清单(TDD,每项先 RED 后 GREEN
### 0. 中文样本模板
- 新建 `scripts/make_zh_template.py`python-docx 镜像日文模板结构
Title/H1×7/H2×6/锚点原样/封面字段),产出 `samples/概要设计书模板_中文.docx` 入库
- 测试:解析 zh 模板断言 7 章 anchor id 与 ja 模板逐一对应
### 1. 语言参数贯通
- `config.py`:新增 `WriterConfig(output_language: "auto"|"zh"|"ja" = "auto")``Settings.writer`
- `models.py``GenerationContext.output_language` 字段;`to_vars()` 输出动态
`{{language_instruction}}`(zh=必须使用中文撰写 / ja=必須日本語で記述 / auto=与标题一致)
- `writer_agent.py`:【语言约束】段落改引该变量
- `context_builder.py` / `orchestrator.py` / `run_trial.py``--output-language {auto,zh,ja}` 逐层透传
### A. 确定性脚本检测 + 失败重试【核心】
- 新建 `src/genesis/writer/language.py`
- `detect_script(text)`:含假名(U+304030FF)→ "ja"CJK 汉字零假名 → "zh";否则 None
- `resolve_expected_language(explicit, title, fallback_texts)`:显式 > 标题假名 > 规则文档主导脚本;
单一事实来源(A 的重试校验与 C 的 QA 维度共用)
- `violations(...)`:仅检 paragraph/note/list 正文块;**表格 rows 不检(照抄原文)、heading 不检(跟随模板)**;
违规判定 = 含 CJK 汉字零假名且长度 ≥12(防误杀短术语)
- `writer_agent.py``generate_chapter` 重试循环内后置校验,违规按生成失败重试;
`max_retries` 默认 1→2(现默认无重试机会,已确认既有测试均显式传参)
### B. `_format_impact()` 标签本地化
-`output_language` 选标签集(zh=受影响 / ja=影響;auto 默认 ja);只影响 prompt 变量,
不影响 impact-report.json 独立产物
### C. QA 第 11 维度 `language_consistency`
- `ChapterArtifact.expected_language: str = ""`(上层解析后传入,scorer 不自行推导)
- `qa/validator.py` 从 orchestrator 透传期望语言;歧义记 unverifiable 计入 detail
- `DEFAULT_THRESHOLDS["language_consistency"] = 1.0`
- ⚠️ 已识别风险:新维度改变 score() 总分均值,`test_eval_scorer.py` / `test_phase5_scorer.py`
具体分数断言需同步修正(诚实集成,不做旁路方法)
## 验收标准
```bash
# 中文概要设计书
python scripts/run_trial.py --template samples/概要设计书模板_中文.docx --output-language zh
# → 正文全中文、表格照抄日文原文、零中日混杂段落(程序化保证)
# 日文(现状默认,行为不变)
python scripts/run_trial.py
```
1. 全量 pytest ≥99% 覆盖率(当前基线 389 passed / 99.04%
2. 真实 LLM 双跑(zh/ja 各一次)+ 扫描脚本确认 output.docx 无违规件
3. 提交 commit + 追加 `_AI_USAGE_LOG.md`
## 涉及文件
- 新增:`scripts/make_zh_template.py``samples/概要设计书模板_中文.docx`
`src/genesis/writer/language.py``tests/test_writer_language.py`
- 修改:`config.py``writer/models.py``writer/writer_agent.py``writer/context_builder.py`
`writer/orchestrator.py``eval/scorer.py``qa/validator.py``scripts/run_trial.py`
- 测试更新:`tests/test_phase5_writer_agent.py``tests/test_phase5_models.py`
`tests/test_eval_scorer.py``tests/test_phase5_scorer.py``tests/test_run_trial.py`
## 状态
- [x] 计划已获用户批准(等待二次确认开工)
- [x] 计划评审(2026-08-24,对照真实代码逐条核验,见下节)
- [x] 步骤 0-3 实施
- [ ] 验收
## 验收记录(2026-08-25
- pytest 全量:**424 passed / 99.15%**(覆盖率门槛 99% 达标)
- Fake 模式双模板试运行:zh 模板 + `--output-language zh` → 7 章 docxja 模板默认 → 7 章 docx,均无崩溃
- 程序化扫描 `output/zh_output.docx`:25 段落,**0 日文假名混入**(语言一致性成立)
- 约束链路单测覆盖:writer 重试/硬失败、QA 第 11 维度、validator 透传、qa_loop 透传
- ⚠️ **未执行项**:真实 LLM 双语双跑需 `GENESIS_INFERENCE__API_KEY`(本环境无 key),确定性强制逻辑已由 e2e + 单测 + 覆盖率间接验证;真实双跑待 key 就绪后补。
## 实施进度(2026-08-25 暂停于步骤 A 修正点)
### 已完成
- **步骤 0(中文模板)**`scripts/make_zh_template.py` + `samples/概要设计书模板_中文.docx`(已生成入库)+ `tests/test_zh_template.py`4 passed)。锚点 id 与 ja 模板逐一对应、译文正确。
- **步骤 1(参数贯通)**`config.py` 新增 `WriterConfig.output_language``Settings.writer``GenerationContext.output_language`+`to_vars().language_instruction`zh/ja/auto 三档);`writer_agent.py`【语言约束】改引 `{{language_instruction}}``context_builder.build_contexts` 透传;`orchestrator.generate``output_language` 参数;`run_trial.py``--output-language``tests/test_language_plumbing.py`5 passed)。
- **步骤 A(检测+强制)部分**`src/genesis/writer/language.py` 已建(detect_script/has_kana/has_cjk/resolve_expected_language/find_language_violations);`writer_agent.generate_chapter` 已接入期望语言推导 + 违规重试 + `max_retries` 默认 1→2。`tests/test_writer_language.py` 多数通过。
### 待修正(暂停点)
- **`language.py``resolve_expected_language` 逻辑需修正**:当前实现在标题为纯汉字(如「DB 設計」,无假名)时 `detect_script` 返回 "zh",导致标题被误当作可靠信号、fallback 失效。
依据计划「显式 > **标题假名** > 规则文档主导脚本」,标题**仅当含假名(detect=="ja"**才可信;纯汉字标题对中/日均可能,应跳过、改看 fallback。
**修正**:将 `resolve_expected_language` 中标题分支改为 `if detect_script(title) == "ja": return "ja"`(不再用纯汉字标题推导 zh),随后才遍历 fallback_texts,最后返回 ""。
- 修正后重跑 `tests/test_writer_language.py` 应全绿(已有 2 个用例因该逻辑失败:`test_resolve_falls_back_to_fallback_text``test_resolve_unverifiable_when_no_hint`)。
### 后续(未动)
- 步骤 A 通过后→ 步骤 B`_format_impact` 标签本地化已预埋参数,待补 zh 标签集使其与 ja 实际不同 + 补测试)。
- 步骤 CQA 第 11 维度 `language_consistency` + `ChapterArtifact.expected_language` + `validator.validate_doc(expected_language=None)` + `DEFAULT_THRESHOLDS` 加项 + 同步修正 `test_eval_scorer.py`/`test_phase5_scorer.py` 总分断言(按评审 R1:不可验证=满分 1.0)。
- 验收:全量 pytest ≥99% + zh/ja 双真实试运行 + 提交 + `_AI_USAGE_LOG.md` 追加。
### 已落盘但未提交
- 上述步骤 0/1/A 的源码与测试改动**尚未 git commit**(暂停前未提交)。
## 评审记录(2026-08-24
### 核验通过的关键假设
1. `config.py` 为 pydantic-settings + `extra="ignore"`,新增 `Settings.writer` 字段无 yaml 时取默认值,零风险
2. 中文模板锚点 idsection:introduction/function_list/screen_list/report_list/db_design/if_definition/batch_list)已从真实日文模板解析确认,镜像后映射/注入器零改动
3. `_format_impact` 无测试断言"受影响"字样,标签本地化安全;`[新規]/[変更]/[削除]/[警告]` 中日通用保留
4. `WriterAgent.max_retries` 既有测试均显式传参(2 或 1),默认值 1→2 不破坏测试
5. prompt 段落名【语言约束】保留、内容改引变量 → 既有模板断言兼容
### 评审修正(2 处)
| # | 原计划 | 修正后 | 依据 |
|---|--------|--------|------|
| R1 | C 维度"歧义记 unverifiable 跳过" | **不可验证 = 满分通过(score 1.0)**,维度恒参与均值;detail 注明 "unverifiable" | 实测 `test_eval_scorer.py L154` / `test_phase5_scorer.py L82` 断言 `total_score == 1.0`——若跳过维度或给非满分,既有用例总分断言即破裂;恒定维度数 + 满分兜底可让多数既有断言不变 |
| R2 | `ChapterArtifact.title: str = ""``expected_language` 两版表述并存 | 统一为 **仅加 `expected_language: str = ""`**,scorer 完全不自行推导 | 单一事实来源原则,避免 scorer 内再出现一套标题猜测逻辑 |
### 评审补充(3 处说明)
| # | 说明 |
|---|------|
| N1 | `validate_doc` 增加**可选** kwarg `expected_language=None`qa_loop.py L55/L69 与 test_phase5_report/test_phase5_validator 共 4 处调用方零破坏);None 时该章按 unverifiable=满分处理 |
| N2 | `max_retries` 默认 2 后,orchestrator.py L72 与 qa_loop.py L28 两处真实管线在失败时会多一次 LLM 调用——与 QALoopController 外层循环叠加的成本上界 = 章 × 2 × QA轮次,可接受;实施时在 WriterAgent docstring 标注 |
| N3 | 运维注意事项写入 README/计划:zh 场景规则文档仍为日文 → auto 回落会判 ja,**中文输出必须显式 `--output-language zh`** |
### 遗留风险(接受)
- auto 模式对纯汉字日文正式文档(无假名标题+无假名规则)无法证明语言——保守跳过强制,由 C 维度报告层呈现 unverifiable
Binary file not shown.
+82
View File
@@ -0,0 +1,82 @@
"""生成中文镜像模板(步骤 0)。
镜像既有日文概要设计书模板的结构与锚点 id(section:xxx 原样保留),
仅将 H1/H2 标题与封面字段翻译为中文,产物 samples/概要设计书模板_中文.docx 入库。
锚点段落文本保持不变是硬约束:Writer 子系统(template_mapper / docx_injector
依赖这些占位符定位章节,任何改动都会破坏映射与注入。
"""
from __future__ import annotations
import shutil
from pathlib import Path
from docx import Document
# H1/H2 标题日文 → 中文(保持编号前缀一一对应)
HEADING_TRANSLATIONS: dict[str, str] = {
"1. はじめに": "1. 前言",
"2. 機能一覧": "2. 功能一览",
"2.1 機能一覧表": "2.1 功能一览表",
"2.2 機能詳細": "2.2 功能详细",
"3. 画面一覧": "3. 画面一览",
"3.1 画面遷移図": "3.1 画面迁移图",
"3.2 画面一覧表": "3.2 画面一览表",
"4. 帳票一覧": "4. 报表一览",
"5. DB設計": "5. DB设计",
"5.1 テーブル一覧": "5.1 表一览",
"5.2 ER図": "5.2 ER图",
"6. IF定義": "6. 接口定义",
"7. バッチ一覧": "7. 批处理一览",
}
# 封面字段(Normal 段落)日文 → 中文
COVER_TRANSLATIONS: dict[str, str] = {
"概要設計書": "概要设计书",
"ドキュメント名: {{doc_title}}": "文档名: {{doc_title}}",
"バージョン: {{version}}": "版本: {{version}}",
"作成日: {{created_at}}": "创建日期: {{created_at}}",
"テンプレート開始": "模板开始",
}
def _translate(text: str) -> str:
"""锚点段落({{section:...}})原样保留;其余按字典翻译,未命中则原样。"""
if text.startswith("{{section:"):
return text
return HEADING_TRANSLATIONS.get(text, COVER_TRANSLATIONS.get(text, text))
def build_zh_template(src: str, dst: str) -> str:
"""镜像 src(日文模板)为 dst(中文模板),返回 dst 路径。"""
src_path = Path(src)
dst_path = Path(dst)
if not src_path.exists():
raise FileNotFoundError(f"源模板不存在: {src_path}")
dst_path.parent.mkdir(parents=True, exist_ok=True)
# 先物理复制(保留样式/编号定义等所有非文本属性),再逐段改写文本
shutil.copyfile(str(src_path), str(dst_path))
doc = Document(str(dst_path))
for para in doc.paragraphs:
new_text = _translate(para.text)
if new_text != para.text:
para.text = new_text
doc.save(str(dst_path))
return str(dst_path)
def main() -> None:
import argparse
parser = argparse.ArgumentParser(description="生成中文镜像模板")
default_src = str(Path(__file__).resolve().parents[1] / "samples" / "概要設計書テンプレート.docx")
default_dst = str(Path(__file__).resolve().parents[1] / "samples" / "概要设计书模板_中文.docx")
parser.add_argument("--src", default=default_src, help="源日文模板")
parser.add_argument("--dst", default=default_dst, help="输出中文模板")
args = parser.parse_args()
out = build_zh_template(args.src, args.dst)
print(f"中文模板已生成: {out}")
if __name__ == "__main__":
main()
+3
View File
@@ -53,6 +53,8 @@ def _parse_args(argv: list[str] | None) -> argparse.Namespace:
p.add_argument("--existing-system", default=str(ROOT / "samples" / "existing-system"), p.add_argument("--existing-system", default=str(ROOT / "samples" / "existing-system"),
help="既有系统源码目录(追加/改修场景)") help="既有系统源码目录(追加/改修场景)")
p.add_argument("--language", default="java", help="既有系统开发语言(默认 java,None=自动探测)") p.add_argument("--language", default="java", help="既有系统开发语言(默认 java,None=自动探测)")
p.add_argument("--output-language", default="auto",
choices=["auto", "zh", "ja"], help="输出语言:auto=与标题一致,zh=简体中文,ja=日文")
p.add_argument("--samples-dir", default=str(ROOT / "samples"), help="样本资产根目录") p.add_argument("--samples-dir", default=str(ROOT / "samples"), help="样本资产根目录")
p.add_argument("--output", default=str(ROOT / "output" / "output.docx"), help="概要设计书输出路径") p.add_argument("--output", default=str(ROOT / "output" / "output.docx"), help="概要设计书输出路径")
p.add_argument("--impact-report", default=str(ROOT / "output" / "impact-report.json"), p.add_argument("--impact-report", default=str(ROOT / "output" / "impact-report.json"),
@@ -86,6 +88,7 @@ def main(argv: list[str] | None = None) -> dict:
samples_dir=args.samples_dir, samples_dir=args.samples_dir,
engine=engine, engine=engine,
template_path=str(Path(args.template)), template_path=str(Path(args.template)),
output_language=args.output_language,
) )
report = ss.impact_report report = ss.impact_report
+15 -1
View File
@@ -2,7 +2,7 @@ from __future__ import annotations
import os import os
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any, Literal
import yaml import yaml
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
@@ -127,6 +127,19 @@ class RagConfig(BaseModel):
rerank: RerankConfig = Field(default_factory=RerankConfig) rerank: RerankConfig = Field(default_factory=RerankConfig)
class WriterConfig(BaseModel):
"""Writer 子系统配置(步骤 1:输出语言参数)。
output_language: 生成概要设计书正文的自然语言
- "auto":与章节标题所用语言保持一致(默认,向后兼容既有日文文档)
- "zh":强制简体中文
- "ja":强制日文
表格数据始终照抄源 Excel 原文(不翻译),见 design.md §7.2。
"""
output_language: Literal["auto", "zh", "ja"] = "auto"
# ---------- 加载辅助 ---------- # ---------- 加载辅助 ----------
def _expand_env(data: Any) -> Any: def _expand_env(data: Any) -> Any:
@@ -192,6 +205,7 @@ class Settings(BaseSettings):
app: AppConfig = Field(default_factory=AppConfig) app: AppConfig = Field(default_factory=AppConfig)
inference: InferenceConfig = Field(default_factory=InferenceConfig) inference: InferenceConfig = Field(default_factory=InferenceConfig)
rag: RagConfig = Field(default_factory=RagConfig) rag: RagConfig = Field(default_factory=RagConfig)
writer: WriterConfig = Field(default_factory=WriterConfig)
@classmethod @classmethod
def from_dir(cls, config_dir: Path | str) -> "Settings": def from_dir(cls, config_dir: Path | str) -> "Settings":
+45
View File
@@ -34,6 +34,9 @@ class ChapterArtifact:
text: str text: str
source_uris: list[str] source_uris: list[str]
template_sections_expected: list[str] template_sections_expected: list[str]
expected_language: str = "" # 期望输出语言("zh"/"ja";空=不可验证,维度记满分)
# 块级 (type, text) 列表:供语言一致性维度排除 heading/table(照抄源/跟随模板)
blocks: list[tuple[str, str]] = field(default_factory=list)
# 逐章评估通过阈值(基于逐章总分) # 逐章评估通过阈值(基于逐章总分)
@@ -56,6 +59,7 @@ DEFAULT_THRESHOLDS: dict[str, float] = {
"traceability": 1.0, "traceability": 1.0,
"placeholder_residue": 1.0, "placeholder_residue": 1.0,
"chapter_completeness": 1.0, "chapter_completeness": 1.0,
"language_consistency": 1.0,
} }
@@ -75,6 +79,7 @@ class ChapterScorer:
dimensions.append(self._traceability(chapters, source)) dimensions.append(self._traceability(chapters, source))
dimensions.append(self._placeholder_residue(chapters)) dimensions.append(self._placeholder_residue(chapters))
dimensions.append(self._completeness(chapters)) dimensions.append(self._completeness(chapters))
dimensions.append(self._language_consistency(chapters))
# LLM 语义维度钩子(每个章节独立评,取该维度平均) # LLM 语义维度钩子(每个章节独立评,取该维度平均)
for name, fn in self.llm_evaluators.items(): for name, fn in self.llm_evaluators.items():
@@ -144,6 +149,45 @@ class ChapterScorer:
detail = " | ".join(s.detail for s in scores) if scores else "no chapters" detail = " | ".join(s.detail for s in scores) if scores else "no chapters"
return DimensionScore(name, round(avg, 4), all(s.passed for s in scores), detail) return DimensionScore(name, round(avg, 4), all(s.passed for s in scores), detail)
# ---------- 语言一致性维度(步骤 C,确定性)----------
def _language_consistency(self, chapters: list[ChapterArtifact]) -> DimensionScore:
"""第 11 维度:输出语言与期望语言一致(确定性,脚本可验证)。
与 writer.language 共用单一检测事实来源。期望语言为空(auto/不可验证)
→ 记满分 1.0 通过(评审 R1:不拉低总分,避免误伤既有断言)。
仅检正文块(heading/table 不检,表格照抄源、标题跟随模板)。
"""
from genesis.writer.language import find_language_violations
if not chapters:
return DimensionScore("language_consistency", 1.0, True, "no chapters")
per: list[DimensionScore] = []
for ch in chapters:
expected = ch.expected_language
if not expected:
per.append(DimensionScore(
"language_consistency", 1.0, True, "unverifiable (no expected language)"))
continue
from genesis.writer.models import ContentBlock
if ch.blocks:
# 优先用块级信息(可排除 heading/table
blocks = [ContentBlock(block_id=str(i), type=t, text=tx)
for i, (t, tx) in enumerate(ch.blocks)]
else:
# 回退:整段正文作为单个 paragraph 块
blocks = [ContentBlock(block_id="0", type="paragraph", text=ch.text or "")]
viol = find_language_violations(blocks, expected)
score = 0.0 if viol else 1.0
passed = score >= self.thresholds["language_consistency"]
per.append(DimensionScore(
"language_consistency", score, passed,
f"期望 {expected},违规 {len(viol)}" if viol else f"期望 {expected},一致"))
avg = sum(p.score for p in per) / len(per)
passed = all(p.passed for p in per)
detail = " | ".join(p.detail for p in per)
return DimensionScore("language_consistency", round(avg, 4), passed, detail)
# ---------- 逐章评估 ---------- # ---------- 逐章评估 ----------
def _score_chapter(self, chapter: ChapterArtifact, source: StructuredSource) -> EvalReport: def _score_chapter(self, chapter: ChapterArtifact, source: StructuredSource) -> EvalReport:
@@ -156,6 +200,7 @@ class ChapterScorer:
self._traceability([chapter], source), self._traceability([chapter], source),
self._placeholder_residue([chapter]), self._placeholder_residue([chapter]),
self._adequacy(chapter), self._adequacy(chapter),
self._language_consistency([chapter]),
] ]
total = sum(d.score for d in dims) / len(dims) if dims else 0.0 total = sum(d.score for d in dims) / len(dims) if dims else 0.0
failed = (not all(d.passed for d in dims)) or total < PASS_THRESHOLD failed = (not all(d.passed for d in dims)) or total < PASS_THRESHOLD
+9 -6
View File
@@ -21,8 +21,8 @@ class QALoop:
def __init__(self, max_rounds: int = DEFAULT_MAX_QA_ROUNDS) -> None: def __init__(self, max_rounds: int = DEFAULT_MAX_QA_ROUNDS) -> None:
self.controller = QALoopController(max_rounds=max_rounds) self.controller = QALoopController(max_rounds=max_rounds)
def _build(self, structured_source, samples_dir, engine, prompt_registry, template_path, output_path, session_id, only_ids=None, prev=None): def _build(self, structured_source, samples_dir, engine, prompt_registry, template_path, output_path, session_id, only_ids=None, prev=None, output_language: str = "auto"):
ctxs = build_contexts(structured_source, samples_dir) ctxs = build_contexts(structured_source, samples_dir, output_language=output_language)
_warn_unanchored(ctxs) _warn_unanchored(ctxs)
state = WriterState([c.chapter_id for c in ctxs]) state = WriterState([c.chapter_id for c in ctxs])
agent = WriterAgent(session_id=session_id, engine=engine, prompt_registry=prompt_registry, state=state) agent = WriterAgent(session_id=session_id, engine=engine, prompt_registry=prompt_registry, state=state)
@@ -47,12 +47,14 @@ class QALoop:
doc.save(output_path) doc.save(output_path)
return [contents_map[cid] for cid in order] return [contents_map[cid] for cid in order]
def run(self, structured_source, output_path, session_id="writer", samples_dir="samples", engine=None, prompt_registry=None, template_path=None) -> QAReport: def run(self, structured_source, output_path, session_id="writer", samples_dir="samples", engine=None, prompt_registry=None, template_path=None, output_language: str = "auto") -> QAReport:
engine = engine or build_inference_engine() engine = engine or build_inference_engine()
prompt_registry = prompt_registry or PromptRegistry() prompt_registry = prompt_registry or PromptRegistry()
validator = QAValidator() validator = QAValidator()
contents = self._build(structured_source, samples_dir, engine, prompt_registry, template_path, output_path, session_id) # auto 不可推导期望语言 → 语言维度记满分(unverifiable);zh/ja 显式强制
report = validator.validate_doc(contents, structured_source) expected = output_language if output_language in ("zh", "ja") else ""
contents = self._build(structured_source, samples_dir, engine, prompt_registry, template_path, output_path, session_id, output_language=output_language)
report = validator.validate_doc(contents, structured_source, expected_language=expected)
while self.controller.can_continue() and report.failed_chapters: while self.controller.can_continue() and report.failed_chapters:
self.controller.advance() self.controller.advance()
contents = self._build( contents = self._build(
@@ -65,6 +67,7 @@ class QALoop:
session_id, session_id,
only_ids=set(report.failed_chapters), only_ids=set(report.failed_chapters),
prev={c.chapter_id: c for c in contents}, prev={c.chapter_id: c for c in contents},
output_language=output_language,
) )
report = validator.validate_doc(contents, structured_source) report = validator.validate_doc(contents, structured_source, expected_language=expected)
return report return report
+20 -8
View File
@@ -10,8 +10,12 @@ class QAValidator:
def __init__(self, scorer: ChapterScorer | None = None) -> None: def __init__(self, scorer: ChapterScorer | None = None) -> None:
self.scorer = scorer or ChapterScorer() self.scorer = scorer or ChapterScorer()
def _to_artifact(self, content: ChapterContent) -> ChapterArtifact: def _to_artifact(self, content: ChapterContent, expected_language: str = "") -> ChapterArtifact:
"""把 ChapterContent 转换为评分器所需的 ChapterArtifact(聚合正文与来源 URI)。""" """把 ChapterContent 转换为评分器所需的 ChapterArtifact(聚合正文与来源 URI)。
expected_language:期望输出语言("zh"/"ja";空=不可验证,维度记满分)。
blocks 保留 (type, text) 供语言维度排除 heading/table。
"""
text = "".join(b.text or "" for b in content.blocks) text = "".join(b.text or "" for b in content.blocks)
source_uris: list[str] = [] source_uris: list[str] = []
for b in content.blocks: for b in content.blocks:
@@ -21,18 +25,26 @@ class QAValidator:
text=text, text=text,
source_uris=source_uris, source_uris=source_uris,
template_sections_expected=[], template_sections_expected=[],
expected_language=expected_language,
blocks=[(b.type, b.text or "") for b in content.blocks],
) )
def validate_chapter(self, content: ChapterContent, structured_source) -> tuple[bool, EvalReport]: def validate_chapter(
report = self.scorer.score([self._to_artifact(content)], structured_source) self, content: ChapterContent, structured_source, expected_language: str = ""
) -> tuple[bool, EvalReport]:
report = self.scorer.score([self._to_artifact(content, expected_language)], structured_source)
passed = content.chapter_id not in report.failed_chapters passed = content.chapter_id not in report.failed_chapters
return passed, report return passed, report
def validate_document(self, contents: list[ChapterContent], structured_source) -> list[tuple[bool, EvalReport]]: def validate_document(
return [self.validate_chapter(c, structured_source) for c in contents] self, contents: list[ChapterContent], structured_source, expected_language: str = ""
) -> list[tuple[bool, EvalReport]]:
return [self.validate_chapter(c, structured_source, expected_language) for c in contents]
def validate_doc(self, contents: list[ChapterContent], structured_source) -> QAReport: def validate_doc(
results = self.validate_document(contents, structured_source) self, contents: list[ChapterContent], structured_source, expected_language: str = ""
) -> QAReport:
results = self.validate_document(contents, structured_source, expected_language)
passed_flags = [ok for ok, _ in results] passed_flags = [ok for ok, _ in results]
reports = [rep for _, rep in results] reports = [rep for _, rep in results]
failed = [c.chapter_id for c, ok in zip(contents, passed_flags) if not ok] failed = [c.chapter_id for c, ok in zip(contents, passed_flags) if not ok]
+6 -1
View File
@@ -7,7 +7,11 @@ from genesis.writer.models import ChapterSpec, GenerationContext
from genesis.writer.template_mapper import map_template from genesis.writer.template_mapper import map_template
def build_contexts(structured_source: StructuredSource, samples_dir: str = "samples") -> list[GenerationContext]: def build_contexts(
structured_source: StructuredSource,
samples_dir: str = "samples",
output_language: str = "auto",
) -> list[GenerationContext]:
specs: list[ChapterSpec] = map_template(structured_source.template) specs: list[ChapterSpec] = map_template(structured_source.template)
rag = CannedRagService(samples_dir=samples_dir) rag = CannedRagService(samples_dir=samples_dir)
out: list[GenerationContext] = [] out: list[GenerationContext] = []
@@ -27,6 +31,7 @@ def build_contexts(structured_source: StructuredSource, samples_dir: str = "samp
template_styles=set(used), template_styles=set(used),
prior_state=None, prior_state=None,
impact_report=getattr(structured_source, "impact_report", None), impact_report=getattr(structured_source, "impact_report", None),
output_language=output_language,
) )
) )
return out return out
+104
View File
@@ -0,0 +1,104 @@
"""输出语言确定性检测与强制(步骤 A)。
设计要点:
- 日文标题多为纯汉字(如「DB設計」无假名),仅凭标题无法判定期望语言;
故 resolve_expected_language 采用两级推导:显式 > 标题假名 > 规则文档主导脚本。
- 检测仅基于「是否含日文假名」:CJK 汉字零假名视为中文(日文不可能不含假名地
使用汉字),反之中日混排含假名判日文。这是确定可机器验证的唯一稳健信号。
- find_language_violations 仅检正文类块(paragraph/note/list);heading 跟随模板、
table 照抄源 Excel 原文,二者均不检(design.md §7.2 内容准确性/可追溯性)。
- 短文本(<12 字)不误杀(如专有术语),阈值见 MIN_VIOLATION_LEN。
"""
from __future__ import annotations
from genesis.writer.models import ContentBlock
# 日文假名 Unicode 区间
_HIRAGANA = (0x3040, 0x309F)
_KATAKANA = (0x30A0, 0x30FF)
# 中日韩统一表意文字(CJK 汉字)
_CJK = (0x4E00, 0x9FFF)
# 受检的正文块类型(heading/table 不检)
_CHECKED_BLOCK_TYPES = {"paragraph", "note", "list"}
# 触发违规判定的最小正文长度(防短术语误杀)
MIN_VIOLATION_LEN = 12
def _in_range(ch: str, lo: int, hi: int) -> bool:
cp = ord(ch)
return lo <= cp <= hi
def has_kana(text: str) -> bool:
"""文本是否含日文假名(平假名/片假名)。"""
return any(_in_range(c, *_HIRAGANA) or _in_range(c, *_KATAKANA) for c in text)
def has_cjk(text: str) -> bool:
"""文本是否含 CJK 汉字。"""
return any(_in_range(c, *_CJK) for c in text)
def detect_script(text: str) -> str | None:
"""检测文本主导自然语言。
含假名 → "ja";含 CJK 汉字但零假名 → "zh";二者皆非(纯 ASCII 等)→ None。
"""
if not text:
return None
if has_kana(text):
return "ja"
if has_cjk(text):
return "zh"
return None
def resolve_expected_language(
explicit: str,
title: str = "",
fallback_texts: tuple[str, ...] | list[str] = (),
) -> str:
"""推导本章期望输出语言(单一事实来源,A 的重试校验与 C 的 QA 维度共用)。
- explicit 为 "zh"/"ja" → 直接采用(用户显式选择优先)
- 否则看标题是否含假名(仅假名可可靠判为日文;纯汉字标题对中/日均可能,不可信)
- 否则看 fallback_texts(如影响调查书/章节数据,通常日文)的主导脚本
- 均无法推导 → 返回 ""(不可验证,交由上层按 unverifiable 处理)
"""
if explicit in ("zh", "ja"):
return explicit
# 标题仅当含假名时可靠指示日文;纯汉字/ASCII 标题跳过,改看 fallback
if has_kana(title or ""):
return "ja"
for text in fallback_texts:
s = detect_script(text or "")
if s:
return s
return ""
def find_language_violations(blocks: list[ContentBlock], expected_language: str) -> list[str]:
"""返回违规正文块文本片段(期望语言非空时才有意义)。
违规判定:
- 期望 "ja":正文块含 CJK 汉字且零假名(即纯中文)且长度 ≥ 阈值
- 期望 "zh":正文块含日文假名
heading/table 块始终跳过(跟随模板 / 照抄源)。
"""
if expected_language not in ("zh", "ja"):
return []
violations: list[str] = []
for b in blocks:
if b.type not in _CHECKED_BLOCK_TYPES:
continue
text = b.text or ""
if len(text) < MIN_VIOLATION_LEN:
continue
if expected_language == "ja":
if has_cjk(text) and not has_kana(text):
violations.append(text)
else: # zh
if has_kana(text):
violations.append(text)
return violations
+40 -5
View File
@@ -109,6 +109,19 @@ class GenerationContext:
template_styles: set[str] template_styles: set[str]
prior_state: object | None = None # WriterState,避免循环 import 用 object prior_state: object | None = None # WriterState,避免循环 import 用 object
impact_report: object | None = None # ImpactReport 影响调查书(生成主上下文) impact_report: object | None = None # ImpactReport 影响调查书(生成主上下文)
output_language: str = "auto" # "auto" | "zh" | "ja"(步骤 1:用户可选输出语言)
def _language_instruction(self) -> str:
"""根据 output_language 生成【语言约束】段的具体指令(步骤 1)。"""
if self.output_language == "zh":
return "必须使用简体中文撰写(标题、正文与所有说明一律中文)。"
if self.output_language == "ja":
return "必ず日本語で記述すること(タイトル・本文・すべての説明は日本語)。"
# auto:沿用与标题语言一致的旧语义(向后兼容既有日文文档)
return (
f"必须与章节标题「{self.title}」所用自然语言保持一致:"
"标题为日文则用日文撰写,为中文则用中文撰写,依此类推。"
)
def to_vars(self) -> dict: def to_vars(self) -> dict:
"""返回供 prompt 渲染的变量字典。""" """返回供 prompt 渲染的变量字典。"""
@@ -125,6 +138,7 @@ class GenerationContext:
f"- {h}" for h in (getattr(tm, "sub_headings", None) or []) f"- {h}" for h in (getattr(tm, "sub_headings", None) or [])
), ),
"prior_state": str(self.prior_state) if self.prior_state is not None else "", "prior_state": str(self.prior_state) if self.prior_state is not None else "",
"language_instruction": self._language_instruction(),
"data": _format_chapter_data( "data": _format_chapter_data(
self.structured_source, self.structured_source,
CHAPTER_SHEET_TYPES.get(self.chapter_id, []), CHAPTER_SHEET_TYPES.get(self.chapter_id, []),
@@ -132,6 +146,7 @@ class GenerationContext:
"impact": _format_impact( "impact": _format_impact(
self.impact_report, self.impact_report,
CHAPTER_IMPACT_ELEMENT.get(self.chapter_id, None), CHAPTER_IMPACT_ELEMENT.get(self.chapter_id, None),
self.output_language,
), ),
} }
@@ -177,11 +192,30 @@ def _format_chapter_data(structured_source: object | None, sheet_types: list[She
return "\n".join(lines).rstrip() return "\n".join(lines).rstrip()
def _format_impact(report: object | None, element_type: ElementType | None = None) -> str: # 影响调查标签本地化(步骤 B):auto/ja 默认日文,zh 中文
# 注:方括号标记 [..] 保留(中日通用),仅标签词本地化
_IMPACT_LABELS: dict[str, dict[str, str]] = {
"zh": {
"new": "新建", "modified": "变更", "deleted": "删除", "warning": "警告",
"affected": "受影响",
},
"ja": {
"new": "新規", "modified": "変更", "deleted": "削除", "warning": "警告",
"affected": "受影响",
},
}
def _format_impact(
report: object | None,
element_type: ElementType | None = None,
output_language: str = "auto",
) -> str:
"""将影响调查书格式化为 prompt 可读文本(无报告/无分析时为空串)。 """将影响调查书格式化为 prompt 可读文本(无报告/无分析时为空串)。
element_type 非 None 时仅保留该类型要素(章节级定向,design.md §6.8 ①); element_type 非 None 时仅保留该类型要素(章节级定向,design.md §6.8 ①);
警告始终保留(不依赖要素类型)。 警告始终保留(不依赖要素类型)。
output_language 控制标签语言(步骤 B);auto 回落到 ja 标签。
""" """
if report is None: if report is None:
return "" return ""
@@ -192,6 +226,7 @@ def _format_impact(report: object | None, element_type: ElementType | None = Non
def keep(el) -> bool: def keep(el) -> bool:
return element_type is None or el.element_type == element_type.value return element_type is None or el.element_type == element_type.value
lab = _IMPACT_LABELS.get(output_language, _IMPACT_LABELS["ja"])
summary = getattr(report, "summary", {}) or {} summary = getattr(report, "summary", {}) or {}
lines = [f"project_type={getattr(ca, 'project_type', '')}"] lines = [f"project_type={getattr(ca, 'project_type', '')}"]
lines.append( lines.append(
@@ -204,15 +239,15 @@ def _format_impact(report: object | None, element_type: ElementType | None = Non
) )
for el in getattr(ca, "new_elements", []) or []: for el in getattr(ca, "new_elements", []) or []:
if keep(el): if keep(el):
lines.append(f"[新規] {el.element_id} {el.element_type} {el.name}") lines.append(f"[{lab['new']}] {el.element_id} {el.element_type} {el.name}")
for el in getattr(ca, "modified_elements", []) or []: for el in getattr(ca, "modified_elements", []) or []:
if keep(el): if keep(el):
impacted = ", ".join(el.impacted_existing) or "-" impacted = ", ".join(el.impacted_existing) or "-"
lines.append(f"[変更] {el.element_id} {el.element_type} {el.name}受影响: {impacted}") lines.append(f"[{lab['modified']}] {el.element_id} {el.element_type} {el.name}{lab['affected']}: {impacted}")
for el in getattr(ca, "deleted_elements", []) or []: for el in getattr(ca, "deleted_elements", []) or []:
if keep(el): if keep(el):
impacted = ", ".join(el.impacted_existing) or "-" impacted = ", ".join(el.impacted_existing) or "-"
lines.append(f"[削除] {el.element_id} {el.element_type} {el.name}受影响: {impacted}") lines.append(f"[{lab['deleted']}] {el.element_id} {el.element_type} {el.name}{lab['affected']}: {impacted}")
for w in getattr(ca, "warnings", []) or []: for w in getattr(ca, "warnings", []) or []:
lines.append(f"[警告] {w.element_id}: {w.issue}") lines.append(f"[{lab['warning']}] {w.element_id}: {w.issue}")
return "\n".join(lines) return "\n".join(lines)
+2 -1
View File
@@ -56,6 +56,7 @@ class WriteOrchestrator:
template_path: str | None = None, template_path: str | None = None,
impact_report=None, impact_report=None,
meta: dict | None = None, meta: dict | None = None,
output_language: str = "auto",
) -> list[ChapterContent]: ) -> list[ChapterContent]:
engine = engine or build_inference_engine() engine = engine or build_inference_engine()
prompt_registry = prompt_registry or PromptRegistry() prompt_registry = prompt_registry or PromptRegistry()
@@ -66,7 +67,7 @@ class WriteOrchestrator:
if impact_report is not None: if impact_report is not None:
# 回填 structured_source,便于 QA/日志/后续下载 # 回填 structured_source,便于 QA/日志/后续下载
structured_source.impact_report = impact_report structured_source.impact_report = impact_report
ctxs = build_contexts(structured_source, samples_dir) ctxs = build_contexts(structured_source, samples_dir, output_language=output_language)
_warn_unanchored(ctxs) _warn_unanchored(ctxs)
state = WriterState([c.chapter_id for c in ctxs]) state = WriterState([c.chapter_id for c in ctxs])
agent = WriterAgent(session_id=session_id, engine=engine, prompt_registry=prompt_registry, state=state) agent = WriterAgent(session_id=session_id, engine=engine, prompt_registry=prompt_registry, state=state)
+26 -5
View File
@@ -13,6 +13,10 @@ from genesis.inference.types import Prompt, StructuredResult
from genesis.writer.models import ChapterContent, GenerationContext from genesis.writer.models import ChapterContent, GenerationContext
from genesis.writer.writer_state import WriterState from genesis.writer.writer_state import WriterState
from genesis.writer.exceptions import WriterGenerationError from genesis.writer.exceptions import WriterGenerationError
from genesis.writer.language import (
find_language_violations,
resolve_expected_language,
)
WRITER_PROMPT_TEMPLATE = ( WRITER_PROMPT_TEMPLATE = (
@@ -33,10 +37,9 @@ WRITER_PROMPT_TEMPLATE = (
"严格以「参考资料(本章对应数据)」中的要件定义数据和「影响调查上下文」为核心依据;" "严格以「参考资料(本章对应数据)」中的要件定义数据和「影响调查上下文」为核心依据;"
"禁止输出与本章无关的系统整体架构、通用设计说明等内容,禁止套用其他章节的主题。" "禁止输出与本章无关的系统整体架构、通用设计说明等内容,禁止套用其他章节的主题。"
"若本章数据为空,则基于规则与影响调查上下文简要撰写,不得虚构数据。\n" "若本章数据为空,则基于规则与影响调查上下文简要撰写,不得虚构数据。\n"
"【语言约束】章节正文(所有 block 的 text 字段)所使用的自然语言" "【语言约束】章节正文(所有 block 的 text 字段)所使用的自然语言{{language_instruction}}\n"
"必须与章节标题「{{title}}」所用语言保持一致:标题为日文则用日文撰写," )
"为中文则用中文撰写,依此类推。"
)
CHAPTER_OUTPUT_SCHEMA = { CHAPTER_OUTPUT_SCHEMA = {
"type": "object", "type": "object",
@@ -70,7 +73,7 @@ class WriterAgent:
engine: InferenceEngine, engine: InferenceEngine,
prompt_registry: PromptRegistry, prompt_registry: PromptRegistry,
state: WriterState, state: WriterState,
max_retries: int = 1, max_retries: int = 2,
) -> None: ) -> None:
self.session_id = session_id self.session_id = session_id
self.engine = engine self.engine = engine
@@ -132,6 +135,15 @@ class WriterAgent:
def generate_chapter(self, context: GenerationContext) -> ChapterContent: def generate_chapter(self, context: GenerationContext) -> ChapterContent:
self._chunk_source(context.structured_source) # 分块可用性验证(真实拼回留待后续) self._chunk_source(context.structured_source) # 分块可用性验证(真实拼回留待后续)
# 步骤 A:推导本章期望输出语言(仅依赖 context,循环前置,稳定)
v = context.to_vars()
expected = resolve_expected_language(
explicit=context.output_language,
title=context.title,
fallback_texts=[v.get("impact", "") or "", v.get("data", "") or ""],
)
last_err: Exception | None = None last_err: Exception | None = None
for _ in range(max(1, self.max_retries)): for _ in range(max(1, self.max_retries)):
try: try:
@@ -148,6 +160,15 @@ class WriterAgent:
# heading 块(prompt 约束为尽力而为,此处程序化强制) # heading 块(prompt 约束为尽力而为,此处程序化强制)
if not (getattr(context.template_marker, "sub_headings", None) or []): if not (getattr(context.template_marker, "sub_headings", None) or []):
content.blocks = [b for b in content.blocks if b.type != "heading"] content.blocks = [b for b in content.blocks if b.type != "heading"]
# 步骤 A:输出语言一致性强制(期望语言可推导时,违规按失败重试)
if expected:
viol = find_language_violations(content.blocks, expected)
if viol:
last_err = WriterGenerationError(
f"章节 {context.chapter_id} 语言不一致(期望 {expected}"
f"发现 {len(viol)} 处违规正文)"
)
continue
self.state.record_success(content) self.state.record_success(content)
return content return content
raise WriterGenerationError(f"章节 {context.chapter_id} 重试耗尽: {last_err}") raise WriterGenerationError(f"章节 {context.chapter_id} 重试耗尽: {last_err}")
+102
View File
@@ -0,0 +1,102 @@
"""步骤 A/C 边界覆盖测试(恢复覆盖率至 ≥99%)。"""
from __future__ import annotations
from pathlib import Path
from types import SimpleNamespace
from genesis.writer.language import find_language_violations
from genesis.writer.models import ChapterContent, ContentBlock, GenerationContext, ChapterSpec
from genesis.writer.writer_agent import WriterAgent
from genesis.writer.writer_state import WriterState
from genesis.writer.exceptions import WriterGenerationError
from genesis.qa.qa_loop import QALoop
from genesis.data_models import StructuredSource
from genesis.parsers.word_template_parser import WordTemplateParser
class _Reg:
@staticmethod
def get_or_create(name, template):
return SimpleNamespace(name=name, version="1", template=template)
def test_find_violations_unexpected_language_code_returns_empty():
blocks = [ContentBlock(block_id="0", type="paragraph", text="中文正文内容充分长度足够")]
# 非 zh/ja 的期望语言 → 不强制,返回空
assert find_language_violations(blocks, "xx") == []
def test_writer_agent_zh_expected_rejects_japanese():
class JpEngine:
def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2):
return SimpleNamespace(
data={"title": variables["title"],
"blocks": [{"type": "paragraph",
"text": "本機能は注文処理を行う画面であり、詳細は以下の通り。"}]},
status="ok",
)
agent = WriterAgent(session_id="s", engine=JpEngine(), prompt_registry=_Reg(),
state=WriterState(["db_design"]), max_retries=1)
ctx = GenerationContext(chapter_id="db_design", title="DB 設計",
template_marker=ChapterSpec(chapter_id="db_design", title="DB 設計"),
structured_source=None, write_rules=[], design_rules=[],
template_styles=set(), output_language="zh")
try:
agent.generate_chapter(ctx)
assert False, "应抛 WriterGenerationError"
except WriterGenerationError:
pass
def test_qa_loop_passes_with_explicit_zh_on_chinese_fake(tmp_path):
"""显式 zh + 中文 Fake → 语言维度通过(验证透传链路)。"""
template = Path("samples/phase5-slice/template.docx")
if not template.exists():
import pytest
pytest.skip("样本模板缺失")
parsed = WordTemplateParser().parse(str(template))
ss = StructuredSource(template=parsed, tables=[], rule_docs=[], image_analyses=[],
existing_system=None, comments=[])
out = tmp_path / "o.docx"
class ZhEngine:
def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2):
return SimpleNamespace(
data={"title": variables["title"],
"blocks": [{"type": "paragraph",
"text": "由 FakeLLM 生成的充分说明内容,满足写入规则要求。"}]},
status="ok",
)
loop = QALoop(max_rounds=1)
report = loop.run(ss, str(out), samples_dir="nonexistent_dir_xyz",
engine=ZhEngine(), template_path=str(template), output_language="zh")
assert report.passed is True
def test_qa_loop_fails_with_explicit_ja_on_chinese_fake(tmp_path):
"""显式 ja + 中文 Fake → 语言维度判失败。"""
template = Path("samples/phase5-slice/template.docx")
if not template.exists():
import pytest
pytest.skip("样本模板缺失")
parsed = WordTemplateParser().parse(str(template))
ss = StructuredSource(template=parsed, tables=[], rule_docs=[], image_analyses=[],
existing_system=None, comments=[])
out = tmp_path / "o.docx"
class ZhEngine:
def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2):
return SimpleNamespace(
data={"title": variables["title"],
"blocks": [{"type": "paragraph",
"text": "由 FakeLLM 生成的充分说明内容,满足写入规则要求。"}]},
status="ok",
)
loop = QALoop(max_rounds=1)
# 显式 ja + 中文 Fake:writer 生成阶段即因语言不一致硬失败(设计语义)
try:
loop.run(ss, str(out), samples_dir="nonexistent_dir_xyz",
engine=ZhEngine(), template_path=str(template), output_language="ja")
assert False, "应抛出 WriterGenerationError"
except WriterGenerationError:
pass
+93
View File
@@ -0,0 +1,93 @@
"""步骤 1:语言参数贯通测试(config → GenerationContext → to_vars → context_builder → run_trial)。"""
from __future__ import annotations
from genesis.data_models import (
CellValue, ExcelTable, ParsedTemplate, Provenance, SheetType, StructuredSource,
)
from genesis.writer.context_builder import build_contexts
from genesis.writer.models import ChapterSpec, GenerationContext
def _src() -> StructuredSource:
return StructuredSource(
tables=[],
template=ParsedTemplate("t.docx", [__import__("genesis.data_models", fromlist=["ChapterMarker"]).ChapterMarker("heading", "1. はじめに", 1)], {}, {"used": []}),
rule_docs=[], image_analyses=[], existing_system=None, comments=[],
)
def test_generation_context_default_output_language_is_auto():
ctx = GenerationContext(
chapter_id="db_design", title="DB 設計",
template_marker=ChapterSpec(chapter_id="db_design", title="DB 設計"),
structured_source=None, write_rules=[], design_rules=[], template_styles=set(),
)
assert ctx.output_language == "auto"
def test_to_vars_language_instruction_explicit_zh():
ctx = GenerationContext(
chapter_id="db_design", title="DB 设計",
template_marker=ChapterSpec(chapter_id="db_design", title="DB 设計"),
structured_source=None, write_rules=[], design_rules=[], template_styles=set(),
output_language="zh",
)
v = ctx.to_vars()
assert "language_instruction" in v
assert "简体中文" in v["language_instruction"]
def test_to_vars_language_instruction_explicit_ja():
ctx = GenerationContext(
chapter_id="db_design", title="DB 設計",
template_marker=ChapterSpec(chapter_id="db_design", title="DB 設計"),
structured_source=None, write_rules=[], design_rules=[], template_styles=set(),
output_language="ja",
)
v = ctx.to_vars()
assert "日本語" in v["language_instruction"]
def test_to_vars_language_instruction_auto_follows_title():
ctx = GenerationContext(
chapter_id="db_design", title="DB 設計",
template_marker=ChapterSpec(chapter_id="db_design", title="DB 設計"),
structured_source=None, write_rules=[], design_rules=[], template_styles=set(),
output_language="auto",
)
v = ctx.to_vars()
assert "标题" in v["language_instruction"] and "一致" in v["language_instruction"]
def test_context_builder_passes_output_language():
src = _src()
ctxs = build_contexts(src, output_language="zh")
assert all(c.output_language == "zh" for c in ctxs)
# 回归:默认仍为 auto
ctxs2 = build_contexts(src)
assert all(c.output_language == "auto" for c in ctxs2)
def test_format_impact_zh_labels_localized():
"""步骤 Boutput_language=zh 时影响调查标签本地化为中文,且不影响 ja 默认。"""
from genesis.data_models import (
ChangeAnalysis, ChangeElement, ChangeType, ImpactReport, ImpactWarning,
)
from genesis.writer.models import _format_impact
ca = ChangeAnalysis(
project_type="enhancement",
new_elements=[ChangeElement("F001", "機能", "止损风控", ChangeType.NEW)],
modified_elements=[ChangeElement("F002", "機能", "下单流程", ChangeType.MODIFIED)],
deleted_elements=[ChangeElement("F030", "機能", "旧功能", ChangeType.DELETED)],
unchanged_elements=[],
warnings=[ImpactWarning("F030", "缺少既存対応")],
)
report = ImpactReport(metadata={}, change_analysis=ca, summary={})
zh = _format_impact(report, None, "zh")
assert "[新建]" in zh and "[变更]" in zh and "[删除]" in zh and "[警告]" in zh
assert "受影响" in zh
# ja / auto 回落仍用日文标签
ja = _format_impact(report, None, "ja")
assert "[新規]" in ja and "[削除]" in ja
+5 -4
View File
@@ -12,13 +12,14 @@ from genesis.qa.qa_loop import QALoop
class FakeEngine: class FakeEngine:
def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2): def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2):
cid = variables.get("title", "x") cid = variables.get("title", "x")
# 日文模板章节(标题含假名)→ 返回日文正文,满足语言一致性强制
return SimpleNamespace( return SimpleNamespace(
data={ data={
"title": cid, "title": cid,
"blocks": [ "blocks": [
{"type": "heading", "level": 2, "text": f"{cid}"}, {"type": "heading", "level": 2, "text": f"{cid}"},
{"type": "paragraph", "text": "FakeLLM 生成的充分说明内容,满足写入规则要求"}, {"type": "paragraph", "text": "本機能はFakeLLMにより生成された十分な説明内容であり、書込規則を満たす"},
{"type": "table", "caption": "示例表", "rows": [["列1", "列2"], ["1", "2"]]}, {"type": "table", "caption": "示例表", "rows": [["列1", "列2"], ["1", "2"]]},
], ],
}, },
status="ok", status="ok",
@@ -50,5 +51,5 @@ def test_phase5_e2e_fake_llm(tmp_path):
assert report.passed is True assert report.passed is True
loaded = Document(str(out)) loaded = Document(str(out))
text = _all_text(loaded) text = _all_text(loaded)
assert "FakeLLM 生成的充分说明内容" in text assert "本機能はFakeLLMにより生成された十分な説明内容" in text
assert "示例表" in text assert "示例表" in text
+92
View File
@@ -0,0 +1,92 @@
"""步骤 CQA 第 11 维度 language_consistency 测试。
评审 R1:不可验证(expected_language 为空/auto)→ 维度满分 1.0 通过,不拉低总分,
使既有 test_eval_scorer / test_phase5_scorer 的 total_score==1.0 断言继续成立。
"""
from __future__ import annotations
from genesis.data_models import (
CellValue, ExcelTable, ParsedTemplate, Provenance, SheetType, StructuredSource,
)
from genesis.eval.scorer import ChapterArtifact, ChapterScorer, EvalReport
def _dim(report: EvalReport, name: str):
for d in report.dimensions:
if d.name == name:
return d
raise AssertionError(f"维度未找到: {name}")
def _source() -> StructuredSource:
cell = CellValue(
value="登録",
provenance=Provenance(file_name="f.xlsx", sheet_name="機能一覧", row=3, column="C", column_header="x"),
)
table = ExcelTable(
name="機能一覧", detected_type=SheetType.FUNCTION,
extraction_method="structured", headers=["v"], rows=[{"v": cell}],
)
return StructuredSource(
tables=[table],
template=ParsedTemplate(file_name="t.docx", sections=[], placeholders={}, styles={}),
rule_docs=[], image_analyses=[], existing_system=None, comments=[],
)
def test_language_dim_unverifiable_is_full_score():
"""expected_language 为空(auto/不可验证)→ 维度满分通过,不拉低总分。"""
src = _source()
artifact = ChapterArtifact(
chapter_id="ch3", text="本機能は注文処理を行う画面である。",
source_uris=[], template_sections_expected=["ch3"],
expected_language="",
)
report = ChapterScorer().score([artifact], src)
dim = _dim(report, "language_consistency")
assert dim.score == 1.0
assert dim.passed is True
# 既有总分断言仍成立(全部确定性维度满分)
assert report.total_score == 1.0
def test_language_dim_ja_expected_but_chinese_fails():
src = _source()
artifact = ChapterArtifact(
chapter_id="ch3", text="这是一段纯中文的章节正文内容,应当判定为语言违规。",
source_uris=[], template_sections_expected=["ch3"],
expected_language="ja",
)
report = ChapterScorer().score([artifact], src)
dim = _dim(report, "language_consistency")
assert dim.score == 0.0
assert dim.passed is False
assert report.total_score < 1.0
def test_language_dim_zh_expected_japanese_fails():
src = _source()
artifact = ChapterArtifact(
chapter_id="ch3", text="本機能は注文処理を行う画面であり、詳細は以下の通り。",
source_uris=[], template_sections_expected=["ch3"],
expected_language="zh",
)
report = ChapterScorer().score([artifact], src)
dim = _dim(report, "language_consistency")
assert dim.score == 0.0
def test_language_dim_threshold_in_defaults():
assert ChapterScorer().thresholds["language_consistency"] == 1.0
def test_dimension_count_increases_by_one():
src = _source()
artifact = ChapterArtifact(
chapter_id="ch3", text="正常内容", source_uris=[], template_sections_expected=["ch3"],
)
before = len(ChapterScorer(thresholds={}, llm_evaluators={}).score([artifact], src).dimensions)
# 默认 scorer 含 language_consistency;对比一个不含该维度的基线不可行,
# 此处仅断言维度中包含 language_consistency 名称
report = ChapterScorer().score([artifact], src)
assert any(d.name == "language_consistency" for d in report.dimensions)
+172
View File
@@ -0,0 +1,172 @@
"""步骤 A:确定性语言检测 + 输出语言强制测试。
- detect_script:含假名→jaCJK 零假名→zh,否则 None
- resolve_expected_language:显式 > 标题假名 > 规则文档主导脚本 > ""(不可验证)
- find_language_violations:仅检正文块(heading/table 不检),长度≥12 防误杀
- WriterAgent.generate_chapter:期望语言非空时违规即重试,耗尽→硬失败
"""
from __future__ import annotations
from genesis.writer.language import (
detect_script,
find_language_violations,
resolve_expected_language,
)
from genesis.writer.models import ChapterContent, ContentBlock, GenerationContext, ChapterSpec
from genesis.writer.writer_agent import WriterAgent
from genesis.writer.writer_state import WriterState
from genesis.writer.exceptions import WriterGenerationError
# ---------- detect_script ----------
def test_detect_script_kana_is_ja():
assert detect_script("機能一覧の説明を記述します。") == "ja"
def test_detect_script_cjk_no_kana_is_zh():
assert detect_script("功能一览的说明内容。") == "zh"
def test_detect_script_ascii_is_none():
assert detect_script("Hello world 123") is None
def test_detect_script_mixed_cjk_kana_is_ja():
# 含假名即判 ja,即使混有汉字(中文不可能含日文假名)
assert detect_script("DB設計の概要を説明する。") == "ja"
# ---------- resolve_expected_language ----------
def test_resolve_explicit_overrides():
assert resolve_expected_language("zh", "DB 設計", ["日本語の影響"]) == "zh"
assert resolve_expected_language("ja", "DB 设計", ["中文影响"]) == "ja"
def test_resolve_falls_back_to_title_kana():
# 标题含假名→ja(注意:纯汉字日文标题如「DB設計」无假名,会落入 fallback)
assert resolve_expected_language("auto", "機能一覧の説明", []) == "ja"
def test_resolve_falls_back_to_fallback_text():
# 标题纯汉字无假名(中文式)→ 看 fallback(日文影响书)→ ja
assert resolve_expected_language("auto", "DB 設計", ["影響調査の結果(日本語)"]) == "ja"
def test_resolve_unverifiable_when_no_hint():
assert resolve_expected_language("auto", "DB 設計", ["123 abc"]) == ""
# ---------- find_language_violations ----------
def _blocks(*specs):
out = []
for i, (t, text) in enumerate(specs):
out.append(ContentBlock(block_id=str(i), type=t, text=text))
return out
def test_violation_ja_expected_but_pure_chinese_paragraph():
blocks = _blocks(("paragraph", "这是一段纯中文的章节正文内容。"))
viol = find_language_violations(blocks, "ja")
assert viol # 期望日文却为中文 → 违规
def test_no_violation_ja_expected_japanese_paragraph():
blocks = _blocks(("paragraph", "本機能は注文処理を行う画面であり、詳細は以下の通り。"))
assert not find_language_violations(blocks, "ja")
def test_heading_and_table_blocks_excluded():
# heading 与 table(照抄原文)即使含中文也不算违规
blocks = _blocks(
("heading", "機能一覧表"),
("table", "機能ID 機能名"), # 表格不检
("paragraph", "本機能は注文処理を行う。"),
)
assert not find_language_violations(blocks, "ja")
def test_short_chinese_not_flagged_under_threshold():
# 长度<12 的短术语不误杀
blocks = _blocks(("paragraph", "中文术语"))
assert not find_language_violations(blocks, "ja")
def test_violation_zh_expected_but_kana_present():
blocks = _blocks(("paragraph", "本機能は注文処理を行う画面である。"))
assert find_language_violations(blocks, "zh")
# ---------- WriterAgent 强制 ----------
class MixedLanguageEngine:
"""返回中英混杂(期望日文时违规)的章节内容。"""
def __init__(self, texts):
self._texts = list(texts)
self._i = 0
def chat_structured(self, *, session_id, prompt, variables, schema, retry_count=2):
from types import SimpleNamespace
text = self._texts[min(self._i, len(self._texts) - 1)]
self._i += 1
return SimpleNamespace(
data={"title": variables["title"], "blocks": [{"type": "paragraph", "text": text}]},
status="ok",
)
class FakePromptRegistry:
@staticmethod
def get_or_create(name, template):
from types import SimpleNamespace
return SimpleNamespace(name=name, version="1", template=template)
def _ctx_with_lang(cid, title, lang):
return GenerationContext(
chapter_id=cid, title=title,
template_marker=ChapterSpec(chapter_id=cid, title=title),
structured_source=None, write_rules=[], design_rules=[],
template_styles=set(), output_language=lang,
)
def test_generate_chapter_retries_on_language_violation_then_passes():
# 第一次返回中文(ja 期望→违规),第二次返回日文(通过)
engine = MixedLanguageEngine([
"这是一段纯中文的章节正文内容,应当被判定为语言违规。",
"本機能は注文処理を行う画面であり、詳細は以下の通り記述する。",
])
agent = WriterAgent(session_id="s", engine=engine,
prompt_registry=FakePromptRegistry(),
state=WriterState(["db_design"]), max_retries=3)
content = agent.generate_chapter(_ctx_with_lang("db_design", "DB 設計", "ja"))
assert content is not None
assert "本機能" in content.blocks[0].text
def test_generate_chapter_hard_fails_when_violation_persists():
# 始终返回中文(ja 期望),重试耗尽→硬失败
engine = MixedLanguageEngine([
"这是一段纯中文的章节正文内容,应当被判定为语言违规。",
"还是一段纯中文的章节正文内容,依旧违规。",
])
agent = WriterAgent(session_id="s", engine=engine,
prompt_registry=FakePromptRegistry(),
state=WriterState(["db_design"]), max_retries=2)
try:
agent.generate_chapter(_ctx_with_lang("db_design", "DB 設計", "ja"))
assert False, "应抛出 WriterGenerationError"
except WriterGenerationError as e:
assert "语言" in str(e) or "language" in str(e).lower()
def test_generate_chapter_auto_unverifiable_not_enforced():
# auto 且无法推导期望语言(标题纯汉字无假名、无日文 fallback)→ 不强制,直接通过
engine = MixedLanguageEngine(["这是中文正文但 auto 不可验证所以放行。"])
agent = WriterAgent(session_id="s", engine=engine,
prompt_registry=FakePromptRegistry(),
state=WriterState(["db_design"]), max_retries=1)
content = agent.generate_chapter(_ctx_with_lang("db_design", "DB 設計", "auto"))
assert content is not None
+77
View File
@@ -0,0 +1,77 @@
"""中文样本模板镜像测试(步骤 0)。
断言 zh 模板与 ja 模板章节结构一致:7 个 H1 章、锚点 id(section:xxx)逐一对应,
且锚点段落文本原样保留(映射/注入器零改动)。
"""
from __future__ import annotations
import shutil
from pathlib import Path
import pytest
from docx import Document
from genesis.parsers.word_template_parser import WordTemplateParser
from genesis.writer.template_mapper import map_template
_SAMPLES = Path(r"D:\00_project\Genesis\samples")
_JA = _SAMPLES / "概要設計書テンプレート.docx"
_ZH = _SAMPLES / "概要设计书模板_中文.docx"
EXPECTED_ANCHORS = [
"introduction", "function_list", "screen_list", "report_list",
"db_design", "if_definition", "batch_list",
]
@pytest.fixture
def zh_template(tmp_path):
"""生成 zh 模板到临时目录并返回路径(不污染 samples/)。"""
from scripts.make_zh_template import build_zh_template
dst = tmp_path / "概要设计书模板_中文.docx"
build_zh_template(str(_JA), str(dst))
return dst
def _parse(path: Path):
return WordTemplateParser().parse(str(path))
def test_zh_template_has_same_seven_chapter_anchors(zh_template):
ja = _parse(_JA)
zh = _parse(zh_template)
ja_ids = [ph for ph in ja.placeholders if ph.startswith("section:")]
zh_ids = [ph for ph in zh.placeholders if ph.startswith("section:")]
assert sorted(zh_ids) == sorted(ja_ids)
assert zh_ids == [f"section:{a}" for a in EXPECTED_ANCHORS]
def test_zh_template_anchor_paragraphs_unchanged(zh_template):
"""锚点段落文本在 zh 模板中必须与 ja 完全一致(映射/注入器依赖它)。"""
ja_doc = Document(str(_JA))
zh_doc = Document(str(zh_template))
ja_anchors = [p.text for p in ja_doc.paragraphs if p.text.startswith("{{section:")]
zh_anchors = [p.text for p in zh_doc.paragraphs if p.text.startswith("{{section:")]
assert zh_anchors == ja_anchors
def test_zh_template_headings_translated(zh_template):
zh_doc = Document(str(zh_template))
texts = {p.text for p in zh_doc.paragraphs if p.style.name.startswith("Heading")}
assert "1. 前言" in texts
assert "2. 功能一览" in texts
assert "5. DB设计" in texts
assert "7. 批处理一览" in texts
def test_zh_template_maps_to_same_seven_chapters(zh_template):
"""template_mapper 对 zh 模板产出 7 个章节,锚点与 ja 对应。"""
pt = _parse(zh_template)
specs = map_template(pt)
assert len(specs) == 7
assert [s.section_placeholder for s in specs] == [
f"section:{a}" for a in EXPECTED_ANCHORS
]