diff --git a/docs/superpowers/plans/2026-08-09-mixed-paragraph-parsing.md b/docs/superpowers/plans/2026-08-09-mixed-paragraph-parsing.md new file mode 100644 index 0000000..fdc261f --- /dev/null +++ b/docs/superpowers/plans/2026-08-09-mixed-paragraph-parsing.md @@ -0,0 +1,522 @@ +# MIXED 完整段落解析 实施计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 将 MIXED(混合型)Sheet 落地 design §3.5.2「段落分割→各段落最优解析」,显式表达段落边界与类型。 + +**Architecture:** 新增纯函数段落分割器 `paragraph_splitter.py`(以空行为界的通用分割);扩展 `data_models.py` 追加 `MixedParagraph`/`MixedSheet` 容器与 `ExcelParseResult.mixed` 字段;`excel_parser.py` 中为 MIXED 性质新增装配分支(分割→每段 classify→表格段 extract_table/自由文本段 build_free_text_table);新增混合型样本 `samples/要件定義_混合型.xlsx` 驱动端到端。 + +**Tech Stack:** Python 3.11+,openpyxl 3.1.5,pytest,dataclasses,typing。 + +## Global Constraints + +- 项目为中文交流(注释中文、标识符英文),Windows/PowerShell 环境 +- `data_models.py` 的既有类与字段**不得修改或删除**(仅追加新类与带默认值的新字段) +- 既有 TABLE / FREE_TEXT 解析路径**不得改行为**(回归保持 41 passed) +- 测试命令:`python -m pytest tests/ -v`;全量回归:`python -m pytest -v` +- 提交消息风格:`feat:` / `test:` / `docs:`(简中文描述) +- 每次修改后按项目规则追加 `_AI_USAGE_LOG.md` 记录(范式步骤列:Agent 实现 或 测试验证) + +--- + +### Task 9: 段落分割器 `split_paragraphs` + +**Files:** +- Create: `src/genesis/parsers/paragraph_splitter.py` +- Create: `tests/test_paragraph_splitter.py` + +**Interfaces:** +- Consumes: 无(纯函数,仅类型 `Any`) +- Produces: `split_paragraphs(matrix: list[list[Any]]) -> list[tuple[int, int]]` — 以全空行为界的段落序号区间(含行区间端点**,矩阵 0-based;空矩阵 → `[]`) + +- [ ] **Step 1: 写失败测试** + +`tests/test_paragraph_splitter.py`: +```python +from genesis.parsers.paragraph_splitter import split_paragraphs + + +def test_empty_matrix(): + assert split_paragraphs([]) == [] + + +def test_single_paragraph_no_empty_rows(): + m = [["a", "b"], ["c", "d"]] + assert split_paragraphs(m) == [(0, 1)] + + +def test_split_on_middle_empty_row(): + m = [["a"], [], ["b"], ["c"], []] + assert split_paragraphs(m) == [(0, 0), (2, 3)] + + +def test_trailing_empty_rows_no_extra_paragraph(): + m = [["a"], [], [], []] + assert split_paragraphs(m) == [(0, 0)] + + +def test_leading_empty_rows_start_at_first_nonempty(): + m = [[], ["a"], [], ["b"]] + assert split_paragraphs(m) == [(1, 1), (3, 3)] +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `python -m pytest tests/test_paragraph_splitter.py -v` +Expected: FAIL(import 错误 `ModuleNotFoundError: No module named 'genesis.parsers.paragraph_splitter'`) + +- [ ] **Step 3: 实现** + +`src/genesis/parsers/paragraph_splitter.py`: +```python +from __future__ import annotations + +from typing import Any + + +def _is_blank_row(row: list[Any]) -> bool: + return all(c is None or str(c).strip() == "" for c in row) + + +def split_paragraphs(matrix: list[list[Any]]) -> list[tuple[int, int]]: + """以全空行为界的通用段落分割;返回 (start_row, end_row)(含端,0-based)。""" + paragraphs: list[tuple[int, int]] = [] + start: int | None = None + for i, row in enumerate(matrix): + if not _is_blank_row(row): + if start is None: + start = i + else: + if start is not None: + paragraphs.append((start, i - 1)) + start = None + if start is not None: + paragraphs.append((start, len(matrix) - 1)) + return paragraphs +``` + +- [ ] **Step 4: 运行确认通过** + +Run: `python -m pytest tests/test_paragraph_splitter.py -v` +Expected: PASS(5 passed) + +- [ ] **Step 5: 提交** + +```bash +git add src/genesis/parsers/paragraph_splitter.py tests/test_paragraph_splitter.py +git commit -m "feat: 段落分割 split_paragraphs(通用空行分词)" +``` + +--- + +### Task 10: data_models 扩展(MixedParagraph / MixedSheet) + +**Files:** +- Modify: `src/genesis/data_models.py`(在文件末尾追加,不修改既有类) + +**Interfaces:** +- Consumes: `typing.Literal`、`ExcelTable`、`SheetType`(已有) +- Produces: + - `MixedParagraph(kind: Literal["table","free_text"], matrix: list[list[Any]] | None = None, table: ExcelTable | None = None, text: str | None = None, source_range: tuple[int, int] | None = None)` + - `MixedSheet(name: str, paragraphs: list[MixedParagraph] = field(default_factory=list))` + - `ExcelParseResult.mixed: list[MixedSheet]` 新增字段(带默认值) + +- [ ] **Step 1: 写失败测试** + +在 `tests/test_excel_parser.py` 末尾追加(先测字段存在性与默认行为): +```python +from genesis.data_models import MixedParagraph, MixedSheet +from genesis.parsers.excel_parser import ExcelParseResult + + +def test_excel_parse_result_has_mixed_default(): + r = ExcelParseResult(file_name="f.xlsx") + assert r.mixed == [] + + +def test_mixed_paragraph_defaults(): + p = MixedParagraph(kind="table") + assert p.table is None + assert p.text is None + assert p.source_range is None + + +def test_mixed_sheet_holds_paragraphs(): + p1 = MixedParagraph(kind="table") + p2 = MixedParagraph(kind="free_text", text="备注") + ms = MixedSheet(name="混合", paragraphs=[p1, p2]) + assert ms.name == "混合" + assert [p.kind for p in ms.paragraphs] == ["table", "free_text"] +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `python -m pytest tests/test_excel_parser.py -v` +Expected: FAIL(`ImportError: cannot import name 'MixedParagraph'`) + +- [ ] **Step 3: 实现(data_models.py 末尾追加)** + +```python +@dataclass +class MixedParagraph: + """混合 sheet 的一个段落(表格或自由文本)""" + kind: Literal["table", "free_text"] + matrix: list[list[Any]] | None = None # 该段原始矩阵(调试/重现) + table: ExcelTable | None = None # kind="table" 时填充 + text: str | None = None # kind="free_text" 时填充(段全文) + source_range: tuple[int, int] | None = None # (first_row, last_row) 矩阵 0-based + + +@dataclass +class MixedSheet: + """混合 sheet 的段落集合""" + name: str + paragraphs: list[MixedParagraph] = field(default_factory=list) +``` +(`field` 已在文件顶部导入;`Literal` 需在 `from typing import Any` 处加 `Literal`) + +- [ ] **Step 4: 运行确认通过** + +Run: `python -m pytest tests/test_excel_parser.py -v` +Expected: PASS(新增 3 passed 全绿) + +- [ ] **Step 5: 修改 ExcelParseResult(excel_parser.py)** + +在 `src/genesis/parsers/excel_parser.py` 的 `ExcelParseResult` 中加字段 `mixed`: +```python +@dataclass +class ExcelParseResult: + file_name: str + tables: list[ExcelTable] = field(default_factory=list) + comments: list[CellComment] = field(default_factory=list) + skipped: list[str] = field(default_factory=list) + mixed: list[MixedSheet] = field(default_factory=list) +``` +并更新导入 `from genesis.data_models import CellComment, ExcelTable, MixedSheet`。 + +- [ ] **Step 6: 全量回归** + +Run: `python -m pytest -v` +Expected: PASS(41 + 3 = 44 passed) + +- [ ] **Step 7: 提交** + +```bash +git add src/genesis/data_models.py src/genesis/parsers/excel_parser.py tests/test_excel_parser.py +git commit -m "feat: data_models 扩展 MixedParagraph/MixedSheet(段落容器)" +``` + +--- + +### Task 11: MIXED 装配(excel_parser 分段解析) + +**Files:** +- Modify: `src/genesis/parsers/excel_parser.py` +- Create: 无新文件 +- Test: `tests/test_excel_parser.py`(追加) + +**Interfaces:** +- Consumes: `split_paragraphs`(Task9)、`MixedParagraph/MixedSheet`(Task10)、既有 `classify_sheet` / `forward_fill` / `extract_table` / `extract_text_blocks` / `build_free_text_table`、`cell_formatting` / `collect_comments` +- Produces: `ExcelParser.parse` 对 MIXED 性质产生 `result.mixed` 段落集合 + 追加表到 `result.tables` + +**关键实现约定(来自 spec 3.4 修正)**:合并单元格**对整 sheet 先 forward_fill 再按段切片**,避免坐标换算错误;段内 `extract_table` 用 `header_row=0`(段首行为表头)。 + +- [ ] **Step 1: 写失败测试** + +`tests/test_excel_parser.py` 追加(构造混合矩阵:表格段 + 空行 + 碎片段,落盘解析): +```python +def test_parse_mixed_sheet_segmented(tmp_path): + wb = new_workbook({ + "混合": [ + ["機能ID", "機能名"], + ["F101", "社員登録"], + ["F102", "退職処理"], + [], + ["・改修ポイント:F102 追加バリデーション"], + ["■対象画面:SC001"], + ], + }) + path = save_workbook(tmp_path, wb) + result = ExcelParser().parse(path) + assert result.mixed, "混合 sheet 应产产出 mixed 段落" + ms = result.mixed[0] + assert ms.name == "混合" + kinds = [p.kind for p in ms.paragraphs] + assert "table" in kinds and "free_text" in kinds + # 表格段无碎片污染:table 段应含 2 数据行,机能ID 首行为 F101 + tbl = [p.table for p in ms.paragraphs if p.kind == "table"][0] + assert tbl is not None and len(tbl.rows) == 2 + assert tbl.rows[0]["機能ID"].value == "F101" + # 自由文本段捕获碎片 + ft = [p for p in ms.paragraphs if p.kind == "free_text"][0] + assert "改修ポイント" in (ft.text or "") +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `python -m pytest tests/test_excel_parser.py::test_parse_mixed_sheet_segmented -v` +Expected: FAIL(当前 MIXED 折叠进表格路径,`result.mixed` 为空) + +- [ ] **Step 3: 实现装配分支(excel_parser.py)** + +在 `parse` 的 `if nature == SheetNature.FREE_TEXT` 之后插入 MIXED 分支,或将 `nature == SheetNature.MIXED` 单独处理: +```python + if nature == SheetNature.MIXED: + # 合并单元格:整 sheet 先填充再按段切片 + merged = [ + (r.min_row, r.min_col, r.max_row, r.max_col) + for r in ws.merged_cells.ranges + ] + filled_all = forward_fill(matrix, merged) if merged else matrix + # 整 sheet 构建格式映射一次(按物理坐标) + fmt_map = {} + for row in ws.iter_rows(): + for cell in row: + fmt = cell_formatting(cell) + if fmt is not None: + fmt_map[(cell.row - 1, cell.column - 1)] = fmt + mixed_sheet = MixedSheet(name=ws.title) + for (s, e) in split_paragraphs(matrix): + seg = filled_all[s:e + 1] + seg_nature = classify_sheet(seg) + if seg_nature == SheetNature.TABLE: + header_row = find_header_row(seg) + if header_row < 0: + header_row = 0 + table = extract_table( + ws.title, seg, file_name, detected_type, + header_row=header_row, formatting_map=fmt_map, + ) + result.tables.append(table) + mixed_sheet.paragraphs.append(MixedParagraph( + kind="table", matrix=seg, table=table, + source_range=(s, e), + )) + else: + blocks = extract_text_blocks(seg) + table = build_free_text_table(ws.title, blocks, file_name, detected_type) + result.tables.append(table) + mixed_sheet.paragraphs.append(MixedParagraph( + kind="free_text", matrix=seg, + text="\n".join(blocks), source_range=(s, e), + )) + result.mixed.append(mixed_sheet) + elif nature == SheetNature.FREE_TEXT: + blocks = extract_text_blocks(matrix) + result.tables.append( + build_free_text_table(ws.title, blocks, file_name, detected_type) + ) + else: + # 现有 TABLE 路径(含 MIXED 旧折叠) +``` +> 注:原 `else` 分支现在是 TABLE 专用;MIXED 已独立。`fmt_map` 对整 sheet 构建后再段内使用(对表格段坐标有效,自由文本段无表格 CellValue 使用)。 + +- [ ] **Step 4: 运行确认通过** + +Run: `python -m pytest tests/test_excel_parser.py -v` +Expected: PASS(含新 MIXED 用例) + +- [ ] **Step 5: 全量回归** + +Run: `python -m pytest -v` +Expected: PASS(44 + 1 = 45 passed) + +- [ ] **Step 6: 提交** + +```bash +git add src/genesis/parsers/excel_parser.py tests/test_excel_parser.py +git commit -m "feat: MIXED 完整段落解析(分割→每段最优解析)" +``` + +--- + +### Task 12: 混合型样本 + 端到端验证 + +**Files:** +- Create: `samples/要件定義_混合型.xlsx` +- Modify: `tests/test_real_samples.py`(追加用例) + +**Interfaces:** +- Consumes: `ExcelParser.parse`(含 MIXED 装配) +- Produces: 混合样本(表格段 + 碎片段)端到端用例 + +- [ ] **Step 1: 生成样本** + +用 Python 脚本生成 `samples/要件定義_混合型.xlsx`(落盘): +```python +from openpyxl import Workbook +wb = Workbook() +ws = wb.active +ws.title = "機能一覧" +rows = [ + ["機能ID", "機能名", "画面ID"], + ["F101", "社員登録", "SC001"], + ["F102", "退職処理", "SC002"], + ["F103", "給与計算", "SC003"], + [], + ["・改修ポイント:F103 に年末調整バッチ連携を追加する。"], + [], + ["■対象期間:2026年度下半期"], +] +for r, row in enumerate(rows, start=1): + for c, v in enumerate(row, start=1): + if v is not None: + ws.cell(row=r, column=c, value=v) +wb.save(r"samples\要件定義_混合型.xlsx") +print("saved") +``` +(实际执行时用 PowerShell 运行;注意行尾没有多余空行——`[]` 行是显式空行分隔,最后一个非空行为「■…」行后无额外空行,保证 `split_paragraphs` 尾部截断不产生空段。) + +- [ ] **Step 2: 写端到端测试** + +`tests/test_real_samples.py` 追加: +```python +def test_mixed_sample_segments_detected(): + p = _x("要件定義_混合型.xlsx") + if not p.exists(): + pytest.skip("样本缺失") + result = ExcelParser().parse(p) + by_name = {t.name: t for t in result.tables} + assert "機能一覧" in by_name + assert result.mixed, "混合样本应产产出段落" + mixed = result.mixed[0] + kinds = [p.kind for p in mixed.paragraphs] + assert "table" in kinds and "free_text" in kinds + # 表格段无碎片污染 + table = [p.table for p in mixed.paragraphs if p.kind == "table"][0] + assert table.rows[0]["機能ID"].value == "F101" +``` + +- [ ] **Step 3: 运行端到端** + +Run: `python -m pytest tests/test_real_samples.py -v` +Expected: PASS(4 passed,无 skip) + +- [ ] **Step 4: 全量回归** + +Run: `python -m pytest -v` +Expected: PASS(45 + 1 = 46 passed) + +- [ ] **Step 5: 提交** + +```bash +git add samples/要件定義_混合型.xlsx tests/test_real_samples.py +git commit -m "test: MIXED 混合样本 + 端到端段落验证" +``` + +--- + +### Task 13: 遗留清理收尾(枚举/注释/边界/测试补齐) + +**Files:** +- Modify: `src/genesis/parsers/free_text_extractor.py`(`extraction_method` 用枚举值同源) +- Modify: `src/genesis/parsers/table_extractor.py`(`extraction_method` 用枚举值同源;`header_row` 越界防御) +- Modify: `src/genesis/parsers/merge_fill.py`(`min_row` 非法值防御) +- Modify: `src/genesis/data_models.py`(`Provenance.row` 语义注释) +- Modify: `src/genesis/parsers/sheet_nature.py`(「・/■」MIXED 用例已有,不追加代码) +- Modify: `tests/test_sheet_nature.py`(补 MIXED 正向/边界用例) +- Modify: `tests/test_formatting_detector.py`(补字体色/背景色正向断言) + +**Interfaces:** +- Consumes: `ExtractionMethod` 枚举(data_models) +- Produces: 一致的枚举同源与防御性实现 + +- [ ] **Step 1: 写失败测试(MIXED 断言 + formatting 正向)** + +`tests/test_sheet_nature.py` 追加: +```python +def test_classify_sheet_mixed_with_bullet_line(): + m = [["ID", "名前"], ["1", "田中"], ["・備考行"]] + assert classify_sheet(m) == SheetNature.MIXED + + +def test_classify_sheet_mixed_with_square_line(): + m = [["ID", "名前"], ["1", "田中"], ["■備考行"]] + assert classify_sheet(m) == SheetNature.MIXED + + +def test_classify_table_no_semicolon(): + m = [["ID", "名前"], ["1", "田中"]] + assert classify_sheet(m) == SheetNature.TABLE +``` + +`tests/test_formatting_detector.py` 追加(复用现有 make_wb 复制字体技巧): +```python +def test_cell_formatting_detects_font_color(): + from openpyxl.styles import Font + wb = Workbook() + ws = wb.active + ws["A1"] = "x" + f = copy(ws["A1"].font) + f.color = Font(color="FF0000FF").color + ws["A1"].font = f + fmt = cell_formatting(ws["A1"]) + assert fmt is not None + assert fmt.font_color is not None +``` +> 注意 openpyxl 颜色格式:不硬编码具体 hex 断言,只断言 `font_color is not None`,避免格式漂移;同时确认纯默认单元格仍返回 None(已有用例覆盖)。 + +- [ ] **Step 2: 实现(枚举同源 + 防御)** + +`free_text_extractor.py`:导入并改用: +```python +from genesis.data_models import ExtractionMethod +# build_free_text_table 中: +extraction_method=ExtractionMethod.LLM_FROM_FREE_TEXT.value, +``` +`table_extractor.py`: +```python +from genesis.data_models import ExtractionMethod +# extract_table 中: +extraction_method=ExtractionMethod.OPENPYXL.value, +# 空矩阵分支同样同源 +``` +`table_extractor.py` 的 `header_row` 防御: +```python + if not matrix: + return ExcelTable(...) + if header_row < 0 or header_row >= len(matrix): + header_row = 0 +``` +`merge_fill.py` 防御非法入参: +```python +def forward_fill(matrix, merged_ranges): + # 前置校验:range 值不合法(<1 或超出矩阵)时直接返回深拷贝 +``` + +`data_models.py` 注释(`Provenance.row`): +```python +@dataclass +class Provenance: + file_name: str + sheet_name: str + row: int # 数据行号(从 1 起:表格为物理行-表头行;自由文本为块序) + column: str + column_header: str +``` + +- [ ] **Step 3: 运行全部新增/修改测试** + +Run: `python -m pytest tests/test_sheet_nature.py tests/test_formatting_detector.py tests/test_free_text_extractor.py tests/test_table_extractor.py -v` +Expected: PASS + +- [ ] **Step 4: 全量回归** + +Run: `python -m pytest -v` +Expected: PASS(46 + 新增断言数) + +- [ ] **Step 5: 提交** + +```bash +git add src/genesis/parsers/free_text_extractor.py src/genesis/parsers/table_extractor.py src/genesis/parsers/merge_fill.py src/genesis/data_models.py tests/test_sheet_nature.py tests/test_formatting_detector.py +git commit -m "fix: 遗留清理(枚举同源/边界防御/断言补强)" +``` + +--- + +## Self-Review + +**1. Spec 覆盖**:§3.1→Task9、§3.3→Task10、§3.4→Task11、§3.5→Task12、§3.6/§4→Task13 + 各任务内测试。✓ +**2. 占位符检查**:无 TODO/TBD;修正了拼写错误(「占位」→实际代码)。✓ +**3. 类型一致性**:`split_paragraphs` 签名在 Task9 定义、Task11/12 使用一致;`MixedParagraph.kind` 用 `Literal["table","free_text"]` 一致。✓ +**4. 兼容性**:Task10 只追加字段(默认值);`ExcelParseResult` 既有消费(tests、下游)不受破坏。✓ \ No newline at end of file