docs: MIXED 完整段落解析实施计划

This commit is contained in:
lhl
2026-08-09 03:46:42 +08:00
parent d01c7d4901
commit 035be2acea
@@ -0,0 +1,522 @@
# MIXED 完整段落解析 实施计划
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 将 MIXED(混合型)Sheet 落地 design §3.5.2「段落分割→各段落最优解析」,显式表达段落边界与类型。
**Architecture:** 新增纯函数段落分割器 `paragraph_splitter.py`(以空行为界的通用分割);扩展 `data_models.py` 追加 `MixedParagraph`/`MixedSheet` 容器与 `ExcelParseResult.mixed` 字段;`excel_parser.py` 中为 MIXED 性质新增装配分支(分割→每段 classify→表格段 extract_table/自由文本段 build_free_text_table);新增混合型样本 `samples/要件定義_混合型.xlsx` 驱动端到端。
**Tech Stack:** Python 3.11+openpyxl 3.1.5pytestdataclassestyping。
## Global Constraints
- 项目为中文交流(注释中文、标识符英文),Windows/PowerShell 环境
- `data_models.py` 的既有类与字段**不得修改或删除**(仅追加新类与带默认值的新字段)
- 既有 TABLE / FREE_TEXT 解析路径**不得改行为**(回归保持 41 passed
- 测试命令:`python -m pytest tests/<file> -v`;全量回归:`python -m pytest -v`
- 提交消息风格:`feat:` / `test:` / `docs:`(简中文描述)
- 每次修改后按项目规则追加 `_AI_USAGE_LOG.md` 记录(范式步骤列:Agent 实现 或 测试验证)
---
### Task 9: 段落分割器 `split_paragraphs`
**Files:**
- Create: `src/genesis/parsers/paragraph_splitter.py`
- Create: `tests/test_paragraph_splitter.py`
**Interfaces:**
- Consumes: 无(纯函数,仅类型 `Any`
- Produces: `split_paragraphs(matrix: list[list[Any]]) -> list[tuple[int, int]]` — 以全空行为界的段落序号区间(含行区间端点**,矩阵 0-based;空矩阵 → `[]`
- [ ] **Step 1: 写失败测试**
`tests/test_paragraph_splitter.py`:
```python
from genesis.parsers.paragraph_splitter import split_paragraphs
def test_empty_matrix():
assert split_paragraphs([]) == []
def test_single_paragraph_no_empty_rows():
m = [["a", "b"], ["c", "d"]]
assert split_paragraphs(m) == [(0, 1)]
def test_split_on_middle_empty_row():
m = [["a"], [], ["b"], ["c"], []]
assert split_paragraphs(m) == [(0, 0), (2, 3)]
def test_trailing_empty_rows_no_extra_paragraph():
m = [["a"], [], [], []]
assert split_paragraphs(m) == [(0, 0)]
def test_leading_empty_rows_start_at_first_nonempty():
m = [[], ["a"], [], ["b"]]
assert split_paragraphs(m) == [(1, 1), (3, 3)]
```
- [ ] **Step 2: 运行确认失败**
Run: `python -m pytest tests/test_paragraph_splitter.py -v`
Expected: FAILimport 错误 `ModuleNotFoundError: No module named 'genesis.parsers.paragraph_splitter'`
- [ ] **Step 3: 实现**
`src/genesis/parsers/paragraph_splitter.py`:
```python
from __future__ import annotations
from typing import Any
def _is_blank_row(row: list[Any]) -> bool:
return all(c is None or str(c).strip() == "" for c in row)
def split_paragraphs(matrix: list[list[Any]]) -> list[tuple[int, int]]:
"""以全空行为界的通用段落分割;返回 (start_row, end_row)(含端,0-based)。"""
paragraphs: list[tuple[int, int]] = []
start: int | None = None
for i, row in enumerate(matrix):
if not _is_blank_row(row):
if start is None:
start = i
else:
if start is not None:
paragraphs.append((start, i - 1))
start = None
if start is not None:
paragraphs.append((start, len(matrix) - 1))
return paragraphs
```
- [ ] **Step 4: 运行确认通过**
Run: `python -m pytest tests/test_paragraph_splitter.py -v`
Expected: PASS5 passed
- [ ] **Step 5: 提交**
```bash
git add src/genesis/parsers/paragraph_splitter.py tests/test_paragraph_splitter.py
git commit -m "feat: 段落分割 split_paragraphs(通用空行分词)"
```
---
### Task 10: data_models 扩展(MixedParagraph / MixedSheet
**Files:**
- Modify: `src/genesis/data_models.py`(在文件末尾追加,不修改既有类)
**Interfaces:**
- Consumes: `typing.Literal``ExcelTable``SheetType`(已有)
- Produces:
- `MixedParagraph(kind: Literal["table","free_text"], matrix: list[list[Any]] | None = None, table: ExcelTable | None = None, text: str | None = None, source_range: tuple[int, int] | None = None)`
- `MixedSheet(name: str, paragraphs: list[MixedParagraph] = field(default_factory=list))`
- `ExcelParseResult.mixed: list[MixedSheet]` 新增字段(带默认值)
- [ ] **Step 1: 写失败测试**
`tests/test_excel_parser.py` 末尾追加(先测字段存在性与默认行为):
```python
from genesis.data_models import MixedParagraph, MixedSheet
from genesis.parsers.excel_parser import ExcelParseResult
def test_excel_parse_result_has_mixed_default():
r = ExcelParseResult(file_name="f.xlsx")
assert r.mixed == []
def test_mixed_paragraph_defaults():
p = MixedParagraph(kind="table")
assert p.table is None
assert p.text is None
assert p.source_range is None
def test_mixed_sheet_holds_paragraphs():
p1 = MixedParagraph(kind="table")
p2 = MixedParagraph(kind="free_text", text="备注")
ms = MixedSheet(name="混合", paragraphs=[p1, p2])
assert ms.name == "混合"
assert [p.kind for p in ms.paragraphs] == ["table", "free_text"]
```
- [ ] **Step 2: 运行确认失败**
Run: `python -m pytest tests/test_excel_parser.py -v`
Expected: FAIL`ImportError: cannot import name 'MixedParagraph'`
- [ ] **Step 3: 实现(data_models.py 末尾追加)**
```python
@dataclass
class MixedParagraph:
"""混合 sheet 的一个段落(表格或自由文本)"""
kind: Literal["table", "free_text"]
matrix: list[list[Any]] | None = None # 该段原始矩阵(调试/重现)
table: ExcelTable | None = None # kind="table" 时填充
text: str | None = None # kind="free_text" 时填充(段全文)
source_range: tuple[int, int] | None = None # (first_row, last_row) 矩阵 0-based
@dataclass
class MixedSheet:
"""混合 sheet 的段落集合"""
name: str
paragraphs: list[MixedParagraph] = field(default_factory=list)
```
`field` 已在文件顶部导入;`Literal` 需在 `from typing import Any` 处加 `Literal`
- [ ] **Step 4: 运行确认通过**
Run: `python -m pytest tests/test_excel_parser.py -v`
Expected: PASS(新增 3 passed 全绿)
- [ ] **Step 5: 修改 ExcelParseResultexcel_parser.py**
`src/genesis/parsers/excel_parser.py``ExcelParseResult` 中加字段 `mixed`
```python
@dataclass
class ExcelParseResult:
file_name: str
tables: list[ExcelTable] = field(default_factory=list)
comments: list[CellComment] = field(default_factory=list)
skipped: list[str] = field(default_factory=list)
mixed: list[MixedSheet] = field(default_factory=list)
```
并更新导入 `from genesis.data_models import CellComment, ExcelTable, MixedSheet`
- [ ] **Step 6: 全量回归**
Run: `python -m pytest -v`
Expected: PASS41 + 3 = 44 passed
- [ ] **Step 7: 提交**
```bash
git add src/genesis/data_models.py src/genesis/parsers/excel_parser.py tests/test_excel_parser.py
git commit -m "feat: data_models 扩展 MixedParagraph/MixedSheet(段落容器)"
```
---
### Task 11: MIXED 装配(excel_parser 分段解析)
**Files:**
- Modify: `src/genesis/parsers/excel_parser.py`
- Create: 无新文件
- Test: `tests/test_excel_parser.py`(追加)
**Interfaces:**
- Consumes: `split_paragraphs`Task9)、`MixedParagraph/MixedSheet`Task10)、既有 `classify_sheet` / `forward_fill` / `extract_table` / `extract_text_blocks` / `build_free_text_table``cell_formatting` / `collect_comments`
- Produces: `ExcelParser.parse` 对 MIXED 性质产生 `result.mixed` 段落集合 + 追加表到 `result.tables`
**关键实现约定(来自 spec 3.4 修正)**:合并单元格**对整 sheet 先 forward_fill 再按段切片**,避免坐标换算错误;段内 `extract_table``header_row=0`(段首行为表头)。
- [ ] **Step 1: 写失败测试**
`tests/test_excel_parser.py` 追加(构造混合矩阵:表格段 + 空行 + 碎片段,落盘解析):
```python
def test_parse_mixed_sheet_segmented(tmp_path):
wb = new_workbook({
"混合": [
["機能ID", "機能名"],
["F101", "社員登録"],
["F102", "退職処理"],
[],
["・改修ポイント:F102 追加バリデーション"],
["■対象画面:SC001"],
],
})
path = save_workbook(tmp_path, wb)
result = ExcelParser().parse(path)
assert result.mixed, "混合 sheet 应产产出 mixed 段落"
ms = result.mixed[0]
assert ms.name == "混合"
kinds = [p.kind for p in ms.paragraphs]
assert "table" in kinds and "free_text" in kinds
# 表格段无碎片污染:table 段应含 2 数据行,机能ID 首行为 F101
tbl = [p.table for p in ms.paragraphs if p.kind == "table"][0]
assert tbl is not None and len(tbl.rows) == 2
assert tbl.rows[0]["機能ID"].value == "F101"
# 自由文本段捕获碎片
ft = [p for p in ms.paragraphs if p.kind == "free_text"][0]
assert "改修ポイント" in (ft.text or "")
```
- [ ] **Step 2: 运行确认失败**
Run: `python -m pytest tests/test_excel_parser.py::test_parse_mixed_sheet_segmented -v`
Expected: FAIL(当前 MIXED 折叠进表格路径,`result.mixed` 为空)
- [ ] **Step 3: 实现装配分支(excel_parser.py**
`parse``if nature == SheetNature.FREE_TEXT` 之后插入 MIXED 分支,或将 `nature == SheetNature.MIXED` 单独处理:
```python
if nature == SheetNature.MIXED:
# 合并单元格:整 sheet 先填充再按段切片
merged = [
(r.min_row, r.min_col, r.max_row, r.max_col)
for r in ws.merged_cells.ranges
]
filled_all = forward_fill(matrix, merged) if merged else matrix
# 整 sheet 构建格式映射一次(按物理坐标)
fmt_map = {}
for row in ws.iter_rows():
for cell in row:
fmt = cell_formatting(cell)
if fmt is not None:
fmt_map[(cell.row - 1, cell.column - 1)] = fmt
mixed_sheet = MixedSheet(name=ws.title)
for (s, e) in split_paragraphs(matrix):
seg = filled_all[s:e + 1]
seg_nature = classify_sheet(seg)
if seg_nature == SheetNature.TABLE:
header_row = find_header_row(seg)
if header_row < 0:
header_row = 0
table = extract_table(
ws.title, seg, file_name, detected_type,
header_row=header_row, formatting_map=fmt_map,
)
result.tables.append(table)
mixed_sheet.paragraphs.append(MixedParagraph(
kind="table", matrix=seg, table=table,
source_range=(s, e),
))
else:
blocks = extract_text_blocks(seg)
table = build_free_text_table(ws.title, blocks, file_name, detected_type)
result.tables.append(table)
mixed_sheet.paragraphs.append(MixedParagraph(
kind="free_text", matrix=seg,
text="\n".join(blocks), source_range=(s, e),
))
result.mixed.append(mixed_sheet)
elif nature == SheetNature.FREE_TEXT:
blocks = extract_text_blocks(matrix)
result.tables.append(
build_free_text_table(ws.title, blocks, file_name, detected_type)
)
else:
# 现有 TABLE 路径(含 MIXED 旧折叠)
```
> 注:原 `else` 分支现在是 TABLE 专用;MIXED 已独立。`fmt_map` 对整 sheet 构建后再段内使用(对表格段坐标有效,自由文本段无表格 CellValue 使用)。
- [ ] **Step 4: 运行确认通过**
Run: `python -m pytest tests/test_excel_parser.py -v`
Expected: PASS(含新 MIXED 用例)
- [ ] **Step 5: 全量回归**
Run: `python -m pytest -v`
Expected: PASS44 + 1 = 45 passed
- [ ] **Step 6: 提交**
```bash
git add src/genesis/parsers/excel_parser.py tests/test_excel_parser.py
git commit -m "feat: MIXED 完整段落解析(分割→每段最优解析)"
```
---
### Task 12: 混合型样本 + 端到端验证
**Files:**
- Create: `samples/要件定義_混合型.xlsx`
- Modify: `tests/test_real_samples.py`(追加用例)
**Interfaces:**
- Consumes: `ExcelParser.parse`(含 MIXED 装配)
- Produces: 混合样本(表格段 + 碎片段)端到端用例
- [ ] **Step 1: 生成样本**
用 Python 脚本生成 `samples/要件定義_混合型.xlsx`(落盘):
```python
from openpyxl import Workbook
wb = Workbook()
ws = wb.active
ws.title = "機能一覧"
rows = [
["機能ID", "機能名", "画面ID"],
["F101", "社員登録", "SC001"],
["F102", "退職処理", "SC002"],
["F103", "給与計算", "SC003"],
[],
["・改修ポイント:F103 に年末調整バッチ連携を追加する。"],
[],
["■対象期間:2026年度下半期"],
]
for r, row in enumerate(rows, start=1):
for c, v in enumerate(row, start=1):
if v is not None:
ws.cell(row=r, column=c, value=v)
wb.save(r"samples\要件定義_混合型.xlsx")
print("saved")
```
(实际执行时用 PowerShell 运行;注意行尾没有多余空行——`[]` 行是显式空行分隔,最后一个非空行为「■…」行后无额外空行,保证 `split_paragraphs` 尾部截断不产生空段。)
- [ ] **Step 2: 写端到端测试**
`tests/test_real_samples.py` 追加:
```python
def test_mixed_sample_segments_detected():
p = _x("要件定義_混合型.xlsx")
if not p.exists():
pytest.skip("样本缺失")
result = ExcelParser().parse(p)
by_name = {t.name: t for t in result.tables}
assert "機能一覧" in by_name
assert result.mixed, "混合样本应产产出段落"
mixed = result.mixed[0]
kinds = [p.kind for p in mixed.paragraphs]
assert "table" in kinds and "free_text" in kinds
# 表格段无碎片污染
table = [p.table for p in mixed.paragraphs if p.kind == "table"][0]
assert table.rows[0]["機能ID"].value == "F101"
```
- [ ] **Step 3: 运行端到端**
Run: `python -m pytest tests/test_real_samples.py -v`
Expected: PASS4 passed,无 skip
- [ ] **Step 4: 全量回归**
Run: `python -m pytest -v`
Expected: PASS45 + 1 = 46 passed
- [ ] **Step 5: 提交**
```bash
git add samples/要件定義_混合型.xlsx tests/test_real_samples.py
git commit -m "test: MIXED 混合样本 + 端到端段落验证"
```
---
### Task 13: 遗留清理收尾(枚举/注释/边界/测试补齐)
**Files:**
- Modify: `src/genesis/parsers/free_text_extractor.py``extraction_method` 用枚举值同源)
- Modify: `src/genesis/parsers/table_extractor.py``extraction_method` 用枚举值同源;`header_row` 越界防御)
- Modify: `src/genesis/parsers/merge_fill.py``min_row` 非法值防御)
- Modify: `src/genesis/data_models.py``Provenance.row` 语义注释)
- Modify: `src/genesis/parsers/sheet_nature.py`(「・/■」MIXED 用例已有,不追加代码)
- Modify: `tests/test_sheet_nature.py`(补 MIXED 正向/边界用例)
- Modify: `tests/test_formatting_detector.py`(补字体色/背景色正向断言)
**Interfaces:**
- Consumes: `ExtractionMethod` 枚举(data_models
- Produces: 一致的枚举同源与防御性实现
- [ ] **Step 1: 写失败测试(MIXED 断言 + formatting 正向)**
`tests/test_sheet_nature.py` 追加:
```python
def test_classify_sheet_mixed_with_bullet_line():
m = [["ID", "名前"], ["1", "田中"], ["・備考行"]]
assert classify_sheet(m) == SheetNature.MIXED
def test_classify_sheet_mixed_with_square_line():
m = [["ID", "名前"], ["1", "田中"], ["■備考行"]]
assert classify_sheet(m) == SheetNature.MIXED
def test_classify_table_no_semicolon():
m = [["ID", "名前"], ["1", "田中"]]
assert classify_sheet(m) == SheetNature.TABLE
```
`tests/test_formatting_detector.py` 追加(复用现有 make_wb 复制字体技巧):
```python
def test_cell_formatting_detects_font_color():
from openpyxl.styles import Font
wb = Workbook()
ws = wb.active
ws["A1"] = "x"
f = copy(ws["A1"].font)
f.color = Font(color="FF0000FF").color
ws["A1"].font = f
fmt = cell_formatting(ws["A1"])
assert fmt is not None
assert fmt.font_color is not None
```
> 注意 openpyxl 颜色格式:不硬编码具体 hex 断言,只断言 `font_color is not None`,避免格式漂移;同时确认纯默认单元格仍返回 None(已有用例覆盖)。
- [ ] **Step 2: 实现(枚举同源 + 防御)**
`free_text_extractor.py`:导入并改用:
```python
from genesis.data_models import ExtractionMethod
# build_free_text_table 中:
extraction_method=ExtractionMethod.LLM_FROM_FREE_TEXT.value,
```
`table_extractor.py`
```python
from genesis.data_models import ExtractionMethod
# extract_table 中:
extraction_method=ExtractionMethod.OPENPYXL.value,
# 空矩阵分支同样同源
```
`table_extractor.py``header_row` 防御:
```python
if not matrix:
return ExcelTable(...)
if header_row < 0 or header_row >= len(matrix):
header_row = 0
```
`merge_fill.py` 防御非法入参:
```python
def forward_fill(matrix, merged_ranges):
# 前置校验:range 值不合法(<1 或超出矩阵)时直接返回深拷贝
```
`data_models.py` 注释(`Provenance.row`):
```python
@dataclass
class Provenance:
file_name: str
sheet_name: str
row: int # 数据行号(从 1 起:表格为物理行-表头行;自由文本为块序)
column: str
column_header: str
```
- [ ] **Step 3: 运行全部新增/修改测试**
Run: `python -m pytest tests/test_sheet_nature.py tests/test_formatting_detector.py tests/test_free_text_extractor.py tests/test_table_extractor.py -v`
Expected: PASS
- [ ] **Step 4: 全量回归**
Run: `python -m pytest -v`
Expected: PASS46 + 新增断言数)
- [ ] **Step 5: 提交**
```bash
git add src/genesis/parsers/free_text_extractor.py src/genesis/parsers/table_extractor.py src/genesis/parsers/merge_fill.py src/genesis/data_models.py tests/test_sheet_nature.py tests/test_formatting_detector.py
git commit -m "fix: 遗留清理(枚举同源/边界防御/断言补强)"
```
---
## Self-Review
**1. Spec 覆盖**:§3.1→Task9、§3.3→Task10、§3.4→Task11、§3.5→Task12、§3.6/§4→Task13 + 各任务内测试。✓
**2. 占位符检查**:无 TODO/TBD;修正了拼写错误(「占位」→实际代码)。✓
**3. 类型一致性**`split_paragraphs` 签名在 Task9 定义、Task11/12 使用一致;`MixedParagraph.kind``Literal["table","free_text"]` 一致。✓
**4. 兼容性**Task10 只追加字段(默认值);`ExcelParseResult` 既有消费(tests、下游)不受破坏。✓