991 lines
31 KiB
Markdown
991 lines
31 KiB
Markdown
# Phase1 里程碑2 Parser Agent - Excel 解析 实施计划
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 实现 ExcelParser，将要件定义 Excel（3 种模式：表格型/自由记述型/混合型）解析为 `ExcelParseResult`（tables + comments），为后代 Agent 提供唯一数据源。
|
||
|
||
**Architecture:** 在已完成的 `data_models.py` 基础上，按 design §3.5 拆分为职责单一模块并由 `ExcelParser` 编排。自由记述型的 LLM 结构化依赖 InferenceEngine（后续里程碑），本里程碑交付可独立测试的文本分段与占位表（`extraction_method="llm_from_free_text"`）。
|
||
|
||
**Tech Stack:** Python ≥3.11、openpyxl ≥3.1、pytest 8。
|
||
|
||
## Global Constraints
|
||
|
||
- 仅读取 `.xlsx`；`.xls` 由 `excel_reader.open_workbook` 抛出 `ValueError`。
|
||
- Sheet 类型判定关键词为日文（design §3.5.1），用于匹配真实日文要件定义（技术必要保留）。
|
||
- 每个单元格生成 `Provenance`；`source_uri` 格式 `file.xlsx#SheetName!CellRef`（design §9.4.5，`build_source_uri` 提供）。
|
||
- 合并单元格下行填充（forward_fill，design §3.5.3）。
|
||
- 取消线/背景色/批注保留在 `CellFormatting` / `CellComment`（design §3.5.4 / §3.5.5）。
|
||
- 每个单元格使用 `data_models` 的 `CellValue`（字段 `value` / `provenance` / `formatting` / `comment`）。
|
||
- 交流语言统一中文（注释/提交信息/本计划正文）；标识符与技术名保留英文/日文关键词。
|
||
- 每个任务结束前 `pytest` 全绿并提交。
|
||
|
||
---
|
||
|
||
### Task 1: ExcelParser 基础结构与来源标注
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/__init__.py`
|
||
- Create: `src/genesis/parsers/provenance.py`
|
||
- Create: `src/genesis/parsers/excel_reader.py`
|
||
- Create: `tests/excel_helpers.py`
|
||
- Create: `tests/test_excel_reader.py`
|
||
|
||
**Interfaces:**
|
||
- Produces:
|
||
- `genesis.parsers.provenance.build_source_uri(file_name: str, sheet_name: str, cell_ref: str) -> str`
|
||
- `genesis.parsers.excel_reader.open_workbook(path: str | Path)` → `openpyxl.Workbook`（仅 `.xlsx`）
|
||
- `genesis.parsers.excel_reader.sheet_matrix(ws) -> list[list[Any]]`
|
||
|
||
测试辅助 `tests/excel_helpers.py` 供全部任务复用。
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/excel_helpers.py`:
|
||
```python
|
||
from openpyxl import Workbook
|
||
|
||
|
||
def new_workbook(sheets: dict[str, list[list]]) -> Workbook:
|
||
"""生成临时 Workbook：key=Sheet 名，value=grid（cell 值）。"""
|
||
wb = Workbook()
|
||
wb.remove(wb.active)
|
||
for name, grid in sheets.items():
|
||
ws = wb.create_sheet(name)
|
||
for r, row in enumerate(grid, start=1):
|
||
for c, value in enumerate(row, start=1):
|
||
ws.cell(row=r, column=c, value=value)
|
||
return wb
|
||
|
||
|
||
def save_workbook(tmp_path, wb: Workbook) -> str:
|
||
"""落盘到 tmp_path 并返回路径字符串。"""
|
||
path = tmp_path / "source.xlsx"
|
||
wb.save(path)
|
||
return str(path)
|
||
```
|
||
|
||
`tests/test_excel_reader.py`:
|
||
```python
|
||
import pytest
|
||
|
||
from genesis.parsers.excel_reader import open_workbook, sheet_matrix
|
||
from genesis.parsers.provenance import build_source_uri
|
||
|
||
from tests.excel_helpers import new_workbook, save_workbook
|
||
|
||
|
||
def test_build_source_uri_format():
|
||
assert build_source_uri("要求.xlsx", "機能一覧", "A5") == "要求.xlsx#機能一覧!A5"
|
||
|
||
|
||
def test_open_workbook_and_sheet_matrix(tmp_path):
|
||
wb = new_workbook({"機能一覧": [["機能ID", "機能名"], ["F001", "社員登録"]]})
|
||
path = save_workbook(tmp_path, wb)
|
||
ws = open_workbook(path)["機能一覧"]
|
||
assert sheet_matrix(ws) == [["機能ID", "機能名"], ["F001", "社員登録"]]
|
||
|
||
|
||
def test_open_workbook_rejects_xls(tmp_path):
|
||
bad = tmp_path / "old.xls"
|
||
bad.write_bytes(b"not really xls")
|
||
with pytest.raises(ValueError):
|
||
open_workbook(bad)
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_excel_reader.py -v`
|
||
Expected: FAIL（`ModuleNotFoundError: No module named 'genesis.parsers'`）
|
||
|
||
- [ ] **Step 3: 实现基础模块**
|
||
|
||
`src/genesis/parsers/__init__.py`:
|
||
```python
|
||
"""Parser Agent：输入资料解析层。"""
|
||
```
|
||
|
||
`src/genesis/parsers/provenance.py`:
|
||
```python
|
||
def build_source_uri(file_name: str, sheet_name: str, cell_ref: str) -> str:
|
||
"""单元格来源 URI：file.xlsx#SheetName!CellRef"""
|
||
return f"{file_name}#{sheet_name}!{cell_ref}"
|
||
```
|
||
|
||
`src/genesis/parsers/excel_reader.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
from openpyxl import load_workbook
|
||
from openpyxl.worksheet.worksheet import Worksheet
|
||
|
||
|
||
def open_workbook(path: str | Path):
|
||
"""普通模式打开 .xlsx（保留公式/样式/批注），.xls 报错。"""
|
||
path = Path(path)
|
||
if path.suffix.lower() != ".xlsx":
|
||
raise ValueError(f"不支持的 Excel 格式: {path.suffix}")
|
||
return load_workbook(path)
|
||
|
||
|
||
def sheet_matrix(ws: Worksheet) -> list[list[Any]]:
|
||
"""整表矩形值（含 None），保留到 max_column。"""
|
||
return [[cell.value for cell in row] for row in ws.iter_rows()]
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_excel_reader.py -v`
|
||
Expected: PASS（3 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers tests/excel_helpers.py tests/test_excel_reader.py
|
||
git commit -m "feat: ExcelParser 基础（workbook/sheet 读取 + source_uri）"
|
||
```
|
||
|
||
> 注：计划中 Step 3 与 Step 4 之间未设独立 Step；Step 3 写完后直接执行 Step 4 运行验证。
|
||
|
||
---
|
||
|
||
### Task 2: SheetDetector（自动识别 Sheet 类型）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/sheet_detector.py`
|
||
- Create: `tests/test_sheet_detector.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `data_models.SheetType`
|
||
- Produces: `detect_sheet_type(sheet_name: str, matrix: list[list[Any]]) -> SheetType`
|
||
（优先级：Sheet 名关键词 → 表头关键词 → `GENERIC`）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_sheet_detector.py`:
|
||
```python
|
||
from genesis.data_models import SheetType
|
||
from genesis.parsers.sheet_detector import detect_sheet_type
|
||
|
||
|
||
def test_detect_by_sheet_name():
|
||
assert detect_sheet_type("機能一覧", []) == SheetType.FUNCTION
|
||
assert detect_sheet_type("画面一覧", []) == SheetType.SCREEN
|
||
assert detect_sheet_type("帳票一覧", []) == SheetType.REPORT
|
||
assert detect_sheet_type("DB定義", []) == SheetType.DATABASE
|
||
assert detect_sheet_type("IF定義", []) == SheetType.INTERFACE
|
||
assert detect_sheet_type("バッチ一覧", []) == SheetType.BATCH
|
||
assert detect_sheet_type("コード管理", []) == SheetType.MASTER
|
||
|
||
|
||
def test_detect_by_header_keyword():
|
||
matrix = [["帳票ID", "帳票名"], ["TB1", "月次"]]
|
||
assert detect_sheet_type("補充シート", matrix) == SheetType.REPORT
|
||
|
||
|
||
def test_unknown_is_generic():
|
||
assert detect_sheet_type("メモ", [["随筆"]]) == SheetType.GENERIC
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_sheet_detector.py -v`
|
||
Expected: FAIL（`ImportError: cannot import name 'detect_sheet_type'`）
|
||
|
||
- [ ] **Step 3: 实现 sheet_detector.py**
|
||
|
||
`src/genesis/parsers/sheet_detector.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from typing import Any
|
||
|
||
from genesis.data_models import SheetType
|
||
|
||
# Sheet 名关键词（design §3.5.1，顺序即优先级）
|
||
NAME_KEYWORDS: list[tuple[str, SheetType]] = [
|
||
("機能", SheetType.FUNCTION),
|
||
("画面", SheetType.SCREEN),
|
||
("帳票", SheetType.REPORT),
|
||
("テーブル", SheetType.DATABASE),
|
||
("DB", SheetType.DATABASE),
|
||
("インターフェース", SheetType.INTERFACE),
|
||
("IF", SheetType.INTERFACE),
|
||
("バッチ", SheetType.BATCH),
|
||
("ジョブ", SheetType.BATCH),
|
||
("マスタ", SheetType.MASTER),
|
||
]
|
||
|
||
# 表头关键词
|
||
HEADER_KEYWORDS: list[tuple[str, SheetType]] = [
|
||
("機能ID", SheetType.FUNCTION),
|
||
("画面ID", SheetType.SCREEN),
|
||
("帳票ID", SheetType.REPORT),
|
||
("テーブルID", SheetType.DATABASE),
|
||
("IF名", SheetType.INTERFACE),
|
||
("バッチID", SheetType.BATCH),
|
||
]
|
||
|
||
|
||
def _name_hit(sheet_name: str) -> SheetType | None:
|
||
for kw, st in NAME_KEYWORDS:
|
||
if kw in sheet_name:
|
||
return st
|
||
return None
|
||
|
||
|
||
def _header_hit(matrix: list[list[Any]]) -> SheetType | None:
|
||
for row in matrix[:3]:
|
||
for cell in row:
|
||
if isinstance(cell, str):
|
||
for kw, st in HEADER_KEYWORDS:
|
||
if kw in cell:
|
||
return st
|
||
return None
|
||
|
||
|
||
def detect_sheet_type(sheet_name: str, matrix: list[list[Any]]) -> SheetType:
|
||
return _name_hit(sheet_name) or _header_hit(matrix) or SheetType.GENERIC
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_sheet_detector.py -v`
|
||
Expected: PASS（3 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/sheet_detector.py tests/test_sheet_detector.py
|
||
git commit -m "feat: SheetDetector 类型识别（名称/表头关键词）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Sheet 性质判定（表格型/自由记述型/混合型）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/sheet_nature.py`
|
||
- Create: `tests/test_sheet_nature.py`
|
||
|
||
**Interfaces:**
|
||
- Produces:
|
||
- `SheetNature(Enum)`：`TABLE` / `FREE_TEXT` / `MIXED`
|
||
- `classify_sheet(matrix: list[list[Any]]) -> SheetNature`
|
||
- `find_header_row(matrix: list[list[Any]]) -> int`（首个连续非空 ≥2 的行的下标；无则 -1）
|
||
|
||
判定启发式（design §3.5.2 落地）：
|
||
- 空行占比 > 30%（`非空行数 / 总行数 < 0.7`）→ FREE_TEXT
|
||
- 仅 A 列使用（`max_cols <= 1`）→ FREE_TEXT
|
||
- 无表头行（`find_header_row == -1`）→ FREE_TEXT
|
||
- 表头行后存在以「・」/「■」开头的碎片行 → MIXED
|
||
- 其余 → TABLE
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_sheet_nature.py`:
|
||
```python
|
||
from genesis.parsers.sheet_nature import SheetNature, classify_sheet, find_header_row
|
||
|
||
|
||
def test_table_detection():
|
||
m = [["機能ID", "機能名"], ["A001", "社員登録"], ["A002", "退職処理"]]
|
||
assert classify_sheet(m) == SheetNature.TABLE
|
||
|
||
|
||
def test_free_text_single_col():
|
||
m = [["新入社員を登録できる。"], ["氏名・所属・入社日を入力する。"]]
|
||
assert classify_sheet(m) == SheetNature.FREE_TEXT
|
||
|
||
|
||
def test_free_text_many_empty_rows():
|
||
m = [["要求A"], [], [], [], ["要求B"]]
|
||
assert classify_sheet(m) == SheetNature.FREE_TEXT
|
||
|
||
|
||
def test_header_row_index():
|
||
m = [["機能ID", "名前"], ["1", "田中"]]
|
||
assert find_header_row(m) == 0
|
||
assert find_header_row([["自由テキスト"]]) == -1
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_sheet_nature.py -v`
|
||
Expected: FAIL（导入错误）
|
||
|
||
- [ ] **Step 3: 实现 sheet_nature.py**
|
||
|
||
`src/genesis/parsers/sheet_nature.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from enum import Enum
|
||
from typing import Any
|
||
|
||
|
||
class SheetNature(Enum):
|
||
TABLE = "table"
|
||
FREE_TEXT = "free_text"
|
||
MIXED = "mixed"
|
||
|
||
|
||
def _non_empty(row: list[Any]) -> list[Any]:
|
||
return [c for c in row if c is not None and str(c).strip() != ""]
|
||
|
||
|
||
def find_header_row(matrix: list[list[Any]]) -> int:
|
||
for i, row in enumerate(matrix):
|
||
if len(_non_empty(row)) >= 2:
|
||
return i
|
||
return -1
|
||
|
||
|
||
def _free_text_like(matrix: list[list[Any]]) -> bool:
|
||
if not matrix:
|
||
return True
|
||
max_cols = max((len(row) for row in matrix), default=0)
|
||
if max_cols <= 1:
|
||
return True
|
||
non_empty_rows = [r for r in matrix if _non_empty(r)]
|
||
if len(non_empty_rows) / len(matrix) < 0.7:
|
||
return True
|
||
return find_header_row(matrix) == -1
|
||
|
||
|
||
def classify_sheet(matrix: list[list[Any]]) -> SheetNature:
|
||
if _free_text_like(matrix):
|
||
return SheetNature.FREE_TEXT
|
||
header_row = find_header_row(matrix)
|
||
if header_row >= 0:
|
||
for row in matrix[header_row + 1:]:
|
||
if any(str(c).strip().startswith(("・", "■")) for c in _non_empty(row)):
|
||
return SheetNature.MIXED
|
||
return SheetNature.TABLE
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_sheet_nature.py -v`
|
||
Expected: PASS（4 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/sheet_nature.py tests/test_sheet_nature.py
|
||
git commit -m "feat: Sheet 性质判定（table/free_text/mixed）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: MergeHandler 与 TableExtractor
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/merge_fill.py`
|
||
- Create: `src/genesis/parsers/table_extractor.py`
|
||
- Create: `tests/test_table_extractor.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `data_models`（`CellValue`、`ExcelTable`、`Provenance`、`SheetType`）、`build_source_uri`
|
||
- Produces:
|
||
- `merge_fill.forward_fill(matrix, merged_ranges) -> list[list[Any]]`
|
||
- `table_extractor.extract_table(sheet_name, matrix, file_name, detected_type) -> ExcelTable`
|
||
- `table_extractor.column_letter(index) -> str`（1→A、27→AA）
|
||
|
||
合并单元格展开：范围 `(min_row, min_col, max_row, max_col)`（1-based）；将范围内全部单元格填为主格（左上角）值。
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_table_extractor.py`:
|
||
```python
|
||
from genesis.data_models import CellValue, ExcelTable, SheetType
|
||
from genesis.parsers.merge_fill import forward_fill
|
||
from genesis.parsers.table_extractor import column_letter, extract_table
|
||
|
||
|
||
def test_forward_fill_vertical():
|
||
matrix = [
|
||
["機能ID", "機能名", "備考"],
|
||
["A001", "", ""],
|
||
["", "", "メモ"],
|
||
]
|
||
filled = forward_fill(matrix, [(1, 2, 2, 2)]) # B1:B2 纵向合并
|
||
assert filled[1][1] == "機能名"
|
||
assert filled[2][2] == "メモ"
|
||
|
||
|
||
def test_forward_fill_horizontal():
|
||
matrix = [["A", "B"], ["x", ""]]
|
||
filled = forward_fill(matrix, [(2, 1, 2, 2)]) # A2:B2 横向合并
|
||
assert filled[1][1] == "x"
|
||
assert filled[1][0] == "x"
|
||
|
||
|
||
def test_column_letter():
|
||
assert column_letter(1) == "A"
|
||
assert column_letter(27) == "AA"
|
||
|
||
|
||
def test_extract_table_basic():
|
||
matrix = [["ID", "名前"], ["1", "田中"], ["2", "佐藤"]]
|
||
table = extract_table("社員一覧", matrix, "f.xlsx", SheetType.FUNCTION)
|
||
assert isinstance(table, ExcelTable)
|
||
assert table.headers == ["ID", "名前"]
|
||
assert table.extraction_method == "openpyxl"
|
||
assert len(table.rows) == 2
|
||
first = table.rows[0]
|
||
assert isinstance(first["ID"], CellValue)
|
||
assert first["ID"].value == "1"
|
||
assert first["ID"].provenance.sheet_name == "社員一覧"
|
||
assert first["ID"].provenance.column == "A"
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_table_extractor.py -v`
|
||
Expected: FAIL（导入错误）
|
||
|
||
- [ ] **Step 3: 实现**
|
||
|
||
`src/genesis/parsers/merge_fill.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from typing import Any
|
||
|
||
|
||
def forward_fill(
|
||
matrix: list[list[Any]],
|
||
merged_ranges: list[tuple[int, int, int, int]],
|
||
) -> list[list[Any]]:
|
||
"""合并单元格：用左上角主格值填充范围内全部单元格。"""
|
||
out = [list(row) for row in matrix]
|
||
for (min_row, min_col, max_row, max_col) in merged_ranges:
|
||
if not out or min_row > len(out) or min_col > len(out[min_row - 1]):
|
||
continue
|
||
main_value = out[min_row - 1][min_col - 1]
|
||
for r in range(min_row, min(max_row, len(out)) + 1):
|
||
row = out[r - 1]
|
||
for c in range(min_col, min(max_col, len(row)) + 1):
|
||
row[c - 1] = main_value
|
||
return out
|
||
```
|
||
|
||
`src/genesis/parsers/table_extractor.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from typing import Any
|
||
|
||
from genesis.data_models import CellValue, ExcelTable, Provenance, SheetType
|
||
|
||
|
||
def column_letter(index: int) -> str:
|
||
"""1 → A、27 → AA。"""
|
||
letters = ""
|
||
while index > 0:
|
||
index, rem = divmod(index - 1, 26)
|
||
letters = chr(65 + rem) + letters
|
||
return letters
|
||
|
||
|
||
def extract_table(
|
||
sheet_name: str,
|
||
matrix: list[list[Any]],
|
||
file_name: str,
|
||
detected_type: SheetType,
|
||
header_row: int = 0,
|
||
) -> ExcelTable:
|
||
"""从矩阵提取表格：首行视为表头，其后为数据行。"""
|
||
if not matrix:
|
||
return ExcelTable(
|
||
name=sheet_name, detected_type=detected_type,
|
||
extraction_method="openpyxl", headers=[], rows=[],
|
||
)
|
||
headers = [str(c) if c is not None else "" for c in matrix[header_row]]
|
||
rows = []
|
||
for r in range(header_row + 1, len(matrix)):
|
||
row_dict = {}
|
||
for c, h in enumerate(headers):
|
||
raw = matrix[r][c] if c < len(matrix[r]) else None
|
||
row_dict[h] = CellValue(
|
||
value=raw,
|
||
provenance=Provenance(
|
||
file_name=file_name,
|
||
sheet_name=sheet_name,
|
||
row=r - header_row, # 数据行号从 1 开始
|
||
column=column_letter(c + 1),
|
||
column_header=h,
|
||
),
|
||
)
|
||
rows.append(row_dict)
|
||
return ExcelTable(
|
||
name=sheet_name, detected_type=detected_type,
|
||
extraction_method="openpyxl", headers=headers, rows=rows,
|
||
)
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_table_extractor.py -v`
|
||
Expected: PASS
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/merge_fill.py src/genesis/parsers/table_extractor.py tests/test_table_extractor.py
|
||
git commit -m "feat: MergeHandler + TableExtractor（forward_fill / 表格提取）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: FormattingDetector（取消线/背景色/批注）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/formatting_detector.py`
|
||
- Create: `tests/test_formatting_detector.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: openpyxl `Worksheet`/`Cell`、`data_models.CellFormatting / CellComment`、`build_source_uri`
|
||
- Produces:
|
||
- `cell_formatting(cell) -> CellFormatting | None`（strike / 字体色 / 背景色，纯默认则 None）
|
||
- `cell_comment(cell, file_name) -> CellComment | None`
|
||
- `collect_comments(ws, file_name) -> list[CellComment]`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_formatting_detector.py`:
|
||
```python
|
||
from openpyxl import Workbook
|
||
from openpyxl.comments import Comment
|
||
|
||
from genesis.data_models import CellComment, CellFormatting
|
||
from genesis.parsers.formatting_detector import (
|
||
cell_comment, cell_formatting, collect_comments,
|
||
)
|
||
|
||
|
||
def make_wb():
|
||
wb = Workbook()
|
||
ws = wb.active
|
||
ws.title = "機能一覧"
|
||
ws["A1"] = "F001"
|
||
ws["A1"].font.strike = True
|
||
ws["A2"] = "F002"
|
||
ws["A2"].comment = Comment("要確認", "reviewer")
|
||
return wb
|
||
|
||
|
||
def test_cell_formatting_detects_strike():
|
||
fmt = cell_formatting(make_wb().active["A1"])
|
||
assert fmt is not None
|
||
assert fmt.strikethrough is True
|
||
|
||
|
||
def test_cell_formatting_none_when_plain():
|
||
wb = Workbook()
|
||
ws = wb.active
|
||
ws["A1"] = "x"
|
||
assert cell_formatting(ws["A1"]) is None
|
||
|
||
|
||
def test_cell_comment_returns_obj():
|
||
wb = make_wb()
|
||
cm = cell_comment(wb.active["A2"], "f.xlsx")
|
||
assert isinstance(cm, CellComment)
|
||
assert cm.author == "reviewer"
|
||
assert cm.text == "要確認"
|
||
assert cm.source_uri == "f.xlsx#機能一覧!A2"
|
||
|
||
|
||
def test_collect_comments():
|
||
comments = collect_comments(make_wb().active, "f.xlsx")
|
||
assert len(comments) == 1
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_formatting_detector.py -v`
|
||
Expected: FAIL（导入错误）
|
||
|
||
- [ ] **Step 3: 实现**
|
||
|
||
`src/genesis/parsers/formatting_detector.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from genesis.data_models import CellComment, CellFormatting
|
||
from genesis.parsers.provenance import build_source_uri
|
||
|
||
|
||
def cell_formatting(cell) -> CellFormatting | None:
|
||
strike = bool(cell.font.strike)
|
||
font_color = None
|
||
if cell.font.color and str(cell.font.color.rgb) not in ("00000000", "FF000000"):
|
||
font_color = str(cell.font.color.rgb)
|
||
bg_color = None
|
||
fill = cell.fill
|
||
if fill and fill.fgColor and str(fill.fgColor.rgb) not in ("00000000", "FF000000"):
|
||
bg_color = str(fill.fgColor.rgb)
|
||
if strike or font_color or bg_color:
|
||
return CellFormatting(
|
||
strikethrough=strike, font_color=font_color, bg_color=bg_color,
|
||
)
|
||
return None
|
||
|
||
|
||
def cell_comment(cell, file_name: str) -> CellComment | None:
|
||
if cell.comment is None:
|
||
return None
|
||
return CellComment(
|
||
author=cell.comment.author or "",
|
||
text=cell.comment.text or "",
|
||
source_uri=build_source_uri(file_name, cell.parent.title, cell.coordinate),
|
||
)
|
||
|
||
|
||
def collect_comments(ws, file_name: str) -> list[CellComment]:
|
||
result = []
|
||
for row in ws.iter_rows():
|
||
for cell in row:
|
||
cm = cell_comment(cell, file_name)
|
||
if cm is not None:
|
||
result.append(cm)
|
||
return result
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_formatting_detector.py -v`
|
||
Expected: PASS（4 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/formatting_detector.py tests/test_formatting_detector.py
|
||
git commit -m "feat: FormattingDetector（取消线/背景色/批注）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: FreeTextExtractor（自由记述型 Sheet 结构占位）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/free_text_extractor.py`
|
||
- Create: `tests/test_free_text_extractor.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `data_models`（`CellValue`、`ExcelTable`、`Provenance`、`SheetType`）
|
||
- Produces:
|
||
- `extract_text_blocks(matrix) -> list[str]`（全空行分段，行内单元格以空格连接）
|
||
- `build_free_text_table(sheet_name, blocks, file_name, detected_type=SheetType.GENERIC) -> ExcelTable`（`extraction_method="llm_from_free_text"`，headers=["text"]）
|
||
|
||
> LLM：本里程碑不调用 LLM（InferenceEngine 未实现）；占位表为后续 LLM 结构化保留入口。
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_free_text_extractor.py`:
|
||
```python
|
||
from genesis.data_models import CellValue, ExcelTable, SheetType
|
||
from genesis.parsers.free_text_extractor import build_free_text_table, extract_text_blocks
|
||
|
||
|
||
def test_extract_blocks_splits_on_empty_rows():
|
||
m = [["新入社員を登録。"], [], ["テスト要件：入力。"], [], []]
|
||
assert extract_text_blocks(m) == ["新入社員を登録。", "テスト要件：入力。"]
|
||
|
||
|
||
def test_build_free_text_table():
|
||
table = build_free_text_table("機能要件", ["A", "B"], "f.xlsx", SheetType.FUNCTION)
|
||
assert isinstance(table, ExcelTable)
|
||
assert table.detected_type == SheetType.FUNCTION
|
||
assert table.extraction_method == "llm_from_free_text"
|
||
assert len(table.rows) == 2
|
||
assert isinstance(table.rows[0]["text"], CellValue)
|
||
assert table.rows[0]["text"].value == "A"
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_free_text_extractor.py -v`
|
||
Expected: FAIL（导入错误）
|
||
|
||
- [ ] **Step 3: 实现**
|
||
|
||
`src/genesis/parsers/free_text_extractor.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from typing import Any
|
||
|
||
from genesis.data_models import CellValue, ExcelTable, Provenance, SheetType
|
||
|
||
|
||
def extract_text_blocks(matrix: list[list[Any]]) -> list[str]:
|
||
"""按全空行分段；行内非空单元格以「 」连接。"""
|
||
blocks: list[str] = []
|
||
current: list[str] = []
|
||
for row in matrix:
|
||
cells = [str(c).strip() for c in row if c is not None and str(c).strip() != ""]
|
||
if not cells:
|
||
if current:
|
||
blocks.append(" ".join(current))
|
||
current = []
|
||
continue
|
||
current.append(" ".join(cells))
|
||
if current:
|
||
blocks.append(" ".join(current))
|
||
return blocks
|
||
|
||
|
||
def build_free_text_table(
|
||
sheet_name: str,
|
||
blocks: list[str],
|
||
file_name: str,
|
||
detected_type: SheetType = SheetType.GENERIC,
|
||
) -> ExcelTable:
|
||
rows = []
|
||
for i, text in enumerate(blocks, start=1):
|
||
rows.append({
|
||
"text": CellValue(
|
||
value=text,
|
||
provenance=Provenance(
|
||
file_name=file_name,
|
||
sheet_name=sheet_name,
|
||
row=i,
|
||
column="A",
|
||
column_header="text",
|
||
),
|
||
),
|
||
})
|
||
return ExcelTable(
|
||
name=sheet_name,
|
||
detected_type=detected_type,
|
||
extraction_method="llm_from_free_text",
|
||
headers=["text"],
|
||
rows=rows,
|
||
)
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_free_text_extractor.py -v`
|
||
Expected: PASS（2 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/free_text_extractor.py tests/test_free_text_extractor.py
|
||
git commit -m "feat: FreeTextExtractor（文本分段 + 占位结构化表）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: ExcelParser 编排器
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/parsers/excel_parser.py`
|
||
- Create: `tests/test_excel_parser.py`
|
||
|
||
**Interfaces:**
|
||
- Produces:
|
||
- `@dataclass ExcelParseResult`：`file_name: str`、`tables: list[ExcelTable]`、`comments: list[CellComment]`、`skipped: list[str]`
|
||
- `class ExcelParser: parse(path) -> ExcelParseResult`
|
||
|
||
`parse` 编排：
|
||
1. `open_workbook(path)`
|
||
2. 每 Sheet：`matrix = sheet_matrix(ws)`；空 → skipped
|
||
3. `detected_type = detect_sheet_type(ws.title, matrix)`
|
||
4. `nature = classify_sheet(matrix)`
|
||
5. `FREE_TEXT` → `extract_text_blocks` + `build_free_text_table(detected_type=detected_type)`
|
||
6. 否则 `forward_fill(matrix, merged)` + `extract_table`
|
||
7. `collect_comments(ws, file_name)` 汇总
|
||
8. 返回 `ExcelParseResult`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
`tests/test_excel_parser.py`:
|
||
```python
|
||
from genesis.data_models import SheetType
|
||
from genesis.parsers.excel_parser import ExcelParseResult, ExcelParser
|
||
|
||
from tests.excel_helpers import new_workbook, save_workbook
|
||
|
||
|
||
def test_parse_table_sheets(tmp_path):
|
||
wb = new_workbook({
|
||
"機能一覧": [["機能ID", "機能名"], ["A001", "社員登録"]],
|
||
"バッチ一覧": [["バッチID", "処理名"], ["B1", "夜間集計"]],
|
||
})
|
||
path = save_workbook(tmp_path, wb)
|
||
result = ExcelParser().parse(path)
|
||
assert isinstance(result, ExcelParseResult)
|
||
by_name = {t.name: t for t in result.tables}
|
||
assert by_name["機能一覧"].detected_type == SheetType.FUNCTION
|
||
assert len(by_name["機能一覧"].rows) == 1
|
||
assert by_name["バッチ一覧"].detected_type == SheetType.BATCH
|
||
|
||
|
||
def test_parse_free_text_sheet(tmp_path):
|
||
wb = new_workbook({"メモ": [["新入社員を登録"], [], [], [], []]})
|
||
path = save_workbook(tmp_path, wb)
|
||
result = ExcelParser().parse(path)
|
||
assert len(result.tables) == 1
|
||
assert result.tables[0].extraction_method == "llm_from_free_text"
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**
|
||
|
||
Run: `python -m pytest tests/test_excel_parser.py -v`
|
||
Expected: FAIL（导入错误）
|
||
|
||
- [ ] **Step 3: 实现 excel_parser.py**
|
||
|
||
`src/genesis/parsers/excel_parser.py`:
|
||
```python
|
||
from __future__ import annotations
|
||
|
||
from dataclasses import dataclass, field
|
||
from pathlib import Path
|
||
|
||
from genesis.data_models import CellComment, ExcelTable
|
||
from genesis.parsers.excel_reader import open_workbook, sheet_matrix
|
||
from genesis.parsers.sheet_detector import detect_sheet_type
|
||
from genesis.parsers.sheet_nature import SheetNature, classify_sheet
|
||
from genesis.parsers.merge_fill import forward_fill
|
||
from genesis.parsers.table_extractor import extract_table
|
||
from genesis.parsers.formatting_detector import collect_comments
|
||
from genesis.parsers.free_text_extractor import build_free_text_table, extract_text_blocks
|
||
|
||
|
||
@dataclass
|
||
class ExcelParseResult:
|
||
file_name: str
|
||
tables: list[ExcelTable] = field(default_factory=list)
|
||
comments: list[CellComment] = field(default_factory=list)
|
||
skipped: list[str] = field(default_factory=list)
|
||
|
||
|
||
class ExcelParser:
|
||
"""要件定义 Excel 解析入口。"""
|
||
|
||
def parse(self, path: str | Path) -> ExcelParseResult:
|
||
wb = open_workbook(path)
|
||
file_name = Path(path).name
|
||
result = ExcelParseResult(file_name=file_name)
|
||
for ws in wb.worksheets:
|
||
matrix = sheet_matrix(ws)
|
||
if not matrix:
|
||
result.skipped.append(ws.title)
|
||
continue
|
||
detected_type = detect_sheet_type(ws.title, matrix)
|
||
nature = classify_sheet(matrix)
|
||
if nature == SheetNature.FREE_TEXT:
|
||
blocks = extract_text_blocks(matrix)
|
||
result.tables.append(
|
||
build_free_text_table(ws.title, blocks, file_name, detected_type)
|
||
)
|
||
else:
|
||
merged = [
|
||
(r.min_row, r.min_col, r.max_row, r.max_col)
|
||
for r in ws.merged_cells.ranges
|
||
]
|
||
filled = forward_fill(matrix, merged) if merged else matrix
|
||
result.tables.append(extract_table(ws.title, filled, file_name, detected_type))
|
||
result.comments.extend(collect_comments(ws, file_name))
|
||
return result
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过**
|
||
|
||
Run: `python -m pytest tests/test_excel_parser.py -v`
|
||
Expected: PASS（2 passed）
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/genesis/parsers/excel_parser.py tests/test_excel_parser.py
|
||
git commit -m "feat: ExcelParser 编排器（类型/性质/提取/汇总）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: 真实样本集成测试
|
||
|
||
**Files:**
|
||
- Create: `tests/test_real_samples.py`
|
||
|
||
**说明:** 使用 `samples/` 下 3 个脱敏要件定义样本做端到端验证。样本缺失时对应用例 `pytest.skip`。
|
||
|
||
- [ ] **Step 1: 写集成测试**
|
||
|
||
`tests/test_real_samples.py`:
|
||
```python
|
||
from pathlib import Path
|
||
|
||
import pytest
|
||
|
||
from genesis.parsers.excel_parser import ExcelParser
|
||
|
||
SAMPLES = Path(__file__).resolve().parents[1] / "samples"
|
||
|
||
|
||
def _x(name: str) -> Path:
|
||
return SAMPLES / name
|
||
|
||
|
||
def test_new_dev_sample_has_tables():
|
||
p = _x("要件定義_新規開発.xlsx")
|
||
if not p.exists():
|
||
pytest.skip("样本缺失")
|
||
result = ExcelParser().parse(p)
|
||
by_name = {t.name: t for t in result.tables}
|
||
assert "機能一覧" in by_name
|
||
assert by_name["機能一覧"].rows
|
||
assert any(t.name == "DB定義" for t in result.tables)
|
||
|
||
|
||
def test_additional_modification_has_tables():
|
||
p = _x("要件定義_追加改修.xlsx")
|
||
if not p.exists():
|
||
pytest.skip("样本缺失")
|
||
result = ExcelParser().parse(p)
|
||
assert result.tables
|
||
assert any(t.rows for t in result.tables)
|
||
|
||
|
||
def test_free_text_sample_detected():
|
||
p = _x("要件定義_自由記述.xlsx")
|
||
if not p.exists():
|
||
pytest.skip("样本缺失")
|
||
result = ExcelParser().parse(p)
|
||
assert any(t.extraction_method == "llm_from_free_text" for t in result.tables)
|
||
```
|
||
|
||
- [ ] **Step 2: 运行集成测试**
|
||
|
||
Run: `python -m pytest tests/test_real_samples.py -v`
|
||
Expected: PASS（样本存在时）或 至少 2 passed 1 skipped
|
||
|
||
- [ ] **Step 3: 全量回归**
|
||
|
||
Run: `python -m pytest -v`
|
||
Expected: PASS（全体收集成功）
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add tests/test_real_samples.py
|
||
git commit -m "test: 真实样本集成测试（3 类型 xlsx）"
|
||
```
|
||
|
||
---
|
||
|
||
## 自审结论（计划完稿时执行）
|
||
|
||
- **Spec 覆盖**：implementation-plan §2.1→Task1、§2.2→Task2、§2.3→Task3、§2.4+§2.5→Task4、§2.7→Task5、§2.6→Task6、§2.9→Task7+Task8；design §3.5.1→T2、§3.5.2→T3、§3.5.3→T4、§3.5.4/§3.5.5→T5；sample-spec §3.1-3.4→T8；验收标准（表格型/自由记述型/混合型解析、合并填充、source_uri）逐项覆盖。
|
||
- **占位符扫描**：无 TBD/TODO；任务指引 Task1 的 Step 编号已显式标注（Step 3 即实现、Step 4 运行），不含空白步骤。
|
||
- **类型一致性**：`CellValue` / `ExcelTable` / `SheetType` / `CellFormatting` / `CellComment` 与 `data_models.py` 完全一致；`build_source_uri(file, sheet, ref)` 全局同名；`extract_table.column` 用 `column_letter` 生成。
|
||
- **依赖边界**：全程不依赖 InferenceEngine；FreeText 表 `extraction_method="llm_from_free_text"` 显式标记后续替换，符合模块边界与 YAGNI。 |