514 lines
18 KiB
Markdown
514 lines
18 KiB
Markdown
# 影响调查 RAG 化 Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 在影响调查（impact investigation）阶段，对项目的既有系统源码/设计文档/规则做切块与向量检索（RAG），把与当前需求/表最相关的片段作为上下文喂给 LLM，提升受影响模块/字段的召回率；在无法检索或无嵌入能力时回退到现有确定性交叉引用（DesignReference），保证行为不退化。
|
||
|
||
**Architecture:** 新增轻量本地 RAG 栈——`Embedder`（优先复用推理引擎的 embed 能力，缺省用确定性的词袋 FakeEmbedder 以保证测试与离线运行）、`RagStore`（SQLite 持久化 chunks + 余弦检索，复用项目既有 SQLite 习惯，不引入 chromadb/faiss 等新重依赖）、`ImpactRAG`（index/retrieve）。`ImpactAgent` 增加 `use_rag` 开关：开启时把检索到的 top-k 片段拼入影响调查 prompt；关闭时走原路径。`GenesisService` 透传 `use_rag`，默认 `False`（向后兼容）。
|
||
|
||
**Tech Stack:** Python `sqlite3` + `json`（向量存储，无新依赖）；现有 `inference` 引擎抽象（取 embed）；`node:test` 仅用于前端（本迭代无前端改动，故无 Node 测试）。
|
||
|
||
## Global Constraints
|
||
|
||
- 中文优先：所有 UI 文案、注释、日志使用中文（专有名词/代码关键字除外）。
|
||
- 不新增重依赖：RAG 存储用标准库 `sqlite3` + `json`；Embedder 优先复用推理引擎，缺省 `FakeEmbedder`（纯标准库）。禁止引入 chromadb / faiss / sentence-transformers 等重依赖。
|
||
- TDD：每个任务先写失败测试再实现；全绿方可 commit。
|
||
- 覆盖率门禁：`fail_under=99`（pyproject 现有），新增 Python 代码须有测试覆盖。
|
||
- 向后兼容：`use_rag` 默认 `False`；未开启时影响调查输出与现状完全一致（DesignReference 路径不变）。
|
||
- 增量索引：同一 scope 重复 index 先清后写，避免重复累积。
|
||
|
||
---
|
||
|
||
### Task 1: Embedder 抽象（含离线 FakeEmbedder）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/rag/embeddings.py`
|
||
- Test: `tests/test_rag_embeddings.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `genesis.inference`（可选，用于真实 embed）
|
||
- Produces:
|
||
- `genesis.rag.embeddings.Embedder` 协议：`embed(texts: list[str]) -> list[list[float]]`
|
||
- `genesis.rag.embeddings.FakeEmbedder`（词袋确定性，dim=64，归一化）
|
||
- `genesis.rag.embeddings.get_embedder(engine) -> Embedder`（有 embed 能力用真实，否则 FakeEmbedder）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/test_rag_embeddings.py
|
||
from genesis.rag.embeddings import FakeEmbedder, get_embedder
|
||
|
||
|
||
def test_fake_embedder_deterministic():
|
||
e = FakeEmbedder()
|
||
a = e.embed(["OrderController 创建订单"])[0]
|
||
b = e.embed(["OrderController 创建订单"])[0]
|
||
assert a == b
|
||
|
||
|
||
def test_fake_embedder_similar_closer_than_unrelated():
|
||
e = FakeEmbedder()
|
||
base = e.embed(["OrderController 处理创建订单请求"])[0]
|
||
sim = e.embed(["OrderController 保存订单到数据库"])[0]
|
||
dif = e.embed(["用户登录认证模块"])[0]
|
||
import math
|
||
def cos(x, y):
|
||
dot = sum(p * q for p, q in zip(x, y))
|
||
nx = math.sqrt(sum(p * p for p in x)); ny = math.sqrt(sum(q * q for q in y))
|
||
return dot / (nx * ny or 1.0)
|
||
assert cos(base, sim) > cos(base, dif)
|
||
|
||
|
||
def test_get_embedder_fake_engine_returns_fake():
|
||
e = get_embedder("fake")
|
||
assert e.__class__.__name__ == "FakeEmbedder"
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `python -m pytest tests/test_rag_embeddings.py -q`
|
||
Expected: FAIL（`ModuleNotFoundError`）
|
||
|
||
- [ ] **Step 3: 实现最小版本**
|
||
|
||
```python
|
||
# src/genesis/rag/embeddings.py
|
||
from __future__ import annotations
|
||
|
||
import hashlib
|
||
import math
|
||
from typing import List, Protocol
|
||
|
||
|
||
class Embedder(Protocol):
|
||
def embed(self, texts: List[str]) -> List[List[float]]: ...
|
||
|
||
|
||
_DIM = 64
|
||
|
||
|
||
def _tokenize(text: str) -> List[str]:
|
||
# 简易分词：去标点、按空白与驼峰边界切分，转小写
|
||
toks = []
|
||
cur = ""
|
||
for ch in text.lower():
|
||
if ch.isalnum():
|
||
cur += ch
|
||
else:
|
||
if cur:
|
||
toks.append(cur); cur = ""
|
||
if cur:
|
||
toks.append(cur)
|
||
# 驼峰拆分（OrderController -> order, controller）
|
||
out = []
|
||
for t in toks:
|
||
idx = 0
|
||
for i, c in enumerate(t):
|
||
if i > 0 and c.isupper():
|
||
out.append(t[idx:i]); idx = i
|
||
out.append(t[idx:])
|
||
return [x for x in out if x]
|
||
|
||
|
||
class FakeEmbedder:
|
||
def embed(self, texts: List[str]) -> List[List[float]]:
|
||
vecs = []
|
||
for t in texts:
|
||
v = [0.0] * _DIM
|
||
for tok in _tokenize(t):
|
||
h = hashlib.md5(tok.encode("utf-8")).digest()
|
||
idx = h[0] % _DIM
|
||
v[idx] += 1.0
|
||
norm = math.sqrt(sum(x * x for x in v)) or 1.0
|
||
vecs.append([x / norm for x in v])
|
||
return vecs
|
||
|
||
|
||
def get_embedder(engine) -> Embedder:
|
||
# 真实引擎若提供 embed API 则在此接入；当前统一回退 FakeEmbedder
|
||
return FakeEmbedder()
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `python -m pytest tests/test_rag_embeddings.py -q`
|
||
Expected: PASS（3 passed）
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add src/genesis/rag/embeddings.py tests/test_rag_embeddings.py
|
||
git commit -m "feat(rag): 新增 Embedder 抽象与离线 FakeEmbedder"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: RagStore（SQLite 向量存储 + 余弦检索）
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/rag/store.py`
|
||
- Test: `tests/test_rag_store.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `embeddings.Embedder`（仅用于维度约定，实际检索用传入向量）
|
||
- Produces:
|
||
- `genesis.rag.store.RagStore(db_path)`：方法 `reset_scope(scope)`、`add(scope, chunks, embeddings)`、`search(scope, query_vec, k) -> list[str]`
|
||
- 向量以 JSON 文本列存储（无新依赖）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/test_rag_store.py
|
||
import math
|
||
from genesis.rag.store import RagStore
|
||
|
||
|
||
def _vec(*ones):
|
||
v = [0.0] * 8
|
||
for i in ones:
|
||
v[i] = 1.0
|
||
n = math.sqrt(sum(x * x for x in v))
|
||
return [x / n for x in v]
|
||
|
||
|
||
def test_search_returns_most_similar_chunk():
|
||
s = RagStore(":memory:")
|
||
s.reset_scope("p1")
|
||
s.add("p1", ["订单模块处理创建", "用户认证登录"], [_vec(0, 1), _vec(4, 5)])
|
||
res = s.search("p1", _vec(0, 1), k=1)
|
||
assert res == ["订单模块处理创建"]
|
||
|
||
|
||
def test_reset_scope_clears():
|
||
s = RagStore(":memory:")
|
||
s.reset_scope("p1")
|
||
s.add("p1", ["a"], [_vec(0)])
|
||
s.reset_scope("p1")
|
||
assert s.search("p1", _vec(0), k=3) == []
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `python -m pytest tests/test_rag_store.py -q`
|
||
Expected: FAIL
|
||
|
||
- [ ] **Step 3: 实现最小版本**
|
||
|
||
```python
|
||
# src/genesis/rag/store.py
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import math
|
||
import sqlite3
|
||
from typing import List
|
||
|
||
|
||
def _cosine(a, b):
|
||
dot = sum(x * y for x, y in zip(a, b))
|
||
na = math.sqrt(sum(x * x for x in a)); nb = math.sqrt(sum(y * y for y in b))
|
||
return dot / (na * nb or 1.0)
|
||
|
||
|
||
class RagStore:
|
||
def __init__(self, db_path: str):
|
||
self.conn = sqlite3.connect(db_path)
|
||
self.conn.execute(
|
||
"CREATE TABLE IF NOT EXISTS rag_chunks ("
|
||
"id INTEGER PRIMARY KEY, scope TEXT, chunk TEXT, embedding TEXT)"
|
||
)
|
||
self.conn.commit()
|
||
|
||
def reset_scope(self, scope: str) -> None:
|
||
self.conn.execute("DELETE FROM rag_chunks WHERE scope=?", (scope,))
|
||
self.conn.commit()
|
||
|
||
def add(self, scope: str, chunks: List[str], embeddings: List[List[float]]) -> None:
|
||
self.reset_scope(scope)
|
||
for c, e in zip(chunks, embeddings):
|
||
self.conn.execute(
|
||
"INSERT INTO rag_chunks(scope, chunk, embedding) VALUES(?,?,?)",
|
||
(scope, c, json.dumps(e)),
|
||
)
|
||
self.conn.commit()
|
||
|
||
def search(self, scope: str, query_vec: List[float], k: int = 5) -> List[str]:
|
||
rows = self.conn.execute(
|
||
"SELECT chunk, embedding FROM rag_chunks WHERE scope=?", (scope,)
|
||
).fetchall()
|
||
scored = []
|
||
for chunk, emb in rows:
|
||
scored.append((_cosine(query_vec, json.loads(emb)), chunk))
|
||
scored.sort(key=lambda x: x[0], reverse=True)
|
||
return [c for _, c in scored[:k]]
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `python -m pytest tests/test_rag_store.py -q`
|
||
Expected: PASS（2 passed）
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add src/genesis/rag/store.py tests/test_rag_store.py
|
||
git commit -m "feat(rag): 新增 RagStore（SQLite 向量存储 + 余弦检索）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: ImpactRAG 索引/检索服务
|
||
|
||
**Files:**
|
||
- Create: `src/genesis/rag/impact_rag.py`
|
||
- Test: `tests/test_impact_rag.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `RagStore`、`Embedder`
|
||
- Produces:
|
||
- `genesis.rag.impact_rag.ImpactRAG(store, embedder)`
|
||
- `index(scope, sources: list[tuple[str, str]])`（sources = [(文件名, 文本), ...]）
|
||
- `retrieve(scope, query: str, k: int = 5) -> list[str]`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/test_impact_rag.py
|
||
from genesis.rag.embeddings import FakeEmbedder
|
||
from genesis.rag.store import RagStore
|
||
from genesis.rag.impact_rag import ImpactRAG
|
||
|
||
|
||
def test_retrieve_returns_relevant_chunk():
|
||
store = RagStore(":memory:")
|
||
rag = ImpactRAG(store, FakeEmbedder())
|
||
sources = [
|
||
("OrderController.java", "public class OrderController { 创建订单 }"),
|
||
("UserAuth.java", "public class UserAuth { 用户登录认证 }"),
|
||
]
|
||
rag.index("p1", sources)
|
||
res = rag.retrieve("p1", "OrderController 的影响范围", k=1)
|
||
assert res and "OrderController" in res[0]
|
||
```
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `python -m pytest tests/test_impact_rag.py -q`
|
||
Expected: FAIL
|
||
|
||
- [ ] **Step 3: 实现最小版本**
|
||
|
||
```python
|
||
# src/genesis/rag/impact_rag.py
|
||
from __future__ import annotations
|
||
|
||
from typing import List, Tuple
|
||
|
||
from genesis.rag.embeddings import Embedder
|
||
from genesis.rag.store import RagStore
|
||
|
||
_CHUNK = 800
|
||
|
||
|
||
def _split(text: str, size: int = _CHUNK) -> List[str]:
|
||
paras = [p.strip() for p in text.split("\n") if p.strip()]
|
||
out, buf = [], ""
|
||
for p in paras:
|
||
if len(buf) + len(p) > size and buf:
|
||
out.append(buf); buf = p
|
||
else:
|
||
buf = (buf + "\n" + p).strip()
|
||
if buf:
|
||
out.append(buf)
|
||
return out or [""]
|
||
|
||
|
||
class ImpactRAG:
|
||
def __init__(self, store: RagStore, embedder: Embedder):
|
||
self.store = store
|
||
self.embedder = embedder
|
||
|
||
def index(self, scope: str, sources: List[Tuple[str, str]]) -> None:
|
||
chunks = []
|
||
for name, text in sources:
|
||
for piece in _split(text):
|
||
chunks.append(f"[{name}]\n{piece}")
|
||
embs = self.embedder.embed(chunks)
|
||
self.store.add(scope, chunks, embs)
|
||
|
||
def retrieve(self, scope: str, query: str, k: int = 5) -> List[str]:
|
||
if not query.strip():
|
||
return []
|
||
qv = self.embedder.embed([query])[0]
|
||
return self.store.search(scope, qv, k)
|
||
```
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `python -m pytest tests/test_impact_rag.py -q`
|
||
Expected: PASS
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add src/genesis/rag/impact_rag.py tests/test_impact_rag.py
|
||
git commit -m "feat(rag): 新增 ImpactRAG 索引/检索服务"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: ImpactAgent 接入 use_rag
|
||
|
||
**Files:**
|
||
- Modify: `src/genesis/impact/impact_agent.py`
|
||
- Test: `tests/test_impact_agent_rag.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `ImpactRAG`（可选）、`FakeEmbedder`/`RagStore`（测试）
|
||
- Produces: `ImpactAgent.__init__` 新增 `rag: ImpactRAG | None = None`；`run_impact(..., use_rag: bool = False)` 开启时把检索片段并入 prompt 上下文
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/test_impact_agent_rag.py
|
||
from genesis.rag.embeddings import FakeEmbedder
|
||
from genesis.rag.store import RagStore
|
||
from genesis.rag.impact_rag import ImpactRAG
|
||
from genesis.impact.impact_agent import ImpactAgent
|
||
|
||
|
||
def _make():
|
||
store = RagStore(":memory:")
|
||
rag = ImpactRAG(store, FakeEmbedder())
|
||
rag.index("p1", [("OrderController.java", "class OrderController { 创建订单 }")])
|
||
# ImpactAgent 构造参数按现有签名；rag 以 kw 注入
|
||
agent = ImpactAgent(rag=rag)
|
||
return agent
|
||
|
||
|
||
def test_run_impact_includes_rag_context(monkeypatch):
|
||
agent = _make()
|
||
captured = {}
|
||
def fake_engine_invoke(prompt):
|
||
captured["prompt"] = prompt
|
||
return '{"impacts":[{"target":"OrderController","type":"修改","detail":"受订单表影响"}]}'
|
||
monkeypatch.setattr(agent, "_engine_invoke", fake_engine_invoke)
|
||
out = agent.run_impact("订单表新增字段", scope="p1", use_rag=True)
|
||
assert "OrderController" in captured["prompt"]
|
||
assert "OrderController" in str(out)
|
||
```
|
||
|
||
（注：`_engine_invoke` 为 ImpactAgent 内部调用 LLM 的既有方法名占位；若实际方法名不同，实现任务中替换为真实方法名，并在测试中同步。）
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `python -m pytest tests/test_impact_agent_rag.py -q`
|
||
Expected: FAIL（`rag` 参数 / `use_rag` 不存在）
|
||
|
||
- [ ] **Step 3: 实现最小版本**
|
||
|
||
`impact_agent.py` 顶部 `from genesis.rag.impact_rag import ImpactRAG`（仅在类型标注用，避免循环导入：若 `rag` 包尚未在 `genesis/rag/__init__` 暴露，使用 `TYPE_CHECKING` 方式或字符串标注）。
|
||
|
||
`ImpactAgent.__init__` 增加：
|
||
|
||
```python
|
||
def __init__(self, ..., rag: "ImpactRAG | None" = None):
|
||
...
|
||
self.rag = rag
|
||
```
|
||
|
||
`run_impact` 在拼装影响调查 prompt 前，若 `use_rag and self.rag`：
|
||
|
||
```python
|
||
context_chunks = self.rag.retrieve(scope, requirement_text, k=5)
|
||
if context_chunks:
|
||
rag_block = "\n".join(f"# 相关代码/文档片段\n{c}" for c in context_chunks)
|
||
prompt = prompt + "\n\n" + rag_block
|
||
```
|
||
|
||
未开启或 `self.rag is None` 时，`prompt` 与现状一致（DesignReference 路径不变）。返回结构不变。
|
||
|
||
- [ ] **Step 4: 运行测试确认通过**
|
||
|
||
Run: `python -m pytest tests/test_impact_agent_rag.py -q`
|
||
Expected: PASS
|
||
|
||
- [ ] **Step 5: 提交**
|
||
|
||
```bash
|
||
git add src/genesis/impact/impact_agent.py tests/test_impact_agent_rag.py
|
||
git commit -m "feat(impact): ImpactAgent 接入 use_rag，检索片段并入 prompt（默认关闭）"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: 服务层接线 + 端到端 + 文档
|
||
|
||
**Files:**
|
||
- Modify: `src/genesis/server/service.py`、`src/genesis/server/app.py`（如需要透传 `use_rag`）
|
||
- Test: `tests/test_impact_rag_e2e.py`
|
||
- Modify: `docs/design.md`、`_AI_USAGE_LOG.md`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `ImpactRAG`、`RagStore`、`FakeEmbedder`、`ImpactAgent(rag=...)`
|
||
- Produces: `GenesisService(use_rag: bool = False)`；会话级影响调查在 `use_rag` 为真时走 RAG
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/test_impact_rag_e2e.py
|
||
from genesis.rag.embeddings import FakeEmbedder
|
||
from genesis.rag.store import RagStore
|
||
from genesis.rag.impact_rag import ImpactRAG
|
||
from genesis.server.service import GenesisService
|
||
from genesis.server.store import ProjectsStore, SessionStore
|
||
|
||
|
||
def test_service_impact_uses_rag_when_enabled(tmp_path):
|
||
store = SessionStore(db_path=str(tmp_path / "s.db"))
|
||
projects = ProjectsStore(db_path=str(tmp_path / "s.db"))
|
||
rag_store = RagStore(str(tmp_path / "rag.db"))
|
||
rag = ImpactRAG(rag_store, FakeEmbedder())
|
||
rag.index("p1", [("OrderController.java", "class OrderController { 创建订单 }")])
|
||
svc = GenesisService(store=store, data_root=str(tmp_path / "data"),
|
||
engine="fake", projects=projects, use_rag=True, rag=rag)
|
||
# 仅验证接线：service 持有 rag 且 use_rag 为真
|
||
assert svc.use_rag is True and svc.rag is rag
|
||
```
|
||
|
||
（完整端到端需真实引擎与样本源码；以接线断言 + 既有 impact 测试不退化为下限。）
|
||
|
||
- [ ] **Step 2: 运行测试确认失败**
|
||
|
||
Run: `python -m pytest tests/test_impact_rag_e2e.py -q`
|
||
Expected: FAIL（`use_rag` / `rag` 参数不存在）
|
||
|
||
- [ ] **Step 3: 实现最小版本**
|
||
|
||
`service.py`：`GenesisService.__init__` 增加 `use_rag: bool = False, rag: ImpactRAG | None = None`；保存 `self.use_rag / self.rag`。在 `run_impact`（或 `_build_impact_agent`）构造 `ImpactAgent(rag=self.rag)` 并将 `use_rag=self.use_rag` 透传给 `run_impact`。
|
||
|
||
`app.py`：`create_app` 增加 `use_rag: bool = False` 参数并透传 `GenesisService(..., use_rag=use_rag)`；`scripts/serve.py` 增加 `--use-rag` 开关（默认关）。
|
||
|
||
- [ ] **Step 4: 运行全量测试确认无回归**
|
||
|
||
Run: `python -m pytest -q -o addopts=""`
|
||
Expected: 全部通过（在现有基线基础上新增用例；既有 impact/design 测试不受默认关闭影响）
|
||
|
||
- [ ] **Step 5: 更新文档 + 提交**
|
||
|
||
`docs/design.md` §12 追加：「影响调查 RAG 化（2026-08-29）：新增 `genesis/rag`（embeddings/store/impact_rag），`ImpactAgent.use_rag` 开启时检索 top-k 片段并入 prompt；默认关闭，回退 DesignReference；存储用 SQLite，无新重依赖。」
|
||
|
||
```bash
|
||
git add src/genesis/server/service.py src/genesis/server/app.py scripts/serve.py tests/test_impact_rag_e2e.py docs/design.md _AI_USAGE_LOG.md
|
||
git commit -m "feat(rag): 服务层接线 use_rag + design.md 记录（默认关闭）"
|
||
```
|
||
|
||
---
|
||
|
||
## Self-Review
|
||
|
||
1. **Spec coverage:** 离线 Embedder（T1）、向量存储（T2）、索引/检索（T3）、Agent 接入与开关（T4）、服务层接线与文档（T5）均覆盖；回退路径在 T4/T5 明确保留。
|
||
2. **Placeholder scan:** 无 TBD；`_engine_invoke` 已在测试中标注为真实方法名占位，实现任务需对齐真实方法名（已在 Step 3 注明）。
|
||
3. **Type consistency:** `ImpactRAG.index(scope, sources)` / `retrieve(scope, query, k)` 在 T3/T4 一致；`RagStore.add(scope, chunks, embeddings)` 维度与 `Embedder.embed` 输出一致；`use_rag` 在 Service/Agent 间透传一致。
|
||
4. **限制:** 默认 `use_rag=False`，不影响现有行为；真实 embed 接入点在 `get_embedder` 预留，待推理引擎提供 embed API 时扩展（不在本迭代）。
|