lhl
a1fefd43eb
按《参赛成果物提交规范·赛道一》§8： - README：新增项目性质：新规声明、项目概述、整体功能说明、效果总结 （431 测试/99.15% 覆盖/双语试运行/影响调查基线）、团队分工、规模与难度自评 - design.md：开发范式流程图（mermaid，6 步与 AI 日志步骤列一致）+ §2.1 Agent 架构图（感知-规划-行动-记忆映射） - _AI_USAGE_LOG.md：回填「待补充」→架构设计、「整体迭代」→反馈迭代 - tests/coverage/ 覆盖率 HTML 报告（99.15%）+ tests/test-execution-log.txt 执行日志 - docs/参赛成果物提交规范-赛道一.md → docs/submission-spec-track1.md（ASCII 化） - pyproject: pytest norecursedirs 排除 test-execution-log.txt 全量 pytest 431 passed / 99.15%
151 lines
20 KiB
HTML
151 lines
20 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
|
<title>Coverage for src\genesis\inference\token.py: 100%</title>
|
|
<link rel="icon" sizes="32x32" href="favicon_32_cb_c827f16f.png">
|
|
<link rel="stylesheet" href="style_cb_0853b3de.css" type="text/css">
|
|
<script src="coverage_html_cb_188fc9a4.js" defer></script>
|
|
</head>
|
|
<body class="pyfile">
|
|
<header>
|
|
<div class="content">
|
|
<h1>
|
|
<span class="text">Coverage for </span><b>src<span class="sep">\</span>genesis<span class="sep">\</span>inference<span class="sep">\</span>token.py</b>:
|
|
<span class="pc_cov">100%</span>
|
|
</h1>
|
|
<aside id="help_panel_wrapper">
|
|
<input id="help_panel_state" type="checkbox">
|
|
<label for="help_panel_state">
|
|
<img id="keyboard_icon" src="keybd_closed_cb_900cfef5.png" alt="Show/hide keyboard shortcuts">
|
|
</label>
|
|
<div id="help_panel">
|
|
<p class="legend">Shortcuts on this page</p>
|
|
<div class="keyhelp">
|
|
<p>
|
|
<kbd>r</kbd>
|
|
<kbd>m</kbd>
|
|
<kbd>x</kbd>
|
|
<kbd>p</kbd>
|
|
toggle line displays
|
|
</p>
|
|
<p>
|
|
<kbd>j</kbd>
|
|
<kbd>k</kbd>
|
|
next/prev highlighted chunk
|
|
</p>
|
|
<p>
|
|
<kbd>0</kbd> (zero) top of page
|
|
</p>
|
|
<p>
|
|
<kbd>1</kbd> (one) first highlighted chunk
|
|
</p>
|
|
<p>
|
|
<kbd>[</kbd>
|
|
<kbd>]</kbd>
|
|
prev/next file
|
|
</p>
|
|
<p>
|
|
<kbd>u</kbd> up to the index
|
|
</p>
|
|
<p>
|
|
<kbd>?</kbd> show/hide this help
|
|
</p>
|
|
</div>
|
|
</div>
|
|
</aside>
|
|
<h2>
|
|
<span class="text">26 statements </span>
|
|
<button type="button" class="run button_toggle_run" value="run" data-shortcut="r" title="Toggle lines run">26<span class="text"> run</span></button>
|
|
<button type="button" class="mis show_mis button_toggle_mis" value="mis" data-shortcut="m" title="Toggle lines missing">0<span class="text"> missing</span></button>
|
|
<button type="button" class="exc show_exc button_toggle_exc" value="exc" data-shortcut="x" title="Toggle lines excluded">0<span class="text"> excluded</span></button>
|
|
<button type="button" class="par run show_par button_toggle_par" value="par" data-shortcut="p" title="Toggle lines partially run">0<span class="text"> partial</span></button>
|
|
</h2>
|
|
<p class="text">
|
|
<a id="prevFileLink" class="nav" href="z_ea3a2206db328682_prompt_registry_py.html">« prev</a>
|
|
<a id="indexLink" class="nav" href="index.html">^ index</a>
|
|
<a id="nextFileLink" class="nav" href="z_ea3a2206db328682_types_py.html">» next</a>
|
|
|
|
<a class="nav" href="https://coverage.readthedocs.io/en/7.15.4">coverage.py v7.15.4</a>,
|
|
created at 2026-08-26 14:20 +0800
|
|
</p>
|
|
<aside class="hidden">
|
|
<button type="button" class="button_next_chunk" data-shortcut="j"></button>
|
|
<button type="button" class="button_prev_chunk" data-shortcut="k"></button>
|
|
<button type="button" class="button_top_of_page" data-shortcut="0"></button>
|
|
<button type="button" class="button_first_chunk" data-shortcut="1"></button>
|
|
<button type="button" class="button_prev_file" data-shortcut="["></button>
|
|
<button type="button" class="button_next_file" data-shortcut="]"></button>
|
|
<button type="button" class="button_to_index" data-shortcut="u"></button>
|
|
<button type="button" class="button_show_hide_help" data-shortcut="?"></button>
|
|
</aside>
|
|
</div>
|
|
</header>
|
|
<main id="source">
|
|
<p class="run"><span class="n"><a id="t1" href="#t1">1</a></span><span class="t"><span class="key">from</span> <span class="nam">__future__</span> <span class="key">import</span> <span class="nam">annotations</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t2" href="#t2">2</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t3" href="#t3">3</a></span><span class="t"><span class="key">import</span> <span class="nam">unicodedata</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t4" href="#t4">4</a></span><span class="t"><span class="key">from</span> <span class="nam">typing</span> <span class="key">import</span> <span class="nam">Callable</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t5" href="#t5">5</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t6" href="#t6">6</a></span><span class="t"><span class="com"># CJK 每字符保守 token 数（T9 整改：cl100k/4字符1token 对中文/日文严重低估）</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t7" href="#t7">7</a></span><span class="t"><span class="com"># 日文/中文实际每字符 1~3 token，取保守 1.5 防止上下文裁剪失效导致 API 超限</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t8" href="#t8">8</a></span><span class="t"><span class="nam">_CJK_TOKENS_PER_CHAR</span> <span class="op">=</span> <span class="num">1.5</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t9" href="#t9">9</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t10" href="#t10">10</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t11" href="#t11">11</a></span><span class="t"><span class="key">def</span> <span class="nam">_is_cjk_char</span><span class="op">(</span><span class="nam">ch</span><span class="op">:</span> <span class="nam">str</span><span class="op">)</span> <span class="op">-></span> <span class="nam">bool</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t12" href="#t12">12</a></span><span class="t"> <span class="str">"""判断字符是否属于 CJK 密集区（中文/日文假名/韩文/全角符号）。"""</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t13" href="#t13">13</a></span><span class="t"> <span class="nam">cp</span> <span class="op">=</span> <span class="nam">ord</span><span class="op">(</span><span class="nam">ch</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t14" href="#t14">14</a></span><span class="t"> <span class="key">return</span> <span class="op">(</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t15" href="#t15">15</a></span><span class="t"> <span class="num">0x4E00</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0x9FFF</span> <span class="com"># CJK 统一表意文字</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t16" href="#t16">16</a></span><span class="t"> <span class="key">or</span> <span class="num">0x3040</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0x30FF</span> <span class="com"># 平假名/片假名</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t17" href="#t17">17</a></span><span class="t"> <span class="key">or</span> <span class="num">0xAC00</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0xD7AF</span> <span class="com"># 韩文音节</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t18" href="#t18">18</a></span><span class="t"> <span class="key">or</span> <span class="num">0xF900</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0xFAFF</span> <span class="com"># CJK 兼容表意文字</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t19" href="#t19">19</a></span><span class="t"> <span class="key">or</span> <span class="num">0xFF00</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0xFFEF</span> <span class="com"># 全角形式（全角标点/字母）</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t20" href="#t20">20</a></span><span class="t"> <span class="key">or</span> <span class="num">0x3400</span> <span class="op"><=</span> <span class="nam">cp</span> <span class="op"><=</span> <span class="num">0x4DBF</span> <span class="com"># CJK 扩展 A</span> </span><span class="r"></span></p>
|
|
<p class="run run2"><span class="n"><a id="t21" href="#t21">21</a></span><span class="t"> <span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t22" href="#t22">22</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t23" href="#t23">23</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t24" href="#t24">24</a></span><span class="t"><span class="key">def</span> <span class="nam">approximate_token_count</span><span class="op">(</span><span class="nam">text</span><span class="op">:</span> <span class="nam">str</span><span class="op">)</span> <span class="op">-></span> <span class="nam">int</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t25" href="#t25">25</a></span><span class="t"> <span class="str">"""内置近似估算（T9 CJK 保守）：CJK 字符按 1.5 token/字符，</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t26" href="#t26">26</a></span><span class="t"><span class="str"> 其余字符按 4 字符 ≈ 1 token；最少 1 token（无外部依赖，可离线）。"""</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t27" href="#t27">27</a></span><span class="t"> <span class="nam">cjk_chars</span> <span class="op">=</span> <span class="nam">sum</span><span class="op">(</span><span class="num">1</span> <span class="key">for</span> <span class="nam">ch</span> <span class="key">in</span> <span class="nam">text</span> <span class="key">if</span> <span class="nam">_is_cjk_char</span><span class="op">(</span><span class="nam">ch</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t28" href="#t28">28</a></span><span class="t"> <span class="nam">other_chars</span> <span class="op">=</span> <span class="nam">len</span><span class="op">(</span><span class="nam">text</span><span class="op">)</span> <span class="op">-</span> <span class="nam">cjk_chars</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t29" href="#t29">29</a></span><span class="t"> <span class="nam">tokens</span> <span class="op">=</span> <span class="nam">cjk_chars</span> <span class="op">*</span> <span class="nam">_CJK_TOKENS_PER_CHAR</span> <span class="op">+</span> <span class="op">(</span><span class="nam">other_chars</span> <span class="op">+</span> <span class="num">3</span><span class="op">)</span> <span class="op">//</span> <span class="num">4</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t30" href="#t30">30</a></span><span class="t"> <span class="key">return</span> <span class="nam">max</span><span class="op">(</span><span class="num">1</span><span class="op">,</span> <span class="nam">int</span><span class="op">(</span><span class="nam">tokens</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t31" href="#t31">31</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t32" href="#t32">32</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t33" href="#t33">33</a></span><span class="t"><span class="key">def</span> <span class="nam">_tiktoken_estimator</span><span class="op">(</span><span class="nam">text</span><span class="op">:</span> <span class="nam">str</span><span class="op">)</span> <span class="op">-></span> <span class="nam">int</span> <span class="op">|</span> <span class="key">None</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t34" href="#t34">34</a></span><span class="t"> <span class="str">"""tiktoken 编码估算；tiktoken 未安装时返回 None。"""</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t35" href="#t35">35</a></span><span class="t"> <span class="key">try</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t36" href="#t36">36</a></span><span class="t"> <span class="key">import</span> <span class="nam">tiktoken</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t37" href="#t37">37</a></span><span class="t"> <span class="key">except</span> <span class="nam">ImportError</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t38" href="#t38">38</a></span><span class="t"> <span class="key">return</span> <span class="key">None</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t39" href="#t39">39</a></span><span class="t"> <span class="key">try</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t40" href="#t40">40</a></span><span class="t"> <span class="nam">enc</span> <span class="op">=</span> <span class="nam">tiktoken</span><span class="op">.</span><span class="nam">get_encoding</span><span class="op">(</span><span class="str">"cl100k_base"</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t41" href="#t41">41</a></span><span class="t"> <span class="key">return</span> <span class="nam">len</span><span class="op">(</span><span class="nam">enc</span><span class="op">.</span><span class="nam">encode</span><span class="op">(</span><span class="nam">text</span><span class="op">)</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t42" href="#t42">42</a></span><span class="t"> <span class="key">except</span> <span class="nam">Exception</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t43" href="#t43">43</a></span><span class="t"> <span class="key">return</span> <span class="key">None</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t44" href="#t44">44</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t45" href="#t45">45</a></span><span class="t"> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t46" href="#t46">46</a></span><span class="t"><span class="key">def</span> <span class="nam">make_estimator</span><span class="op">(</span><span class="nam">backend</span><span class="op">:</span> <span class="nam">str</span> <span class="op">=</span> <span class="str">"tiktoken"</span><span class="op">)</span> <span class="op">-></span> <span class="nam">Callable</span><span class="op">[</span><span class="op">[</span><span class="nam">str</span><span class="op">]</span><span class="op">,</span> <span class="nam">int</span><span class="op">]</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t47" href="#t47">47</a></span><span class="t"> <span class="str">"""按配置选择估算器：backend="tiktoken"（默认）优先 tiktoken，</span> </span><span class="r"></span></p>
|
|
<p class="pln"><span class="n"><a id="t48" href="#t48">48</a></span><span class="t"><span class="str"> 缺失或异常回落内置 approximate；backend="approximate" 直接用近似。"""</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t49" href="#t49">49</a></span><span class="t"> <span class="key">if</span> <span class="nam">backend</span> <span class="op">==</span> <span class="str">"approximate"</span><span class="op">:</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t50" href="#t50">50</a></span><span class="t"> <span class="key">return</span> <span class="nam">approximate_token_count</span> </span><span class="r"></span></p>
|
|
<p class="run"><span class="n"><a id="t51" href="#t51">51</a></span><span class="t"> <span class="key">return</span> <span class="key">lambda</span> <span class="nam">text</span><span class="op">:</span> <span class="nam">_tiktoken_estimator</span><span class="op">(</span><span class="nam">text</span><span class="op">)</span> <span class="key">or</span> <span class="nam">approximate_token_count</span><span class="op">(</span><span class="nam">text</span><span class="op">)</span> </span><span class="r"></span></p>
|
|
</main>
|
|
<footer>
|
|
<div class="content">
|
|
<p>
|
|
<a class="nav" href="z_ea3a2206db328682_prompt_registry_py.html">« prev</a>
|
|
<a class="nav" href="index.html">^ index</a>
|
|
<a class="nav" href="z_ea3a2206db328682_types_py.html">» next</a>
|
|
|
|
<a class="nav" href="https://coverage.readthedocs.io/en/7.15.4">coverage.py v7.15.4</a>,
|
|
created at 2026-08-26 14:20 +0800
|
|
</p>
|
|
</div>
|
|
</footer>
|
|
</body>
|
|
</html>
|