315 lines
28 KiB
JSON
315 lines
28 KiB
JSON
|
{
|
|||
|
|
"topic": "ai-engineering",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"schema_version": "1.0.0",
|
|||
|
|
"generated": "2026-09-09T10:00:00+08:00",
|
|||
|
|
"questions": [
|
|||
|
|
{
|
|||
|
|
"id": "cr-001",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"difficulty": 3,
|
|||
|
|
"tags": [
|
|||
|
|
"tool-calling",
|
|||
|
|
"ai-agent",
|
|||
|
|
"harness-engineering"
|
|||
|
|
],
|
|||
|
|
"question": "以下是一个 AI Agent 调用外部工具的 Python 实现。该 Agent 接收用户查询，根据 LLM 判断是否需要调用工具，并在获取工具结果后生成最终回复。请阅读代码并回答子问题。",
|
|||
|
|
"code": "import json\nfrom typing import Any, Callable\n\n# 工具注册表：name -> {description, parameters, handler}\ntool_registry: dict[str, dict] = {}\n\ndef register_tool(name: str, description: str, params: dict, handler: Callable):\n \"\"\"注册一个可被 Agent 调用的工具\"\"\"\n tool_registry[name] = {\n \"description\": description,\n \"parameters\": params,\n \"handler\": handler,\n }\n\ndef call_tool(name: str, arguments: dict[str, Any]) -> str:\n \"\"\"执行工具调用，带参数校验和异常处理\"\"\"\n if name not in tool_registry:\n return json.dumps({\"error\": f\"Unknown tool: {name}\"})\n tool = tool_registry[name]\n try:\n # 按 schema 校验必填参数\n for param, spec in tool[\"parameters\"].items():\n if spec.get(\"required\") and param not in arguments:\n return json.dumps({\"error\": f\"Missing required param: {param}\"})\n result = tool[\"handler\"](**arguments)\n return json.dumps({\"result\": result})\n except Exception as e:\n return json.dumps({\"error\": str(e)})\n\ndef agent_loop(user_query: str, llm_client) -> str:\n \"\"\"Agent 主循环：查询 -> 判断 -> 调用工具 -> 拼接回复\"\"\"\n tools_schema = [\n {\"name\": k, \"description\": v[\"description\"], \"parameters\": v[\"parameters\"]}\n for k, v in tool_registry.items()\n ]\n response = llm_client.chat(user_query, tools=tools_schema)\n\n # LLM 返回 tool_calls 时执行工具\n if response.get(\"tool_calls\"):\n tool_results = []\n for tc in response[\"tool_calls\"]:\n result = call_tool(tc[\"name\"], tc[\"arguments\"])\n tool_results.append({\"tool\": tc[\"name\"], \"output\": result})\n # 将工具结果反馈给 LLM 生成最终回答\n followup = llm_client.chat(\n f\"Tools returned: {json.dumps(tool_results)}\",\n tools=tools_schema,\n )\n return followup[\"content\"]\n return response[\"content\"]",
|
|||
|
|
"language": "python",
|
|||
|
|
"sub_questions": [
|
|||
|
|
{
|
|||
|
|
"index": 1,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "当 LLM 返回的 tool_calls 中包含多个工具调用时，当前代码对这些调用的执行方式是？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "并行异步执行，使用 asyncio.gather",
|
|||
|
|
"B": "串行顺序执行，逐个调用",
|
|||
|
|
"C": "只执行第一个工具调用，忽略其余",
|
|||
|
|
"D": "随机选择一个工具调用执行"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "代码中使用 for tc in response[\"tool_calls\"] 串行遍历并逐个调用 call_tool()，所有工具调用按顺序依次执行，没有使用异步或并发机制。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 2,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "call_tool 函数在遇到工具不存在时的处理策略是？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "抛出 KeyError 异常中断流程",
|
|||
|
|
"B": "返回包含 error 字段的 JSON 字符串",
|
|||
|
|
"C": "自动注册该工具并返回空结果",
|
|||
|
|
"D": "从 tool_registry 中删除该条目后重试"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "代码中 if name not in tool_registry 分支直接返回 json.dumps({\"error\": f\"Unknown tool: {name}\"})，即返回一个描述错误信息的 JSON 字符串，不会抛出异常。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 3,
|
|||
|
|
"type": "short_answer",
|
|||
|
|
"question": "当前 agent_loop 在执行完工具调用后，将结果反馈给 LLM 生成最终回复。请指出这种设计中可能存在的一个安全风险，并简要说明如何缓解。",
|
|||
|
|
"answer": "LLM 可能产生幻觉，在最终回复中编造不存在的工具调用结果",
|
|||
|
|
"keywords": [
|
|||
|
|
"prompt injection",
|
|||
|
|
"工具结果注入",
|
|||
|
|
"结果篡改",
|
|||
|
|
"二次验证",
|
|||
|
|
"权限控制",
|
|||
|
|
"沙箱"
|
|||
|
|
],
|
|||
|
|
"scoring_rubric": "答出工具结果可能被恶意利用（如 prompt injection）或 LLM 可能忽略/篡改工具返回内容得 2 分；提出缓解措施（如对工具返回值做转义、限制 LLM 只能引用原始返回、结果二次验证等）再得 2 分；总分 4 分。",
|
|||
|
|
"explanation": "工具返回的内容被直接拼入 prompt，攻击者可能通过操控工具返回来实施 prompt injection，或 LLM 可能忽略/错误引用工具结果。"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"explanation": "这段代码展示了一个经典的 Tool Calling Agent 架构：工具注册 -> LLM 判断 -> 工具执行 -> 结果回传。核心要点包括：1) 工具通过注册表管理，支持动态扩展；2) call_tool 包含参数校验和异常兜底，保证鲁棒性；3) 串行执行工具调用在简单场景够用，但高延迟场景需要并发优化；4) 工具结果直接拼入 prompt 可能带来 prompt injection 风险。",
|
|||
|
|
"source": null,
|
|||
|
|
"related": []
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"id": "cr-002",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"difficulty": 4,
|
|||
|
|
"tags": [
|
|||
|
|
"ai-agent",
|
|||
|
|
"context-engineering",
|
|||
|
|
"harness-engineering"
|
|||
|
|
],
|
|||
|
|
"question": "以下是一个 RAG（检索增强生成）流程的核心实现，包含向量检索、上下文拼接和最终生成。请阅读代码并回答子问题。",
|
|||
|
|
"code": "from dataclasses import dataclass\nfrom typing import Optional\n\n@dataclass\nclass RetrievalResult:\n content: str\n score: float\n source: str\n\ndef retrieve(\n query: str,\n vector_store, # 向量数据库客户端\n top_k: int = 5,\n score_threshold: float = 0.7,\n) -> list[RetrievalResult]:\n \"\"\"从向量库中检索相关文档片段\"\"\"\n embeddings = vector_store.encode(query)\n raw_results = vector_store.search(embeddings, k=top_k * 2) # 过量召回\n\n # 过滤低分结果并去重\n seen = set()\n filtered = []\n for doc, score in raw_results:\n if score < score_threshold:\n continue\n dedup_key = hash(doc.content[:100])\n if dedup_key in seen:\n continue\n seen.add(dedup_key)\n filtered.append(RetrievalResult(doc.content, score, doc.source))\n return filtered[:top_k]\n\ndef build_prompt(\n query: str,\n contexts: list[RetrievalResult],\n max_context_tokens: int = 3000,\n) -> str:\n \"\"\"将检索结果拼接进 LLM prompt\"\"\"\n header = \"根据以下参考资料回答用户问题。如果资料不足请说明。\\n\\n\"\n context_parts = []\n token_budget = max_context_tokens\n\n for i, ctx in enumerate(contexts):\n chunk = f\"[参考{i+1}] (来源: {ctx.source}, 相关度: {ctx.score:.2f})\\n{ctx.content}\\n\"\n # 粗略估算 token 数（1 中文 ≈ 2 token）\n est_tokens = len(chunk) * 2\n if est_tokens > token_budget:\n break\n context_parts.append(chunk)\n token_budget -= est_tokens\n\n context_block = \"\\n\".join(context_parts)\n return f\"{header}--- 参考资料 ---\\n{context_block}\\n---\\n\\n用户问题: {query}\\n\"",
|
|||
|
|
"language": "python",
|
|||
|
|
"sub_questions": [
|
|||
|
|
{
|
|||
|
|
"index": 1,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "retrieve 函数中为什么要用 top_k * 2 进行过量召回（oversampling）？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "为了提高向量搜索的速度",
|
|||
|
|
"B": "因为后续的分数过滤和去重可能淘汰部分结果，需预留余量",
|
|||
|
|
"C": "为了减少向量数据库的查询次数",
|
|||
|
|
"D": "为了让 LLM 有更多 token 可用"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "过量召回是为了给后续的 score_threshold 过滤和 hash 去重留出余量。如果恰好召回 top_k 条，经过过滤后可能不足 top_k 条有效结果，导致返回结果数少于预期。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 2,
|
|||
|
|
"type": "short_answer",
|
|||
|
|
"question": "build_prompt 使用了 token_budget 机制来控制上下文长度。请说明这种策略的潜在问题，并提出至少一种改进方案。",
|
|||
|
|
"answer": "粗略的字符数估算不够精确，可能严重高估或低估实际 token 数",
|
|||
|
|
"keywords": [
|
|||
|
|
"token 计算",
|
|||
|
|
"字符估算",
|
|||
|
|
"tiktoken",
|
|||
|
|
"tokenizer",
|
|||
|
|
"动态截断",
|
|||
|
|
"压缩",
|
|||
|
|
"摘要"
|
|||
|
|
],
|
|||
|
|
"scoring_rubric": "指出字符级估算的不准确性（如中英文混合、标点符号差异）得 2 分；提出改进方案（使用 tiktoken 精确计算、对长 chunk 做摘要压缩、按语义段落截断等）得 2 分；总分 4 分。",
|
|||
|
|
"explanation": "字符级 token 估算在中英文混合场景误差大，应使用 tokenizer 库精确计算。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 3,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "以下关于这段 RAG 代码的说法，哪一项最准确地描述了它在生产环境中的局限？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "代码没有调用 LLM，所以无法生成最终回答",
|
|||
|
|
"B": "缺少对检索结果的相关性重排序（re-ranking），在噪声文档较多时检索质量会下降",
|
|||
|
|
"C": "代码使用了同步调用，无法支持并发请求",
|
|||
|
|
"D": "vector_store 的 search 方法不支持批量查询"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "这段代码仅靠向量相似度排序，没有使用 cross-encoder 或 LLM-based re-ranking 对结果进行精排。在实际生产中，向量检索的初始召回往往包含噪声，re-ranking 可以显著提升最终送入 LLM 的上下文质量。"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"explanation": "这段 RAG 代码涵盖了三个核心阶段：1) 过量召回 + 过滤去重的检索策略，保证结果质量；2) 基于 token budget 的上下文拼接，防止超出模型窗口限制；3) 结构化 prompt 模板引导 LLM 利用参考资料。生产级 RAG 还需考虑 re-ranking、query rewriting、混合检索（向量+关键词）、上下文压缩等优化。",
|
|||
|
|
"source": null,
|
|||
|
|
"related": []
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"id": "cr-003",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"difficulty": 3,
|
|||
|
|
"tags": [
|
|||
|
|
"mcp",
|
|||
|
|
"ai-agent",
|
|||
|
|
"tool-calling"
|
|||
|
|
],
|
|||
|
|
"question": "以下是一个 MCP（Model Context Protocol）Server 的 Python 实现，展示了如何注册和提供 Tool 给外部 Agent 调用。请阅读代码并回答子问题。",
|
|||
|
|
"code": "from mcp.server import Server\nfrom mcp.types import Tool, TextContent\nimport json, os\n\nserver = Server(\"weather-server\")\n\n# 定义 MCP Tool 的元信息和参数 schema\nWEATHER_TOOL = Tool(\n name=\"get_weather\",\n description=\"获取指定城市的当前天气信息\",\n inputSchema={\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\", \"description\": \"城市名称，如 Beijing\"},\n \"unit\": {\n \"type\": \"string\",\n \"enum\": [\"celsius\", \"fahrenheit\"],\n \"description\": \"温度单位\",\n \"default\": \"celsius\",\n },\n },\n \"required\": [\"city\"],\n },\n)\n\n# 注册可用工具列表\n@server.list_tools()\nasync def list_tools() -> list[Tool]:\n return [WEATHER_TOOL]\n\n# 处理工具调用请求\n@server.call_tool()\nasync def handle_call(name: str, arguments: dict) -> list[TextContent]:\n if name != \"get_weather\":\n raise ValueError(f\"Unsupported tool: {name}\")\n city = arguments[\"city\"]\n unit = arguments.get(\"unit\", \"celsius\")\n # 模拟调用外部天气 API\n api_key = os.environ.get(\"WEATHER_API_KEY\")\n if not api_key:\n return [TextContent(type=\"text\", text=json.dumps({\"error\": \"Missing API key\"}))]\n weather_data = fetch_weather_from_api(city, unit, api_key) # 外部 API 调用\n return [TextContent(type=\"text\", text=json.dumps(weather_data))]",
|
|||
|
|
"language": "python",
|
|||
|
|
"sub_questions": [
|
|||
|
|
{
|
|||
|
|
"index": 1,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "MCP Server 中 inputSchema 的作用是什么？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "定义 MCP Server 自身的配置参数",
|
|||
|
|
"B": "描述 Tool 接受的输入参数结构，供调用方（如 LLM）生成符合格式的请求",
|
|||
|
|
"C": "定义 Tool 返回结果的数据格式",
|
|||
|
|
"D": "限制调用方只能调用该 Tool 一次"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "inputSchema 是 JSON Schema 格式的参数描述，MCP Client（通常是 LLM Agent）会读取这个 schema 来了解 tool 需要哪些参数、参数类型和是否必填，从而生成正确的调用参数。这与 OpenAI Function Calling 中的 parameters 字段作用一致。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 2,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "handle_call 函数返回的是 list[TextContent] 而非单个字符串，这样设计的主要原因是？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "为了支持异步并发调用",
|
|||
|
|
"B": "为了兼容 MCP 协议中一次 Tool 调用可以返回多个内容块的设计",
|
|||
|
|
"C": "因为 Python 的 json.dumps 只能处理列表",
|
|||
|
|
"D": "为了隐藏实际的返回数据"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "MCP 协议设计上支持 Tool 返回多种类型的内容块（文本、图片、嵌入数据等），所以返回类型是 list[Content]。当前只返回 TextContent 是最简单的情况，但如果需要返回混合内容（如图表+文本），这种设计提供了扩展性。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 3,
|
|||
|
|
"type": "short_answer",
|
|||
|
|
"question": "在生产环境中，这个 MCP Server 缺少哪些关键的健壮性保障措施？请列举至少两项。",
|
|||
|
|
"answer": "缺少输入参数校验、请求限流、API Key 安全管理、超时控制、调用日志记录、错误重试",
|
|||
|
|
"keywords": [
|
|||
|
|
"参数校验",
|
|||
|
|
"限流",
|
|||
|
|
"rate limit",
|
|||
|
|
"超时",
|
|||
|
|
"timeout",
|
|||
|
|
"日志",
|
|||
|
|
"logging",
|
|||
|
|
"重试",
|
|||
|
|
"安全",
|
|||
|
|
"认证"
|
|||
|
|
],
|
|||
|
|
"scoring_rubric": "每正确列举一项健壮性措施并简要说明得 1.5 分，最高 4 分。示例：缺少参数校验（应验证 city 非空且长度合理）1.5 分；缺少限流（应防止恶意高频调用）1.5 分；缺少超时控制（外部 API 调用应有 timeout）1.5 分；缺少调用日志（应记录每次调用的入参和结果）1.5 分。",
|
|||
|
|
"explanation": "生产级 MCP Server 需要完整的健壮性保障，包括参数校验、限流、超时、日志等。"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"explanation": "这段代码展示了 MCP Server 的核心模式：1) 通过 Tool 元信息（name、description、inputSchema）声明式地注册工具能力；2) 使用装饰器模式分别处理 list_tools 和 call_tool 请求；3) 返回标准化的 Content 类型。MCP 的关键设计是将 Tool 定义与实现分离，使得 Agent 可以在运行时动态发现和调用 Tool，实现即插即用的能力扩展。",
|
|||
|
|
"source": null,
|
|||
|
|
"related": []
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"id": "cr-004",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"difficulty": 3,
|
|||
|
|
"tags": [
|
|||
|
|
"guardrail",
|
|||
|
|
"ai-agent",
|
|||
|
|
"llm-ops"
|
|||
|
|
],
|
|||
|
|
"question": "以下是一个 LLM 输出 Guardrail 过滤器的 Python 实现，用于在将 LLM 回复发送给用户之前进行安全检查。请阅读代码并回答子问题。",
|
|||
|
|
"code": "import re\nfrom dataclasses import dataclass, field\nfrom enum import Enum\n\nclass SafetyLevel(Enum):\n SAFE = \"safe\"\n WARNING = \"warning\"\n BLOCKED = \"blocked\"\n\n@dataclass\nclass GuardrailResult:\n level: SafetyLevel\n original: str\n sanitized: str = \"\"\n violations: list[str] = field(default_factory=list)\n\ndef check_pii(text: str) -> list[str]:\n \"\"\"检测文本中的个人身份信息（PII）\"\"\"\n violations = []\n # 检测手机号（中国大陆 11 位）\n if re.search(r'1[3-9]\\d{9}', text):\n violations.append(\"phone_number\")\n # 检测身份证号（18 位）\n if re.search(r'\\d{17}[\\dXx]', text):\n violations.append(\"id_card\")\n # 检测邮箱地址\n if re.search(r'[\\w.-]+@[\\w.-]+\\.\\w+', text):\n violations.append(\"email\")\n return violations\n\ndef check_prompt_leakage(text: str) -> list[str]:\n \"\"\"检测是否泄露系统提示词\"\"\"\n leak_patterns = [\n r'(?i)system\\s*prompt',\n r'(?i)you\\s+are\\s+a\\s+',\n r'(?i)ignore\\s+(previous|above|all)\\s+instructions',\n ]\n return [\"prompt_leakage\"] if any(re.search(p, text) for p in leak_patterns) else []\n\ndef apply_guardrails(output: str, strict: bool = False) -> GuardrailResult:\n \"\"\"应用所有安全检查规则，返回过滤结果\"\"\"\n violations = []\n violations.extend(check_pii(output))\n violations.extend(check_prompt_leakage(output))\n\n if not violations:\n return GuardrailResult(SafetyLevel.SAFE, output)\n # PII 敏感信息直接脱敏\n sanitized = re.sub(r'1[3-9]\\d{9}', '[手机号已脱敏]', output)\n sanitized = re.sub(r'\\d{17}[\\dXx]', '[身份证已脱敏]', sanitized)\n sanitized = re.sub(r'[\\w.-]+@[\\w.-]+\\.\\w+', '[邮箱已脱敏]', sanitized)\n # 在非严格模式下，PII 只警告不拦截；prompt leakage 始终拦截\n has_leakage = \"prompt_leakage\" in violations\n if has_leakage or strict:\n return GuardrailResult(SafetyLevel.BLOCKED, output, \"\", violations)\n return GuardrailResult(SafetyLevel.WARNING, output, sanitized, violations)",
|
|||
|
|
"language": "python",
|
|||
|
|
"sub_questions": [
|
|||
|
|
{
|
|||
|
|
"index": 1,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "当 strict=False 且 LLM 输出仅包含 PII 信息（不含 prompt leakage）时，apply_guardrails 的返回结果是？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "level=BLOCKED，sanitized 为空字符串",
|
|||
|
|
"B": "level=WARNING，sanitized 包含脱敏后的文本",
|
|||
|
|
"C": "level=SAFE，原样返回",
|
|||
|
|
"D": "抛出异常，要求必须在 strict 模式下运行"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "代码中，当 strict=False 且没有 prompt_leakage 时，走 else 分支返回 GuardrailResult(SafetyLevel.WARNING, output, sanitized, violations)，level 为 WARNING，sanitized 包含脱敏后的文本。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 2,
|
|||
|
|
"type": "short_answer",
|
|||
|
|
"question": "当前的 check_pii 函数使用正则表达式检测手机号。请指出这种方法在实际生产中可能遗漏的一个 PII 类型，并简要说明为什么它难以用简单正则捕获。",
|
|||
|
|
"answer": "自然语言中嵌入的地址信息难以用正则捕获",
|
|||
|
|
"keywords": [
|
|||
|
|
"地址",
|
|||
|
|
"姓名",
|
|||
|
|
"自然语言",
|
|||
|
|
"NER",
|
|||
|
|
"上下文",
|
|||
|
|
"语义理解",
|
|||
|
|
"NLP"
|
|||
|
|
],
|
|||
|
|
"scoring_rubric": "指出一种难以用正则检测的 PII 类型（如地址、姓名、银行卡号、IP 地址等）得 2 分；解释为什么难以捕获（如需要语义理解、NER 模型、上下文推理等）得 2 分；总分 4 分。",
|
|||
|
|
"explanation": "地址、姓名等自然语言中的 PII 缺乏固定格式，需要 NER 模型进行语义级检测。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 3,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "关于这段 Guardrail 代码的设计，以下哪项评价最准确？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "使用了 LLM-as-Judge 方案，判断精度最高",
|
|||
|
|
"B": "基于规则的过滤方案，速度快但无法检测语义级别的安全问题",
|
|||
|
|
"C": "完全依赖外部 API 进行安全检查，延迟最低",
|
|||
|
|
"D": "使用了深度学习模型，对所有类型的内容都有效"
|
|||
|
|
},
|
|||
|
|
"answer": "B",
|
|||
|
|
"explanation": "该 Guardrail 使用正则表达式和关键词匹配等规则方法，优点是延迟低、可控性强、可解释性好，但缺点是无法处理语义级别的问题（如隐晦的有害内容、方言表达、上下文相关的攻击等），这些需要 NLU 模型或 LLM-based guardrail 来补充。"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"explanation": "这段 Guardrail 代码展示了生产级 LLM 输出过滤的常见模式：1) 分层检查（PII + Prompt Leakage 分开检测）；2) 脱敏而非拦截（对 PII 优先脱敏而非直接阻断，平衡安全与用户体验）；3) 严格/宽松模式切换（strict 模式用于高安全场景）。实际生产中，还需补充有害内容检测（toxicity）、幻觉检测、事实性校验等模块，通常会组合使用规则引擎 + 分类模型 + LLM Judge 的多层架构。",
|
|||
|
|
"source": null,
|
|||
|
|
"related": []
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"id": "cr-005",
|
|||
|
|
"type": "code_reading",
|
|||
|
|
"difficulty": 4,
|
|||
|
|
"tags": [
|
|||
|
|
"llm-ops",
|
|||
|
|
"ai-agent",
|
|||
|
|
"harness-engineering"
|
|||
|
|
],
|
|||
|
|
"question": "以下是一个 AI Agent 调用 LLM 时的重试与降级机制的 Python 实现。当主模型调用失败或返回异常时，系统会自动重试或切换到备用模型。请阅读代码并回答子问题。",
|
|||
|
|
"code": "import time\nimport logging\nfrom dataclasses import dataclass\nfrom typing import Optional\n\nlogger = logging.getLogger(__name__)\n\n@dataclass\nclass LLMConfig:\n provider: str # 如 \"openai\", \"anthropic\", \"local\"\n model: str # 如 \"gpt-4o\", \"claude-3\"\n api_key: str\n max_retries: int = 2\n timeout: float = 30.0\n\ndef call_llm_with_retry(\n prompt: str,\n configs: list[LLMConfig], # 按优先级排列的模型配置列表\n fallback_response: str = \"抱歉，服务暂时不可用，请稍后重试。\",\n) -> str:\n \"\"\"带重试和降级的 LLM 调用\"\"\"\n last_error = None\n\n for config in configs:\n for attempt in range(config.max_retries + 1):\n try:\n logger.info(f\"Calling {config.provider}/{config.model} (attempt {attempt + 1})\")\n response = _call_provider(config, prompt)\n\n # 检查响应质量（空内容或错误标记）\n if not response or response.strip() == \"\":\n raise ValueError(\"Empty response from LLM\")\n if response.startswith(\"[ERROR]\"):\n raise ValueError(f\"LLM returned error: {response}\")\n\n logger.info(f\"Success with {config.provider}/{config.model}\")\n return response\n\n except Exception as e:\n last_error = e\n logger.warning(\n f\"{config.provider}/{config.model} failed \"\n f\"(attempt {attempt + 1}): {e}\"\n )\n # 指数退避：等待 1s, 2s, 4s ...\n if attempt < config.max_retries:\n wait = 2 ** attempt\n logger.info(f\"Retrying in {wait}s...\")\n time.sleep(wait)\n\n # 当前 provider 的所有重试耗尽，切换到下一个\n logger.warning(f\"All retries exhausted for {config.provider}, falling back...\")\n\n # 所有 provider 都失败，返回兜底响应\n logger.error(f\"All LLM providers failed. Last error: {last_error}\")\n return fallback_response\n\ndef _call_provider(config: LLMConfig, prompt: str) -> str:\n \"\"\"实际调用 LLM provider（此处为占位实现）\"\"\"\n if config.provider == \"openai\":\n # 实际会调用 OpenAI API\n return _call_openai(config.model, prompt, config.api_key, config.timeout)\n elif config.provider == \"anthropic\":\n return _call_anthropic(config.model, prompt, config.api_key, config.timeout)\n else:\n raise ValueError(f\"Unknown provider: {config.provider}\")",
|
|||
|
|
"language": "python",
|
|||
|
|
"sub_questions": [
|
|||
|
|
{
|
|||
|
|
"index": 1,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "代码中使用了 2 ** attempt 作为重试间隔。当某个 provider 的 max_retries=3 时，前四次尝试（含首次）的等待时间序列是？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "0s, 1s, 2s, 4s",
|
|||
|
|
"B": "1s, 2s, 4s, 8s",
|
|||
|
|
"C": "1s, 1s, 1s, 1s",
|
|||
|
|
"D": "0s, 0s, 0s, 0s"
|
|||
|
|
},
|
|||
|
|
"answer": "A",
|
|||
|
|
"explanation": "attempt 从 0 开始：第 1 次 attempt=0（首次调用，不等待），第 2 次 attempt=0 失败后 wait=2^0=1s，第 3 次 attempt=1 失败后 wait=2^1=2s，第 4 次 attempt=2 失败后 wait=2^2=4s。所以等待序列为 0s, 1s, 2s, 4s。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 2,
|
|||
|
|
"type": "single_choice",
|
|||
|
|
"question": "当 configs 列表中有 2 个 provider 且都调用失败时，函数最终返回什么？",
|
|||
|
|
"options": {
|
|||
|
|
"A": "抛出异常，不返回任何值",
|
|||
|
|
"B": "返回最后一个 provider 最后一次调用的原始异常信息",
|
|||
|
|
"C": "返回 fallback_response 参数指定的兜底文本",
|
|||
|
|
"D": "返回空字符串"
|
|||
|
|
},
|
|||
|
|
"answer": "C",
|
|||
|
|
"explanation": "两个 for 循环（provider 循环 + attempt 循环）都执行完后，函数执行最后的 return fallback_response，返回调用方传入的兜底响应文本。这是防御性编程的典型模式——永远给用户一个有意义的回复。"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"index": 3,
|
|||
|
|
"type": "short_answer",
|
|||
|
|
"question": "当前实现使用了 time.sleep() 进行同步阻塞等待。如果这个函数在异步 Web 服务中被调用，会有什么问题？请提出一种不阻塞事件循环的替代方案。",
|
|||
|
|
"answer": "time.sleep 会阻塞整个事件循环，导致同进程内其他并发请求无法处理",
|
|||
|
|
"keywords": [
|
|||
|
|
"阻塞",
|
|||
|
|
"事件循环",
|
|||
|
|
"asyncio",
|
|||
|
|
"异步",
|
|||
|
|
"await",
|
|||
|
|
"asyncio.sleep",
|
|||
|
|
"线程池"
|
|||
|
|
],
|
|||
|
|
"scoring_rubric": "正确指出 time.sleep 会阻塞事件循环导致并发能力丧失得 2 分；提出 asyncio.sleep 或将重试逻辑放入线程池（run_in_executor）等非阻塞方案得 2 分；总分 4 分。",
|
|||
|
|
"explanation": "同步阻塞等待在异步框架中是致命的，会严重影响服务并发能力，应使用 asyncio.sleep 替代。"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"explanation": "这段代码展示了生产级 LLM 调用的重试与降级策略：1) 多级 fallback：按配置优先级依次尝试不同 provider；2) 指数退避：避免失败时立即重试造成 provider 过载；3) 响应质量检查：不仅捕获异常，还检查空响应和错误标记；4) 兜底响应：所有渠道失败时仍给用户一个有意义的回复。实际生产中还需考虑 circuit breaker（熔断器）、成功率监控、不同 provider 的 token 计费差异、以及异步化改造以适配高并发场景。",
|
|||
|
|
"source": null,
|
|||
|
|
"related": []
|
|||
|
|
}
|
|||
|
|
]
|
|||
|
|
}
|