Compare commits
50 Commits
ac11aa0f99
...
feat/mnn-s
| Author | SHA1 | Date | |
|---|---|---|---|
|
c47e62b658 | ||
|
|
0fd469f5c0 | ||
|
|
ef5364d8c3 | ||
|
|
198570186e | ||
|
|
32180d7c0e | ||
|
|
558682c8f8 | ||
|
|
404abbf10b | ||
|
|
e179a369f6 | ||
|
|
30f75dc2cd | ||
|
|
abacf5c4f5 | ||
|
|
52db6fb85a | ||
|
|
de19d7abcd | ||
|
|
b3777d508d | ||
|
|
9d856fcfc4 | ||
|
|
6c6a950140 | ||
|
|
f58d6064ba | ||
|
|
c3f8ec400c | ||
|
|
69de5faf4b | ||
|
|
477a64ecb4 | ||
|
|
6405733358 | ||
|
|
2e27677f80 | ||
|
|
2e90139df7 | ||
|
|
77139f5e32 | ||
|
|
0dd60d6021 | ||
|
|
43cdde9bab | ||
|
|
0a824610cf | ||
|
|
7e8e692695 | ||
|
|
3f9a2af279 | ||
|
|
a65c63947b | ||
|
|
8494e51823 | ||
|
|
070e016f81 | ||
|
|
8c8599e77d | ||
|
|
b7e8ab33ec | ||
|
|
db327afd79 | ||
|
|
5eb724ab86 | ||
|
|
cfeb25247a | ||
|
|
26a7d53b1b | ||
|
|
e603738330 | ||
|
|
7f0a76098a | ||
|
|
b79ae54b7b | ||
|
|
ca5a3fa38b | ||
|
|
836f3d4234 | ||
|
|
b919404412 | ||
|
|
ddfd474bb3 | ||
|
|
cbacd9461a | ||
|
|
39b1521f00 | ||
|
|
9da3fbc87e | ||
|
|
f6c0ba7077 | ||
|
|
afc6a79dd7 | ||
|
|
06484d09ff |
73
.agents/skills/vibe-submission-collector/SKILL.md
Normal file
73
.agents/skills/vibe-submission-collector/SKILL.md
Normal file
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: vibe-submission-collector
|
||||
description: Collect Vibe-a-thon contestant information and optional gift-shipping details through staged Chinese Q&A, isolate private fulfillment data, surface missing scoring evidence without blocking optional omissions, and generate a consistently organized submission folder. Use when contestants prepare or revise Top 30 materials, organizers need standardized packages or gift recipient data, or video and files over 10 MB must be represented by shareable links.
|
||||
---
|
||||
|
||||
# Vibe Submission Collector
|
||||
|
||||
Create one standardized submission folder without requiring contestants to design their own document structure.
|
||||
|
||||
## Required resources
|
||||
|
||||
Read [references/submission-spec.md](references/submission-spec.md) completely before interviewing or editing a package. Read only the matching entries in [references/field-help.md](references/field-help.md) when a contestant asks what a field means or how to answer it. Use [scripts/submission_tool.py](scripts/submission_tool.py) to initialize, render, and validate folders.
|
||||
|
||||
## Workflow
|
||||
|
||||
Use an available Python 3 interpreter for the bundled script: prefer `python3`; if that command is unavailable, use `python`. Replace `<python>` in the commands below with the working command.
|
||||
|
||||
1. Ask for the output root and the organizer-provided submission ID. If no ID exists, derive a filesystem-safe ID from the team or project name and ask the contestant to confirm it.
|
||||
2. Initialize the folder:
|
||||
|
||||
```bash
|
||||
<python> <skill-dir>/scripts/submission_tool.py init --root <output-root> --id <submission-id> [--title <optional-project-title>]
|
||||
```
|
||||
|
||||
3. Interview in the rounds defined by the specification. Ask at most five questions per round. Accept rough notes, draft concise answers for the contestant, and require confirmation before recording claims as final.
|
||||
4. Edit `00-submission.json` as the canonical source for submission and scoring information. Edit gift-delivery personal data only in `private/09-gift-shipping.json`. Preserve schema versions and keys. Use empty strings, empty arrays, or `null` for unavailable information; never invent evidence, users, test results, links, or technical usage. Certificate name, mobile number, participation mode, team-member certificate names when applicable, and mandatory declarations must be completed; other unanswered fields do not affect package eligibility.
|
||||
5. For video, source archives, datasets, and other large files, request a stable cloud-drive or streaming URL with review access. Do not copy large files into the package.
|
||||
6. Never request or store API keys, personal passwords, production credentials, or real-user data unrelated to the submission. If a Demo needs authentication, ask for a dedicated temporary test account and record it only in `private/08-test-access.md`. Store gift recipient name, phone, and shipping address only in `private/09-gift-shipping.json`. Never repeat either private file in generated summaries or review outputs.
|
||||
7. Render all human-readable files after each completed interview round:
|
||||
|
||||
```bash
|
||||
<python> <skill-dir>/scripts/submission_tool.py render <submission-folder>
|
||||
```
|
||||
|
||||
8. Run validation before handoff:
|
||||
|
||||
```bash
|
||||
<python> <skill-dir>/scripts/submission_tool.py validate <submission-folder>
|
||||
```
|
||||
|
||||
9. Resolve every validation error. Validation errors are limited to package integrity, mandatory organizer-contact/certificate fields, and mandatory declarations. Report scoring-impact warnings explicitly, show the contestant a concise claim-and-link checklist, and obtain final confirmation.
|
||||
10. Only after the contestant explicitly confirms the final submission information, send this fixed handoff reminder:
|
||||
|
||||
> 请通过邮件将整理好的信息包发回给组织方：**gdghangzhou@163.com**。如果文件夹太大，可以先上传云网盘，然后发送链接。（请选手自行确保发送的链接可以公开访问）
|
||||
|
||||
If a publicly accessible cloud-drive link is used, exclude the entire `private/` directory from that public package. Send `private/08-test-access.md` and `private/09-gift-shipping.json`, when present, to the organizer as separate email attachments or through a separate access-controlled link. Do not show the handoff reminder before final confirmation.
|
||||
|
||||
## Interview behavior
|
||||
|
||||
- Communicate in Chinese unless the contestant requests another language.
|
||||
- Ask with field labels and direct questions only. Do not proactively display field definitions, explanatory notes, examples, or the help catalog.
|
||||
- If the contestant asks what a field means, why it is needed, or how to fill it, look up only that field in `references/field-help.md`, explain it briefly, then resume the current question.
|
||||
- Explain why a question affects review when the contestant is unsure.
|
||||
- Prefer concrete prompts such as “法国首次支付用户” over broad prompts such as “海外用户”.
|
||||
- Separate facts, contestant claims, and unvalidated assumptions.
|
||||
- Do not collect Vibe Coding chat logs or Prompt histories. Prompt or Agent workflow evidence is optional only when it is itself a core technical mechanism.
|
||||
- Do not use likes, views, favorites, or other propagation metrics as review evidence.
|
||||
- Do not treat polished prose as evidence. Ask for a link, file, timecode, code path, screenshot, test record, or explicit “not yet validated”.
|
||||
- Do not block handoff when scoring information is absent. Explain which judging dimensions cannot receive adequate evidence and preserve the omission as a warning.
|
||||
|
||||
## Output contract
|
||||
|
||||
Produce exactly the required filenames from the specification. Keep participant materials under one `<submission-id>/` folder. Do not create alternative summaries, README files, or renamed variants. `00-submission.json` is authoritative; all Markdown files are generated views.
|
||||
|
||||
Finish by reporting:
|
||||
|
||||
- absolute folder path;
|
||||
- validation result;
|
||||
- missing or unverified evidence;
|
||||
- links that still need permission checks;
|
||||
- whether a dedicated test account was supplied securely.
|
||||
|
||||
After this report and the contestant's explicit final confirmation, append the fixed email handoff reminder from Workflow step 10.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Vibe 选手材料收集"
|
||||
short_description: "分轮收集必填联系与证书信息，按需解释字段并生成统一材料包"
|
||||
default_prompt: "使用 $vibe-submission-collector 通过分轮问答帮我收集参赛资料和礼品邮寄信息。"
|
||||
@@ -0,0 +1,158 @@
|
||||
# 选手字段按需解释
|
||||
|
||||
本文件仅供问答助手按需查询。默认不要向选手展示字段说明、示例或整份目录。选手询问某个字段时，只解释对应字段，然后继续当前问题。
|
||||
|
||||
## 目录
|
||||
|
||||
1. 基础与联系信息
|
||||
2. 礼品邮寄信息
|
||||
3. 用户问题
|
||||
4. 解决方案与核心功能
|
||||
5. Demo 与体验
|
||||
6. Google 技术应用
|
||||
7. 创新性
|
||||
8. 出海落地与本地化
|
||||
9. 演示视频
|
||||
10. 原创性与访问声明
|
||||
|
||||
## 基础与联系信息
|
||||
|
||||
- **作品编号**：组委会用于唯一识别材料包的编号；没有编号时可由项目或选手名称生成一个文件名安全的编号。
|
||||
- **作品或产品名称**：作品对外展示的名称；尚未确定可以暂不提供。
|
||||
- **团队名称**：团队对外使用的名称，不等于团队成员姓名；没有团队名称可留空。
|
||||
- **选手姓名（获奖证书）**：证书上需要印制的主联系人真实姓名，请确认汉字、英文名和大小写准确；必填。
|
||||
- **手机号**：组委会联系选手、确认入围和颁奖信息使用的常用手机号码；必填。
|
||||
- **微信号**：可被搜索或添加的微信账号，不是微信昵称或二维码；选填。
|
||||
- **邮箱**：能够正常收取赛事通知的常用邮箱；选填。
|
||||
- **是否团队参赛**：选择个人参赛或团队参赛，用于决定是否必须填写团队成员信息；必填。
|
||||
- **团队成员姓名（获奖证书）**：团队参赛时，填写所有需要出现在获奖证书上的成员姓名并确认顺序和拼写；团队参赛必填。
|
||||
- **参赛赛道**：从“真、善、美”中选择最符合作品价值方向的一项。
|
||||
- **当前状态**：说明作品现在处于概念、原型、可运行 Demo、试运行或已发布等哪个阶段。
|
||||
- **一句话介绍**：用一句话说清服务谁、在什么场景使用、解决什么问题。
|
||||
- **目标国家或地区**：作品计划服务或首先验证的海外国家、地区或文化市场。
|
||||
- **产品使用语言**：产品界面、内容或交互当前支持的语言；选填。
|
||||
- **小红书作品链接**：参赛作品在小红书发布后的可访问链接，不需要填写点赞或播放数据。
|
||||
|
||||
## 礼品邮寄信息
|
||||
|
||||
- **礼品收件人姓名**：实际签收赛事礼品的人名，可以与选手证书姓名不同。
|
||||
- **礼品收件电话**：快递员联系收件人使用的电话号码，可以与参赛联系人手机号不同。
|
||||
- **礼品邮寄地址**：能够正常收件的完整地址，建议包含省、市、区、街道、门牌号和必要的公司或园区信息。
|
||||
|
||||
礼品邮寄信息为受限个人信息，只用于组委会履约寄送，不进入评审材料。
|
||||
|
||||
## 用户问题
|
||||
|
||||
- **具体目标用户**：实际会使用产品的一类明确人群，避免只写“海外用户”或“所有人”。
|
||||
- **典型使用场景**：目标用户在什么时间、地点、任务或情境下遇到问题并使用产品。
|
||||
- **用户当前如何解决问题**：没有本作品时，用户现在使用的人工方法、工具、产品或替代方案。
|
||||
- **现有方式的具体不足**：当前方案在成本、效率、准确性、可访问性、体验或文化适配上的缺陷。
|
||||
- **核心用户痛点**：对用户影响最大、最值得优先解决的困难。
|
||||
- **用户问题证据**：访谈、反馈、公开资料、数据、真实案例或观察记录；没有证据可以明确填写“尚未验证”。
|
||||
- **访谈或测试人数**：实际参加访谈或体验测试的人数，不包含团队内部未经说明的成员。
|
||||
- **代表性用户反馈**：最能说明问题或产品价值的一条真实反馈，必要时做匿名和脱敏处理。
|
||||
- **尚未验证的假设**：团队当前相信但还没有通过用户或数据证实的判断。
|
||||
|
||||
## 解决方案与核心功能
|
||||
|
||||
- **解决方案概述**：说明产品用什么方式解决前述问题，不需要写市场宣传口号。
|
||||
- **用户价值**：用户使用产品后能够获得的具体改善或结果。
|
||||
- **完整用户流程**：从用户进入产品、完成关键操作到得到结果的连续步骤。
|
||||
- **核心功能**：直接支撑产品主要价值的功能，最多填写五项，辅助设置或装饰功能不必列入。
|
||||
- **功能名称**：便于评委识别该功能的简短名称。
|
||||
- **功能输入**：用户、系统或外部数据提供给功能的内容。
|
||||
- **系统处理**：功能对输入执行的关键处理或 AI 能力。
|
||||
- **功能输出**：用户最终能够看到、下载或使用的结果。
|
||||
- **可运行状态**：说明功能目前完整可用、部分可用、模拟演示或尚未实现。
|
||||
- **功能证据位置**：能够验证功能的 Demo 步骤、视频时间点、截图或代码位置。
|
||||
|
||||
## Demo 与体验
|
||||
|
||||
- **在线 Demo 链接**：评审可以直接打开并体验作品的网址。
|
||||
- **可运行文件链接**：无法在线体验时，提供安装包或运行文件的网盘链接；超过 10 MB 不要直接放入材料包。
|
||||
- **已发布渠道**：产品已经正式发布的平台或入口，例如 App Store、小红书小程序或网页应用。
|
||||
- **已发布产品名称**：评委在发布渠道中需要搜索的准确名称。
|
||||
- **访问方式**：说明评委通过浏览器、手机 App、桌面软件、小程序或本地运行等哪种方式体验。
|
||||
- **推荐运行环境**：建议使用的设备、系统、浏览器版本、网络或其他环境条件。
|
||||
- **预计体验时间**：评委完成核心流程大约需要多少分钟。
|
||||
- **体验步骤**：评委需要依次执行的三至五个操作。
|
||||
- **每一步预期结果**：执行该操作后应该看到或得到的结果，用于判断 Demo 是否正常。
|
||||
- **已知问题和当前限制**：已经发现但尚未修复的故障、兼容性问题或功能边界。
|
||||
- **备用 Demo 或下载链接**：主链接失效时可使用的替代访问地址。
|
||||
- **评审期间保持可用确认**：确认链接、服务和测试账号将在评审期内可访问。
|
||||
- **专用临时测试账号**：仅为评委创建、与个人或生产账号隔离的登录账号。
|
||||
- **临时密码**：专用测试账号的临时密码，不得填写个人密码、生产密码或 API Key。
|
||||
- **账号有效期**：临时账号可以正常使用到哪个日期和时间。
|
||||
- **登录说明**：验证码、角色选择或特殊入口等评委登录时必须知道的步骤。
|
||||
|
||||
## Google 技术应用
|
||||
|
||||
- **Google 产品或技术名称**：实际使用的 Google 工具、平台、API、模型或开发框架名称。
|
||||
- **具体模型或能力**：使用的模型名称、版本或具体能力，例如多模态理解、结构化输出或 Firebase Authentication。
|
||||
- **用于哪个产品功能**：这项 Google 技术支撑的用户可见功能或系统能力。
|
||||
- **输入是什么**：发送给该技术的文本、图片、音频、数据或其他内容。
|
||||
- **进行了什么处理**：该技术在核心流程中执行的分析、生成、识别、存储或开发工作。
|
||||
- **输出是什么**：技术处理后返回并被产品使用的结果。
|
||||
- **核心能力或支撑能力**：核心表示移除后主要价值无法成立；支撑表示用于部署、存储、开发或非核心环节。
|
||||
- **为什么选择这项技术**：说明它与目标场景的匹配点，而不是只写“效果好”。
|
||||
- **移除后会影响什么**：说明不用这项技术时哪些功能失效或价值下降。
|
||||
- **技术证据位置**：能够核验使用情况的代码路径、架构图、截图、视频时间点或 Demo 行为。
|
||||
- **代码仓库**：可供核验实现的代码仓库链接；无法公开时可以说明原因并提交其他证据。
|
||||
- **评审版本或 Commit ID**：本次评审对应的固定代码版本，避免评审期间版本变化。
|
||||
- **部署依赖**：运行作品必须使用的第三方服务、数据、模型、平台或环境。
|
||||
- **技术架构图**：展示客户端、服务端、Google 技术、数据和主要调用关系的图。
|
||||
- **核心代码证据位置**：能够核验关键功能实现的文件、目录或代码片段位置。
|
||||
- **部署说明**：评委或技术人员理解作品如何运行、部署或复现所需的说明。
|
||||
- **保密限制说明**：无法公开部分代码、数据或配置时，说明限制范围和可提供的替代证据。
|
||||
|
||||
## 创新性
|
||||
|
||||
- **技术创新**：技术架构、AI 使用方式、工程实现或能力组合上与常见方案不同的部分。
|
||||
- **产品与体验创新**：问题定义、功能机制、交互或用户体验上与现有方案不同的部分。
|
||||
- **创新点名称**：概括创新内容的短标题。
|
||||
- **常见方案如何处理**：现有产品或通常做法解决同一问题的方式。
|
||||
- **本作品有什么不同**：与常见方案相比的明确差异，不要只写“更智能”。
|
||||
- **带来的实际效果**：该差异为用户、产品或技术实现带来的可观察改善。
|
||||
- **创新证据位置**：验证创新已经实现的 Demo、代码、截图、视频或材料位置。
|
||||
|
||||
## 出海落地与本地化
|
||||
|
||||
- **首个计划进入的国家或地区**：团队准备首先验证或落地的具体海外市场。
|
||||
- **为什么选择这个市场**：目标用户、问题强度、团队资源或市场条件与选择之间的关系。
|
||||
- **首批用户如何获得**：准备通过社区、合作伙伴、应用商店、内容平台或其他渠道接触第一批用户。
|
||||
- **用户使用条件**：使用产品需要的设备、网络、语言、支付、账号或环境条件。
|
||||
- **主要落地障碍**：进入目标市场可能遇到的法规、文化、渠道、成本、技术或信任问题。
|
||||
- **预计成本范围**：产品运行、获客、模型调用或服务用户的大致成本区间；选填。
|
||||
- **社会价值路径**：产品如何通过具体使用过程为目标人群带来改善；选填。
|
||||
- **影响指标**：未来用于判断产品是否真的产生价值的可衡量指标；选填。
|
||||
- **未来六个月计划**：团队接下来六个月准备验证、开发或落地的关键事项；选填。
|
||||
- **目标文化或地区**：本地化设计具体针对的文化群体或区域。
|
||||
- **当地用户习惯**：会影响产品内容、流程或交互的当地行为习惯。
|
||||
- **当地用户的情绪、顾虑或限制**：用户在该场景中的信任、隐私、安全、支付、无障碍或资源限制。
|
||||
- **语言翻译之外的产品适配**：除翻译外，对内容、UI、流程、支付、隐私、交互或信息结构做出的调整。
|
||||
- **本地化研究或测试证据**：当地用户反馈、访谈、公开研究、测试记录或其他依据。
|
||||
- **是否经过当地用户测试**：是否让目标地区或文化背景的真实用户体验过作品。
|
||||
- **根据测试做出的修改**：团队根据本地用户反馈实际调整了哪些内容。
|
||||
|
||||
## 演示视频
|
||||
|
||||
- **2–5 分钟演示视频链接**：可在线播放或下载的视频地址，视频文件不要直接放入材料包。
|
||||
- **Demo 演示**：视频中需要展示核心功能实际运行，而不只是口头或 PPT 说明。
|
||||
- **用户问题说明**：说明目标用户、使用场景和需要解决的问题。
|
||||
- **Google 技术应用说明**：说明使用了哪些 Google 技术以及它们参与了什么核心能力。
|
||||
- **创新点说明**：展示技术或产品体验与常见方案的差异。
|
||||
- **本地化说明**：说明针对目标国家、地区或文化做出的具体适配。
|
||||
- **视频时间点**：对应内容在视频中出现的时间位置；选填，用于帮助评委快速定位。
|
||||
|
||||
## 原创性与访问声明
|
||||
|
||||
- **作品是否在比赛前已经存在**：如实选择作品在比赛开始前是否已有原型、代码、产品或上线版本；必填。
|
||||
- **比赛期间新增了哪些内容**：说明本次比赛期间新完成的功能、设计、技术或验证工作；必填。
|
||||
- **原创性确认**：确认作品由本人或团队完成，并对提交内容的原创性负责；必填。
|
||||
- **同意评委访问和测试**：授权组委会和评委在评审期内打开链接、使用测试账号并体验作品；必填。
|
||||
|
||||
## 附件索引
|
||||
|
||||
- **附件路径**：小型证据文件在材料包中的相对位置。
|
||||
- **附件类型**：用户研究、技术、本地化或其他证据类别。
|
||||
- **附件说明**：说明附件证明什么内容，避免评委仅凭文件名猜测。
|
||||
@@ -0,0 +1,159 @@
|
||||
# Vibe-a-thon submission package specification
|
||||
|
||||
## Contents
|
||||
|
||||
1. Package contract
|
||||
2. Requiredness policy
|
||||
3. Interview rounds
|
||||
4. Evidence and security rules
|
||||
5. Completion rules
|
||||
6. Confirmed-submission handoff
|
||||
|
||||
## 1. Package contract
|
||||
|
||||
Every new package uses schema `vibe-a-thon-submission/v3`:
|
||||
|
||||
```text
|
||||
<submission-id>/
|
||||
├── 00-submission.json
|
||||
├── 01-overview.md
|
||||
├── 02-user-problem.md
|
||||
├── 03-product-demo.md
|
||||
├── 04-google-technology.md
|
||||
├── 05-market-localization.md
|
||||
├── 06-innovation.md
|
||||
├── 07-declarations.md
|
||||
├── private/
|
||||
│ ├── 08-test-access.md
|
||||
│ └── 09-gift-shipping.json
|
||||
└── evidence/
|
||||
├── user-research/
|
||||
├── technical/
|
||||
└── localization/
|
||||
```
|
||||
|
||||
Use `00-submission.json` as the canonical source. Regenerate the numbered Markdown files with the bundled script.
|
||||
|
||||
Treat both files under `private/` as restricted personal information. Share test access only with the organizing committee and assigned judges. Share gift-shipping information only with organizers responsible for fulfillment; do not expose it to judges or public showcase materials.
|
||||
|
||||
## 2. Requiredness policy
|
||||
|
||||
These organizer-contact and certificate fields block submission:
|
||||
|
||||
- contestant name exactly as it should appear on the certificate;
|
||||
- mobile number used by organizers to contact the contestant;
|
||||
- whether the entry is individual or team-based;
|
||||
- all team-member names exactly as they should appear on certificates, when the entry is team-based.
|
||||
|
||||
These contestant declarations also block submission:
|
||||
|
||||
- whether the work existed before the contest;
|
||||
- what was newly created during the contest;
|
||||
- originality confirmation;
|
||||
- consent for judges to access and test the work.
|
||||
|
||||
All other contestant-facing fields are optional. Missing information must not block handoff or cause automatic disqualification. Missing scoring evidence produces a scoring-impact warning. Missing optional administrative information produces an administrative warning.
|
||||
|
||||
Schema version and submission ID remain technically required so the two Skills can exchange and aggregate packages reliably.
|
||||
|
||||
## 3. Interview rounds
|
||||
|
||||
Ask no more than five questions per round. Allow “暂不提供” only for optional questions. Do not show field explanations unless the contestant explicitly asks. Summarize and confirm each round before continuing.
|
||||
|
||||
### Round A: contestant and project information
|
||||
|
||||
1. Contestant name exactly as it should appear on a certificate and mobile number; optionally collect WeChat ID and email.
|
||||
2. Whether this is an individual or team entry. For a team entry, collect every team-member name exactly as it should appear on certificates.
|
||||
3. Optional gift-shipping recipient name, phone, and full address.
|
||||
4. Track (`真`, `善`, or `美`), current status, and one-sentence description in the form “为谁，在什么场景，解决什么问题”.
|
||||
5. Target countries or regions and Xiaohongshu post URL.
|
||||
|
||||
Project/product title and team name may be recorded when supplied but are not required.
|
||||
|
||||
### Round B: user problem
|
||||
|
||||
1. Specific target user.
|
||||
2. Typical usage scenario.
|
||||
3. Current user workaround.
|
||||
4. Concrete shortcomings of the current approach.
|
||||
5. Core user pain points; optionally add interviews, feedback, public sources, or other evidence.
|
||||
|
||||
No user evidence blocks submission. Flag its absence for the external judge.
|
||||
|
||||
### Round C: solution and functioning Demo
|
||||
|
||||
1. Solution summary, user value, and complete user flow.
|
||||
2. Up to five core features.
|
||||
3. Online Demo or runnable-file cloud URL. If released, record the distribution channel and exact product name, such as App Store or Xiaohongshu.
|
||||
4. Access method, recommended browser/device/environment, expected review duration, and three to five steps with expected results.
|
||||
5. Known limitations, fallback URL, and availability confirmation.
|
||||
|
||||
If login is required, request a dedicated temporary account, temporary password, validity period, and login instructions.
|
||||
|
||||
### Round D: Google technology
|
||||
|
||||
For each technology actually used, collect:
|
||||
|
||||
- Google product or technology name;
|
||||
- specific model or capability;
|
||||
- product function supported;
|
||||
- input, processing, and output;
|
||||
- core or supporting role;
|
||||
- selection reason;
|
||||
- impact if removed;
|
||||
- code, architecture, screenshot, video timecode, or Demo evidence location.
|
||||
|
||||
### Round E: innovation, overseas viability, and localization
|
||||
|
||||
1. At least one technical innovation when available.
|
||||
2. At least one product or experience innovation when available.
|
||||
3. First planned overseas market, rationale, initial user-acquisition route, usage conditions, and main barriers.
|
||||
4. Target culture, local habits, user emotions or constraints, and adaptations beyond translation.
|
||||
5. Localization research or test evidence and changes made after testing.
|
||||
|
||||
For each innovation, collect title, common approach, difference, actual effect, and evidence location.
|
||||
|
||||
### Round F: video and declarations
|
||||
|
||||
1. A 2–5 minute video URL containing the Demo, user problem, Google technology, innovation, and localization explanation.
|
||||
2. Whether the work existed before the contest.
|
||||
3. What was newly created during the contest.
|
||||
4. Originality confirmation.
|
||||
5. Consent for judges to access and test the submitted work.
|
||||
|
||||
Video section timecodes are optional.
|
||||
|
||||
## 4. Evidence and security rules
|
||||
|
||||
- Do not collect Vibe Coding chat logs or Prompt histories. Prompt or Agent workflow evidence is optional only when it is itself a core technical mechanism.
|
||||
- Do not collect or score likes, views, favorites, shares, or other propagation metrics.
|
||||
- Submit videos, source archives, datasets, install packages, and every file larger than 10 MB as a stable streaming, store, or cloud-drive URL.
|
||||
- Never collect API keys, production credentials, personal passwords, or real-user personal data.
|
||||
- Use only a dedicated temporary test account for restricted Demos. Never repeat its secret in generated summaries.
|
||||
- Store gift recipient name, phone, and full shipping address only in `private/09-gift-shipping.json`. Never include them in `00-submission.json`, generated Markdown, AI evidence summaries, judge scorecards, or ranking CSV files.
|
||||
- Distinguish contestant claims from verified evidence and explicitly preserve “not provided” or “not tested”.
|
||||
|
||||
## 5. Completion rules
|
||||
|
||||
Validation errors block handoff only for:
|
||||
|
||||
- invalid or unsupported package schema;
|
||||
- missing submission ID;
|
||||
- missing contestant certificate name;
|
||||
- missing mobile number;
|
||||
- unanswered individual/team participation mode;
|
||||
- missing team-member certificate names when team participation is selected;
|
||||
- unanswered `existed_before_contest` declaration;
|
||||
- missing description of work created during the contest;
|
||||
- originality not confirmed;
|
||||
- judge access and testing not authorized.
|
||||
|
||||
Everything else is a warning. Missing scoring material may reduce the relevant score; missing optional contact or gift information is administrative only. Examples include missing user problem, Demo, video, Google technology evidence, innovation, overseas-market reasoning, localization, invalid optional URLs, incomplete Demo steps, or files larger than 10 MB.
|
||||
|
||||
## 6. Confirmed-submission handoff
|
||||
|
||||
Do not show the handoff reminder during collection, drafting, validation, or correction. Show it only after the contestant has explicitly confirmed that the submission information is final:
|
||||
|
||||
> 请通过邮件将整理好的信息包发回给组织方：**gdghangzhou@163.com**。如果文件夹太大，可以先上传云网盘，然后发送链接。（请选手自行确保发送的链接可以公开访问）
|
||||
|
||||
When a cloud-drive link is used, the contestant must verify that the main submission package can be opened without requesting additional permission. Because the link may be publicly accessible, the uploaded public package must exclude the entire `private/` directory. When either restricted file is present, send `private/08-test-access.md` and `private/09-gift-shipping.json` separately to the organizer as email attachments or through a separate access-controlled link. Never publish either restricted file through the public submission link.
|
||||
661
.agents/skills/vibe-submission-collector/scripts/submission_tool.py
Executable file
661
.agents/skills/vibe-submission-collector/scripts/submission_tool.py
Executable file
@@ -0,0 +1,661 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Initialize, render, and validate Vibe-a-thon submission packages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
SCHEMA = "vibe-a-thon-submission/v3"
|
||||
LEGACY_SCHEMAS = {"vibe-a-thon-submission/v1", "vibe-a-thon-submission/v2"}
|
||||
REQUIRED_MARKDOWN = [
|
||||
"01-overview.md",
|
||||
"02-user-problem.md",
|
||||
"03-product-demo.md",
|
||||
"04-google-technology.md",
|
||||
"05-market-localization.md",
|
||||
"06-innovation.md",
|
||||
"07-declarations.md",
|
||||
]
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
|
||||
|
||||
|
||||
def valid_url(value: str) -> bool:
|
||||
parsed = urlparse(value or "")
|
||||
return parsed.scheme in {"http", "https"} and bool(parsed.netloc)
|
||||
|
||||
|
||||
def blank_submission(submission_id: str, title: str) -> dict:
|
||||
return {
|
||||
"schema_version": SCHEMA,
|
||||
"submission_id": submission_id,
|
||||
"updated_at": now_iso(),
|
||||
"basic": {
|
||||
"title": title,
|
||||
"team_name": "",
|
||||
"track": "",
|
||||
"one_liner": "",
|
||||
"contestant_name_certificate": "",
|
||||
"has_team": None,
|
||||
"team_member_names_certificate": [],
|
||||
"contact": {"name": "", "email": "", "phone": "", "wechat": ""},
|
||||
"members": [],
|
||||
"target_markets": [],
|
||||
"languages": [],
|
||||
"status": "",
|
||||
"xiaohongshu_url": "",
|
||||
},
|
||||
"problem": {
|
||||
"target_users": "",
|
||||
"scenario": "",
|
||||
"current_workaround": "",
|
||||
"pain_points": [],
|
||||
"evidence_summary": "",
|
||||
"evidence_links": [],
|
||||
"research_count": 0,
|
||||
"representative_feedback": "",
|
||||
"unvalidated_assumptions": [],
|
||||
},
|
||||
"solution": {"summary": "", "user_value": "", "user_flow": [], "core_features": []},
|
||||
"demo": {
|
||||
"url": "",
|
||||
"access_method": "",
|
||||
"recommended_environment": "",
|
||||
"experience_steps": [],
|
||||
"expected_duration_minutes": None,
|
||||
"known_issues": [],
|
||||
"fallback_download_url": "",
|
||||
"published_channel": "",
|
||||
"published_product_name": "",
|
||||
"support_contact": "",
|
||||
"availability_confirmed": False,
|
||||
"automated_testing_consent": False,
|
||||
"access_restricted": False,
|
||||
"private_access_file": "private/08-test-access.md",
|
||||
},
|
||||
"video": {
|
||||
"url": "",
|
||||
"duration_minutes": None,
|
||||
"contents_confirmed": {"demo": False, "user_problem": False, "google_technology": False, "innovation": False, "localization": False},
|
||||
"timecodes": {"problem": "", "demo": "", "google_technology": "", "innovation": "", "localization": ""},
|
||||
},
|
||||
"google_technology": [],
|
||||
"technical": {
|
||||
"architecture_file": "",
|
||||
"repository_url": "",
|
||||
"commit_id": "",
|
||||
"code_evidence_paths": [],
|
||||
"deployment_notes": "",
|
||||
"dependencies": [],
|
||||
"confidentiality_notes": "",
|
||||
},
|
||||
"innovation": {"technical": [], "product_experience": []},
|
||||
"market": {
|
||||
"first_market": "",
|
||||
"market_rationale": "",
|
||||
"user_acquisition": "",
|
||||
"usage_conditions": "",
|
||||
"barriers": [],
|
||||
"cost_range": "",
|
||||
"social_impact_path": "",
|
||||
"impact_metrics": [],
|
||||
"six_month_plan": "",
|
||||
},
|
||||
"localization": {
|
||||
"target_culture": "",
|
||||
"habits": [],
|
||||
"user_concerns": [],
|
||||
"adaptations": [],
|
||||
"evidence": [],
|
||||
"tested_with_local_users": False,
|
||||
"test_count": 0,
|
||||
"changes_after_testing": "",
|
||||
},
|
||||
"declarations": {
|
||||
"existed_before_contest": None,
|
||||
"new_work_during_contest": "",
|
||||
"originality_confirmed": False,
|
||||
"review_access_consent": False,
|
||||
},
|
||||
"attachments": [],
|
||||
}
|
||||
|
||||
|
||||
def load_submission(folder: Path) -> dict:
|
||||
path = folder / "00-submission.json"
|
||||
if not path.is_file():
|
||||
raise ValueError(f"missing {path}")
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if data.get("schema_version") in LEGACY_SCHEMAS | {SCHEMA}:
|
||||
basic = data.setdefault("basic", {})
|
||||
basic.setdefault("contestant_name_certificate", basic.get("contact", {}).get("name", ""))
|
||||
basic.setdefault("team_member_names_certificate", [
|
||||
member.get("name", "") for member in basic.get("members", []) if isinstance(member, dict) and member.get("name")
|
||||
])
|
||||
basic.setdefault("has_team", bool(basic.get("team_member_names_certificate")))
|
||||
demo = data.setdefault("demo", {})
|
||||
demo.setdefault("published_channel", "")
|
||||
demo.setdefault("published_product_name", "")
|
||||
video = data.setdefault("video", {})
|
||||
video.setdefault("contents_confirmed", {"demo": False, "user_problem": False, "google_technology": False, "innovation": False, "localization": False})
|
||||
if data.get("schema_version") in LEGACY_SCHEMAS:
|
||||
data["schema_version"] = SCHEMA
|
||||
return data
|
||||
|
||||
|
||||
def bullets(values, empty="未提供") -> str:
|
||||
if not values:
|
||||
return f"- {empty}"
|
||||
result = []
|
||||
for value in values:
|
||||
if isinstance(value, dict):
|
||||
result.append(f"- {json.dumps(value, ensure_ascii=False)}")
|
||||
else:
|
||||
result.append(f"- {value}")
|
||||
return "\n".join(result)
|
||||
|
||||
|
||||
def text(value, empty="未提供") -> str:
|
||||
if value is None or value == "":
|
||||
return empty
|
||||
if isinstance(value, bool):
|
||||
return "是" if value else "否"
|
||||
return str(value)
|
||||
|
||||
|
||||
def render(folder: Path, data: dict) -> None:
|
||||
basic, problem, solution = data["basic"], data["problem"], data["solution"]
|
||||
demo, video = data["demo"], data["video"]
|
||||
tech, market, loc = data["technical"], data["market"], data["localization"]
|
||||
decl = data["declarations"]
|
||||
|
||||
overview = f"""# 作品概览
|
||||
|
||||
- 作品编号：{text(data['submission_id'])}
|
||||
- 作品名称：{text(basic['title'])}
|
||||
- 团队名称：{text(basic['team_name'])}
|
||||
- 选手姓名（获奖证书）：{text(basic['contestant_name_certificate'])}
|
||||
- 参赛形式：{'团队参赛' if basic['has_team'] is True else '个人参赛' if basic['has_team'] is False else '未提供'}
|
||||
- 团队成员姓名（获奖证书）：{', '.join(basic['team_member_names_certificate']) or '未提供'}
|
||||
- 赛道：{text(basic['track'])}
|
||||
- 当前状态：{text(basic['status'])}
|
||||
- 一句话介绍：{text(basic['one_liner'])}
|
||||
- 目标市场：{', '.join(basic['target_markets']) or '未提供'}
|
||||
- 产品语言：{', '.join(basic['languages']) or '未提供'}
|
||||
- 小红书链接：{text(basic['xiaohongshu_url'])}
|
||||
|
||||
## 团队成员
|
||||
|
||||
{bullets(basic['members'])}
|
||||
|
||||
## 联系人
|
||||
|
||||
- 姓名：{text(basic['contact']['name'])}
|
||||
- 邮箱：{text(basic['contact']['email'])}
|
||||
- 手机：{text(basic['contact']['phone'])}
|
||||
- 微信：{text(basic['contact']['wechat'])}
|
||||
"""
|
||||
user_problem = f"""# 用户问题
|
||||
|
||||
## 目标用户
|
||||
|
||||
{text(problem['target_users'])}
|
||||
|
||||
## 使用场景
|
||||
|
||||
{text(problem['scenario'])}
|
||||
|
||||
## 当前解决方式
|
||||
|
||||
{text(problem['current_workaround'])}
|
||||
|
||||
## 具体痛点
|
||||
|
||||
{bullets(problem['pain_points'])}
|
||||
|
||||
## 问题证据
|
||||
|
||||
{text(problem['evidence_summary'])}
|
||||
|
||||
{bullets(problem['evidence_links'], '未提供证据链接')}
|
||||
|
||||
- 访谈或测试人数：{text(problem['research_count'], '0')}
|
||||
- 代表性反馈：{text(problem['representative_feedback'])}
|
||||
|
||||
## 尚未验证的假设
|
||||
|
||||
{bullets(problem['unvalidated_assumptions'])}
|
||||
"""
|
||||
feature_lines = []
|
||||
for index, feature in enumerate(solution["core_features"], 1):
|
||||
feature_lines.append(
|
||||
f"### {index}. {text(feature.get('name'))}\n\n"
|
||||
f"- 解决的问题：{text(feature.get('problem'))}\n"
|
||||
f"- 输入：{text(feature.get('input'))}\n"
|
||||
f"- 系统处理：{text(feature.get('processing'))}\n"
|
||||
f"- 输出：{text(feature.get('output'))}\n"
|
||||
f"- 当前状态：{text(feature.get('runnable_status'))}\n"
|
||||
f"- 证据位置：{text(feature.get('evidence_location'))}"
|
||||
)
|
||||
step_lines = []
|
||||
for index, step in enumerate(demo["experience_steps"], 1):
|
||||
step_lines.append(f"{index}. {text(step.get('action'))}\n - 预期结果：{text(step.get('expected_result'))}")
|
||||
timecodes = video["timecodes"]
|
||||
product_demo = f"""# 产品与 Demo
|
||||
|
||||
## 解决方案
|
||||
|
||||
{text(solution['summary'])}
|
||||
|
||||
## 用户价值
|
||||
|
||||
{text(solution['user_value'])}
|
||||
|
||||
## 用户流程
|
||||
|
||||
{bullets(solution['user_flow'])}
|
||||
|
||||
## 核心功能
|
||||
|
||||
{chr(10).join(feature_lines) or '未提供'}
|
||||
|
||||
## Demo
|
||||
|
||||
- Demo链接：{text(demo['url'])}
|
||||
- 访问方式：{text(demo['access_method'])}
|
||||
- 推荐环境：{text(demo['recommended_environment'])}
|
||||
- 预计体验时长：{text(demo['expected_duration_minutes'])} 分钟
|
||||
- 备用下载链接：{text(demo['fallback_download_url'])}
|
||||
- 已发布渠道：{text(demo['published_channel'])}
|
||||
- 已发布产品名称：{text(demo['published_product_name'])}
|
||||
- 技术支持联系人：{text(demo['support_contact'])}
|
||||
- 受限访问：{text(demo['access_restricted'])}
|
||||
- 测试账号文件：{text(demo['private_access_file'])}
|
||||
|
||||
### 体验步骤
|
||||
|
||||
{chr(10).join(step_lines) or '未提供'}
|
||||
|
||||
### 已知问题
|
||||
|
||||
{bullets(demo['known_issues'])}
|
||||
|
||||
## 演示视频
|
||||
|
||||
- 视频链接：{text(video['url'])}
|
||||
- 时长：{text(video['duration_minutes'])} 分钟
|
||||
- 包含 Demo 演示：{text(video['contents_confirmed']['demo'])}
|
||||
- 包含用户问题：{text(video['contents_confirmed']['user_problem'])}
|
||||
- 包含 Google 技术说明：{text(video['contents_confirmed']['google_technology'])}
|
||||
- 包含创新点说明：{text(video['contents_confirmed']['innovation'])}
|
||||
- 包含本地化说明：{text(video['contents_confirmed']['localization'])}
|
||||
- 用户问题：{text(timecodes['problem'])}
|
||||
- Demo：{text(timecodes['demo'])}
|
||||
- Google技术：{text(timecodes['google_technology'])}
|
||||
- 创新点：{text(timecodes['innovation'])}
|
||||
- 本地化：{text(timecodes['localization'])}
|
||||
"""
|
||||
google_lines = []
|
||||
for index, item in enumerate(data["google_technology"], 1):
|
||||
google_lines.append(
|
||||
f"## {index}. {text(item.get('name'))}\n\n"
|
||||
f"- 具体能力或模型：{text(item.get('capability'))}\n"
|
||||
f"- 对应功能：{text(item.get('feature'))}\n"
|
||||
f"- 输入：{text(item.get('input'))}\n"
|
||||
f"- 处理：{text(item.get('processing'))}\n"
|
||||
f"- 输出：{text(item.get('output'))}\n"
|
||||
f"- 核心程度：{text(item.get('role'))}\n"
|
||||
f"- 选型原因：{text(item.get('selection_reason'))}\n"
|
||||
f"- 移除后的影响：{text(item.get('removal_impact'))}\n"
|
||||
f"- 证据位置：{text(item.get('evidence_location'))}"
|
||||
)
|
||||
google_technology = f"""# Google 技术与实现证据
|
||||
|
||||
{chr(10).join(google_lines) or '未提供 Google 技术记录'}
|
||||
|
||||
## 技术材料
|
||||
|
||||
- 架构图：{text(tech['architecture_file'])}
|
||||
- 代码仓库：{text(tech['repository_url'])}
|
||||
- 评审版本：{text(tech['commit_id'])}
|
||||
- 部署说明：{text(tech['deployment_notes'])}
|
||||
- 保密限制：{text(tech['confidentiality_notes'])}
|
||||
|
||||
### 核心代码或证据位置
|
||||
|
||||
{bullets(tech['code_evidence_paths'])}
|
||||
|
||||
### 外部依赖
|
||||
|
||||
{bullets(tech['dependencies'])}
|
||||
"""
|
||||
market_localization = f"""# 出海市场、社会价值与本地化
|
||||
|
||||
## 出海落地
|
||||
|
||||
- 首个市场：{text(market['first_market'])}
|
||||
- 市场选择原因：{text(market['market_rationale'])}
|
||||
- 首批用户获取方式：{text(market['user_acquisition'])}
|
||||
- 使用条件：{text(market['usage_conditions'])}
|
||||
- 成本范围：{text(market['cost_range'])}
|
||||
- 六个月计划：{text(market['six_month_plan'])}
|
||||
|
||||
### 主要障碍
|
||||
|
||||
{bullets(market['barriers'])}
|
||||
|
||||
## 社会价值
|
||||
|
||||
{text(market['social_impact_path'])}
|
||||
|
||||
### 影响指标
|
||||
|
||||
{bullets(market['impact_metrics'])}
|
||||
|
||||
## 本地化
|
||||
|
||||
- 目标文化或地区：{text(loc['target_culture'])}
|
||||
- 是否经过当地用户测试：{text(loc['tested_with_local_users'])}
|
||||
- 测试人数：{text(loc['test_count'], '0')}
|
||||
- 测试后的修改：{text(loc['changes_after_testing'])}
|
||||
|
||||
### 当地习惯
|
||||
|
||||
{bullets(loc['habits'])}
|
||||
|
||||
### 用户顾虑
|
||||
|
||||
{bullets(loc['user_concerns'])}
|
||||
|
||||
### 非翻译型适配
|
||||
|
||||
{bullets(loc['adaptations'])}
|
||||
|
||||
### 本地化证据
|
||||
|
||||
{bullets(loc['evidence'])}
|
||||
"""
|
||||
innovation_lines = []
|
||||
for label, values in [("技术创新", data["innovation"]["technical"]), ("产品与体验创新", data["innovation"]["product_experience"])]:
|
||||
innovation_lines.append(f"## {label}")
|
||||
if not values:
|
||||
innovation_lines.append("\n未提供")
|
||||
for index, item in enumerate(values, 1):
|
||||
innovation_lines.append(
|
||||
f"\n### {index}. {text(item.get('title'))}\n\n"
|
||||
f"- 常见做法：{text(item.get('common_approach'))}\n"
|
||||
f"- 本作品的不同：{text(item.get('difference'))}\n"
|
||||
f"- 实际效果：{text(item.get('effect'))}\n"
|
||||
f"- 证据位置：{text(item.get('evidence_location'))}"
|
||||
)
|
||||
declarations = f"""# 原创性与合规声明
|
||||
|
||||
- 赛前已存在：{text(decl['existed_before_contest'])}
|
||||
- 比赛期间新增：{text(decl['new_work_during_contest'])}
|
||||
- 原创性确认：{text(decl['originality_confirmed'])}
|
||||
- 评委访问和测试授权：{text(decl['review_access_consent'])}
|
||||
|
||||
## 小型附件索引
|
||||
|
||||
{bullets(data['attachments'])}
|
||||
"""
|
||||
outputs = {
|
||||
"01-overview.md": overview,
|
||||
"02-user-problem.md": user_problem,
|
||||
"03-product-demo.md": product_demo,
|
||||
"04-google-technology.md": google_technology,
|
||||
"05-market-localization.md": market_localization,
|
||||
"06-innovation.md": "# 创新说明\n\n" + "\n".join(innovation_lines) + "\n",
|
||||
"07-declarations.md": declarations,
|
||||
}
|
||||
for filename, content in outputs.items():
|
||||
(folder / filename).write_text(content.strip() + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def validate(folder: Path, data: dict) -> dict:
|
||||
errors, warnings = [], []
|
||||
|
||||
def missing(value) -> bool:
|
||||
return value is None or value == "" or value == []
|
||||
|
||||
def warn_missing(value, label):
|
||||
if missing(value):
|
||||
warnings.append(f"missing scoring information: {label}")
|
||||
|
||||
if missing(data.get("schema_version")):
|
||||
errors.append("missing schema_version")
|
||||
if data.get("schema_version") != SCHEMA:
|
||||
errors.append(f"unsupported schema_version: {data.get('schema_version')}")
|
||||
if missing(data.get("submission_id")):
|
||||
errors.append("missing submission_id")
|
||||
|
||||
basic = data.get("basic", {})
|
||||
warn_missing(basic.get("contact", {}).get("wechat"), "微信号")
|
||||
if missing(basic.get("contestant_name_certificate")):
|
||||
errors.append("missing basic.contestant_name_certificate")
|
||||
if missing(basic.get("contact", {}).get("phone")):
|
||||
errors.append("missing basic.contact.phone")
|
||||
if not isinstance(basic.get("has_team"), bool):
|
||||
errors.append("basic.has_team must be answered true or false")
|
||||
elif basic.get("has_team") and missing(basic.get("team_member_names_certificate")):
|
||||
errors.append("missing basic.team_member_names_certificate for team entry")
|
||||
elif not basic.get("has_team") and basic.get("team_member_names_certificate"):
|
||||
warnings.append("inconsistent administrative information: 个人参赛不应填写团队成员证书姓名")
|
||||
warn_missing(basic.get("track"), "参赛赛道")
|
||||
if basic.get("track") and basic["track"] not in {"真", "善", "美"}:
|
||||
warnings.append("invalid scoring information: 参赛赛道应为真、善或美")
|
||||
warn_missing(basic.get("status"), "当前状态")
|
||||
warn_missing(basic.get("one_liner"), "一句话介绍")
|
||||
warn_missing(basic.get("contact", {}).get("email"), "邮箱")
|
||||
warn_missing(basic.get("target_markets"), "目标国家或地区")
|
||||
warn_missing(basic.get("xiaohongshu_url"), "小红书作品链接")
|
||||
if basic.get("xiaohongshu_url") and not valid_url(basic["xiaohongshu_url"]):
|
||||
warnings.append("invalid optional URL: 小红书作品链接")
|
||||
|
||||
contact = basic.get("contact", {})
|
||||
if not contact.get("email") and not contact.get("wechat"):
|
||||
warnings.append("missing administrative information: 邮箱或微信号")
|
||||
|
||||
problem = data.get("problem", {})
|
||||
warn_missing(problem.get("target_users"), "具体目标用户")
|
||||
warn_missing(problem.get("scenario"), "典型使用场景")
|
||||
warn_missing(problem.get("current_workaround"), "用户当前解决方式")
|
||||
warn_missing(problem.get("pain_points"), "核心用户痛点和现有方式不足")
|
||||
if not problem.get("evidence_summary") and not problem.get("evidence_links"):
|
||||
warnings.append("scoring evidence missing: 用户问题证据")
|
||||
|
||||
solution = data.get("solution", {})
|
||||
warn_missing(solution.get("summary"), "解决方案概述")
|
||||
warn_missing(solution.get("user_value"), "用户价值")
|
||||
warn_missing(solution.get("user_flow"), "完整用户流程")
|
||||
warn_missing(solution.get("core_features"), "核心功能")
|
||||
|
||||
demo = data.get("demo", {})
|
||||
warn_missing(demo.get("url"), "Demo或可运行文件链接")
|
||||
if demo.get("url") and not valid_url(demo["url"]):
|
||||
warnings.append("invalid optional URL: Demo链接")
|
||||
if demo.get("published_channel") and not demo.get("published_product_name"):
|
||||
warnings.append("incomplete scoring information: 已发布渠道缺少产品名称")
|
||||
if demo.get("published_product_name") and not demo.get("published_channel"):
|
||||
warnings.append("incomplete scoring information: 已发布产品缺少发布渠道")
|
||||
warn_missing(demo.get("access_method"), "Demo访问方式")
|
||||
warn_missing(demo.get("recommended_environment"), "推荐运行环境")
|
||||
warn_missing(demo.get("expected_duration_minutes"), "预计体验时间")
|
||||
steps = demo.get("experience_steps", [])
|
||||
if not steps:
|
||||
warnings.append("missing scoring information: Demo体验步骤")
|
||||
elif not 3 <= len(steps) <= 5:
|
||||
warnings.append("incomplete scoring information: Demo体验步骤建议为3至5步")
|
||||
for index, step in enumerate(steps, 1):
|
||||
if missing(step.get("action")):
|
||||
warnings.append(f"incomplete scoring information: Demo第{index}步缺少操作")
|
||||
if missing(step.get("expected_result")):
|
||||
warnings.append(f"incomplete scoring information: Demo第{index}步缺少预期结果")
|
||||
if not demo.get("availability_confirmed"):
|
||||
warnings.append("missing scoring information: Demo评审期可用确认")
|
||||
if demo.get("access_restricted") and not (folder / demo.get("private_access_file", "")).is_file():
|
||||
warnings.append("restricted Demo has no private test-access file")
|
||||
if not demo.get("fallback_download_url"):
|
||||
warnings.append("missing scoring information: 备用Demo或下载链接")
|
||||
elif not valid_url(demo["fallback_download_url"]):
|
||||
warnings.append("invalid optional URL: 备用Demo或下载链接")
|
||||
|
||||
video = data.get("video", {})
|
||||
warn_missing(video.get("url"), "2至5分钟演示视频链接")
|
||||
if video.get("url") and not valid_url(video["url"]):
|
||||
warnings.append("invalid optional URL: 演示视频链接")
|
||||
duration = video.get("duration_minutes")
|
||||
if duration is None:
|
||||
warnings.append("missing scoring information: 演示视频时长")
|
||||
elif not 2 <= duration <= 5:
|
||||
warnings.append("incomplete scoring information: 演示视频应为2至5分钟")
|
||||
video_contents = video.get("contents_confirmed", {})
|
||||
for key, label in [("demo", "Demo演示"), ("user_problem", "用户问题"), ("google_technology", "Google技术说明"), ("innovation", "创新点说明"), ("localization", "本地化说明")]:
|
||||
if not video_contents.get(key):
|
||||
warnings.append(f"missing scoring information: 视频未确认包含{label}")
|
||||
|
||||
google_items = data.get("google_technology", [])
|
||||
if not google_items:
|
||||
warnings.append("missing scoring information: Google技术应用")
|
||||
required_google = ["name", "capability", "feature", "input", "processing", "output", "role", "selection_reason", "removal_impact", "evidence_location"]
|
||||
for index, item in enumerate(google_items, 1):
|
||||
for key in required_google:
|
||||
if missing(item.get(key)):
|
||||
warnings.append(f"incomplete scoring information: Google技术第{index}项缺少{key}")
|
||||
if item.get("role") and item["role"] not in {"核心", "支撑"}:
|
||||
warnings.append(f"invalid scoring information: Google技术第{index}项角色应为核心或支撑")
|
||||
|
||||
innovation = data.get("innovation", {})
|
||||
warn_missing(innovation.get("technical"), "技术创新")
|
||||
warn_missing(innovation.get("product_experience"), "产品与体验创新")
|
||||
|
||||
market = data.get("market", {})
|
||||
warn_missing(market.get("first_market"), "首个计划进入的市场")
|
||||
warn_missing(market.get("market_rationale"), "市场选择原因")
|
||||
warn_missing(market.get("user_acquisition"), "首批用户获取方式")
|
||||
warn_missing(market.get("usage_conditions"), "用户使用条件")
|
||||
warn_missing(market.get("barriers"), "主要落地障碍")
|
||||
|
||||
localization = data.get("localization", {})
|
||||
warn_missing(localization.get("target_culture"), "目标文化或地区")
|
||||
warn_missing(localization.get("habits"), "当地用户习惯")
|
||||
warn_missing(localization.get("user_concerns"), "当地用户情绪、顾虑或限制")
|
||||
warn_missing(localization.get("adaptations"), "语言翻译之外的产品适配")
|
||||
if not localization.get("tested_with_local_users"):
|
||||
warnings.append("scoring evidence missing: 未经过当地用户测试")
|
||||
if not localization.get("evidence"):
|
||||
warnings.append("scoring evidence missing: 本地化研究或测试证据")
|
||||
|
||||
declarations = data.get("declarations", {})
|
||||
if not isinstance(declarations.get("existed_before_contest"), bool):
|
||||
errors.append("declarations.existed_before_contest must be answered true or false")
|
||||
if missing(declarations.get("new_work_during_contest")):
|
||||
errors.append("missing declarations.new_work_during_contest")
|
||||
if declarations.get("originality_confirmed") is not True:
|
||||
errors.append("declarations.originality_confirmed must be true")
|
||||
if declarations.get("review_access_consent") is not True:
|
||||
errors.append("declarations.review_access_consent must be true")
|
||||
|
||||
for path in folder.rglob("*"):
|
||||
if path.is_file() and path.stat().st_size > 10 * 1024 * 1024:
|
||||
warnings.append(f"large local file should be a cloud link: {path.relative_to(folder)}")
|
||||
|
||||
shipping_path = folder / "private/09-gift-shipping.json"
|
||||
if not shipping_path.is_file():
|
||||
warnings.append("missing administrative information: 礼品邮寄信息文件")
|
||||
else:
|
||||
try:
|
||||
shipping = json.loads(shipping_path.read_text(encoding="utf-8"))
|
||||
for key, label in [("recipient_name", "礼品收件人姓名"), ("phone", "礼品收件电话"), ("address", "礼品邮寄地址")]:
|
||||
if missing(shipping.get(key)):
|
||||
warnings.append(f"missing administrative information: {label}")
|
||||
except json.JSONDecodeError:
|
||||
warnings.append("invalid administrative information: 礼品邮寄信息文件不是有效JSON")
|
||||
return {"schema_version": SCHEMA, "submission_id": data.get("submission_id"), "valid": not errors, "errors": errors, "warnings": warnings}
|
||||
|
||||
|
||||
def command_init(args) -> int:
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{1,63}", args.id):
|
||||
raise ValueError("submission ID must be 2-64 filesystem-safe characters")
|
||||
folder = Path(args.root).expanduser().resolve() / args.id
|
||||
if folder.exists() and any(folder.iterdir()):
|
||||
raise ValueError(f"refusing to overwrite non-empty folder: {folder}")
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
for subdir in ["private", "evidence/user-research", "evidence/technical", "evidence/localization"]:
|
||||
(folder / subdir).mkdir(parents=True, exist_ok=True)
|
||||
data = blank_submission(args.id, args.title)
|
||||
(folder / "00-submission.json").write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
(folder / "private/08-test-access.md").write_text(
|
||||
"# 评审测试账号（敏感）\n\n"
|
||||
"> 仅填写专用临时测试账号。禁止填写 API Key、个人密码、生产凭据或真实用户数据。\n\n"
|
||||
"- Demo：\n- 测试账号：\n- 临时密码：\n- 有效期：\n- 额外说明：\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(folder / "private/09-gift-shipping.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": "vibe-a-thon-gift-shipping/v1",
|
||||
"submission_id": args.id,
|
||||
"recipient_name": "",
|
||||
"phone": "",
|
||||
"address": "",
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
render(folder, data)
|
||||
print(folder)
|
||||
return 0
|
||||
|
||||
|
||||
def command_render(args) -> int:
|
||||
folder = Path(args.folder).expanduser().resolve()
|
||||
data = load_submission(folder)
|
||||
data["updated_at"] = now_iso()
|
||||
(folder / "00-submission.json").write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
render(folder, data)
|
||||
print(f"rendered {folder}")
|
||||
return 0
|
||||
|
||||
|
||||
def command_validate(args) -> int:
|
||||
folder = Path(args.folder).expanduser().resolve()
|
||||
result = validate(folder, load_submission(folder))
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0 if result["valid"] else 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
init_parser = subparsers.add_parser("init")
|
||||
init_parser.add_argument("--root", required=True)
|
||||
init_parser.add_argument("--id", required=True)
|
||||
init_parser.add_argument("--title", default="")
|
||||
init_parser.set_defaults(func=command_init)
|
||||
render_parser = subparsers.add_parser("render")
|
||||
render_parser.add_argument("folder")
|
||||
render_parser.set_defaults(func=command_render)
|
||||
validate_parser = subparsers.add_parser("validate")
|
||||
validate_parser.add_argument("folder")
|
||||
validate_parser.set_defaults(func=command_validate)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
return args.func(args)
|
||||
except (ValueError, KeyError, json.JSONDecodeError) as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
@@ -2,3 +2,6 @@
|
||||
/Models/
|
||||
/build/
|
||||
.DS_Store
|
||||
|
||||
# MNN 预编译二进制:由 scripts/build-mnn-xcframework.sh 本地生成,不入库防历史膨胀
|
||||
/Frameworks/MNN.xcframework/
|
||||
|
||||
19
AGENTS.md
19
AGENTS.md
@@ -22,9 +22,9 @@
|
||||
| UI | SwiftUI | iOS 17+,用 `@Observable` / `@Model` |
|
||||
| 持久化 | SwiftData | 见 §5 数据模型 |
|
||||
| 图表 | Swift Charts | iOS 16+ 原生 |
|
||||
| **AI 运行时** | **MLX Swift (Apple 官方)** | 不要建议 Core ML / llama.cpp / Ollama |
|
||||
| LLM | Qwen3-1.7B 4bit (HF: `mlx-community/Qwen3-1.7B-4bit`) | ~1.0GB,负责文本生成、关键词抽取、趋势解读 |
|
||||
| VL | Qwen2.5-VL-3B-Instruct 4bit (HF: `mlx-community/Qwen2.5-VL-3B-Instruct-4bit`) | ~2.0GB,负责拍照→结构化指标 |
|
||||
| **AI 运行时(主)** | **MNN (alibaba) + Arm SME2 + CPU** | 挑战赛考核点:Qwen + MNN + SME2 端侧 CPU 推理。device-only(xcframework 见 `scripts/build-mnn-xcframework.sh`),A19/iPhone17 启用 SME2、A17 回退 NEON。经 `MNNLLMBridge`(ObjC++)→ `MNNBackend` |
|
||||
| **AI 运行时(兜底)** | **MLX Swift (Apple 官方,Metal GPU)** | 双后端:`InferenceEngine` 切换,模拟器/兜底用 MLX。不要建议 Core ML / llama.cpp / Ollama |
|
||||
| 模型 | **Qwen3.5-2B**(一个多模态模型,文本+视觉一肩挑) | 真机主用:`taobao-mnn/Qwen3.5-2B-MNN`(~1.2GB);MLX 兜底:`mlx-community/Qwen3.5-2B-4bit`(~1.7GB)。**已废弃**:Qwen3-1.7B / Qwen2.5-VL-3B / Qwen3-VL-4B(4B 实测过慢退回 2B) |
|
||||
| 文档扫描 | VisionKit `VNDocumentCameraView` | 不要自己写透视校正 |
|
||||
| Face ID | LocalAuthentication | |
|
||||
| Live Activity | ActivityKit + WidgetExtension | demo 杀手锏,真机才能测 |
|
||||
@@ -38,13 +38,14 @@
|
||||
### 3.1 模块边界(强制)
|
||||
|
||||
```
|
||||
UI → CaptureService / AskService / TrendService → AIRuntime → MLX
|
||||
UI → CaptureService / AskService / TrendService → AIRuntime → MNN / MLX
|
||||
↓
|
||||
Persistence
|
||||
```
|
||||
|
||||
- **UI 永远不直接调 `AIRuntime`**。所有 AI 调用必须经过 `*Service` 层,这样 UI 可以注入 mock、可以预览。
|
||||
- **`AIRuntime` 是 `actor` 单例,串行化**。同一时刻只允许一个推理任务,MLX 共享显存,并发会 OOM。CaptureService 拍照时如果 AskService 正在流式生成,要在队列里排队。
|
||||
- **`AIRuntime` 是 `actor` 单例,串行化**。同一时刻只允许一个推理任务(模型共享内存/Metal 显存,并发会 OOM 被 jetsam 杀)。CaptureService 拍照时如果 AskService 正在流式生成,要在队列里排队。**真正落地**是 actor 内信号量闸门 `acquireGate()/releaseGate()`,所有占显存的重活(解码 + 模型加载)进入前先 await,且加载 VL 前先卸 LLM。
|
||||
- **引擎选择**:`InferenceEngine.current` 由偏好(`.auto`/`.mnn`/`.mlx`)+ 设备可用性解析,真机默认 `.mnn`(SME2/NEON),模拟器回退 `.mlx`。
|
||||
- **`*Service` 不直接读写 SwiftData 主上下文**。要么传入 `ModelContext`,要么走 ServiceLocator,方便测试。
|
||||
|
||||
### 3.2 VL pipeline(拍一张 = 一条流程)
|
||||
@@ -66,7 +67,7 @@ VL prompt 必须:
|
||||
### 3.3 RAG(结构化检索,不做 embedding)
|
||||
|
||||
**两段式调用**:
|
||||
1. 用 Qwen3-1.7B 抽取意图 + 关键词,输出 JSON `{indicators, time_range, intent}`,~50 token,<1s
|
||||
1. 用 Qwen3.5-2B 抽取意图 + 关键词,输出 JSON `{indicators, time_range, intent}`,~50 token,<1s
|
||||
2. SwiftData 按关键词检索 ≤ 10 条记录,拼 `ChatRAG` prompt,流式生成回答
|
||||
|
||||
**第 1 步失败时**回退到"近 30 天全表扫描",不卡死。
|
||||
@@ -84,7 +85,9 @@ VL prompt 必须:
|
||||
## 4. 模型分发
|
||||
|
||||
- 模型放 `Application Support/Models/`,首启动用 `URLSession.downloadTask` 拉,带断点续传 + 进度条
|
||||
- 总体积 ~4GB(LLM ~1.0GB + VL ~3.1GB),WiFi 提示必须有
|
||||
- **用户面只有一个模型**:Qwen3.5-2B-MNN(~1.2GB,`ModelKind.userFacing = [.mnnLLM]`)。多模态,文本+视觉全包,下载全部 / 就绪计数只算它
|
||||
- MLX 兜底版 Qwen3.5-2B-4bit(~1.7GB)仅模拟器与兜底用,不展示、不计入「下载全部」,但旁路导入仍可单独导
|
||||
- WiFi 提示必须有
|
||||
- App 在模型未就绪时**仍可启动**,但所有 AI 入口显示"模型未就绪,前往下载"
|
||||
- `ModelStore` 必须提供**旁路接口**:允许把模型预拷进沙盒(demo 现场重装时用)
|
||||
|
||||
@@ -259,7 +262,7 @@ C2 解读 Tab 底部显示一段 diff 文本,**由 `ReportCompareService` 计算
|
||||
|
||||
| 周次 | 必交付 |
|
||||
|---|---|
|
||||
| W1 末 / W2 当前 | 项目结构、MLX 跑通 Qwen3-1.7B、首个 token 在设备吐出 |
|
||||
| W1 末 / W2 当前 | 项目结构、跑通 Qwen3.5-2B(MLX/MNN)、首个 token 在设备吐出 |
|
||||
| W2-W3 | AIRuntime + LLMSession,文字日记 + 基础 RAG 问答(打字机效果)(W2 进行中) |
|
||||
| W3-W4 | VLSession + 统一拍照流程(单项 + 整份)、Asset / FileVault |
|
||||
| W4 末 | **C1 ArchiveListView**(分类 chip + 年份分组,接 @Query) |
|
||||
|
||||
23
CLAUDE.md
23
CLAUDE.md
@@ -22,9 +22,12 @@
|
||||
| UI | SwiftUI | iOS 17+,用 `@Observable` / `@Model` |
|
||||
| 持久化 | SwiftData | 见 §5 数据模型 |
|
||||
| 图表 | Swift Charts | iOS 16+ 原生 |
|
||||
| **AI 运行时** | **MLX Swift (Apple 官方)** | 不要建议 Core ML / llama.cpp / Ollama |
|
||||
| LLM | Qwen3-1.7B 4bit (HF: `mlx-community/Qwen3-1.7B-4bit`) | ~1.0GB,负责文本生成、关键词抽取、趋势解读 |
|
||||
| VL | Qwen2.5-VL-3B-Instruct 4bit (HF: `mlx-community/Qwen2.5-VL-3B-Instruct-4bit`) | ~2.0GB,负责拍照→结构化指标 |
|
||||
| **AI 运行时(主)** | **MNN (alibaba) + Arm SME2 + CPU** | 挑战赛考核点:Qwen + MNN + SME2 端侧 CPU 推理。device-only(xcframework 见 `scripts/build-mnn-xcframework.sh`),A19/iPhone17 启用 SME2、A17 回退 NEON。经 `MNNLLMBridge`(ObjC++)→ `MNNBackend` |
|
||||
| **AI 运行时(兜底)** | **MLX Swift (Apple 官方,Metal GPU)** | 双后端:`InferenceEngine` 切换,模拟器/兜底用 MLX。不要建议 Core ML / llama.cpp / Ollama |
|
||||
| **统一模型(文本+视觉)** | **Qwen3.5-2B 多模态,一个模型全包** | 同一个 Qwen3.5-2B 同时做文本生成 / 关键词抽取 / 趋势解读 **和** 拍照→结构化指标。两种格式两种引擎,按设备选(见下两行)。代码:`ModelKind` |
|
||||
| ├ MNN 主(iPhone17+/SME2) | `taobao-mnn/Qwen3.5-2B-MNN`(~1.1GiB,含 `visual.mnn`) | 挑战赛考核路径,真机默认。文本 + 图→文都走它。`ModelKind.mnnLLM`,唯一对用户暴露(`userFacing`) |
|
||||
| └ MLX 兜底 / 模拟器 | `mlx-community/Qwen3.5-2B-4bit`(~1.7GB,多模态) | Metal GPU。走 `qwen3_5`,文本与 VL 复用同一模型。`ModelKind.llm`。4B 实测过慢已退回 2B |
|
||||
| ~~VL(独立)~~ | ~~`mlx-community/Qwen3-VL-4B-Instruct-4bit`~~ **已废弃** | MLX VL 已改复用统一 Qwen3.5-2B 多模态;`ModelKind.vl` 仅保留枚举避免动穷举 switch,不再下载/展示 |
|
||||
| 文档扫描 | VisionKit `VNDocumentCameraView` | 不要自己写透视校正 |
|
||||
| Face ID | LocalAuthentication | |
|
||||
| Live Activity | ActivityKit + WidgetExtension | demo 杀手锏,真机才能测 |
|
||||
@@ -38,13 +41,13 @@
|
||||
### 3.1 模块边界(强制)
|
||||
|
||||
```
|
||||
UI → CaptureService / AskService / TrendService → AIRuntime → MLX
|
||||
UI → CaptureService / AskService / TrendService → AIRuntime → MNN(主) / MLX(兜底)
|
||||
↓
|
||||
Persistence
|
||||
```
|
||||
|
||||
- **UI 永远不直接调 `AIRuntime`**。所有 AI 调用必须经过 `*Service` 层,这样 UI 可以注入 mock、可以预览。
|
||||
- **`AIRuntime` 是 `actor` 单例,串行化**。同一时刻只允许一个推理任务,MLX 共享显存,并发会 OOM。CaptureService 拍照时如果 AskService 正在流式生成,要在队列里排队。
|
||||
- **`AIRuntime` 是 `actor` 单例,串行化**。同一时刻只允许一个推理任务(`InferenceEngine` 选 MNN/SME2 主或 MLX/GPU 兜底,共享内存/显存,并发会 OOM)。CaptureService 拍照时如果 AskService 正在流式生成,要在队列里排队。
|
||||
- **`*Service` 不直接读写 SwiftData 主上下文**。要么传入 `ModelContext`,要么走 ServiceLocator,方便测试。
|
||||
|
||||
### 3.2 VL pipeline(拍一张 = 一条流程)
|
||||
@@ -66,7 +69,7 @@ VL prompt 必须:
|
||||
### 3.3 RAG(结构化检索,不做 embedding)
|
||||
|
||||
**两段式调用**:
|
||||
1. 用 Qwen3-1.7B 抽取意图 + 关键词,输出 JSON `{indicators, time_range, intent}`,~50 token,<1s
|
||||
1. 用统一 Qwen3.5-2B(MNN 主 / MLX 兜底)抽取意图 + 关键词,输出 JSON `{indicators, time_range, intent}`,~50 token,<1s
|
||||
2. SwiftData 按关键词检索 ≤ 10 条记录,拼 `ChatRAG` prompt,流式生成回答
|
||||
|
||||
**第 1 步失败时**回退到"近 30 天全表扫描",不卡死。
|
||||
@@ -84,7 +87,7 @@ VL prompt 必须:
|
||||
## 4. 模型分发
|
||||
|
||||
- 模型放 `Application Support/Models/`,首启动用 `URLSession.downloadTask` 拉,带断点续传 + 进度条
|
||||
- 总体积 ~4GB(LLM ~1.0GB + VL ~3.1GB),WiFi 提示必须有
|
||||
- **用户侧只下载统一模型 Qwen3.5-2B(MNN,~1.1GiB,含视觉)**——不再是 ~4GB 两模型。`ModelKind.userFacing = [.mnnLLM]`,「下载全部」/ 就绪计数只算它。MLX 兜底模型 `Qwen3.5-2B-4bit`(~1.7GB)仅模拟器 / 旁路导入用,不计入用户下载;`Qwen3-VL-4B` 已废弃,不再分发。WiFi 提示仍保留
|
||||
- App 在模型未就绪时**仍可启动**,但所有 AI 入口显示"模型未就绪,前往下载"
|
||||
- `ModelStore` 必须提供**旁路接口**:允许把模型预拷进沙盒(demo 现场重装时用)
|
||||
|
||||
@@ -249,7 +252,7 @@ C2 解读 Tab 底部显示一段 diff 文本,**由 `ReportCompareService` 计算
|
||||
3. **UI 不直接调 AIRuntime**——必须经过 Service
|
||||
4. **AIRuntime 必须 actor 化**——禁止 class + lock
|
||||
5. **VL/LLM prompt 必须有 few-shot + 失败回退**——不能让用户卡在 AI 错误屏
|
||||
6. **新功能必须问"清单里有吗"**——清单外的功能(用药提醒、多 profile、暗黑模式、iCloud 同步……)默认不做,要做必须先讨论。**已加回的例外**:报告对比(16.1,§7.2)、症状追踪(Symptom @Model)、长期监测指标(MonitorMetric / IndicatorQuickSheet,W2)、个人资料(UserProfile,W2)
|
||||
6. **新功能必须问"清单里有吗"**——清单外的功能(多 profile、暗黑模式、iCloud 同步……)默认不做,要做必须先讨论。**已加回的例外**:报告对比(16.1,§7.2)、症状追踪(Symptom @Model)、长期监测指标(MonitorMetric / IndicatorQuickSheet,W2)、个人资料(UserProfile,W2)、**用药提醒**(记录 · 用药记录点药 → 复用自由提醒 `CustomReminder` / `CustomReminderEditSheet`,只到点提示,**仍不给剂量/频次建议**,守 §1 "不做剂量推荐")
|
||||
7. **不要在 6 周里重构现有 Tab/RecordSheet 骨架**——增量加东西,不要推倒重来
|
||||
8. **报告详情(C2)与归档元信息编辑(B3)是两个 View**——B3 是 draft 编辑(写),C2 是 detail 浏览(读),不要合并复用主框架
|
||||
|
||||
@@ -259,7 +262,7 @@ C2 解读 Tab 底部显示一段 diff 文本,**由 `ReportCompareService` 计算
|
||||
|
||||
| 周次 | 必交付 |
|
||||
|---|---|
|
||||
| W1 末 / W2 当前 | 项目结构、MLX 跑通 Qwen3-1.7B、首个 token 在设备吐出 |
|
||||
| W1 末 / W2 当前 | 项目结构、跑通 Qwen3.5-2B(MLX/MNN)、首个 token 在设备吐出 |
|
||||
| W2-W3 | AIRuntime + LLMSession,文字日记 + 基础 RAG 问答(打字机效果)(W2 进行中) |
|
||||
| W3-W4 | VLSession + 统一拍照流程(单项 + 整份)、Asset / FileVault |
|
||||
| W4 末 | **C1 ArchiveListView**(分类 chip + 年份分组,接 @Query) |
|
||||
@@ -281,7 +284,7 @@ C2 解读 Tab 底部显示一段 diff 文本,**由 `ReportCompareService` 计算
|
||||
## 12. 评委 PPT 卖点排序(写代码时记住为什么这么做)
|
||||
|
||||
1. 影像档案系统(统一 VL 拍照 + 归档) — 核心创意
|
||||
2. 100% 本地 + SME2 加速 — 技术亮点
|
||||
2. 100% 本地 + **MNN + Arm SME2 端侧 CPU 加速**(挑战赛考核点,MLX/GPU 兜底) — 技术亮点
|
||||
3. 本地 RAG 长期记忆 — 端侧不可替代性
|
||||
4. 隐私三件套(系统级加密 + Face ID + 永久删除) — 信任建立
|
||||
5. AI 趋势解读 — 长期价值
|
||||
|
||||
11
KangkangWidget-src/KangkangWidgetBundle.swift
Normal file
11
KangkangWidget-src/KangkangWidgetBundle.swift
Normal file
@@ -0,0 +1,11 @@
|
||||
import WidgetKit
|
||||
import SwiftUI
|
||||
|
||||
/// KangkangWidget extension 入口。
|
||||
/// W5 做 Live Activity 时,把 ActivityConfiguration 也注册进这个 Bundle。
|
||||
@main
|
||||
struct KangkangWidgetBundle: WidgetBundle {
|
||||
var body: some Widget {
|
||||
PinnedIndicatorsWidget()
|
||||
}
|
||||
}
|
||||
249
KangkangWidget-src/PinnedIndicatorsWidget.swift
Normal file
249
KangkangWidget-src/PinnedIndicatorsWidget.swift
Normal file
@@ -0,0 +1,249 @@
|
||||
import WidgetKit
|
||||
import SwiftUI
|
||||
|
||||
// MARK: - 快照模型(主 App 的独立拷贝)
|
||||
//
|
||||
// ⚠️ 同步契约:与主 App `康康/Persistence/WidgetSnapshot.swift` 字段必须一致。
|
||||
// extension 不引主 App 代码(免去 target membership 配置),改字段时两边一起改。
|
||||
|
||||
private struct WidgetSnapshot: Codable, Equatable {
|
||||
struct Item: Codable, Equatable {
|
||||
var name: String
|
||||
var value: String
|
||||
var unit: String
|
||||
var statusRaw: String // high|low|normal
|
||||
var capturedAt: Date
|
||||
}
|
||||
|
||||
var updatedAt: Date
|
||||
var items: [Item]
|
||||
|
||||
static let appGroupID = "group.com.xuhuayong.kangkang"
|
||||
static let storeKey = "kk.widget.snapshot.v1"
|
||||
|
||||
static func load() -> WidgetSnapshot? {
|
||||
guard let defaults = UserDefaults(suiteName: appGroupID),
|
||||
let data = defaults.data(forKey: storeKey) else { return nil }
|
||||
return try? JSONDecoder().decode(WidgetSnapshot.self, from: data)
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 调色(镜像主 App Tj.Palette,extension 不引 DesignSystem)
|
||||
|
||||
private enum KkColor {
|
||||
static let sand = Color(red: 0.976, green: 0.969, blue: 0.949)
|
||||
static let ink = Color(red: 0.165, green: 0.153, blue: 0.137)
|
||||
static let text = Color(red: 0.149, green: 0.137, blue: 0.118)
|
||||
static let text2 = Color(red: 0.420, green: 0.408, blue: 0.384)
|
||||
static let text3 = Color(red: 0.616, green: 0.604, blue: 0.580)
|
||||
static let brick = Color(red: 0.886, green: 0.388, blue: 0.314) // high
|
||||
static let amber = Color(red: 0.871, green: 0.627, blue: 0.314) // low
|
||||
static let leaf = Color(red: 0.180, green: 0.357, blue: 0.518) // normal
|
||||
}
|
||||
|
||||
private func statusColor(_ raw: String) -> Color {
|
||||
switch raw {
|
||||
case "high": return KkColor.brick
|
||||
case "low": return KkColor.amber
|
||||
default: return KkColor.leaf
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Timeline
|
||||
|
||||
private struct PinnedEntry: TimelineEntry {
|
||||
let date: Date
|
||||
let items: [WidgetSnapshot.Item]
|
||||
let updatedAt: Date?
|
||||
}
|
||||
|
||||
private struct PinnedProvider: TimelineProvider {
|
||||
func placeholder(in context: Context) -> PinnedEntry {
|
||||
PinnedEntry(date: .now, items: Self.sampleItems, updatedAt: .now)
|
||||
}
|
||||
|
||||
func getSnapshot(in context: Context, completion: @escaping (PinnedEntry) -> Void) {
|
||||
if context.isPreview {
|
||||
completion(placeholder(in: context))
|
||||
} else {
|
||||
completion(currentEntry())
|
||||
}
|
||||
}
|
||||
|
||||
func getTimeline(in context: Context, completion: @escaping (Timeline<PinnedEntry>) -> Void) {
|
||||
// 数据由主 App 写快照后 reloadAllTimelines 主动推;这里 30 分钟兜底刷一次
|
||||
// (只为让"x 天前"的相对时间不至于太陈旧)。
|
||||
let entry = currentEntry()
|
||||
let next = Calendar.current.date(byAdding: .minute, value: 30, to: .now) ?? .now
|
||||
completion(Timeline(entries: [entry], policy: .after(next)))
|
||||
}
|
||||
|
||||
private func currentEntry() -> PinnedEntry {
|
||||
let snap = WidgetSnapshot.load()
|
||||
return PinnedEntry(date: .now, items: snap?.items ?? [], updatedAt: snap?.updatedAt)
|
||||
}
|
||||
|
||||
static let sampleItems: [WidgetSnapshot.Item] = [
|
||||
.init(name: "收缩压", value: "128", unit: "mmHg", statusRaw: "normal",
|
||||
capturedAt: .now.addingTimeInterval(-3600 * 5)),
|
||||
.init(name: "空腹血糖", value: "6.4", unit: "mmol/L", statusRaw: "high",
|
||||
capturedAt: .now.addingTimeInterval(-3600 * 30)),
|
||||
.init(name: "体重", value: "68.5", unit: "kg", statusRaw: "normal",
|
||||
capturedAt: .now.addingTimeInterval(-3600 * 50)),
|
||||
.init(name: "尿酸", value: "486", unit: "μmol/L", statusRaw: "high",
|
||||
capturedAt: .now.addingTimeInterval(-3600 * 80)),
|
||||
]
|
||||
}
|
||||
|
||||
// MARK: - Views
|
||||
|
||||
private struct PinnedIndicatorsView: View {
|
||||
@Environment(\.widgetFamily) private var family
|
||||
let entry: PinnedEntry
|
||||
|
||||
var body: some View {
|
||||
Group {
|
||||
if entry.items.isEmpty {
|
||||
emptyView
|
||||
} else {
|
||||
switch family {
|
||||
case .systemMedium: mediumView
|
||||
default: smallView
|
||||
}
|
||||
}
|
||||
}
|
||||
.containerBackground(for: .widget) { KkColor.sand }
|
||||
}
|
||||
|
||||
private var emptyView: some View {
|
||||
VStack(spacing: 6) {
|
||||
Image(systemName: "chart.line.uptrend.xyaxis")
|
||||
.font(.system(size: 22))
|
||||
.foregroundStyle(KkColor.text3)
|
||||
Text("在康康里关注指标后\n这里会显示最新值")
|
||||
.font(.system(size: 11))
|
||||
.multilineTextAlignment(.center)
|
||||
.foregroundStyle(KkColor.text3)
|
||||
}
|
||||
}
|
||||
|
||||
/// 小尺寸:首条放大 + 其余最多 2 条小行。
|
||||
private var smallView: some View {
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
header
|
||||
if let first = entry.items.first {
|
||||
VStack(alignment: .leading, spacing: 1) {
|
||||
Text(first.name)
|
||||
.font(.system(size: 11))
|
||||
.foregroundStyle(KkColor.text2)
|
||||
HStack(alignment: .firstTextBaseline, spacing: 3) {
|
||||
Text(first.value)
|
||||
.font(.system(size: 24, weight: .semibold, design: .rounded))
|
||||
.foregroundStyle(statusColor(first.statusRaw))
|
||||
Text(first.unit)
|
||||
.font(.system(size: 10))
|
||||
.foregroundStyle(KkColor.text3)
|
||||
}
|
||||
}
|
||||
}
|
||||
ForEach(entry.items.dropFirst().prefix(2), id: \.name) { item in
|
||||
compactRow(item)
|
||||
}
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
|
||||
}
|
||||
|
||||
/// 中尺寸:两列网格,最多 6 条。
|
||||
private var mediumView: some View {
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
header
|
||||
LazyVGrid(columns: [GridItem(.flexible(), spacing: 12), GridItem(.flexible())],
|
||||
alignment: .leading, spacing: 8) {
|
||||
ForEach(entry.items.prefix(6), id: \.name) { item in
|
||||
gridCell(item)
|
||||
}
|
||||
}
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading)
|
||||
}
|
||||
|
||||
private var header: some View {
|
||||
HStack(spacing: 4) {
|
||||
Text("康康 · 长期监测")
|
||||
.font(.system(size: 10, weight: .semibold))
|
||||
.foregroundStyle(KkColor.text3)
|
||||
Spacer()
|
||||
if let updatedAt = entry.updatedAt {
|
||||
Text(updatedAt, style: .relative)
|
||||
.font(.system(size: 9))
|
||||
.foregroundStyle(KkColor.text3)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func compactRow(_ item: WidgetSnapshot.Item) -> some View {
|
||||
HStack(spacing: 4) {
|
||||
Circle()
|
||||
.fill(statusColor(item.statusRaw))
|
||||
.frame(width: 5, height: 5)
|
||||
Text(item.name)
|
||||
.font(.system(size: 10))
|
||||
.foregroundStyle(KkColor.text2)
|
||||
.lineLimit(1)
|
||||
Spacer(minLength: 2)
|
||||
Text(item.value)
|
||||
.font(.system(size: 11, weight: .semibold, design: .rounded))
|
||||
.foregroundStyle(KkColor.text)
|
||||
}
|
||||
}
|
||||
|
||||
private func gridCell(_ item: WidgetSnapshot.Item) -> some View {
|
||||
VStack(alignment: .leading, spacing: 1) {
|
||||
HStack(spacing: 4) {
|
||||
Circle()
|
||||
.fill(statusColor(item.statusRaw))
|
||||
.frame(width: 5, height: 5)
|
||||
Text(item.name)
|
||||
.font(.system(size: 10))
|
||||
.foregroundStyle(KkColor.text2)
|
||||
.lineLimit(1)
|
||||
}
|
||||
HStack(alignment: .firstTextBaseline, spacing: 2) {
|
||||
Text(item.value)
|
||||
.font(.system(size: 15, weight: .semibold, design: .rounded))
|
||||
.foregroundStyle(KkColor.text)
|
||||
Text(item.unit)
|
||||
.font(.system(size: 8))
|
||||
.foregroundStyle(KkColor.text3)
|
||||
.lineLimit(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Widget
|
||||
|
||||
struct PinnedIndicatorsWidget: Widget {
|
||||
var body: some WidgetConfiguration {
|
||||
StaticConfiguration(kind: "PinnedIndicatorsWidget", provider: PinnedProvider()) { entry in
|
||||
PinnedIndicatorsView(entry: entry)
|
||||
}
|
||||
.configurationDisplayName("长期监测")
|
||||
.description("展示你关注的健康指标最新值。数据 100% 在本机。")
|
||||
.supportedFamilies([.systemSmall, .systemMedium])
|
||||
}
|
||||
}
|
||||
|
||||
#Preview("small", as: .systemSmall) {
|
||||
PinnedIndicatorsWidget()
|
||||
} timeline: {
|
||||
PinnedEntry(date: .now, items: PinnedProvider.sampleItems, updatedAt: .now)
|
||||
}
|
||||
|
||||
#Preview("medium", as: .systemMedium) {
|
||||
PinnedIndicatorsWidget()
|
||||
} timeline: {
|
||||
PinnedEntry(date: .now, items: PinnedProvider.sampleItems, updatedAt: .now)
|
||||
}
|
||||
52
docs/Widget接入步骤.md
Normal file
52
docs/Widget接入步骤.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# 桌面 Widget 接入步骤（约 3 分钟，Xcode 操作）
|
||||
|
||||
代码已全部写好。主 App 侧（快照写入 + RootView hook）已自动编译生效；
|
||||
Widget extension 需要你在 Xcode 里建一次 target，再放入两个源文件。
|
||||
|
||||
## 1. 创建 Widget Extension target
|
||||
|
||||
1. Xcode 打开 `康康.xcodeproj` → 菜单 **File → New → Target…**
|
||||
2. 选 **iOS → Widget Extension**，点 Next
|
||||
3. Product Name 填 **`KangkangWidget`**
|
||||
- ❌ 不勾 "Include Live Activity"（W5 做 Live Activity 时再往这个 target 里加，Bundle 入口已留好注释）
|
||||
- ❌ 不勾 "Include Configuration App Intent"（我们用 StaticConfiguration）
|
||||
4. 点 Finish；弹出 "Activate scheme?" 选 **Activate**
|
||||
|
||||
## 2. 替换模板代码
|
||||
|
||||
Xcode 会在工程根目录生成 `KangkangWidget/` 文件夹（含模板 swift 文件）。
|
||||
|
||||
1. 删除模板生成的所有 `.swift` 文件（`KangkangWidget.swift`、`KangkangWidgetBundle.swift`、`AppIntent.swift` 等，**保留 `Info.plist` 和 Assets**），选 "Move to Trash"
|
||||
2. 把 `KangkangWidget-src/` 里的两个文件拖进 Xcode 的 `KangkangWidget` 文件夹（勾选 target：KangkangWidget）：
|
||||
- `KangkangWidgetBundle.swift`
|
||||
- `PinnedIndicatorsWidget.swift`
|
||||
3. 拖完后可删掉暂存目录 `KangkangWidget-src/`
|
||||
|
||||
## 3. 配置 App Group（两个 target 都要）
|
||||
|
||||
数据通过 App Group UserDefaults 传递，ID 固定为 **`group.com.xuhuayong.kangkang`**。
|
||||
|
||||
1. 选中工程 → target **康康** → Signing & Capabilities → **+ Capability → App Groups** → ＋ 添加 `group.com.xuhuayong.kangkang`
|
||||
2. target **KangkangWidget** → 同样添加 App Groups → 勾选同一个 `group.com.xuhuayong.kangkang`
|
||||
3. KangkangWidget 的 **iOS Deployment Target 改成 17.0**（模板默认可能更高）
|
||||
|
||||
> 个人开发者账号下 App Group 会自动注册；如签名报错，在两个 target 的 Signing 里确认 Team 一致。
|
||||
|
||||
## 4. 验证
|
||||
|
||||
1. scheme 切回 **康康**，跑真机/模拟器
|
||||
2. 进 App（首页出现即写入快照），回到桌面 → 长按 → 添加小组件 → 找 **康康 · 长期监测**
|
||||
3. 小/中两个尺寸都支持。没有任何 pinned 指标时显示引导文案；
|
||||
在趋势页关注指标（或 C2「关联到趋势」）后，回桌面即可看到最新值
|
||||
|
||||
## 故障排查
|
||||
|
||||
- **小组件空白/不出现**：先确认两个 target 的 App Group 勾的是同一个 ID；再确认主 App 至少前台打开过一次（快照由主 App 写）
|
||||
- **数据不更新**：快照在 App 进后台时刷新；强杀 App 不触发 `scenePhase == .background`，正常 Home 手势退出即可
|
||||
- **编译报 `containerBackground` 不存在**：KangkangWidget 的 Deployment Target 没改成 17.0
|
||||
|
||||
## 架构备忘（给后续会话）
|
||||
|
||||
- 主 App 写快照：`康康/Persistence/WidgetSnapshot.swift`（数据契约）+ `WidgetSnapshotRefresher.swift`（pinned 指标 → App Group，RootView 在启动和进后台时调用）
|
||||
- Widget 读快照：`KangkangWidget/PinnedIndicatorsWidget.swift` 内有 `WidgetSnapshot` 的**独立拷贝**（extension 不引主 App 代码）。⚠️ 改字段两边同步
|
||||
- Widget 不读 SwiftData：store 有文件保护且在主 App 沙盒，extension 锁屏时读不到；快照 = 最后一次看到的值，锁屏也能显示
|
||||
107
docs/release/Google赛道-重定位与改造说明.md
Normal file
107
docs/release/Google赛道-重定位与改造说明.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# 康康 · Google 赛道重定位与改造说明
|
||||
|
||||
> 面向「入围 Top 30」评审表(完成度与传播 25% / Google AI 深度 25% / 创新与 Vibe Coding 20% / Tech for Good 出海 20% / 第 5 维度 ~10% 待补全)。
|
||||
> 本文是把原 MNN/SME2 赛道作品改投 Google 赛道的定位与技术映射。改造日期:2026-07-01。
|
||||
|
||||
---
|
||||
|
||||
## 0. 一句话定位(新)
|
||||
|
||||
**A privacy-first, offline-capable health companion for low-connectivity and underserved communities — understand your medical reports and medicines even without a doctor or internet, in your own language.**
|
||||
|
||||
中文:**给弱网/缺医地区与跨国语言障碍人群的隐私优先健康助手——没有医生、没有网络,也能读懂自己的化验单和药盒,用你的母语。**
|
||||
|
||||
> 注意叙事转向:旧版讲「中国人不想把体检报告传云端」(纯国内隐私焦虑);新版讲「全球低资源人群 + 跨境语言障碍」(出海 + Tech for Good)。同一套 App、同一套 UI/数据,**主要改模型底座 + 叙事**,代码改动集中在 AI 层。
|
||||
|
||||
---
|
||||
|
||||
## 1. hybrid 架构:为什么这样用 Google 技术(评审 25% 核心)
|
||||
|
||||
评审明确要求「Google 技术参与**核心能力构建**,不是包装卖点」并「说明**为什么**用这些 Google 技术」。康康的回答:
|
||||
|
||||
| 能力 | 用的 Google 技术 | 为什么非它不可 |
|
||||
|---|---|---|
|
||||
| **端侧离线推理(默认)** | **Gemma-3n E2B**(Google 开源多模态小模型,端侧 4bit) | 弱网/无网地区要「飞行模式也能用」。Gemma-3n 是 Google 专为手机优化的开源模型,**离线、隐私、不依赖账号**——这是低资源场景的刚需,云模型做不到。 |
|
||||
| **读报告/药盒原图 → 结构化**(联网增强) | **Gemini 2.5 Flash 多模态 API** | 端侧小模型读密集化验单小字不稳;Gemini 多模态能直接读图出结构化指标 + 证据位置框。**这是产品第一卖点「拍一张→变档案」的真正承载者。** |
|
||||
| **深度解读 / 多语言** | **Gemini 2.5 Flash** | 跨国用户拿到外语报告,需要「翻译 + 大白话解读」。Gemini 的多语言与推理能力正对口出海语言障碍痛点。 |
|
||||
| **Prompt 设计 / 调试** | **Google AI Studio** | 所有 Gemini/Gemma 的 prompt 在 AI Studio 里迭代;API key 也由 AI Studio 免费签发。 |
|
||||
|
||||
**hybrid 的产品逻辑**:隐私优先——**默认端侧 Gemma-3n,数据不出设备**;只有用户在「我的 · 云端 AI」**主动开启**后,「读原图 / 深度解读 / 多语言」才走 Gemini 云端。**断网或额度耗尽自动回退端侧,功能不中断。** 这既拿满「Google 产品深度」,又守住隐私品牌,还天然适配「时有时无的网络」这一低资源现实。
|
||||
|
||||
> 代码落点:`GeminiBackend.swift`(REST 直调,SSE 流式 + 多模态)、`AIRuntime.generateCloud / analyzeReportCloud`(不进 OOM 闸门,可与端侧并发)、`CaptureService.runVL`(云端优先读图、失败回退端侧 OCR+文本)、`InferenceSettingsView`(云端开关 + key)。
|
||||
|
||||
### 生产级升级路径(写给评委看工程成熟度)
|
||||
demo 用 AI Studio 直发 key 的 REST 方案(零新增 SPM 依赖、即时可编译)。生产应升级到 **Firebase AI Logic**(Swift SDK):App Check 防盗用、客户端不裸存 key、内建端侧↔云端 hybrid 回退。迁移点已在 `GeminiBackend` 注释标注,换 SDK 不动上层 Service。
|
||||
|
||||
---
|
||||
|
||||
## 2. 重定位:目标用户 / 市场 / 影响路径(评审 Tech for Good 20% + 创新 20%)
|
||||
|
||||
评审要「真实人群 / 弱势 / 低资源 / 全球化」+「海外落地可行性,不只是善意叙事」+「目标用户、场景、影响路径、后续扩展」。
|
||||
|
||||
- **目标人群**
|
||||
1. 低资源/弱网地区居民:缺医少药、网络不稳,拿到化验单无人解读。
|
||||
2. 跨境人群:移民、留学生、外派、旅居者,面对**外语**医疗文件看不懂。
|
||||
3. 慢病/老人照护者:替家人留存报告、复查前整理重点。
|
||||
- **真实问题**:看不懂(术语+外语)、找不到(报告散落)、不敢传(隐私)、没医生在身边。
|
||||
- **海外落地可行性(不是善意叙事)**
|
||||
- 离线即可用 → 适配低带宽市场,不依赖稳定网络与数据套餐。
|
||||
- 隐私默认端侧 → 契合 GDPR 等严监管市场的健康数据合规。
|
||||
- 多语言 → 一套产品覆盖多语种,边际成本低。
|
||||
- **影响路径**:个人读懂自己 → 家庭健康档案管理 → 低资源社区的基层健康工作者辅助工具。
|
||||
- **后续扩展**:更多语种、更多文档类型(疫苗本/处方/影像)、与本地公共健康项目对接、可选导出给当地医生。
|
||||
|
||||
---
|
||||
|
||||
## 3. 逐维度自检(改造后该拿的分)
|
||||
|
||||
| 维度 | 权重 | 改造后状态 |
|
||||
|---|---|---|
|
||||
| 完成度与传播 | 25% | 可运行 App + 闭环(拍报告→识别→档案→趋势→问答→摘要);Gemini 恢复真·读图。**待补:3-5 分钟 demo 视频(见 §4)+ 重写图文(§5)** |
|
||||
| Google AI 深度 | 25% | Gemma-3n(端侧)+ Gemini 多模态(读图)+ Gemini(深度/多语言)+ AI Studio。**均在核心链路,非包装。** |
|
||||
| 创新 & Vibe Coding | 20% | hybrid 隐私优先 + 离线可用 + 多语言;Claude Code/Gemini Code Assist 全程 vibe coding。叙事已转出海。 |
|
||||
| Tech for Good 出海 | 20% | 低资源/弱网/跨国语言障碍,见 §2。 |
|
||||
| 第 5 维度(~10%) | ? | **未知,待补全(见 §6 开放项)** |
|
||||
|
||||
---
|
||||
|
||||
## 4. 现场 Demo 脚本(3-5 分钟,先证「能跑」)
|
||||
|
||||
> 评审第一条:核心功能可实际运行,3-5 分钟讲清主要价值。把最强证据放最前。
|
||||
|
||||
1. **(0:00-0:30)问题**:一句话——「全球很多人拿到化验单看不懂,身边没医生,也不敢把隐私报告传云端。」
|
||||
2. **(0:30-1:30)拍一张 → 变档案**:现场拍/导入一张报告 → Gemini 多模态读出结构化指标 + 异常高亮 → 存进档案。**这是第一卖点。**
|
||||
3. **(1:30-2:15)飞行模式离线**:开飞行模式,演示端侧 Gemma-3n 仍能做文本解读/问答——「没网也能用」。这是低资源场景的杀手锏。
|
||||
4. **(2:15-3:00)多语言**:拿一张**外语**报告 → Gemini 翻译 + 母语大白话解读。出海痛点直球。
|
||||
5. **(3:00-3:45)长期价值**:趋势页 AI 解读 + 「就诊前 30 秒整理重点」摘要。
|
||||
6. **(3:45-4:30)隐私 + 技术收尾**:「我的 · 云端 AI」开关讲 hybrid;一句话点出 Gemma-3n + Gemini + AI Studio。
|
||||
|
||||
视频素材:片头/转场可用 **Veo / Google Flow** 生成;录屏需真机配合(端侧 + 飞行模式只能真机)。
|
||||
|
||||
---
|
||||
|
||||
## 5. 传播物料改写要点(小红书图文)
|
||||
|
||||
- 标题从「MNN/SME2 端侧」改为「不上传、断网也能用的健康 AI(Gemma + Gemini)」。
|
||||
- 9 宫格替换:把旧「MNN-SME2 性能自检」图换成「云端 AI 设置页(Gemini 开关)」+「飞行模式离线生成」+「外语报告→母语解读」。
|
||||
- 正文三个创新点改为:① 隐私优先 hybrid;② 端侧离线可用(低资源);③ 多语言出海。
|
||||
- 删除一切「100% 本地、不上云」绝对化表述,统一为「隐私优先,云端可选」。
|
||||
|
||||
---
|
||||
|
||||
## 6. 你需要做的事 + 开放项(不阻塞已完成代码)
|
||||
|
||||
**必须由你操作(我无法代办)**
|
||||
1. 去 **aistudio.google.com** 免费签发 Gemini API Key。
|
||||
2. App「我的 · 推理引擎 · 云端 AI · Gemini」打开开关、粘贴 key。
|
||||
3. 真机录 demo 视频(端侧 + 飞行模式只能真机)。
|
||||
|
||||
**开放项(需要你提供信息)**
|
||||
4. **评分表第 5 个维度(~10%)被截断**——把完整评审标准或比赛出处发我,补进 §3,别再漏考核点。
|
||||
5. **平台权衡**:当前保 iOS。若评委更看重 Android 生态(Android Studio/Flutter/Gemini Nano),可再评估,但 2-3 周内不建议重写。
|
||||
|
||||
---
|
||||
|
||||
## 7. 一句话给你自己
|
||||
|
||||
上一个比赛输在「为命题做的东西没按被考核的点证明」。这次的红线是:**Gemini 必须在核心链路里被看到在跑(读图/多语言),而不是设置页里一个没人点的开关。** demo 视频里 Gemini 读出一张报告的那 10 秒,比任何文案都值钱。
|
||||
462
docs/release/build_creative_proposal_doc.py
Normal file
462
docs/release/build_creative_proposal_doc.py
Normal file
@@ -0,0 +1,462 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from docx import Document
|
||||
from docx.enum.section import WD_SECTION
|
||||
from docx.enum.table import WD_TABLE_ALIGNMENT, WD_CELL_VERTICAL_ALIGNMENT
|
||||
from docx.enum.text import WD_ALIGN_PARAGRAPH
|
||||
from docx.oxml import OxmlElement
|
||||
from docx.oxml.ns import qn
|
||||
from docx.shared import Cm, Inches, Pt, RGBColor
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
OUT = ROOT / "docs" / "release" / "康康创意方案文档.docx"
|
||||
PICS = ROOT / "docs" / "release" / "xhs-9grid"
|
||||
|
||||
BLUE = RGBColor(46, 116, 181)
|
||||
DARK_BLUE = RGBColor(31, 77, 120)
|
||||
INK = RGBColor(31, 31, 31)
|
||||
MUTED = RGBColor(95, 95, 95)
|
||||
LIGHT = "F4F6F9"
|
||||
PALE_BLUE = "E8EEF5"
|
||||
GREEN = RGBColor(58, 112, 76)
|
||||
BRICK = RGBColor(154, 78, 65)
|
||||
|
||||
|
||||
def set_run_font(run, size=None, bold=None, color=None, font="PingFang SC"):
|
||||
run.font.name = font
|
||||
run._element.rPr.rFonts.set(qn("w:ascii"), "Calibri")
|
||||
run._element.rPr.rFonts.set(qn("w:hAnsi"), "Calibri")
|
||||
run._element.rPr.rFonts.set(qn("w:eastAsia"), font)
|
||||
if size is not None:
|
||||
run.font.size = Pt(size)
|
||||
if bold is not None:
|
||||
run.bold = bold
|
||||
if color is not None:
|
||||
run.font.color.rgb = color
|
||||
|
||||
|
||||
def set_paragraph_font(paragraph, size=11, color=INK, font="PingFang SC"):
|
||||
for run in paragraph.runs:
|
||||
set_run_font(run, size=size, color=color, font=font)
|
||||
|
||||
|
||||
def set_cell_shading(cell, fill):
|
||||
tc_pr = cell._tc.get_or_add_tcPr()
|
||||
shd = tc_pr.find(qn("w:shd"))
|
||||
if shd is None:
|
||||
shd = OxmlElement("w:shd")
|
||||
tc_pr.append(shd)
|
||||
shd.set(qn("w:fill"), fill)
|
||||
|
||||
|
||||
def set_cell_margins(cell, top=80, start=120, bottom=80, end=120):
|
||||
tc = cell._tc
|
||||
tc_pr = tc.get_or_add_tcPr()
|
||||
tc_mar = tc_pr.first_child_found_in("w:tcMar")
|
||||
if tc_mar is None:
|
||||
tc_mar = OxmlElement("w:tcMar")
|
||||
tc_pr.append(tc_mar)
|
||||
for m, v in [("top", top), ("start", start), ("bottom", bottom), ("end", end)]:
|
||||
node = tc_mar.find(qn(f"w:{m}"))
|
||||
if node is None:
|
||||
node = OxmlElement(f"w:{m}")
|
||||
tc_mar.append(node)
|
||||
node.set(qn("w:w"), str(v))
|
||||
node.set(qn("w:type"), "dxa")
|
||||
|
||||
|
||||
def set_table_geometry(table, widths):
|
||||
table.alignment = WD_TABLE_ALIGNMENT.CENTER
|
||||
table.autofit = False
|
||||
tbl = table._tbl
|
||||
tbl_pr = tbl.tblPr
|
||||
tbl_w = tbl_pr.find(qn("w:tblW"))
|
||||
if tbl_w is None:
|
||||
tbl_w = OxmlElement("w:tblW")
|
||||
tbl_pr.append(tbl_w)
|
||||
tbl_w.set(qn("w:w"), str(sum(widths)))
|
||||
tbl_w.set(qn("w:type"), "dxa")
|
||||
tbl_grid = tbl.tblGrid
|
||||
if tbl_grid is None:
|
||||
tbl_grid = OxmlElement("w:tblGrid")
|
||||
tbl.append(tbl_grid)
|
||||
for child in list(tbl_grid):
|
||||
tbl_grid.remove(child)
|
||||
for width in widths:
|
||||
grid_col = OxmlElement("w:gridCol")
|
||||
grid_col.set(qn("w:w"), str(width))
|
||||
tbl_grid.append(grid_col)
|
||||
for row in table.rows:
|
||||
for idx, cell in enumerate(row.cells):
|
||||
tc_pr = cell._tc.get_or_add_tcPr()
|
||||
tc_w = tc_pr.find(qn("w:tcW"))
|
||||
if tc_w is None:
|
||||
tc_w = OxmlElement("w:tcW")
|
||||
tc_pr.append(tc_w)
|
||||
tc_w.set(qn("w:w"), str(widths[idx]))
|
||||
tc_w.set(qn("w:type"), "dxa")
|
||||
set_cell_margins(cell)
|
||||
cell.vertical_alignment = WD_CELL_VERTICAL_ALIGNMENT.CENTER
|
||||
|
||||
|
||||
def table_border(table, color="DADCE0", size="6"):
|
||||
tbl_pr = table._tbl.tblPr
|
||||
borders = tbl_pr.first_child_found_in("w:tblBorders")
|
||||
if borders is None:
|
||||
borders = OxmlElement("w:tblBorders")
|
||||
tbl_pr.append(borders)
|
||||
for edge in ["top", "left", "bottom", "right", "insideH", "insideV"]:
|
||||
tag = f"w:{edge}"
|
||||
element = borders.find(qn(tag))
|
||||
if element is None:
|
||||
element = OxmlElement(tag)
|
||||
borders.append(element)
|
||||
element.set(qn("w:val"), "single")
|
||||
element.set(qn("w:sz"), size)
|
||||
element.set(qn("w:space"), "0")
|
||||
element.set(qn("w:color"), color)
|
||||
|
||||
|
||||
def add_para(doc, text="", style=None, size=11, bold=False, color=INK, align=None, after=8):
|
||||
p = doc.add_paragraph(style=style)
|
||||
p.paragraph_format.space_after = Pt(after)
|
||||
p.paragraph_format.line_spacing = 1.25
|
||||
if align is not None:
|
||||
p.alignment = align
|
||||
run = p.add_run(text)
|
||||
set_run_font(run, size=size, bold=bold, color=color)
|
||||
return p
|
||||
|
||||
|
||||
def add_heading(doc, text, level=1):
|
||||
p = doc.add_paragraph(style=f"Heading {level}")
|
||||
p.paragraph_format.keep_with_next = True
|
||||
run = p.add_run(text)
|
||||
if level == 1:
|
||||
set_run_font(run, 16, True, BLUE)
|
||||
p.paragraph_format.space_before = Pt(18)
|
||||
p.paragraph_format.space_after = Pt(10)
|
||||
elif level == 2:
|
||||
set_run_font(run, 13, True, BLUE)
|
||||
p.paragraph_format.space_before = Pt(12)
|
||||
p.paragraph_format.space_after = Pt(6)
|
||||
else:
|
||||
set_run_font(run, 12, True, DARK_BLUE)
|
||||
p.paragraph_format.space_before = Pt(8)
|
||||
p.paragraph_format.space_after = Pt(4)
|
||||
return p
|
||||
|
||||
|
||||
def add_bullet(doc, text, level=0):
|
||||
p = doc.add_paragraph(style="List Bullet")
|
||||
p.paragraph_format.left_indent = Inches(0.375)
|
||||
p.paragraph_format.first_line_indent = Inches(-0.194)
|
||||
p.paragraph_format.space_after = Pt(4)
|
||||
p.paragraph_format.line_spacing = 1.208
|
||||
run = p.add_run(text)
|
||||
set_run_font(run, 10.5, color=INK)
|
||||
return p
|
||||
|
||||
|
||||
def add_callout(doc, title, body, fill=LIGHT, title_color=DARK_BLUE):
|
||||
table = doc.add_table(rows=1, cols=1)
|
||||
set_table_geometry(table, [9360])
|
||||
table_border(table, color="E1E5EA", size="4")
|
||||
cell = table.cell(0, 0)
|
||||
set_cell_shading(cell, fill)
|
||||
cell.text = ""
|
||||
p = cell.paragraphs[0]
|
||||
p.paragraph_format.space_after = Pt(4)
|
||||
r = p.add_run(title)
|
||||
set_run_font(r, 11, True, title_color)
|
||||
p2 = cell.add_paragraph()
|
||||
p2.paragraph_format.space_after = Pt(0)
|
||||
p2.paragraph_format.line_spacing = 1.22
|
||||
r2 = p2.add_run(body)
|
||||
set_run_font(r2, 10.5, color=INK)
|
||||
doc.add_paragraph().paragraph_format.space_after = Pt(4)
|
||||
return table
|
||||
|
||||
|
||||
def add_label_table(doc, rows, widths=(2300, 7060), header=None):
|
||||
table = doc.add_table(rows=0, cols=2)
|
||||
set_table_geometry(table, list(widths))
|
||||
table_border(table, color="DADCE0", size="5")
|
||||
if header:
|
||||
row = table.add_row()
|
||||
row.cells[0].merge(row.cells[1])
|
||||
cell = row.cells[0]
|
||||
set_cell_shading(cell, PALE_BLUE)
|
||||
p = cell.paragraphs[0]
|
||||
p.paragraph_format.space_after = Pt(0)
|
||||
r = p.add_run(header)
|
||||
set_run_font(r, 10.5, True, DARK_BLUE)
|
||||
for label, value in rows:
|
||||
row = table.add_row()
|
||||
for cell in row.cells:
|
||||
set_cell_shading(cell, "FFFFFF")
|
||||
p0 = row.cells[0].paragraphs[0]
|
||||
p0.paragraph_format.space_after = Pt(0)
|
||||
r0 = p0.add_run(label)
|
||||
set_run_font(r0, 10, True, DARK_BLUE)
|
||||
p1 = row.cells[1].paragraphs[0]
|
||||
p1.paragraph_format.space_after = Pt(0)
|
||||
p1.paragraph_format.line_spacing = 1.18
|
||||
r1 = p1.add_run(value)
|
||||
set_run_font(r1, 10, color=INK)
|
||||
doc.add_paragraph().paragraph_format.space_after = Pt(4)
|
||||
return table
|
||||
|
||||
|
||||
def add_matrix(doc, headers, rows, widths):
|
||||
table = doc.add_table(rows=1, cols=len(headers))
|
||||
set_table_geometry(table, widths)
|
||||
table_border(table, color="DADCE0", size="5")
|
||||
for idx, h in enumerate(headers):
|
||||
cell = table.cell(0, idx)
|
||||
set_cell_shading(cell, PALE_BLUE)
|
||||
p = cell.paragraphs[0]
|
||||
p.paragraph_format.space_after = Pt(0)
|
||||
r = p.add_run(h)
|
||||
set_run_font(r, 9.5, True, DARK_BLUE)
|
||||
for row_data in rows:
|
||||
row = table.add_row()
|
||||
for idx, text in enumerate(row_data):
|
||||
cell = row.cells[idx]
|
||||
set_cell_shading(cell, "FFFFFF")
|
||||
p = cell.paragraphs[0]
|
||||
p.paragraph_format.space_after = Pt(0)
|
||||
p.paragraph_format.line_spacing = 1.16
|
||||
r = p.add_run(text)
|
||||
set_run_font(r, 9.2, color=INK)
|
||||
doc.add_paragraph().paragraph_format.space_after = Pt(4)
|
||||
return table
|
||||
|
||||
|
||||
def setup_styles(doc):
|
||||
styles = doc.styles
|
||||
normal = styles["Normal"]
|
||||
normal.font.name = "Calibri"
|
||||
normal._element.rPr.rFonts.set(qn("w:eastAsia"), "PingFang SC")
|
||||
normal.font.size = Pt(11)
|
||||
normal.font.color.rgb = INK
|
||||
normal.paragraph_format.space_after = Pt(8)
|
||||
normal.paragraph_format.line_spacing = 1.333
|
||||
normal.paragraph_format.alignment = WD_ALIGN_PARAGRAPH.JUSTIFY
|
||||
|
||||
for style_name in ["Heading 1", "Heading 2", "Heading 3", "List Bullet"]:
|
||||
style = styles[style_name]
|
||||
style.font.name = "Calibri"
|
||||
style._element.rPr.rFonts.set(qn("w:eastAsia"), "PingFang SC")
|
||||
|
||||
|
||||
def set_header_footer(section):
|
||||
header = section.header.paragraphs[0]
|
||||
header.text = ""
|
||||
header.alignment = WD_ALIGN_PARAGRAPH.RIGHT
|
||||
run = header.add_run("康康 Kangkang 创意方案")
|
||||
set_run_font(run, 9, color=MUTED)
|
||||
footer = section.footer.paragraphs[0]
|
||||
footer.text = ""
|
||||
footer.alignment = WD_ALIGN_PARAGRAPH.CENTER
|
||||
run = footer.add_run("本方案仅描述健康记录与科普式解读能力，不构成医疗诊断或用药建议")
|
||||
set_run_font(run, 8.5, color=MUTED)
|
||||
|
||||
|
||||
def add_title_page(doc):
|
||||
add_para(doc, "手机上的 AI 创意方案", size=12, bold=True, color=GREEN, align=WD_ALIGN_PARAGRAPH.CENTER, after=10)
|
||||
title = doc.add_paragraph()
|
||||
title.alignment = WD_ALIGN_PARAGRAPH.CENTER
|
||||
title.paragraph_format.space_after = Pt(6)
|
||||
r = title.add_run("康康：本地优先的个人健康档案")
|
||||
set_run_font(r, 24, True, DARK_BLUE)
|
||||
subtitle = doc.add_paragraph()
|
||||
subtitle.alignment = WD_ALIGN_PARAGRAPH.CENTER
|
||||
subtitle.paragraph_format.space_after = Pt(18)
|
||||
r2 = subtitle.add_run("100% 本地推理 · 个人健康影像档案 · 大白话解读 · 结构化 RAG 问答")
|
||||
set_run_font(r2, 12.5, color=MUTED)
|
||||
|
||||
add_label_table(
|
||||
doc,
|
||||
[
|
||||
("项目形态", "iOS 原生 App，SwiftUI + SwiftData。"),
|
||||
("目标用户", "不愿把体检报告、化验单、症状和用药记录交给云端的普通用户。"),
|
||||
("核心主张", "健康数据默认留在手机里，本地模型负责整理、解释和检索。"),
|
||||
("技术主线", "Qwen3.5-2B + MNN + Arm SME2/NEON，MLX Swift 作为模拟器和兜底后端。"),
|
||||
("边界声明", "不做医疗诊断、剂量推荐、急诊判断、医生预约、账号系统或数据上云。"),
|
||||
],
|
||||
header="项目概览",
|
||||
)
|
||||
|
||||
add_callout(
|
||||
doc,
|
||||
"一句话创意",
|
||||
"把体检报告、化验指标、症状、日记和用药记录收进一个本地健康档案，再让手机端本地大模型用普通人能听懂的语言帮助整理、回顾和就诊前准备。",
|
||||
fill="F7FAF7",
|
||||
title_color=GREEN,
|
||||
)
|
||||
|
||||
doc.add_page_break()
|
||||
|
||||
|
||||
def add_sources(doc):
|
||||
add_heading(doc, "依据来源", 1)
|
||||
add_bullet(doc, "AGENTS.md：项目定位、技术选型、AI 链路、数据模型、隐私边界和六周 demo 目标。")
|
||||
add_bullet(doc, "康康/AI/InferenceEngine.swift 与 AIRuntime.swift：MNN/MLX 双后端、SME2 探测、actor 串行推理闸门。")
|
||||
add_bullet(doc, "康康/Services/HealthExportService.swift：本地 RAG、意图抽取、结构化检索、就诊摘要生成和失败兜底。")
|
||||
add_bullet(doc, "docs/release/app-store-metadata.md 与小红书项目介绍：用户价值、隐私表达和非医疗器械边界。")
|
||||
|
||||
|
||||
def main():
|
||||
doc = Document()
|
||||
section = doc.sections[0]
|
||||
section.page_width = Inches(8.5)
|
||||
section.page_height = Inches(11)
|
||||
section.top_margin = Inches(1)
|
||||
section.bottom_margin = Inches(1)
|
||||
section.left_margin = Inches(1)
|
||||
section.right_margin = Inches(1)
|
||||
section.header_distance = Inches(0.492)
|
||||
section.footer_distance = Inches(0.492)
|
||||
setup_styles(doc)
|
||||
set_header_footer(section)
|
||||
|
||||
add_title_page(doc)
|
||||
|
||||
add_heading(doc, "1. 应用场景", 1)
|
||||
add_para(doc, "康康面向的是普通人的日常健康信息管理，而不是医院内部系统或专业诊疗工具。产品重点放在体检后、复查前、慢性指标长期观察、看医生前准备等高频生活场景。")
|
||||
add_matrix(
|
||||
doc,
|
||||
["场景", "用户动作", "康康提供的价值"],
|
||||
[
|
||||
("体检或化验后", "拍摄报告或从相册导入。", "本地识别关键指标、参考范围和异常状态，形成可检索的报告档案。"),
|
||||
("长期指标记录", "记录血压、血糖、体重、血脂等指标。", "自动沉淀趋势曲线，用大白话说明最近变化，降低读图门槛。"),
|
||||
("症状与日记", "随手记身体感受、睡眠、疼痛、用药等。", "把零散记录整理成时间线，必要时由本地 AI 追问补全。"),
|
||||
("就诊前准备", "输入“把最近一个月整理给医生看”。", "本地检索症状、指标、报告和用药，生成可复制/分享的就诊摘要。"),
|
||||
("家庭健康管理", "为父母或自己保存报告和复查信息。", "减少纸质报告丢失、相册翻找和口头回忆遗漏。"),
|
||||
],
|
||||
[1600, 2600, 5160],
|
||||
)
|
||||
|
||||
add_heading(doc, "2. 用户痛点", 1)
|
||||
add_bullet(doc, "看不懂：体检报告和化验单充满缩写、箭头、参考范围，普通人很难快速判断哪些值得关注。")
|
||||
add_bullet(doc, "找不到：健康信息散落在纸质报告、相册截图、备忘录和聊天记录里，复查或就医时很难完整回溯。")
|
||||
add_bullet(doc, "不敢传：健康报告包含年龄、医院、检查结果、既往问题等高度敏感信息，上传给云端 AI 会带来隐私顾虑。")
|
||||
add_bullet(doc, "记不全：看医生时常常忘记症状持续时间、近期指标变化、在服药物和过敏史。")
|
||||
add_bullet(doc, "工具割裂：传统健康记录 App 重记录轻解释，AI 聊天工具会解释但不沉淀长期个人档案。")
|
||||
|
||||
doc.add_page_break()
|
||||
add_heading(doc, "3. 技术方案", 1)
|
||||
add_heading(doc, "3.1 模型选型", 2)
|
||||
add_para(doc, "主模型选择 Qwen3.5-2B，原因是它在移动端内存和速度预算内更适合 demo 落地，同时承担文本解读和视觉识别两类任务，避免拆分多套模型造成下载、存储和加载复杂度上升。")
|
||||
add_label_table(
|
||||
doc,
|
||||
[
|
||||
("主路径模型", "Qwen3.5-2B-MNN，约 1.2GB，面向真机端侧推理。"),
|
||||
("兜底模型", "MLX Swift 对应的 Qwen3.5-2B-4bit，主要用于模拟器、调试和 MNN 不可用时的兜底。"),
|
||||
("能力覆盖", "报告/药盒等图片识别、健康文本整理、趋势解释、身体档案问答和就诊摘要。"),
|
||||
("取舍逻辑", "不用更大模型追求极限效果，而是优先保证端侧速度、内存可控、下载可接受和 demo 可复现。"),
|
||||
],
|
||||
header="模型选型摘要",
|
||||
)
|
||||
|
||||
add_heading(doc, "3.2 推理框架", 2)
|
||||
add_para(doc, "推理框架采用双后端策略：真机主路径为阿里开源 MNN，面向挑战赛的 Qwen + MNN + SME2 端侧 CPU 推理；MLX Swift 作为 Apple 官方 Metal GPU 兜底，用于模拟器和对照测试。")
|
||||
add_matrix(
|
||||
doc,
|
||||
["模块", "方案", "作用"],
|
||||
[
|
||||
("MNNBackend", "ObjC++ 桥接 MNN LLM 能力。", "在真机上加载 Qwen3.5-2B-MNN，支撑文本生成与视觉分析。"),
|
||||
("InferenceEngine", "auto / mnn / mlx 三种偏好。", "真机优先 MNN，MNN 不可用时自动回退 MLX。"),
|
||||
("AIRuntime", "actor 单例 + 推理闸门。", "串行化模型加载、文本生成和视觉推理，避免并发 OOM。"),
|
||||
("ModelStore", "Application Support/Models。", "管理模型下载、就绪校验和现场旁路导入。"),
|
||||
],
|
||||
[1900, 3200, 4260],
|
||||
)
|
||||
|
||||
add_heading(doc, "3.3 端侧适配思路", 2)
|
||||
add_bullet(doc, "设备优先级：支持 MNN 的真机默认走 MNN；支持 SME2 的 A19/iPhone17 走 SME2 加速，旧设备回退 NEON；模拟器走 MLX。")
|
||||
add_bullet(doc, "内存控制：AIRuntime 使用 actor 内信号量，保证同一时间只有一个重推理任务占用模型内存；加载 LLM 前卸载 VL，加载 MNN 前卸载 MLX 侧模型。")
|
||||
add_bullet(doc, "体验兜底：模型未就绪时 App 仍可启动，AI 入口提示前往模型管理；意图抽取失败时回退近 30 天全表扫描；识别失败时回退手动录入。")
|
||||
add_bullet(doc, "结构化 RAG：不引入 embedding 模型，先由本地模型抽取意图和关键词，再用 SwiftData 检索指标、报告、症状和日记，最后拼接上下文生成回答。")
|
||||
add_bullet(doc, "隐私保护：报告原图只保存到本地 Vault，SwiftData 存结构化记录；使用 iOS completeFileProtection、Face ID 启动锁和永久删除。")
|
||||
|
||||
add_heading(doc, "4. 创新点", 1)
|
||||
add_matrix(
|
||||
doc,
|
||||
["创新点", "说明", "差异化价值"],
|
||||
[
|
||||
("本地优先健康 AI", "AI 不依赖云端 API，核心解读和问答在手机内完成。", "解决健康数据上传焦虑，形成隐私可信的 AI 体验。"),
|
||||
("影像档案 + AI 解读闭环", "拍报告、识别、归档、趋势、问答、摘要串成一个系统。", "不是单点 OCR 或聊天框，而是长期可用的个人健康档案。"),
|
||||
("MNN + SME2 端侧推理", "将 Qwen 模型通过 MNN 接入 iPhone CPU/SME2 路径。", "展示大模型在移动 CPU 上可复现运行的技术亮点。"),
|
||||
("轻量结构化 RAG", "利用 SwiftData 已有结构化记录检索，不额外引入 embedding 模型。", "降低端侧存储和计算成本，响应更可控。"),
|
||||
("明确医疗边界", "只做记录、整理和科普式解释，不做诊断、用药和急诊判断。", "既保留实用价值，也降低合规和误导风险。"),
|
||||
],
|
||||
[1800, 3900, 3660],
|
||||
)
|
||||
|
||||
add_heading(doc, "5. 预期效果", 1)
|
||||
add_callout(
|
||||
doc,
|
||||
"用户侧效果",
|
||||
"用户可以把报告、指标、症状和用药从碎片化记录变成可检索、可回顾、可解释的个人健康档案；在看医生前，用一份摘要减少遗漏和重复描述。",
|
||||
fill="F7FAF7",
|
||||
title_color=GREEN,
|
||||
)
|
||||
add_callout(
|
||||
doc,
|
||||
"技术侧效果",
|
||||
"验证 Qwen3.5-2B 在 MNN + SME2/NEON 端侧 CPU 路径上的可用性，形成模型下载、引擎选择、性能自检、推理串行、失败兜底和本地数据检索的一体化方案。",
|
||||
fill="F4F6F9",
|
||||
title_color=DARK_BLUE,
|
||||
)
|
||||
add_callout(
|
||||
doc,
|
||||
"展示侧效果",
|
||||
"比赛或路演时可以通过飞行模式、本地模型管理页、性能自检 tok/s、报告识别前后对比和身体档案摘要，清晰证明“不是云端套壳，而是端侧 AI 创意应用”。",
|
||||
fill="FFF6F2",
|
||||
title_color=BRICK,
|
||||
)
|
||||
|
||||
add_heading(doc, "6. 方案完整性", 1)
|
||||
add_para(doc, "康康的完整性体现在产品、技术和安全边界三条线同时闭合：用户能完成从采集到回顾再到就诊摘要的闭环；技术上有模型、运行时、数据、服务层和 UI 的分层；安全上明确不引入云服务、不做账号、不做自研密码学。")
|
||||
add_matrix(
|
||||
doc,
|
||||
["层级", "已覆盖能力", "设计原则"],
|
||||
[
|
||||
("采集层", "拍报告、记录指标、写日记、记症状、药品识别。", "入口轻，失败可手动补录。"),
|
||||
("数据层", "SwiftData 模型：Indicator、Report、DiaryEntry、Symptom、Asset、ChatTurn、UserProfile 等。", "结构化保存，便于检索和趋势计算。"),
|
||||
("AI 层", "CaptureService、HealthExportService、TrendInsightService 经 AIRuntime 调用本地模型。", "UI 不直连模型，推理统一排队。"),
|
||||
("展示层", "首页、记录、趋势、我的、模型管理、身体档案摘要。", "围绕 demo 核心卖点组织信息。"),
|
||||
("隐私层", "本地 Vault、completeFileProtection、Face ID、永久删除、无账号无云。", "使用系统能力，不自造密码学。"),
|
||||
],
|
||||
[1500, 5000, 2860],
|
||||
)
|
||||
|
||||
add_heading(doc, "7. 风险与边界控制", 1)
|
||||
add_bullet(doc, "AI 输出仅作为健康记录整理与科普式解释，不作为医疗诊断、治疗、用药或剂量建议。")
|
||||
add_bullet(doc, "识别结果必须允许用户核对和编辑，避免把模型错误直接落成事实。")
|
||||
add_bullet(doc, "端侧模型受设备性能、内存和电量影响，需要模型未就绪、加载失败、低性能设备等状态提示。")
|
||||
add_bullet(doc, "健康类内容表达要避免“治疗”“诊断”“疗效”等容易误导的词汇。")
|
||||
|
||||
# Visual appendix with the generated 9-grid overview if available.
|
||||
overview = PICS / "00-9宫格总览.png"
|
||||
if overview.exists():
|
||||
doc.add_page_break()
|
||||
add_heading(doc, "附录：展示素材示意", 1)
|
||||
add_para(doc, "以下为小红书 9 宫格展示图总览，可用于说明产品闭环与技术亮点。", size=10.5, color=MUTED)
|
||||
p = doc.add_paragraph()
|
||||
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
|
||||
run = p.add_run()
|
||||
run.add_picture(str(overview), width=Inches(5.6))
|
||||
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(OUT)
|
||||
print(OUT)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
BIN
docs/release/pics/01-字体大小设置.png
Normal file
BIN
docs/release/pics/01-字体大小设置.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 842 KiB |
BIN
docs/release/pics/02-Xcode开发调试界面.png
Normal file
BIN
docs/release/pics/02-Xcode开发调试界面.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 955 KiB |
BIN
docs/release/pics/03-健康日记-语音输入键盘.jpg
Normal file
BIN
docs/release/pics/03-健康日记-语音输入键盘.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
BIN
docs/release/pics/04-关于页面-项目说明.png
Normal file
BIN
docs/release/pics/04-关于页面-项目说明.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
BIN
docs/release/pics/05-字体大小设置-副本.png
Normal file
BIN
docs/release/pics/05-字体大小设置-副本.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 842 KiB |
BIN
docs/release/pics/06-推理引擎-MNN-SME2性能自检.png
Normal file
BIN
docs/release/pics/06-推理引擎-MNN-SME2性能自检.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.0 MiB |
BIN
docs/release/pics/07-模型管理-Qwen已就绪.png
Normal file
BIN
docs/release/pics/07-模型管理-Qwen已就绪.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 556 KiB |
BIN
docs/release/pics/08-提醒列表-服药提醒.png
Normal file
BIN
docs/release/pics/08-提醒列表-服药提醒.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 437 KiB |
BIN
docs/release/pics/09-趋势详情-体重AI解读.png
Normal file
BIN
docs/release/pics/09-趋势详情-体重AI解读.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 737 KiB |
BIN
docs/release/pics/10-身体档案-已知背景与指标趋势.png
Normal file
BIN
docs/release/pics/10-身体档案-已知背景与指标趋势.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 351 KiB |
BIN
docs/release/pics/11-身体档案-就诊摘要生成结果.png
Normal file
BIN
docs/release/pics/11-身体档案-就诊摘要生成结果.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 357 KiB |
BIN
docs/release/pics/12-药品库-编辑药品.png
Normal file
BIN
docs/release/pics/12-药品库-编辑药品.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
BIN
docs/release/pics/13-记录指标-长期监测预设.png
Normal file
BIN
docs/release/pics/13-记录指标-长期监测预设.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 274 KiB |
BIN
docs/release/pics/14-健康日记-AI追问补充.png
Normal file
BIN
docs/release/pics/14-健康日记-AI追问补充.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 291 KiB |
BIN
docs/release/pics/15-健康日记-本地AI整理中.png
Normal file
BIN
docs/release/pics/15-健康日记-本地AI整理中.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 239 KiB |
BIN
docs/release/pics/16-用药记录-记录用药表单.png
Normal file
BIN
docs/release/pics/16-用药记录-记录用药表单.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 587 KiB |
BIN
docs/release/pics/17-新建菜单-记录入口列表.png
Normal file
BIN
docs/release/pics/17-新建菜单-记录入口列表.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 523 KiB |
BIN
docs/release/pics/18-症状开始-常见症状表单.png
Normal file
BIN
docs/release/pics/18-症状开始-常见症状表单.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 202 KiB |
BIN
docs/release/pics/19-报告归档-核对报告信息.png
Normal file
BIN
docs/release/pics/19-报告归档-核对报告信息.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
BIN
docs/release/pics/20-药品识别-核对药品.png
Normal file
BIN
docs/release/pics/20-药品识别-核对药品.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 321 KiB |
BIN
docs/release/pics/21-首页-健康日历趋势提醒.png
Normal file
BIN
docs/release/pics/21-首页-健康日历趋势提醒.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 454 KiB |
BIN
docs/release/pics/22-框选异常指标-本地识别中.png
Normal file
BIN
docs/release/pics/22-框选异常指标-本地识别中.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 6.9 MiB |
83
docs/release/sensevoice-integration.md
Normal file
83
docs/release/sensevoice-integration.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# 记录问诊 · 本地 SenseVoice 转写接入
|
||||
|
||||
> 「记录问诊」录完整段录音后,在本机用 **SenseVoice**(经 **sherpa-mnn** 跑在 **MNN 后端**)离线转写成文字,
|
||||
> 再交本地 LLM 整理成问诊小结。全程本机、无网络。
|
||||
>
|
||||
> 代码已全部就位且**默认编译通过**:未接入 sherpa-mnn 时 `SenseVoiceBridge` 走桩,问诊自动回退系统端侧识别(SFSpeech)。
|
||||
> 本文档是把真实 SenseVoice 跑起来的「构建 + 设备验证」步骤。
|
||||
|
||||
## 架构(按本项目模块边界 §3.1)
|
||||
|
||||
```
|
||||
ConsultationSheet(UI)
|
||||
→ 录音(ConsultationRecorder,m4a 落 Vault)
|
||||
→ SenseVoiceASRService.transcribe(file) // 离线整段转写,无实时字幕
|
||||
→ AIRuntime.runExclusiveForASR { … } // 进推理闸门 + 卸常驻 LLM/VL 腾内存(防 OOM)
|
||||
→ SenseVoiceBridge(ObjC++) // sherpa-mnn C-API
|
||||
→ sherpa-mnn → MNN 后端(CPU/SME2)
|
||||
→ DiaryAssistService.organizeConsultation(text) // 本地 LLM 整理成问诊小结
|
||||
→ 存为带「问诊」tag 的 DiaryEntry(录音挂 audio Asset)
|
||||
```
|
||||
|
||||
- 引擎/模型未就绪 → 自动回退 SFSpeech;再不行 → 手动文字录入。任何一步都不卡死(红线 #5)。
|
||||
- SenseVoice 是**非流式**:录音中只显示声纹动效,不显示实时字幕,结束后整段转写。
|
||||
|
||||
## 一、构建 sherpa-mnn.xcframework
|
||||
|
||||
```sh
|
||||
MNN_SRC=/Users/xuhuayong/apps/MNN-src sh scripts/build-sherpa-mnn-xcframework.sh
|
||||
```
|
||||
|
||||
- 复用 MNN 源码自带的 `apps/frameworks/sherpa-mnn/build-ios.sh`,产出 `Frameworks/sherpa-mnn.xcframework`
|
||||
(已含 device arm64 + simulator,libtool 合并好的静态库 + `Headers/`)。
|
||||
- 若 sherpa cmake 报找不到 MNN:先 `sh scripts/build-mnn-xcframework.sh` 构建 MNN,
|
||||
再 `export MNN_LIB_DIR=<含 libMNN.a + include 的目录>` 重跑。
|
||||
- Apple Silicon 上若不需要 Intel 模拟器,可在 `build-ios.sh` 里去掉 `simulator_x86_64` 段加速。
|
||||
|
||||
## 二、加进 Xcode 工程
|
||||
|
||||
1. 把 `Frameworks/sherpa-mnn.xcframework` 拖进 target → **Frameworks, Libraries, and Embedded Content**,
|
||||
选 **Do Not Embed**(静态库)。
|
||||
2. **Build Settings → Header Search Paths** 追加(recursive,Debug + Release):
|
||||
```
|
||||
$(PROJECT_DIR)/Frameworks/sherpa-mnn.xcframework/Headers
|
||||
```
|
||||
这样 `SenseVoiceBridge.mm` 里的 `#if __has_include(<sherpa-mnn/c-api/c-api.h>)` 命中,自动切到真实实现。
|
||||
3. sherpa-mnn 用到 C++ 标准库,确保 target 的 **Other Linker Flags** 含 `-lc++`(通常已隐式链接)。
|
||||
4. `MNN.xcframework` 仍需在工程里(sherpa-mnn 依赖它)。即便主 LLM 已切 Gemma-3n/MLX,
|
||||
**不要从工程移除 MNN.xcframework**,否则问诊转写退回 SFSpeech。
|
||||
|
||||
> 工程用 PBXFileSystemSynchronizedRootGroup:`SenseVoiceBridge.{h,mm}` 放在 `康康/AI/MNN/` 下已自动参与编译,
|
||||
> 桥接已在 `康康/康康-Bridging-Header.h` 暴露给 Swift,无需手改 pbxproj。
|
||||
|
||||
## 三、转换并安装 SenseVoice 模型
|
||||
|
||||
```sh
|
||||
MNN_SRC=/Users/xuhuayong/apps/MNN-src sh scripts/convert-sensevoice-mnn.sh
|
||||
```
|
||||
|
||||
产出 `build/SenseVoice/{model.mnn, tokens.txt}`(转的是 **fp32** `model.onnx`,8bit 权重量化;勿转 int8 版)。
|
||||
安装到沙盒 `Application Support/Models/SenseVoice/`:
|
||||
|
||||
- **模拟器**:拷到
|
||||
`~/Library/Developer/CoreSimulator/Devices/<id>/data/Containers/Data/Application/<id>/Library/Application Support/Models/SenseVoice/`
|
||||
- **真机**:经「我的 · 模型管理」旁路导入,或用调试构建预拷进沙盒。
|
||||
|
||||
就位判定:`SenseVoiceASRService.isModelInstalled`(model.mnn + tokens.txt 都在)。
|
||||
引擎+模型都就绪后 `SenseVoiceASRService.isAvailable == true`,问诊即走 SenseVoice。
|
||||
|
||||
## 四、设备验证
|
||||
|
||||
1. 真机打开「记一笔 → 记录问诊 → 开始录音」,说几句(含数值/药名,验数字保真)。
|
||||
2. 「结束并整理」后应看到「正在转写录音 · 本地 SenseVoice」,而非「本机识别」。
|
||||
3. 转写稿交本地 LLM 整理成问诊小结,保存后在记录详情里能回放原声、看小结。
|
||||
4. 故意不装模型 → 应自动回退「本机识别」(SFSpeech),功能仍可用。
|
||||
|
||||
## 备注:与 Gemma-3n/MLX 主线的关系
|
||||
|
||||
- 转写(SenseVoice/MNN)与文本生成(Gemma-3n/MLX)互相独立:ASR 用 MNN 后端,LLM 整理用 MLX。
|
||||
二者经 `AIRuntime` 闸门**串行**,不会同时常驻内存(§3.1)。
|
||||
- 这是目前工程里**唯一**仍依赖 MNN 的路径。若决定彻底移除 MNN,问诊转写会退回 SFSpeech;
|
||||
桥接的 `__has_include` 守卫保证那时仍能编译。
|
||||
- 若希望改走 sherpa-onnx(onnxruntime,不依赖 MNN),只需把 `SenseVoiceBridge.mm` 里的
|
||||
`SherpaMnn*` C-API 换成对应的 `SherpaOnnx*`(结构同名),其余 Swift 层不动。
|
||||
BIN
docs/release/xhs-9grid/00-9宫格总览.png
Normal file
BIN
docs/release/xhs-9grid/00-9宫格总览.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 489 KiB |
BIN
docs/release/xhs-9grid/01-封面-本地健康AI档案.png
Normal file
BIN
docs/release/xhs-9grid/01-封面-本地健康AI档案.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 293 KiB |
BIN
docs/release/xhs-9grid/02-拍报告自动变档案.png
Normal file
BIN
docs/release/xhs-9grid/02-拍报告自动变档案.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 374 KiB |
BIN
docs/release/xhs-9grid/03-Qwen-MNN-SME2性能自检.png
Normal file
BIN
docs/release/xhs-9grid/03-Qwen-MNN-SME2性能自检.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 197 KiB |
BIN
docs/release/xhs-9grid/04-身体档案就诊摘要.png
Normal file
BIN
docs/release/xhs-9grid/04-身体档案就诊摘要.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 281 KiB |
BIN
docs/release/xhs-9grid/05-记录指标到趋势解读.png
Normal file
BIN
docs/release/xhs-9grid/05-记录指标到趋势解读.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 251 KiB |
BIN
docs/release/xhs-9grid/06-药盒识别入药品库.png
Normal file
BIN
docs/release/xhs-9grid/06-药盒识别入药品库.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 229 KiB |
BIN
docs/release/xhs-9grid/07-语音日记AI追问.png
Normal file
BIN
docs/release/xhs-9grid/07-语音日记AI追问.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 183 KiB |
BIN
docs/release/xhs-9grid/08-记录入口和症状追踪.png
Normal file
BIN
docs/release/xhs-9grid/08-记录入口和症状追踪.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 203 KiB |
BIN
docs/release/xhs-9grid/09-开发记录端侧推理.png
Normal file
BIN
docs/release/xhs-9grid/09-开发记录端侧推理.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 216 KiB |
137
docs/release/小红书文案.md
Normal file
137
docs/release/小红书文案.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# 康康 · 小红书发布文案(比赛评审用)
|
||||
|
||||
> 使用说明:
|
||||
> - `◻︎` 处填真机实测数字(打开 我的 → 模型管理 → 性能自检,截图同时把数字抄进来)
|
||||
> - `#比赛官方话题#` 和 `@官方账号` 替换成组委会指定的话题和账号(评审通常按官方话题检索作品,**漏带话题可能查不到你的帖子**)
|
||||
> - 主推版做主帖;技术版可隔 2~3 天发第二篇,小红书对"同一项目多角度连发"权重友好
|
||||
> - 发布时间建议:工作日 12:00–13:30 或 20:00–22:30
|
||||
|
||||
---
|
||||
|
||||
## 版本 A · 主推版(大众 + 评委兼顾)
|
||||
|
||||
### 标题(三选一,均 ≤ 20 字)
|
||||
|
||||
1. 体检报告拍一下,AI 解读不联网📱
|
||||
2. 我做了个不上传的健康 AI,飞行模式都能用
|
||||
3. 爸妈的体检报告,终于有 AI 肯"离线"看了
|
||||
|
||||
### 正文
|
||||
|
||||
体检报告上一堆↑↓箭头,看得懂的没几个;
|
||||
想让 AI 帮忙解读,又得把化验单拍给云端——
|
||||
等于把自己最隐私的数据交出去了。
|
||||
|
||||
所以我做了「康康」:一个 **100% 本地推理** 的健康档案 App🍃
|
||||
所有 AI 都跑在 iPhone 自己的芯片上,**开飞行模式照样用**,数据一个字节都不出手机。
|
||||
|
||||
✅ 它能做什么👇
|
||||
|
||||
📷 **拍一张,报告变档案**
|
||||
化验单/体检报告对着拍,OCR + 端侧大模型自动抽出每项指标、参考范围、偏高偏低,归档成可检索的电子档案。
|
||||
|
||||
📈 **趋势看得见**
|
||||
血压、血糖、体重……长期指标自动画折线,AI 用大白话告诉你"这半年在变好还是变差"。
|
||||
|
||||
💬 **问它,它真的记得你**
|
||||
"我去年尿酸多少?""最近三次血脂对比一下"——它从你自己的历史记录里检索回答,每句话都带引用,点一下能跳回原始报告。
|
||||
|
||||
🗣️ **嘴说就能记**
|
||||
"昨晚头疼,睡得不好"——说一句,自动整理成日记;药盒扫一下,自动录入正在吃的药。
|
||||
|
||||
🏥 **看病前 30 秒**
|
||||
一键生成给医生看的就诊摘要:近期症状 + 关键指标 + 用药过敏史,门诊不再大脑空白。
|
||||
|
||||
🔐 **隐私三件套**
|
||||
系统级硬件加密 + Face ID 锁 + 永久删除。没有账号、没有云、没有"用户协议第 38 条"。
|
||||
|
||||
⚙️ 技术控看这里:
|
||||
端侧跑的是 Qwen3.5 大模型,推理框架是阿里开源的 MNN,在 iPhone 17 上吃满了 Arm 最新的 SME2 矩阵指令——纯 CPU 解码 ◻︎ tok/s,锁屏界面实时显示生成速度,推理快到不像没联网😎
|
||||
|
||||
这是我参加 #比赛官方话题# 的参赛作品,从设计到代码一个人肝了六周。
|
||||
如果你也觉得"健康数据就该留在自己手机里",求个赞和收藏🙏
|
||||
有想要的功能评论区告诉我,下个版本安排!
|
||||
|
||||
⚠️ 康康只做记录和科普式解读,不做诊断不替代医生,身体不舒服请及时就医。
|
||||
|
||||
### 话题标签
|
||||
|
||||
\#比赛官方话题# #端侧AI #本地大模型 #健康管理 #体检报告解读 #隐私保护 #iOS开发 #独立开发者 #AI应用 #数字健康
|
||||
|
||||
### 配图脚本(9 宫格)
|
||||
|
||||
| # | 内容 | 备注 |
|
||||
|---|------|------|
|
||||
| 1 | 封面:手机展示首页 + 大字标题"体检报告 AI 解读,不联网" | 封面字要大,缩略图能读清 |
|
||||
| 2 | 拍照识别报告全流程(拍摄→指标确认页) | 可两张拼一张 |
|
||||
| 3 | 报告详情 C2:原图/解读/指标 三 Tab | 露出"对比上次"区块 |
|
||||
| 4 | 趋势页折线图 + AI 一句话解读 | |
|
||||
| 5 | AI 问答:带 [1][2] 引用 Pill 的回答 | 体现"检索自己的记录" |
|
||||
| 6 | **控制中心飞行模式开启 + App 正常生成回答** 同屏 | 全帖最有说服力的一张 |
|
||||
| 7 | 性能自检卡:SME2 标识 + prefill/decode tok/s | 评委重点看这张 |
|
||||
| 8 | 锁屏 Live Activity 实时 tok/s | |
|
||||
| 9 | 隐私设置页:Face ID + 永久删除 | |
|
||||
|
||||
---
|
||||
|
||||
## 版本 B · 技术圈层版(隔 2~3 天发)
|
||||
|
||||
### 标题(二选一)
|
||||
|
||||
1. 在 iPhone 的 CPU 上,我把大模型跑到 ◻︎ tok/s
|
||||
2. 不用 GPU,iPhone 17 纯 CPU 跑通 Qwen3.5🔥
|
||||
|
||||
### 正文
|
||||
|
||||
最近所有人都在卷云端大模型,我反着来:
|
||||
把整套健康 AI——视觉识别、RAG 问答、趋势解读——全部塞进 iPhone 本地,**纯 CPU 推理**。
|
||||
|
||||
为什么是 CPU 不是 GPU?
|
||||
因为 Arm 在新一代芯片里加了 SME2(可伸缩矩阵扩展):专为矩阵乘法设计的指令集,大模型推理的核心运算正好是它的主场。
|
||||
|
||||
我的技术栈👇
|
||||
🔹 模型:Qwen3.5-2B(多模态,一个模型同时干文本 + 看图识报告)
|
||||
🔹 推理框架:MNN(阿里开源),iPhone 17/A19 走 SME2,老机型自动回退 NEON
|
||||
🔹 兜底:MLX(Apple 官方,Metal GPU),双后端运行时无感切换
|
||||
🔹 应用层:SwiftUI + SwiftData,RAG 用结构化检索(意图抽取→按关键词查库→拼 prompt),不引入 embedding 模型,首响更快
|
||||
|
||||
实测数据(iPhone 17,可在 App 内"性能自检"复现):
|
||||
⚡ prefill ◻︎ tok/s / decode ◻︎ tok/s
|
||||
⚡ 拍一张化验单到出结构化指标:约 ◻︎ 秒
|
||||
⚡ 模型常驻互斥 + actor 串行闸门,长时间使用不 OOM
|
||||
|
||||
几个有意思的坑:
|
||||
1️⃣ MNN 默认 enable_thinking=true,模型疯狂输出 <think> 吃光 token 预算,要在 bridge 层 set_config 关掉
|
||||
2️⃣ 长文本逐行复读死循环——采样器默认不带 repetition penalty,MNN 要显式写进 mixed_samplers
|
||||
3️⃣ LLM 和 VL 同时驻留必 jetsam,做了常驻互斥 + 推理优先级闸门(交互任务可插队后台预生成)
|
||||
|
||||
做这个项目的初衷很简单:健康数据是最不该上云的数据。
|
||||
端侧推理已经到了"真能用"的拐点,这是我给 #比赛官方话题# 交的答卷。
|
||||
|
||||
代码细节/性能调优有兴趣的评论区聊👇
|
||||
|
||||
⚠️ App 仅做记录与科普式解读,不提供诊断建议。
|
||||
|
||||
### 话题标签
|
||||
|
||||
\#比赛官方话题# #端侧AI #MNN #Qwen #ArmSME2 #大模型推理 #iOS开发 #SwiftUI #独立开发者 #本地大模型
|
||||
|
||||
### 配图脚本
|
||||
|
||||
1. 封面:性能自检卡大图,tok/s 数字放大做封面字
|
||||
2. 架构图:UI → Service → AIRuntime → MNN(SME2)/MLX 双后端
|
||||
3. 飞行模式 + 流式生成同屏
|
||||
4. 锁屏 Live Activity tok/s
|
||||
5. 拍照识别报告前后对比(原图 → 结构化指标)
|
||||
6. Xcode/代码截图:MNNLLMBridge 或 actor 闸门片段(打码无关信息)
|
||||
7. 老机型 NEON vs iPhone 17 SME2 速度对比(如有数据)
|
||||
|
||||
---
|
||||
|
||||
## 发布贴士
|
||||
|
||||
1. **官方话题必带且放第一位**,正文里也 @官方账号 一次
|
||||
2. 封面图决定 80% 点击:大字 + 高对比,别用纯截图
|
||||
3. 发布后 1 小时内回评论(尤其问"怎么下载"的,回复"比赛 demo 阶段,关注我等上架"),互动率影响推荐量
|
||||
4. 不要写"治疗""诊断""疗效"等词,健康类内容平台审得严,现有文案已规避
|
||||
5. 主帖发出后把链接填进比赛报名系统/问卷(如果章程要求回填链接)
|
||||
86
docs/release/小红书长文-Google赛道版.md
Normal file
86
docs/release/小红书长文-Google赛道版.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# 康康 · 小红书长文(Google 赛道版)
|
||||
|
||||
> 改写自原 MNN 版。核心转向:出海 / 低资源 / 多语言 / Google 技术(Gemma + Gemini)。
|
||||
> 发布前替换:比赛官方话题、@官方账号、真机数字、demo 视频。
|
||||
|
||||
---
|
||||
|
||||
## 标题备选
|
||||
|
||||
1. 我做了个 App:化验单看不懂?拍一下,断网也能读
|
||||
2. 把 Google 的 AI 塞进手机,帮你读懂外语体检报告
|
||||
3. Day_ | 不上传、断网也能用的健康 AI(Gemma + Gemini)
|
||||
4. 缺医、弱网、看不懂报告——我用端侧 AI 做了个解法
|
||||
|
||||
推荐主标题:**Day_ | 不上传、断网也能用的健康 AI**
|
||||
封面副字:**Gemma 端侧离线 + Gemini 读图/翻译 · 你的母语**
|
||||
|
||||
---
|
||||
|
||||
## 正文主稿
|
||||
|
||||
很多人拿到一张化验单,第一反应是:这些箭头、缩写、参考范围,到底哪项要紧?
|
||||
|
||||
如果身边没有医生,或者报告还是外语的,就更难。而最常见的「拍给云端 AI」方式,又得把姓名、年龄、医院、检查结果这些最私密的信息传上去。
|
||||
|
||||
所以我做了「康康」——一个**默认在手机本地跑、隐私优先**的健康助手。目标人群不只是国内,而是**弱网/缺医地区,和在国外看不懂医疗文件的人**。
|
||||
|
||||
它现在能完成一个完整闭环:
|
||||
|
||||
拍一张报告 → 读出指标(数值、单位、参考范围、异常)→ 存成可检索的档案 → 看趋势 → 问它「最近哪些不正常、就诊前帮我整理重点」。
|
||||
|
||||
技术上我用的是 **Google 的 AI,双路 hybrid**:
|
||||
|
||||
- **端侧默认:Gemma-3n**(Google 开源的手机端多模态小模型)。**飞行模式也能用**——这对网络不稳的地方是刚需。数据不出设备。
|
||||
- **联网增强:Gemini 2.5 Flash**。你主动开启后,**拍报告原图直接让 Gemini 读图**出结构化指标;**外语报告**也能 Gemini 翻译 + 用你的母语讲明白。
|
||||
- 断网或额度用尽,**自动回退端侧**,功能不中断。
|
||||
|
||||
为什么是这套组合?因为低资源场景同时要「离线能用」和「读得准、能翻译」两件事:前者只有端侧小模型能给(Gemma-3n),后者需要强多模态和多语言(Gemini)。一个负责隐私与可用性,一个负责能力,正好互补。
|
||||
|
||||
我特别在意一点:这不是一个「会聊天的健康机器人」,而是把健康资料变成**你自己的长期记忆**——不用再翻相册找去年的化验单,看医生前 30 秒就能整理出重点。它不替你诊断,也不给用药建议,只负责把你已有的信息收好、读懂、讲清楚。
|
||||
|
||||
整个 App 是我用 AI 辅助(vibe coding)在几周内做出来的。
|
||||
|
||||
健康数据,应该先属于你自己——哪怕没有网络,哪怕你在异国他乡。
|
||||
|
||||
#比赛官方话题# #端侧AI #Gemma #Gemini #GoogleAI #本地大模型 #健康管理 #出海 #隐私保护 #iOS开发
|
||||
|
||||
---
|
||||
|
||||
## 9 宫格图片排序(Google 赛道)
|
||||
|
||||
| 顺序 | 图片 | 图中文字 | 作用 |
|
||||
|---|---|---|---|
|
||||
| 1 | 首页 + 封面大字 | 不上传、断网也能用的健康 AI | 先讲是什么 |
|
||||
| 2 | 拍报告 → Gemini 读图结构化 | 拍一张,报告变档案 | 第一卖点(Gemini 核心) |
|
||||
| 3 | 飞行模式 + 正在生成 | 断网也能 AI 解读 | 低资源杀手锏(Gemma 端侧) |
|
||||
| 4 | 外语报告 → 母语解读 | 看不懂的外语报告,翻译+讲明白 | 出海痛点(Gemini 多语言) |
|
||||
| 5 | 我的 · 云端 AI(Gemini 开关) | 隐私优先,云端可选 | hybrid + Google 产品证据 |
|
||||
| 6 | 趋势页 AI 解读 | 长期变化看得懂 | 长期价值 |
|
||||
| 7 | 就诊摘要 | 看医生前自动整理 | 应用价值 |
|
||||
| 8 | 药盒识别 | 药盒也能入档 | 扩展场景 |
|
||||
| 9 | 架构/开发图 | Gemma + Gemini + SwiftUI | 收尾,vibe coding |
|
||||
|
||||
---
|
||||
|
||||
## 封面字方案
|
||||
|
||||
**技术向**
|
||||
- 主字:把 Google AI 塞进手机
|
||||
- 副字:Gemma 端侧离线 + Gemini 读图翻译
|
||||
- 角标:断网也能用
|
||||
|
||||
**大众向**
|
||||
- 主字:化验单看不懂?拍一下
|
||||
- 副字:不上传,断网也能读,还能翻译外语报告
|
||||
- 角标:几周独立开发 demo
|
||||
|
||||
---
|
||||
|
||||
## 仍需补的素材(对应评审「完成度与传播」)
|
||||
|
||||
1. **Gemini 读一张报告的 10 秒录屏**——最值钱的证据,务必有。
|
||||
2. 飞行模式 + 端侧生成。
|
||||
3. 外语报告 → 母语解读。
|
||||
4. 云端 AI 设置页(Gemini 开关)。
|
||||
5. 片头/转场可用 Veo / Google Flow 生成。
|
||||
318
docs/release/小红书长文-项目介绍.md
Normal file
318
docs/release/小红书长文-项目介绍.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# 康康 · 小红书长文正文与 9 宫格方案
|
||||
|
||||
> 参考风格:「Day5｜30小时,用 AI 开发的 APP 完工了」这类独立开发记录口吻。
|
||||
> 使用前替换:比赛官方话题、@官方账号、真机 tok/s 数字、开发耗时。
|
||||
|
||||
---
|
||||
|
||||
## 标题备选
|
||||
|
||||
1. Day42｜我做了个不上传的健康 AI App
|
||||
2. 6 周做完:体检报告 AI 解读,但不联网
|
||||
3. 把大模型塞进 iPhone,做了个健康档案 App
|
||||
4. 体检报告别再乱拍给云端 AI 了
|
||||
5. 用 MNN 在本地跑健康 AI,我把 demo 做完了
|
||||
|
||||
推荐主标题:
|
||||
|
||||
**Day42｜我做了个不上传的健康 AI App**
|
||||
|
||||
副标题可放封面小字:
|
||||
|
||||
**Qwen + MNN + SME2,体检报告在手机本地解读**
|
||||
|
||||
---
|
||||
|
||||
## 正文主稿
|
||||
|
||||
Day42,这个 6 周 demo 终于能完整跑起来了。
|
||||
|
||||
我做了一个叫「康康」的 iOS App。它的目标很简单:把体检报告、化验指标、症状、日记和用药记录整理成一个本地健康档案,再让 AI 用大白话帮你看懂。
|
||||
|
||||
但我给自己加了一条硬限制:
|
||||
|
||||
**健康数据不上传。**
|
||||
|
||||
这件事一开始听起来有点拧巴。因为现在最常见的 AI 解读方式,就是把报告拍照发给云端模型。可体检报告里有姓名、年龄、医院、检查指标、既往问题,基本是一个人最私密的数据之一。
|
||||
|
||||
所以我想做一个反过来的版本:
|
||||
|
||||
不是把报告交给云端 AI,而是把 AI 放进手机里。
|
||||
|
||||
---
|
||||
|
||||
这版「康康」现在能完成一个比较完整的闭环:
|
||||
|
||||
1. 拍一张体检报告或化验单
|
||||
2. 本地识别图片里的指标
|
||||
3. 把数值、单位、参考范围、异常状态整理成结构化档案
|
||||
4. 后续可以看趋势,比如体重、血糖、血脂、血压
|
||||
5. 也可以问它:最近哪些指标异常?就诊前帮我整理一份摘要
|
||||
|
||||
整个过程不需要账号,不接云服务,没有广告 SDK,也没有第三方分析 SDK。
|
||||
|
||||
数据存在 iPhone 本地。报告原图进本地 Vault,SwiftData 存结构化记录,Face ID 可以给 App 加启动锁。
|
||||
|
||||
---
|
||||
|
||||
技术上最核心的一点是端侧推理。
|
||||
|
||||
这次我主路径用的是:
|
||||
|
||||
**Qwen3.5-2B + MNN + iPhone CPU/SME2**
|
||||
|
||||
MNN 是阿里开源的推理框架。我的目标不是“能接上一个 API 就算 AI App”,而是让模型真的在手机本地吐 token。
|
||||
|
||||
App 里做了一个推理引擎页,可以看到当前后端:
|
||||
|
||||
- 自动模式:真机优先走 MNN
|
||||
- MNN:CPU + SME2/NEON
|
||||
- MLX:兜底和对照
|
||||
|
||||
在支持 SME2 的设备上,MNN 会走端侧 CPU 加速。性能自检页会跑固定 prompt,直接显示 prefill 和 decode 速度。这里不是 PPT 数字,是 App 内可以复现的实测。
|
||||
|
||||
目前真机结果:
|
||||
|
||||
**读入:122 tok/s,生成:33.0 tok/s,总耗时:1.1s**
|
||||
|
||||
这个速度已经足够支撑健康档案里的短问答、趋势解读和报告整理。
|
||||
|
||||
---
|
||||
|
||||
为了让它不是“套壳聊天框”,我把 AI 链路拆成了几层:
|
||||
|
||||
UI 不直接调模型。
|
||||
|
||||
拍照识别走 CaptureService,身体档案问答走 HealthExportService,趋势解读走 TrendInsightService,最后统一进入 AIRuntime。
|
||||
|
||||
AIRuntime 是 actor,里面做了推理闸门。同一时间只允许一个重推理任务占模型内存。因为端侧模型最怕的不是慢,而是多个任务同时加载,内存峰值上去以后直接被系统杀掉。
|
||||
|
||||
这也是做端侧 AI 和做云端 API 最大的区别之一:
|
||||
|
||||
云端是请求排队。
|
||||
|
||||
手机上是内存、功耗、模型加载、UI 响应一起排队。
|
||||
|
||||
---
|
||||
|
||||
我觉得这个项目真正有价值的地方,不是“AI 能不能讲两句漂亮话”,而是它把健康资料变成了自己的长期记忆。
|
||||
|
||||
比如:
|
||||
|
||||
你不用再翻相册找去年那张化验单。
|
||||
|
||||
也不用每次看医生前,临时回忆“我最近到底哪几项不正常”。
|
||||
|
||||
你可以把症状、用药、指标、报告都记在一个地方,需要时生成一份就诊摘要。它不会替医生判断,也不会给剂量建议,只负责把你已有的信息整理清楚。
|
||||
|
||||
对普通人来说,这比一个会聊天的健康机器人更实用。
|
||||
|
||||
---
|
||||
|
||||
这次我最想表达的创新点有三个:
|
||||
|
||||
第一,健康 AI 应该默认本地优先。
|
||||
|
||||
不是所有数据都适合上传。健康数据尤其应该有“不出设备也能用”的选择。
|
||||
|
||||
第二,端侧大模型已经能做完整产品闭环。
|
||||
|
||||
不是只跑一个 demo prompt,而是接进真实 App:模型管理、性能自检、拍照识别、趋势解读、问答摘要、失败兜底、隐私设置,这些都要一起工作。
|
||||
|
||||
第三,RAG 不一定非要 embedding。
|
||||
|
||||
康康里的健康记录本来就是结构化数据。指标名、时间、报告类型、症状、用药都很明确。很多问题可以先抽取意图,再从 SwiftData 里查相关记录,最后让本地模型生成回答。这样更轻,也更适合手机。
|
||||
|
||||
---
|
||||
|
||||
现在还有很多地方可以继续打磨,比如报告详情页、飞行模式演示图、Live Activity 的实时 tok/s、更多真机性能对比。
|
||||
|
||||
但这版已经证明了一件事:
|
||||
|
||||
**个人健康档案 + 本地大模型 + MNN 端侧推理,是可以组成一个完整应用的。**
|
||||
|
||||
如果说云端 AI 更像一个很聪明的外部顾问,那我希望康康更像一个安静放在手机里的私人健康档案员。
|
||||
|
||||
它不替你诊断。
|
||||
|
||||
不替医生做决定。
|
||||
|
||||
它只帮你把自己的身体记录收好、看懂、在需要的时候讲清楚。
|
||||
|
||||
这就是我这次做「康康」的原因。
|
||||
|
||||
参加 #比赛官方话题# 的作品记录。后面如果有时间,我会继续补飞行模式演示、Live Activity 和完整 demo 视频。
|
||||
|
||||
声明:康康只做健康信息记录、整理和科普式解读,不是医疗器械,不提供诊断、治疗、用药或剂量建议。任何健康决策请咨询专业医生。
|
||||
|
||||
---
|
||||
|
||||
## 精简版正文
|
||||
|
||||
我做了一个叫「康康」的 iOS App。
|
||||
|
||||
它能把体检报告、化验指标、症状、日记和用药记录整理成一个本地健康档案,再用 AI 帮你做大白话解读。
|
||||
|
||||
最关键的是:不上传。
|
||||
|
||||
我不想把体检报告这种高度隐私的数据拍给云端模型,所以这次把 Qwen3.5-2B 放进了手机本地,主推理路径用 MNN,在支持 SME2 的 iPhone 上走端侧 CPU 加速。
|
||||
|
||||
现在 App 里已经能看到完整闭环:
|
||||
|
||||
拍报告 → 本地识别 → 结构化指标 → 趋势图 → 身体档案问答 → 就诊摘要。
|
||||
|
||||
技术上做了双后端:
|
||||
|
||||
- MNN:真机主路径,CPU/SME2
|
||||
- MLX:模拟器和兜底
|
||||
|
||||
App 里还有性能自检页,当前真机实测:
|
||||
|
||||
读入 122 tok/s,生成 33.0 tok/s,总耗时 1.1s。
|
||||
|
||||
我觉得这个项目最有意思的地方,不是做了一个 AI 聊天框,而是把本地大模型真正接进了健康档案系统。
|
||||
|
||||
它不会替你诊断,也不会给用药建议。它只做三件事:
|
||||
|
||||
帮你收好记录,帮你看懂变化,帮你在看医生前把重点讲清楚。
|
||||
|
||||
健康数据应该先属于自己。
|
||||
|
||||
这就是我做「康康」的原因。
|
||||
|
||||
#比赛官方话题# #端侧AI #MNN #Qwen #本地大模型 #健康管理 #iOS开发 #SwiftUI #隐私保护 #独立开发
|
||||
|
||||
---
|
||||
|
||||
## 9 宫格图片排序
|
||||
|
||||
### 当前素材可发版
|
||||
|
||||
| 顺序 | 图片 | 封面字/图中文字 | 作用 |
|
||||
|---|---|---|---|
|
||||
| 1 | `21-首页-健康日历趋势提醒.png` | 不上传的健康 AI 档案 | 封面,先讲产品是什么 |
|
||||
| 2 | `19-报告归档-核对报告信息.png` | 拍报告,自动变档案 | 展示核心创意:报告归档 |
|
||||
| 3 | `22-框选异常指标-本地识别中.png` | 图片在本地识别 | 展示视觉识别过程 |
|
||||
| 4 | `06-推理引擎-MNN-SME2性能自检.png` | Qwen + MNN + SME2 | 技术证据和比赛亮点 |
|
||||
| 5 | `07-模型管理-Qwen已就绪.png` | 1.19GB 模型在手机里 | 证明不是云端 API |
|
||||
| 6 | `11-身体档案-就诊摘要生成结果.png` | 就诊前 30 秒整理重点 | 展示应用价值 |
|
||||
| 7 | `09-趋势详情-体重AI解读.png` | 长期趋势,AI 讲人话 | 展示长期使用价值 |
|
||||
| 8 | `20-药品识别-核对药品.png` | 药盒也能本地识别 | 展示扩展场景 |
|
||||
| 9 | `02-Xcode开发调试界面.png` | 6 周独立开发记录 | 收尾,增强真实开发感 |
|
||||
|
||||
### 如果补图后的更强版
|
||||
|
||||
| 顺序 | 建议图片 | 封面字/图中文字 |
|
||||
|---|---|---|
|
||||
| 1 | 首页 + 大字封面 | 不上传的健康 AI 档案 |
|
||||
| 2 | 飞行模式 + 正在生成 | 断网也能 AI 解读 |
|
||||
| 3 | 报告拍照前后对比 | 拍一下,报告变档案 |
|
||||
| 4 | 报告详情三 Tab | 原图、解读、指标都留存 |
|
||||
| 5 | MNN 性能自检 | Qwen + MNN + SME2 |
|
||||
| 6 | 模型管理页 | 模型真的在本机 |
|
||||
| 7 | 身体档案摘要 | 看医生前自动整理 |
|
||||
| 8 | 趋势页 | 长期变化看得懂 |
|
||||
| 9 | Xcode/架构图 | SwiftUI + SwiftData + MNN |
|
||||
|
||||
---
|
||||
|
||||
## 封面字方案
|
||||
|
||||
### 方案 A:大众向
|
||||
|
||||
主字:
|
||||
|
||||
**体检报告 AI 解读**
|
||||
|
||||
副字:
|
||||
|
||||
**不联网,不上传,只在手机里跑**
|
||||
|
||||
角标:
|
||||
|
||||
**6 周独立开发 demo**
|
||||
|
||||
### 方案 B:技术向
|
||||
|
||||
主字:
|
||||
|
||||
**把 Qwen 塞进 iPhone**
|
||||
|
||||
副字:
|
||||
|
||||
**MNN + SME2 本地推理健康档案**
|
||||
|
||||
角标:
|
||||
|
||||
**decode 33.0 tok/s**
|
||||
|
||||
### 方案 C:参考 Day 风格
|
||||
|
||||
主字:
|
||||
|
||||
**Day42 做完一个本地健康 AI**
|
||||
|
||||
副字:
|
||||
|
||||
**体检报告拍一下,但不交给云端**
|
||||
|
||||
角标:
|
||||
|
||||
**SwiftUI + MNN + Qwen**
|
||||
|
||||
推荐使用方案 C,更贴近你给的参考笔记风格,同时还能保留项目技术点。
|
||||
|
||||
---
|
||||
|
||||
## 每张图建议加字
|
||||
|
||||
1. `21-首页-健康日历趋势提醒.png`
|
||||
- 主字:Day42 做完一个本地健康 AI
|
||||
- 小字:体检报告、指标、症状、用药都存在手机里
|
||||
|
||||
2. `19-报告归档-核对报告信息.png`
|
||||
- 主字:拍报告,自动变档案
|
||||
- 小字:原图和结构化信息一起保存
|
||||
|
||||
3. `22-框选异常指标-本地识别中.png`
|
||||
- 主字:图片识别也在本地
|
||||
- 小字:异常指标可框选识别
|
||||
|
||||
4. `06-推理引擎-MNN-SME2性能自检.png`
|
||||
- 主字:Qwen + MNN + SME2
|
||||
- 小字:生成 33.0 tok/s
|
||||
|
||||
5. `07-模型管理-Qwen已就绪.png`
|
||||
- 主字:模型真的在手机里
|
||||
- 小字:Qwen3.5-2B 本地就绪
|
||||
|
||||
6. `11-身体档案-就诊摘要生成结果.png`
|
||||
- 主字:看医生前自动整理
|
||||
- 小字:症状、指标、用药一份摘要带走
|
||||
|
||||
7. `09-趋势详情-体重AI解读.png`
|
||||
- 主字:长期变化看得懂
|
||||
- 小字:折线图 + AI 大白话解读
|
||||
|
||||
8. `20-药品识别-核对药品.png`
|
||||
- 主字:药盒也能入档
|
||||
- 小字:只记录,不提供用药建议
|
||||
|
||||
9. `02-Xcode开发调试界面.png`
|
||||
- 主字:不是套壳聊天框
|
||||
- 小字:SwiftUI + SwiftData + MNN 端侧推理
|
||||
|
||||
---
|
||||
|
||||
## 仍建议补充的截图
|
||||
|
||||
1. 飞行模式打开 + App 正在流式生成,这是“不上传”的最强证据。
|
||||
2. 报告详情页三 Tab:原图 / 解读 / 指标,用于证明档案浏览闭环。
|
||||
3. Face ID 启动锁页面,用于补足隐私三件套。
|
||||
4. 如果 Live Activity 已能跑,补锁屏 tok/s,非常适合比赛记忆点。
|
||||
|
||||
---
|
||||
|
||||
## 标签
|
||||
|
||||
#比赛官方话题# #端侧AI #MNN #Qwen #本地大模型 #健康管理 #体检报告解读 #隐私保护 #iOS开发 #SwiftUI #独立开发 #数字健康
|
||||
BIN
docs/release/康康创意方案文档.docx
Normal file
BIN
docs/release/康康创意方案文档.docx
Normal file
Binary file not shown.
41
docs/research/mnn-kv-cache-prefix.md
Normal file
41
docs/research/mnn-kv-cache-prefix.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# MNN 前缀 KV Cache 调研(2026-06-10)
|
||||
|
||||
## 结论
|
||||
|
||||
当前打包的 MNN.xcframework 已暴露 prefix cache 能力,技术上可以把每个场景**固定的
|
||||
system prompt + few-shot 模板**的 prefill 结果缓存到磁盘,二次调用跳过这部分 prefill。
|
||||
**建议 W6 polish 阶段、用性能自检卡量化 prefill 占比之后再决定是否接入**;当前瓶颈在
|
||||
decode 而非二次 prefill,优先级低于 C1/C2/Live Activity。
|
||||
|
||||
## 依据(`Frameworks/MNN.xcframework/ios-arm64/MNN.framework/Headers/llm/llm.hpp`)
|
||||
|
||||
| API | 行号 | 含义 |
|
||||
|---|---|---|
|
||||
| `bool setPrefixCacheFile(const std::string& filename, int flag = 0)` | :161 | 指定前缀缓存文件;配套私有成员 `mPrefixCacheMode` / `mPrefixLength` / `mIsPrefixFileExist` / `completePrefixWrite()`(:250-255)印证:命中时 prefill 只算增量部分 |
|
||||
| `bool reuse_kv()` | :171 | 读 config 开关 `reuse_kv`,多轮对话内复用 KV(同一会话增量 prefill) |
|
||||
| `void syncPromptCache(const ChatMessages&)` | :176 | decode 结束后同步缓存文本——注释明确说明 cache 在 generate() 后自更新,此接口供做过后处理(如 deleteThinkPart)的调用方提供更准确版本 |
|
||||
| `void setKVCacheInfo(size_t add, size_t remove, ...)` / `eraseHistory(begin, end)` | :158-160 | 更底层的 KV 区间管理,可做部分历史擦除 |
|
||||
|
||||
## 对本项目的适用性
|
||||
|
||||
- 我们所有调用都是「固定模板前缀 + 可变数据后缀」的单轮 `response()`,与 prefix cache
|
||||
的模型吻合。
|
||||
- 模板体量(估):报告识别 ~900 tok、导出报告 ~700 tok、意图抽取 ~300 tok。
|
||||
按性能自检卡实测的 prefill 速率推算,每次调用预计省 **1~3s**。
|
||||
- 多场景共用一个 cache 文件是否支持多前缀未知;最坏情况只对单一场景(建议选「报告识别」,
|
||||
模板最长、调用最频繁)生效。
|
||||
|
||||
## 风险
|
||||
|
||||
1. `flag` 参数语义在头文件无注释,需读 MNN 源码或实验确认。
|
||||
2. OMNI(多模态)分支下行为未验证——我们的 MNN 模型是 Omni 构建。
|
||||
3. cache 文件与模型权重版本绑定:模型更新/重下载后必须失效,否则可能输出乱码。
|
||||
4. `<img>` 标签在 prompt 前部(`analyzeImages` 把图片标签拼在最前),意味着报告识别场景的
|
||||
"固定前缀" 实际不固定 —— **文本场景(导出/意图抽取)才是干净的 prefix cache 候选**。
|
||||
|
||||
## 建议的接入步骤(W6,如性能自检显示 prefill 占比 >30%)
|
||||
|
||||
1. `MNNLLMBridge` init 后调 `setPrefixCacheFile(<AppSupport>/mnn-prefix.cache)`(仅文本场景)。
|
||||
2. 真机 A/B:同一导出报告各跑 3 次,对比 `LlmContext.prefill_us`。
|
||||
3. 异常处理:加载失败或输出劣化时删除 cache 文件并禁用,回退现状。
|
||||
4. `ModelDownloadService.importModel` / 重下载路径上顺手删除旧 cache 文件。
|
||||
1626
docs/superpowers/plans/2026-06-10-competition-optimizations.md
Normal file
1626
docs/superpowers/plans/2026-06-10-competition-optimizations.md
Normal file
File diff suppressed because it is too large
Load Diff
930
docs/superpowers/plans/2026-06-10-voice-diary.md
Normal file
930
docs/superpowers/plans/2026-06-10-voice-diary.md
Normal file
@@ -0,0 +1,930 @@
|
||||
# 语音健康日记 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 在「健康记录」(`DiaryQuickSheet`)加语音输入:iOS 端侧流式语音识别实时转写,停止后由本地 LLM(Qwen3.5-2B,经 AIRuntime)整理成健康日记草稿,追加进输入框,可一键回退原话。
|
||||
|
||||
**Architecture:** `DiaryQuickSheet`(mic 按钮 + 状态机)→ `SpeechDictationService`(新,AVAudioEngine + SFSpeechRecognizer 端侧流式转写,不落盘音频)→ `DiaryAssistService.organize(transcript:)`(新方法,经 AIRuntime actor 队列)。Spec:`docs/superpowers/specs/2026-06-10-voice-diary-design.md`。
|
||||
|
||||
**Tech Stack:** SwiftUI、Speech framework(`requiresOnDeviceRecognition = true`)、AVFoundation、Swift Testing(`康康Tests`)。
|
||||
|
||||
**工程约定(执行前必读):**
|
||||
- 工程是 Xcode 16 同步组(`PBXFileSystemSynchronizedRootGroup`):`康康/`、`康康Tests/` 下新建文件**自动入 target,不要改 pbxproj 的文件列表**(权限键除外,见 Task 1)。
|
||||
- CLI 编译/测试必须:`export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer`,且加 `-derivedDataPath ./build/cli-dd`(避免和 Xcode 抢 build.db 锁)。
|
||||
- 工程 `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor`:类型默认 MainActor;系统回调闭包(audio tap、recognitionTask handler)是 nonisolated,**闭包内只碰局部捕获变量,回主线程用 `Task { @MainActor in }`**。
|
||||
- 用户可见文案用 `String(appLoc: "...")`;字号用 `Font.tjScaled(...)`,禁止裸 `.system(size:)`;颜色只用 `Tj.Palette.*`。**不要手改 `Localizable.xcstrings`**(键缺失时回退键名本身,中文键名即兜底文案)。
|
||||
- `git status` 里已有 `康康/Localizable.xcstrings` 的无关改动——**任何 commit 都不要带上它**(逐文件 `git add`)。
|
||||
- spec 偏差说明(已确认的两处小调整):① CLAUDE.md 提到的 `DebugAIRunner` 已不在工程中,prompt 自检改为 `康康Tests` 单元测试 + 真机手测清单;② mic 按钮放「内容」section 标签行右侧(而非输入框内右下角 overlay),避免与文字重叠,仍属"输入框旁"。
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 建独立分支
|
||||
|
||||
**Files:** 无(纯 git)
|
||||
|
||||
- [x] **Step 1: 从当前分支建 `feat/voice-diary`**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
git checkout -b feat/voice-diary
|
||||
```
|
||||
|
||||
Expected: `Switched to a new branch 'feat/voice-diary'`(`Localizable.xcstrings` 的本地改动会跟着工作区走,不影响)。
|
||||
|
||||
---
|
||||
|
||||
### Task 1: 新增麦克风 + 语音识别权限描述(pbxproj)
|
||||
|
||||
**Files:**
|
||||
- Modify: `康康.xcodeproj/project.pbxproj:430`、`康康.xcodeproj/project.pbxproj:486`(Debug + Release 两个构建配置)
|
||||
|
||||
pbxproj 的 `INFOPLIST_KEY_*` 按字母序排列:Microphone 插在 `NSHealthUpdateUsageDescription` 之后,SpeechRecognition 插在 `NSPhotoLibraryUsageDescription` 之后。每个锚点行在文件中出现 **2 次**(Debug/Release),用 replace_all 一次改两处。
|
||||
|
||||
- [x] **Step 1: 插入 NSMicrophoneUsageDescription(replace_all)**
|
||||
|
||||
用 Edit 工具,`replace_all: true`:
|
||||
|
||||
old_string(注意行首是 4 个 tab):
|
||||
```
|
||||
INFOPLIST_KEY_NSHealthUpdateUsageDescription = "康康不会写入 Apple 健康数据。此说明用于满足 HealthKit 权限校验,你的健康资料只保留在本机。";
|
||||
```
|
||||
|
||||
new_string:
|
||||
```
|
||||
INFOPLIST_KEY_NSHealthUpdateUsageDescription = "康康不会写入 Apple 健康数据。此说明用于满足 HealthKit 权限校验,你的健康资料只保留在本机。";
|
||||
INFOPLIST_KEY_NSMicrophoneUsageDescription = "康康需要使用麦克风进行语音记录,识别全程在本机完成,声音不会上传。";
|
||||
```
|
||||
|
||||
- [x] **Step 2: 插入 NSSpeechRecognitionUsageDescription(replace_all)**
|
||||
|
||||
old_string:
|
||||
```
|
||||
INFOPLIST_KEY_NSPhotoLibraryUsageDescription = "康康需要读取你已有的体检/化验报告照片用于本地识别,不会上传。";
|
||||
```
|
||||
|
||||
new_string:
|
||||
```
|
||||
INFOPLIST_KEY_NSPhotoLibraryUsageDescription = "康康需要读取你已有的体检/化验报告照片用于本地识别,不会上传。";
|
||||
INFOPLIST_KEY_NSSpeechRecognitionUsageDescription = "语音转文字使用 iOS 端侧识别,内容不会发送给 Apple 或任何服务器。";
|
||||
```
|
||||
|
||||
- [x] **Step 3: 验证两个键各出现 2 次**
|
||||
|
||||
```bash
|
||||
grep -c "NSMicrophoneUsageDescription\|NSSpeechRecognitionUsageDescription" 康康.xcodeproj/project.pbxproj
|
||||
```
|
||||
|
||||
Expected: `4`
|
||||
|
||||
- [x] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康.xcodeproj/project.pbxproj
|
||||
git commit -m "feat(语音日记): 新增麦克风与语音识别权限描述(端侧识别文案)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: organize prompt(TDD)
|
||||
|
||||
**Files:**
|
||||
- Test: `康康Tests/DiaryOrganizePromptTests.swift`(新建)
|
||||
- Modify: `康康/AI/Prompts/DiaryAssistPrompts.swift`(文件末尾 `}` 前加方法)
|
||||
|
||||
- [x] **Step 1: 写失败测试**
|
||||
|
||||
新建 `康康Tests/DiaryOrganizePromptTests.swift`:
|
||||
|
||||
```swift
|
||||
import Testing
|
||||
@testable import 康康
|
||||
|
||||
struct DiaryOrganizePromptTests {
|
||||
@Test func organizePromptContainsTranscriptAndHardRules() {
|
||||
let prompt = DiaryAssistPrompts.organize(transcript: "今天早上头晕量了血压140 90")
|
||||
|
||||
#expect(prompt.contains("今天早上头晕量了血压140 90"))
|
||||
// 健康数据红线:数值/单位/药名/时间不许改,必须写进 prompt
|
||||
#expect(prompt.contains("数值"))
|
||||
#expect(prompt.contains("药名"))
|
||||
// 自适应样式两条规则都在
|
||||
#expect(prompt.contains("一段通顺的话"))
|
||||
#expect(prompt.contains("分行"))
|
||||
// 项目 prompt 规范:禁思考标签
|
||||
#expect(prompt.contains("/no_think"))
|
||||
}
|
||||
|
||||
@Test func organizePromptTruncatesLongTranscript() {
|
||||
let long = String(repeating: "头晕", count: 2000) // 4000 字符,超过上限
|
||||
let prompt = DiaryAssistPrompts.organize(transcript: long)
|
||||
|
||||
// 整条 prompt 里口述部分被截断到 organizeTranscriptLimit
|
||||
let expectedTail = String(long.prefix(DiaryAssistPrompts.organizeTranscriptLimit))
|
||||
#expect(prompt.contains(expectedTail))
|
||||
#expect(!prompt.contains(String(long.prefix(DiaryAssistPrompts.organizeTranscriptLimit + 2))))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: 跑测试确认编译失败(方法还不存在)**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||
xcodebuild test -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-only-testing:'康康Tests/DiaryOrganizePromptTests' \
|
||||
-derivedDataPath ./build/cli-dd CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20
|
||||
```
|
||||
|
||||
Expected: 编译错误 `type 'DiaryAssistPrompts' has no member 'organize'`(TEST FAILED)。
|
||||
|
||||
- [x] **Step 3: 实现 organize prompt**
|
||||
|
||||
在 `康康/AI/Prompts/DiaryAssistPrompts.swift` 的 enum 末尾(`suggest` 方法后、收尾 `}` 前)加:
|
||||
|
||||
```swift
|
||||
// MARK: - 语音口述 → 日记整理
|
||||
|
||||
/// 口述转写稿截断上限(字符)。2B 模型 context 保护:超长口述只取前面部分。
|
||||
static let organizeTranscriptLimit = 1200
|
||||
|
||||
/// 把语音转写稿整理成健康日记草稿。自适应样式:内容少 → 一段通顺的话;
|
||||
/// 多方面 → 按「方面:内容」分行。
|
||||
/// 红线(spec §2):只重组语言,严禁增删改任何数值、单位、药名、时间——
|
||||
/// 2B 模型把 140/90 改成 130/90 即健康数据事故,所以规则放第一条并配 few-shot 强化。
|
||||
static func organize(transcript: String) -> String {
|
||||
let trimmed = String(transcript.prefix(organizeTranscriptLimit))
|
||||
return """
|
||||
你是健康记录助手。下面是用户口述身体状态的语音转写原话,可能口语化、有重复、缺标点。
|
||||
请把它整理成一条清晰的健康日记。
|
||||
|
||||
硬性规则:
|
||||
- 【绝对不许】增加、删除或改动任何数值、单位、药名、时间——原话说 140/90 就必须写 140/90。
|
||||
- 只重组语言:去掉口头语和重复;用第一人称;不加入原话没有的事实。
|
||||
- 内容只涉及一两个方面 → 整理成一段通顺的话(2-4 句)。
|
||||
- 内容涉及多个方面(症状/用药/饮食/睡眠/运动等) → 按「方面:内容」分行。
|
||||
- 不诊断、不给用药建议、不写「建议就医」。
|
||||
- 只输出整理后的日记正文,不要解释、不要 markdown 围栏、不要 <think> 标签。
|
||||
|
||||
示例 1(口述:那个今天早上起来有点头晕然后我量了下血压140 90比平时高一点没吃早饭就出门了):
|
||||
今天早上起来有点头晕,量了血压 140/90,比平时高一点。没吃早饭就出门了。
|
||||
|
||||
示例 2(口述:今天头晕了一上午下午好点了血压早上量的140 90嗯缬沙坦吃了降脂药忘了吃早饭没吃中午吃的清淡晚上散步了半小时):
|
||||
症状:头晕了一上午,下午好转。
|
||||
血压:早上 140/90。
|
||||
用药:缬沙坦已服,降脂药忘服。
|
||||
饮食:早饭未吃,午餐清淡。
|
||||
运动:晚上散步半小时。
|
||||
|
||||
【口述原话】:
|
||||
\(trimmed)
|
||||
|
||||
Output: /no_think
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: 跑测试确认通过**
|
||||
|
||||
同 Step 2 命令。Expected: `** TEST SUCCEEDED **`,2 个用例通过。
|
||||
|
||||
- [x] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康Tests/DiaryOrganizePromptTests.swift 康康/AI/Prompts/DiaryAssistPrompts.swift
|
||||
git commit -m "feat(语音日记): organize prompt(自适应样式 + 数值不可改红线)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: DiaryAssistService.organize
|
||||
|
||||
**Files:**
|
||||
- Modify: `康康/Services/DiaryAssistService.swift:99` 之后(`suggest` 方法后、struct 收尾 `}` 前)
|
||||
|
||||
无新单测(纯转发 AIRuntime,LLM 行为靠真机手测;解析逻辑只有 strip + trim,复用已测过的 `stripThinkBlocks`)。
|
||||
|
||||
- [x] **Step 1: 加 organize 方法**
|
||||
|
||||
在 `suggest` 方法的收尾 `}` 之后、struct 收尾 `}` 之前加:
|
||||
|
||||
```swift
|
||||
/// 把语音转写稿整理成健康日记草稿(spec 2026-06-10-voice-diary)。
|
||||
/// 失败(模型未就绪 / 输出为空)抛错,调用方回退为直接使用原话,不卡死。
|
||||
/// 与 suggest 同样走 AIRuntime actor 队列,自然与追问/拍照串行。
|
||||
func organize(transcript: String) async throws -> (text: String, decodeRate: Double) {
|
||||
do {
|
||||
try await AIRuntime.shared.prepare()
|
||||
} catch {
|
||||
throw AssistError.modelNotReady
|
||||
}
|
||||
|
||||
let prompt = DiaryAssistPrompts.organize(transcript: transcript)
|
||||
var collected = ""
|
||||
var lastRate: Double = 0
|
||||
let stream = await AIRuntime.shared.generate(prompt: prompt, maxTokens: 400)
|
||||
for try await chunk in stream {
|
||||
collected += chunk.text
|
||||
if chunk.decodeRate > 0 { lastRate = chunk.decodeRate }
|
||||
}
|
||||
|
||||
let text = HealthExportService.stripThinkBlocks(collected)
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !text.isEmpty else { throw AssistError.empty }
|
||||
return (text, lastRate)
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: 编译验证**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||
xcodebuild -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-configuration Debug build -derivedDataPath ./build/cli-dd \
|
||||
CODE_SIGNING_ALLOWED=NO 2>&1 | grep -E "\.swift:[0-9]+:[0-9]+: (error|warning):|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
|
||||
Expected: `BUILD SUCCEEDED`,无新增 warning。
|
||||
|
||||
- [x] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康/Services/DiaryAssistService.swift
|
||||
git commit -m "feat(语音日记): DiaryAssistService.organize 转写稿整理"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: SpeechDictationService(端侧流式转写)
|
||||
|
||||
**Files:**
|
||||
- Create: `康康/Services/SpeechDictationService.swift`
|
||||
|
||||
硬件绑定,无单测;模拟器路径(`isAvailable == false`)与真机路径在 Task 7 手测。
|
||||
|
||||
- [x] **Step 1: 新建 SpeechDictationService.swift**
|
||||
|
||||
```swift
|
||||
import Foundation
|
||||
import Speech
|
||||
import AVFoundation
|
||||
|
||||
/// 端侧流式语音转写(spec 2026-06-10-voice-diary)。
|
||||
/// AVAudioEngine 麦克风 buffer → SFSpeechAudioBufferRecognitionRequest,
|
||||
/// `requiresOnDeviceRecognition = true` 硬性端侧,识别内容不出设备;**不落盘任何音频**。
|
||||
///
|
||||
/// 生命周期:start(onPartial:) 开始录音并实时回调 partial;stop() 结束并返回最终稿。
|
||||
/// 调用方:DiaryQuickSheet。工程默认 MainActor 隔离,本类型即 MainActor;
|
||||
/// audio tap 与识别回调在系统线程,闭包内只碰局部捕获对象,回主线程统一走 Task { @MainActor }。
|
||||
final class SpeechDictationService {
|
||||
|
||||
enum DictationError: Error, LocalizedError {
|
||||
case unavailable
|
||||
case audioEngineStartFailed(String)
|
||||
|
||||
var errorDescription: String? {
|
||||
switch self {
|
||||
case .unavailable:
|
||||
return String(appLoc: "本机不支持端侧语音识别")
|
||||
case .audioEngineStartFailed(let m):
|
||||
return String(appLoc: "录音启动失败:\(m)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 优先系统语言;系统语言不支持端侧时兜底中文(demo 机即使系统是英文也能用)。
|
||||
private static func makeRecognizer() -> SFSpeechRecognizer? {
|
||||
if let r = SFSpeechRecognizer(locale: .current), r.supportsOnDeviceRecognition {
|
||||
return r
|
||||
}
|
||||
if let r = SFSpeechRecognizer(locale: Locale(identifier: "zh-CN")),
|
||||
r.supportsOnDeviceRecognition {
|
||||
return r
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
/// 本机是否支持端侧识别。false(模拟器/老机型)时 UI 隐藏 mic 入口,静默降级。
|
||||
static var isAvailable: Bool { makeRecognizer() != nil }
|
||||
|
||||
private let audioEngine = AVAudioEngine()
|
||||
private var request: SFSpeechAudioBufferRecognitionRequest?
|
||||
private var task: SFSpeechRecognitionTask?
|
||||
/// 识别回调持续刷新;isFinal 或出错时置 didFinish。stop() 用「final 优先、partial 兜底」。
|
||||
private var latestText = ""
|
||||
private var didFinish = false
|
||||
|
||||
private(set) var isRecording = false
|
||||
|
||||
/// 麦克风 + 语音识别两个权限一起申请。任一被拒返回 false。
|
||||
func requestAuthorization() async -> Bool {
|
||||
let speech = await withCheckedContinuation { (c: CheckedContinuation<SFSpeechRecognizerAuthorizationStatus, Never>) in
|
||||
SFSpeechRecognizer.requestAuthorization { c.resume(returning: $0) }
|
||||
}
|
||||
guard speech == .authorized else { return false }
|
||||
return await AVAudioApplication.requestRecordPermission()
|
||||
}
|
||||
|
||||
/// 开始录音 + 流式识别。partial 结果在主线程回调(录音面板实时字幕)。
|
||||
func start(onPartial: @escaping (String) -> Void) throws {
|
||||
guard !isRecording else { return }
|
||||
guard let recognizer = Self.makeRecognizer(), recognizer.isAvailable else {
|
||||
throw DictationError.unavailable
|
||||
}
|
||||
|
||||
let session = AVAudioSession.sharedInstance()
|
||||
do {
|
||||
try session.setCategory(.record, mode: .measurement, options: .duckOthers)
|
||||
try session.setActive(true, options: .notifyOthersOnDeactivation)
|
||||
} catch {
|
||||
throw DictationError.audioEngineStartFailed(error.localizedDescription)
|
||||
}
|
||||
|
||||
let request = SFSpeechAudioBufferRecognitionRequest()
|
||||
request.requiresOnDeviceRecognition = true // 红线:识别不出设备
|
||||
request.shouldReportPartialResults = true
|
||||
request.addsPunctuation = true
|
||||
self.request = request
|
||||
latestText = ""
|
||||
didFinish = false
|
||||
|
||||
let input = audioEngine.inputNode
|
||||
let format = input.outputFormat(forBus: 0)
|
||||
// tap 在音频线程跑:只碰局部捕获的 request,不碰 self
|
||||
input.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, _ in
|
||||
request.append(buffer)
|
||||
}
|
||||
audioEngine.prepare()
|
||||
do {
|
||||
try audioEngine.start()
|
||||
} catch {
|
||||
input.removeTap(onBus: 0)
|
||||
deactivateSession()
|
||||
throw DictationError.audioEngineStartFailed(error.localizedDescription)
|
||||
}
|
||||
|
||||
task = recognizer.recognitionTask(with: request) { [weak self] result, error in
|
||||
// 系统线程 → 主线程
|
||||
Task { @MainActor in
|
||||
guard let self else { return }
|
||||
if let result {
|
||||
self.latestText = result.bestTranscription.formattedString
|
||||
onPartial(self.latestText)
|
||||
if result.isFinal { self.didFinish = true }
|
||||
}
|
||||
if error != nil { self.didFinish = true }
|
||||
}
|
||||
}
|
||||
isRecording = true
|
||||
}
|
||||
|
||||
/// 停止录音,等待最终识别结果(最多 1.5s,超时用最新 partial),返回最终稿。
|
||||
/// 中途识别出错时已拿到的 partial 一样返回(spec 错误表:照常进整理流程)。
|
||||
func stop() async -> String {
|
||||
guard isRecording else { return "" }
|
||||
isRecording = false
|
||||
|
||||
audioEngine.stop()
|
||||
audioEngine.inputNode.removeTap(onBus: 0)
|
||||
request?.endAudio()
|
||||
|
||||
let deadline = Date().addingTimeInterval(1.5)
|
||||
while !didFinish && Date() < deadline {
|
||||
try? await Task.sleep(nanoseconds: 100_000_000)
|
||||
}
|
||||
task?.cancel()
|
||||
task = nil
|
||||
request = nil
|
||||
deactivateSession()
|
||||
return latestText
|
||||
}
|
||||
|
||||
/// 用户直接关 sheet 时的清理:不关心结果,立即停。
|
||||
func abort() {
|
||||
guard isRecording else { return }
|
||||
isRecording = false
|
||||
audioEngine.stop()
|
||||
audioEngine.inputNode.removeTap(onBus: 0)
|
||||
request?.endAudio()
|
||||
task?.cancel()
|
||||
task = nil
|
||||
request = nil
|
||||
deactivateSession()
|
||||
}
|
||||
|
||||
private func deactivateSession() {
|
||||
try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: 编译验证**
|
||||
|
||||
同 Task 3 Step 2 命令。Expected: `BUILD SUCCEEDED`。若出现 actor 隔离 warning(标注 error in Swift 6 language mode 的不阻塞),按提示把回调内对 self 的访问收进 `Task { @MainActor in }`,不许用 `nonisolated(unsafe)` 糊。
|
||||
|
||||
- [x] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康/Services/SpeechDictationService.swift
|
||||
git commit -m "feat(语音日记): SpeechDictationService 端侧流式转写(不落盘音频)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: DiaryVoicePanel(录音/整理面板视图)
|
||||
|
||||
**Files:**
|
||||
- Create: `康康/Features/Diary/DiaryVoicePanel.swift`
|
||||
|
||||
纯展示组件,状态全部外部传入,DiaryQuickSheet(已 600+ 行)不再膨胀。
|
||||
|
||||
- [x] **Step 1: 新建 DiaryVoicePanel.swift**
|
||||
|
||||
```swift
|
||||
import SwiftUI
|
||||
|
||||
/// 「健康记录」语音输入面板(spec 2026-06-10-voice-diary)。
|
||||
/// 两种模式:recording(实时字幕 + 计时 + 停止)/ organizing(AI 整理中,可取消)。
|
||||
/// 纯展示:状态由 DiaryQuickSheet 持有并传入。
|
||||
struct DiaryVoicePanel: View {
|
||||
enum Mode: Equatable {
|
||||
case recording(elapsedSeconds: Int)
|
||||
case organizing
|
||||
}
|
||||
|
||||
let mode: Mode
|
||||
/// recording 时为实时字幕;organizing 时为已定稿的转写稿(置灰展示)。
|
||||
let transcript: String
|
||||
let onStop: () -> Void
|
||||
let onCancelOrganize: () -> Void
|
||||
|
||||
/// 录音上限 3 分钟(超时由 DiaryQuickSheet 的看门狗触发 onStop)。
|
||||
static let maxRecordingSeconds = 180
|
||||
|
||||
var body: some View {
|
||||
VStack(alignment: .leading, spacing: 10) {
|
||||
header
|
||||
transcriptArea
|
||||
if case .recording = mode {
|
||||
stopButton
|
||||
}
|
||||
}
|
||||
.padding(12)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
.overlay(alignment: .bottom) {
|
||||
if mode == .organizing {
|
||||
AIFlowBar().padding(.horizontal, 1)
|
||||
}
|
||||
}
|
||||
.clipShape(RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous))
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private var header: some View {
|
||||
switch mode {
|
||||
case .recording(let elapsed):
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "waveform")
|
||||
.font(.tjScaled(12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
Text("正在听 · 识别在本机完成")
|
||||
.font(.tjScaled(13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
Text(Self.format(elapsed))
|
||||
.font(.tjScaled(12, design: .monospaced))
|
||||
.foregroundStyle(elapsed >= Self.maxRecordingSeconds - 30
|
||||
? Tj.Palette.brick : Tj.Palette.text3)
|
||||
}
|
||||
case .organizing:
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "sparkles")
|
||||
.font(.tjScaled(12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.symbolEffect(.pulse, options: .repeating)
|
||||
Text("AI 整理中 · 本地推理")
|
||||
.font(.tjScaled(13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
Button("取消") { onCancelOrganize() }
|
||||
.font(.tjScaled(12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private var transcriptArea: some View {
|
||||
ScrollViewReader { proxy in
|
||||
ScrollView(showsIndicators: false) {
|
||||
Text(transcript.isEmpty ? String(appLoc: "开始说话…") : transcript)
|
||||
.font(.tjScaled(14))
|
||||
.foregroundStyle(transcriptColor)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
Color.clear.frame(height: 1).id("tail")
|
||||
}
|
||||
.frame(maxHeight: 120)
|
||||
.onChange(of: transcript) { _, _ in
|
||||
proxy.scrollTo("tail", anchor: .bottom)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var transcriptColor: Color {
|
||||
if transcript.isEmpty { return Tj.Palette.text3 }
|
||||
return mode == .organizing ? Tj.Palette.text3 : Tj.Palette.text
|
||||
}
|
||||
|
||||
private var stopButton: some View {
|
||||
Button(action: onStop) {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "stop.circle.fill")
|
||||
Text("说完了,整理成日记")
|
||||
}
|
||||
.font(.tjScaled(14, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
.frame(maxWidth: .infinity)
|
||||
.padding(.vertical, 12)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.brick)
|
||||
)
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
private static func format(_ seconds: Int) -> String {
|
||||
String(format: "%d:%02d", seconds / 60, seconds % 60)
|
||||
}
|
||||
}
|
||||
|
||||
#Preview("录音中") {
|
||||
DiaryVoicePanel(mode: .recording(elapsedSeconds: 23),
|
||||
transcript: "今天早上起来有点头晕,量了血压一百四九十",
|
||||
onStop: {}, onCancelOrganize: {})
|
||||
.padding()
|
||||
}
|
||||
|
||||
#Preview("整理中") {
|
||||
DiaryVoicePanel(mode: .organizing,
|
||||
transcript: "今天早上起来有点头晕,量了血压一百四九十",
|
||||
onStop: {}, onCancelOrganize: {})
|
||||
.padding()
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: 编译验证**
|
||||
|
||||
同 Task 3 Step 2 命令。Expected: `BUILD SUCCEEDED`。
|
||||
|
||||
- [x] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康/Features/Diary/DiaryVoicePanel.swift
|
||||
git commit -m "feat(语音日记): DiaryVoicePanel 录音/整理面板"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: DiaryQuickSheet 接入(mic 按钮 + 状态机 + 回退 pill)
|
||||
|
||||
**Files:**
|
||||
- Modify: `康康/Features/Diary/DiaryQuickSheet.swift`
|
||||
|
||||
改 5 处:① 状态 + 录音流程函数;② 「内容」标签行加 mic 按钮;③ 输入框下方挂面板 / 提示条 / 回退 pill;④ `canRequestSuggest` 把 organizing 排除;⑤ onDisappear 清理。
|
||||
|
||||
- [x] **Step 1: 加语音状态(`@FocusState` 行之后、`hasContent` 之前)**
|
||||
|
||||
在 `DiaryQuickSheet.swift:38`(`@FocusState private var contentFocused: Bool`)之后插入:
|
||||
|
||||
```swift
|
||||
|
||||
// MARK: 语音输入状态(spec 2026-06-10-voice-diary)
|
||||
|
||||
enum VoicePhase: Equatable { case idle, recording, organizing }
|
||||
@State private var voicePhase: VoicePhase = .idle
|
||||
@State private var liveTranscript = ""
|
||||
@State private var recordingSeconds = 0
|
||||
/// 最近一次最终转写稿,「改用原话」回退用;再次录音时覆盖。
|
||||
@State private var rawTranscript: String?
|
||||
/// 刚追加进正文的整理稿,用于「改用原话」时在正文中定位替换。
|
||||
/// 用户手动编辑掉该段(正文中找不到了)时 pill 自然消失。
|
||||
@State private var organizedAppended: String?
|
||||
/// 一次性提示条文案(整理失败已填原话 / 没听清等),开始新录音时清掉。
|
||||
@State private var voiceNote: String?
|
||||
@State private var voiceDeniedAlert = false
|
||||
@State private var voiceFlowTask: Task<Void, Never>?
|
||||
@State private var recordingWatchdog: Task<Void, Never>?
|
||||
private let dictation = SpeechDictationService()
|
||||
```
|
||||
|
||||
- [x] **Step 2: 「内容」标签行加 mic 按钮**
|
||||
|
||||
把(`DiaryQuickSheet.swift:79-80` 附近):
|
||||
|
||||
```swift
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
sectionLabel(String(appLoc: "内容"))
|
||||
```
|
||||
|
||||
改为:
|
||||
|
||||
```swift
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
HStack {
|
||||
sectionLabel(String(appLoc: "内容"))
|
||||
Spacer()
|
||||
if SpeechDictationService.isAvailable, voicePhase == .idle {
|
||||
Button(action: startVoice) {
|
||||
HStack(spacing: 4) {
|
||||
Image(systemName: "mic.fill")
|
||||
.font(.tjScaled(11, weight: .semibold))
|
||||
Text("说一段")
|
||||
.font(.tjScaled(12, weight: .semibold))
|
||||
}
|
||||
.foregroundStyle(isLoading ? Tj.Palette.text3 : Tj.Palette.brick)
|
||||
.padding(.horizontal, 10)
|
||||
.padding(.vertical, 5)
|
||||
.background(Capsule().strokeBorder(
|
||||
isLoading ? Tj.Palette.line : Tj.Palette.brick.opacity(0.5),
|
||||
lineWidth: 1))
|
||||
.contentShape(Capsule())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(isLoading) // AI 追问生成中不抢 AIRuntime 队列
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`TextField` 那段不动,仍在该 VStack 内。)
|
||||
|
||||
- [x] **Step 3: 输入框下方挂面板 / 提示条 / 回退 pill**
|
||||
|
||||
在 TextField 的 `.overlay(...)` 闭合后、该 VStack 的收尾 `}` 之前(即原 `DiaryQuickSheet.swift:95` `)` 与 `:96` `}` 之间)插入:
|
||||
|
||||
```swift
|
||||
|
||||
if voicePhase != .idle {
|
||||
DiaryVoicePanel(
|
||||
mode: voicePhase == .organizing
|
||||
? .organizing
|
||||
: .recording(elapsedSeconds: recordingSeconds),
|
||||
transcript: liveTranscript,
|
||||
onStop: stopVoiceAndOrganize,
|
||||
onCancelOrganize: cancelOrganize
|
||||
)
|
||||
}
|
||||
|
||||
if let note = voiceNote {
|
||||
HStack(spacing: 6) {
|
||||
Image(systemName: "info.circle")
|
||||
.font(.tjScaled(11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Text(note)
|
||||
.font(.tjScaled(11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
}
|
||||
|
||||
if let organized = organizedAppended,
|
||||
rawTranscript != nil,
|
||||
content.range(of: organized) != nil {
|
||||
Button(action: revertToRawTranscript) {
|
||||
HStack(spacing: 4) {
|
||||
Image(systemName: "arrow.uturn.backward")
|
||||
.font(.tjScaled(10, weight: .semibold))
|
||||
Text("改用原话")
|
||||
.font(.tjScaled(11, weight: .semibold))
|
||||
}
|
||||
.foregroundStyle(Tj.Palette.ink)
|
||||
.padding(.horizontal, 10)
|
||||
.padding(.vertical, 5)
|
||||
.background(Capsule().strokeBorder(Tj.Palette.line, lineWidth: 1))
|
||||
.contentShape(Capsule())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: organizing 期间禁用「AI 追问」+ 关 sheet 清理 + 权限 alert**
|
||||
|
||||
把 `DiaryQuickSheet.swift:48`:
|
||||
|
||||
```swift
|
||||
private var canRequestSuggest: Bool { hasContent && !isLoading }
|
||||
```
|
||||
|
||||
改为:
|
||||
|
||||
```swift
|
||||
private var canRequestSuggest: Bool { hasContent && !isLoading && voicePhase == .idle }
|
||||
```
|
||||
|
||||
把 `DiaryQuickSheet.swift:146`:
|
||||
|
||||
```swift
|
||||
.onDisappear { suggestTask?.cancel() }
|
||||
```
|
||||
|
||||
改为:
|
||||
|
||||
```swift
|
||||
.onDisappear {
|
||||
suggestTask?.cancel()
|
||||
voiceFlowTask?.cancel()
|
||||
recordingWatchdog?.cancel()
|
||||
dictation.abort()
|
||||
}
|
||||
.alert(String(appLoc: "需要麦克风与语音识别权限"), isPresented: $voiceDeniedAlert) {
|
||||
Button(String(appLoc: "前往设置")) {
|
||||
if let url = URL(string: UIApplication.openSettingsURLString) {
|
||||
UIApplication.shared.open(url)
|
||||
}
|
||||
}
|
||||
Button(String(appLoc: "取消"), role: .cancel) {}
|
||||
} message: {
|
||||
Text("语音记录全程在本机完成,声音和文字都不会上传。请在设置中允许麦克风和语音识别。")
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 5: 加流程函数(`// MARK: - Actions` 区,`requestSuggestions` 之前)**
|
||||
|
||||
在 `DiaryQuickSheet.swift` 的 `sectionLabel` 函数后插入:
|
||||
|
||||
```swift
|
||||
|
||||
// MARK: 语音输入流程
|
||||
|
||||
private func startVoice() {
|
||||
contentFocused = false
|
||||
voiceNote = nil
|
||||
voiceFlowTask = Task { @MainActor in
|
||||
guard await dictation.requestAuthorization() else {
|
||||
voiceDeniedAlert = true
|
||||
return
|
||||
}
|
||||
do {
|
||||
liveTranscript = ""
|
||||
recordingSeconds = 0
|
||||
try dictation.start { partial in liveTranscript = partial }
|
||||
withAnimation(.snappy(duration: 0.2)) { voicePhase = .recording }
|
||||
// 计时 + 3 分钟看门狗(到点自动停,行为与点「停止」一致)
|
||||
recordingWatchdog = Task { @MainActor in
|
||||
while !Task.isCancelled {
|
||||
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
||||
guard !Task.isCancelled, voicePhase == .recording else { return }
|
||||
recordingSeconds += 1
|
||||
if recordingSeconds >= DiaryVoicePanel.maxRecordingSeconds {
|
||||
stopVoiceAndOrganize()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
voiceNote = error.localizedDescription
|
||||
voicePhase = .idle
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func stopVoiceAndOrganize() {
|
||||
guard voicePhase == .recording else { return }
|
||||
recordingWatchdog?.cancel()
|
||||
voiceFlowTask = Task { @MainActor in
|
||||
let transcript = (await dictation.stop())
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
liveTranscript = transcript
|
||||
guard !transcript.isEmpty else {
|
||||
withAnimation(.snappy(duration: 0.2)) { voicePhase = .idle }
|
||||
voiceNote = String(appLoc: "没听清,再试一次")
|
||||
return
|
||||
}
|
||||
rawTranscript = transcript
|
||||
withAnimation(.snappy(duration: 0.2)) { voicePhase = .organizing }
|
||||
do {
|
||||
let result = try await DiaryAssistService.shared.organize(transcript: transcript)
|
||||
guard !Task.isCancelled else { return }
|
||||
appendToContent(result.text)
|
||||
organizedAppended = result.text
|
||||
lastRate = result.decodeRate
|
||||
} catch is CancellationError {
|
||||
// cancelOrganize 已处理回退,这里只收尾
|
||||
} catch {
|
||||
guard !Task.isCancelled else { return }
|
||||
appendToContent(transcript) // 红线 #5:整理失败回退原话,不卡死
|
||||
organizedAppended = nil
|
||||
voiceNote = String(appLoc: "AI 整理失败,已填入原话")
|
||||
}
|
||||
withAnimation(.snappy(duration: 0.2)) { voicePhase = .idle }
|
||||
}
|
||||
}
|
||||
|
||||
/// 取消整理:中断 LLM,直接填原话(与失败回退同路径)。
|
||||
private func cancelOrganize() {
|
||||
guard voicePhase == .organizing else { return }
|
||||
voiceFlowTask?.cancel()
|
||||
if let raw = rawTranscript {
|
||||
appendToContent(raw)
|
||||
organizedAppended = nil
|
||||
voiceNote = String(appLoc: "已取消整理,填入原话")
|
||||
}
|
||||
withAnimation(.snappy(duration: 0.2)) { voicePhase = .idle }
|
||||
}
|
||||
|
||||
/// 「改用原话」:把刚追加的整理稿替换为原始转写稿(spec §2:LLM 改数兜底)。
|
||||
private func revertToRawTranscript() {
|
||||
guard let raw = rawTranscript,
|
||||
let organized = organizedAppended,
|
||||
let range = content.range(of: organized, options: .backwards) else { return }
|
||||
withAnimation(.snappy(duration: 0.18)) {
|
||||
content = content.replacingCharacters(in: range, with: raw)
|
||||
organizedAppended = nil
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 6: 编译验证(touch 强制重编拿全量警告)**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
touch 康康/Features/Diary/DiaryQuickSheet.swift
|
||||
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||
xcodebuild -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-configuration Debug build -derivedDataPath ./build/cli-dd \
|
||||
CODE_SIGNING_ALLOWED=NO 2>&1 | grep -E "\.swift:[0-9]+:[0-9]+: (error|warning):|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
|
||||
Expected: `BUILD SUCCEEDED`,无新增 warning。
|
||||
|
||||
- [x] **Step 7: 跑全量单测(确认没碰坏别的)**
|
||||
|
||||
```bash
|
||||
xcodebuild test -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-derivedDataPath ./build/cli-dd CODE_SIGNING_ALLOWED=NO 2>&1 | tail -5
|
||||
```
|
||||
|
||||
Expected: `** TEST SUCCEEDED **`。
|
||||
|
||||
- [x] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add 康康/Features/Diary/DiaryQuickSheet.swift
|
||||
git commit -m "feat(语音日记): DiaryQuickSheet 接入语音输入(录音→整理→回退原话)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: 验证与手测清单
|
||||
|
||||
**Files:** 无新增代码
|
||||
|
||||
- [x] **Step 1: 模拟器降级路径验证**
|
||||
|
||||
模拟器跑 App(或 Xcode Preview `DiaryQuickSheet`),打开「+ 新建 → 写日记」:
|
||||
- `SpeechDictationService.isAvailable` 在模拟器多半为 false → 「说一段」按钮应**整体不显示**,其余功能照旧。
|
||||
- 若模拟器恰好支持端侧识别(部分 macOS/Xcode 组合会),按钮出现也算通过——继续验证录音面板出现、无崩溃即可。
|
||||
|
||||
- [ ] **Step 2: 真机手测清单(连 iPhone 跑,逐项确认)**
|
||||
|
||||
1. 首次点「说一段」→ 依次弹语音识别 + 麦克风两个系统权限框,文案是 Task 1 写的端侧说明
|
||||
2. 拒绝权限 → 再点按钮弹「前往设置」alert,能跳系统设置
|
||||
3. 录音中:实时字幕逐字上屏、计时走动、说话时 waveform 动画
|
||||
4. 点「说完了,整理成日记」→ 面板转「AI 整理中」(AIFlowBar 流动)→ 整理稿**追加**进输入框(已有手打内容不被覆盖)
|
||||
5. 口述含数值(如"血压一百四九十")→ 整理稿数值未被改动(说 3 条不同口述各验一次)
|
||||
6. 「改用原话」pill 出现;点击 → 整理稿被替换为原始转写稿;再手动编辑正文该段 → pill 消失
|
||||
7. 飞行模式(模型已下载)→ 全流程照常,验证 100% 本地
|
||||
8. 一个字不说就点停止 → 「没听清,再试一次」,回 idle 不卡死
|
||||
9. 模型未下载(或长按删除模型后)→ 整理失败 → 原话直接入框 + 提示
|
||||
10. 录音中直接下滑关 sheet → 无崩溃,再次打开正常
|
||||
11. 「AI 整理中」点取消 → 原话入框 + 「已取消整理,填入原话」
|
||||
|
||||
- [ ] **Step 3: 把手测结果记进 commit(若有 fix,随 fix 一起提)**
|
||||
|
||||
```bash
|
||||
git commit --allow-empty -m "test(语音日记): 真机手测清单通过(见 plan Task 7)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review 记录
|
||||
|
||||
- **Spec 覆盖**:权限(T1)、organize prompt + 自适应 + 数值红线(T2)、Service(T3)、端侧转写不落盘 + 3 分钟上限 + zh 兜底(T4)、面板 + 实时字幕(T5)、mic 入口 + 状态机 + 追加不覆盖 + 改用原话 + 全部错误回退 + organizing 禁用追问(T6)、手测含飞行模式/空转写/取消(T7)。spec 各节均有对应任务。
|
||||
- **占位符**:无 TBD/TODO;所有代码步骤给了完整代码。
|
||||
- **类型一致性**:`SpeechDictationService.isAvailable/requestAuthorization/start(onPartial:)/stop()/abort()` 在 T4 定义、T6 使用一致;`DiaryVoicePanel.Mode`/`maxRecordingSeconds` T5 定义、T6 使用一致;`organize(transcript:) -> (text:, decodeRate:)` T3 定义、T6 解构一致;`AssistError` 复用现有定义。
|
||||
296
docs/superpowers/plans/2026-06-10-voice-export-composer.md
Normal file
296
docs/superpowers/plans/2026-06-10-voice-export-composer.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# 「身体档案」输入框语音听写 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 在「身体档案」(`HealthExportSheet`)底部聊天输入框加端侧语音听写:点 mic 开始、识别文字实时流进输入框、再点停止,不调 LLM、不自动发送。
|
||||
|
||||
**Architecture:** 复用 `SpeechDictationService`(@State 持有);新增 static 纯函数 `merge(prefix:partial:)` 处理"已有文字 + 听写文字"拼接(唯一可单测逻辑);`HealthExportSheet` 加 6 个 @State + mic 按钮 + 3 个流程函数。Spec:`docs/superpowers/specs/2026-06-10-voice-export-composer-design.md`。
|
||||
|
||||
**Tech Stack:** SwiftUI、Speech(经 SpeechDictationService)、Swift Testing。
|
||||
|
||||
**工程约定:** 同 `2026-06-10-voice-diary.md` 的「执行前必读」(同步组免改 pbxproj、CLI 用 `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` + `-derivedDataPath ./build/cli-dd`、全量并行测试不可信要 `-only-testing` 定点跑、commit 逐文件 add 不带 `Localizable.xcstrings`)。**当前环境注意**:xcode-select 已指向完整 Xcode 且许可证未接受——`git` 用 `DEVELOPER_DIR=/Library/Developer/CommandLineTools` 前缀绕过;`xcodebuild` 必须先让用户跑 `sudo xcodebuild -license accept`。直接在 `feat/mnn-sme2-runtime` 分支上做(上一功能合并后该分支即集成分支,不再另开分支避免并发会话分支错位)。
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `merge(prefix:partial:)`(TDD)
|
||||
|
||||
**Files:**
|
||||
- Test: `康康Tests/SpeechDictationMergeTests.swift`(新建)
|
||||
- Modify: `康康/Services/SpeechDictationService.swift`(`isAvailable` 之后加 static 方法)
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
新建 `康康Tests/SpeechDictationMergeTests.swift`:
|
||||
|
||||
```swift
|
||||
import Testing
|
||||
@testable import 康康
|
||||
|
||||
struct SpeechDictationMergeTests {
|
||||
@Test func emptyPrefixReturnsPartial() {
|
||||
#expect(SpeechDictationService.merge(prefix: "", partial: "今天头晕") == "今天头晕")
|
||||
}
|
||||
|
||||
@Test func plainPrefixJoinsWithSpace() {
|
||||
#expect(SpeechDictationService.merge(prefix: "已有内容", partial: "新听写")
|
||||
== "已有内容 新听写")
|
||||
}
|
||||
|
||||
@Test func whitespaceTerminatedPrefixConcatsDirectly() {
|
||||
#expect(SpeechDictationService.merge(prefix: "第一行\n", partial: "新听写")
|
||||
== "第一行\n新听写")
|
||||
}
|
||||
|
||||
@Test func emptyPartialKeepsPrefix() {
|
||||
#expect(SpeechDictationService.merge(prefix: "已有内容", partial: "") == "已有内容")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 跑测试确认编译失败**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||
xcodebuild test -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-only-testing:'康康Tests/SpeechDictationMergeTests' \
|
||||
-derivedDataPath ./build/cli-dd CODE_SIGNING_ALLOWED=NO 2>&1 | grep -E "error:|TEST (SUCCEEDED|FAILED)" | head -5
|
||||
```
|
||||
|
||||
Expected: `error: type 'SpeechDictationService' has no member 'merge'`(TEST FAILED)。
|
||||
|
||||
- [ ] **Step 3: 实现 merge**
|
||||
|
||||
在 `康康/Services/SpeechDictationService.swift` 的 `static var isAvailable` 行之后加:
|
||||
|
||||
```swift
|
||||
|
||||
/// 听写文本拼接:听写开始时输入框已有 prefix,partial 持续拼在其后。
|
||||
/// prefix 空 → 直接用 partial;prefix 以空白/换行结尾 → 直接连接;否则补一个空格。
|
||||
static func merge(prefix: String, partial: String) -> String {
|
||||
guard !partial.isEmpty else { return prefix }
|
||||
guard !prefix.isEmpty else { return partial }
|
||||
if let last = prefix.unicodeScalars.last,
|
||||
CharacterSet.whitespacesAndNewlines.contains(last) {
|
||||
return prefix + partial
|
||||
}
|
||||
return prefix + " " + partial
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 跑测试确认通过**
|
||||
|
||||
同 Step 2 命令。Expected: `** TEST SUCCEEDED **`,4 个用例通过。
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
DEVELOPER_DIR=/Library/Developer/CommandLineTools git add 康康Tests/SpeechDictationMergeTests.swift 康康/Services/SpeechDictationService.swift
|
||||
DEVELOPER_DIR=/Library/Developer/CommandLineTools git commit -m "feat(语音听写): SpeechDictationService.merge 前缀拼接(TDD)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: HealthExportSheet 接入
|
||||
|
||||
**Files:**
|
||||
- Modify: `康康/Features/Archive/HealthExportSheet.swift`(状态区 :27-30、canAsk :38、canGenerateReport :49、快捷问答 chip :133、onDisappear :103、alert :104、composer :410)
|
||||
|
||||
- [ ] **Step 1: 加听写状态(「快捷问答」状态块之后、`init` 之前)**
|
||||
|
||||
在 `@State private var newPromptText = ""` 之后插入:
|
||||
|
||||
```swift
|
||||
|
||||
// 语音听写(spec 2026-06-10-voice-export-composer)。
|
||||
// dictation 必须 @State:struct View 重建时普通 let 会换新实例(日记踩过的坑)。
|
||||
@State private var dictation = SpeechDictationService()
|
||||
@State private var isDictating = false
|
||||
/// 听写开始时输入框已有文字,partial 始终拼在它后面。
|
||||
@State private var dictationPrefix = ""
|
||||
@State private var dictationTask: Task<Void, Never>?
|
||||
@State private var dictationWatchdog: Task<Void, Never>?
|
||||
@State private var dictationDeniedAlert = false
|
||||
/// 录音上限,与日记一致(防麦克风悬挂)。
|
||||
private static let dictationMaxSeconds = 180
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 录音中禁发送/生成/chip**
|
||||
|
||||
`canAsk` 加条件:
|
||||
|
||||
```swift
|
||||
private var canAsk: Bool {
|
||||
!isAnswering &&
|
||||
!isGeneratingReport &&
|
||||
!isDictating &&
|
||||
!draftQuestion.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
|
||||
}
|
||||
```
|
||||
|
||||
`canGenerateReport` 的 `!isGeneratingReport &&` 后加 `!isDictating &&`。
|
||||
|
||||
快捷问答 chip 动作(`draftQuestion = p.prompt` 处)改为:
|
||||
|
||||
```swift
|
||||
guard !isDictating else { return }
|
||||
draftQuestion = p.prompt
|
||||
```
|
||||
|
||||
- [ ] **Step 3: composer 加 mic 按钮 + TextField 录音中禁用**
|
||||
|
||||
TextField 的 `.disabled(isAnswering || isGeneratingReport)` 改为 `.disabled(isAnswering || isGeneratingReport || isDictating)`。
|
||||
|
||||
TextField 与发送 Button 之间插入:
|
||||
|
||||
```swift
|
||||
|
||||
if SpeechDictationService.isAvailable {
|
||||
Button { toggleDictation() } label: {
|
||||
Image(systemName: isDictating ? "stop.fill" : "mic.fill")
|
||||
.font(.tjScaled(15, weight: .semibold))
|
||||
.foregroundStyle(isDictating ? Tj.Palette.paper : Tj.Palette.brick)
|
||||
.frame(width: 40, height: 40)
|
||||
.background(Circle().fill(isDictating ? Tj.Palette.brick : Tj.Palette.brickSoft))
|
||||
.symbolEffect(.pulse, options: .repeating, isActive: isDictating)
|
||||
}
|
||||
.disabled(isAnswering || isGeneratingReport)
|
||||
.accessibilityLabel(isDictating ? String(appLoc: "停止听写") : String(appLoc: "语音输入"))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 生命周期 + 权限 alert**
|
||||
|
||||
`.onDisappear { task?.cancel() }` 改为:
|
||||
|
||||
```swift
|
||||
.onDisappear {
|
||||
task?.cancel()
|
||||
dictationTask?.cancel()
|
||||
dictationWatchdog?.cancel()
|
||||
dictation.abort()
|
||||
}
|
||||
```
|
||||
|
||||
现有「添加快捷问答」alert 的 `}` 闭合之后追加:
|
||||
|
||||
```swift
|
||||
.alert(String(appLoc: "需要麦克风与语音识别权限"), isPresented: $dictationDeniedAlert) {
|
||||
Button(String(appLoc: "前往设置")) {
|
||||
if let url = URL(string: UIApplication.openSettingsURLString) {
|
||||
UIApplication.shared.open(url)
|
||||
}
|
||||
}
|
||||
Button(String(appLoc: "取消"), role: .cancel) {}
|
||||
} message: {
|
||||
Text("语音输入全程在本机完成,声音和文字都不会上传。请在设置中允许麦克风和语音识别。")
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: 流程函数(`// MARK: - Actions` 之后、`sendQuestion` 之前)**
|
||||
|
||||
```swift
|
||||
// MARK: 语音听写
|
||||
|
||||
private func toggleDictation() {
|
||||
if isDictating { stopDictation() } else { startDictation() }
|
||||
}
|
||||
|
||||
private func startDictation() {
|
||||
questionFocused = false
|
||||
dictationTask = Task { @MainActor in
|
||||
guard await dictation.requestAuthorization() else {
|
||||
dictationDeniedAlert = true
|
||||
return
|
||||
}
|
||||
do {
|
||||
dictationPrefix = draftQuestion
|
||||
try dictation.start { partial in
|
||||
draftQuestion = SpeechDictationService.merge(prefix: dictationPrefix,
|
||||
partial: partial)
|
||||
}
|
||||
withAnimation(.snappy(duration: 0.2)) { isDictating = true }
|
||||
dictationWatchdog = Task { @MainActor in
|
||||
try? await Task.sleep(nanoseconds: UInt64(Self.dictationMaxSeconds) * 1_000_000_000)
|
||||
guard !Task.isCancelled, isDictating else { return }
|
||||
stopDictation()
|
||||
}
|
||||
} catch {
|
||||
isDictating = false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func stopDictation() {
|
||||
guard isDictating else { return }
|
||||
dictationWatchdog?.cancel()
|
||||
dictationTask = Task { @MainActor in
|
||||
let final = (await dictation.stop()).trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if !final.isEmpty {
|
||||
draftQuestion = SpeechDictationService.merge(prefix: dictationPrefix,
|
||||
partial: final)
|
||||
}
|
||||
// final 为空:partial 已实时在输入框,保持现状即天然兜底(spec:不提示「没听清」)
|
||||
withAnimation(.snappy(duration: 0.2)) { isDictating = false }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: touch 强制重编验证**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
touch 康康/Features/Archive/HealthExportSheet.swift
|
||||
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||
xcodebuild -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-configuration Debug build -derivedDataPath ./build/cli-dd \
|
||||
CODE_SIGNING_ALLOWED=NO 2>&1 | grep -E "\.swift:[0-9]+:[0-9]+: (error|warning):|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
|
||||
Expected: `BUILD SUCCEEDED`,无新增 warning。
|
||||
|
||||
- [ ] **Step 7: 定点回归(语音相关全部测试)**
|
||||
|
||||
```bash
|
||||
xcodebuild test -project 康康.xcodeproj -scheme 康康 \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 17' \
|
||||
-only-testing:'康康Tests/SpeechDictationMergeTests' \
|
||||
-only-testing:'康康Tests/SpeechDictationAvailabilityTests' \
|
||||
-only-testing:'康康Tests/DiaryOrganizePromptTests' \
|
||||
-derivedDataPath ./build/cli-dd CODE_SIGNING_ALLOWED=NO 2>&1 | grep -E "Test case.*(passed|failed)|TEST (SUCCEEDED|FAILED)"
|
||||
```
|
||||
|
||||
Expected: `** TEST SUCCEEDED **`,7 用例通过。
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/xuhuayong/apps/康康
|
||||
DEVELOPER_DIR=/Library/Developer/CommandLineTools git add 康康/Features/Archive/HealthExportSheet.swift
|
||||
DEVELOPER_DIR=/Library/Developer/CommandLineTools git commit -m "feat(语音听写): 身体档案输入框听写实时上屏"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: 真机手测清单
|
||||
|
||||
- [ ] **Step 1: 真机逐项确认**
|
||||
|
||||
1. 「身体档案」composer 出现 mic 按钮(模拟器不支持端侧识别时隐藏)
|
||||
2. 点 mic → 说话 → 字实时出现在输入框;输入框已有文字时保留并以空格衔接
|
||||
3. 录音中:输入框/发送/「生成整理报告」/快捷问答 chip 均不可用;mic 为红色停止态
|
||||
4. 再点 mic → 停止,文字落定,点发送正常走问答
|
||||
5. 权限拒绝 → alert 跳设置
|
||||
6. 录音中直接关 sheet → 无崩溃、麦克风指示灯熄灭
|
||||
7. 3 分钟自动停止
|
||||
|
||||
---
|
||||
|
||||
## Self-Review 记录
|
||||
|
||||
- **Spec 覆盖**:merge 纯函数+单测(T1)、@State 持有/实时上屏/停止落定/空结果保持现状(T2 S5)、mic 隐藏与禁用矩阵(T2 S2-S3)、权限 alert + onDisappear abort + 看门狗(T2 S4-S5)、真机清单(T3)。无缺口。
|
||||
- **占位符**:无;所有代码步骤给全。
|
||||
- **类型一致性**:`merge(prefix:partial:)` T1 定义、T2 S5 调用一致;`dictationMaxSeconds`/`isDictating`/`dictationPrefix` 命名前后一致;`SpeechDictationService.isAvailable/requestAuthorization/start/stop/abort` 与现有实现签名一致。
|
||||
121
docs/superpowers/specs/2026-06-10-voice-diary-design.md
Normal file
121
docs/superpowers/specs/2026-06-10-voice-diary-design.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# 语音健康日记(语音转文字 + AI 整理)设计
|
||||
|
||||
> 2026-06-10 · 在「健康记录」(`DiaryQuickSheet`)里加语音输入:iOS 端侧语音识别实时转写,停止后由本地 LLM 整理成健康日记草稿,可编辑后保存。
|
||||
|
||||
## 背景
|
||||
|
||||
「健康记录」目前只能手打文字(`DiaryQuickSheet` → `DiaryEntry`),已有「AI 医生角度多轮追问」辅助。口述比打字门槛低得多,尤其适合身体不适时记录。
|
||||
|
||||
现有两个本地模型(Qwen3.5-2B 文本、Qwen3-VL 视觉)都没有音频编码器,无法做 ASR;引入 Whisper 类模型要 +0.5~1.5GB 体积和一条新推理链路,不可接受。`SFSpeechRecognizer` 支持强制端侧识别(`requiresOnDeviceRecognition = true`),中文质量够用、零体积,与「100% 本地」卖点完全一致。
|
||||
|
||||
## 决策(已与用户确认)
|
||||
|
||||
| 维度 | 决定 |
|
||||
|---|---|
|
||||
| 交互形态 | 说完 → 自动调 LLM 整理成日记草稿(非纯听写) |
|
||||
| 整理样式 | 自适应:口述短 → 一段通顺的话;口述长且多方面 → 自动分点 |
|
||||
| 入口 | `DiaryQuickSheet` 输入框旁麦克风按钮(不动 RecordSheet 骨架) |
|
||||
| 转写链路 | 流式实时转写(AVAudioEngine buffer → 实时字幕),不落盘音频 |
|
||||
| ASR 引擎 | `SFSpeechRecognizer` 端侧;不引入 Whisper;不做云端回退 |
|
||||
|
||||
## 架构
|
||||
|
||||
```
|
||||
DiaryQuickSheet(mic 按钮 + 录音面板)
|
||||
├─► SpeechDictationService(新)── AVAudioEngine + SFSpeechRecognizer(端侧)
|
||||
└─► DiaryAssistService.organize(transcript:)(新方法)──► AIRuntime ──► MNN/MLX
|
||||
```
|
||||
|
||||
符合模块边界:UI 不直接碰 AIRuntime;语音采集是系统能力,封装成独立 Service。
|
||||
|
||||
## 组件
|
||||
|
||||
### 1. `SpeechDictationService`(新,`Services/`,`@MainActor`)
|
||||
|
||||
封装 AVAudioEngine 麦克风采集 + `SFSpeechAudioBufferRecognitionRequest` 流式识别。
|
||||
|
||||
接口:
|
||||
|
||||
- `static var isAvailable: Bool` — 本机是否支持**端侧**中文识别(`supportsOnDeviceRecognition` + locale 检查;模拟器/老机型为 false)
|
||||
- `func requestAuthorization() async -> Bool` — 麦克风 + 语音识别两个权限一起申请
|
||||
- `func start(onPartial: @escaping (String) -> Void) throws` — 开始录音,partial 结果实时回调(录音面板字幕)
|
||||
- `func stop() async -> String` — 停止并返回最终转写稿
|
||||
|
||||
实现要点:
|
||||
|
||||
- `requiresOnDeviceRecognition = true`(硬性,识别内容不出设备)
|
||||
- `addsPunctuation = true`(自动标点)
|
||||
- locale 跟随系统,不支持端侧时 `isAvailable = false`
|
||||
- **不写任何音频文件**,buffer 即用即弃
|
||||
- 录音上限 3 分钟,到点自动 stop
|
||||
|
||||
### 2. `DiaryAssistService.organize(transcript:)`(新方法)
|
||||
|
||||
```swift
|
||||
func organize(transcript: String) async throws -> (text: String, decodeRate: Double)
|
||||
```
|
||||
|
||||
- prompt 加在 `AI/Prompts/DiaryAssistPrompts.swift`:`organizePrompt(transcript:)`
|
||||
- few-shot 两例:短口述 → 一段第一人称通顺文本;长口述(症状/用药/饮食多方面)→ 分点
|
||||
- **硬性约束写进 prompt:只重组语言,不得增删改任何数值、单位、药名、时间**(健康数据,2B 模型改数即事故)
|
||||
- 转写稿超长先截断(保护 context),非流式,await 完整结果
|
||||
- 走 AIRuntime actor 队列,与「多轮追问」「拍照识别」自然串行
|
||||
|
||||
### 3. `DiaryQuickSheet` UI 改动
|
||||
|
||||
- 内容输入框 trailing 加 mic 按钮(`isAvailable == false` 时整个隐藏)
|
||||
- 录音态:输入框下方展开录音面板 —— 实时字幕区 + 脉冲动画(sparkles/waveform `symbolEffect`)+「停止」按钮
|
||||
- 整理态:面板转「AI 整理中」(复用 `AIFlowBar` + tok/s),可取消
|
||||
- 完成:整理稿**追加**进输入框(沿用 `appendToContent`,不覆盖已写内容);面板收起
|
||||
- 完成后显示一次性「改用原话」pill:点击把刚追加的整理稿替换为原始转写稿(原始稿在本次 sheet 生命周期内持有;再次录音或手动编辑该段后 pill 消失)
|
||||
- 整理稿入框后,既有「AI 多轮追问」功能照常可用,无需特殊处理
|
||||
|
||||
## 状态机
|
||||
|
||||
```
|
||||
idle ──(点 mic,权限 OK)──► recording ──(停止/3min 到点)──► organizing ──► done(回 idle)
|
||||
```
|
||||
|
||||
- 实时字幕只显示在录音面板,**停止前不进输入框**
|
||||
- `organizing` 期间 mic 按钮与「AI 追问」按钮禁用(AIRuntime 串行,避免排队困惑)
|
||||
|
||||
## 错误处理(红线 #5:全部有回退,不卡死)
|
||||
|
||||
| 故障 | 行为 |
|
||||
|---|---|
|
||||
| 权限被拒 | 弹说明 alert + 「前往设置」跳系统设置 |
|
||||
| 本机不支持端侧识别(含模拟器) | mic 按钮隐藏,静默降级为纯手打 |
|
||||
| 识别中途出错 | 已拿到的 partial 文本照常进 organizing |
|
||||
| 转写结果为空 | 提示「没听清,再试一次」,回 idle |
|
||||
| LLM 未就绪 / 整理失败 | **原始转写稿直接追加进输入框** + 提示「AI 整理失败,已填入原话」 |
|
||||
|
||||
不做云端识别回退(红线 #1:不引入云服务)。
|
||||
|
||||
## 权限(project.pbxproj 新增两条 INFOPLIST_KEY)
|
||||
|
||||
- `NSMicrophoneUsageDescription`:康康需要使用麦克风进行语音记录,识别全程在本机完成,声音不会上传。
|
||||
- `NSSpeechRecognitionUsageDescription`:语音转文字使用 iOS 端侧识别,内容不会发送给 Apple 或任何服务器。
|
||||
|
||||
## 测试
|
||||
|
||||
- `organize` prompt:`DebugAIRunner` 加自检入口(短/长两条样例口述,肉眼验自适应样式 + 数值不被改动)
|
||||
- 录音链路:真机手测清单(权限首次申请、录音字幕、3 分钟自动停、整理失败回退、「改用原话」)
|
||||
- 模拟器:验证 `isAvailable == false` 时 mic 按钮隐藏
|
||||
|
||||
## 范围边界(不做)
|
||||
|
||||
- 症状 / AI 问答的语音入口
|
||||
- 音频文件保存或回放
|
||||
- Whisper / 任何新模型
|
||||
- Live Activity 集成(前台短流程,无必要)
|
||||
- 多语言听写优化(locale 跟系统,不支持即降级)
|
||||
|
||||
## 卖点映射(§12)
|
||||
|
||||
1. 降低记录门槛 → 卖点 1(影像档案之外的日常记录闭环)
|
||||
2. 「系统端侧 ASR + 本地 LLM 整理」全链路不出设备 → 卖点 2(100% 本地)
|
||||
3. 日记语料变多 → 卖点 3(本地 RAG 长期记忆)
|
||||
|
||||
## 排期
|
||||
|
||||
清单外新功能(红线 #6),本设计即立项讨论结论。工作量约 1~1.5 天,独立小分支插队,不挤占 C1/VL 主线。
|
||||
@@ -0,0 +1,47 @@
|
||||
# 「身体档案」输入框语音输入 设计
|
||||
|
||||
> 2026-06-10 · 在「身体档案」(`HealthExportSheet`)底部聊天输入框加端侧语音听写,复用 `SpeechDictationService`,识别文字实时流进输入框。
|
||||
|
||||
## 背景
|
||||
|
||||
「身体档案」composer 是聊天式输入(提问/诉求 → 发送 → LLM 对话/生成报告)。与日记不同,这里输入的内容马上交给 LLM,**不需要"整理"加工**;口述原话直接进输入框即正确行为(类似系统键盘听写)。
|
||||
|
||||
## 决策(已与用户确认)
|
||||
|
||||
| 维度 | 决定 |
|
||||
|---|---|
|
||||
| 交互 | 听写直接流进输入框:点 mic 开始,实时上屏;再点停止;用户自查后手动发送 |
|
||||
| LLM | 不调用(无整理步骤、不自动发送) |
|
||||
| 复用 | `SpeechDictationService`(**@State 持有**,防视图重建丢实例)、权限 alert 文案、3 分钟看门狗、onDisappear abort |
|
||||
| UI | mic 按钮放 TextField 与发送键之间;`isAvailable == false` 隐藏;录音中变红色停止态(脉冲动画) |
|
||||
|
||||
## 组件
|
||||
|
||||
### 1. `SpeechDictationService.merge(prefix:partial:)`(新,static 纯函数)
|
||||
|
||||
听写文本拼接规则,唯一可单测的逻辑:
|
||||
- `prefix` 为空 → 返回 `partial`
|
||||
- `prefix` 以空白/换行结尾 → `prefix + partial`
|
||||
- 其余 → `prefix + " " + partial`
|
||||
|
||||
### 2. `HealthExportSheet` 改动
|
||||
|
||||
- `@State dictation` + `isDictating` + `dictationPrefix` + 看门狗 Task
|
||||
- 点 mic:申请权限(拒绝 → alert 跳设置,与日记同文案)→ 记录 `dictationPrefix = draftQuestion` → start,每个 partial:`draftQuestion = merge(prefix:partial:)`
|
||||
- 再点:`stop()`,最终稿同 merge 落定;**stop 返回空时保留输入框现状**(partial 已实时在框里,天然兜底,不提示「没听清」)
|
||||
- 3 分钟看门狗自动停(防麦克风悬挂)
|
||||
|
||||
## 冲突防护
|
||||
|
||||
- 录音中:TextField 与发送按钮、「生成整理报告」按钮禁用(防手输与 partial 互相覆盖、防录音中发送)
|
||||
- `isAnswering / isGeneratingReport` 时 mic 禁用
|
||||
- `onDisappear` abort
|
||||
|
||||
## 测试
|
||||
|
||||
- `merge(prefix:partial:)` 3 个单测(空前缀 / 空白结尾前缀 / 普通前缀)
|
||||
- 真机手测:听写上屏、停止落定、已有文字保留、权限拒绝、3 分钟自动停
|
||||
|
||||
## 不做(YAGNI)
|
||||
|
||||
快捷问答弹窗 / 个人资料 Form 等其他输入处的语音;自动发送;录音面板;LLM 整理。
|
||||
116
docs/小红书创作笔记-GDG出海创想赛.md
Normal file
116
docs/小红书创作笔记-GDG出海创想赛.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# 小红书创作笔记 · GDG 出海创想赛
|
||||
|
||||
> 用途：小红书发布「创作笔记」，展示创作过程 + 最终效果，并在报名表提交作品链接。
|
||||
> 发布必带话题：**#GDG出海创想赛 #Google开发者大会**
|
||||
> 建议：先发笔记 → 复制笔记链接 → 填报名表。
|
||||
|
||||
---
|
||||
|
||||
## 一、正文（可直接复制发布）
|
||||
|
||||
### 📌 标题（三选一，都够炸）
|
||||
|
||||
1. 我用 Google 的 Gemma 做了个「看得懂体检报告」的 App，还全程不上云 🔒
|
||||
2. 6 周独立开发丨一个把化验单读成大白话的本地 AI 健康 App
|
||||
3. 拍一张体检报告，AI 帮我读懂 + 存档 + 追趋势（100% 本地）📱
|
||||
|
||||
### ✍️ 正文
|
||||
|
||||
家里体检报告、化验单越攒越多，看不懂又不敢乱传网上😮💨
|
||||
所以我做了一个 App —— **康康**，一个「只属于你自己」的健康影像档案本 📖
|
||||
|
||||
它最特别的地方：**AI 全程在手机里跑，报告不出你的手机。**
|
||||
|
||||
用的是 Google 的 **Gemma 4** 端侧模型（跑在 iPhone 本地），
|
||||
需要更强解读时，才可选接 **Gemini** 云端增强 —— 而且**默认是关的**，开不开你说了算。
|
||||
隐私优先，这是我做它的底线 ✅
|
||||
|
||||
—— 它能干这些事 👇
|
||||
|
||||
📷 **拍一张，自动读懂 + 归档**
|
||||
对着体检报告 / 化验单拍照，AI 自动把「血压 128、血糖 5.6、甘油三酯偏高…」这些指标结构化提取出来，异常项直接标红，一份份归进档案库。不用手动录入。
|
||||
|
||||
💬 **像聊天一样问自己的健康**
|
||||
「我最近血糖怎么样？」「上次体检哪些偏高？」直接问，它翻你自己的历史记录来答，还能点引用跳回原始报告。这是本地 RAG —— 你的健康长期记忆。
|
||||
|
||||
📈 **趋势一眼看 + AI 一句话解读**
|
||||
血压、血糖、体重这些长期指标自动连成折线图，AI 补一句大白话解读，不是干巴巴的数字。
|
||||
|
||||
🗂 **报告档案 + 对比上次**
|
||||
按体检 / 化验 / 影像 / 处方分类归档，还能和「上一份同类报告」自动对比：哪项升了、哪项降了、哪项从正常变偏高，一目了然。
|
||||
|
||||
🩺 **还有这些贴心的**
|
||||
· 问诊录音自动转文字，看病时不用手忙脚乱记
|
||||
· 症状 / 日记 / 用药提醒，随手记一笔
|
||||
· 一键导出健康摘要，看医生时直接给（只给文字，不给原图）
|
||||
|
||||
🔐 **隐私三件套**
|
||||
系统级硬件加密 + Face ID 启动锁 + 永久删除。你的健康数据，真·你自己的。
|
||||
|
||||
—— 创作过程我也放进图里了，从第一版画的信息架构，到 Gemma 在真机吐出第一个 token 的那一刻，再到现在的完整界面，6 周一点点长出来的 🌱
|
||||
|
||||
这是我给 **GDG 出海创想赛** 交的作品，做的时候一直在想一件事：
|
||||
**AI 能不能既聪明，又让人放心？** 康康就是我的答案。
|
||||
|
||||
有同样「不想把健康数据交出去」的朋友吗？评论区聊聊你会用吗 👇
|
||||
|
||||
#GDG出海创想赛 #Google开发者大会 #独立开发 #AI应用 #端侧AI #Gemma #Gemini #健康管理 #隐私保护 #iOS开发 #SwiftUI
|
||||
|
||||
---
|
||||
|
||||
## 二、配图建议（小红书是图片优先，建议 6–9 张）
|
||||
|
||||
小红书笔记权重很吃图，按「过程 → 效果」讲故事。推荐顺序：
|
||||
|
||||
| 顺序 | 内容 | 说明 |
|
||||
|---|---|---|
|
||||
| 封面 | App 首页真机截图 + 大字标题「不上云的 AI 健康档案」 | 封面决定点击率，字要大、对比强 |
|
||||
| 图 2 | 拍报告 → AI 自动提取指标的对比图（原图 vs 结构化结果） | 最有冲击力的「效果」页 |
|
||||
| 图 3 | 健康问答聊天界面（带引用回链的那种） | 展示本地 RAG |
|
||||
| 图 4 | 趋势折线图 + AI 解读那一句 | 长期价值 |
|
||||
| 图 5 | 报告详情「对比上次」区块 | 差异化亮点 |
|
||||
| 图 6 | 隐私设置页（Face ID / 云端 AI 默认关 开关） | 强化「隐私优先」记忆点 |
|
||||
| 图 7 | 创作过程：手绘/白板信息架构草图 | 「创作笔记」硬要求：展示过程 |
|
||||
| 图 8 | 创作过程：Gemma 在真机吐第一个 token 的截图 / tok/s 画面 | 技术真实感，扣 Google 主题 |
|
||||
| 图 9 | 6 周时间线 / 从草图到成品的拼图 | 收尾，强化「独立开发」故事 |
|
||||
|
||||
> 小技巧：截图统一加个手机边框、统一背景色，成套出现更高级。
|
||||
|
||||
---
|
||||
|
||||
## 三、核心卖点速记（写文案 / 答评论时对齐口径）
|
||||
|
||||
按对评委和用户的重要性排序：
|
||||
|
||||
1. **影像档案系统** —— 拍一张，AI 读懂 + 结构化归档（核心创意）
|
||||
2. **Google AI 全家桶** —— 端侧 Gemma 4 打底 + 可选 Gemini 云端增强（扣赛题）
|
||||
3. **本地 RAG 健康长期记忆** —— 问自己的历史数据，端侧不可替代
|
||||
4. **隐私三件套** —— 系统级加密 + Face ID + 永久删除（信任）
|
||||
5. **AI 趋势解读** —— 长期健康价值
|
||||
6. **对比上次** —— 报告 diff，即时可见
|
||||
|
||||
一句话定位：
|
||||
> **康康 —— 一个 100% 本地推理的个人健康影像档案，帮你把看不懂的体检报告读成大白话，还全程不出手机。**
|
||||
|
||||
---
|
||||
|
||||
## 四、明确「不做」的边界（被问到时的标准回答）
|
||||
|
||||
避免被误解成医疗产品，也是负责任 AI 的加分项：
|
||||
|
||||
- ❌ 不做医疗诊断、不给用药剂量 / 频次建议、不做急诊判断
|
||||
- ❌ 不上传数据、不做账号系统、不接任何云分析 SDK
|
||||
- ✅ 只做「记录 + 读懂 + 追踪」，把判断权留给你和你的医生
|
||||
|
||||
> 标准话术：「康康是健康**记录**工具，不是诊断工具。它帮你读懂和留存数据，但看病还是要找医生。」
|
||||
|
||||
---
|
||||
|
||||
## 五、发布 & 报名 Checklist
|
||||
|
||||
- [ ] 笔记正文含话题 **#GDG出海创想赛 #Google开发者大会**
|
||||
- [ ] 配图 ≥ 6 张，含「创作过程」页（报名硬要求）
|
||||
- [ ] 封面字大、卖点清晰
|
||||
- [ ] 发布后复制笔记链接
|
||||
- [ ] 打开报名表，提交作品链接 + 个人信息
|
||||
- [ ] （可选）正文提一句「Google 出海创想赛作品」，方便主办方检索
|
||||
210
docs/踩坑与排查记录.md
Normal file
210
docs/踩坑与排查记录.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# 康康 · 踩坑与排查记录
|
||||
|
||||
> 本地推理 / SwiftData / 端侧模型这类问题不好复现也不好搜,踩过的坑按统一模板记在这里,方便回查。
|
||||
> 新增条目往最上面加(倒序),模板见文末。
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-09 · 生成身体档案报告时,LLM 逐行复读死循环
|
||||
|
||||
### 现象
|
||||
多轮「身体档案」对话点生成报告后,「## 关键指标」整段陷入死循环:同一行
|
||||
`⚠️ 收缩压 (107 mmHg)` 连续重复几十遍,最后被 maxTokens 截断成半行「⚠️ 收缩」。
|
||||
(本质是小模型 **repetition / degeneration loop**,不是数据真有几十条。)
|
||||
|
||||
### 根因(确认)
|
||||
采样器**完全没有重复惩罚**,叠加低温 → 几乎必然复读。两个后端都有问题:
|
||||
|
||||
| 后端 | 位置 | 原配置 | 问题 |
|
||||
|---|---|---|---|
|
||||
| MNN(主) | `MNNLLMBridge.mm` `initWithConfigPath` 的 `set_config` | `temperature 0.3, topP 0.85` | 无 `penalty` |
|
||||
| MLX(兜底) | `LLMSession.swift` `GenerateParameters` | `temperature 0.3, topP 0.85` | 无 `repetitionPenalty` |
|
||||
|
||||
关键细节(读 MNN 源码 `transformers/llm/engine/src/`):
|
||||
- `llmconfig.hpp`:`mixed_samplers` 默认 `{topK, tfs, typical, topP, min_p, temperature}` —— **不含 `penalty`**;
|
||||
`penalty` / `ngram_factor` 默认 `1.0`(=全关)。
|
||||
- `sampler.cpp` `configMixed`:只会把 `penalty`「**移到链首(如果存在)**」,**不会自动插入**。
|
||||
所以光设 `"penalty":1.1` 没用,必须把 `"penalty"` 显式写进 `mixed_samplers`。
|
||||
- `sampler.cpp` `stepPenalty`:`repetition_penalty` 对 logits 乘法惩罚;**n-gram 命中整段重复时惩罚直接升到 `max_penalty`** —— 这正是掐断「整行复读」最有效的开关。
|
||||
|
||||
**为什么低温反而更糟**:temperature 0.3 接近贪心,一旦吐出 `收缩压 (107 mmHg)\n`,
|
||||
最高概率的后续就是再吐一遍同样的行,无惩罚就永远出不来。
|
||||
|
||||
### 排查过程(可复用思路)
|
||||
1. 看现象先判定是「数据重复」还是「生成复读」—— 被截断成半行 `收缩` 说明是 token 级复读,不是数据。
|
||||
2. `grep -niE "penalty|temperature|top_?p|sampler" 康康/AI/` 一把定位两个后端的采样配置 → 都没 penalty。
|
||||
3. 不猜 MNN 配置键,直接读构建用的源码 `MNN_SRC=/Users/xuhuayong/apps/MNN-src`
|
||||
的 `llmconfig.hpp` / `sampler.cpp`,确认键名、默认值、`mixed_samplers` 不自动插 penalty。
|
||||
4. MLX 侧读 SPM checkout 的 `MLXLMCommon/Evaluate.swift`,确认 `GenerateParameters` 有
|
||||
`repetitionPenalty: Float?` + `repetitionContextSize: Int`。
|
||||
|
||||
### 修复
|
||||
- **MNN** `MNNLLMBridge.mm`:`set_config` 显式开重复惩罚 +
|
||||
把 `penalty` 放进 mixed 链首:
|
||||
```jsonc
|
||||
{
|
||||
"jinja":{"context":{"enable_thinking":false}},
|
||||
"sampler_type":"mixed",
|
||||
"mixed_samplers":["penalty","topK","topP","temperature"],
|
||||
"temperature":0.3,"topP":0.85,"topK":40,
|
||||
"penalty":1.1,"n_gram":8,"ngram_factor":1.05
|
||||
}
|
||||
```
|
||||
(注意:JSON merge-patch 对数组是**整体替换**,所以这里会覆盖掉默认 `mixed_samplers`,符合预期。)
|
||||
- **MLX** `LLMSession.swift`:`GenerateParameters(..., repetitionPenalty: 1.1, repetitionContextSize: 64)`。
|
||||
|
||||
取值都偏保守:`penalty 1.1` / `ngram_factor 1.05` 是业界常用档(MNN 自带 omni 默认 1.05),
|
||||
低温 + 轻惩罚既能掐复读,又不破坏 JSON / 结构化输出的稳定性。
|
||||
|
||||
### 验证
|
||||
- `xcodebuild ... -destination generic/platform=iOS` 编译通过(两个后端均编进)。
|
||||
- ⚠️ **真机/模拟器跑一遍多轮导出生成报告**,确认不再复读 —— 复读属推理期行为,单测覆盖不到,必须实跑。
|
||||
|
||||
### 预防 / 相关注意
|
||||
- 任何新增的「长文本生成」(非 JSON 抽取)都走同一套带惩罚的采样参数,别再裸 temperature。
|
||||
- **相关隐患(未修,留观)**:`HealthExportService.retrieveDialogueSnapshot` 取指标时
|
||||
**没有 `prefix` 截断**(窗口检索版 `retrieve` 截了 `prefix(20)`)。指标极多时 prompt 会膨胀、
|
||||
也更易诱发复读。若复发,优先给 dialogue snapshot 也加上限。
|
||||
|
||||
---
|
||||
|
||||
> 以下几条据 W1–W2(2026-05~06)记忆补记,细节以代码/提交为准。
|
||||
|
||||
## 2026-06-09 · MNN 路径 Qwen3.5 强制思考,只吐 `<think>` / JSON 解析失败
|
||||
|
||||
### 现象
|
||||
MNN 真机路径上模型自检只显示 `<think>` 思考过程,AI 辅助拿不到 JSON(解析失败);
|
||||
同样的 prompt 走 MLX 兜底却正常。
|
||||
|
||||
### 根因
|
||||
模型自带 `config.json`(taobao-mnn 预转换件)写死 `"jinja":{"context":{"enable_thinking":true}}`,
|
||||
Qwen3.5 聊天模板据此每个 assistant 回合硬塞 `<think>\n` 开思考,吞掉 token 预算。
|
||||
**prompt 里的 `/no_think` 对 MNN 无效** —— 模板只读 `enable_thinking`,不看文本软开关。
|
||||
只在真机爆是因为 MLX 经 swift-transformers 套模板时不传 `enable_thinking` → 走 else 空 think 块,天然不思考。
|
||||
(这点从仓库代码看不出来,config.json 是下载/旁路导入的模型产物,不在 git 里。)
|
||||
|
||||
### 修复
|
||||
`MNNLLMBridge.mm` 在 `createLLM` 后、`load()` 前 merge-patch 关闭:
|
||||
`set_config("{\"jinja\":{\"context\":{\"enable_thinking\":false}}}")`。不改模型文件、不动字节校验。`stripThink` 保留兜底。
|
||||
|
||||
### 预防
|
||||
再遇 MNN 只出思考 / JSON 解析失败,先查 `config.json` 的 `enable_thinking`,别去调 `/no_think` 或加大预算。
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-07 · 「记录指标·拍照识别」VL 直读化验单不稳 → 改 Vision OCR + LLM
|
||||
|
||||
### 现象
|
||||
Qwen-VL 直读密集小字化验单经常返回 `{"indicators":[]}`(读不出指标)。
|
||||
|
||||
### 根因 / 决策
|
||||
小模型 VL 对密集中文小字不稳。改链路:`DocumentScanner 整页扫描 → Apple Vision OCR(zh-Hans/Hant/en)
|
||||
→ Qwen3 LLM 解析(VLPrompts.indicatorsFromText)→ stripThink → parseIndicatorsJSON → 确认页人工校对 → 存`。
|
||||
Vision OCR 是系统框架、100% 本地,不违反隐私红线。
|
||||
|
||||
### 预防
|
||||
这条路**不要改回 VL 直读**。VL 仍只用于「体检报告归档」整份解读,两者分开。OCR 行分组偶有错位,靠确认页人工校正兜底。
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-01 · git 全量 push 撞 HTTP 413(历史里有 165MB 构建产物)
|
||||
|
||||
### 现象
|
||||
`git push` 到 myv0(Gitea 反代有上传体积限制)报 **HTTP 413**。
|
||||
|
||||
### 根因
|
||||
旧 commit 误把 `build/` 构建产物提交进库(最大单文件 xcarchive DWARF **165MB**),后来虽 `git rm --cached` + `.gitignore`,
|
||||
但对象仍留在历史 → `.git` 87MB,全量 push 超反代上限。
|
||||
|
||||
### 修复
|
||||
对主仓库 `git filter-repo --path build/ --invert-paths --force` 从全历史剥离 → `.git` 87M→2.9M,不再 413。
|
||||
注意:① 重写了所有 commit hash(内容不变),旧克隆需重新 clone;② filter-repo 会移除所有 remote,事后须重新 `git remote add origin`;③ 凭证不写入 `.git/config`。
|
||||
|
||||
### 预防
|
||||
`build/` 必须在 `.gitignore`;别把构建产物 / 大二进制提交进库。
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-31 · 快拍 VL 识别时 App 自动退出(jetsam OOM,非崩溃)
|
||||
|
||||
### 现象
|
||||
iPhone 15 Pro Max 上 VL 识别时 App 直接退出。
|
||||
|
||||
### 根因
|
||||
不是代码崩溃(catch 只切 warning 屏,Swift 报错不会杀进程),是 **OS 内存超限 jetsam kill**。三因叠加:
|
||||
① 无 entitlement(8GB 设备默认单 App 上限 ~3GB,VL ~3GB 常驻冲过);② 从不卸载模型(LLM ~1GB + VL ~3GB 同驻 → 4GB+);③ 没设 MLX cache 上限。
|
||||
|
||||
### 修复
|
||||
① 新建 `康康.entitlements` 加 `com.apple.developer.kernel.increased-memory-limit=true`;
|
||||
② `AIRuntime` 加 `unloadLLM/unloadVL` 做**常驻互斥**(两大模型永不同时驻留)+ actor 内**串行推理闸门**(GPU 同一时刻只一个解码/加载);
|
||||
③ `GPU.set(cacheLimit: 256MB)`,启动调一次。
|
||||
|
||||
### 验证
|
||||
编译 + 单测通过。⚠️ **真机 OOM 是否真消失仍需 iPhone 15 Pro Max 实测**(本机无法跑真机)。
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-30 · 每次重打包 SwiftData 数据被清空
|
||||
|
||||
### 现象
|
||||
W2 期每次重新打包安装,本地数据全没了。
|
||||
|
||||
### 根因
|
||||
`KangkangApp.swift` 里 `ModelContainer` 创建失败的 catch 块原本**直接删 store 文件**。
|
||||
SwiftData 只对纯增量改动自动轻量迁移;一旦 schema 改动超纲(最常见:**给已存在 `@Model` 新增「非可选且无内联默认值」属性**)→ 迁移抛错 → 进 catch → 删库。
|
||||
|
||||
### 修复
|
||||
catch 改为把旧 store(含 `-wal`/`-shm`)挪到 `Application Support/StoreBackups/<时间戳>/` 再重建,不删除。
|
||||
|
||||
### 预防
|
||||
给已存在 `@Model` 加属性**一律给可选或内联默认值**(如 `var x: String = "daily"`),才走轻量迁移。正式发布前升级为 `VersionedSchema` + `SchemaMigrationPlan`。
|
||||
|
||||
---
|
||||
|
||||
## (无明确日期)· 编辑 Localizable.xcstrings 炸出上万行噪声 diff
|
||||
|
||||
### 现象
|
||||
改 `Localizable.xcstrings` 新增 3 个 key,却产生 ~15000 行 diff。
|
||||
|
||||
### 根因
|
||||
仓库里该文件是 **Xcode 规范格式**(`"key" : {` 冒号两侧带空格、2 空格缩进、key 按 Xcode 排序、结尾无换行);
|
||||
用 `python json.dump(indent=2)` 重写会把分隔符变成 `": "` 且顺序不同 → 几乎每行都 diff。
|
||||
|
||||
### 修复 / 正确做法
|
||||
基于 HEAD 原始文本做**文本插入**:把新 key 块按 Xcode 格式(` "<key>" : ` + `separators=(',', ' : ')` 的 value)拼到 strings 段末尾,保持结尾无换行。**不要整文件 json.dump 回写**。
|
||||
|
||||
---
|
||||
|
||||
## 附:命令行编译方式(排查时拿真实错误/警告)
|
||||
|
||||
- 系统默认是 Command Line Tools,裸 `xcodebuild` 不可用,需显式指向完整 Xcode:
|
||||
`export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer`
|
||||
- **必须用独立 derivedDataPath**(如 `-derivedDataPath /tmp/kk-derived-xxx`),否则和 Xcode 抢同一把 `build.db` 锁报 `database is locked`(不是代码错)。
|
||||
- 增量编译会吞警告:要看某文件警告先 `touch` 它强制重编,再 grep `error:|warning:|BUILD (SUCCEEDED|FAILED)`。
|
||||
- 工程是 Swift 5 + `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor`;跨到 `nonisolated` 调 MainActor 成员的隔离警告(标 "error in Swift 6 mode")在 Swift 5 下不阻塞构建。
|
||||
|
||||
---
|
||||
|
||||
## 模板(复制下面这段新增条目)
|
||||
|
||||
```markdown
|
||||
## YYYY-MM-DD · 一句话标题
|
||||
|
||||
### 现象
|
||||
（用户看到什么 / 怎么触发）
|
||||
|
||||
### 根因(确认)
|
||||
（定位到的真正原因,不是猜测;贴关键文件:行)
|
||||
|
||||
### 排查过程
|
||||
（怎么一步步定位的,方便下次复用思路)
|
||||
|
||||
### 修复
|
||||
（改了什么,贴 diff 要点或配置)
|
||||
|
||||
### 验证
|
||||
（怎么确认修好了;不能单测的要写明需实跑)
|
||||
|
||||
### 预防 / 相关注意
|
||||
（怎么避免再犯;顺带发现的隐患)
|
||||
```
|
||||
51
scripts/build-mnn-xcframework.sh
Normal file
51
scripts/build-mnn-xcframework.sh
Normal file
@@ -0,0 +1,51 @@
|
||||
#!/bin/sh
|
||||
# 构建 MNN.xcframework(device arm64 + simulator arm64),含 LLM 引擎 + SME2。
|
||||
# 产物输出到 康康/../Frameworks/MNN.xcframework(被 .gitignore,不入库,防历史膨胀)。
|
||||
#
|
||||
# 用法:
|
||||
# MNN_SRC=/path/to/MNN sh scripts/build-mnn-xcframework.sh
|
||||
# 需求:CMake 3.14+、Xcode、约 10-40 分钟。
|
||||
#
|
||||
# 关键 flag:
|
||||
# MNN_BUILD_LLM=ON —— 编入 llm 引擎(并导出 llm/llm.hpp),自动开 MNN_LOW_MEMORY
|
||||
# MNN_BUILD_LLM_OMNI=ON —— VL(图→文)所需:多模态 Omni + OpenCV 图像解码。
|
||||
# 统一模型(Qwen3.5-2B-MNN 一肩挑文本+视觉)必须开。
|
||||
# MNN_SME2=ON —— CMake 默认 ON,A19/iPhone17 运行时经 KleidiAI 自动启用,A17 回退 NEON
|
||||
# MNN_METAL=OFF —— 考核走 CPU+SME2,关 Metal 保持精简
|
||||
set -e
|
||||
|
||||
MNN_SRC="${MNN_SRC:-/Users/xuhuayong/apps/MNN-src}"
|
||||
OUT_DIR="$(cd "$(dirname "$0")/.." && pwd)/Frameworks"
|
||||
TOOLCHAIN_NEW="${MNN_SRC}/cmake/ios.toolchain.new.cmake"
|
||||
EXTRA="-DMNN_BUILD_LLM=ON -DMNN_BUILD_LLM_OMNI=ON -DMNN_METAL=OFF -DMNN_ARM82=true -DMNN_SME2=ON"
|
||||
COMMON="-DCMAKE_BUILD_TYPE=Release -DENABLE_BITCODE=0 -DMNN_AAPL_FMWK=1 -DMNN_SEP_BUILD=0 -DMNN_BUILD_SHARED_LIBS=false -DMNN_USE_THREAD_POOL=OFF"
|
||||
|
||||
export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
|
||||
cd "$MNN_SRC"
|
||||
|
||||
# 新版 ios-cmake toolchain(支持 SIMULATORARM64;MNN 自带的旧版只支持 x86_64 模拟器)
|
||||
if [ ! -f "$TOOLCHAIN_NEW" ]; then
|
||||
curl -sL "https://raw.githubusercontent.com/leetal/ios-cmake/master/ios.toolchain.cmake" -o "$TOOLCHAIN_NEW"
|
||||
fi
|
||||
|
||||
# device arm64
|
||||
rm -rf build-dev-arm64 && mkdir build-dev-arm64 && cd build-dev-arm64
|
||||
cmake .. $COMMON $EXTRA -DCMAKE_TOOLCHAIN_FILE="$TOOLCHAIN_NEW" -DPLATFORM=OS64 -DDEPLOYMENT_TARGET=17.0
|
||||
make MNN -j16
|
||||
cd ..
|
||||
|
||||
# simulator arm64
|
||||
rm -rf build-sim-arm64 && mkdir build-sim-arm64 && cd build-sim-arm64
|
||||
cmake .. $COMMON $EXTRA -DCMAKE_TOOLCHAIN_FILE="$TOOLCHAIN_NEW" -DPLATFORM=SIMULATORARM64 -DDEPLOYMENT_TARGET=17.0
|
||||
make MNN -j16
|
||||
cd ..
|
||||
|
||||
# 合成 xcframework
|
||||
rm -rf "$OUT_DIR/MNN.xcframework"
|
||||
mkdir -p "$OUT_DIR"
|
||||
xcrun xcodebuild -create-xcframework \
|
||||
-framework build-dev-arm64/MNN.framework \
|
||||
-framework build-sim-arm64/MNN.framework \
|
||||
-output "$OUT_DIR/MNN.xcframework"
|
||||
|
||||
echo "✅ 输出: $OUT_DIR/MNN.xcframework"
|
||||
52
scripts/build-sherpa-mnn-xcframework.sh
Executable file
52
scripts/build-sherpa-mnn-xcframework.sh
Executable file
@@ -0,0 +1,52 @@
|
||||
#!/bin/sh
|
||||
# 构建 sherpa-mnn.xcframework(端侧 SenseVoice ASR,跑在 MNN 后端),供「记录问诊」离线转写用。
|
||||
# 产物:Frameworks/sherpa-mnn.xcframework(device arm64 + simulator),被 .gitignore 不入库。
|
||||
#
|
||||
# 这是「全代码已就位、构建/真机验证由你来跑」中的构建步骤(详见 docs/release/sensevoice-integration.md)。
|
||||
# App 侧 SenseVoiceBridge.mm 用 __has_include(<sherpa-mnn/c-api/c-api.h>) 探测:
|
||||
# - 未接入(当前默认)→ 编为桩,问诊自动回退系统端侧识别(SFSpeech),不影响编译/运行;
|
||||
# - 接入本脚本产物 + 设好 HEADER_SEARCH_PATHS → 自动切到真实 SenseVoice。
|
||||
#
|
||||
# 用法:
|
||||
# MNN_SRC=/Users/xuhuayong/apps/MNN-src sh scripts/build-sherpa-mnn-xcframework.sh
|
||||
#
|
||||
# 需求:已 clone MNN 源码(含 apps/frameworks/sherpa-mnn)、CMake、Xcode。约 15-40 分钟。
|
||||
set -e
|
||||
|
||||
MNN_SRC="${MNN_SRC:-/Users/xuhuayong/apps/MNN-src}"
|
||||
SHERPA_DIR="${MNN_SRC}/apps/frameworks/sherpa-mnn"
|
||||
APP_FRAMEWORKS="$(cd "$(dirname "$0")/.." && pwd)/Frameworks"
|
||||
export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
|
||||
|
||||
[ -d "$SHERPA_DIR" ] || { echo "❌ 找不到 sherpa-mnn:$SHERPA_DIR(检查 MNN_SRC)"; exit 1; }
|
||||
|
||||
# ① 准备 MNN 静态库供 sherpa-mnn 链接。
|
||||
# sherpa-mnn 的 build-ios.sh 用 MNN_LIB_DIR 找 libMNN.a + 头文件。
|
||||
# 这里复用本项目的 MNN device/sim 构建产物;若尚未构建,先跑 scripts/build-mnn-xcframework.sh。
|
||||
# 注意:sherpa 的 build-ios.sh 默认还会编 simulator x86_64,Apple Silicon 上通常用不到——
|
||||
# 如只在 arm64 Mac/真机验证,可在 sherpa 的 build-ios.sh 里去掉 simulator_x86_64 段,加快构建。
|
||||
if [ -z "${MNN_LIB_DIR}" ]; then
|
||||
echo "ℹ️ 未显式指定 MNN_LIB_DIR,默认用 MNN_SRC 内的构建产物目录。"
|
||||
echo " 若 sherpa cmake 报找不到 MNN,请先构建 MNN(scripts/build-mnn-xcframework.sh)"
|
||||
echo " 并 export MNN_LIB_DIR=指向含 libMNN.a + include 的目录。"
|
||||
export MNN_LIB_DIR="${MNN_SRC}/project/ios/build"
|
||||
fi
|
||||
echo "MNN_LIB_DIR=${MNN_LIB_DIR}"
|
||||
|
||||
# ② 调 sherpa-mnn 自带的 iOS 构建(已含 libtool 合并 + create-xcframework + 拷头到 Headers/)。
|
||||
cd "$SHERPA_DIR"
|
||||
sh build-ios.sh
|
||||
|
||||
[ -d "$SHERPA_DIR/sherpa-mnn.xcframework" ] || { echo "❌ 构建未产出 sherpa-mnn.xcframework"; exit 1; }
|
||||
|
||||
# ③ 拷进 App 的 Frameworks/(供 Xcode 链接 + HEADER_SEARCH_PATHS 指向其 Headers/)。
|
||||
mkdir -p "$APP_FRAMEWORKS"
|
||||
rm -rf "$APP_FRAMEWORKS/sherpa-mnn.xcframework"
|
||||
cp -R "$SHERPA_DIR/sherpa-mnn.xcframework" "$APP_FRAMEWORKS/"
|
||||
|
||||
echo "✅ 输出: $APP_FRAMEWORKS/sherpa-mnn.xcframework"
|
||||
echo "下一步(Xcode,见 docs/release/sensevoice-integration.md):"
|
||||
echo " 1) 把 sherpa-mnn.xcframework 拖进 target 的 Frameworks, Libraries, and Embedded Content(Do Not Embed,静态库)"
|
||||
echo " 2) 在 Build Settings → HEADER_SEARCH_PATHS 追加(recursive):"
|
||||
echo " \$(PROJECT_DIR)/Frameworks/sherpa-mnn.xcframework/Headers"
|
||||
echo " 3) 转换并安装 SenseVoice 模型:sh scripts/convert-sensevoice-mnn.sh"
|
||||
65
scripts/convert-sensevoice-mnn.sh
Executable file
65
scripts/convert-sensevoice-mnn.sh
Executable file
@@ -0,0 +1,65 @@
|
||||
#!/bin/sh
|
||||
# 把官方 SenseVoice ONNX 模型转换成 MNN 格式,产出「记录问诊」端侧转写要用的两件套:
|
||||
# SenseVoice/model.mnn —— MNNConvert 量化转换(weightQuant 8bit)的图
|
||||
# SenseVoice/tokens.txt —— id↔token 映射(直接拷 onnx 包里的)
|
||||
#
|
||||
# 产物目录可:
|
||||
# - 旁路导入:拷到模拟器/真机沙盒 Application Support/Models/SenseVoice/(demo 现场重装兜底);
|
||||
# - 或上传到你的镜像,后续做成 App 内下载项(本期默认旁路导入,见集成文档)。
|
||||
#
|
||||
# 用法:
|
||||
# MNN_SRC=/Users/xuhuayong/apps/MNN-src sh scripts/convert-sensevoice-mnn.sh
|
||||
#
|
||||
# 需求:MNNConvert(从 MNN 源码 -DMNN_BUILD_CONVERTER=ON 编出)、wget/curl、tar。
|
||||
set -e
|
||||
|
||||
MNN_SRC="${MNN_SRC:-/Users/xuhuayong/apps/MNN-src}"
|
||||
OUT_DIR="$(cd "$(dirname "$0")/.." && pwd)/build/SenseVoice"
|
||||
WORK_DIR="$(cd "$(dirname "$0")/.." && pwd)/build/sensevoice-src"
|
||||
MODEL_TAR="sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17"
|
||||
MODEL_URL="https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/${MODEL_TAR}.tar.bz2"
|
||||
|
||||
# ① 找 MNNConvert。
|
||||
MNNCONVERT="${MNNCONVERT:-${MNN_SRC}/build/MNNConvert}"
|
||||
if [ ! -x "$MNNCONVERT" ]; then
|
||||
echo "❌ 找不到 MNNConvert:$MNNCONVERT"
|
||||
echo " 先编译转换器(在 MNN 源码里):"
|
||||
echo " cd \"$MNN_SRC\" && mkdir -p build && cd build \\"
|
||||
echo " && cmake .. -DMNN_BUILD_CONVERTER=ON -DMNN_LOW_MEMORY=ON -DMNN_SEP_BUILD=OFF && make MNNConvert -j8"
|
||||
echo " 或 export MNNCONVERT=/path/to/MNNConvert 后重跑。"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ② 下载 + 解包 SenseVoice ONNX(含 fp32 model.onnx / int8 / tokens.txt)。
|
||||
mkdir -p "$WORK_DIR"
|
||||
cd "$WORK_DIR"
|
||||
if [ ! -d "$MODEL_TAR" ]; then
|
||||
echo "⬇️ 下载 SenseVoice ONNX 模型…"
|
||||
if command -v wget >/dev/null 2>&1; then
|
||||
wget -c "$MODEL_URL"
|
||||
else
|
||||
curl -L -O "$MODEL_URL"
|
||||
fi
|
||||
tar xvf "${MODEL_TAR}.tar.bz2"
|
||||
fi
|
||||
|
||||
SRC_ONNX="${WORK_DIR}/${MODEL_TAR}/model.onnx"
|
||||
SRC_TOKENS="${WORK_DIR}/${MODEL_TAR}/tokens.txt"
|
||||
[ -f "$SRC_ONNX" ] || { echo "❌ 缺 model.onnx(应为 fp32,勿用 int8):$SRC_ONNX"; exit 1; }
|
||||
[ -f "$SRC_TOKENS" ] || { echo "❌ 缺 tokens.txt:$SRC_TOKENS"; exit 1; }
|
||||
|
||||
# ③ 转换:fp32 onnx → mnn(权重 8bit 量化,降体积+配合 MNN_LOW_MEMORY 降运行内存)。
|
||||
# 红线:转 fp32 的 model.onnx,不要转 model.int8.onnx(README 明确)。
|
||||
mkdir -p "$OUT_DIR"
|
||||
echo "🔧 转换 model.onnx → model.mnn(weightQuant 8bit)…"
|
||||
"$MNNCONVERT" -f ONNX \
|
||||
--modelFile "$SRC_ONNX" \
|
||||
--MNNModel "$OUT_DIR/model.mnn" \
|
||||
--weightQuantBits=8 --weightQuantBlock=64
|
||||
|
||||
cp "$SRC_TOKENS" "$OUT_DIR/tokens.txt"
|
||||
|
||||
echo "✅ 输出: $OUT_DIR/{model.mnn, tokens.txt}"
|
||||
echo "安装到沙盒(见 docs/release/sensevoice-integration.md):"
|
||||
echo " - 模拟器:拷到 ~/Library/Developer/CoreSimulator/.../Application Support/Models/SenseVoice/"
|
||||
echo " - 真机:经「我的 · 模型管理」旁路导入,或预拷进沙盒 Models/SenseVoice/"
|
||||
@@ -53,7 +53,8 @@ xcodebuild archive \
|
||||
-configuration "$CONFIGURATION" \
|
||||
-destination 'generic/platform=iOS' \
|
||||
-archivePath "$ARCHIVE_PATH" \
|
||||
-allowProvisioningUpdates
|
||||
-allowProvisioningUpdates \
|
||||
-skipMacroValidation
|
||||
|
||||
echo "==> [2/3] 生成 ExportOptions.plist..."
|
||||
cat > "$EXPORT_PLIST" <<EOF
|
||||
|
||||
@@ -10,6 +10,10 @@
|
||||
FEED000000000000DEAD0001 /* MLXLLM in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD0003 /* MLXLLM */; };
|
||||
FEED000000000000DEAD0002 /* MLXLMCommon in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD0004 /* MLXLMCommon */; };
|
||||
FEED000000000000DEAD0005 /* MLXVLM in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD0006 /* MLXVLM */; };
|
||||
FEED000000000000DEAD0007 /* MLXHuggingFace in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD0008 /* MLXHuggingFace */; };
|
||||
FEED000000000000DEAD0009 /* Tokenizers in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD000A /* Tokenizers */; };
|
||||
FEED000000000000DEAD000B /* HuggingFace in Frameworks */ = {isa = PBXBuildFile; productRef = FEED000000000000DEAD000C /* HuggingFace */; };
|
||||
FEEDFACE000000000000F002 /* MNN.xcframework in Frameworks */ = {isa = PBXBuildFile; fileRef = FEEDFACE000000000000F001 /* MNN.xcframework */; };
|
||||
/* End PBXBuildFile section */
|
||||
|
||||
/* Begin PBXContainerItemProxy section */
|
||||
@@ -33,6 +37,7 @@
|
||||
5E463CF92FC403BB0089145B /* 康康.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = "康康.app"; sourceTree = BUILT_PRODUCTS_DIR; };
|
||||
5E463D082FC403BC0089145B /* 康康Tests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = "康康Tests.xctest"; sourceTree = BUILT_PRODUCTS_DIR; };
|
||||
5E463D122FC403BC0089145B /* 康康UITests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = "康康UITests.xctest"; sourceTree = BUILT_PRODUCTS_DIR; };
|
||||
FEEDFACE000000000000F001 /* MNN.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; name = MNN.xcframework; path = Frameworks/MNN.xcframework; sourceTree = "<group>"; };
|
||||
/* End PBXFileReference section */
|
||||
|
||||
/* Begin PBXFileSystemSynchronizedRootGroup section */
|
||||
@@ -61,6 +66,10 @@
|
||||
FEED000000000000DEAD0001 /* MLXLLM in Frameworks */,
|
||||
FEED000000000000DEAD0002 /* MLXLMCommon in Frameworks */,
|
||||
FEED000000000000DEAD0005 /* MLXVLM in Frameworks */,
|
||||
FEED000000000000DEAD0007 /* MLXHuggingFace in Frameworks */,
|
||||
FEED000000000000DEAD0009 /* Tokenizers in Frameworks */,
|
||||
FEED000000000000DEAD000B /* HuggingFace in Frameworks */,
|
||||
FEEDFACE000000000000F002 /* MNN.xcframework in Frameworks */,
|
||||
);
|
||||
runOnlyForDeploymentPostprocessing = 0;
|
||||
};
|
||||
@@ -88,6 +97,7 @@
|
||||
5E463D0B2FC403BC0089145B /* 康康Tests */,
|
||||
5E463D152FC403BC0089145B /* 康康UITests */,
|
||||
5E463CFA2FC403BB0089145B /* Products */,
|
||||
FEEDFACE000000000000F001 /* MNN.xcframework */,
|
||||
);
|
||||
sourceTree = "<group>";
|
||||
};
|
||||
@@ -124,6 +134,9 @@
|
||||
FEED000000000000DEAD0003 /* MLXLLM */,
|
||||
FEED000000000000DEAD0004 /* MLXLMCommon */,
|
||||
FEED000000000000DEAD0006 /* MLXVLM */,
|
||||
FEED000000000000DEAD0008 /* MLXHuggingFace */,
|
||||
FEED000000000000DEAD000A /* Tokenizers */,
|
||||
FEED000000000000DEAD000C /* HuggingFace */,
|
||||
);
|
||||
productName = "康康";
|
||||
productReference = 5E463CF92FC403BB0089145B /* 康康.app */;
|
||||
@@ -183,7 +196,7 @@
|
||||
attributes = {
|
||||
BuildIndependentTargetsInParallel = 1;
|
||||
LastSwiftUpdateCheck = 2600;
|
||||
LastUpgradeCheck = 2600;
|
||||
LastUpgradeCheck = 2650;
|
||||
TargetAttributes = {
|
||||
5E463CF82FC403BB0089145B = {
|
||||
CreatedOnToolsVersion = 26.0.1;
|
||||
@@ -211,7 +224,9 @@
|
||||
mainGroup = 5E463CF02FC403BB0089145B;
|
||||
minimizedProjectReferenceProxies = 1;
|
||||
packageReferences = (
|
||||
5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-examples" */,
|
||||
5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */,
|
||||
5E9A1F882FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-transformers" */,
|
||||
5E9A1F892FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-huggingface" */,
|
||||
);
|
||||
preferredProjectObjectVersion = 77;
|
||||
productRefGroup = 5E463CFA2FC403BB0089145B /* Products */;
|
||||
@@ -292,6 +307,7 @@
|
||||
buildSettings = {
|
||||
ALWAYS_SEARCH_USER_PATHS = NO;
|
||||
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
|
||||
CLANG_ANALYZER_LOCALIZABILITY_NONLOCALIZED = YES;
|
||||
CLANG_ANALYZER_NONNULL = YES;
|
||||
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
|
||||
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
|
||||
@@ -321,6 +337,7 @@
|
||||
CLANG_WARN_UNREACHABLE_CODE = YES;
|
||||
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
|
||||
COPY_PHASE_STRIP = NO;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEBUG_INFORMATION_FORMAT = dwarf;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
ENABLE_STRICT_OBJC_MSGSEND = YES;
|
||||
@@ -344,6 +361,7 @@
|
||||
MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
|
||||
MTL_FAST_MATH = YES;
|
||||
ONLY_ACTIVE_ARCH = YES;
|
||||
STRING_CATALOG_GENERATE_SYMBOLS = YES;
|
||||
SWIFT_ACTIVE_COMPILATION_CONDITIONS = "DEBUG $(inherited)";
|
||||
SWIFT_OPTIMIZATION_LEVEL = "-Onone";
|
||||
};
|
||||
@@ -354,6 +372,7 @@
|
||||
buildSettings = {
|
||||
ALWAYS_SEARCH_USER_PATHS = NO;
|
||||
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
|
||||
CLANG_ANALYZER_LOCALIZABILITY_NONLOCALIZED = YES;
|
||||
CLANG_ANALYZER_NONNULL = YES;
|
||||
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
|
||||
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
|
||||
@@ -383,6 +402,7 @@
|
||||
CLANG_WARN_UNREACHABLE_CODE = YES;
|
||||
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
|
||||
COPY_PHASE_STRIP = NO;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
ENABLE_NS_ASSERTIONS = NO;
|
||||
@@ -399,6 +419,7 @@
|
||||
LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
|
||||
MTL_ENABLE_DEBUG_INFO = NO;
|
||||
MTL_FAST_MATH = YES;
|
||||
STRING_CATALOG_GENERATE_SYMBOLS = YES;
|
||||
SWIFT_COMPILATION_MODE = wholemodule;
|
||||
};
|
||||
name = Release;
|
||||
@@ -410,12 +431,14 @@
|
||||
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
|
||||
CODE_SIGN_ENTITLEMENTS = "康康/康康.entitlements";
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
ENABLE_APP_SANDBOX = YES;
|
||||
ENABLE_HARDENED_RUNTIME = YES;
|
||||
ENABLE_PREVIEWS = YES;
|
||||
ENABLE_USER_SELECTED_FILES = readonly;
|
||||
FRAMEWORK_SEARCH_PATHS = "$(PROJECT_DIR)/Frameworks";
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
INFOPLIST_KEY_CFBundleDisplayName = "康康";
|
||||
INFOPLIST_KEY_ITSAppUsesNonExemptEncryption = NO;
|
||||
@@ -423,8 +446,10 @@
|
||||
INFOPLIST_KEY_NSFaceIDUsageDescription = "用于解锁你的健康档案,数据始终保留在本机。";
|
||||
INFOPLIST_KEY_NSHealthShareUsageDescription = "康康会读取 Apple 健康中的生日、性别、身高和血型,用于本地填充个人资料,不会上传。";
|
||||
INFOPLIST_KEY_NSHealthUpdateUsageDescription = "康康不会写入 Apple 健康数据。此说明用于满足 HealthKit 权限校验,你的健康资料只保留在本机。";
|
||||
INFOPLIST_KEY_NSMicrophoneUsageDescription = "康康需要使用麦克风进行语音记录,识别全程在本机完成,声音不会上传。";
|
||||
INFOPLIST_KEY_NSPhotoLibraryAddUsageDescription = "康康会把识别后的报告原图加密保存到 App 沙盒,不会写入你的相册。";
|
||||
INFOPLIST_KEY_NSPhotoLibraryUsageDescription = "康康需要读取你已有的体检/化验报告照片用于本地识别,不会上传。";
|
||||
INFOPLIST_KEY_NSSpeechRecognitionUsageDescription = "语音转文字使用 iOS 端侧识别,内容不会发送给 Apple 或任何服务器。";
|
||||
"INFOPLIST_KEY_UIApplicationSceneManifest_Generation[sdk=iphoneos*]" = YES;
|
||||
"INFOPLIST_KEY_UIApplicationSceneManifest_Generation[sdk=iphonesimulator*]" = YES;
|
||||
"INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents[sdk=iphoneos*]" = YES;
|
||||
@@ -439,7 +464,7 @@
|
||||
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
|
||||
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
REGISTER_APP_GROUPS = YES;
|
||||
@@ -450,6 +475,7 @@
|
||||
SWIFT_APPROACHABLE_CONCURRENCY = YES;
|
||||
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
|
||||
SWIFT_EMIT_LOC_STRINGS = YES;
|
||||
SWIFT_OBJC_BRIDGING_HEADER = "康康/康康-Bridging-Header.h";
|
||||
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
|
||||
SWIFT_VERSION = 5.0;
|
||||
TARGETED_DEVICE_FAMILY = "1,2";
|
||||
@@ -464,12 +490,14 @@
|
||||
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
|
||||
CODE_SIGN_ENTITLEMENTS = "康康/康康.entitlements";
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
ENABLE_APP_SANDBOX = YES;
|
||||
ENABLE_HARDENED_RUNTIME = YES;
|
||||
ENABLE_PREVIEWS = YES;
|
||||
ENABLE_USER_SELECTED_FILES = readonly;
|
||||
FRAMEWORK_SEARCH_PATHS = "$(PROJECT_DIR)/Frameworks";
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
INFOPLIST_KEY_CFBundleDisplayName = "康康";
|
||||
INFOPLIST_KEY_ITSAppUsesNonExemptEncryption = NO;
|
||||
@@ -477,8 +505,10 @@
|
||||
INFOPLIST_KEY_NSFaceIDUsageDescription = "用于解锁你的健康档案,数据始终保留在本机。";
|
||||
INFOPLIST_KEY_NSHealthShareUsageDescription = "康康会读取 Apple 健康中的生日、性别、身高和血型,用于本地填充个人资料,不会上传。";
|
||||
INFOPLIST_KEY_NSHealthUpdateUsageDescription = "康康不会写入 Apple 健康数据。此说明用于满足 HealthKit 权限校验,你的健康资料只保留在本机。";
|
||||
INFOPLIST_KEY_NSMicrophoneUsageDescription = "康康需要使用麦克风进行语音记录,识别全程在本机完成,声音不会上传。";
|
||||
INFOPLIST_KEY_NSPhotoLibraryAddUsageDescription = "康康会把识别后的报告原图加密保存到 App 沙盒,不会写入你的相册。";
|
||||
INFOPLIST_KEY_NSPhotoLibraryUsageDescription = "康康需要读取你已有的体检/化验报告照片用于本地识别,不会上传。";
|
||||
INFOPLIST_KEY_NSSpeechRecognitionUsageDescription = "语音转文字使用 iOS 端侧识别,内容不会发送给 Apple 或任何服务器。";
|
||||
"INFOPLIST_KEY_UIApplicationSceneManifest_Generation[sdk=iphoneos*]" = YES;
|
||||
"INFOPLIST_KEY_UIApplicationSceneManifest_Generation[sdk=iphonesimulator*]" = YES;
|
||||
"INFOPLIST_KEY_UIApplicationSupportsIndirectInputEvents[sdk=iphoneos*]" = YES;
|
||||
@@ -493,7 +523,7 @@
|
||||
LD_RUNPATH_SEARCH_PATHS = "@executable_path/Frameworks";
|
||||
"LD_RUNPATH_SEARCH_PATHS[sdk=macosx*]" = "@executable_path/../Frameworks";
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
REGISTER_APP_GROUPS = YES;
|
||||
@@ -504,6 +534,7 @@
|
||||
SWIFT_APPROACHABLE_CONCURRENCY = YES;
|
||||
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
|
||||
SWIFT_EMIT_LOC_STRINGS = YES;
|
||||
SWIFT_OBJC_BRIDGING_HEADER = "康康/康康-Bridging-Header.h";
|
||||
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
|
||||
SWIFT_VERSION = 5.0;
|
||||
TARGETED_DEVICE_FAMILY = "1,2";
|
||||
@@ -516,12 +547,13 @@
|
||||
buildSettings = {
|
||||
BUNDLE_LOADER = "$(TEST_HOST)";
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
IPHONEOS_DEPLOYMENT_TARGET = 17.0;
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang.Tests;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
SDKROOT = auto;
|
||||
@@ -543,12 +575,13 @@
|
||||
buildSettings = {
|
||||
BUNDLE_LOADER = "$(TEST_HOST)";
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
IPHONEOS_DEPLOYMENT_TARGET = 17.0;
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang.Tests;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
SDKROOT = auto;
|
||||
@@ -569,12 +602,13 @@
|
||||
isa = XCBuildConfiguration;
|
||||
buildSettings = {
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
IPHONEOS_DEPLOYMENT_TARGET = 17.0;
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang.UITests;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
SDKROOT = auto;
|
||||
@@ -595,12 +629,13 @@
|
||||
isa = XCBuildConfiguration;
|
||||
buildSettings = {
|
||||
CODE_SIGN_STYLE = Automatic;
|
||||
CURRENT_PROJECT_VERSION = 5;
|
||||
CURRENT_PROJECT_VERSION = 6;
|
||||
DEAD_CODE_STRIPPING = YES;
|
||||
DEVELOPMENT_TEAM = F2C8C774FG;
|
||||
GENERATE_INFOPLIST_FILE = YES;
|
||||
IPHONEOS_DEPLOYMENT_TARGET = 17.0;
|
||||
MACOSX_DEPLOYMENT_TARGET = 26.0;
|
||||
MARKETING_VERSION = 1.0;
|
||||
MARKETING_VERSION = 1.1;
|
||||
PRODUCT_BUNDLE_IDENTIFIER = com.xuhuayong.kangkang.UITests;
|
||||
PRODUCT_NAME = "$(TARGET_NAME)";
|
||||
SDKROOT = auto;
|
||||
@@ -659,12 +694,28 @@
|
||||
/* End XCConfigurationList section */
|
||||
|
||||
/* Begin XCRemoteSwiftPackageReference section */
|
||||
5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-examples" */ = {
|
||||
5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */ = {
|
||||
isa = XCRemoteSwiftPackageReference;
|
||||
repositoryURL = "https://github.com/ml-explore/mlx-swift-examples";
|
||||
repositoryURL = "https://github.com/ml-explore/mlx-swift-lm";
|
||||
requirement = {
|
||||
kind = upToNextMajorVersion;
|
||||
minimumVersion = 2.29.1;
|
||||
kind = exactVersion;
|
||||
version = 3.31.4;
|
||||
};
|
||||
};
|
||||
5E9A1F882FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-transformers" */ = {
|
||||
isa = XCRemoteSwiftPackageReference;
|
||||
repositoryURL = "https://github.com/huggingface/swift-transformers";
|
||||
requirement = {
|
||||
kind = exactVersion;
|
||||
version = 1.2.1;
|
||||
};
|
||||
};
|
||||
5E9A1F892FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-huggingface" */ = {
|
||||
isa = XCRemoteSwiftPackageReference;
|
||||
repositoryURL = "https://github.com/huggingface/swift-huggingface.git";
|
||||
requirement = {
|
||||
kind = exactVersion;
|
||||
version = 0.9.0;
|
||||
};
|
||||
};
|
||||
/* End XCRemoteSwiftPackageReference section */
|
||||
@@ -672,19 +723,34 @@
|
||||
/* Begin XCSwiftPackageProductDependency section */
|
||||
FEED000000000000DEAD0003 /* MLXLLM */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-examples" */;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */;
|
||||
productName = MLXLLM;
|
||||
};
|
||||
FEED000000000000DEAD0004 /* MLXLMCommon */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-examples" */;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */;
|
||||
productName = MLXLMCommon;
|
||||
};
|
||||
FEED000000000000DEAD0006 /* MLXVLM */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-examples" */;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */;
|
||||
productName = MLXVLM;
|
||||
};
|
||||
FEED000000000000DEAD0008 /* MLXHuggingFace */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F872FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "mlx-swift-lm" */;
|
||||
productName = MLXHuggingFace;
|
||||
};
|
||||
FEED000000000000DEAD000A /* Tokenizers */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F882FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-transformers" */;
|
||||
productName = Tokenizers;
|
||||
};
|
||||
FEED000000000000DEAD000C /* HuggingFace */ = {
|
||||
isa = XCSwiftPackageProductDependency;
|
||||
package = 5E9A1F892FC43C9A0097DD29 /* XCRemoteSwiftPackageReference "swift-huggingface" */;
|
||||
productName = HuggingFace;
|
||||
};
|
||||
/* End XCSwiftPackageProductDependency section */
|
||||
};
|
||||
rootObject = 5E463CF12FC403BB0089145B /* Project object */;
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"originHash" : "6b8265ebd61c6fdfca835dd1f90f17439ca9abc5c11a8b7b5db8790be0349e4d",
|
||||
"originHash" : "170c2e0a1709c1ce83072661d2e376c3474847f0e091357712d91122c6658589",
|
||||
"pins" : [
|
||||
{
|
||||
"identity" : "gzipswift",
|
||||
"identity" : "eventsource",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/1024jp/GzipSwift",
|
||||
"location" : "https://github.com/mattt/EventSource.git",
|
||||
"state" : {
|
||||
"revision" : "731037f6cc2be2ec01562f6597c1d0aa3fe6fd05",
|
||||
"version" : "6.0.1"
|
||||
"revision" : "a3a85a85214caf642abaa96ae664e4c772a59f6e",
|
||||
"version" : "1.4.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -15,17 +15,35 @@
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/ml-explore/mlx-swift",
|
||||
"state" : {
|
||||
"revision" : "072b684acaae80b6a463abab3a103732f33774bf",
|
||||
"version" : "0.29.1"
|
||||
"revision" : "dc43e62d7055353c7f99fa071a4e71d29dfddc44",
|
||||
"version" : "0.31.4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "mlx-swift-examples",
|
||||
"identity" : "mlx-swift-lm",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/ml-explore/mlx-swift-examples",
|
||||
"location" : "https://github.com/ml-explore/mlx-swift-lm",
|
||||
"state" : {
|
||||
"revision" : "9bff95ca5f0b9e8c021acc4d71a2bbe4a7441631",
|
||||
"version" : "2.29.1"
|
||||
"revision" : "bd4b7434e6bdb588c7ef55706ff8904cb7fd4c57",
|
||||
"version" : "3.31.4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-asn1",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-asn1.git",
|
||||
"state" : {
|
||||
"revision" : "a9a5efd40eaf558a2bcd48d64b1d1646be686008",
|
||||
"version" : "1.7.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-atomics",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-atomics.git",
|
||||
"state" : {
|
||||
"revision" : "0442cb5a3f98ab802acb777929fdb446bda11a34",
|
||||
"version" : "1.3.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -33,8 +51,26 @@
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-collections.git",
|
||||
"state" : {
|
||||
"revision" : "fea17c02d767f46b23070fdfdacc28a03a39232a",
|
||||
"version" : "1.5.1"
|
||||
"revision" : "a0cb0954ecb21e4e31b0070e6ed5674e8556685a",
|
||||
"version" : "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-crypto",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-crypto.git",
|
||||
"state" : {
|
||||
"revision" : "1b6b2e274e85105bfa155183145a1dcfd63331f1",
|
||||
"version" : "4.5.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-huggingface",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/huggingface/swift-huggingface.git",
|
||||
"state" : {
|
||||
"revision" : "b721959445b617d0bf03910b2b4aced345fd93bf",
|
||||
"version" : "0.9.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -46,6 +82,15 @@
|
||||
"version" : "2.3.6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-nio",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-nio.git",
|
||||
"state" : {
|
||||
"revision" : "cd3e1152083706d77b223fb29110e590efcc70c0",
|
||||
"version" : "2.101.2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-numerics",
|
||||
"kind" : "remoteSourceControl",
|
||||
@@ -55,13 +100,40 @@
|
||||
"version" : "1.1.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-syntax",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/swiftlang/swift-syntax.git",
|
||||
"state" : {
|
||||
"revision" : "79e4b74a295b6eb74a8b585e3a39d29e70c1dbd1",
|
||||
"version" : "603.0.2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-system",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/apple/swift-system.git",
|
||||
"state" : {
|
||||
"revision" : "7502b711c92a17741fa625d722b0ccbd595d8ed1",
|
||||
"version" : "1.7.2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "swift-transformers",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/huggingface/swift-transformers",
|
||||
"state" : {
|
||||
"revision" : "a2e184dddb4757bc943e77fbe99ac6786c53f0b2",
|
||||
"version" : "1.0.0"
|
||||
"revision" : "58c4bc11963a140358d791f678a60a2745a23146",
|
||||
"version" : "1.2.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"identity" : "yyjson",
|
||||
"kind" : "remoteSourceControl",
|
||||
"location" : "https://github.com/ibireme/yyjson.git",
|
||||
"state" : {
|
||||
"revision" : "8b4a38dc994a110abaec8a400615567bd996105f",
|
||||
"version" : "0.12.0"
|
||||
}
|
||||
}
|
||||
],
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Scheme
|
||||
LastUpgradeVersion = "2600"
|
||||
LastUpgradeVersion = "2650"
|
||||
version = "1.7">
|
||||
<BuildAction
|
||||
parallelizeBuildables = "YES"
|
||||
|
||||
@@ -15,6 +15,13 @@ enum AIRuntimeError: Error, LocalizedError {
|
||||
}
|
||||
}
|
||||
|
||||
/// 推理优先级。interactive = 用户正在屏幕前等(识别/问答/自检);
|
||||
/// background = 预生成(报告摘要等),排队让行、解码中可被协作式抢占。
|
||||
nonisolated enum InferencePriority: Sendable, Equatable {
|
||||
case interactive
|
||||
case background
|
||||
}
|
||||
|
||||
actor AIRuntime {
|
||||
static let shared = AIRuntime()
|
||||
|
||||
@@ -27,11 +34,32 @@ actor AIRuntime {
|
||||
|
||||
private(set) var status: Status = .notReady
|
||||
private(set) var vlStatus: Status = .notReady
|
||||
private(set) var lastDecodeRate: Double = 0
|
||||
|
||||
/// 末次文本生成的性能统计(性能自检页消费;两后端归一)。
|
||||
private(set) var lastGenerateStats: GenerateStats?
|
||||
|
||||
/// 当前实际生效的后端标签(性能自检 / PPT 截图用)。
|
||||
/// 端侧统一走 MLX/GPU(Gemma 4 E2B);MNN/SME2 LLM 路径已移除。
|
||||
var activeBackendLabel: String {
|
||||
#if targetEnvironment(simulator)
|
||||
return "MLX · CPU(模拟器)"
|
||||
#else
|
||||
return "MLX · GPU"
|
||||
#endif
|
||||
}
|
||||
|
||||
private var llmSession: LLMSession?
|
||||
private var vlSession: VLSession?
|
||||
|
||||
// MARK: - Gemini 云端后端(hybrid:端侧 Gemma 默认,云端按需增强)
|
||||
// 云端调用不占本机显存,不进 OOM 闸门,可与端侧推理并发。用于「云端深度解读 / 多语言」
|
||||
// 与「拍报告/药盒多模态读图」——后者恢复端侧 Gemma 4(MLX 文本版)丢掉的真·视觉能力。
|
||||
private let gemini = GeminiBackend()
|
||||
/// 云端是否可用(用户已开启 + 有 key)。UI 与调用方据此决定走云还是端侧。
|
||||
nonisolated var cloudAvailable: Bool { CloudAI.isConfigured }
|
||||
/// 云端后端标签(性能自检 / 截图用)。
|
||||
nonisolated static var cloudLabel: String { "Gemini · \(CloudAI.model)" }
|
||||
|
||||
// MARK: - 串行推理闸门(§3.1 OOM 防护的真正落地)
|
||||
//
|
||||
// actor 只串行化「方法入口」,但 generate() 同步返回流、真正解码在内部 Task;
|
||||
@@ -42,30 +70,56 @@ actor AIRuntime {
|
||||
// 这里用 actor 内信号量(count = 1):所有「会占显存的重活」(解码 + 模型加载)
|
||||
// 进入前先 await acquireGate(),结束后 releaseGate()。actor 串行执行保证
|
||||
// gateBusy / gateWaiters 的读写天然无并发。
|
||||
private struct GateWaiter {
|
||||
let priority: InferencePriority
|
||||
let cont: CheckedContinuation<Void, Never>
|
||||
}
|
||||
private var gateBusy = false
|
||||
private var gateWaiters: [CheckedContinuation<Void, Never>] = []
|
||||
private var gateHolderPriority: InferencePriority = .interactive
|
||||
private var preemptRequested = false
|
||||
private var gateWaiters: [GateWaiter] = []
|
||||
|
||||
private func acquireGate() async {
|
||||
/// interactive 排到所有 background 等待者之前;同优先级保持 FIFO。纯函数,单测覆盖。
|
||||
nonisolated static func gateInsertionIndex(of priority: InferencePriority,
|
||||
in waiting: [InferencePriority]) -> Int {
|
||||
guard priority == .interactive else { return waiting.count }
|
||||
return waiting.firstIndex(of: .background) ?? waiting.count
|
||||
}
|
||||
|
||||
private func acquireGate(_ priority: InferencePriority = .interactive) async {
|
||||
if !gateBusy {
|
||||
gateBusy = true
|
||||
gateHolderPriority = priority
|
||||
return
|
||||
}
|
||||
// 前台请求撞上后台持有者:请其让位 —— 后台解码循环在下一个 token 抛 CancellationError。
|
||||
if priority == .interactive, gateHolderPriority == .background {
|
||||
preemptRequested = true
|
||||
}
|
||||
await withCheckedContinuation { (cont: CheckedContinuation<Void, Never>) in
|
||||
gateWaiters.append(cont)
|
||||
let idx = Self.gateInsertionIndex(of: priority, in: gateWaiters.map(\.priority))
|
||||
gateWaiters.insert(GateWaiter(priority: priority, cont: cont), at: idx)
|
||||
}
|
||||
// 被 releaseGate 唤醒时即已持有闸门(gateBusy 保持 true)。
|
||||
}
|
||||
|
||||
private func releaseGate() {
|
||||
preemptRequested = false
|
||||
if gateWaiters.isEmpty {
|
||||
gateBusy = false
|
||||
} else {
|
||||
// 把闸门直接交给队首等待者,gateBusy 维持 true,不留空窗。
|
||||
let next = gateWaiters.removeFirst()
|
||||
next.resume()
|
||||
gateHolderPriority = next.priority
|
||||
next.cont.resume()
|
||||
}
|
||||
}
|
||||
|
||||
/// 后台持有者每收到一个 token 查一次:前台在排队就让位。
|
||||
private func shouldPreempt(_ priority: InferencePriority) -> Bool {
|
||||
priority == .background && preemptRequested
|
||||
}
|
||||
|
||||
private init() {}
|
||||
|
||||
/// App 启动时调用一次:给 MLX 的 GPU 缓冲池设上限,避免 reuse cache 在大模型常驻之上
|
||||
@@ -74,11 +128,11 @@ actor AIRuntime {
|
||||
nonisolated static func configureMLXMemory() {
|
||||
#if !targetEnvironment(simulator)
|
||||
// 256MB cache 上限:够复用、不至于在 3GB 模型之上再囤几百 MB 空闲缓冲。
|
||||
MLX.GPU.set(cacheLimit: 256 * 1024 * 1024)
|
||||
MLX.Memory.cacheLimit = 256 * 1024 * 1024
|
||||
#endif
|
||||
}
|
||||
|
||||
/// 加载模型。首次调用会真正加载,后续幂等。
|
||||
/// 加载文本模型(端侧 MLX/GPU,Gemma 4 E2B)。首次调用会真正加载,后续幂等。
|
||||
func prepare() async throws {
|
||||
// 已有其他调用方在加载时,轮询等其结束再判定结果。
|
||||
// 不能像旧实现那样裸 return:那会让调用方误以为已 ready,随后 generate 的
|
||||
@@ -121,7 +175,10 @@ actor AIRuntime {
|
||||
|
||||
/// 流式生成。调用前应先 await prepare()。
|
||||
/// 注意:返回流是同步创建的,但跨 actor 调用 LLMSession 需要 await。
|
||||
func generate(prompt: String, maxTokens: Int = 256) -> AsyncThrowingStream<TokenChunk, Error> {
|
||||
/// priority = .background 时排队让行、解码中可被前台请求按 token 抢占(CancellationError 透传)。
|
||||
func generate(prompt: String,
|
||||
maxTokens: Int = 256,
|
||||
priority: InferencePriority = .interactive) -> AsyncThrowingStream<TokenChunk, Error> {
|
||||
// 在 actor 隔离上下文中捕获快照,Task 内不再访问 self.status / self.llmSession
|
||||
let snapshotStatus = status
|
||||
let snapshotSession = llmSession
|
||||
@@ -133,7 +190,9 @@ actor AIRuntime {
|
||||
return
|
||||
}
|
||||
// 进闸门:保证本次 LLM 解码与任何 VL 解码 / 模型加载串行,绝不并发占显存。
|
||||
await self.acquireGate()
|
||||
await self.acquireGate(priority)
|
||||
// defer 保证正常结束 / 异常 / 取消都释放闸门;杜绝未来新增 early-return 导致全局推理死锁。
|
||||
defer { self.releaseGate() }
|
||||
do {
|
||||
// session.generate 跨 actor 边界,需要 await
|
||||
let stream = await session.generate(prompt: prompt, maxTokens: maxTokens)
|
||||
@@ -141,26 +200,74 @@ actor AIRuntime {
|
||||
// 消费者(UI)提前关闭/取消时,下面的 checkCancellation 让本 Task 尽快退出,
|
||||
// 连带丢弃 session 流并触发其 onTermination,停止底层 MLX 解码,不空耗 GPU。
|
||||
try Task.checkCancellation()
|
||||
// Task 闭包在 generate() 内启动,继承 AIRuntime 的 actor 隔离;
|
||||
// 调用同 actor 的 recordRate 不需要 await
|
||||
self.recordRate(chunk.decodeRate)
|
||||
// 后台任务让位:前台请求在排队时,下一个 token 处主动退出。
|
||||
if self.shouldPreempt(priority) { throw CancellationError() }
|
||||
continuation.yield(chunk)
|
||||
}
|
||||
self.lastGenerateStats = await session.lastStats
|
||||
continuation.finish()
|
||||
} catch is CancellationError {
|
||||
// 取消/抢占以 CancellationError 透传,调用方据此区分「让位」与「真失败」。
|
||||
continuation.finish(throwing: CancellationError())
|
||||
} catch {
|
||||
continuation.finish(throwing: AIRuntimeError.inferenceFailed("\(error)"))
|
||||
}
|
||||
// 正常结束 / 异常 / 取消(checkCancellation 抛出后被上面 catch 吞掉)都会走到这,
|
||||
// 闸门一定释放,不会死锁后续推理。
|
||||
self.releaseGate()
|
||||
}
|
||||
// 消费者取消/流终止时取消内部 Task(与 LLMSession / HealthExportService 一致)。
|
||||
continuation.onTermination = { _ in task.cancel() }
|
||||
}
|
||||
}
|
||||
|
||||
private func recordRate(_ rate: Double) {
|
||||
if rate > 0 { lastDecodeRate = rate }
|
||||
// MARK: - Gemini 云端(hybrid 增强)
|
||||
|
||||
/// 云端流式生成(深度解读 / 多语言)。不进 OOM 闸门,可与端侧并发。
|
||||
/// 调用前应确认 `cloudAvailable`;失败时调用方回退端侧 `generate`。
|
||||
func generateCloud(prompt: String, maxTokens: Int = 512) -> AsyncThrowingStream<TokenChunk, Error> {
|
||||
AsyncThrowingStream { continuation in
|
||||
let task = Task {
|
||||
do {
|
||||
let stream = await self.gemini.generate(prompt: prompt, maxTokens: maxTokens)
|
||||
for try await chunk in stream {
|
||||
try Task.checkCancellation()
|
||||
continuation.yield(chunk)
|
||||
}
|
||||
self.lastGenerateStats = await self.gemini.lastStats
|
||||
continuation.finish()
|
||||
} catch is CancellationError {
|
||||
continuation.finish(throwing: CancellationError())
|
||||
} catch {
|
||||
continuation.finish(throwing: AIRuntimeError.inferenceFailed("\(error)"))
|
||||
}
|
||||
}
|
||||
continuation.onTermination = { _ in task.cancel() }
|
||||
}
|
||||
}
|
||||
|
||||
/// 云端多模态识别:图片直传 Gemini 读出结构化文本(通常 JSON)。
|
||||
/// 恢复端侧丢掉的真·视觉读图;失败时调用方回退端侧 OCR+文本(§3.2)。
|
||||
func analyzeReportCloud(imageURLs: [URL], prompt: String, maxTokens: Int = 1024) async throws -> String {
|
||||
do {
|
||||
return try await gemini.analyze(imageURLs: imageURLs, prompt: prompt, maxTokens: maxTokens)
|
||||
} catch {
|
||||
throw AIRuntimeError.inferenceFailed("\(error)")
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 端侧 ASR(SenseVoice via sherpa-mnn)互斥入口
|
||||
|
||||
/// 给端侧语音转写(SenseVoice,占 CPU + 内存)用:进推理闸门串行 + 卸掉常驻 LLM/VL 腾内存,
|
||||
/// 避免与文本/视觉模型同时常驻冲过单 App 内存上限被 jetsam 杀(§3.1 OOM 防护)。
|
||||
///
|
||||
/// 问诊流程是「先转写 → 再 organize(会重载 LLM)」严格串行,所以这里卸掉 LLM 是安全的:
|
||||
/// body 跑完一次转写返回后,调用方走 DiaryAssistService.organizeConsultation 时会自然重新 prepare。
|
||||
/// body 内部应自行 hop 到后台线程做阻塞解码(本 actor 在 await 期间不阻塞,但闸门保持持有,
|
||||
/// 其它推理请求会排队等待,杜绝并发占内存)。
|
||||
func runExclusiveForASR<T: Sendable>(_ body: @Sendable () async throws -> T) async rethrows -> T {
|
||||
await acquireGate(.interactive)
|
||||
defer { releaseGate() }
|
||||
unloadLLM()
|
||||
unloadVL()
|
||||
return try await body()
|
||||
}
|
||||
|
||||
// MARK: - VL
|
||||
@@ -172,14 +279,15 @@ actor AIRuntime {
|
||||
}
|
||||
if vlStatus == .ready { return }
|
||||
|
||||
// 同 prepare():用 isComplete 排除半下载(避免在残缺权重上崩溃),与下载服务判据一致。
|
||||
guard ModelStore.shared.isComplete(for: .vl) else {
|
||||
// MLX VL 改用 .llm 的 Qwen3.5-2B 多模态(VLMModelFactory 走 qwen3_5 视觉路径),
|
||||
// 不再单独需要 Qwen3-VL-4B。用 isComplete 排除半下载,与下载服务判据一致。
|
||||
guard ModelStore.shared.isComplete(for: .llm) else {
|
||||
vlStatus = .error("VL 模型未就绪")
|
||||
throw AIRuntimeError.notReady
|
||||
}
|
||||
|
||||
// 进闸门:等所有在跑的推理(可能是 LLM 文本流)结束,再卸 LLM + 载 VL。
|
||||
// —— 这正是「异常项快拍识别时 App 自动退出」的主因防护。
|
||||
// —— 这正是「指标速记识别时 App 自动退出」的主因防护。
|
||||
await acquireGate()
|
||||
defer { releaseGate() }
|
||||
if vlStatus == .ready { return }
|
||||
@@ -190,7 +298,7 @@ actor AIRuntime {
|
||||
vlStatus = .loading
|
||||
do {
|
||||
let session = try await VLSession.load(
|
||||
folderURL: ModelStore.shared.localURL(for: .vl)
|
||||
folderURL: ModelStore.shared.localURL(for: .llm)
|
||||
)
|
||||
self.vlSession = session
|
||||
vlStatus = .ready
|
||||
@@ -208,7 +316,7 @@ actor AIRuntime {
|
||||
guard llmSession != nil else { return }
|
||||
llmSession = nil
|
||||
status = .notReady
|
||||
MLX.GPU.clearCache()
|
||||
MLX.Memory.clearCache()
|
||||
}
|
||||
|
||||
/// 卸载 VL,释放 ModelContainer 引用并清 MLX 显存缓存。幂等。
|
||||
@@ -216,7 +324,7 @@ actor AIRuntime {
|
||||
guard vlSession != nil else { return }
|
||||
vlSession = nil
|
||||
vlStatus = .notReady
|
||||
MLX.GPU.clearCache()
|
||||
MLX.Memory.clearCache()
|
||||
}
|
||||
|
||||
/// 图像 → JSON 字符串(由 VLPrompts.reportExtraction 引导)。
|
||||
|
||||
@@ -28,6 +28,7 @@ final class FileDownloader: NSObject, URLSessionDataDelegate, @unchecked Sendabl
|
||||
private let lock = NSLock()
|
||||
private var handle: FileHandle?
|
||||
private var written: Int = 0
|
||||
private var expectedTotal: Int = 0
|
||||
private var onProgress: ((Int) -> Void)?
|
||||
private var responseError: Error?
|
||||
private var continuation: CheckedContinuation<Void, Error>?
|
||||
@@ -77,24 +78,33 @@ final class FileDownloader: NSObject, URLSessionDataDelegate, @unchecked Sendabl
|
||||
lock.withLock {
|
||||
self.handle = fileHandle
|
||||
self.written = offset
|
||||
self.expectedTotal = expectedBytes
|
||||
self.onProgress = onProgress
|
||||
self.responseError = nil
|
||||
}
|
||||
|
||||
// offset 0 也发 `bytes=0-`:诱导服务器回 206 + Content-Range 报告资源总大小,
|
||||
// didReceive 里与清单比对不符立即失败 —— 上游仓库更新导致大小漂移时(2026-07-14 实发),
|
||||
// 秒级报 sizeMismatch,而不是把 3.5GB 下完才在收尾校验发现。
|
||||
var request = URLRequest(url: url)
|
||||
if offset > 0 {
|
||||
request.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
|
||||
}
|
||||
request.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
|
||||
|
||||
let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
|
||||
defer { session.finishTasksAndInvalidate() }
|
||||
|
||||
// 句柄在 didCompleteWithError 内关闭(同一 delegate 队列,串行于 didReceive)。
|
||||
try await withCheckedThrowingContinuation { (cont: CheckedContinuation<Void, Error>) in
|
||||
lock.lock()
|
||||
self.continuation = cont
|
||||
lock.unlock()
|
||||
session.dataTask(with: request).resume()
|
||||
do {
|
||||
try await withCheckedThrowingContinuation { (cont: CheckedContinuation<Void, Error>) in
|
||||
lock.lock()
|
||||
self.continuation = cont
|
||||
lock.unlock()
|
||||
session.dataTask(with: request).resume()
|
||||
}
|
||||
} catch let error as DownloadError {
|
||||
// 响应头快速失败的大小不符:删 .part,防止换源续传时把两个版本的字节拼进同一文件。
|
||||
// badStatus(如临时 503)保留 .part,便于重试续传。
|
||||
if case .sizeMismatch = error { try? fm.removeItem(at: part) }
|
||||
throw error
|
||||
}
|
||||
|
||||
let finalSize = Self.fileSize(at: part)
|
||||
@@ -119,9 +129,27 @@ final class FileDownloader: NSObject, URLSessionDataDelegate, @unchecked Sendabl
|
||||
if let http = response as? HTTPURLResponse, http.statusCode >= 400 {
|
||||
lock.lock(); responseError = DownloadError.badStatus(http.statusCode); lock.unlock()
|
||||
completionHandler(.cancel)
|
||||
} else {
|
||||
completionHandler(.allow)
|
||||
return
|
||||
}
|
||||
// 快速失败:206 的 Content-Range(`bytes a-b/total`)报告了资源真实总大小,与清单不符
|
||||
// 说明服务器上的文件版本变了,立即取消(收尾校验必失败,没必要下完)。
|
||||
// 只信 206 —— 200 的 Content-Length 可能是压缩后长度(gzip),比对会误伤,仍走收尾校验。
|
||||
if let http = response as? HTTPURLResponse, http.statusCode == 206,
|
||||
let contentRange = http.value(forHTTPHeaderField: "Content-Range"),
|
||||
let totalPart = contentRange.split(separator: "/").last,
|
||||
let serverTotal = Int(totalPart) {
|
||||
lock.lock()
|
||||
let expected = expectedTotal
|
||||
lock.unlock()
|
||||
if serverTotal != expected {
|
||||
lock.lock()
|
||||
responseError = DownloadError.sizeMismatch(expected: expected, got: serverTotal)
|
||||
lock.unlock()
|
||||
completionHandler(.cancel)
|
||||
return
|
||||
}
|
||||
}
|
||||
completionHandler(.allow)
|
||||
}
|
||||
|
||||
nonisolated func urlSession(
|
||||
|
||||
215
康康/AI/GeminiBackend.swift
Normal file
215
康康/AI/GeminiBackend.swift
Normal file
@@ -0,0 +1,215 @@
|
||||
import Foundation
|
||||
|
||||
/// 云端 AI(Google Gemini)配置。**隐私优先:默认关闭**,用户在「我的 · 云端 AI」显式开启并填入
|
||||
/// AI Studio 的 API key 后才会启用。key 优先取 UserDefaults(用户填写),其次 Info.plist
|
||||
/// `GEMINI_API_KEY` / 环境变量(开发期注入),三处都没有则视为未配置。
|
||||
///
|
||||
/// 设计取舍:demo 阶段用 AI Studio 直发 API key 的 REST 方案,零新增 SPM 依赖、即时可编译;
|
||||
/// 生产级应升级到 Firebase AI Logic(App Check 防盗用、不在客户端裸存 key、内建 hybrid),
|
||||
/// 见 `docs/release` 的接入说明。
|
||||
nonisolated enum CloudAI {
|
||||
private static let enabledKey = "cloud_ai_gemini_enabled"
|
||||
private static let apiKeyKey = "cloud_ai_gemini_key"
|
||||
private static let modelKey = "cloud_ai_gemini_model"
|
||||
|
||||
/// 默认模型:快、便宜、原生多模态(文本+图像)。可在设置覆盖。
|
||||
static let defaultModel = "gemini-2.5-flash"
|
||||
|
||||
/// 用户是否开启云端增强(默认 false —— 不上云是默认态)。
|
||||
static var isEnabled: Bool {
|
||||
get { UserDefaults.standard.bool(forKey: enabledKey) }
|
||||
set { UserDefaults.standard.set(newValue, forKey: enabledKey) }
|
||||
}
|
||||
|
||||
/// Gemini API key。用户填写优先,其次构建期注入(Info.plist / 环境变量)。
|
||||
static var apiKey: String? {
|
||||
get {
|
||||
if let k = UserDefaults.standard.string(forKey: apiKeyKey),
|
||||
!k.trimmingCharacters(in: .whitespaces).isEmpty { return k }
|
||||
if let k = Bundle.main.object(forInfoDictionaryKey: "GEMINI_API_KEY") as? String,
|
||||
!k.isEmpty { return k }
|
||||
if let k = ProcessInfo.processInfo.environment["GEMINI_API_KEY"],
|
||||
!k.isEmpty { return k }
|
||||
return nil
|
||||
}
|
||||
set { UserDefaults.standard.set(newValue, forKey: apiKeyKey) }
|
||||
}
|
||||
|
||||
static var model: String {
|
||||
get { UserDefaults.standard.string(forKey: modelKey) ?? defaultModel }
|
||||
set { UserDefaults.standard.set(newValue, forKey: modelKey) }
|
||||
}
|
||||
|
||||
/// 云端是否可用:已开启 + 有 key。具体网络可达性由调用方在失败时回退端侧。
|
||||
static var isConfigured: Bool {
|
||||
isEnabled && apiKey != nil
|
||||
}
|
||||
}
|
||||
|
||||
enum GeminiError: Error, LocalizedError {
|
||||
case notConfigured
|
||||
case http(Int, String)
|
||||
case decode(String)
|
||||
|
||||
var errorDescription: String? {
|
||||
switch self {
|
||||
case .notConfigured: return String(appLoc: "云端 AI 未配置(请在「我的 · 云端 AI」开启并填入 key)")
|
||||
case .http(let c, let m): return String(appLoc: "Gemini 请求失败(\(c)):\(m)")
|
||||
case .decode(let m): return String(appLoc: "Gemini 响应解析失败:\(m)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Google Gemini 云端后端。经 REST(URLSession)调用 Generative Language API:
|
||||
/// - `generate`：`streamGenerateContent`(SSE 流式),用于「云端深度解读 / 多语言」。
|
||||
/// - `analyze`：`generateContent`(多模态),把报告/药盒图片直传 Gemini 读出结构化结果——
|
||||
/// 恢复端侧 Gemma 4(MLX 文本版)丢掉的真·视觉能力(§拍照→结构化)。
|
||||
///
|
||||
/// 云端调用不占本机显存,**不进 AIRuntime 的 OOM 闸门**,可与端侧推理并发。
|
||||
actor GeminiBackend {
|
||||
private let endpointBase = "https://generativelanguage.googleapis.com/v1beta/models"
|
||||
|
||||
private(set) var lastStats: GenerateStats?
|
||||
|
||||
// MARK: - 流式文本生成
|
||||
|
||||
/// 流式生成。返回流被取消时内部 Task 取消、连带断开底层连接。
|
||||
func generate(prompt: String, maxTokens: Int) -> AsyncThrowingStream<TokenChunk, Error> {
|
||||
AsyncThrowingStream { continuation in
|
||||
let task = Task {
|
||||
do {
|
||||
guard let key = CloudAI.apiKey else { throw GeminiError.notConfigured }
|
||||
let model = CloudAI.model
|
||||
var req = URLRequest(url: URL(string:
|
||||
"\(endpointBase)/\(model):streamGenerateContent?alt=sse&key=\(key)")!)
|
||||
req.httpMethod = "POST"
|
||||
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
|
||||
req.httpBody = try Self.requestBody(textParts: [prompt],
|
||||
imageParts: [],
|
||||
maxTokens: maxTokens)
|
||||
|
||||
let (bytes, response) = try await URLSession.shared.bytes(for: req)
|
||||
if let http = response as? HTTPURLResponse, http.statusCode != 200 {
|
||||
var body = ""
|
||||
for try await line in bytes.lines { body += line }
|
||||
throw GeminiError.http(http.statusCode, String(body.prefix(300)))
|
||||
}
|
||||
|
||||
let start = Date()
|
||||
var firstAt: Date?
|
||||
var produced = 0
|
||||
var usage: GeminiResponse.Usage?
|
||||
|
||||
for try await line in bytes.lines {
|
||||
if Task.isCancelled { break }
|
||||
guard line.hasPrefix("data:") else { continue }
|
||||
let payload = line.dropFirst(5).trimmingCharacters(in: .whitespaces)
|
||||
guard !payload.isEmpty, payload != "[DONE]",
|
||||
let data = payload.data(using: .utf8) else { continue }
|
||||
let chunk = try? JSONDecoder().decode(GeminiResponse.self, from: data)
|
||||
if let u = chunk?.usageMetadata { usage = u }
|
||||
let text = chunk?.candidates?.first?.content?.parts?
|
||||
.compactMap(\.text).joined() ?? ""
|
||||
guard !text.isEmpty else { continue }
|
||||
if firstAt == nil { firstAt = Date() }
|
||||
produced += 1
|
||||
let elapsed = Date().timeIntervalSince(firstAt ?? start)
|
||||
let rate = elapsed > 0 ? Double(produced) / elapsed : 0
|
||||
continuation.yield(TokenChunk(text: text, decodeRate: rate))
|
||||
}
|
||||
|
||||
// 归一统计:prefill = 首 chunk 前耗时;decode = 其后耗时;token 数取 usageMetadata。
|
||||
let ttf = (firstAt ?? Date()).timeIntervalSince(start)
|
||||
let total = Date().timeIntervalSince(start)
|
||||
self.lastStats = GenerateStats(
|
||||
promptTokens: usage?.promptTokenCount ?? 0,
|
||||
genTokens: usage?.candidatesTokenCount ?? produced,
|
||||
prefillSeconds: max(ttf, 0.0001),
|
||||
decodeSeconds: max(total - ttf, 0.0001)
|
||||
)
|
||||
continuation.finish()
|
||||
} catch is CancellationError {
|
||||
continuation.finish(throwing: CancellationError())
|
||||
} catch {
|
||||
continuation.finish(throwing: error)
|
||||
}
|
||||
}
|
||||
continuation.onTermination = { _ in task.cancel() }
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 多模态(图 → 文)
|
||||
|
||||
/// 多模态识别:图片 + prompt → 文本(通常是 JSON)。非流式,一次返回。
|
||||
/// 调用方负责解析 + 失败回退端侧(§3.2)。
|
||||
func analyze(imageURLs: [URL], prompt: String, maxTokens: Int) async throws -> String {
|
||||
guard let key = CloudAI.apiKey else { throw GeminiError.notConfigured }
|
||||
let model = CloudAI.model
|
||||
var req = URLRequest(url: URL(string:
|
||||
"\(endpointBase)/\(model):generateContent?key=\(key)")!)
|
||||
req.httpMethod = "POST"
|
||||
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
|
||||
req.httpBody = try Self.requestBody(textParts: [prompt],
|
||||
imageParts: imageURLs,
|
||||
maxTokens: maxTokens)
|
||||
|
||||
let (data, response) = try await URLSession.shared.data(for: req)
|
||||
if let http = response as? HTTPURLResponse, http.statusCode != 200 {
|
||||
let body = String(data: data, encoding: .utf8) ?? ""
|
||||
throw GeminiError.http(http.statusCode, String(body.prefix(300)))
|
||||
}
|
||||
let decoded = try JSONDecoder().decode(GeminiResponse.self, from: data)
|
||||
if let msg = decoded.error?.message { throw GeminiError.http(0, msg) }
|
||||
guard let text = decoded.candidates?.first?.content?.parts?
|
||||
.compactMap(\.text).joined(), !text.isEmpty else {
|
||||
throw GeminiError.decode(String(appLoc: "无文本返回"))
|
||||
}
|
||||
if let u = decoded.usageMetadata {
|
||||
self.lastStats = GenerateStats(promptTokens: u.promptTokenCount ?? 0,
|
||||
genTokens: u.candidatesTokenCount ?? 0,
|
||||
prefillSeconds: 0.0001, decodeSeconds: 0.0001)
|
||||
}
|
||||
return text
|
||||
}
|
||||
|
||||
// MARK: - 请求体
|
||||
|
||||
private static func requestBody(textParts: [String],
|
||||
imageParts: [URL],
|
||||
maxTokens: Int) throws -> Data {
|
||||
var parts: [[String: Any]] = textParts.map { ["text": $0] }
|
||||
for url in imageParts {
|
||||
guard let raw = try? Data(contentsOf: url) else { continue }
|
||||
// 控制单图体积,避免请求过大;Vault 原图已是 JPEG。
|
||||
let mime = url.pathExtension.lowercased() == "png" ? "image/png" : "image/jpeg"
|
||||
parts.append(["inline_data": ["mime_type": mime,
|
||||
"data": raw.base64EncodedString()]])
|
||||
}
|
||||
let body: [String: Any] = [
|
||||
"contents": [["role": "user", "parts": parts]],
|
||||
"generationConfig": [
|
||||
"maxOutputTokens": maxTokens,
|
||||
"temperature": 0.3,
|
||||
"topP": 0.85
|
||||
]
|
||||
]
|
||||
return try JSONSerialization.data(withJSONObject: body)
|
||||
}
|
||||
}
|
||||
|
||||
/// Gemini `GenerateContentResponse` 的最小可解码子集。
|
||||
/// nonisolated:工程默认 MainActor 隔离,不标会把 Decodable 合成实现推成 MainActor,
|
||||
/// 在 GeminiBackend 的后台解码上下文用不了(Swift 6 报错)。
|
||||
private nonisolated struct GeminiResponse: Decodable {
|
||||
nonisolated struct Candidate: Decodable { let content: Content? }
|
||||
nonisolated struct Content: Decodable { let parts: [Part]? }
|
||||
nonisolated struct Part: Decodable { let text: String? }
|
||||
nonisolated struct Usage: Decodable {
|
||||
let promptTokenCount: Int?
|
||||
let candidatesTokenCount: Int?
|
||||
}
|
||||
nonisolated struct APIError: Decodable { let message: String? }
|
||||
let candidates: [Candidate]?
|
||||
let usageMetadata: Usage?
|
||||
let error: APIError?
|
||||
}
|
||||
19
康康/AI/GenerateStats.swift
Normal file
19
康康/AI/GenerateStats.swift
Normal file
@@ -0,0 +1,19 @@
|
||||
import Foundation
|
||||
|
||||
/// 单次生成的性能统计,两后端(MNN / MLX)归一。
|
||||
/// MNN 取自 LlmContext(prefill_us / decode_us);MLX 取自 GenerateCompletionInfo。
|
||||
struct GenerateStats: Sendable, Equatable {
|
||||
var promptTokens: Int
|
||||
var genTokens: Int
|
||||
/// prefill(读入 prompt)耗时,秒。
|
||||
var prefillSeconds: Double
|
||||
/// decode(逐 token 生成)耗时,秒。
|
||||
var decodeSeconds: Double
|
||||
|
||||
var prefillTokensPerSecond: Double {
|
||||
prefillSeconds > 0 ? Double(promptTokens) / prefillSeconds : 0
|
||||
}
|
||||
var decodeTokensPerSecond: Double {
|
||||
decodeSeconds > 0 ? Double(genTokens) / decodeSeconds : 0
|
||||
}
|
||||
}
|
||||
@@ -2,12 +2,22 @@ import Foundation
|
||||
import MLX
|
||||
import MLXLLM
|
||||
import MLXLMCommon
|
||||
// mlx-swift-lm 3.x 把 HF Hub/分词器栈拆成 opt-in 宏(MLXHuggingFace),
|
||||
// `#hubDownloader()` / `#huggingFaceTokenizerLoader()` 展开处需要 HuggingFace / Tokenizers 在作用域。
|
||||
import MLXHuggingFace
|
||||
import HuggingFace
|
||||
import Tokenizers
|
||||
|
||||
/// 封装 MLX 语言模型的流式生成,actor 保证单线程访问。
|
||||
/// 基于 mlx-swift-examples 2.29.1(commit 9bff95ca)的 API。
|
||||
/// 基于 mlx-swift-lm 3.31.4 的 API(MLXLLM / MLXLMCommon)。
|
||||
actor LLMSession {
|
||||
let container: ModelContainer
|
||||
|
||||
/// 末次生成统计(取自流末尾的 .info 完成事件,性能自检用)。
|
||||
private(set) var lastStats: GenerateStats?
|
||||
|
||||
private func record(_ s: GenerateStats) { lastStats = s }
|
||||
|
||||
init(container: ModelContainer) {
|
||||
self.container = container
|
||||
}
|
||||
@@ -26,10 +36,20 @@ actor LLMSession {
|
||||
}
|
||||
|
||||
/// 从本地目录加载模型(包含 config.json + weights + tokenizer)。
|
||||
/// Gemma 4 改了回合标记:聊天回合以 `<turn|>`(token 106)结束(3n 时代是 `<end_of_turn>`),
|
||||
/// 而分词器自带的 eos 仍是 `<eos>`(1);不显式补 `<turn|>` 会让解码停不下来、把 maxTokens
|
||||
/// 跑满还吐回合分隔噪声。与上游 mlx-swift-lm 预置 `gemma4_e2b_it_4bit` 的 extraEOSTokens 一致。
|
||||
static func load(folderURL: URL) async throws -> LLMSession {
|
||||
let configuration = ModelConfiguration(directory: folderURL)
|
||||
let configuration = ModelConfiguration(
|
||||
directory: folderURL,
|
||||
extraEOSTokens: ["<turn|>"]
|
||||
)
|
||||
// 3.31.4:loadContainer 需显式传入 Downloader + TokenizerLoader。本地目录配置(.directory)
|
||||
// 不会真正下载(resolve 直接用目录),Downloader 仅占位;分词器由 HF AutoTokenizer 从目录加载。
|
||||
let container = try await withDeviceOverride {
|
||||
try await LLMModelFactory.shared.loadContainer(
|
||||
from: #hubDownloader(),
|
||||
using: #huggingFaceTokenizerLoader(),
|
||||
configuration: configuration
|
||||
)
|
||||
}
|
||||
@@ -45,10 +65,16 @@ actor LLMSession {
|
||||
let task = Task {
|
||||
do {
|
||||
try await Self.withDeviceOverride {
|
||||
// 低温:本 App 文本任务多为"直答/JSON 抽取",高温随机性会经常吐成非 JSON。
|
||||
// 0.3 + topP 0.85 让输出更确定、JSON 更稳(与 MNN set_config 降温对齐)。
|
||||
// repetitionPenalty:低温 + 无惩罚时,长文本(如「关键指标」列表)会逐行复读
|
||||
// 进入死循环;1.1 的重复惩罚 + 64 token 上下文窗口掐断复读(与 MNN penalty 对齐)。
|
||||
let parameters = GenerateParameters(
|
||||
maxTokens: maxTokens,
|
||||
temperature: Float(0.6),
|
||||
topP: Float(0.9)
|
||||
temperature: Float(0.3),
|
||||
topP: Float(0.85),
|
||||
repetitionPenalty: Float(1.1),
|
||||
repetitionContextSize: 64
|
||||
)
|
||||
|
||||
try await container.perform { (context: ModelContext) in
|
||||
@@ -72,9 +98,14 @@ actor LLMSession {
|
||||
let rate = elapsed > 0 ? Double(produced) / elapsed : 0
|
||||
continuation.yield(TokenChunk(text: text, decodeRate: rate))
|
||||
|
||||
case .info:
|
||||
case .info(let info):
|
||||
// 生成完成统计,是流的最后一个事件
|
||||
break
|
||||
await self.record(GenerateStats(
|
||||
promptTokens: info.promptTokenCount,
|
||||
genTokens: info.generationTokenCount,
|
||||
prefillSeconds: info.promptTime,
|
||||
decodeSeconds: info.generateTime
|
||||
))
|
||||
|
||||
case .toolCall:
|
||||
// 纯文本生成不会触发,switch 穷举
|
||||
|
||||
40
康康/AI/MNN/SenseVoiceBridge.h
Normal file
40
康康/AI/MNN/SenseVoiceBridge.h
Normal file
@@ -0,0 +1,40 @@
|
||||
//
|
||||
// SenseVoiceBridge.h
|
||||
// 康康
|
||||
//
|
||||
// Objective-C 封装:端侧 SenseVoice ASR(语音→文字),经 sherpa-mnn 在 MNN 后端跑。
|
||||
// 「记录问诊」用它把整段录音离线转写成文字,再交本地 LLM 整理成问诊小结。
|
||||
//
|
||||
// 与 MNNLLMBridge 同一套思路:真实实现在 .mm 里以 C 调用 <sherpa-mnn/c-api/c-api.h>;
|
||||
// 当工程尚未链接 sherpa-mnn(当前默认)时,以 __has_include 编为桩(isAvailable 返回 NO),
|
||||
// 上层 SenseVoiceASRService 自动回退到系统端侧识别(SFSpeech),App 不受影响。
|
||||
// 构建/接入 sherpa-mnn.xcframework 的步骤见 docs/release/sensevoice-integration.md。
|
||||
//
|
||||
|
||||
#import <Foundation/Foundation.h>
|
||||
|
||||
NS_ASSUME_NONNULL_BEGIN
|
||||
|
||||
@interface SenseVoiceBridge : NSObject
|
||||
|
||||
/// 本构建是否含真实 sherpa-mnn ASR(已链接 + 非桩=YES)。NO 时上层回退系统识别。
|
||||
+ (BOOL)isAvailable;
|
||||
|
||||
/// 用转换好的 SenseVoice MNN 模型 + tokens 创建离线识别器(贪心解码)。
|
||||
/// modelPath: MNNConvert 产出的 model.mnn;tokensPath: tokens.txt;
|
||||
/// language: "auto" / "zh" / "en" / "ja" / "ko" / "yue"。失败返回 nil。
|
||||
/// 识别器随实例常驻,dealloc 时释放——调用方按「一次问诊建一个」用,用完即释放降内存峰值。
|
||||
- (nullable instancetype)initWithModelPath:(NSString *)modelPath
|
||||
tokensPath:(NSString *)tokensPath
|
||||
language:(NSString *)language;
|
||||
|
||||
/// 转写一段单声道 PCM(float32,取值 [-1, 1])。sherpa 内部按 sampleRate 重采样到 16k 再抽 fbank。
|
||||
/// samples 在本调用期间需保持有效。返回识别文本(可能含 <|lang|> 等标签,由上层清洗);失败返回 nil。
|
||||
/// 同步阻塞直到解码结束——调用方务必放到后台线程,且经 AIRuntime 闸门与 LLM/VL 互斥(防 OOM)。
|
||||
- (nullable NSString *)transcribeSamples:(const float *)samples
|
||||
count:(int)count
|
||||
sampleRate:(int)sampleRate;
|
||||
|
||||
@end
|
||||
|
||||
NS_ASSUME_NONNULL_END
|
||||
112
康康/AI/MNN/SenseVoiceBridge.mm
Normal file
112
康康/AI/MNN/SenseVoiceBridge.mm
Normal file
@@ -0,0 +1,112 @@
|
||||
//
|
||||
// SenseVoiceBridge.mm
|
||||
// 康康
|
||||
//
|
||||
// ObjC++ 实现。链接 sherpa-mnn 时走真实 C-API;否则编为桩(返回不可用,上层回退系统识别)。
|
||||
//
|
||||
// 可用性闸门:仅当能找到 sherpa-mnn 的 C-API 头时才编真实路径。这样在尚未接入 sherpa-mnn
|
||||
// 的工程里本文件也能正常编过(走桩),接入后(把 sherpa-mnn.xcframework 加进 Frameworks/ 并
|
||||
// 让 HEADER_SEARCH_PATHS 找到其头)自动切到真实实现。见 docs/release/sensevoice-integration.md。
|
||||
//
|
||||
|
||||
#import "SenseVoiceBridge.h"
|
||||
|
||||
#if __has_include(<sherpa-mnn/c-api/c-api.h>)
|
||||
#define KK_SHERPA_MNN_AVAILABLE 1
|
||||
#else
|
||||
#define KK_SHERPA_MNN_AVAILABLE 0
|
||||
#endif
|
||||
|
||||
#if !KK_SHERPA_MNN_AVAILABLE
|
||||
|
||||
// ============ 桩:工程未链接 sherpa-mnn ============
|
||||
@implementation SenseVoiceBridge
|
||||
+ (BOOL)isAvailable { return NO; }
|
||||
- (nullable instancetype)initWithModelPath:(NSString *)modelPath
|
||||
tokensPath:(NSString *)tokensPath
|
||||
language:(NSString *)language { return nil; }
|
||||
- (nullable NSString *)transcribeSamples:(const float *)samples
|
||||
count:(int)count
|
||||
sampleRate:(int)sampleRate { return nil; }
|
||||
@end
|
||||
|
||||
#else
|
||||
|
||||
// ============ 真实:sherpa-mnn 离线 SenseVoice ============
|
||||
#include <sherpa-mnn/c-api/c-api.h>
|
||||
#include <string>
|
||||
|
||||
@implementation SenseVoiceBridge {
|
||||
const SherpaMnnOfflineRecognizer *_recognizer;
|
||||
}
|
||||
|
||||
+ (BOOL)isAvailable { return YES; }
|
||||
|
||||
- (nullable instancetype)initWithModelPath:(NSString *)modelPath
|
||||
tokensPath:(NSString *)tokensPath
|
||||
language:(NSString *)language {
|
||||
self = [super init];
|
||||
if (!self) return nil;
|
||||
|
||||
std::string model = modelPath.UTF8String;
|
||||
std::string tokens = tokensPath.UTF8String;
|
||||
std::string lang = language.length ? language.UTF8String : "auto";
|
||||
|
||||
SherpaMnnOfflineSenseVoiceModelConfig senseVoice;
|
||||
memset(&senseVoice, 0, sizeof(senseVoice));
|
||||
senseVoice.model = model.c_str();
|
||||
senseVoice.language = lang.c_str();
|
||||
senseVoice.use_itn = 1; // 逆文本规整:把「一百四十」之类还原成 140,数值更利于阅读/给医生看
|
||||
|
||||
SherpaMnnOfflineModelConfig modelConfig;
|
||||
memset(&modelConfig, 0, sizeof(modelConfig));
|
||||
modelConfig.tokens = tokens.c_str();
|
||||
modelConfig.num_threads = 2; // 端侧保守取 2,避免与 UI 抢核
|
||||
modelConfig.debug = 0;
|
||||
modelConfig.provider = "cpu"; // sherpa-mnn 后端即 MNN(CPU/SME2),provider 维持 "cpu"
|
||||
modelConfig.sense_voice = senseVoice;
|
||||
|
||||
SherpaMnnOfflineRecognizerConfig config;
|
||||
memset(&config, 0, sizeof(config));
|
||||
config.decoding_method = "greedy_search";
|
||||
config.model_config = modelConfig;
|
||||
|
||||
_recognizer = SherpaMnnCreateOfflineRecognizer(&config);
|
||||
if (_recognizer == nullptr) return nil;
|
||||
return self;
|
||||
}
|
||||
|
||||
- (void)dealloc {
|
||||
if (_recognizer) {
|
||||
SherpaMnnDestroyOfflineRecognizer(_recognizer);
|
||||
_recognizer = nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
- (nullable NSString *)transcribeSamples:(const float *)samples
|
||||
count:(int)count
|
||||
sampleRate:(int)sampleRate {
|
||||
if (_recognizer == nullptr || samples == nullptr || count <= 0) return nil;
|
||||
|
||||
const SherpaMnnOfflineStream *stream = SherpaMnnCreateOfflineStream(_recognizer);
|
||||
if (stream == nullptr) return nil;
|
||||
|
||||
SherpaMnnAcceptWaveformOffline(stream, sampleRate, samples, count);
|
||||
SherpaMnnDecodeOfflineStream(_recognizer, stream);
|
||||
|
||||
NSString *text = nil;
|
||||
const SherpaMnnOfflineRecognizerResult *result =
|
||||
SherpaMnnGetOfflineStreamResult(stream);
|
||||
if (result) {
|
||||
if (result->text) {
|
||||
text = [NSString stringWithUTF8String:result->text];
|
||||
}
|
||||
SherpaMnnDestroyOfflineRecognizerResult(result);
|
||||
}
|
||||
SherpaMnnDestroyOfflineStream(stream);
|
||||
return text;
|
||||
}
|
||||
|
||||
@end
|
||||
|
||||
#endif
|
||||
@@ -11,23 +11,54 @@ struct ModelFile: Equatable, Sendable {
|
||||
/// 字节数与服务器素材逐一核对一致,见
|
||||
/// docs/superpowers/specs/2026-05-29-model-download-design.md 附录 A。
|
||||
nonisolated enum ModelManifest {
|
||||
/// 自建 Caddy 静态服务(用户自建 HTTPS 反代)。
|
||||
/// 自建 Caddy 静态镜像。现仅 `.vl`(已废弃,不再下载)回退用;主模型 / MLX 兜底已切魔搭官方。
|
||||
/// 备选纯 IP(需 App 端 ATS 例外): http://101.132.124.52:5244/
|
||||
static let baseURL = URL(string: "https://file.myv0.com/")!
|
||||
|
||||
/// 各模型的魔搭(ModelScope)官方仓库 id。大陆可达;`resolve/master` 会 302 跳阿里 OSS,
|
||||
/// OSS 支持 Range,断点续传穿透重定向(实测 206 + 精确分片)。各文件大小与本清单逐一核对一致(2026-06 实测)。
|
||||
/// 注意组织名:MNN 模型在魔搭组织为 `MNN`(非 HuggingFace 的 taobao-mnn);MLX 沿用 mlx-community。
|
||||
static func modelScopeRepo(for kind: ModelKind) -> String? {
|
||||
switch kind {
|
||||
case .llm: return "mlx-community/gemma-4-e2b-it-4bit" // 主模型,大陆可达
|
||||
case .mnnLLM: return "MNN/Qwen3.5-2B-MNN" // 已停用,保留源备查
|
||||
case .vl: return nil // 已废弃,不再下载 / 分发,不提供魔搭源
|
||||
}
|
||||
}
|
||||
|
||||
/// hf-mirror(hf-mirror.com)回退源仓库 id。仅 mlx-community 系与 HF 同 org/name 可直接镜像;
|
||||
/// MNN 源在 HuggingFace 组织名不同(taobao-mnn),且已停用,不配回退。revision 用 HF 的 `main`。
|
||||
static func huggingFaceMirrorRepo(for kind: ModelKind) -> String? {
|
||||
switch kind {
|
||||
case .llm: return kind.huggingFaceRepo // mlx-community/gemma-4-e2b-it-4bit
|
||||
case .vl, .mnnLLM: return nil
|
||||
}
|
||||
}
|
||||
|
||||
static func files(for kind: ModelKind) -> [ModelFile] {
|
||||
switch kind {
|
||||
case .llm:
|
||||
// gemma-4-e2b-it-4bit:Gemma 4 E2B(instruct,4bit)。config.json 顶层 model_type = "gemma4",
|
||||
// 经 LLMModelFactory 的 "gemma4" 路径加载(mlx-swift-lm ≥3.31.4 已注册;Gemma4Model 只取
|
||||
// language_model 权重,跳过视觉/音频)。e2b 检查点无视觉权重(视觉在 gemma4_unified 12B)。
|
||||
// 字节数取自 ModelScope mlx-community/gemma-4-e2b-it-4bit 仓库实际 blob 大小
|
||||
//(Range 请求 Content-Range 总长,2026-07-14 与魔搭 / hf-mirror 双源核对一致)。
|
||||
// 上游 2026-07 中旬更新过一次 checkpoint(权重 3.581G→3.551G,config/tokenizer_config/
|
||||
// processor_config/index 同步变化;model_type 仍 gemma4、4bit,加载路径不受影响)——
|
||||
// 清单值过期会让 FileDownloader 校验 sizeMismatch,表现为「下载失败」,更新须双源重新核对。
|
||||
// 排除 README.md / .gitattributes / configuration.json(后者为 ModelScope 元数据,MLX 加载
|
||||
// 用不到)。Gemma 4 布局与 3n 不同:无 tokenizer.model / special_tokens_map.json,
|
||||
// tokenizer_config.json 精简(chat template 独立在 chat_template.jinja),新增
|
||||
// processor_config.json。全部运行文件按上游 mlx-swift-lm 预置仓库快照镜像,避免漏文件。
|
||||
return [
|
||||
ModelFile(path: "config.json", bytes: 937),
|
||||
ModelFile(path: "model.safetensors", bytes: 968_080_210),
|
||||
ModelFile(path: "model.safetensors.index.json", bytes: 49_731),
|
||||
ModelFile(path: "tokenizer.json", bytes: 11_422_654),
|
||||
ModelFile(path: "tokenizer_config.json", bytes: 9_706),
|
||||
ModelFile(path: "vocab.json", bytes: 2_776_833),
|
||||
ModelFile(path: "merges.txt", bytes: 1_671_853),
|
||||
ModelFile(path: "special_tokens_map.json", bytes: 613),
|
||||
ModelFile(path: "added_tokens.json", bytes: 707),
|
||||
ModelFile(path: "config.json", bytes: 6_395),
|
||||
ModelFile(path: "generation_config.json", bytes: 208),
|
||||
ModelFile(path: "model.safetensors", bytes: 3_550_670_554),
|
||||
ModelFile(path: "model.safetensors.index.json", bytes: 218_323),
|
||||
ModelFile(path: "tokenizer.json", bytes: 32_169_626),
|
||||
ModelFile(path: "tokenizer_config.json", bytes: 2_740),
|
||||
ModelFile(path: "chat_template.jinja", bytes: 17_336),
|
||||
ModelFile(path: "processor_config.json", bytes: 1_316),
|
||||
]
|
||||
case .vl:
|
||||
// Qwen3-VL-4B-Instruct-4bit:字节数取自 mlx-community 仓库实际 blob 大小
|
||||
@@ -52,6 +83,19 @@ nonisolated enum ModelManifest {
|
||||
ModelFile(path: "preprocessor_config.json", bytes: 782),
|
||||
ModelFile(path: "video_preprocessor_config.json", bytes: 817),
|
||||
]
|
||||
case .mnnLLM:
|
||||
// taobao-mnn/Qwen3.5-2B-MNN 预转换 MNN 格式(HF API 实测,2026-06)。
|
||||
// 运行时必需:config.json(MNN llm 配置)+ llm_config.json(超参)+ llm.mnn(图)
|
||||
// + llm.mnn.weight(量化权重 ~1.1GB)+ tokenizer.txt + visual.mnn(多模态,文本路径不用但配置含 mllm)。
|
||||
// 排除 README/.gitattributes 与可读 dump(llm.mnn.json / export_args.json)。
|
||||
return [
|
||||
ModelFile(path: "config.json", bytes: 652),
|
||||
ModelFile(path: "llm_config.json", bytes: 8_692),
|
||||
ModelFile(path: "llm.mnn", bytes: 2_148_136),
|
||||
ModelFile(path: "llm.mnn.weight", bytes: 1_176_647_702),
|
||||
ModelFile(path: "tokenizer.txt", bytes: 6_465_727),
|
||||
ModelFile(path: "visual.mnn", bytes: 488_096),
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,10 +103,35 @@ nonisolated enum ModelManifest {
|
||||
files(for: kind).reduce(0) { $0 + $1.bytes }
|
||||
}
|
||||
|
||||
/// 单个文件的下载 URL = baseURL / <仓库名> / <相对路径>。
|
||||
/// 单个文件的**首选**下载 URL(= 候选源列表的第一项)。主模型走魔搭官方 `resolve/master`。
|
||||
/// 下载编排应改用 `fileURLs` 拿到「魔搭优先 + hf-mirror 回退」的有序候选源;此方法保留给
|
||||
/// 只需首选源的调用方(如测试断言主源路径)。
|
||||
static func fileURL(for kind: ModelKind, file: ModelFile) -> URL {
|
||||
baseURL
|
||||
.appendingPathComponent(kind.rawValue, isDirectory: true)
|
||||
.appendingPathComponent(file.path)
|
||||
fileURLs(for: kind, file: file)[0]
|
||||
}
|
||||
|
||||
/// 单个文件的有序候选下载源:**魔搭(ModelScope,resolve/master)优先,hf-mirror(resolve/main)回退**。
|
||||
/// 两源同为 mlx-community 仓库同一份权重,字节完全一致 —— 断点续传的 `.part` 可跨源复用
|
||||
/// (换源时发 `Range: bytes=offset-` 继续,不必重下)。用户选择:魔搭优先 + hf-mirror 回退。
|
||||
/// 已废弃的 `.vl`(无魔搭 / 无 hf 回退)落到自建镜像 `baseURL`(实际不会被下载)。
|
||||
static func fileURLs(for kind: ModelKind, file: ModelFile) -> [URL] {
|
||||
var urls: [URL] = []
|
||||
if let repo = modelScopeRepo(for: kind) {
|
||||
urls.append(
|
||||
URL(string: "https://modelscope.cn/models/\(repo)/resolve/master/")!
|
||||
.appendingPathComponent(file.path))
|
||||
}
|
||||
if let hfRepo = huggingFaceMirrorRepo(for: kind) {
|
||||
urls.append(
|
||||
URL(string: "https://hf-mirror.com/\(hfRepo)/resolve/main/")!
|
||||
.appendingPathComponent(file.path))
|
||||
}
|
||||
if urls.isEmpty {
|
||||
urls.append(
|
||||
baseURL
|
||||
.appendingPathComponent(kind.rawValue, isDirectory: true)
|
||||
.appendingPathComponent(file.path))
|
||||
}
|
||||
return urls
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,14 +1,22 @@
|
||||
import Foundation
|
||||
|
||||
nonisolated enum ModelKind: String, CaseIterable {
|
||||
/// 与 HuggingFace mlx-community 仓库名一一对应,也是沙盒 Models/ 下的子目录名。
|
||||
case llm = "Qwen3-1.7B-4bit"
|
||||
case vl = "Qwen3-VL-4B-Instruct-4bit"
|
||||
/// 也是沙盒 Models/ 下的子目录名 / CDN 路径段。
|
||||
/// 主模型已从 Qwen3.5-2B → Gemma-3n → **Gemma 4 E2B(端侧 4bit,只走 MLX/GPU)**:
|
||||
/// - llm:**用户唯一下载的主模型**。MLX 文本引擎,经 LLMModelFactory 的 `gemma4` 路径加载
|
||||
/// (mlx-swift-lm ≥3.31.4 已注册;`Gemma4Model` 只取 `language_model` 权重,跳过视觉/音频)。
|
||||
/// e2b 检查点无视觉权重(视觉在 gemma4_unified 12B),故拍报告解读仍走 OCR + 文本 LLM。
|
||||
/// - vl:已废弃,保留枚举避免动一圈穷举 switch,不再下载/展示。
|
||||
/// - mnnLLM:已停用。本项目已放弃 MNN/SME2 路径;保留枚举只为兼容旧引擎判断,不再分发/计入下载。
|
||||
case llm = "gemma-4-e2b-it-4bit"
|
||||
case vl = "Qwen3-VL-4B-Instruct-4bit"
|
||||
case mnnLLM = "Qwen3.5-2B-MNN"
|
||||
|
||||
var displayName: String {
|
||||
switch self {
|
||||
case .llm: return "Qwen3-1.7B"
|
||||
case .vl: return "Qwen3-VL-4B"
|
||||
case .llm: return "Gemma 4 E2B (MLX)"
|
||||
case .vl: return "Qwen3-VL-4B"
|
||||
case .mnnLLM: return "Qwen3.5-2B (MNN/SME2)"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +25,10 @@ nonisolated enum ModelKind: String, CaseIterable {
|
||||
|
||||
/// 用于判定该模型是否已就绪的最小标志文件
|
||||
var sentinelFilename: String { "config.json" }
|
||||
|
||||
/// 面向用户的模型集合:模型管理页 / 下载全部 / 就绪计数对外只暴露统一的主模型
|
||||
/// Gemma 4 E2B(MLX,文本)。.vl/.mnnLLM 已停用,不展示、不计入「下载全部」与就绪计数。
|
||||
static let userFacing: [ModelKind] = [.llm]
|
||||
}
|
||||
|
||||
/// `@unchecked Sendable`:rootURL 是 let,方法只读 filesystem(线程安全),
|
||||
|
||||
54
康康/AI/Prompts/ConsultationPrompts.swift
Normal file
54
康康/AI/Prompts/ConsultationPrompts.swift
Normal file
@@ -0,0 +1,54 @@
|
||||
import Foundation
|
||||
|
||||
/// 「记录问诊」录音转写 → 结构化问诊小结的 prompt(2026-06-28)。
|
||||
///
|
||||
/// 与 `DiaryAssistPrompts.organize`(口述日记)同源同魂,但产物不同:
|
||||
/// 问诊录音里通常**两个人在说话**(本人 + 医生),要按就诊场景归到固定小节,方便日后翻看/给下一位医生看。
|
||||
///
|
||||
/// 红线(同 §1 / §10):
|
||||
/// - 这是**转写整理**,不是 App 在做诊断或给药建议——只忠实复述录音里医生/本人说过的话,
|
||||
/// 【绝对不许】自行新增任何诊断、用药、剂量、频次或「建议就医」。模型没听到的就不写。
|
||||
/// - 数值、单位、药名、剂量、时间一字不改(140/90 永远是 140/90)。
|
||||
enum ConsultationPrompts {
|
||||
|
||||
/// 转写稿截断上限(字符)。2B context 保护:超长问诊只取前段整理。
|
||||
static let organizeTranscriptLimit = 2000
|
||||
|
||||
static func organize(transcript: String) -> String {
|
||||
let trimmed = String(transcript.prefix(organizeTranscriptLimit))
|
||||
return """
|
||||
你是健康记录助手。下面是用户在医院/诊所就诊时的录音转写原话,通常包含本人和医生两个人的对话,
|
||||
可能口语化、有重复、缺标点、说话人没标注。请把它整理成一份清晰的【问诊小结】,方便本人日后回看。
|
||||
|
||||
硬性规则:
|
||||
- 这只是**转写整理**:只复述录音里【确实说过】的内容,【绝对不许】自己新增诊断、用药、剂量、频次或「建议就医」。
|
||||
- 输出语言必须与【录音转写原话】一致:原话是中文就用中文,是英文就用英文,对应语言不翻译(小节名可保留中文标签)。
|
||||
- 数值、单位、药名、剂量、时间【一字不改】——原话说 140/90 就写 140/90,说一天两次就写一天两次。
|
||||
- 录音里没提到的小节就【整节省略】,不要写「无」「未提及」之类占位,也不要硬凑。
|
||||
- 区分说话人:本人的主诉用第一人称「我」;医生说的话写成「医生:…」。
|
||||
- 不确定/听不清的地方保留原样,不要脑补。
|
||||
|
||||
按下面这些小节整理(只保留录音里出现过的,用「小节名:内容」分行):
|
||||
主诉 —— 本人这次来看什么、哪里不舒服
|
||||
现病史 —— 起病时间、过程、变化
|
||||
医生判断 —— 医生当场说的判断或解释(原话复述,不替医生下结论)
|
||||
医生建议 —— 医生给的检查、复查、生活建议
|
||||
用药 —— 医生提到的药名/用法(原样复述,不补剂量)
|
||||
复查与注意 —— 下次复查时间、注意事项
|
||||
|
||||
只输出整理后的小结正文,不要解释、不要 markdown 围栏、不要 <think> 标签。
|
||||
|
||||
示例(转写:就那个我最近老是胸口闷上楼梯就喘大概有半个月了医生说听着心率有点快先做个心电图和验血下周三来复查这两天别太累):
|
||||
主诉:最近胸口闷,上楼梯就喘。
|
||||
现病史:已持续约半个月。
|
||||
医生判断:医生:听着心率有点快。
|
||||
医生建议:先做心电图和验血。
|
||||
复查与注意:下周三复查,这两天别太累。
|
||||
|
||||
【录音转写原话】:
|
||||
\(trimmed)
|
||||
|
||||
Output: /no_think
|
||||
"""
|
||||
}
|
||||
}
|
||||
@@ -11,12 +11,32 @@ enum DiaryAssistPrompts {
|
||||
|
||||
/// 固定问诊维度清单。模型每条问题必须正好归属其中一个;UI 累积已覆盖维度回传下一轮。
|
||||
/// 顺序即展示/示例顺序,改动需同步 few-shot。
|
||||
/// 注意:这是**受控词表 + 去重键**,模型固定输出中文维度名(prompt 已约束),
|
||||
/// 内部匹配/去重都按中文原名进行,只在**展示**时经 `localizedDimension` 翻成界面语言。
|
||||
static let dimensions: [String] = [
|
||||
"起病诱因", "症状性质", "伴随症状", "加重缓解",
|
||||
"持续频率", "既往家族史", "用药过敏", "生活方式",
|
||||
]
|
||||
|
||||
/// - content: 患者当前全文。
|
||||
/// 把中文维度名翻成界面语言,仅用于 UI 展示(如追问输入框的占位提示)。
|
||||
/// 走 `String(appLoc:)` 查表:这些维度名已作为 key 收进 Localizable.xcstrings。
|
||||
/// 词表外的自定义 dim(模型偶发)原样返回,不至于丢字。
|
||||
nonisolated static func localizedDimension(_ dim: String) -> String {
|
||||
let d = dim.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
switch d {
|
||||
case "起病诱因": return String(appLoc: "起病诱因")
|
||||
case "症状性质": return String(appLoc: "症状性质")
|
||||
case "伴随症状": return String(appLoc: "伴随症状")
|
||||
case "加重缓解": return String(appLoc: "加重缓解")
|
||||
case "持续频率": return String(appLoc: "持续频率")
|
||||
case "既往家族史": return String(appLoc: "既往家族史")
|
||||
case "用药过敏": return String(appLoc: "用药过敏")
|
||||
case "生活方式": return String(appLoc: "生活方式")
|
||||
default: return d
|
||||
}
|
||||
}
|
||||
|
||||
/// - content: 用户当前全文。
|
||||
/// - coveredDimensions: 之前各轮已经问过(或记录里已写明)的维度名,本轮必须避开。
|
||||
/// 第一轮传空数组。
|
||||
static func suggest(content: String, coveredDimensions: [String] = []) -> String {
|
||||
@@ -30,8 +50,8 @@ enum DiaryAssistPrompts {
|
||||
: "\n- 已问过的维度【不要再问】:\(covered.joined(separator: "、"))。本轮只能从这些还没问的维度里挑:\(allowedLine)。"
|
||||
|
||||
return """
|
||||
你是社区医生的小助手。患者写了一段身体状态的健康记录,信息可能不够完整。
|
||||
请从医生问诊角度提出 3-4 个最值得追问的问题,帮患者把这条记录补全。
|
||||
你是社区医生的小助手。用户写了一段身体状态的健康记录,信息可能不够完整。
|
||||
请从医生问诊角度提出 3-4 个最值得追问的问题,帮用户把这条记录补全。
|
||||
|
||||
【问诊维度清单】每个问题必须正好归属其中一个,并用 dim 标注:
|
||||
1. 起病诱因 —— 何时开始、有无诱因
|
||||
@@ -72,6 +92,48 @@ enum DiaryAssistPrompts {
|
||||
本轮可选维度:\(allowedLine)
|
||||
【最新记录】:
|
||||
\(content)
|
||||
\(aiOutputLanguageDirective())
|
||||
（注意:dim 维度名仍固定用上面清单里的中文原名;只有 q 和 fill 面向用户的文字用上述语言。）
|
||||
Output: /no_think
|
||||
"""
|
||||
}
|
||||
|
||||
// MARK: - 语音口述 → 日记整理
|
||||
|
||||
/// 口述转写稿截断上限(字符)。2B 模型 context 保护:超长口述只取前面部分。
|
||||
static let organizeTranscriptLimit = 1200
|
||||
|
||||
/// 把语音转写稿整理成健康日记草稿。自适应样式:内容少 → 一段通顺的话;
|
||||
/// 多方面 → 按「方面:内容」分行。
|
||||
/// 红线(spec 2026-06-10-voice-diary §2):只重组语言,严禁增删改任何数值、单位、药名、时间——
|
||||
/// 2B 模型把 140/90 改成 130/90 即健康数据事故,所以规则放第一条并配 few-shot 强化。
|
||||
static func organize(transcript: String) -> String {
|
||||
let trimmed = String(transcript.prefix(organizeTranscriptLimit))
|
||||
return """
|
||||
你是健康记录助手。下面是用户口述身体状态的语音转写原话,可能口语化、有重复、缺标点。
|
||||
请把它整理成一条清晰的健康日记。
|
||||
|
||||
硬性规则:
|
||||
- 【绝对不许】增加、删除或改动任何数值、单位、药名、时间——原话说 140/90 就必须写 140/90。
|
||||
- 输出语言必须与【口述原话】完全一致:原话是中文就用中文,是英文就用英文,是日文/韩文就用对应语言,不要翻译。
|
||||
- 只重组语言:去掉口头语和重复;用第一人称;不加入原话没有的事实。
|
||||
- 内容只涉及一两个方面 → 整理成一段通顺的话(2-4 句)。
|
||||
- 内容涉及多个方面(症状/用药/饮食/睡眠/运动等) → 按「方面:内容」分行。
|
||||
- 不诊断、不给用药建议、不写「建议就医」。
|
||||
- 只输出整理后的日记正文,不要解释、不要 markdown 围栏、不要 <think> 标签。
|
||||
|
||||
示例 1(口述:那个今天早上起来有点头晕然后我量了下血压140 90比平时高一点没吃早饭就出门了):
|
||||
今天早上起来有点头晕,量了血压 140/90,比平时高一点。没吃早饭就出门了。
|
||||
|
||||
示例 2(口述:今天头晕了一上午下午好点了血压早上量的140 90嗯缬沙坦吃了降脂药忘了吃早饭没吃中午吃的清淡晚上散步了半小时):
|
||||
症状:头晕了一上午,下午好转。
|
||||
血压:早上 140/90。
|
||||
用药:缬沙坦已服,降脂药忘服。
|
||||
饮食:早饭未吃,午餐清淡。
|
||||
运动:晚上散步半小时。
|
||||
|
||||
【口述原话】:
|
||||
\(trimmed)
|
||||
|
||||
Output: /no_think
|
||||
"""
|
||||
|
||||
143
康康/AI/Prompts/DiaryChatPrompts.swift
Normal file
143
康康/AI/Prompts/DiaryChatPrompts.swift
Normal file
@@ -0,0 +1,143 @@
|
||||
import Foundation
|
||||
|
||||
/// 「聊着记」用到的三个 LLM prompt:全屏动画小形象「康康」和用户语音聊天,
|
||||
/// 基于已有健康记录做有记忆的追问,聊完蒸馏成一条健康日记。
|
||||
///
|
||||
/// 人设铁律(贯穿三个 prompt):康康是「陪伴记录的小伙伴」不是医生——
|
||||
/// 不诊断、不荐药、不写「建议就医」、不评价病情轻重、不用「患者」称呼。
|
||||
///
|
||||
/// 数值红线沿用 `DiaryAssistPrompts.organize`:蒸馏时严禁增删改任何数值/单位/药名/时间;
|
||||
/// 语言规则对齐 §「不用「患者」称呼」与 aiOutputLanguageDirective(App/Localization.swift)。
|
||||
///
|
||||
/// nonisolated:纯字符串拼装,无共享状态;`DiaryChatService` 的 nonisolated 纯函数
|
||||
/// (roundsUsed/isWrappedUp)要引用 `maxRounds`,不标会被默认 MainActor 隔离挡住。
|
||||
nonisolated enum DiaryChatPrompts {
|
||||
|
||||
/// 收尾前允许的追问总轮数(开场那句提问不计入)。
|
||||
static let maxRounds = 4
|
||||
|
||||
/// 对话稿截断上限(字符),2B/E2B 端侧模型 context 保护。
|
||||
static let distillTranscriptLimit = 2000
|
||||
|
||||
/// AI 全挂时的确定性开场白(View 兜底用,不经 LLM)。
|
||||
static var staticOpening: String { String(appLoc: "今天身体感觉怎么样?想到什么就说什么,我帮你记着。") }
|
||||
|
||||
// MARK: - 开场白
|
||||
|
||||
/// 用户刚进「聊着记」,康康说的第一句话:自然打招呼 + 基于最近一条记录关心地追问 1 个问题。
|
||||
static func opening(dataJSON: String) -> String {
|
||||
"""
|
||||
你是「康康」,用户健康 App 里陪伴记录的小伙伴,你不是医生。
|
||||
用户刚打开「聊着记」,你要说第一句话:先自然地打个招呼,再基于下面【健康记录】里最近一条记录,像老朋友一样关心地问 1 个问题。
|
||||
|
||||
铁律:
|
||||
- 只能提到【健康记录】里真实出现过的内容,记录里没有的绝不编造。
|
||||
- 数值、日期、药名必须与记录完全一致,一个字都不能改。
|
||||
- 不诊断、不给用药或剂量建议、不写「建议就医」、不评价病情轻重、不用「患者」称呼。
|
||||
- 最多 2 句话,只问 1 个问题;不用列表、不用 Markdown、不用表情符号。
|
||||
- 记录里 diaries / indicators 全为空时,不提任何具体记录,只简单问今天感觉怎么样。
|
||||
|
||||
示例 1(记录里有「2026-07-11 日记:头疼了一下午」):
|
||||
前天你记过头疼了一下午,现在好些了吗?
|
||||
|
||||
示例 2(记录里有血压 145/92 偏高):
|
||||
昨天量的血压 145/92 有点偏高,今天有没有再量一次?
|
||||
|
||||
示例 3(记录为空):
|
||||
嗨,今天身体感觉怎么样?想到什么就跟我说说,我帮你记下来。
|
||||
|
||||
【健康记录】:
|
||||
\(dataJSON)
|
||||
|
||||
直接输出这句话,不要引号、不要解释、不要 <think> 标签。
|
||||
\(aiOutputLanguageDirective())
|
||||
/no_think
|
||||
"""
|
||||
}
|
||||
|
||||
// MARK: - 追问回复
|
||||
|
||||
/// 用户每说一句,康康的回复:先接住,再按剩余轮次决定是否追问。
|
||||
/// - roundsLeft: 收尾前还剩几轮追问机会;> 0 可再追问,== 0 必须温和收尾。
|
||||
static func reply(transcript: String, latest: String, dataJSON: String, roundsLeft: Int) -> String {
|
||||
// 轮次分支在 Swift 里拼,不让模型自己算:两段互斥,收尾分支绝不出现「追问」字样。
|
||||
let roundRule: String = roundsLeft > 0
|
||||
? "本轮还可以追问,最多问 1 个新问题。追问只围绕「把这条记录补完整」:时间、部位、程度、吃没吃药、睡眠饮食等;可以结合【健康记录】做有记忆的追问(如「上次你记过 XX,这次也一样吗」)。"
|
||||
: "【不要再提任何问题】。用 1-2 句话温和收尾:先接住用户刚说的话,再告诉 TA 今天先聊到这儿,点下面的「聊完了」,我帮你整理成一条记录。"
|
||||
return """
|
||||
你是「康康」,用户健康 App 里陪伴记录的小伙伴,你不是医生。
|
||||
你正在「聊着记」里陪用户聊天。任务:先用一小句接住用户刚说的话(共情或确认),再按【本轮规则】决定要不要追问。
|
||||
|
||||
铁律:
|
||||
- 只能提到【健康记录】里真实出现过的内容,记录里没有的绝不编造;数值、日期、药名必须与记录完全一致。
|
||||
- 不诊断、不给用药或剂量建议、不写「建议就医」、不评价病情轻重、不用「患者」称呼。
|
||||
- 总共最多 2 句话;不用列表、不用 Markdown、不用表情符号。
|
||||
|
||||
【本轮规则】:
|
||||
\(roundRule)
|
||||
|
||||
示例(还能追问,用户说「下午头就不疼了,不过晚上只睡了五个小时」):
|
||||
不疼了就好。昨晚只睡五个小时,是入睡难还是半夜醒了?
|
||||
|
||||
示例(收尾):
|
||||
好,记下啦。今天先聊到这儿,点下面的「聊完了」,我帮你把这些整理成一条记录。
|
||||
|
||||
【健康记录】:
|
||||
\(dataJSON)
|
||||
|
||||
【对话】:
|
||||
\(transcript.isEmpty ? "无" : transcript)
|
||||
|
||||
【用户刚说的】:
|
||||
\(latest)
|
||||
|
||||
直接输出你要说的话,不要引号、不要解释、不要 <think> 标签。\(aiOutputLanguageDirective())
|
||||
/no_think
|
||||
"""
|
||||
}
|
||||
|
||||
// MARK: - 蒸馏成日记
|
||||
|
||||
/// 聊完后把整段对话蒸馏成一条健康日记草稿。结构与红线对齐 `DiaryAssistPrompts.organize`:
|
||||
/// 只搬运不改写、严禁增删改数值,且**不拼** aiOutputLanguageDirective——语言跟「我:」原话走。
|
||||
static func distill(transcript: String) -> String {
|
||||
let trimmed = String(transcript.prefix(distillTranscriptLimit))
|
||||
return """
|
||||
你是「康康」,用户健康 App 里陪伴记录的小伙伴。下面是你和用户在「聊着记」里的对话稿。
|
||||
请把用户说的话蒸馏成一条健康日记草稿。
|
||||
|
||||
硬性规则:
|
||||
- 【只取「我:」说的内容作为事实】——「康康:」的话只是提问和陪伴,绝不能把康康提到的记录、猜测或问题当成用户今天发生的事写进日记。
|
||||
- 【绝对不许】增加、删除或改动任何数值、单位、药名、时间——原话说 140/90 就必须写 140/90。
|
||||
- 输出语言必须与「我:」的原话完全一致:原话是中文就用中文,是英文就用英文,是日文/韩文就用对应语言,不要翻译。
|
||||
- 用第一人称;去掉口头语和重复;不加入对话里没有的事实。
|
||||
- 内容只涉及一两个方面 → 整理成一段通顺的话(2-4 句)。
|
||||
- 内容涉及多个方面(症状/用药/饮食/睡眠/运动等) → 按「方面:内容」分行。
|
||||
- 不诊断、不给用药建议、不写「建议就医」。
|
||||
- 只输出日记正文,不要开头语、不要解释、不要 markdown 围栏、不要 <think> 标签。
|
||||
|
||||
示例 1(单方面 → 成段):
|
||||
我: 今天头有点晕早上量了血压140 90
|
||||
康康: 记下啦。上次你记过没吃早饭,今天吃了吗?
|
||||
我: 吃了吃了 喝了粥
|
||||
输出:
|
||||
今天头有点晕,早上量了血压 140/90。早饭吃了,喝了粥。
|
||||
|
||||
示例 2(多方面 → 分行;注意康康的话不进日记):
|
||||
我: 今天头晕了一上午下午好点了血压早上量的140 90缬沙坦吃了
|
||||
康康: 记下啦。上次你血压也偏高,是不是又没睡好?昨晚睡了几个小时?
|
||||
我: 就睡了五个小时中午吃的清淡
|
||||
输出:
|
||||
症状:头晕了一上午,下午好转。
|
||||
血压:早上 140/90。
|
||||
用药:缬沙坦已服。
|
||||
睡眠:只睡了五个小时。
|
||||
饮食:午餐清淡。
|
||||
|
||||
【对话记录】:
|
||||
\(trimmed)
|
||||
|
||||
Output: /no_think
|
||||
"""
|
||||
}
|
||||
}
|
||||
@@ -70,19 +70,20 @@ enum HealthExportPrompts {
|
||||
- 严禁编造或推测任何数字、日期、症状、药物、检查结果、诊断,哪怕看起来很合理。
|
||||
- JSON 里没有的信息,对应小节一律写「无记录」,不要补全、不要举例、不要套用常见病例模板。
|
||||
- 数值必须原样照搬(含单位与参考范围);status 为 high/low/abnormal 的指标前加 ⚠️。
|
||||
- 「主诉」「患者疑问」可参考【患者原话】,但不得加入原话与数据里都没有的症状。
|
||||
- 「主诉」「本人疑问」可参考【本人原话】,但不得加入原话与数据里都没有的症状。
|
||||
- diaries 里 kind 为「问诊」的是既往看医生的问诊记录,可据此补充「主诉」「本人背景」,但同样只搬运、不编造。
|
||||
|
||||
输出格式:
|
||||
- 严格 Markdown,标题用 # / ##,不要 markdown 围栏,不要输出 JSON,不写「数据」二字。
|
||||
- 不给诊断意见、用药建议或「建议就医」。全文中文,简洁,医生 30 秒能扫完。
|
||||
- 不给诊断意见、用药建议或「建议就医」。简洁,医生 30 秒能扫完(输出语言见文末说明)。
|
||||
- 严格按以下 6 段(顺序与标题固定):
|
||||
\(labelLine)
|
||||
## 主诉
|
||||
## 患者背景
|
||||
## 本人背景
|
||||
## 近期症状(按时间倒序)
|
||||
## 关键指标(异常项优先)
|
||||
## 在服药与过敏
|
||||
## 患者疑问
|
||||
## 本人疑问
|
||||
|
||||
—— 格式示例(只示范「无记录」与数值写法,内容请勿照抄)——
|
||||
真实数据:{"profile":{},"symptoms":[],"indicators":[{"name":"体温","value":"38.5","unit":"℃","range":"36-37.2","status":"high","date":"2026-05-01"}],"reports":[],"diaries":[],"time_window":{"from":"2026-04-02","to":"2026-05-02"}}
|
||||
@@ -90,7 +91,7 @@ enum HealthExportPrompts {
|
||||
# 就诊摘要 — 近期健康摘要
|
||||
## 主诉
|
||||
无记录
|
||||
## 患者背景
|
||||
## 本人背景
|
||||
无记录
|
||||
## 近期症状(按时间倒序)
|
||||
无记录
|
||||
@@ -98,7 +99,7 @@ enum HealthExportPrompts {
|
||||
⚠️ 体温 38.5 ℃(参考 36-37.2,2026-05-01)
|
||||
## 在服药与过敏
|
||||
无记录
|
||||
## 患者疑问
|
||||
## 本人疑问
|
||||
无记录
|
||||
—— 示例结束(以上咳嗽/体温等仅示范格式,切勿出现在你的输出里)——
|
||||
|
||||
@@ -107,10 +108,10 @@ enum HealthExportPrompts {
|
||||
【真实数据】:
|
||||
\(dataJSON)
|
||||
|
||||
【患者原话】:\(userPrompt)
|
||||
【本人原话】:\(userPrompt)
|
||||
|
||||
再次强调:只整理上面【真实数据】里真实出现过的内容,禁止编造任何数字/日期/症状/药物。
|
||||
直接输出 Markdown,不要思考过程,不要 <think> 标签:
|
||||
直接输出 Markdown,不要思考过程,不要 <think> 标签:\(aiOutputLanguageDirective())
|
||||
/no_think
|
||||
"""
|
||||
}
|
||||
@@ -141,7 +142,7 @@ enum HealthExportPrompts {
|
||||
【用户最新问题】:
|
||||
\(latestQuestion)
|
||||
|
||||
直接输出中文回答,不要 Markdown 标题,不要 <think>:
|
||||
直接输出回答,不要 Markdown 标题,不要 <think>:\(aiOutputLanguageDirective())
|
||||
/no_think
|
||||
"""
|
||||
}
|
||||
@@ -156,20 +157,23 @@ enum HealthExportPrompts {
|
||||
铁律:
|
||||
- 只能使用【本地健康记录】和【多轮对话】里真实出现的信息。
|
||||
- 禁止编造数字、日期、症状、药物、检查结果、诊断。
|
||||
- 日期一律照搬【本地健康记录】JSON 里的完整 `date` 字段(格式 yyyy-MM-dd,即年-月-日);严禁只写年份或省略月、日。多轮对话里若把日期说得不全,一律以 JSON 的完整日期为准。
|
||||
- 禁止给诊断意见、用药建议、剂量建议或急诊判断。
|
||||
- JSON 里没有的信息,对应小节写「无记录」。
|
||||
- 指标 status 为 high/low/abnormal 的项目前加 ⚠️。
|
||||
|
||||
输出要求:
|
||||
- 严格 Markdown,不要 markdown 围栏,不要输出 JSON。
|
||||
- 中文,简洁,医生 30 秒能扫完。
|
||||
- 简洁,医生 30 秒能扫完(输出语言见文末说明)。
|
||||
- 「相关健康日记」每条单独一行,格式为「2026-05-01:正文摘要」,日期照抄 JSON 的 date 字段,精确到日。
|
||||
- diaries 里 kind 为「问诊」的是既往问诊记录,优先列入「相关健康日记」并标注「(问诊)」。
|
||||
- 严格按以下段落:
|
||||
# 就诊摘要
|
||||
## 本次想解决的问题
|
||||
## 相关健康日记
|
||||
## 相关指标
|
||||
## 已知背景
|
||||
## 患者关心的问题
|
||||
## 本人关心的问题
|
||||
## 可带给医生确认的要点
|
||||
|
||||
【本地健康记录】:
|
||||
@@ -178,7 +182,7 @@ enum HealthExportPrompts {
|
||||
【多轮对话】:
|
||||
\(transcript.isEmpty ? "无" : transcript)
|
||||
|
||||
直接输出 Markdown,不要思考过程,不要 <think>:
|
||||
直接输出 Markdown,不要思考过程,不要 <think>:\(aiOutputLanguageDirective())
|
||||
/no_think
|
||||
"""
|
||||
}
|
||||
|
||||
44
康康/AI/Prompts/InsightPrompts.swift
Normal file
44
康康/AI/Prompts/InsightPrompts.swift
Normal file
@@ -0,0 +1,44 @@
|
||||
import Foundation
|
||||
|
||||
/// 本地解读类 prompt:报告大白话摘要 + 趋势一句话解读。
|
||||
/// 红线:不诊断、不荐药;称呼「你」,不出现「患者」(产品定位:自我健康记录)。
|
||||
nonisolated enum InsightPrompts {
|
||||
|
||||
/// 报告整体大白话摘要(归档后台预生成,写回 Report.summary)。
|
||||
static func reportPlainSummary(title: String, typeLabel: String, indicatorLines: String) -> String {
|
||||
"""
|
||||
你是健康档案助手。下面是一份报告的指标列表,请用大白话给本人(称「你」)写 2~3 句整体解读:
|
||||
- 第 1 句:总体情况(共几项、几项异常)。
|
||||
- 之后:点名最值得留意的异常项,用生活化语言说明偏高/偏低意味着什么方向。
|
||||
- 不诊断疾病、不推荐药物或剂量;异常较多时建议「带上报告咨询医生」。
|
||||
- 只输出正文文字,不要标题、列表、JSON、markdown。
|
||||
|
||||
示例:
|
||||
输入:血常规(化验单),指标:白细胞 5.2 (3.5-9.5) normal;血红蛋白 118 (130-175) low;血小板 210 (125-350) normal
|
||||
输出:这份血常规共 3 项,2 项正常,血红蛋白略低于参考范围。血红蛋白偏低通常与贫血方向有关,平时可以多补充含铁食物;如果还伴随乏力头晕,建议带上报告咨询医生。
|
||||
|
||||
现在的报告:\(title)(\(typeLabel))
|
||||
指标:
|
||||
\(indicatorLines)
|
||||
只输出 2~3 句正文。\(aiOutputLanguageDirective())/no_think
|
||||
"""
|
||||
}
|
||||
|
||||
/// 趋势一句话解读(TrendDetailView,按数据指纹缓存)。
|
||||
static func trendInsight(title: String, unit: String, rangeText: String, dataLines: String) -> String {
|
||||
"""
|
||||
你是健康档案助手。下面是「\(title)」的历史记录(单位 \(unit)\(rangeText)),请用大白话给本人(称「你」)写 1~2 句趋势解读:
|
||||
- 说清整体走向(上升/下降/平稳/波动)和当前值与参考范围的关系。
|
||||
- 不诊断疾病、不推荐药物;持续异常时温和建议「复查或咨询医生」。
|
||||
- 只输出正文文字,不要标题、列表、JSON。
|
||||
|
||||
示例:
|
||||
输入:体重,单位 kg,记录:2026-04-01 72.5 / 2026-04-15 71.8 / 2026-05-01 71.2
|
||||
输出:近一个月你的体重稳步下降了约 1.3kg,节奏平缓,继续保持现在的习惯就好。
|
||||
|
||||
现在的记录:
|
||||
\(dataLines)
|
||||
只输出 1~2 句正文。\(aiOutputLanguageDirective())/no_think
|
||||
"""
|
||||
}
|
||||
}
|
||||
58
康康/AI/Prompts/IntentPrompts.swift
Normal file
58
康康/AI/Prompts/IntentPrompts.swift
Normal file
@@ -0,0 +1,58 @@
|
||||
import Foundation
|
||||
|
||||
/// 「长按 + 语音直达」prompt:端侧语音转写文本 → LLM(MNN/SME2 主链路)分类到新建入口。
|
||||
/// 输出契约:严格 JSON `{"intent":"…"}`;解析失败/超时 → VoiceIntentService 回退关键词匹配(§3.2)。
|
||||
nonisolated enum IntentPrompts {
|
||||
|
||||
static func classify(_ utterance: String) -> String {
|
||||
classifyTemplate.replacingOccurrences(of: "{{TEXT}}", with: String(utterance.prefix(120)))
|
||||
}
|
||||
|
||||
private static let classifyTemplate: String = #"""
|
||||
你是健康 App 的语音意图分类器。用户长按「新建」按钮说了一句话,判断 ta 想打开哪个功能。
|
||||
请只输出一段合法 JSON,格式 {"intent":"<分类>"},不要解释、不要 markdown 围栏、不要任何前后缀文字。
|
||||
|
||||
分类(只能选下面其中一个):
|
||||
- "diary" 写日记,记录今天的感受、饮食、睡眠、身体状态
|
||||
- "medication" 记一次用药/服药、吃了什么药、拍药盒(凡涉及「吃药/服药/用药」都归这里)
|
||||
- "symptom" 记录身体症状,哪里不舒服(头疼、咳嗽、发烧、头晕…),与吃药无关
|
||||
- "indicator" 记录指标数值(血压、血糖、体重、心率、体温…)
|
||||
- "archive" 归档整份体检报告/化验单(拍报告存档)
|
||||
- "export" 生成给医生看的身体档案/健康总结
|
||||
- "reminder" 设置周期提醒
|
||||
- "unknown" 无法判断
|
||||
|
||||
规则:
|
||||
- 说到「提醒我…」一律 "reminder",即使内容涉及吃药或量血压。
|
||||
- 凡是「记录/记一次用药、服药、吃药、吃了药」→ "medication",哪怕没说具体药名。
|
||||
- 「记录/记一次」+ 动作时,先看这个动作是什么(吃药→medication、量血压→indicator、
|
||||
哪里疼→symptom),不要因为出现「记录」二字就归类成 symptom。
|
||||
- 明确说出具体身体症状(头疼、咳嗽、发烧、头晕、拉肚子…)才算 "symptom";
|
||||
与吃药/用药无关。只是泛泛说今天的状态、心情、饮食、睡眠、累不累、舒不舒服 → "diary"。
|
||||
- 既像日记又提到具体数值时,以数值为准 → "indicator"。
|
||||
- 含否定或「忘了/没顾上」的吃药(「没吃药」「忘了吃药」「不用吃药」)不是记录用药 → "diary"。
|
||||
- 只有明确要「拍下/存档这份报告或化验单」时才算 "archive";只是顺口提到体检或报告
|
||||
(「下周去体检」「医生说报告没问题」)不要归 archive,按日记或提醒处理。
|
||||
- 拿不准、又不明确属于其它类别时,默认 "diary"(日记是最常见、最自由的入口)。
|
||||
尤其 "medication" 和 "archive" 会直接打开相机,把握不大时宁可归 "diary",不要误开相机。
|
||||
|
||||
示例:
|
||||
"帮我记一下今天的血压,高压128低压85" → {"intent":"indicator"}
|
||||
"我今天有点头疼,想记录一下" → {"intent":"symptom"}
|
||||
"我要记录一次用药" → {"intent":"medication"}
|
||||
"记一下今天吃的药" → {"intent":"medication"}
|
||||
"今天有点累,感觉不太舒服" → {"intent":"diary"}
|
||||
"刚买了一盒降压药,拍一下存进去" → {"intent":"medication"}
|
||||
"今天睡得不错,写个日记" → {"intent":"diary"}
|
||||
"把这份体检报告存档" → {"intent":"archive"}
|
||||
"今天太忙,忘了吃药" → {"intent":"diary"}
|
||||
"下周打算去做个体检" → {"intent":"diary"}
|
||||
"医生说我报告没什么大问题" → {"intent":"diary"}
|
||||
"每天早上八点提醒我量血压" → {"intent":"reminder"}
|
||||
"整理一份给医生看的健康总结" → {"intent":"export"}
|
||||
|
||||
现在判断下面这句话,只输出 JSON。/no_think
|
||||
|
||||
用户的话:{{TEXT}}
|
||||
"""#
|
||||
}
|
||||
64
康康/AI/Prompts/MedicationPrompts.swift
Normal file
64
康康/AI/Prompts/MedicationPrompts.swift
Normal file
@@ -0,0 +1,64 @@
|
||||
import Foundation
|
||||
|
||||
/// 「拍药盒入档」prompt:Vision OCR 出药盒/说明书/处方文字后,
|
||||
/// 交 LLM(Qwen,MNN/SME2 主链路)结构化抽药品名 + 规格 + 用法。
|
||||
/// 输出契约:严格 JSON;解析失败 → UI 回退手动录入(§3.2 失败回退红线)。
|
||||
/// 注意:只做"识别入档",不做剂量推荐/用药提醒(§1 明确不做)。
|
||||
nonisolated enum MedicationPrompts {
|
||||
|
||||
static func medicationsFromText(_ ocrText: String) -> String {
|
||||
// ≤5 张合并 OCR,放宽到 2400(单张 1200 偏小,易把背面用法截掉)。
|
||||
medicationsFromTextTemplate
|
||||
.replacingOccurrences(of: "{{OCR_TEXT}}", with: VLPrompts.clipOCR(ocrText, limit: 2400))
|
||||
}
|
||||
|
||||
private static let medicationsFromTextTemplate: String = #"""
|
||||
你是药品包装识别助手。下面是对一种药品的多张照片(药盒正面/背面/说明书/处方单)做 OCR 得到的纯文本,各张之间用「---」分隔,可能有错字、换行混乱或无关噪声。
|
||||
请从中提取药品信息,只输出一段合法 JSON,不要解释、不要 markdown 围栏、不要任何前后缀文字。
|
||||
|
||||
JSON schema(严格):
|
||||
{
|
||||
"medications": [
|
||||
{
|
||||
"name": string, // 药品名,见下方「name 怎么填」
|
||||
"strength": string, // 规格,如 "80mg"、"0.5g×24片";识别不出填 ""
|
||||
"usage": string // 用法用量,如 "每日一次,一次一粒";包装上没有就填 ""
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
name 怎么填(关键,别搞混):
|
||||
- 药品名 = 通用名(化学/药典名),这是要填进 name 的主体。中文药名照中文写,英文药名(如 "Metformin"、"Amoxicillin")就照英文原样抄,不要翻译、不要丢。
|
||||
- 若包装上同时印有商品名/商标名(厂商起的牌子名,如 "代文""泰诺""Tylenol"),把它放在通用名后的括号里,例如 "缬沙坦胶囊(代文)"。只读到商品名、读不到通用名时,就直接用商品名当 name。
|
||||
- 生产厂家/公司名/品牌 LOGO 文字(如 "XX药业有限公司""诺华""拜耳")不是药名,一律不要当 name,也不要塞进括号。
|
||||
|
||||
通用规则:
|
||||
- 只提取药品本身;"国药准字"批准文号、生产厂家、批号、有效期、条形码、贮藏、二维码一律忽略。
|
||||
- 多张照片通常是同一种药的不同面,合并成一条,不要因为来自不同照片就重复输出;处方单可能有多种药,才分多条。
|
||||
- 不要发明药品。名称读不清的整条跳过;strength / usage 读不清就填 "",不要编造。
|
||||
- 不要输出任何服药建议或剂量调整建议,只抄录包装上已有的文字。
|
||||
- 同一药品只输出一次。
|
||||
|
||||
示例 1(药盒,含商品名 + 厂商):
|
||||
输入 OCR 文本: 代文 缬沙坦胶囊 80mg×7粒 国药准字H20103521 北京诺华制药有限公司
|
||||
输出:
|
||||
{"medications":[{"name":"缬沙坦胶囊(代文)","strength":"80mg×7粒","usage":""}]}
|
||||
|
||||
示例 2(说明书含用法):
|
||||
输入 OCR 文本: 二甲双胍缓释片 0.5g×30片 用法用量:口服,一次1片,一日2次,随餐服用
|
||||
输出:
|
||||
{"medications":[{"name":"二甲双胍缓释片","strength":"0.5g×30片","usage":"口服,一次1片,一日2次,随餐服用"}]}
|
||||
|
||||
示例 3(英文药名,正反两张合并):
|
||||
输入 OCR 文本: Amoxicillin Capsules 500mg GSK
|
||||
---
|
||||
Dosage: Take one capsule three times daily
|
||||
输出:
|
||||
{"medications":[{"name":"Amoxicillin","strength":"500mg","usage":"Take one capsule three times daily"}]}
|
||||
|
||||
现在请解析下面这段 OCR 文本,只输出 JSON。/no_think
|
||||
|
||||
OCR 文本:
|
||||
{{OCR_TEXT}}
|
||||
"""#
|
||||
}
|
||||
@@ -3,7 +3,7 @@ import Foundation
|
||||
/// VL 模型(Qwen3-VL)用于体检 / 化验单识别的 prompt 模板。
|
||||
/// 输出契约:严格 JSON,无任何解释文字、markdown 围栏或前后缀。
|
||||
/// 解析失败 → CaptureService 回退到手动录入(§3.2 失败回退红线)。
|
||||
enum VLPrompts {
|
||||
nonisolated enum VLPrompts {
|
||||
|
||||
/// 输出 JSON 的字段定义(写进 prompt 里教模型):
|
||||
/// ```
|
||||
@@ -31,12 +31,38 @@ enum VLPrompts {
|
||||
|
||||
/// VL 模型不知"今天"是哪天,且 few-shot 示例里写死了日期,
|
||||
/// 必须把当天日期显式注入 prompt,模型在无报告日期时才会用对正确的回退值。
|
||||
static func reportExtraction(today: Date = .now) -> String {
|
||||
/// ocrText 非空时把 Vision OCR 的结果作为参考文本注入 —— Vision 抄数字比
|
||||
/// 2B 多模态读密集小字稳;版面与表格结构仍以图片为准。
|
||||
static func reportExtraction(today: Date = .now, ocrText: String = "") -> String {
|
||||
let f = DateFormatter()
|
||||
f.locale = Locale(identifier: "en_US_POSIX")
|
||||
f.dateFormat = "yyyy-MM-dd"
|
||||
let todayStr = f.string(from: today)
|
||||
return reportExtractionTemplate.replacingOccurrences(of: "{{TODAY}}", with: todayStr)
|
||||
let ocrSection: String
|
||||
if ocrText.isEmpty {
|
||||
ocrSection = ""
|
||||
} else {
|
||||
ocrSection = """
|
||||
|
||||
|
||||
OCR 参考文本(系统对同一报告做文字识别的结果,可能有错字、串行或漏行;版面与表格结构以图片为准,但数值、小数点以 OCR 文字更可靠):
|
||||
\(clipOCR(ocrText))
|
||||
|
||||
"""
|
||||
}
|
||||
return reportExtractionTemplate
|
||||
.replacingOccurrences(of: "{{TODAY}}", with: todayStr)
|
||||
.replacingOccurrences(of: "{{OCR_SECTION}}", with: ocrSection)
|
||||
}
|
||||
|
||||
/// OCR 文本截断:限制进入 prompt 的体量(2B 模型上下文有限)。截到最后一个完整行。
|
||||
static func clipOCR(_ text: String, limit: Int = 1800) -> String {
|
||||
guard text.count > limit else { return text }
|
||||
let clipped = String(text.prefix(limit))
|
||||
if let lastNewline = clipped.lastIndex(of: "\n") {
|
||||
return String(clipped[..<lastNewline]) + "\n(后续内容过长已截断)"
|
||||
}
|
||||
return clipped + "\n(后续内容过长已截断)"
|
||||
}
|
||||
|
||||
private static let reportExtractionTemplate: String = #"""
|
||||
@@ -84,13 +110,120 @@ JSON schema(严格):
|
||||
输入: 一份春季体检,3 项可读
|
||||
输出:
|
||||
{"title":"春季年度体检","type":"checkup","report_date":"2026-04-12","institution":"协和医院","page_count":1,"summary":"血脂偏高、其他正常","indicators":[{"name":"低密度脂蛋白","value":"3.84","unit":"mmol/L","range":"< 3.40","status":"high","source_page":1,"source_box":[0.12,0.31,0.76,0.07]},{"name":"谷丙转氨酶","value":"32","unit":"U/L","range":"9 - 50","status":"normal","source_page":1,"source_box":[0.12,0.39,0.76,0.07]},{"name":"空腹血糖","value":"5.2","unit":"mmol/L","range":"3.9 - 6.1","status":"normal","source_page":1,"source_box":[0.12,0.47,0.76,0.07]}]}
|
||||
|
||||
{{OCR_SECTION}}
|
||||
现在请识别图片并输出 JSON:
|
||||
"""#
|
||||
|
||||
// MARK: - 局部小框识别(异常项快拍)
|
||||
// MARK: - 报告整份解读(OCR 文本 → 完整 ParsedReport,纯文本 LLM)
|
||||
|
||||
/// 异常项快拍专用:输入是报告/化验单的**局部照片**(常常只有一两行指标)。
|
||||
/// C2「重新解读」/ 拍照整份识别走这条:主模型 Gemma 4 E2B 无视觉权重(e2b 检查点仅文本,
|
||||
/// 视觉在 gemma4_unified 12B),故先 Vision OCR 出纯文本,再交文本 LLM 一次抽出报告级 meta + 全部指标。
|
||||
/// 与 reportExtraction(多模态)同 schema,但去掉 source_page / source_box —— OCR 文本没有版面坐标,
|
||||
/// 让模型编框会污染原图证据高亮,统一不输出。
|
||||
static func reportExtractionFromText(_ ocrText: String, today: Date = .now) -> String {
|
||||
let f = DateFormatter()
|
||||
f.locale = Locale(identifier: "en_US_POSIX")
|
||||
f.dateFormat = "yyyy-MM-dd"
|
||||
let todayStr = f.string(from: today)
|
||||
return reportExtractionFromTextTemplate
|
||||
.replacingOccurrences(of: "{{TODAY}}", with: todayStr)
|
||||
.replacingOccurrences(of: "{{OCR_TEXT}}", with: clipOCR(ocrText, limit: 2200))
|
||||
}
|
||||
|
||||
private static let reportExtractionFromTextTemplate: String = #"""
|
||||
你是医学体检/化验报告识别助手。下面是对一份报告做 OCR 得到的纯文本,可能有错字、错位、噪声或换行混乱。
|
||||
请从中提取报告的元信息与所有指标,只输出一段合法 JSON,不要解释、不要 markdown 围栏、不要任何前后缀文字。
|
||||
|
||||
今天的日期是 {{TODAY}}。
|
||||
|
||||
JSON schema(严格):
|
||||
{
|
||||
"title": string,
|
||||
"type": "checkup" | "lab" | "imaging" | "prescription" | "other",
|
||||
"report_date": "YYYY-MM-DD",
|
||||
"institution": string,
|
||||
"page_count": number,
|
||||
"summary": string,
|
||||
"indicators": [
|
||||
{
|
||||
"name": string,
|
||||
"value": string,
|
||||
"unit": string,
|
||||
"range": string,
|
||||
"status": "high" | "low" | "normal"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
规则:
|
||||
- status 优先看箭头/标记(↑/H/偏高 → "high",↓/L/偏低 → "low");没有标记时用 value 与 range 比较;都没有 → "normal"。
|
||||
- range 保留原文(如 "< 3.40"、"3.9 - 6.1"、"0 - 5");OCR 把破折号写成 "--" / "~" 都归一成 " - ";没有参考范围就填 ""。
|
||||
- 凡是「指标名 + 明确数值」可读的都要提取——**没有参考范围不是跳过的理由**,结论页叙述式文字(如「总胆红素: 23.0(μmol/L)↑」)同样提取。只有数值本身是乱码无法判断才跳过,绝不发明指标,同一指标只输出一次。
|
||||
- title / institution / summary 读不出就填 "";report_date 挑「报告/检查/采样日期」其一统一成 YYYY-MM-DD,只有年月就补 -01,实在读不出填上面给出的「今天」({{TODAY}})。
|
||||
- type:化验单→"lab";体检套餐→"checkup";影像(B超/CT/X光/MRI)→"imaging";处方→"prescription";拿不准→"other"。
|
||||
- summary 用一句话概括整体(如「血脂偏高,其余正常」);页眉、医生签名、栏目标题、OCR 噪声一律忽略。
|
||||
|
||||
示例 OCR 文本:
|
||||
协和医院体检中心 健康体检报告 体检日期:2026-04-12 低密度脂蛋白 3.84 mmol/L <3.40 ↑ 空腹血糖 5.2 mmol/L 3.9-6.1
|
||||
对应输出:
|
||||
{"title":"健康体检报告","type":"checkup","report_date":"2026-04-12","institution":"协和医院体检中心","page_count":1,"summary":"血脂偏高,血糖正常","indicators":[{"name":"低密度脂蛋白","value":"3.84","unit":"mmol/L","range":"< 3.40","status":"high"},{"name":"空腹血糖","value":"5.2","unit":"mmol/L","range":"3.9 - 6.1","status":"normal"}]}
|
||||
|
||||
现在请解析下面这段 OCR 文本,只输出 JSON。
|
||||
|
||||
OCR 文本:
|
||||
{{OCR_TEXT}}
|
||||
"""#
|
||||
|
||||
// MARK: - 报告归档 · 轻量 meta(只抽日期/机构/类型/标题,不识别指标)
|
||||
|
||||
/// 报告归档新链路:只保存原图,**不逐项识别指标**(逐项多模态识别在 2B 上易 OOM / 卡死)。
|
||||
/// 用 Vision OCR 出纯文本后,交文本 LLM 只抽报告级 meta —— 输出极小(~50 token),快且稳。
|
||||
/// 识别不到就留空,UI 用占位(今天 / 空机构),用户可手填。
|
||||
static func reportMetaFromText(_ ocrText: String, today: Date = .now) -> String {
|
||||
let f = DateFormatter()
|
||||
f.locale = Locale(identifier: "en_US_POSIX")
|
||||
f.dateFormat = "yyyy-MM-dd"
|
||||
let todayStr = f.string(from: today)
|
||||
return reportMetaTemplate
|
||||
.replacingOccurrences(of: "{{TODAY}}", with: todayStr)
|
||||
.replacingOccurrences(of: "{{OCR_TEXT}}", with: clipOCR(ocrText, limit: 1500))
|
||||
}
|
||||
|
||||
private static let reportMetaTemplate: String = #"""
|
||||
你是体检/化验报告归档助手。下面是对一份报告做 OCR 得到的纯文本,可能有错字、错位、噪声。
|
||||
请只提取这份报告的「元信息」,**不要提取任何具体指标/数值**。只输出一段合法 JSON,不要解释、不要 markdown 围栏、不要任何前后缀文字。
|
||||
|
||||
今天的日期是 {{TODAY}}。
|
||||
|
||||
JSON schema(严格):
|
||||
{
|
||||
"title": string, // 报告抬头,如 "春季年度体检";读不出就填 ""
|
||||
"type": "checkup" | "lab" | "imaging" | "prescription" | "other",
|
||||
"report_date": "YYYY-MM-DD", // 报告/采样/体检日期;实在读不出就填 ""
|
||||
"institution": string // 医院/体检机构名;读不出就填 ""
|
||||
}
|
||||
|
||||
规则:
|
||||
- 只输出上面 4 个字段,绝不输出 indicators / 数值 / 参考范围。
|
||||
- type:化验单→"lab";体检套餐→"checkup";影像(B超/CT/X光/MRI)→"imaging";处方→"prescription";拿不准→"other"。
|
||||
- 日期挑「报告日期 / 检查日期 / 采样日期」其一,统一成 YYYY-MM-DD;只有年月就补 -01;读不出填 ""。
|
||||
- institution 取医院/体检中心全称,去掉「检验科/报告单」等栏目词;读不出填 ""。
|
||||
- 不要编造;读不出的字段填 ""。
|
||||
|
||||
示例 OCR 文本:
|
||||
协和医院体检中心 健康体检报告 姓名:张三 体检日期:2026-04-12 低密度脂蛋白 3.84 ↑ ...
|
||||
输出:
|
||||
{"title":"健康体检报告","type":"checkup","report_date":"2026-04-12","institution":"协和医院体检中心"}
|
||||
|
||||
现在请解析下面这段 OCR 文本,只输出 JSON。/no_think
|
||||
|
||||
OCR 文本:
|
||||
{{OCR_TEXT}}
|
||||
"""#
|
||||
|
||||
// MARK: - 局部小框识别(指标速记)
|
||||
|
||||
/// 指标速记专用:输入是报告/化验单的**局部照片**(常常只有一两行指标)。
|
||||
/// 只要 indicators 数组,不要报告标题/机构/日期等元信息 —— 这条路径只存数值,不建 Report。
|
||||
static func regionExtraction(today: Date = .now) -> String {
|
||||
let f = DateFormatter()
|
||||
|
||||
@@ -2,6 +2,10 @@ import Foundation
|
||||
import MLX
|
||||
import MLXVLM
|
||||
import MLXLMCommon
|
||||
// 3.x HF Hub/分词器栈拆成 opt-in 宏(见 LLMSession 注释)。
|
||||
import MLXHuggingFace
|
||||
import HuggingFace
|
||||
import Tokenizers
|
||||
|
||||
/// 封装 MLX VL 模型(Qwen3-VL)的图像 → 文本推理。
|
||||
/// 与 LLMSession 同款 actor 隔离,串行化由上游 AIRuntime 统一保证。
|
||||
@@ -25,8 +29,11 @@ actor VLSession {
|
||||
/// 从本地目录加载 VL 模型(包含 config.json + weights + tokenizer + processor)。
|
||||
static func load(folderURL: URL) async throws -> VLSession {
|
||||
let configuration = ModelConfiguration(directory: folderURL)
|
||||
// 3.31.4:同 LLMSession,显式传 Downloader(占位)+ HF 分词器加载器。
|
||||
let container = try await withDeviceOverride {
|
||||
try await VLMModelFactory.shared.loadContainer(
|
||||
from: #hubDownloader(),
|
||||
using: #huggingFaceTokenizerLoader(),
|
||||
configuration: configuration
|
||||
)
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ struct KangkangApp: App {
|
||||
CustomMonitorMetric.self,
|
||||
HealthExport.self,
|
||||
CustomReminder.self,
|
||||
Medication.self,
|
||||
])
|
||||
let config = ModelConfiguration(schema: schema, isStoredInMemoryOnly: false)
|
||||
// 建库后给 store 文件补 .completeUnlessOpen 保护(§6),两条创建路径共用。
|
||||
@@ -102,6 +103,9 @@ struct KangkangApp: App {
|
||||
// 语言 / 字体档位切换 → 整树重建,即时生效(固定字号经 tjScaled 读新倍率)。
|
||||
.id("\(lang.current.rawValue)-\(fontScale.scale.rawValue)")
|
||||
}
|
||||
// 设计系统是纯浅色(背景恒为 sand)。锁定 light:否则系统深色模式下,
|
||||
// 未显式设色的 Text/TextField 走 .primary 变白,在浅背景上看不见(如日记输入框)。
|
||||
.preferredColorScheme(.light)
|
||||
}
|
||||
.modelContainer(sharedModelContainer)
|
||||
}
|
||||
|
||||
@@ -118,6 +118,48 @@ extension String {
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - AI 输出语言(供 Service/Prompt 层在非 MainActor 上下文读取当前界面语言)
|
||||
|
||||
/// 当前界面语言解析出的「输出语言」,给本地 LLM 的 prompt 用:
|
||||
/// 让 AI 生成的追问 / 解读 / 摘要跟随用户所选界面语言,而不是永远中文。
|
||||
/// 判定只看 `appLocLocale`(LanguageManager 切换时刷新的 nonisolated 快照);
|
||||
/// `system` 跟随系统时,按系统首选语言归一到最接近的受支持语言,兜底中文。
|
||||
nonisolated enum AIOutputLanguage: String {
|
||||
case zhHans, en, ja, ko
|
||||
|
||||
/// 从当前 appLocLocale 解析。system(autoupdatingCurrent)时读系统语言码。
|
||||
static var current: AIOutputLanguage {
|
||||
let code = appLocLocale.language.languageCode?.identifier ?? "zh"
|
||||
switch code {
|
||||
case "en": return .en
|
||||
case "ja": return .ja
|
||||
case "ko": return .ko
|
||||
default: return .zhHans // zh 及其它未支持语言统一回中文(few-shot 也是中文)
|
||||
}
|
||||
}
|
||||
|
||||
/// prompt 里对模型自称的目标语言名(用该语言自己的说法,减少模型误判)。
|
||||
var promptName: String {
|
||||
switch self {
|
||||
case .zhHans: return "简体中文"
|
||||
case .en: return "English"
|
||||
case .ja: return "日本語"
|
||||
case .ko: return "한국어"
|
||||
}
|
||||
}
|
||||
|
||||
/// 追加到 prompt 末尾的「用某语言输出」硬指令。中文时返回空串(few-shot 已是中文,不必额外声明)。
|
||||
/// 用于「生成给用户看的内容」(追问 / 解读 / 摘要),让输出语言=界面语言。
|
||||
var outputDirective: String {
|
||||
self == .zhHans ? "" : "\n\n重要:请全程用\(promptName)输出所有面向用户的文字(JSON 的字段值也用\(promptName)),不要用中文。"
|
||||
}
|
||||
}
|
||||
|
||||
/// 便捷:当前界面语言的「用某语言输出」指令(见 `AIOutputLanguage.outputDirective`)。
|
||||
nonisolated func aiOutputLanguageDirective() -> String {
|
||||
AIOutputLanguage.current.outputDirective
|
||||
}
|
||||
|
||||
// MARK: - Bundle 运行时重定向(供 Text / NSLocalizedString 双保险)
|
||||
|
||||
/// 关联对象 key(需稳定地址,文件级全局即可)。
|
||||
|
||||
36
康康/DesignSystem/AIDisclaimer.swift
Normal file
36
康康/DesignSystem/AIDisclaimer.swift
Normal file
@@ -0,0 +1,36 @@
|
||||
import SwiftUI
|
||||
|
||||
/// 全 App 统一的 AI 免责声明(上架合规:医疗类内容必须声明不做诊断)。
|
||||
/// 任何展示 AI 生成的归纳/解读/建议文本的地方,都应在结果末尾附上 `AIDisclaimerFooter`;
|
||||
/// 离开 App 的文本(复制/分享)用 `AIDisclaimer.appended(to:)` 把声明带上。
|
||||
enum AIDisclaimer {
|
||||
/// 面向用户展示的完整声明文本(随「我的 · 语言」本地化)。
|
||||
/// 注意:必须是**单个** `String(appLoc:)` key(勿再用字符串拼接),否则无法整句进 String Catalog 翻译;
|
||||
/// 且展示处用 `Text(AIDisclaimer.text)`(传 String 变量),SwiftUI 不会再做本地化 —— 这里先本地化好。
|
||||
static var text: String {
|
||||
String(appLoc: "本内容由本机本地 AI 依据你录入的健康记录自动归纳整理,仅供个人健康管理与就医沟通参考,不构成医学诊断、治疗建议或专业医疗意见;具体健康问题请咨询执业医师。")
|
||||
}
|
||||
|
||||
/// 复制/分享时把声明追加到正文末尾(分隔线 + 声明),让文本离开 App 也带着免责。
|
||||
static func appended(to body: String) -> String {
|
||||
let trimmed = body.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
return "\(trimmed)\n\n———\n\(text)"
|
||||
}
|
||||
}
|
||||
|
||||
/// AI 结果下方的免责声明脚注:小字、弱色、信息图标。放在任何 AI 生成文本块末尾。
|
||||
struct AIDisclaimerFooter: View {
|
||||
var body: some View {
|
||||
HStack(alignment: .top, spacing: 6) {
|
||||
Image(systemName: "info.circle")
|
||||
.font(.tjScaled( 10))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Text(AIDisclaimer.text)
|
||||
.font(.tjScaled( 10))
|
||||
.lineSpacing(2)
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
}
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
}
|
||||
}
|
||||
52
康康/DesignSystem/AIFlowBar.swift
Normal file
52
康康/DesignSystem/AIFlowBar.swift
Normal file
@@ -0,0 +1,52 @@
|
||||
import SwiftUI
|
||||
|
||||
/// Apple Intelligence 式多彩流光线:蓝→紫→粉→橙→青,横向无缝循环流动。
|
||||
/// 全 App「AI 计算中」时刻的统一视觉点缀(日记 AI 辅助、身体档案报告生成/检索等待等)。
|
||||
///
|
||||
/// 注意:这条线的颜色是刻意走出 `Tj.Palette` 单色系统的 AI 高光点缀(应产品要求的
|
||||
/// Apple 风格),仅此组件如此;其余 UI 仍严格守 §9 单色 token。
|
||||
///
|
||||
/// 驱动方式用 `TimelineView(.animation)` 而非 `.onAppear` + `repeatForever`:这条线出现的
|
||||
/// 场景(流式回答、tok/s 每 0.5s 刷新等)父视图都在高频重绘,隐式 `repeatForever` 动画会被
|
||||
/// 反复打断/重置 → 看起来「几乎不动」。TimelineView 按显示刷新率直接从时间算偏移,与父视图
|
||||
/// 重绘完全解耦,任何场景下都匀速流动。
|
||||
struct AIFlowBar: View {
|
||||
var height: CGFloat = 3
|
||||
/// 颜色平移一整圈(一个完整色序)的秒数,越小越快。
|
||||
var cycle: Double = 0.6
|
||||
|
||||
private static let base: [Color] = [
|
||||
Color(red: 0.35, green: 0.47, blue: 0.98), // 蓝
|
||||
Color(red: 0.62, green: 0.36, blue: 0.92), // 紫
|
||||
Color(red: 0.96, green: 0.40, blue: 0.62), // 粉
|
||||
Color(red: 1.00, green: 0.55, blue: 0.30), // 橙
|
||||
Color(red: 0.30, green: 0.80, blue: 0.92), // 青
|
||||
]
|
||||
|
||||
/// 色序重复两遍并以首色收尾(共 11 个 stop,均匀分布):一个色周期恰好占据画布宽度,
|
||||
/// 平移一个画布宽度后首尾同色,循环完全无缝。
|
||||
private static let gradient: Gradient = {
|
||||
let colors = base + base + [base[0]]
|
||||
let last = CGFloat(colors.count - 1)
|
||||
return Gradient(stops: colors.enumerated().map { i, c in
|
||||
Gradient.Stop(color: c, location: CGFloat(i) / last)
|
||||
})
|
||||
}()
|
||||
|
||||
var body: some View {
|
||||
TimelineView(.animation) { timeline in
|
||||
GeometryReader { geo in
|
||||
let w = geo.size.width
|
||||
let t = timeline.date.timeIntervalSinceReferenceDate
|
||||
let progress = CGFloat(t.truncatingRemainder(dividingBy: cycle) / cycle)
|
||||
Capsule()
|
||||
.fill(LinearGradient(gradient: Self.gradient,
|
||||
startPoint: .leading, endPoint: .trailing))
|
||||
.frame(width: w * 2)
|
||||
.offset(x: -w * progress)
|
||||
}
|
||||
}
|
||||
.frame(height: height)
|
||||
.clipShape(Capsule())
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,12 @@ enum Tj {
|
||||
static let leaf = Color(red: 0.180, green: 0.357, blue: 0.518)
|
||||
static let leafSoft = Color(red: 0.867, green: 0.910, blue: 0.941)
|
||||
static let darkBg = Color(red: 0.051, green: 0.063, blue: 0.059)
|
||||
// 数据折线主色:低饱和「桉叶 / 鼠尾草」青绿,沉静、契合健康、在米色背景上有高级感。
|
||||
// 替代原本近黑的 ink 折线;异常点仍用 brick 红点点缀,冷线 + 暖点对比清楚。
|
||||
static let teal = Color(red: 0.337, green: 0.529, blue: 0.494)
|
||||
static let tealSoft = Color(red: 0.808, green: 0.878, blue: 0.863)
|
||||
// 统一柔和阴影色:暖灰褐,低饱和。代替近黑的 ink 阴影,投影更柔、不发脏。
|
||||
static let shadow = Color(red: 0.376, green: 0.345, blue: 0.298)
|
||||
}
|
||||
|
||||
enum Radius {
|
||||
@@ -64,7 +70,7 @@ extension View {
|
||||
RoundedRectangle(cornerRadius: radius, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: bordered ? 1 : 0)
|
||||
)
|
||||
.shadow(color: bordered ? .clear : Color(red: 0.196, green: 0.157, blue: 0.098).opacity(0.05),
|
||||
.shadow(color: bordered ? .clear : Tj.Palette.shadow.opacity(0.06),
|
||||
radius: 2, x: 0, y: 1)
|
||||
}
|
||||
}
|
||||
|
||||
48
康康/DesignSystem/VaultImage.swift
Normal file
48
康康/DesignSystem/VaultImage.swift
Normal file
@@ -0,0 +1,48 @@
|
||||
import SwiftUI
|
||||
|
||||
/// 从加密 Vault 异步加载并降采样显示原图的通用组件。
|
||||
///
|
||||
/// 替代「在 body 里 `try? FileVault.shared.loadImage(...)` 同步读盘 + 全量解码」的旧写法,
|
||||
/// 解决两个真实问题:
|
||||
/// 1. **OOM**:全分辨率位图(4000×3000 ≈ 48MB)进内存,翻几页就触发 jetsam。这里按 `maxPixel`
|
||||
/// 降采样,缩略图几百 KB,全屏图几 MB。
|
||||
/// 2. **主线程卡顿**:读盘 + JPEG 解码在主线程会掉帧。这里放到后台线程,主线程只拿结果绘制。
|
||||
///
|
||||
/// 区分「加载中」与「读取失败」两态:加载中显示中性占位,只有真正失败才显示「原图无法读取」,
|
||||
/// 不会一打开就闪一下吓人的错误文案。`content` 拿到 `UIImage`(而非 `Image`),
|
||||
/// 方便需要 `image.size` 的调用方(如证据高亮 overlay)按真实宽高比定位。
|
||||
struct VaultImage<Content: View, Placeholder: View>: View {
|
||||
let relativePath: String
|
||||
/// 降采样目标最大边(像素)。缩略图给 ~400,全屏查看器给 ~2000。
|
||||
var maxPixel: CGFloat = 1024
|
||||
|
||||
@ViewBuilder var content: (UIImage) -> Content
|
||||
/// 占位回调,`isLoading == true` 表示仍在加载,`false` 表示加载完成但失败。
|
||||
@ViewBuilder var placeholder: (_ isLoading: Bool) -> Placeholder
|
||||
|
||||
@State private var image: UIImage?
|
||||
@State private var loading = true
|
||||
|
||||
var body: some View {
|
||||
Group {
|
||||
if let image {
|
||||
content(image)
|
||||
} else {
|
||||
placeholder(loading)
|
||||
}
|
||||
}
|
||||
// id 变了(TabView 翻到新页 / 行复用换 asset / 同一 path 改目标尺寸)就重新加载;
|
||||
// 把 maxPixel 纳入 id(与 FileVault 缓存 key 同构),避免同 path 切尺寸时不刷新显示旧图。
|
||||
.task(id: "\(relativePath)@\(Int(maxPixel))") {
|
||||
loading = true
|
||||
let path = relativePath
|
||||
let mp = maxPixel
|
||||
let loaded = await Task.detached(priority: .userInitiated) {
|
||||
try? FileVault.shared.loadDownsampledImage(relativePath: path, maxPixelSize: mp)
|
||||
}.value
|
||||
guard !Task.isCancelled else { return }
|
||||
image = loaded
|
||||
loading = false
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -23,66 +23,92 @@ struct ArchiveListView: View {
|
||||
@Query(sort: \MetricReminder.updatedAt, order: .reverse)
|
||||
private var metricReminders: [MetricReminder]
|
||||
|
||||
@Query(sort: \Medication.updatedAt, order: .reverse)
|
||||
private var medications: [Medication]
|
||||
|
||||
/// 记录页内的 push 目的地。用单个 `navigationDestination(item:)` 驱动——
|
||||
/// 多个 `navigationDestination(isPresented:)` 并存时 SwiftUI 行为未定义(会误触发)。
|
||||
private enum Route: Hashable { case exports, reminders }
|
||||
private enum Route: Hashable { case exports, reminders, medicationLibrary }
|
||||
|
||||
@State private var filter: TimelineKind? = nil
|
||||
@State private var endingSymptom: Symptom?
|
||||
|
||||
/// 默认无参;从首页「我的报告档案」进入时传 `.report` 预选分类 chip。
|
||||
/// 仅在视图创建时生效(RootView 切 tab 会重建 ArchiveListView)。
|
||||
init(initialFilter: TimelineKind? = nil) {
|
||||
_filter = State(initialValue: initialFilter)
|
||||
}
|
||||
@State private var selectedEntry: TimelineEntry?
|
||||
@State private var selectedGroup: IndicatorGroup?
|
||||
@State private var route: Route?
|
||||
|
||||
/// 顶部搜索:点放大镜展开搜索框,按条目标题(指标/报告/症状/日记名)实时过滤,与分类 chip 叠加。
|
||||
@State private var searching = false
|
||||
@State private var query = ""
|
||||
|
||||
@MainActor
|
||||
private var allEntries: [TimelineEntry] {
|
||||
let mapped =
|
||||
TimelineEntry.from(indicators: indicators) +
|
||||
TimelineEntry.aggregatedIndicators(indicators) +
|
||||
reports.map(TimelineEntry.from(report:)) +
|
||||
diaries.map(TimelineEntry.from(diary:)) +
|
||||
symptoms.map(TimelineEntry.from(symptom:))
|
||||
let filtered = filter.map { kind in mapped.filter { $0.kind == kind } } ?? mapped
|
||||
return filtered.sorted { $0.date > $1.date }
|
||||
let byKind = filter.map { kind in mapped.filter { $0.kind == kind } } ?? mapped
|
||||
let q = query.trimmingCharacters(in: .whitespaces)
|
||||
let byQuery = q.isEmpty ? byKind : byKind.filter { $0.title.localizedCaseInsensitiveContains(q) }
|
||||
return byQuery.sorted { $0.date > $1.date }
|
||||
}
|
||||
|
||||
private var grouped: [(section: DateSection, items: [TimelineEntry])] {
|
||||
TimelineGrouping.group(allEntries)
|
||||
}
|
||||
|
||||
private var totalCount: Int { allEntries.count }
|
||||
|
||||
var body: some View {
|
||||
NavigationStack {
|
||||
content
|
||||
.navigationDestination(item: $route) { route in
|
||||
switch route {
|
||||
case .exports: HealthExportListView()
|
||||
case .reminders: RemindersListView()
|
||||
case .exports: HealthExportListView()
|
||||
case .reminders: RemindersListView()
|
||||
case .medicationLibrary: MedicationLibraryView()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var content: some View {
|
||||
VStack(alignment: .leading, spacing: 0) {
|
||||
header
|
||||
// 聚合(含血压配对 O(m²))+ 分类/搜索过滤在一次 body 内只算一次。原先 .isEmpty、分组、
|
||||
// 计数各调一遍 allEntries,等于全表聚合三次;搜索时每次按键都翻三倍,这里收敛成一次。
|
||||
let entries = allEntries
|
||||
let groups = TimelineGrouping.group(entries)
|
||||
return VStack(alignment: .leading, spacing: 0) {
|
||||
header(total: entries.count)
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.top, 8)
|
||||
.padding(.bottom, 14)
|
||||
|
||||
if reminderTotal > 0 {
|
||||
reminderBoard
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.bottom, 10)
|
||||
}
|
||||
|
||||
// 药品库入口:始终显示——它是「管理常用药」的浏览/管理目的地,空库时也要能找到来添加。
|
||||
medicationBoard
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.bottom, 14)
|
||||
|
||||
filterChips
|
||||
.padding(.bottom, searching ? 10 : 14)
|
||||
|
||||
if searching {
|
||||
searchField
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.bottom, 14)
|
||||
}
|
||||
|
||||
filterChips
|
||||
.padding(.bottom, 14)
|
||||
|
||||
if allEntries.isEmpty {
|
||||
if entries.isEmpty {
|
||||
emptyState
|
||||
} else {
|
||||
ScrollView(showsIndicators: false) {
|
||||
LazyVStack(alignment: .leading, spacing: 18, pinnedViews: [.sectionHeaders]) {
|
||||
ForEach(grouped, id: \.section) { group in
|
||||
ForEach(groups, id: \.section) { group in
|
||||
Section {
|
||||
VStack(spacing: 10) {
|
||||
ForEach(group.items) { entry in
|
||||
@@ -109,6 +135,9 @@ struct ArchiveListView: View {
|
||||
TimelineEntryDetailView(detail: d)
|
||||
}
|
||||
}
|
||||
.sheet(item: $selectedGroup) { group in
|
||||
IndicatorSeriesDetailView(group: group)
|
||||
}
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
@@ -123,9 +152,14 @@ struct ArchiveListView: View {
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
} else {
|
||||
// 其余条目(报告/指标/日记/已结束症状):点 → 只读详情
|
||||
// 其余条目:指标 → 同类聚合详情(横向翻页 + 趋势);报告/日记/已结束症状 → 只读详情
|
||||
Button {
|
||||
if detail(for: entry) != nil { selectedEntry = entry }
|
||||
guard let d = detail(for: entry) else { return }
|
||||
switch d {
|
||||
case .indicator(let i): selectedGroup = IndicatorGroup.of(i)
|
||||
case .bloodPressure(let sys, _): selectedGroup = IndicatorGroup.of(sys)
|
||||
default: selectedEntry = entry
|
||||
}
|
||||
} label: {
|
||||
TimelineRow(entry: entry)
|
||||
}
|
||||
@@ -140,12 +174,12 @@ struct ArchiveListView: View {
|
||||
diaries: diaries, symptoms: symptoms)
|
||||
}
|
||||
|
||||
private var header: some View {
|
||||
private func header(total: Int) -> some View {
|
||||
HStack(alignment: .lastTextBaseline) {
|
||||
Text("记录")
|
||||
.font(.tjTitle(26))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
Text(totalCount == 0 ? "" : String(appLoc: "\(totalCount) 条"))
|
||||
Text(total == 0 ? "" : String(appLoc: "\(total) 条"))
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Spacer()
|
||||
@@ -164,9 +198,57 @@ struct ArchiveListView: View {
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
searchToggle
|
||||
}
|
||||
}
|
||||
|
||||
private var searchToggle: some View {
|
||||
Button {
|
||||
withAnimation(.easeInOut(duration: 0.18)) {
|
||||
searching.toggle()
|
||||
if !searching { query = "" }
|
||||
}
|
||||
} label: {
|
||||
Image(systemName: searching ? "xmark" : "magnifyingglass")
|
||||
.font(.tjScaled( 14, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.frame(width: 32, height: 32)
|
||||
.background(Circle().fill(Tj.Palette.sand2))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.accessibilityLabel(searching ? String(appLoc: "关闭搜索") : String(appLoc: "搜索记录"))
|
||||
}
|
||||
|
||||
private var searchField: some View {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "magnifyingglass")
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
TextField(String(appLoc: "搜索指标 / 报告 / 症状名"), text: $query)
|
||||
.textInputAutocapitalization(.never)
|
||||
.autocorrectionDisabled()
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.tint(Tj.Palette.ink)
|
||||
if !query.isEmpty {
|
||||
Button { query = "" } label: {
|
||||
Image(systemName: "xmark.circle.fill")
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
}
|
||||
.padding(.horizontal, 12)
|
||||
.padding(.vertical, 10)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1)
|
||||
)
|
||||
}
|
||||
|
||||
// MARK: - 提醒任务汇总卡
|
||||
|
||||
/// 两类提醒(自由 + 指标记录)合计,含已关闭。
|
||||
@@ -232,13 +314,73 @@ struct ArchiveListView: View {
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
// MARK: - 药品库入口卡
|
||||
|
||||
/// 主标题:空库「药品库」,有药「药品库 · N 种常用药」。
|
||||
private var medicationCountLabel: String {
|
||||
medications.isEmpty
|
||||
? String(appLoc: "药品库")
|
||||
: String(appLoc: "药品库 · \(medications.count) 种常用药")
|
||||
}
|
||||
|
||||
/// 副标题:空库给引导文案;有药取前 3 个药名预览(药名是用户数据,不本地化)。
|
||||
private var medicationPreviewLine: String {
|
||||
if medications.isEmpty { return String(appLoc: "拍药盒或手动添加常用药") }
|
||||
let names = medications.prefix(3).map(\.name).joined(separator: " · ")
|
||||
return medications.count > 3 ? names + " …" : names
|
||||
}
|
||||
|
||||
/// 点击进药品库(MedicationLibraryView,push 形态)统一管理;卡片本身只展示。
|
||||
private var medicationBoard: some View {
|
||||
Button { route = .medicationLibrary } label: {
|
||||
HStack(spacing: 12) {
|
||||
ZStack {
|
||||
Circle().fill(medications.isEmpty ? Tj.Palette.sand2 : Tj.Palette.leafSoft)
|
||||
Image(systemName: "pills.fill")
|
||||
.font(.tjScaled( 16))
|
||||
.foregroundStyle(medications.isEmpty ? Tj.Palette.text3 : Tj.Palette.ink)
|
||||
}
|
||||
.frame(width: 36, height: 36)
|
||||
|
||||
VStack(alignment: .leading, spacing: 2) {
|
||||
Text(medicationCountLabel)
|
||||
.font(.tjScaled( 15, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.lineLimit(1)
|
||||
Text(medicationPreviewLine)
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.lineLimit(1)
|
||||
}
|
||||
|
||||
Spacer(minLength: 0)
|
||||
|
||||
Image(systemName: "chevron.right")
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
.padding(14)
|
||||
.contentShape(Rectangle())
|
||||
.tjCard()
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
/// 分类过滤条(2026-06-28 重组):把原先扁平的一排标签按语义归三组——
|
||||
/// 自述(日记/症状/问诊)· 检查(指标/报告)· 用药——组间加竖线分隔、每枚标签带图标 + 类别色,
|
||||
/// 让「一堆标签」读成有结构的三块。过滤逻辑仍是单选某个 TimelineKind(选中再点取消)。
|
||||
private var filterChips: some View {
|
||||
ScrollView(.horizontal, showsIndicators: false) {
|
||||
HStack(spacing: 8) {
|
||||
chip(label: String(appLoc: "全部"), selected: filter == nil) { filter = nil }
|
||||
ForEach(TimelineKind.allCases) { kind in
|
||||
chip(label: kind.label, selected: filter == kind) {
|
||||
filter = filter == kind ? nil : kind
|
||||
allChip
|
||||
ForEach(TimelineKind.Group.allCases) { group in
|
||||
groupSeparator
|
||||
// 单成员组(用药)不另标组名:组名与唯一标签重复,直接出标签即可。
|
||||
if group.kinds.count > 1 {
|
||||
groupCaption(group.caption)
|
||||
}
|
||||
ForEach(group.kinds) { kind in
|
||||
kindChip(kind)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -246,23 +388,53 @@ struct ArchiveListView: View {
|
||||
}
|
||||
}
|
||||
|
||||
private func chip(label: String, selected: Bool, action: @escaping () -> Void) -> some View {
|
||||
Button(action: action) {
|
||||
Text(label)
|
||||
private var allChip: some View {
|
||||
let selected = filter == nil
|
||||
return Button { filter = nil } label: {
|
||||
Text(String(appLoc: "全部"))
|
||||
.font(.tjScaled( 13, weight: selected ? .semibold : .regular))
|
||||
.foregroundStyle(selected ? Tj.Palette.paper : Tj.Palette.text)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 8)
|
||||
.background(
|
||||
Capsule().fill(selected ? Tj.Palette.ink : Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
Capsule().strokeBorder(Tj.Palette.line, lineWidth: selected ? 0 : 1)
|
||||
)
|
||||
.background(Capsule().fill(selected ? Tj.Palette.ink : Tj.Palette.paper))
|
||||
.overlay(Capsule().strokeBorder(selected ? Color.clear : Tj.Palette.line, lineWidth: 1))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
/// 单枚分类标签:图标 + 名称,用该类别的 accent 着色;选中填充 accent。
|
||||
private func kindChip(_ kind: TimelineKind) -> some View {
|
||||
let selected = filter == kind
|
||||
return Button { filter = selected ? nil : kind } label: {
|
||||
HStack(spacing: 5) {
|
||||
Image(systemName: kind.icon)
|
||||
.font(.tjScaled( 11, weight: .semibold))
|
||||
.foregroundStyle(selected ? Tj.Palette.paper : kind.accent)
|
||||
Text(kind.label)
|
||||
.font(.tjScaled( 13, weight: selected ? .semibold : .regular))
|
||||
.foregroundStyle(selected ? Tj.Palette.paper : Tj.Palette.text)
|
||||
}
|
||||
.padding(.horizontal, 12)
|
||||
.padding(.vertical, 8)
|
||||
.background(Capsule().fill(selected ? kind.accent : Tj.Palette.paper))
|
||||
.overlay(Capsule().strokeBorder(selected ? Color.clear : Tj.Palette.line, lineWidth: 1))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
private var groupSeparator: some View {
|
||||
RoundedRectangle(cornerRadius: 1, style: .continuous)
|
||||
.fill(Tj.Palette.lineSoft)
|
||||
.frame(width: 1, height: 18)
|
||||
.padding(.horizontal, 2)
|
||||
}
|
||||
|
||||
private func groupCaption(_ text: String) -> some View {
|
||||
Text(text)
|
||||
.font(.tjScaled( 11, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
|
||||
private func sectionHeader(_ section: DateSection, count: Int) -> some View {
|
||||
HStack {
|
||||
Text(section.label)
|
||||
@@ -282,13 +454,19 @@ struct ArchiveListView: View {
|
||||
}
|
||||
|
||||
private var emptyState: some View {
|
||||
VStack(spacing: 14) {
|
||||
let q = query.trimmingCharacters(in: .whitespaces)
|
||||
let isSearchMiss = !q.isEmpty
|
||||
return VStack(spacing: 14) {
|
||||
Spacer()
|
||||
TjPlaceholder(label: String(appLoc: "还没有任何记录\n点底部 + 号开始"))
|
||||
TjPlaceholder(label: isSearchMiss
|
||||
? String(appLoc: "没有匹配「\(q)」的记录")
|
||||
: String(appLoc: "还没有任何记录\n点底部 + 号开始"))
|
||||
.frame(width: 240, height: 140)
|
||||
Text(filter == nil ? String(appLoc: "记录会按时间归类显示") : String(appLoc: "这个类别下没有记录"))
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
if !isSearchMiss {
|
||||
Text(filter == nil ? String(appLoc: "记录会按时间归类显示") : String(appLoc: "这个类别下没有记录"))
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
Spacer()
|
||||
}
|
||||
.frame(maxWidth: .infinity)
|
||||
|
||||
@@ -29,6 +29,8 @@ struct HealthExportDetailView: View {
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
|
||||
AIDisclaimerFooter()
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 16)
|
||||
@@ -117,7 +119,7 @@ struct HealthExportDetailView: View {
|
||||
}
|
||||
.buttonStyle(TjGhostButton(height: 44, fontSize: 13, horizontalPadding: 14))
|
||||
|
||||
ShareLink(item: export.content) {
|
||||
ShareLink(item: AIDisclaimer.appended(to: export.content)) {
|
||||
Label("分享", systemImage: "square.and.arrow.up")
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.tracking(1)
|
||||
@@ -149,7 +151,7 @@ struct HealthExportDetailView: View {
|
||||
}
|
||||
|
||||
private func copy() {
|
||||
UIPasteboard.general.string = export.content
|
||||
UIPasteboard.general.string = AIDisclaimer.appended(to: export.content)
|
||||
copiedFlash = true
|
||||
DispatchQueue.main.asyncAfter(deadline: .now() + 1.4) {
|
||||
copiedFlash = false
|
||||
@@ -175,9 +177,9 @@ struct HealthExportDetailView: View {
|
||||
# 就诊摘要 — 感冒就诊
|
||||
|
||||
## 主诉
|
||||
患者男,38 岁,感冒 3 天未愈。
|
||||
本人男,38 岁,感冒 3 天未愈。
|
||||
|
||||
## 患者背景
|
||||
## 本人背景
|
||||
- 高血压 2 年
|
||||
- 在服药:缬沙坦 80mg qd
|
||||
""",
|
||||
|
||||
@@ -117,11 +117,17 @@ struct HealthExportRow: View {
|
||||
.tjCard()
|
||||
}
|
||||
|
||||
static func relativeDate(_ d: Date) -> String {
|
||||
/// 复用单个 formatter:RelativeDateTimeFormatter 初始化较贵,列表每行每次重绘都 new 会累积开销。
|
||||
/// 用系统 Locale.current(与原实现一致),进程内不变,可安全缓存。
|
||||
private static let relativeFormatter: RelativeDateTimeFormatter = {
|
||||
let f = RelativeDateTimeFormatter()
|
||||
f.locale = Locale.current
|
||||
f.unitsStyle = .full
|
||||
return f.localizedString(for: d, relativeTo: .now)
|
||||
return f
|
||||
}()
|
||||
|
||||
static func relativeDate(_ d: Date) -> String {
|
||||
relativeFormatter.localizedString(for: d, relativeTo: .now)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -19,9 +19,17 @@ struct HealthExportSheet: View {
|
||||
@State private var error: Error?
|
||||
@State private var completed: Bool = false
|
||||
@State private var copiedFlash: Bool = false
|
||||
@State private var lastScrollAt: Date = .distantPast // 流式滚动节流时间戳
|
||||
@State private var answeringTurnID: UUID?
|
||||
@State private var retrieval: HealthExportService.RetrievalSummary?
|
||||
@State private var turnRetrievals: [UUID: HealthExportService.RetrievalSummary] = [:]
|
||||
@FocusState private var questionFocused: Bool
|
||||
|
||||
// 快捷问答
|
||||
@State private var promptStore = QuickPromptStore.shared
|
||||
@State private var showAddPrompt = false
|
||||
@State private var newPromptText = ""
|
||||
|
||||
init(initialPrompt: String = "") {
|
||||
self.initialPrompt = initialPrompt
|
||||
}
|
||||
@@ -33,10 +41,16 @@ struct HealthExportSheet: View {
|
||||
!isGeneratingReport &&
|
||||
!draftQuestion.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
|
||||
}
|
||||
/// 已有有效用户对话内容。
|
||||
private var hasUserContent: Bool {
|
||||
turns.contains(where: { $0.role == .user && !$0.text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty })
|
||||
}
|
||||
|
||||
/// 可生成报告:有对话内容,或输入框里有文字(允许跳过多轮对话直接生成)。
|
||||
private var canGenerateReport: Bool {
|
||||
!isAnswering &&
|
||||
!isGeneratingReport &&
|
||||
turns.contains(where: { $0.role == .user && !$0.text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty })
|
||||
(hasUserContent || !draftQuestion.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty)
|
||||
}
|
||||
|
||||
var body: some View {
|
||||
@@ -44,7 +58,7 @@ struct HealthExportSheet: View {
|
||||
header
|
||||
ScrollViewReader { proxy in
|
||||
ScrollView {
|
||||
VStack(alignment: .leading, spacing: 18) {
|
||||
LazyVStack(alignment: .leading, spacing: 18) {
|
||||
introSection
|
||||
|
||||
ForEach(turns) { turn in
|
||||
@@ -63,15 +77,15 @@ struct HealthExportSheet: View {
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 16)
|
||||
}
|
||||
.onChange(of: content) { _, _ in
|
||||
withAnimation(.easeOut(duration: 0.12)) {
|
||||
proxy.scrollTo("bottom", anchor: .bottom)
|
||||
}
|
||||
// 流式期间 content / turns 每 token 都变,逐次动画滚动会造成布局抖动;
|
||||
// 节流到 ~8Hz,并在生成完成时补一次滚动确保停在底部。
|
||||
.onChange(of: content) { _, _ in throttledScrollToBottom(proxy) }
|
||||
.onChange(of: turns) { _, _ in throttledScrollToBottom(proxy) }
|
||||
.onChange(of: completed) { _, done in
|
||||
if done { scrollToBottom(proxy) }
|
||||
}
|
||||
.onChange(of: turns) { _, _ in
|
||||
withAnimation(.easeOut(duration: 0.12)) {
|
||||
proxy.scrollTo("bottom", anchor: .bottom)
|
||||
}
|
||||
.onChange(of: answeringTurnID) { _, id in
|
||||
if id == nil { scrollToBottom(proxy) }
|
||||
}
|
||||
}
|
||||
if completed {
|
||||
@@ -88,6 +102,75 @@ struct HealthExportSheet: View {
|
||||
questionFocused = true
|
||||
}
|
||||
.onDisappear { task?.cancel() }
|
||||
.alert("添加快捷问答", isPresented: $showAddPrompt) {
|
||||
TextField("输入一句常用问题…", text: $newPromptText)
|
||||
Button("取消", role: .cancel) { newPromptText = "" }
|
||||
Button("添加") {
|
||||
promptStore.add(prompt: newPromptText)
|
||||
newPromptText = ""
|
||||
}
|
||||
} message: {
|
||||
Text("保存后点一下,就能把这句话填进输入框")
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 快捷问答
|
||||
|
||||
/// 内置 + 自定义快捷问答 chip 行;点 chip 填入输入框,末尾「＋ 自定义」追加,长按自定义删除。
|
||||
private var quickPromptRow: some View {
|
||||
ScrollView(.horizontal, showsIndicators: false) {
|
||||
HStack(spacing: 8) {
|
||||
ForEach(promptStore.all) { p in
|
||||
quickPromptChip(p)
|
||||
}
|
||||
addQuickPromptChip
|
||||
}
|
||||
.padding(.vertical, 1) // 给 chip 描边留出像素,避免被 ScrollView 裁切
|
||||
}
|
||||
}
|
||||
|
||||
private func quickPromptChip(_ p: QuickPrompt) -> some View {
|
||||
Button {
|
||||
draftQuestion = p.prompt
|
||||
questionFocused = true
|
||||
} label: {
|
||||
Text(p.title)
|
||||
.font(.tjScaled( 12, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.lineLimit(1)
|
||||
.padding(.horizontal, 12)
|
||||
.padding(.vertical, 7)
|
||||
.background(Capsule().fill(Tj.Palette.sand2))
|
||||
.overlay(Capsule().strokeBorder(Tj.Palette.lineSoft, lineWidth: 1))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.contextMenu {
|
||||
if !p.isBuiltin {
|
||||
Button(role: .destructive) {
|
||||
promptStore.delete(p)
|
||||
} label: {
|
||||
Label("删除", systemImage: "trash")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var addQuickPromptChip: some View {
|
||||
Button { showAddPrompt = true } label: {
|
||||
Label("自定义", systemImage: "plus")
|
||||
.font(.tjScaled( 12, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
.padding(.horizontal, 12)
|
||||
.padding(.vertical, 7)
|
||||
.background(Capsule().fill(Tj.Palette.paper))
|
||||
.overlay(
|
||||
Capsule().strokeBorder(
|
||||
Tj.Palette.line,
|
||||
style: StrokeStyle(lineWidth: 1, dash: [3])
|
||||
)
|
||||
)
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
// MARK: - Header
|
||||
@@ -128,14 +211,7 @@ struct HealthExportSheet: View {
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
Text("例:最近血压波动大吗？")
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Text("例:把我最近头晕、睡眠和指标变化整理给医生")
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
quickPromptRow
|
||||
|
||||
Text("上下文:全部记录指标 + 健康日记 · 本地 RAG · 不上传任何数据")
|
||||
.font(.tjScaled( 11))
|
||||
@@ -161,12 +237,17 @@ struct HealthExportSheet: View {
|
||||
Text(turn.role.transcriptLabel)
|
||||
.font(.tjScaled( 11, weight: .semibold))
|
||||
.foregroundStyle(isUser ? Tj.Palette.paper.opacity(0.8) : Tj.Palette.text3)
|
||||
if !isUser, let summary = turnRetrievals[turn.id] {
|
||||
RetrievalChipsView(summary: summary)
|
||||
}
|
||||
if turn.id == answeringTurnID && turn.text.isEmpty {
|
||||
HStack(spacing: 8) {
|
||||
ProgressView()
|
||||
Text("正在查看本地记录…")
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
Text(turnRetrievals[turn.id] == nil
|
||||
? "正在查看本地记录…"
|
||||
: "正在根据这些记录回答…")
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
AIFlowBar()
|
||||
}
|
||||
} else {
|
||||
Text(turn.text)
|
||||
@@ -196,6 +277,11 @@ struct HealthExportSheet: View {
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
MarkdownView(text: content)
|
||||
|
||||
if completed {
|
||||
Divider().background(Tj.Palette.lineSoft)
|
||||
AIDisclaimerFooter()
|
||||
}
|
||||
}
|
||||
.padding(16)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
@@ -220,6 +306,9 @@ struct HealthExportSheet: View {
|
||||
arrow
|
||||
phasePill(.generating)
|
||||
}
|
||||
if let retrieval {
|
||||
RetrievalChipsView(summary: retrieval)
|
||||
}
|
||||
if phase == .generating && rate > 0 {
|
||||
Text(String(format: String(appLoc: "本地推理 · %.1f tok/s"), rate))
|
||||
.font(.tjScaled( 11, design: .monospaced))
|
||||
@@ -229,6 +318,9 @@ struct HealthExportSheet: View {
|
||||
.font(.tjScaled( 11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
|
||||
// AI 计算中:多彩流光线(与日记 AI 辅助同一组件)
|
||||
AIFlowBar().padding(.top, 2)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -267,7 +359,7 @@ struct HealthExportSheet: View {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "exclamationmark.triangle.fill")
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
Text(err.localizedDescription)
|
||||
Text(friendlyMessage(for: err))
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
}
|
||||
@@ -291,7 +383,7 @@ struct HealthExportSheet: View {
|
||||
}
|
||||
.buttonStyle(TjGhostButton(height: 44, fontSize: 13, horizontalPadding: 14))
|
||||
|
||||
ShareLink(item: content) {
|
||||
ShareLink(item: AIDisclaimer.appended(to: content)) {
|
||||
Label("分享", systemImage: "square.and.arrow.up")
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.tracking(1)
|
||||
@@ -319,7 +411,7 @@ struct HealthExportSheet: View {
|
||||
private var composer: some View {
|
||||
VStack(spacing: 10) {
|
||||
HStack(spacing: 8) {
|
||||
TextField("继续提问或补充情况…", text: $draftQuestion, axis: .vertical)
|
||||
TextField("写下要整理什么,或先提问补充情况…", text: $draftQuestion, axis: .vertical)
|
||||
.font(.tjScaled( 14))
|
||||
.lineLimit(1...4)
|
||||
.padding(.horizontal, 12)
|
||||
@@ -342,12 +434,28 @@ struct HealthExportSheet: View {
|
||||
.accessibilityLabel("发送问题")
|
||||
}
|
||||
|
||||
Button { startReportGeneration() } label: {
|
||||
Label("生成整理报告", systemImage: "doc.text.below.ecg")
|
||||
if isGeneratingReport {
|
||||
Button { stopGeneration() } label: {
|
||||
Label("停止生成", systemImage: "stop.fill")
|
||||
.font(.tjScaled( 14, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.frame(maxWidth: .infinity)
|
||||
.frame(height: 44)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.brick, lineWidth: 1)
|
||||
)
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
} else {
|
||||
Button { startReportGeneration() } label: {
|
||||
Label("生成整理报告", systemImage: "doc.text.below.ecg")
|
||||
}
|
||||
.buttonStyle(TjPrimaryButton(height: 44, fontSize: 14))
|
||||
.disabled(!canGenerateReport)
|
||||
.opacity(canGenerateReport ? 1 : 0.45)
|
||||
}
|
||||
.buttonStyle(TjPrimaryButton(height: 44, fontSize: 14))
|
||||
.disabled(!canGenerateReport)
|
||||
.opacity(canGenerateReport ? 1 : 0.45)
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 12)
|
||||
@@ -380,16 +488,33 @@ struct HealthExportSheet: View {
|
||||
task?.cancel()
|
||||
task = Task { @MainActor in
|
||||
do {
|
||||
for try await chunk in stream {
|
||||
appendToTurn(id: assistantTurn.id, text: chunk.text)
|
||||
if chunk.decodeRate > 0 { rate = chunk.decodeRate }
|
||||
for try await event in stream {
|
||||
switch event {
|
||||
case .retrieved(let summary):
|
||||
withAnimation(.snappy(duration: 0.25)) {
|
||||
turnRetrievals[assistantTurn.id] = summary
|
||||
}
|
||||
case .token(let chunk):
|
||||
appendToTurn(id: assistantTurn.id, text: chunk.text)
|
||||
if chunk.decodeRate > 0 { rate = chunk.decodeRate }
|
||||
case .phaseChanged, .completed:
|
||||
break
|
||||
}
|
||||
}
|
||||
answeringTurnID = nil
|
||||
questionFocused = true
|
||||
} catch {
|
||||
answeringTurnID = nil
|
||||
appendToTurn(id: assistantTurn.id, text: error.localizedDescription)
|
||||
questionFocused = true
|
||||
if error is CancellationError { return }
|
||||
#if DEBUG
|
||||
print("[HealthExport] answer failed: \(error)")
|
||||
#endif
|
||||
// 已有部分回答就保留;否则给一句友好兜底,绝不把技术异常当成 AI 的回答展示。
|
||||
if let idx = turns.firstIndex(where: { $0.id == assistantTurn.id }),
|
||||
turns[idx].text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
|
||||
turns[idx].text = String(appLoc: "这次没能回答上来,请换个说法再试一次。")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -402,10 +527,20 @@ struct HealthExportSheet: View {
|
||||
private func startReportGeneration() {
|
||||
guard canGenerateReport else { return }
|
||||
questionFocused = false
|
||||
|
||||
// 直接生成:输入框里有文字(快捷问答/手输)就把它作为一条诉求追加进对话,
|
||||
// 不必先走多轮问答 —— 用户点一下「生成报告」即可。
|
||||
let draft = draftQuestion.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if !draft.isEmpty {
|
||||
turns.append(.user(draft))
|
||||
draftQuestion = ""
|
||||
}
|
||||
|
||||
content = ""
|
||||
rate = 0 // 重新生成时清零,避免旧 tok/s 残留显示
|
||||
error = nil
|
||||
completed = false
|
||||
retrieval = nil
|
||||
phase = .retrieving
|
||||
|
||||
let stream = HealthExportService.shared.export(conversation: turns, in: ctx)
|
||||
@@ -416,6 +551,8 @@ struct HealthExportSheet: View {
|
||||
switch event {
|
||||
case .phaseChanged(let ph):
|
||||
phase = ph
|
||||
case .retrieved(let summary):
|
||||
withAnimation(.snappy(duration: 0.25)) { retrieval = summary }
|
||||
case .token(let chunk):
|
||||
content += chunk.text
|
||||
if chunk.decodeRate > 0 { rate = chunk.decodeRate }
|
||||
@@ -424,6 +561,10 @@ struct HealthExportSheet: View {
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
if error is CancellationError { return }
|
||||
#if DEBUG
|
||||
print("[HealthExport] export failed: \(error)")
|
||||
#endif
|
||||
self.error = error
|
||||
self.phase = nil
|
||||
}
|
||||
@@ -435,6 +576,17 @@ struct HealthExportSheet: View {
|
||||
startReportGeneration()
|
||||
}
|
||||
|
||||
/// 停止正在进行的报告生成:取消推理任务,回到可重新生成的干净态(已写的诉求保留在对话里)。
|
||||
private func stopGeneration() {
|
||||
task?.cancel()
|
||||
task = nil
|
||||
phase = nil
|
||||
rate = 0
|
||||
completed = false
|
||||
content = ""
|
||||
retrieval = nil
|
||||
}
|
||||
|
||||
private func reset() {
|
||||
task?.cancel()
|
||||
task = nil
|
||||
@@ -444,11 +596,13 @@ struct HealthExportSheet: View {
|
||||
error = nil
|
||||
completed = false
|
||||
answeringTurnID = nil
|
||||
retrieval = nil
|
||||
turnRetrievals = [:]
|
||||
questionFocused = true
|
||||
}
|
||||
|
||||
private func copy() {
|
||||
UIPasteboard.general.string = content
|
||||
UIPasteboard.general.string = AIDisclaimer.appended(to: content)
|
||||
copiedFlash = true
|
||||
DispatchQueue.main.asyncAfter(deadline: .now() + 1.4) {
|
||||
copiedFlash = false
|
||||
@@ -459,6 +613,65 @@ struct HealthExportSheet: View {
|
||||
task?.cancel()
|
||||
dismiss()
|
||||
}
|
||||
|
||||
// MARK: - 滚动
|
||||
|
||||
private func scrollToBottom(_ proxy: ScrollViewProxy) {
|
||||
lastScrollAt = Date()
|
||||
withAnimation(.easeOut(duration: 0.12)) {
|
||||
proxy.scrollTo("bottom", anchor: .bottom)
|
||||
}
|
||||
}
|
||||
|
||||
/// 节流滚动:流式 token 高频触发,限制到约 8Hz,避免每 token 一次动画布局。
|
||||
private func throttledScrollToBottom(_ proxy: ScrollViewProxy) {
|
||||
guard Date().timeIntervalSince(lastScrollAt) > 0.12 else { return }
|
||||
scrollToBottom(proxy)
|
||||
}
|
||||
|
||||
/// 把推理过程中抛出的技术异常翻译成用户能看懂的一句话(取消不算错误)。
|
||||
private func friendlyMessage(for error: Error) -> String {
|
||||
if error is CancellationError { return "" }
|
||||
return String(appLoc: "这次没能生成成功,请稍后重试。")
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 检索结果 chips(本地 RAG 可视化)
|
||||
|
||||
/// 生成开始前先把「本地 RAG 找到了什么」演出来:N 条记录 + 记录名 chips。
|
||||
/// 结构化检索(不用 embedding)的天然优势 —— 每条命中都可解释、可展示(§12 卖点 3)。
|
||||
private struct RetrievalChipsView: View {
|
||||
let summary: HealthExportService.RetrievalSummary
|
||||
|
||||
var body: some View {
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
if summary.totalCount == 0 {
|
||||
Text("本地档案中暂无相关记录,将仅按你的描述整理")
|
||||
.font(.tjScaled( 11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
} else {
|
||||
Text(String(appLoc: "已在本地档案中找到 \(summary.totalCount) 条相关记录"))
|
||||
.font(.tjScaled( 11, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.leaf)
|
||||
ScrollView(.horizontal, showsIndicators: false) {
|
||||
HStack(spacing: 6) {
|
||||
ForEach(Array(summary.chips.enumerated()), id: \.offset) { _, chip in
|
||||
Text(chip)
|
||||
.font(.tjScaled( 11))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
.lineLimit(1)
|
||||
.padding(.horizontal, 8)
|
||||
.padding(.vertical, 4)
|
||||
.background(Capsule().fill(Tj.Palette.sand2))
|
||||
.overlay(Capsule().strokeBorder(Tj.Palette.lineSoft, lineWidth: 1))
|
||||
}
|
||||
}
|
||||
.padding(.vertical, 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
.transition(.opacity.combined(with: .move(edge: .top)))
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 简易 Markdown 渲染(行级)
|
||||
@@ -578,15 +791,30 @@ struct MarkdownView: View {
|
||||
return nil
|
||||
}
|
||||
|
||||
/// 行内样式缓存:流式生成时整段会被反复重渲染,缓存命中可避免对同一行重复跑昂贵的
|
||||
/// markdown 解析。键为行文本,稳定的历史行命中缓存,只有正在增长的尾行才真正解析。
|
||||
private final class AttrBox { let value: AttributedString; init(_ v: AttributedString) { value = v } }
|
||||
private static let inlineCache: NSCache<NSString, AttrBox> = {
|
||||
let c = NSCache<NSString, AttrBox>()
|
||||
c.countLimit = 256
|
||||
return c
|
||||
}()
|
||||
|
||||
private func inline(_ s: String) -> AttributedString {
|
||||
// 快路径:整段报告里绝大多数行(标题、普通条目、纯正文)都没有内联标记,
|
||||
// 直接走纯文本,跳过 AttributedString(markdown:) —— 这是流式期间最大的一笔开销。
|
||||
if !s.contains(where: { $0 == "*" || $0 == "_" || $0 == "[" || $0 == "`" }) {
|
||||
return AttributedString(s)
|
||||
}
|
||||
let key = s as NSString
|
||||
if let hit = Self.inlineCache.object(forKey: key) { return hit.value }
|
||||
// **bold** / *italic* / [text](url) 走 AttributedString markdown 解析
|
||||
if let attr = try? AttributedString(
|
||||
let attr = (try? AttributedString(
|
||||
markdown: s,
|
||||
options: .init(interpretedSyntax: .inlineOnlyPreservingWhitespace)
|
||||
) {
|
||||
return attr
|
||||
}
|
||||
return AttributedString(s)
|
||||
)) ?? AttributedString(s)
|
||||
Self.inlineCache.setObject(AttrBox(attr), forKey: key)
|
||||
return attr
|
||||
}
|
||||
|
||||
// MARK: - 行级解析
|
||||
@@ -641,9 +869,9 @@ struct MarkdownView: View {
|
||||
# 就诊摘要 — 感冒就诊
|
||||
|
||||
## 主诉
|
||||
患者男,38 岁,感冒 3 天未愈,主诉鼻塞、咳嗽、低烧。
|
||||
本人男,38 岁,感冒 3 天未愈,主诉鼻塞、咳嗽、低烧。
|
||||
|
||||
## 患者背景
|
||||
## 本人背景
|
||||
- 高血压 2 年
|
||||
- 在服药:**缬沙坦 80mg qd**
|
||||
- 过敏:青霉素
|
||||
|
||||
92
康康/Features/Archive/QuickPrompt.swift
Normal file
92
康康/Features/Archive/QuickPrompt.swift
Normal file
@@ -0,0 +1,92 @@
|
||||
import Foundation
|
||||
import Observation
|
||||
|
||||
/// 「身体档案」里的快捷问答:点一下把一句常用问题填进输入框。
|
||||
/// 内置 3 条(不可删),用户可自定义追加(可删)。
|
||||
struct QuickPrompt: Identifiable, Codable, Equatable {
|
||||
let id: UUID
|
||||
var title: String // chip 上显示的短标签
|
||||
var prompt: String // 点击后填入输入框的完整问题
|
||||
var isBuiltin: Bool
|
||||
|
||||
init(id: UUID = UUID(), title: String, prompt: String, isBuiltin: Bool) {
|
||||
self.id = id
|
||||
self.title = title
|
||||
self.prompt = prompt
|
||||
self.isBuiltin = isBuiltin
|
||||
}
|
||||
}
|
||||
|
||||
/// 快捷问答存储:内置常量 + 自定义条目(UserDefaults JSON,无 SwiftData schema 迁移风险)。
|
||||
/// 自定义条目只是 UI 便利项、不是健康记录,故不进 SwiftData。
|
||||
@Observable
|
||||
final class QuickPromptStore {
|
||||
static let shared = QuickPromptStore()
|
||||
|
||||
private let defaults = UserDefaults.standard
|
||||
private let storageKey = "kk.quickPrompts.custom.v1"
|
||||
|
||||
private(set) var custom: [QuickPrompt]
|
||||
|
||||
private init() {
|
||||
if let data = defaults.data(forKey: storageKey),
|
||||
let decoded = try? JSONDecoder().decode([QuickPrompt].self, from: data) {
|
||||
custom = decoded
|
||||
} else {
|
||||
custom = []
|
||||
}
|
||||
}
|
||||
|
||||
/// 内置在前、自定义在后,供 chip 行展示。
|
||||
var all: [QuickPrompt] { Self.builtins + custom }
|
||||
|
||||
/// 追加一条自定义问答。空白忽略;标签自动取问题前几个字。
|
||||
func add(prompt rawPrompt: String) {
|
||||
let trimmed = rawPrompt.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else { return }
|
||||
custom.append(QuickPrompt(title: Self.deriveTitle(trimmed),
|
||||
prompt: trimmed,
|
||||
isBuiltin: false))
|
||||
persist()
|
||||
}
|
||||
|
||||
/// 删除一条自定义问答(内置不可删)。
|
||||
func delete(_ p: QuickPrompt) {
|
||||
guard !p.isBuiltin else { return }
|
||||
custom.removeAll { $0.id == p.id }
|
||||
persist()
|
||||
}
|
||||
|
||||
private func persist() {
|
||||
if let data = try? JSONEncoder().encode(custom) {
|
||||
defaults.set(data, forKey: storageKey)
|
||||
}
|
||||
}
|
||||
|
||||
/// 自定义条目的短标签:压成单行,取前 8 个字,超出补省略号。
|
||||
static func deriveTitle(_ prompt: String) -> String {
|
||||
let oneLine = prompt.replacingOccurrences(of: "\n", with: " ")
|
||||
.trimmingCharacters(in: .whitespaces)
|
||||
let head = oneLine.prefix(8)
|
||||
return oneLine.count > 8 ? "\(head)…" : String(head)
|
||||
}
|
||||
|
||||
/// 内置 3 条(首屏):覆盖「就诊 / 解读 / 速查」三类,数据依赖稳、不碰诊断红线。
|
||||
static let builtins: [QuickPrompt] = [
|
||||
QuickPrompt(
|
||||
title: "就诊摘要",
|
||||
prompt: "根据我最近的身体症状，结合历史指标，整理一份让门诊医生快速了解我情况的就诊摘要。",
|
||||
isBuiltin: true
|
||||
),
|
||||
QuickPrompt(
|
||||
title: "趋势解读",
|
||||
prompt: "把我血压最近半年的变化讲清楚：是变好还是变差、要注意什么。",
|
||||
isBuiltin: true
|
||||
),
|
||||
QuickPrompt(
|
||||
title: "速答清单",
|
||||
prompt: "把我的过敏史、正在吃的药、慢性病整理成一句话清单，方便就诊时快速回答医生。",
|
||||
isBuiltin: true
|
||||
),
|
||||
]
|
||||
}
|
||||
@@ -8,9 +8,12 @@ struct CaptureReviewForm: View {
|
||||
@State var parsed: ParsedReport
|
||||
let assets: [FileVault.SavedAsset]
|
||||
let warning: String?
|
||||
/// 归档模式:只存原图 + 基本信息(标题/类型/日期/机构),隐藏指标区与摘要。
|
||||
/// 报告归档不再逐项识别(逐项多模态在 2B 上易 OOM 卡死),见 CaptureService.extractReportMeta。
|
||||
var metaOnly: Bool = false
|
||||
let onSave: (ParsedReport) -> Void
|
||||
let onCancel: () -> Void
|
||||
/// 「重新识别」回调。assets 为空(写图失败)时传 nil,banner 上不显示该按钮。
|
||||
/// 「重新识别信息」回调。assets 为空(写图失败)时传 nil,banner 上不显示该按钮。
|
||||
var onReanalyze: (() -> Void)? = nil
|
||||
|
||||
var body: some View {
|
||||
@@ -23,7 +26,9 @@ struct CaptureReviewForm: View {
|
||||
pageThumbnails
|
||||
}
|
||||
metaSection
|
||||
indicatorSection
|
||||
if !metaOnly {
|
||||
indicatorSection
|
||||
}
|
||||
Spacer(minLength: 8)
|
||||
actions
|
||||
}
|
||||
@@ -68,20 +73,26 @@ struct CaptureReviewForm: View {
|
||||
private var pageThumbnails: some View {
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
sectionLabel(String(appLoc: "已保存 \(assets.count) 页(端侧加密)"))
|
||||
if metaOnly {
|
||||
Text("原图已加密保存,详情页随时可翻看放大。系统只识别报告日期与机构作为标签,不逐项录入数值。")
|
||||
.font(.tjScaled( 11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
}
|
||||
ScrollView(.horizontal, showsIndicators: false) {
|
||||
HStack(spacing: 10) {
|
||||
ForEach(Array(assets.enumerated()), id: \.offset) { _, asset in
|
||||
if let img = try? FileVault.shared.loadImage(relativePath: asset.relativePath) {
|
||||
Image(uiImage: img)
|
||||
.resizable()
|
||||
.scaledToFill()
|
||||
.frame(width: 84, height: 110)
|
||||
.clipShape(RoundedRectangle(cornerRadius: 8, style: .continuous))
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: 8, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1)
|
||||
)
|
||||
VaultImage(relativePath: asset.relativePath, maxPixel: 400) { img in
|
||||
Image(uiImage: img).resizable().scaledToFill()
|
||||
} placeholder: { _ in
|
||||
Tj.Palette.paper
|
||||
}
|
||||
.frame(width: 84, height: 110)
|
||||
.clipShape(RoundedRectangle(cornerRadius: 8, style: .continuous))
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: 8, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1)
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -117,9 +128,11 @@ struct CaptureReviewForm: View {
|
||||
labeledField(String(appLoc: "机构(可选)")) {
|
||||
TextField("如:协和医院", text: $parsed.institution)
|
||||
}
|
||||
labeledField(String(appLoc: "摘要(可选)")) {
|
||||
TextField("一句话总结", text: $parsed.summary, axis: .vertical)
|
||||
.lineLimit(1...3)
|
||||
if !metaOnly {
|
||||
labeledField(String(appLoc: "摘要(可选)")) {
|
||||
TextField("一句话总结", text: $parsed.summary, axis: .vertical)
|
||||
.lineLimit(1...3)
|
||||
}
|
||||
}
|
||||
}
|
||||
.padding(12)
|
||||
|
||||
@@ -32,8 +32,14 @@ struct PhotoPickerSheet: View {
|
||||
.clipShape(Capsule())
|
||||
}
|
||||
|
||||
Button("取消", action: onCancel)
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Button(action: onCancel) {
|
||||
Text("取消")
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.padding(.horizontal, 24)
|
||||
.frame(minHeight: 44) // HIG 最小命中区
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
|
||||
if loading {
|
||||
ProgressView().tint(Tj.Palette.ink)
|
||||
|
||||
@@ -62,7 +62,7 @@ struct UnifiedCaptureFlow: View {
|
||||
switch phase {
|
||||
case .idle: return String(appLoc: "拍摄报告")
|
||||
case .analyzing: return String(appLoc: "本地识别中…")
|
||||
case .editing: return String(appLoc: "核对识别结果")
|
||||
case .editing: return String(appLoc: "核对报告信息")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,6 +86,7 @@ struct UnifiedCaptureFlow: View {
|
||||
parsed: parsed,
|
||||
assets: assets,
|
||||
warning: warning,
|
||||
metaOnly: true, // 归档只存原图 + meta,不逐项识别(§见 CaptureService.extractReportMeta)
|
||||
onSave: { final in saveAll(parsed: final, assets: assets) },
|
||||
onCancel: cancelAll,
|
||||
onReanalyze: assets.isEmpty ? nil : { reanalyze(assets: assets) }
|
||||
@@ -152,10 +153,21 @@ struct UnifiedCaptureFlow: View {
|
||||
phase = .analyzing(images: images, assets: nil)
|
||||
let timeout = analyzeTimeoutSeconds
|
||||
analyzeTask = Task {
|
||||
// Step 1: 先把图写进 Vault。
|
||||
// 在 UI 这一层写,而不是塞进 CaptureService.analyze —— 这样取消/失败回退时,
|
||||
// assets 已经在 phase 里,cancelAll 能清理孤儿,editingFallback 也不必再补写。
|
||||
let assets = images.compactMap { try? FileVault.shared.writeJPEG($0) }
|
||||
// Step 1: 把图写进 Vault(归档核心价值:先把原图存下来),并立刻降采样出预览缩略图。
|
||||
// 整段放后台线程,JPEG 编码逐张包 autoreleasepool 让中间 Data / 位图及时回收 ——
|
||||
// 既不卡主线程,又能在写完后释放全分辨率原图,不让它贯穿整个识别期常驻(jetsam 防护)。
|
||||
let inputBox = UncheckedImageBox(images: images)
|
||||
let written: (assets: [FileVault.SavedAsset], thumbs: UncheckedImageBox) =
|
||||
await Task.detached(priority: .userInitiated) {
|
||||
let assets = inputBox.images.compactMap { img in
|
||||
autoreleasepool { try? FileVault.shared.writeJPEG(img) }
|
||||
}
|
||||
let thumbs = assets.compactMap {
|
||||
try? FileVault.shared.loadDownsampledImage(relativePath: $0.relativePath, maxPixelSize: 600)
|
||||
}
|
||||
return (assets, UncheckedImageBox(images: thumbs))
|
||||
}.value
|
||||
let assets = written.assets
|
||||
// 极端情况:用户在写图过程中按了「取消」,View 已 dismiss、cancelAll 看到的
|
||||
// phase 还是 .analyzing(_, nil),清不到这批刚写完的图 — 这里手动收尾。
|
||||
if Task.isCancelled {
|
||||
@@ -167,61 +179,50 @@ struct UnifiedCaptureFlow: View {
|
||||
phase = .editing(
|
||||
parsed: .empty(),
|
||||
assets: [],
|
||||
warning: String(appLoc: "图片保存失败,手动录入并保留文本")
|
||||
warning: String(appLoc: "图片保存失败,请重试")
|
||||
)
|
||||
}
|
||||
return
|
||||
}
|
||||
// 把 assets 暴露给 phase,使工具栏「取消」也能找到孤儿清理。
|
||||
// 原图已落盘:phase 改持 600px 缩略图(释放全分辨率原图),同时把 assets 暴露给「取消」做孤儿清理。
|
||||
await MainActor.run {
|
||||
if case .analyzing(let imgs, _) = phase {
|
||||
phase = .analyzing(images: imgs, assets: assets)
|
||||
}
|
||||
phase = .analyzing(images: written.thumbs.images, assets: assets)
|
||||
}
|
||||
|
||||
// Step 2: VL 推理(timeout 哨兵到点 cancel 父任务,VLSession 在下一个 token break)。
|
||||
// Step 2: 轻量 meta 提取(OCR + 文本 LLM,只抽日期/机构/类型/标题)。
|
||||
// 不再跑多模态逐项识别 —— 那在 2B 上又慢又会 OOM 卡死。watchdog 到点 cancel。
|
||||
let watchdog = Task {
|
||||
try? await Task.sleep(for: .seconds(timeout))
|
||||
analyzeTask?.cancel()
|
||||
}
|
||||
defer { watchdog.cancel() }
|
||||
|
||||
do {
|
||||
let parsed = try await CaptureService.shared.reanalyze(assets: assets)
|
||||
if Task.isCancelled {
|
||||
await editingFallback(assets: assets,
|
||||
msg: String(appLoc: "识别超时(>\(timeout)s),先手动录入"))
|
||||
return
|
||||
}
|
||||
let (meta, recognized) = await CaptureService.shared.extractReportMeta(assets: assets)
|
||||
if Task.isCancelled {
|
||||
await MainActor.run {
|
||||
phase = .editing(
|
||||
parsed: parsed,
|
||||
assets: assets,
|
||||
warning: parsed.isEmpty ? String(appLoc: "识别没有读出指标,请手动补充") : nil
|
||||
)
|
||||
phase = .editing(parsed: .empty(), assets: assets,
|
||||
warning: String(appLoc: "识别超时,已保存原图,请手动填写信息"))
|
||||
}
|
||||
} catch let CaptureError.parseFailed(msg) {
|
||||
await editingFallback(assets: assets, msg: String(appLoc: "VL 输出无法解析:\(msg)"))
|
||||
} catch let CaptureError.inferenceFailed(msg) {
|
||||
await editingFallback(assets: assets,
|
||||
msg: Task.isCancelled
|
||||
? String(appLoc: "识别超时(>\(timeout)s),先手动录入")
|
||||
: String(appLoc: "推理失败:\(msg)"))
|
||||
} catch CaptureError.modelNotReady {
|
||||
await editingFallback(assets: assets, msg: String(appLoc: "VL 模型未就绪,先手动录入"))
|
||||
} catch {
|
||||
await editingFallback(assets: assets,
|
||||
msg: String(appLoc: "未知错误:\(error.localizedDescription)"))
|
||||
return
|
||||
}
|
||||
await MainActor.run {
|
||||
phase = .editing(
|
||||
parsed: meta,
|
||||
assets: assets,
|
||||
warning: recognized ? nil
|
||||
: String(appLoc: "未能自动识别报告信息,已保存原图,可手动填写日期 / 机构")
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 「重新识别」:复用已存 assets,不再写图,只重跑 VL。
|
||||
/// 「重新识别信息」:复用已存 assets,不再写图,只重跑一次轻量 meta 提取。
|
||||
private func reanalyze(assets: [FileVault.SavedAsset]) {
|
||||
analyzeTask?.cancel()
|
||||
// 这里没有原始 UIImage,AnalyzingView 显示首张缩略图即可
|
||||
// 这里没有原始 UIImage,AnalyzingView 只把首张缩略图模糊后当背景,降采样到 600px 足够,
|
||||
// 避免为「重新识别」把整页全分辨率原图(数十 MB)载进内存。
|
||||
let thumbnails: [UIImage] = assets.compactMap {
|
||||
try? FileVault.shared.loadImage(relativePath: $0.relativePath)
|
||||
try? FileVault.shared.loadDownsampledImage(relativePath: $0.relativePath, maxPixelSize: 600)
|
||||
}
|
||||
phase = .analyzing(images: thumbnails, assets: assets)
|
||||
let timeout = analyzeTimeoutSeconds
|
||||
@@ -232,40 +233,19 @@ struct UnifiedCaptureFlow: View {
|
||||
}
|
||||
defer { watchdog.cancel() }
|
||||
|
||||
do {
|
||||
let parsed = try await CaptureService.shared.reanalyze(assets: assets)
|
||||
if Task.isCancelled {
|
||||
await editingFallback(assets: assets,
|
||||
msg: String(appLoc: "识别超时(>\(timeout)s),保留旧编辑"))
|
||||
return
|
||||
}
|
||||
let (meta, recognized) = await CaptureService.shared.extractReportMeta(assets: assets)
|
||||
if Task.isCancelled {
|
||||
await MainActor.run {
|
||||
phase = .editing(
|
||||
parsed: parsed,
|
||||
assets: assets,
|
||||
warning: parsed.isEmpty ? String(appLoc: "重新识别没有读出新指标") : nil
|
||||
)
|
||||
phase = .editing(parsed: .empty(), assets: assets,
|
||||
warning: String(appLoc: "识别超时,已保留原图"))
|
||||
}
|
||||
} catch CaptureError.modelNotReady {
|
||||
await editingFallback(assets: assets, msg: String(appLoc: "VL 模型未就绪"))
|
||||
} catch let CaptureError.parseFailed(msg) {
|
||||
await editingFallback(assets: assets, msg: String(appLoc: "VL 输出无法解析:\(msg)"))
|
||||
} catch let CaptureError.inferenceFailed(msg) {
|
||||
await editingFallback(assets: assets,
|
||||
msg: Task.isCancelled
|
||||
? String(appLoc: "识别超时(>\(timeout)s)")
|
||||
: String(appLoc: "推理失败:\(msg)"))
|
||||
} catch {
|
||||
await editingFallback(assets: assets,
|
||||
msg: String(appLoc: "未知错误:\(error.localizedDescription)"))
|
||||
return
|
||||
}
|
||||
await MainActor.run {
|
||||
phase = .editing(parsed: meta, assets: assets,
|
||||
warning: recognized ? nil
|
||||
: String(appLoc: "未能自动识别报告信息,可手动填写"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// reanalyze 失败时回到 editing,保留 assets 但清空 parsed。
|
||||
private func editingFallback(assets: [FileVault.SavedAsset], msg: String) async {
|
||||
await MainActor.run {
|
||||
phase = .editing(parsed: .empty(), assets: assets, warning: msg)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -311,10 +291,18 @@ struct UnifiedCaptureFlow: View {
|
||||
}
|
||||
|
||||
try? ctx.save()
|
||||
// 后台预生成大白话摘要:用户继续操作,详情页打开时秒开。
|
||||
// 低优先级 —— 任何前台 AI 任务(再次拍照/问答)都会让它在下一个 token 让位。
|
||||
Task { await ReportInsightService.shared.pregenerateIfNeeded(report: report, in: ctx) }
|
||||
onClose()
|
||||
}
|
||||
}
|
||||
|
||||
/// 跨 detached 边界安全携带 UIImage 数组:图片只读、不并发改,封装免 Sendable 报错。
|
||||
private struct UncheckedImageBox: @unchecked Sendable {
|
||||
let images: [UIImage]
|
||||
}
|
||||
|
||||
// MARK: - 分析中视图
|
||||
|
||||
private struct AnalyzingView: View {
|
||||
@@ -359,10 +347,16 @@ private struct AnalyzingView: View {
|
||||
.foregroundStyle(Tj.Palette.amber)
|
||||
}
|
||||
}
|
||||
Button("取消识别 · 改为手动录入", action: onCancel)
|
||||
.font(.tjScaled( 13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.padding(.top, 4)
|
||||
Button(action: onCancel) {
|
||||
Text("取消识别 · 改为手动录入")
|
||||
.font(.tjScaled( 13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.padding(.horizontal, 20)
|
||||
.frame(minHeight: 44) // HIG 最小命中区
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.padding(.top, 4)
|
||||
Spacer()
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
|
||||
489
康康/Features/Chat/HealthChatView.swift
Normal file
489
康康/Features/Chat/HealthChatView.swift
Normal file
@@ -0,0 +1,489 @@
|
||||
import SwiftUI
|
||||
import SwiftData
|
||||
|
||||
/// 「和康康聊聊」——全局健康记录问答(RAG)。
|
||||
///
|
||||
/// 你问,康康基于你本机的健康记录(指标 / 报告 / 症状 / 日记)检索后作答,100% 端侧。
|
||||
/// 复用 `HealthExportService.answer`(检索快照 → dialogueAnswer prompt → 流式作答 + 检索可视化),
|
||||
/// 卡通形象 `CompanionAvatarView` 放在顶栏当「对话对象」,情绪随生成状态切换。
|
||||
///
|
||||
/// 长期记忆:每轮问答落 `ChatTurn`,进场读回最近 10 轮,跨会话续聊(§12#3 本地 RAG 长期记忆)。
|
||||
/// 边界:UI 不直接调 AIRuntime(红线 #3),全部经 `HealthExportService`;失败不卡屏(红线 #5)。
|
||||
struct HealthChatView: View {
|
||||
let onClose: () -> Void
|
||||
|
||||
@Environment(\.modelContext) private var ctx
|
||||
|
||||
/// 本场对话(含进场读回的历史 + 本次新问答)。assistant 空文本 = 正在生成的占位轮。
|
||||
@State private var turns: [HealthExportDialogueTurn] = []
|
||||
/// 每条 assistant 回答对应的「查阅到的记录」摘要,展示在气泡下方做 RAG 可视化。
|
||||
@State private var turnRetrievals: [UUID: HealthExportService.RetrievalSummary] = [:]
|
||||
@State private var draft = ""
|
||||
/// 正在流式作答的 assistant 轮 id;非 nil = 正在回答(锁输入、形象进入思考/说话态)。
|
||||
@State private var answeringTurnID: UUID?
|
||||
@State private var rate: Double = 0
|
||||
@State private var task: Task<Void, Never>?
|
||||
@FocusState private var inputFocused: Bool
|
||||
|
||||
// MARK: 语音提问(复用 SpeechDictationService,与写日记同一套本机转写)
|
||||
@State private var isRecording = false
|
||||
/// 必须 @State:struct View 重建时普通 let 会换新实例,导致 stop() 落在没在录音的新服务上。
|
||||
@State private var dictation = SpeechDictationService()
|
||||
@State private var micDeniedAlert = false
|
||||
|
||||
private var isAnswering: Bool { answeringTurnID != nil }
|
||||
|
||||
/// 端侧模型就绪才能问(answer 走本地推理);未就绪时空态给下载指引。
|
||||
private var modelReady: Bool { ModelStore.shared.isComplete(for: .llm) }
|
||||
|
||||
/// 形象情绪:录音→listening;检索/等首 token→thinking;流式吐字→speaking;其余→idle。
|
||||
private var mood: CompanionAvatarView.Mood {
|
||||
if isRecording { return .listening }
|
||||
guard let id = answeringTurnID else { return .idle }
|
||||
let streaming = turns.first { $0.id == id }?.text.isEmpty == false
|
||||
return streaming ? .speaking : .thinking
|
||||
}
|
||||
|
||||
/// 空态引导:点一下即作为问题发出,降低「不知道能问啥」的启动成本。
|
||||
private let starters = [
|
||||
"帮我总结一下最近的身体状况",
|
||||
"我最近的血压怎么样?",
|
||||
"上一次体检有哪些异常?",
|
||||
"最近的血糖有变化吗?"
|
||||
]
|
||||
|
||||
var body: some View {
|
||||
VStack(spacing: 0) {
|
||||
header
|
||||
transcript
|
||||
}
|
||||
.background(Tj.Palette.sand.ignoresSafeArea())
|
||||
.safeAreaInset(edge: .bottom) { inputBar }
|
||||
.onAppear(perform: loadHistory)
|
||||
.onDisappear {
|
||||
task?.cancel()
|
||||
dictation.abort()
|
||||
}
|
||||
.alert(String(appLoc: "需要麦克风与语音识别权限"), isPresented: $micDeniedAlert) {
|
||||
Button(String(appLoc: "前往设置")) {
|
||||
if let url = URL(string: UIApplication.openSettingsURLString) {
|
||||
UIApplication.shared.open(url)
|
||||
}
|
||||
}
|
||||
Button(String(appLoc: "取消"), role: .cancel) {}
|
||||
} message: {
|
||||
Text("语音提问全程在本机完成,声音和文字都不会上传。")
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - 顶栏(卡通形象 + 状态)
|
||||
|
||||
private var header: some View {
|
||||
HStack(spacing: 12) {
|
||||
CompanionAvatarView(mood: mood, size: 44)
|
||||
|
||||
VStack(alignment: .leading, spacing: 1) {
|
||||
Text("康康")
|
||||
.font(.tjScaled(15, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
Text(headerSubtitle)
|
||||
.font(.tjScaled(11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.lineLimit(1)
|
||||
}
|
||||
|
||||
Spacer(minLength: 8)
|
||||
|
||||
Button(action: onClose) {
|
||||
Image(systemName: "xmark")
|
||||
.font(.tjScaled(14, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.frame(width: 32, height: 32)
|
||||
.background(Circle().fill(Tj.Palette.sand2))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 12)
|
||||
.background(Tj.Palette.paper)
|
||||
.overlay(alignment: .bottom) {
|
||||
Rectangle().fill(Tj.Palette.lineSoft).frame(height: 1)
|
||||
}
|
||||
}
|
||||
|
||||
private var headerSubtitle: String {
|
||||
if isRecording { return String(appLoc: "正在听你说…") }
|
||||
if isAnswering {
|
||||
return rate > 0
|
||||
? String(appLoc: "正在翻你的记录 · \(String(format: "%.1f", rate)) tok/s")
|
||||
: String(appLoc: "正在翻你的记录…")
|
||||
}
|
||||
return String(appLoc: "问问你的健康记录 · 全程在本机")
|
||||
}
|
||||
|
||||
// MARK: - 对话区
|
||||
|
||||
private var transcript: some View {
|
||||
ScrollViewReader { proxy in
|
||||
ScrollView {
|
||||
if turns.isEmpty {
|
||||
emptyState
|
||||
.padding(.top, 28)
|
||||
.padding(.horizontal, 20)
|
||||
} else {
|
||||
LazyVStack(alignment: .leading, spacing: 14) {
|
||||
ForEach(turns) { turn in
|
||||
bubble(turn)
|
||||
}
|
||||
Color.clear.frame(height: 1).id(bottomAnchor)
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 16)
|
||||
}
|
||||
}
|
||||
// 每吐一个 token / 新增一轮都滚到底,保证最新内容始终可见。
|
||||
.onChange(of: turns.last?.text) { _, _ in scrollToBottom(proxy) }
|
||||
.onChange(of: turns.count) { _, _ in scrollToBottom(proxy) }
|
||||
}
|
||||
}
|
||||
|
||||
private let bottomAnchor = "chat.bottom"
|
||||
|
||||
private func scrollToBottom(_ proxy: ScrollViewProxy) {
|
||||
withAnimation(.easeOut(duration: 0.2)) {
|
||||
proxy.scrollTo(bottomAnchor, anchor: .bottom)
|
||||
}
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private func bubble(_ turn: HealthExportDialogueTurn) -> some View {
|
||||
if turn.role == .user {
|
||||
HStack {
|
||||
Spacer(minLength: 40)
|
||||
Text(turn.text)
|
||||
.font(.tjScaled(14))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 10)
|
||||
.background(RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.ink))
|
||||
}
|
||||
} else {
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
// 空文本 + 正在回答 = 首 token 未到,显示「思考」占位,不给用户空白焦虑。
|
||||
if turn.text.isEmpty && answeringTurnID == turn.id {
|
||||
thinkingBubble
|
||||
} else {
|
||||
// 康康的回答按 Markdown 渲染(加粗 / 要点列表 / 异常值高亮),
|
||||
// 复用报告页那套流式优化过的 MarkdownView(带内联缓存,逐 token 重渲不卡)。
|
||||
MarkdownView(text: turn.text)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 10)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper))
|
||||
.overlay(RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1))
|
||||
}
|
||||
|
||||
if let summary = turnRetrievals[turn.id], summary.totalCount > 0 {
|
||||
retrievalChips(summary)
|
||||
}
|
||||
}
|
||||
.padding(.trailing, 32)
|
||||
}
|
||||
}
|
||||
|
||||
private var thinkingBubble: some View {
|
||||
HStack(spacing: 5) {
|
||||
Text("正在查阅你的记录")
|
||||
.font(.tjScaled(13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
ProgressView().scaleEffect(0.7)
|
||||
}
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 10)
|
||||
.background(RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper))
|
||||
.overlay(RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1))
|
||||
}
|
||||
|
||||
/// RAG 可视化:这条回答查阅了哪些本机记录(§12#3 端侧不可替代性的现场证据)。
|
||||
private func retrievalChips(_ summary: HealthExportService.RetrievalSummary) -> some View {
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
Text("查阅了本机 \(summary.totalCount) 条记录")
|
||||
.font(.tjScaled(10, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
FlowChips(items: summary.chips)
|
||||
}
|
||||
.padding(.leading, 2)
|
||||
}
|
||||
|
||||
// MARK: - 空态(引导起手问题)
|
||||
|
||||
private var emptyState: some View {
|
||||
VStack(spacing: 16) {
|
||||
CompanionAvatarView(mood: .idle, size: 110)
|
||||
VStack(spacing: 6) {
|
||||
Text("我是康康")
|
||||
.font(.tjScaled(20, weight: .semibold, design: .serif))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
Text(modelReady
|
||||
? String(appLoc: "问我关于你健康记录的事,我只看你本机的数据来回答")
|
||||
: String(appLoc: "先到「我的 · 模型管理」下载模型,就能和我聊了"))
|
||||
.font(.tjScaled(13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.multilineTextAlignment(.center)
|
||||
}
|
||||
.padding(.horizontal, 12)
|
||||
|
||||
if modelReady {
|
||||
VStack(spacing: 10) {
|
||||
ForEach(starters, id: \.self) { q in
|
||||
Button { send(q) } label: {
|
||||
HStack {
|
||||
Text(q)
|
||||
.font(.tjScaled(13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
Spacer()
|
||||
Image(systemName: "arrow.up.right")
|
||||
.font(.tjScaled(11, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 12)
|
||||
.tjCard(bordered: true)
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
}
|
||||
.padding(.top, 4)
|
||||
}
|
||||
}
|
||||
.frame(maxWidth: .infinity)
|
||||
}
|
||||
|
||||
// MARK: - 输入栏
|
||||
|
||||
private var inputBar: some View {
|
||||
HStack(spacing: 10) {
|
||||
if SpeechDictationService.isAvailable {
|
||||
Button(action: toggleMic) {
|
||||
Image(systemName: isRecording ? "stop.fill" : "mic.fill")
|
||||
.font(.tjScaled(15, weight: .semibold))
|
||||
.foregroundStyle(isRecording ? Tj.Palette.paper : Tj.Palette.text3)
|
||||
.frame(width: 40, height: 40)
|
||||
.background(Circle().fill(isRecording ? Tj.Palette.brick : Tj.Palette.sand2))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(isAnswering)
|
||||
.opacity(isAnswering ? 0.4 : 1)
|
||||
}
|
||||
|
||||
TextField(String(appLoc: "问问你的健康记录…"), text: $draft, axis: .vertical)
|
||||
.font(.tjScaled(14))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.lineLimit(1...4)
|
||||
.focused($inputFocused)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 10)
|
||||
.background(Capsule().fill(Tj.Palette.sand2))
|
||||
.disabled(isRecording)
|
||||
|
||||
Button { send(draft) } label: {
|
||||
Image(systemName: "arrow.up")
|
||||
.font(.tjScaled(16, weight: .bold))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
.frame(width: 40, height: 40)
|
||||
.background(Circle().fill(canSend ? Tj.Palette.ink : Tj.Palette.line))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(!canSend)
|
||||
}
|
||||
.padding(.horizontal, 16)
|
||||
.padding(.vertical, 10)
|
||||
.background(Tj.Palette.paper)
|
||||
.overlay(alignment: .top) {
|
||||
Rectangle().fill(Tj.Palette.lineSoft).frame(height: 1)
|
||||
}
|
||||
}
|
||||
|
||||
private var canSend: Bool {
|
||||
!draft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
|
||||
&& !isAnswering && !isRecording && modelReady
|
||||
}
|
||||
|
||||
// MARK: - 发送 / 流式作答(镜像 HealthExportSheet.sendQuestion)
|
||||
|
||||
private func send(_ raw: String) {
|
||||
let question = raw.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !question.isEmpty, !isAnswering, modelReady else { return }
|
||||
draft = ""
|
||||
inputFocused = false
|
||||
|
||||
let userTurn = HealthExportDialogueTurn.user(question)
|
||||
let assistantTurn = HealthExportDialogueTurn.assistant("")
|
||||
turns.append(userTurn)
|
||||
turns.append(assistantTurn)
|
||||
answeringTurnID = assistantTurn.id
|
||||
rate = 0
|
||||
|
||||
// 只把最近若干轮喂给模型:端侧小模型 context 有限,历史全塞会溢出/变慢。
|
||||
let conversationForPrompt = Array(
|
||||
turns.filter { $0.id != assistantTurn.id }.suffix(8)
|
||||
)
|
||||
let stream = HealthExportService.shared.answer(
|
||||
question: question,
|
||||
conversation: conversationForPrompt,
|
||||
in: ctx
|
||||
)
|
||||
task?.cancel()
|
||||
task = Task { @MainActor in
|
||||
do {
|
||||
for try await event in stream {
|
||||
switch event {
|
||||
case .retrieved(let summary):
|
||||
withAnimation(.snappy(duration: 0.25)) {
|
||||
turnRetrievals[assistantTurn.id] = summary
|
||||
}
|
||||
case .token(let chunk):
|
||||
appendToTurn(id: assistantTurn.id, text: chunk.text)
|
||||
if chunk.decodeRate > 0 { rate = chunk.decodeRate }
|
||||
case .phaseChanged, .completed:
|
||||
break
|
||||
}
|
||||
}
|
||||
answeringTurnID = nil
|
||||
persist(question: question, answerID: assistantTurn.id)
|
||||
} catch {
|
||||
answeringTurnID = nil
|
||||
if error is CancellationError { return }
|
||||
// 已有部分回答就保留;否则给一句友好兜底,绝不把技术异常当 AI 回答展示(红线 #5)。
|
||||
if let idx = turns.firstIndex(where: { $0.id == assistantTurn.id }),
|
||||
turns[idx].text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
|
||||
turns[idx].text = String(appLoc: "这次没能回答上来,请换个说法再试一次。")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func appendToTurn(id: UUID, text: String) {
|
||||
guard let idx = turns.firstIndex(where: { $0.id == id }) else { return }
|
||||
turns[idx].text += text
|
||||
}
|
||||
|
||||
/// 一轮问答落库成 ChatTurn(跨会话记忆)。空回答 / 兜底话术不入库,避免污染历史。
|
||||
private func persist(question: String, answerID: UUID) {
|
||||
guard let answer = turns.first(where: { $0.id == answerID })?.text
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines), !answer.isEmpty else { return }
|
||||
let turn = ChatTurn(question: question, answer: answer, decodeRate: rate)
|
||||
ctx.insert(turn)
|
||||
try? ctx.save()
|
||||
}
|
||||
|
||||
// MARK: - 历史读回
|
||||
|
||||
/// 进场读回最近 10 轮问答,续上「长期记忆」。只在 onAppear 读一次,新问答只追加,不会重复。
|
||||
private func loadHistory() {
|
||||
var desc = FetchDescriptor<ChatTurn>(sortBy: [SortDescriptor(\.createdAt, order: .reverse)])
|
||||
desc.fetchLimit = 10
|
||||
let recent = (try? ctx.fetch(desc)) ?? []
|
||||
guard !recent.isEmpty else { return }
|
||||
var loaded: [HealthExportDialogueTurn] = []
|
||||
for ct in recent.reversed() {
|
||||
loaded.append(.user(ct.question))
|
||||
loaded.append(.assistant(ct.answer))
|
||||
}
|
||||
turns = loaded
|
||||
}
|
||||
|
||||
// MARK: - 语音提问
|
||||
|
||||
private func toggleMic() {
|
||||
if isRecording {
|
||||
Task { @MainActor in
|
||||
let final = await dictation.stop()
|
||||
isRecording = false
|
||||
let merged = final.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if !merged.isEmpty { draft = merged }
|
||||
}
|
||||
} else {
|
||||
Task { @MainActor in
|
||||
guard await dictation.requestAuthorization() else {
|
||||
micDeniedAlert = true
|
||||
return
|
||||
}
|
||||
do {
|
||||
inputFocused = false
|
||||
draft = ""
|
||||
try dictation.start { partial in draft = partial }
|
||||
isRecording = true
|
||||
} catch {
|
||||
isRecording = false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 检索 chip 的自适应换行排布(简单流式布局,依赖 iOS 16+ Layout)。
|
||||
private struct FlowChips: View {
|
||||
let items: [String]
|
||||
|
||||
var body: some View {
|
||||
FlowLayout(spacing: 6) {
|
||||
ForEach(items, id: \.self) { chip in
|
||||
Text(chip)
|
||||
.font(.tjScaled(10, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
.padding(.horizontal, 8)
|
||||
.padding(.vertical, 4)
|
||||
.background(Capsule().fill(Tj.Palette.sand2))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 极简流式布局:逐行摆放,超出可用宽度就换行。仅用于检索 chip,不追求通用。
|
||||
private struct FlowLayout: Layout {
|
||||
var spacing: CGFloat = 6
|
||||
|
||||
func sizeThatFits(proposal: ProposedViewSize, subviews: Subviews, cache: inout Void) -> CGSize {
|
||||
let maxWidth = proposal.width ?? .infinity
|
||||
var x: CGFloat = 0, y: CGFloat = 0, rowHeight: CGFloat = 0
|
||||
for sub in subviews {
|
||||
let size = sub.sizeThatFits(.unspecified)
|
||||
if x + size.width > maxWidth, x > 0 {
|
||||
x = 0
|
||||
y += rowHeight + spacing
|
||||
rowHeight = 0
|
||||
}
|
||||
x += size.width + spacing
|
||||
rowHeight = max(rowHeight, size.height)
|
||||
}
|
||||
return CGSize(width: maxWidth == .infinity ? x : maxWidth, height: y + rowHeight)
|
||||
}
|
||||
|
||||
func placeSubviews(in bounds: CGRect, proposal: ProposedViewSize, subviews: Subviews, cache: inout Void) {
|
||||
let maxWidth = bounds.width
|
||||
var x: CGFloat = 0, y: CGFloat = 0, rowHeight: CGFloat = 0
|
||||
for sub in subviews {
|
||||
let size = sub.sizeThatFits(.unspecified)
|
||||
if x + size.width > maxWidth, x > 0 {
|
||||
x = 0
|
||||
y += rowHeight + spacing
|
||||
rowHeight = 0
|
||||
}
|
||||
sub.place(at: CGPoint(x: bounds.minX + x, y: bounds.minY + y),
|
||||
anchor: .topLeading, proposal: ProposedViewSize(size))
|
||||
x += size.width + spacing
|
||||
rowHeight = max(rowHeight, size.height)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#Preview {
|
||||
HealthChatView(onClose: {})
|
||||
}
|
||||
130
康康/Features/Consultation/ConsultationAudioPlayer.swift
Normal file
130
康康/Features/Consultation/ConsultationAudioPlayer.swift
Normal file
@@ -0,0 +1,130 @@
|
||||
import SwiftUI
|
||||
import AVFoundation
|
||||
import Combine
|
||||
|
||||
/// Vault 内一段问诊录音的回放控件(2026-06-28)。播放/暂停 + 进度条 + 时间。
|
||||
/// 录音存在加密 Vault 里,App 前台即解锁可读;读不到(损坏/异常)显示占位,不崩。
|
||||
/// 不做拖拽 seek(降复杂度):进度条只读,demo 够用。
|
||||
struct ConsultationAudioPlayer: View {
|
||||
let asset: Asset
|
||||
|
||||
@State private var player: AVAudioPlayer?
|
||||
@State private var isPlaying = false
|
||||
@State private var currentTime: Double = 0
|
||||
@State private var duration: Double = 0
|
||||
@State private var loadFailed = false
|
||||
|
||||
/// 播放时刷新进度/时间;不播放时空转(开销可忽略)。AVAudioPlayer 播完 isPlaying 自然转 false。
|
||||
private let ticker = Timer.publish(every: 0.2, on: .main, in: .common).autoconnect()
|
||||
|
||||
var body: some View {
|
||||
VStack(alignment: .leading, spacing: 10) {
|
||||
HStack(spacing: 6) {
|
||||
Image(systemName: "waveform")
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.ink2)
|
||||
Text(String(appLoc: "问诊录音"))
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer()
|
||||
Text(String(appLoc: "本机存储"))
|
||||
.font(.tjScaled( 11)).foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
|
||||
if loadFailed {
|
||||
Text("录音无法读取")
|
||||
.font(.tjScaled( 13)).foregroundStyle(Tj.Palette.text3)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
} else {
|
||||
HStack(spacing: 12) {
|
||||
Button(action: togglePlay) {
|
||||
Image(systemName: isPlaying ? "pause.fill" : "play.fill")
|
||||
.font(.tjScaled( 16, weight: .bold))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
.frame(width: 40, height: 40)
|
||||
.background(Circle().fill(Tj.Palette.ink2))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
GeometryReader { geo in
|
||||
ZStack(alignment: .leading) {
|
||||
Capsule().fill(Tj.Palette.sand2)
|
||||
Capsule().fill(Tj.Palette.ink2)
|
||||
.frame(width: geo.size.width * progressFraction)
|
||||
}
|
||||
}
|
||||
.frame(height: 5)
|
||||
HStack {
|
||||
Text(Self.timeText(currentTime))
|
||||
Spacer()
|
||||
Text(Self.timeText(duration))
|
||||
}
|
||||
.font(.tjScaled( 11, design: .monospaced))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
.padding(14)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
.onAppear(perform: load)
|
||||
.onDisappear {
|
||||
player?.stop()
|
||||
isPlaying = false
|
||||
}
|
||||
.onReceive(ticker) { _ in
|
||||
guard let p = player, isPlaying else { return }
|
||||
currentTime = p.currentTime
|
||||
if !p.isPlaying { // 播完:复位
|
||||
isPlaying = false
|
||||
currentTime = 0
|
||||
p.currentTime = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var progressFraction: CGFloat {
|
||||
guard duration > 0 else { return 0 }
|
||||
return CGFloat(min(max(currentTime / duration, 0), 1))
|
||||
}
|
||||
|
||||
private func load() {
|
||||
guard player == nil, !loadFailed else { return }
|
||||
do {
|
||||
let url = try FileVault.shared.absoluteURL(forReading: asset.relativePath)
|
||||
let p = try AVAudioPlayer(contentsOf: url)
|
||||
p.prepareToPlay()
|
||||
player = p
|
||||
duration = p.duration
|
||||
} catch {
|
||||
loadFailed = true
|
||||
}
|
||||
}
|
||||
|
||||
private func togglePlay() {
|
||||
guard let p = player else { return }
|
||||
if isPlaying {
|
||||
p.pause()
|
||||
isPlaying = false
|
||||
} else {
|
||||
try? AVAudioSession.sharedInstance().setCategory(.playback, options: [])
|
||||
try? AVAudioSession.sharedInstance().setActive(true)
|
||||
p.play()
|
||||
isPlaying = true
|
||||
}
|
||||
}
|
||||
|
||||
private static func timeText(_ seconds: Double) -> String {
|
||||
let s = Int(seconds.rounded())
|
||||
return String(format: "%d:%02d", s / 60, s % 60)
|
||||
}
|
||||
}
|
||||
657
康康/Features/Consultation/ConsultationSheet.swift
Normal file
657
康康/Features/Consultation/ConsultationSheet.swift
Normal file
@@ -0,0 +1,657 @@
|
||||
import SwiftUI
|
||||
import SwiftData
|
||||
|
||||
/// 「记录问诊」录入 sheet(2026-06-28)。
|
||||
///
|
||||
/// 流程:开始录音 → 端侧实时转写(`ConsultationRecorder`)→ 停止 → AI 整理成问诊小结
|
||||
/// (`DiaryAssistService.organizeConsultation`)→ 审阅可编辑 → 保存为带「问诊」tag 的 DiaryEntry
|
||||
/// (录音作为 audio/m4a Asset 落加密 Vault)。
|
||||
///
|
||||
/// 守红线:UI 不直接调 AIRuntime(经 DiaryAssistService);识别 + 录音全程本机;
|
||||
/// AI 整理失败回退录音原文(#5);录音落盘尽力而为,失败也照常存文字。
|
||||
/// 本机不支持端侧识别(模拟器/老机型)→ 退化为手动文字录入,功能仍可用。
|
||||
struct ConsultationSheet: View {
|
||||
@Environment(\.modelContext) private var ctx
|
||||
@Environment(\.dismiss) private var dismiss
|
||||
|
||||
enum Phase: Equatable { case idle, recording, transcribing, organizing, review }
|
||||
@State private var phase: Phase = .idle
|
||||
|
||||
@State private var liveTranscript = ""
|
||||
@State private var recordingSeconds = 0
|
||||
/// 审阅区可编辑的问诊小结(整理稿;整理失败时是录音原文)。最终落库的就是它。
|
||||
@State private var note = ""
|
||||
/// 最终转写原话:「改用原话」回退用,也作整理失败兜底。
|
||||
@State private var rawTranscript = ""
|
||||
/// 录音临时文件(tmp 里的 m4a)。保存时 importFile 搬进 Vault;未保存则 onDisappear 删掉。
|
||||
@State private var audioTempURL: URL?
|
||||
@State private var createdAt: Date = .now
|
||||
@State private var decodeRate: Double = 0
|
||||
@State private var voiceNote: String?
|
||||
@State private var deniedAlert = false
|
||||
@State private var saved = false
|
||||
|
||||
@State private var recordTask: Task<Void, Never>?
|
||||
@State private var organizeTask: Task<Void, Never>?
|
||||
@State private var watchdog: Task<Void, Never>?
|
||||
/// 必须 @State:struct View 重建时普通 let 会换新实例,导致 stop() 落在没在录音的新实例上、
|
||||
/// 而真正录音的老实例关不掉麦克风悬挂(同 DiaryQuickSheet.dictation 的注释)。
|
||||
@State private var recorder = ConsultationRecorder()
|
||||
@FocusState private var noteFocused: Bool
|
||||
|
||||
/// 录音上限 10 分钟(超时由看门狗触发停止并整理)。
|
||||
private let maxSeconds = 600
|
||||
|
||||
/// 本机不支持端侧识别 → 手动录入模式(进入即可打字,无录音)。
|
||||
private var manualMode: Bool { !ConsultationRecorder.isAvailable }
|
||||
|
||||
private var canSave: Bool {
|
||||
!note.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
|
||||
}
|
||||
|
||||
var body: some View {
|
||||
VStack(spacing: 0) {
|
||||
grabber
|
||||
header
|
||||
ScrollView(showsIndicators: false) {
|
||||
VStack(alignment: .leading, spacing: 16) {
|
||||
if let note = voiceNote { noteBanner(note) }
|
||||
phaseContent
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.top, 4)
|
||||
.padding(.bottom, 8)
|
||||
}
|
||||
.scrollDismissesKeyboard(.interactively)
|
||||
bottomBar
|
||||
}
|
||||
.background(
|
||||
Tj.Palette.sand
|
||||
.clipShape(RoundedRectangle(cornerRadius: Tj.Radius.xl, style: .continuous))
|
||||
.ignoresSafeArea(edges: .bottom)
|
||||
)
|
||||
.presentationDetents([.large])
|
||||
.presentationDragIndicator(.hidden)
|
||||
.presentationBackground(Tj.Palette.sand)
|
||||
.presentationCornerRadius(Tj.Radius.xl)
|
||||
.onDisappear(perform: cleanup)
|
||||
.alert(String(appLoc: "需要麦克风与语音识别权限"), isPresented: $deniedAlert) {
|
||||
Button(String(appLoc: "前往设置")) {
|
||||
if let url = URL(string: UIApplication.openSettingsURLString) {
|
||||
UIApplication.shared.open(url)
|
||||
}
|
||||
}
|
||||
Button(String(appLoc: "取消"), role: .cancel) {}
|
||||
} message: {
|
||||
Text("问诊录音全程在本机完成,声音和文字都不会上传。请在设置中允许麦克风和语音识别。")
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Header
|
||||
|
||||
private var grabber: some View {
|
||||
Capsule()
|
||||
.fill(Tj.Palette.line)
|
||||
.frame(width: 40, height: 4)
|
||||
.padding(.top, 10)
|
||||
.padding(.bottom, 14)
|
||||
}
|
||||
|
||||
private var header: some View {
|
||||
HStack(alignment: .center, spacing: 12) {
|
||||
Button { dismiss() } label: {
|
||||
Image(systemName: "xmark")
|
||||
.font(.tjScaled( 15, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
.frame(width: 32, height: 32)
|
||||
.background(Circle().fill(Tj.Palette.sand2))
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
VStack(alignment: .leading, spacing: 2) {
|
||||
Text("记录问诊")
|
||||
.font(.tjH2())
|
||||
.foregroundStyle(Tj.Palette.text)
|
||||
Text("录音 → 本机转写 → AI 整理成问诊小结")
|
||||
.font(.tjScaled( 11))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
Spacer()
|
||||
TjLockChip()
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.bottom, 12)
|
||||
}
|
||||
|
||||
// MARK: - Phase content
|
||||
|
||||
@ViewBuilder
|
||||
private var phaseContent: some View {
|
||||
switch phase {
|
||||
case .idle:
|
||||
if manualMode { manualEntry } else { idleStart }
|
||||
case .recording:
|
||||
recordingCard
|
||||
case .transcribing:
|
||||
transcribingCard
|
||||
case .organizing:
|
||||
organizingCard
|
||||
case .review:
|
||||
reviewArea
|
||||
}
|
||||
}
|
||||
|
||||
/// 待开始:大录音按钮 + 三条说明(本机识别 / 可回放 / 不上传)。
|
||||
private var idleStart: some View {
|
||||
VStack(spacing: 18) {
|
||||
Button(action: startRecording) {
|
||||
VStack(spacing: 10) {
|
||||
ZStack {
|
||||
Circle().fill(Tj.Palette.ink2)
|
||||
.frame(width: 76, height: 76)
|
||||
.shadow(color: Tj.Palette.ink2.opacity(0.3), radius: 12, y: 4)
|
||||
Image(systemName: "mic.fill")
|
||||
.font(.tjScaled( 30, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
}
|
||||
Text("开始录音")
|
||||
.font(.tjScaled( 15, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.ink2)
|
||||
}
|
||||
.frame(maxWidth: .infinity)
|
||||
.padding(.vertical, 22)
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
|
||||
VStack(alignment: .leading, spacing: 10) {
|
||||
bullet("stethoscope", String(appLoc: "看医生时录下来,事后不怕记不全"))
|
||||
bullet("waveform", String(appLoc: "录音结束后在本机转写,原声存进加密档案"))
|
||||
bullet("sparkles", String(appLoc: "康康自动整理成主诉/医生建议/复查小节"))
|
||||
}
|
||||
.padding(16)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
|
||||
Text("仅供个人记录,不构成诊断或用药建议。")
|
||||
.font(.tjScaled( 11)).foregroundStyle(Tj.Palette.text3)
|
||||
}
|
||||
.padding(.top, 6)
|
||||
}
|
||||
|
||||
private func bullet(_ icon: String, _ text: String) -> some View {
|
||||
HStack(alignment: .top, spacing: 10) {
|
||||
Image(systemName: icon)
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.ink2)
|
||||
.frame(width: 20)
|
||||
Text(text)
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// 录音中:计时 + 实时字幕 + 结束按钮。
|
||||
private var recordingCard: some View {
|
||||
VStack(alignment: .leading, spacing: 12) {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "waveform")
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
Text("正在录音 · 声音不上传")
|
||||
.font(.tjScaled( 13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
Text(Self.timeText(recordingSeconds))
|
||||
.font(.tjScaled( 13, design: .monospaced))
|
||||
.foregroundStyle(recordingSeconds >= maxSeconds - 30 ? Tj.Palette.brick : Tj.Palette.text3)
|
||||
}
|
||||
|
||||
// 离线转写(SenseVoice)无实时字幕:录音中只给声纹动效 + 说明,结束后整段转写。
|
||||
VStack(spacing: 10) {
|
||||
Image(systemName: "waveform")
|
||||
.font(.tjScaled( 44, weight: .regular))
|
||||
.foregroundStyle(Tj.Palette.ink2)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
Text("正在录音,结束后会在本机把整段录音转成文字")
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.multilineTextAlignment(.center)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
}
|
||||
.frame(maxWidth: .infinity, minHeight: 150)
|
||||
.padding(.vertical, 8)
|
||||
|
||||
Button(action: stopAndOrganize) {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "stop.circle.fill")
|
||||
Text("结束并整理")
|
||||
}
|
||||
.font(.tjScaled( 15, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.paper)
|
||||
.frame(maxWidth: .infinity)
|
||||
.padding(.vertical, 13)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.brick)
|
||||
)
|
||||
.contentShape(Rectangle())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
.padding(14)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
}
|
||||
|
||||
/// 录音转写中:整段录音离线转文字(优先本地 SenseVoice,不可用回退本机识别)。无实时字幕,统一转写。
|
||||
private var transcribingCard: some View {
|
||||
VStack(alignment: .leading, spacing: 12) {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "waveform")
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
Text(SenseVoiceASRService.isAvailable
|
||||
? String(appLoc: "正在转写录音 · 本地 SenseVoice")
|
||||
: String(appLoc: "正在转写录音 · 本机识别"))
|
||||
.font(.tjScaled( 13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
Text("录音已结束,正在本机把整段录音转成文字,请稍候…")
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
AIFlowBar()
|
||||
}
|
||||
.padding(14)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
}
|
||||
|
||||
/// AI 整理中:呼吸流光 + 置灰原话预览 + 取消(取消即用原话)。
|
||||
private var organizingCard: some View {
|
||||
VStack(alignment: .leading, spacing: 12) {
|
||||
HStack(spacing: 8) {
|
||||
Image(systemName: "sparkles")
|
||||
.font(.tjScaled( 13, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.brick)
|
||||
.symbolEffect(.pulse, options: .repeating)
|
||||
Text(decodeRate > 0
|
||||
? String(format: String(appLoc: "正在整理问诊小结 · %.1f tok/s"), decodeRate)
|
||||
: String(appLoc: "正在整理问诊小结 · 本地推理"))
|
||||
.font(.tjScaled( 13, weight: .medium))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
Button(String(appLoc: "用原话")) { cancelOrganize() }
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
Text(rawTranscript)
|
||||
.font(.tjScaled( 14))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
.lineLimit(6)
|
||||
AIFlowBar()
|
||||
}
|
||||
.padding(14)
|
||||
.frame(maxWidth: .infinity, alignment: .leading)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.md, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.lineSoft, lineWidth: 1)
|
||||
)
|
||||
}
|
||||
|
||||
/// 审阅:录音已附提示 + 可编辑小结 + (改用原话 / 重新整理) + 时间。
|
||||
private var reviewArea: some View {
|
||||
VStack(alignment: .leading, spacing: 14) {
|
||||
if audioTempURL != nil {
|
||||
HStack(spacing: 6) {
|
||||
Image(systemName: "waveform.circle.fill")
|
||||
.font(.tjScaled( 13))
|
||||
.foregroundStyle(Tj.Palette.ink2)
|
||||
Text("录音已保存,可在记录详情里回放")
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
}
|
||||
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
HStack {
|
||||
Text(String(appLoc: "问诊小结"))
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.tracking(0.3)
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
Spacer()
|
||||
if decodeRate > 0 {
|
||||
Text(String(format: "%.1f tok/s", decodeRate))
|
||||
.font(.tjScaled( 10, design: .monospaced))
|
||||
.foregroundStyle(Tj.Palette.leaf)
|
||||
}
|
||||
}
|
||||
TextField(String(appLoc: "整理后的问诊小结,可在这里修改…"),
|
||||
text: $note, axis: .vertical)
|
||||
.font(.tjScaled( 15))
|
||||
.lineLimit(6...16)
|
||||
.focused($noteFocused)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 12)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1)
|
||||
)
|
||||
|
||||
HStack(spacing: 10) {
|
||||
if !rawTranscript.isEmpty && note != rawTranscript {
|
||||
reviewChip("arrow.uturn.backward", String(appLoc: "改用原话")) {
|
||||
withAnimation(.snappy(duration: 0.18)) { note = rawTranscript }
|
||||
}
|
||||
}
|
||||
if !rawTranscript.isEmpty {
|
||||
reviewChip("arrow.clockwise", String(appLoc: "重新整理")) { reorganize() }
|
||||
}
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
}
|
||||
|
||||
VStack(alignment: .leading, spacing: 8) {
|
||||
Text(String(appLoc: "时间"))
|
||||
.font(.tjScaled( 12, weight: .semibold))
|
||||
.tracking(0.3)
|
||||
.foregroundStyle(Tj.Palette.text2)
|
||||
DatePicker("", selection: $createdAt, in: ...Date.now)
|
||||
.datePickerStyle(.compact)
|
||||
.labelsHidden()
|
||||
}
|
||||
|
||||
AIDisclaimerFooter()
|
||||
}
|
||||
}
|
||||
|
||||
private func reviewChip(_ icon: String, _ label: String, action: @escaping () -> Void) -> some View {
|
||||
Button(action: action) {
|
||||
HStack(spacing: 4) {
|
||||
Image(systemName: icon).font(.tjScaled( 10, weight: .semibold))
|
||||
Text(label).font(.tjScaled( 11, weight: .semibold))
|
||||
}
|
||||
.foregroundStyle(Tj.Palette.ink)
|
||||
.padding(.horizontal, 10)
|
||||
.padding(.vertical, 5)
|
||||
.background(Capsule().strokeBorder(Tj.Palette.line, lineWidth: 1))
|
||||
.contentShape(Capsule())
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
}
|
||||
|
||||
/// 手动模式(本机不支持识别):直接打字录入问诊小结。
|
||||
private var manualEntry: some View {
|
||||
VStack(alignment: .leading, spacing: 12) {
|
||||
HStack(spacing: 6) {
|
||||
Image(systemName: "info.circle")
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Text("本机暂不支持录音转写,可手动记录这次问诊")
|
||||
.font(.tjScaled( 12))
|
||||
.foregroundStyle(Tj.Palette.text3)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
TextField(String(appLoc: "记录这次问诊:主诉、医生说了什么、用药、复查…"),
|
||||
text: $note, axis: .vertical)
|
||||
.font(.tjScaled( 15))
|
||||
.lineLimit(6...16)
|
||||
.focused($noteFocused)
|
||||
.padding(.horizontal, 14)
|
||||
.padding(.vertical, 12)
|
||||
.background(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.fill(Tj.Palette.paper)
|
||||
)
|
||||
.overlay(
|
||||
RoundedRectangle(cornerRadius: Tj.Radius.sm, style: .continuous)
|
||||
.strokeBorder(Tj.Palette.line, lineWidth: 1)
|
||||
)
|
||||
DatePicker("", selection: $createdAt, in: ...Date.now)
|
||||
.datePickerStyle(.compact)
|
||||
.labelsHidden()
|
||||
}
|
||||
.padding(.top, 6)
|
||||
}
|
||||
|
||||
// MARK: - Bottom bar
|
||||
|
||||
@ViewBuilder
|
||||
private var bottomBar: some View {
|
||||
switch phase {
|
||||
case .idle where manualMode:
|
||||
HStack(spacing: 8) {
|
||||
Button(String(appLoc: "取消")) { dismiss() }
|
||||
.buttonStyle(TjGhostButton(height: 44, fontSize: 15, horizontalPadding: 14))
|
||||
Button(String(appLoc: "保存")) { save() }
|
||||
.buttonStyle(TjPrimaryButton(height: 44, fontSize: 15, horizontalPadding: 14))
|
||||
.disabled(!canSave)
|
||||
.opacity(canSave ? 1 : 0.4)
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 14)
|
||||
case .idle:
|
||||
// 非手动模式的待开始页:开始录音是内容区大按钮,关闭走 header 的 ✕,底部不再占一行。
|
||||
EmptyView()
|
||||
case .review:
|
||||
HStack(spacing: 8) {
|
||||
Button(String(appLoc: "重新录音")) { restartRecording() }
|
||||
.buttonStyle(TjGhostButton(height: 44, fontSize: 15, horizontalPadding: 14))
|
||||
Button(String(appLoc: "保存")) { save() }
|
||||
.buttonStyle(TjPrimaryButton(height: 44, fontSize: 15, horizontalPadding: 14))
|
||||
.disabled(!canSave)
|
||||
.opacity(canSave ? 1 : 0.4)
|
||||
}
|
||||
.padding(.horizontal, 20)
|
||||
.padding(.vertical, 14)
|
||||
case .recording, .transcribing, .organizing:
|
||||
EmptyView()
|
||||
}
|
||||
}
|
||||
|
||||
private func noteBanner(_ text: String) -> some View {
|
||||
HStack(spacing: 6) {
|
||||
Image(systemName: "exclamationmark.circle.fill")
|
||||
.font(.tjScaled( 11)).foregroundStyle(Tj.Palette.amber)
|
||||
Text(text).font(.tjScaled( 12)).foregroundStyle(Tj.Palette.text2)
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Actions
|
||||
|
||||
private func startRecording() {
|
||||
voiceNote = nil
|
||||
noteFocused = false
|
||||
recordTask = Task { @MainActor in
|
||||
guard await recorder.requestAuthorization() else {
|
||||
deniedAlert = true
|
||||
return
|
||||
}
|
||||
do {
|
||||
liveTranscript = ""
|
||||
recordingSeconds = 0
|
||||
try recorder.start { partial in liveTranscript = partial }
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .recording }
|
||||
watchdog = Task { @MainActor in
|
||||
while !Task.isCancelled {
|
||||
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
||||
guard !Task.isCancelled, phase == .recording else { return }
|
||||
recordingSeconds += 1
|
||||
if recordingSeconds >= maxSeconds { stopAndOrganize(); return }
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
#if DEBUG
|
||||
print("[Consultation] recorder start failed: \(error)")
|
||||
#endif
|
||||
voiceNote = String(appLoc: "无法开始录音,请检查麦克风 / 语音识别权限")
|
||||
phase = .idle
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func stopAndOrganize() {
|
||||
guard phase == .recording else { return }
|
||||
watchdog?.cancel()
|
||||
organizeTask = Task { @MainActor in
|
||||
let result = await recorder.stop()
|
||||
audioTempURL = result.audioTempURL
|
||||
|
||||
// 系统端侧识别(SFSpeech,录音时顺带跑)作为 SenseVoice 不可用 / 失败时的自动回退稿。
|
||||
var sfFallback = result.transcript.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if sfFallback.isEmpty {
|
||||
sfFallback = liveTranscript.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
}
|
||||
|
||||
// 离线转写整段录音:优先本地 SenseVoice(MNN),不可用 / 失败 / 空 → 回退 SFSpeech(守红线 #5)。
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .transcribing }
|
||||
var transcript = ""
|
||||
if SenseVoiceASRService.isAvailable, let url = result.audioTempURL {
|
||||
transcript = (try? await SenseVoiceASRService.shared.transcribe(audioFileURL: url)) ?? ""
|
||||
}
|
||||
if Task.isCancelled { return }
|
||||
if transcript.isEmpty { transcript = sfFallback }
|
||||
rawTranscript = transcript
|
||||
|
||||
guard !transcript.isEmpty else {
|
||||
// 没听清:进审阅让用户手动补;录音(若有)仍可保存。
|
||||
voiceNote = String(appLoc: "没听清,可手动补充或重新录音")
|
||||
note = ""
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .review }
|
||||
return
|
||||
}
|
||||
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .organizing }
|
||||
await runOrganize(transcript: transcript)
|
||||
}
|
||||
}
|
||||
|
||||
/// 调用 LLM 整理;成功填小结,取消/失败回退原话(红线 #5)。供首次整理与「重新整理」复用。
|
||||
private func runOrganize(transcript: String) async {
|
||||
do {
|
||||
let organized = try await DiaryAssistService.shared.organizeConsultation(transcript: transcript)
|
||||
guard !Task.isCancelled else { return }
|
||||
note = organized.text
|
||||
decodeRate = organized.decodeRate
|
||||
} catch is CancellationError {
|
||||
return // cancelOrganize 已处理回退
|
||||
} catch {
|
||||
note = transcript
|
||||
voiceNote = String(appLoc: "AI 整理没成功,已填入录音原文")
|
||||
}
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .review }
|
||||
}
|
||||
|
||||
private func cancelOrganize() {
|
||||
guard phase == .organizing else { return }
|
||||
organizeTask?.cancel()
|
||||
note = rawTranscript
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .review }
|
||||
}
|
||||
|
||||
/// 审阅区「重新整理」:从原话再跑一遍 LLM(首次整理结果不满意时)。
|
||||
private func reorganize() {
|
||||
guard !rawTranscript.isEmpty else { return }
|
||||
organizeTask?.cancel()
|
||||
decodeRate = 0
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .organizing }
|
||||
organizeTask = Task { @MainActor in
|
||||
await runOrganize(transcript: rawTranscript)
|
||||
}
|
||||
}
|
||||
|
||||
/// 审阅区「重新录音」:丢弃当前录音/小结,回到待开始。
|
||||
private func restartRecording() {
|
||||
organizeTask?.cancel()
|
||||
if let url = audioTempURL { try? FileManager.default.removeItem(at: url) }
|
||||
audioTempURL = nil
|
||||
note = ""
|
||||
rawTranscript = ""
|
||||
liveTranscript = ""
|
||||
decodeRate = 0
|
||||
voiceNote = nil
|
||||
withAnimation(.snappy(duration: 0.2)) { phase = .idle }
|
||||
}
|
||||
|
||||
private func save() {
|
||||
let content = note.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !content.isEmpty else { return }
|
||||
let entry = DiaryEntry(content: content,
|
||||
createdAt: createdAt,
|
||||
tags: [DiaryEntry.consultationTag])
|
||||
ctx.insert(entry)
|
||||
// 录音搬进加密 Vault 并挂为 Asset(尽力而为:失败也照常存文字)。
|
||||
if let temp = audioTempURL,
|
||||
let savedAsset = try? FileVault.shared.importFile(at: temp, preferredExtension: "m4a") {
|
||||
let asset = Asset(relativePath: savedAsset.relativePath,
|
||||
mimeType: "audio/m4a",
|
||||
bytes: savedAsset.bytes)
|
||||
ctx.insert(asset)
|
||||
entry.assets.append(asset)
|
||||
audioTempURL = nil // 已搬走,cleanup 不再删
|
||||
}
|
||||
try? ctx.save()
|
||||
saved = true
|
||||
dismiss()
|
||||
}
|
||||
|
||||
/// 关闭时收尾:停录音、撤任务、删未保存的录音临时文件。
|
||||
private func cleanup() {
|
||||
recordTask?.cancel()
|
||||
organizeTask?.cancel()
|
||||
watchdog?.cancel()
|
||||
recorder.abort()
|
||||
if !saved, let url = audioTempURL {
|
||||
try? FileManager.default.removeItem(at: url)
|
||||
}
|
||||
}
|
||||
|
||||
private static func timeText(_ seconds: Int) -> String {
|
||||
String(format: "%d:%02d", seconds / 60, seconds % 60)
|
||||
}
|
||||
}
|
||||
|
||||
#Preview {
|
||||
ConsultationSheet()
|
||||
.modelContainer(for: [DiaryEntry.self, Asset.self], inMemory: true)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user