feat(AI): MNN 4B 多模态一肩挑文本+视觉,合并为单模型(MLX 仍兜底)
利用 Qwen3.5-4B-MNN 本身是多模态(含 visual.mnn),让同一个 MNN 模型 同时做文本生成与拍照识别 → MNN 路径只需下 1 个模型(7.4GB→2.64GB)。 MLX(.llm/.vl)保留作兜底,尤其开发机 iPhone 15 Pro(A17 无 SME2)。 - MNN.xcframework 重建为 OMNI(MNN_BUILD_LLM_OMNI=ON,加 OpenCV 图像解码); 构建脚本同步加 OMNI flag - MNNLLMBridge.analyzeImages:把图片路径拼成 <img>路径</img> 标签 + response, Omni 内部 CV::imread 加载(无需桥接 include OpenCV);与 generateText 共用 runResponse - MNNBackend.analyze:detached 线程跑 blocking VL 调用,聚合为字符串 - AIRuntime:engine=.mnn 且就绪时,prepareVL→prepareMNN、analyzeReport→mnn.analyze; 否则回退 MLX VL device + 模拟器 BUILD SUCCEEDED,0 error,OMNI 框架链接干净。 VL 实际识别质量需真机用化验单 A/B(demo 核心)。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -47,12 +47,34 @@ actor MNNBackend {
|
||||
}
|
||||
}
|
||||
|
||||
/// 图→文(VL)。当前 MNN 文本构建未含 OMNI,直接抛错让上层回退 MLX VL。
|
||||
func analyze(imageURLs: [URL], prompt: String, maxTokens: Int) throws -> String {
|
||||
throw AIRuntimeError.inferenceFailed("MNN 当前构建不支持 VL(需 OMNI)")
|
||||
/// 图→文(VL)。一次性收集(JSON 抽取不需流式)。桥接里把图片路径拼成 <img> 标签,
|
||||
/// MNN Omni 内部 imread 加载(需 OMNI 框架);blocking 调用放 detached 线程。
|
||||
func analyze(imageURLs: [URL], prompt: String, maxTokens: Int) async throws -> String {
|
||||
guard let bridge else { throw AIRuntimeError.notReady }
|
||||
let paths = imageURLs.map(\.path)
|
||||
let box = MNNUncheckedBox(bridge)
|
||||
return try await withCheckedThrowingContinuation { cont in
|
||||
Task.detached(priority: .userInitiated) {
|
||||
let sink = MNNTextSink()
|
||||
do {
|
||||
_ = try box.value.analyzeImages(paths, prompt: prompt, maxTokens: Int32(maxTokens)) { piece in
|
||||
sink.append(piece)
|
||||
}
|
||||
cont.resume(returning: sink.text)
|
||||
} catch {
|
||||
cont.resume(throwing: AIRuntimeError.inferenceFailed(error.localizedDescription))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 单线程串行回调聚合文本,无竞争。
|
||||
private nonisolated final class MNNTextSink: @unchecked Sendable {
|
||||
private(set) var text = ""
|
||||
func append(_ s: String) { text += s }
|
||||
}
|
||||
|
||||
/// 把非 Sendable 的 ObjC 桥对象安全带过 detached 边界。
|
||||
/// 安全性来自 `AIRuntime` 闸门:同一时刻只有一个生成在跑,桥不会被并发访问。
|
||||
private nonisolated struct MNNUncheckedBox<T>: @unchecked Sendable {
|
||||
|
||||
Reference in New Issue
Block a user