feat(AI): MNN 4B 多模态一肩挑文本+视觉,合并为单模型(MLX 仍兜底)
利用 Qwen3.5-4B-MNN 本身是多模态(含 visual.mnn),让同一个 MNN 模型 同时做文本生成与拍照识别 → MNN 路径只需下 1 个模型(7.4GB→2.64GB)。 MLX(.llm/.vl)保留作兜底,尤其开发机 iPhone 15 Pro(A17 无 SME2)。 - MNN.xcframework 重建为 OMNI(MNN_BUILD_LLM_OMNI=ON,加 OpenCV 图像解码); 构建脚本同步加 OMNI flag - MNNLLMBridge.analyzeImages:把图片路径拼成 <img>路径</img> 标签 + response, Omni 内部 CV::imread 加载(无需桥接 include OpenCV);与 generateText 共用 runResponse - MNNBackend.analyze:detached 线程跑 blocking VL 调用,聚合为字符串 - AIRuntime:engine=.mnn 且就绪时,prepareVL→prepareMNN、analyzeReport→mnn.analyze; 否则回退 MLX VL device + 模拟器 BUILD SUCCEEDED,0 error,OMNI 框架链接干净。 VL 实际识别质量需真机用化验单 A/B(demo 核心)。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -255,6 +255,11 @@ actor AIRuntime {
|
||||
|
||||
/// 加载 VL 模型。幂等,首调真正 load。
|
||||
func prepareVL() async throws {
|
||||
// 选了 MNN 且多模态模型就绪:VL 复用同一个 MNN 模型(文本+视觉一肩挑),走 prepareMNN。
|
||||
if InferenceEngine.current == .mnn, ModelStore.shared.isComplete(for: .mnnLLM) {
|
||||
try await prepareMNN()
|
||||
return
|
||||
}
|
||||
while vlStatus == .loading {
|
||||
try await Task.sleep(nanoseconds: 80_000_000)
|
||||
}
|
||||
@@ -314,6 +319,16 @@ actor AIRuntime {
|
||||
func analyzeReport(imageURLs: [URL],
|
||||
prompt: String,
|
||||
maxTokens: Int = 512) async throws -> String {
|
||||
// 选了 MNN 且就绪:图→文走同一个 MNN 多模态模型。
|
||||
if InferenceEngine.current == .mnn, mnnStatus == .ready {
|
||||
await acquireGate()
|
||||
defer { releaseGate() }
|
||||
do {
|
||||
return try await mnn.analyze(imageURLs: imageURLs, prompt: prompt, maxTokens: maxTokens)
|
||||
} catch {
|
||||
throw AIRuntimeError.inferenceFailed("\(error)")
|
||||
}
|
||||
}
|
||||
guard vlStatus == .ready, let session = vlSession else {
|
||||
throw AIRuntimeError.notReady
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user