Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,24 @@ jobs:
run: test -x apps/desktop/resources/backend/techspar-api && test -f apps/desktop/resources/backend/onnxruntime/onnxruntime_binding.node
- name: 校验 Electron 打包配置
run: bun run --cwd apps/desktop pack

voice-input-windows:
runs-on: windows-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14"
- name: 安装锁定依赖和 Electron 运行时
run: |
bun install --frozen-lockfile
node apps/desktop/node_modules/electron/install.js
- name: Windows Electron 合成音频验收
run: bun run smoke:voice
- uses: actions/upload-artifact@v4
if: always()
with:
name: windows-voice-input-smoke
path: dist/voice-input-smoke/
if-no-files-found: ignore
Binary file added docs/validation/issue-95-recording.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 52 additions & 0 deletions docs/validation/issue-95-voice-input.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Issue #95:模拟面试语音输入验收

## 行为与修复

模拟面试的麦克风用于录音后转成文字:点击开始,再次点击停止,识别结果追加到回答框。它不播放录音或朗读面试官的问题。

- 录音会话统一管理权限请求、录音器、音频轨道、音量检测和转写请求。
- 设备断开、录音器意外停止或报错后释放资源,显示可重试的提示。
- 取消、离开页面、结束面试或切换题目后,不启动迟到的录音,也不回填迟到的转写。
- 转写超过 60 秒会取消请求并恢复可重试状态;保留全局鉴权和额度错误处理。
- 提供输入设备选择、实际采集设备名称和音量反馈,帮助区分权限允许与实际有声音。
- 安静的 Opus 录音可能小于 1000 字节,结合实际录音时长判断过短输入;不把音量阈值当成人声识别结果。
- 编码格式与上传扩展名保持一致,超过社区版短语音 7 MiB 上限时给出提示。

## 自动验收

```sh
bun test ./frontend/tests/voice-input.test.tsx
bun run check
bun run smoke:voice
```

回归测试覆盖正常转写、权限拒绝、设备缺失、设备占用、延迟授权、重复点击、启动失败、设备结束、自然停止、录音器错误、录音大小、编码兼容、空结果、HTTP/网络错误、超时、取消、页面卸载、音量显示及设备选择。

`smoke:voice` 使用真实 Electron/Chromium、仓库的权限处理和实际 Interview 页面,注入 16 kHz 合成 WAV;ASR 使用本地模拟接口,不采集物理麦克风、不调用付费服务。

Windows x64 本地验收环境:Electron 43.4.1 / Chromium 150.0.7871.224。

| 输入/故障 | 验证结果 |
| --- | --- |
| 有声输入,简历面试与专项训练 | 音量变化、WebM 上传、文字回填、轨道释放 |
| 静音输入,简历面试与专项训练 | 显示无声音、允许低字节数的有效录音上传、空转写提示、不回填文字 |
| HTTP 500、空转写 | 显示错误,同一页面重新录音后成功 |
| 真实 Chromium 音频轨道停止 | 退出录音状态,显示中断提示 |
| 转写中取消 | 恢复可操作状态,迟到结果不回填 |

截图及上传字节记录输出至 `dist/voice-input-smoke/`。CI 的 `voice-input-windows` 任务重复运行该验收并保存产物。

![简历面试中的输入设备、采集状态和音量](issue-95-recording.png)

## 发布后实机确认

上述测试证明代码和合成音频链路正常;它不替代原反馈用户的物理麦克风、蓝牙驱动和官方云端 ASR 验收。官方安装包还需要同步相应前端修复。

1. 在受影响的 Windows 电脑上打开包含修复的客户端,选择实际麦克风。
2. 说话时确认设备名称及音量变化;无变化时检查输入音量、静音开关和设备连接。
3. 录制一句话,停止后确认文字回填;分别验证简历面试和专项训练。
4. 录音中拔掉设备,确认出现提示;重新接入后可再次录音。
5. 验证切换题目、取消和离开页面后麦克风释放,旧转写不会进入新回答。
6. 若仍失败,记录客户端版本、所选设备、可见错误和 `/api/transcribe` 的状态码,对照服务端请求日志定位。

在原用户的物理设备和官方服务尚未确认前,PR 使用 `Related to #95`,不自动关闭原反馈。
8 changes: 6 additions & 2 deletions frontend/src/api/interview.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@ export { authFetch } from "./client";
// ── Speech-to-text ──

export async function transcribeAudio(
audioBlob: Blob
audioBlob: Blob,
signal?: AbortSignal,
): Promise<ApiResponse<"/api/transcribe", "post">> {
const form = new FormData();
form.append("file", audioBlob, "recording.webm");
const extension = audioBlob.type.startsWith("audio/ogg") ? "ogg"
: audioBlob.type.startsWith("audio/mp4") ? "m4a" : "webm";
form.append("file", audioBlob, `recording.${extension}`);
const res = await authFetch(`${API_BASE}/transcribe`, {
method: "POST",
body: form,
signal,
});
if (!res.ok) throw new Error(await res.text());
return res.json();
Expand Down
42 changes: 42 additions & 0 deletions frontend/src/components/VoiceInputFeedback.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import { useId } from "react";
import type useVoiceInput from "../hooks/useVoiceInput";

export default function VoiceInputFeedback({ voice }: { voice: ReturnType<typeof useVoiceInput> }) {
const id = useId();
if (!voice.isSupported) return null;
return (
<div className="w-full max-w-3xl space-y-2">
<div className="flex flex-wrap items-center gap-2 text-xs text-dim">
<label htmlFor={id}>输入麦克风</label>
<select id={id} value={voice.microphoneId} disabled={voice.isBusy}
onChange={(event) => voice.setMicrophoneId(event.target.value)}
className="max-w-64 rounded border border-border bg-surface px-2 py-1">
<option value="">系统默认麦克风</option>
{voice.microphoneId && !voice.microphones.some((device) => device.deviceId === voice.microphoneId) && (
<option value={voice.microphoneId}>所选麦克风已断开,请重新选择</option>
)}
{voice.microphones.filter((device) => device.deviceId && device.deviceId !== "default")
.map((device, index) => <option key={device.deviceId} value={device.deviceId}>{device.label || `麦克风 ${index + 1}`}</option>)}
</select>
{voice.isBusy && <button type="button" onClick={voice.cancel} className="text-text underline">取消语音输入</button>}
{voice.error && window.techsparDesktop?.openAudioSettings && (
<button type="button" onClick={() => {
void window.techsparDesktop?.openAudioSettings?.("microphone").catch(() => {});
}} className="text-text underline">打开麦克风设置</button>
)}
</div>
{voice.isStarting && <p role="status" className="text-xs text-dim">正在请求麦克风权限,可在授权窗口中允许访问。</p>}
{voice.isListening && (
<div className="space-y-1 text-xs text-dim">
<p role="status">正在录音 · {voice.inputLabel}。再次点击麦克风停止并转成文字。</p>
{voice.level !== null && <div className="flex items-center gap-2">
<meter aria-label="麦克风输入音量" min={0} max={1} value={voice.level} className="w-28" />
<span>{voice.heard ? "已检测到声音" : "暂未检测到声音,请说话并检查输入音量或静音开关。"}</span>
</div>}
</div>
)}
{voice.isTranscribing && <p role="status" className="text-xs text-dim">正在转成文字,请稍候。</p>}
{voice.error && <p role="alert" className="text-sm text-red">{voice.error}</p>}
</div>
);
}
189 changes: 55 additions & 134 deletions frontend/src/hooks/useVoiceInput.ts
Original file line number Diff line number Diff line change
@@ -1,152 +1,73 @@
import { useState, useRef, useCallback, useEffect } from "react";
import { transcribeAudio } from "../api/interview";
import { HandledApiError } from "../api/client";
import { VoiceInputSession, initialVoiceInputState } from "../lib/voice-input";

interface VoiceInputOptions {
onResult?: (text: string) => void;
}

export default function useVoiceInput({ onResult }: VoiceInputOptions = {}) {
const [isListening, setIsListening] = useState(false);
const [isTranscribing, setIsTranscribing] = useState(false);
const [error, setError] = useState("");
const mediaRecorderRef = useRef<MediaRecorder | null>(null);
const chunksRef = useRef<Blob[]>([]);
const streamRef = useRef<MediaStream | null>(null);
const [state, setState] = useState(initialVoiceInputState);
const [microphones, setMicrophones] = useState<MediaDeviceInfo[]>([]);
const [microphoneId, setMicrophoneId] = useState("");
const sessionRef = useRef<VoiceInputSession | null>(null);
const onResultRef = useRef(onResult);
const refreshDevicesRef = useRef<() => void>(() => {});

useEffect(() => {
onResultRef.current = onResult;
}, [onResult]);

// Check if getUserMedia is available
const isSupported =
typeof navigator !== "undefined" && !!navigator.mediaDevices?.getUserMedia &&
typeof MediaRecorder !== "undefined";

const stopListening = useCallback(() => {
setIsListening(false);

const recorder = mediaRecorderRef.current;
if (!recorder || recorder.state === "inactive") return;
setIsTranscribing(true);

// Stop recording — triggers ondataavailable + onstop
recorder.onstop = async () => {
// Build audio blob from chunks
const blob = new Blob(chunksRef.current, { type: recorder.mimeType });
chunksRef.current = [];

// Release microphone
streamRef.current?.getTracks().forEach((t) => t.stop());
streamRef.current = null;
mediaRecorderRef.current = null;
useEffect(() => { onResultRef.current = onResult; }, [onResult]);

// Very small recordings may not contain enough audio to transcribe.
if (blob.size < 1000) {
setError("录音内容太少,请说完一句话后再点击停止。");
setIsTranscribing(false);
return;
}
const isSupported = typeof navigator !== "undefined" &&
!!navigator.mediaDevices?.getUserMedia && typeof MediaRecorder !== "undefined";

// Send to backend for transcription
useEffect(() => {
let mounted = true;
let revision = 0;
const media = navigator.mediaDevices;
const refresh = async () => {
const current = ++revision;
try {
const { text } = (await transcribeAudio(blob)) as { text?: string };
if (text?.trim()) {
onResultRef.current?.(text);
} else {
setError("未识别到语音,请检查系统默认麦克风和输入音量后重试。");
}
} catch (err) {
console.error("Transcription failed:", err);
if (!(err instanceof HandledApiError)) {
setError("语音识别失败,请检查网络或语音服务配置后重试。");
}
} finally {
setIsTranscribing(false);
}
const devices = await media?.enumerateDevices?.();
if (mounted && current === revision) setMicrophones(devices?.filter((device) => device.kind === "audioinput") || []);
} catch { /* Recording can still use the system default device. */ }
};

recorder.stop();
}, []);

const startListening = useCallback(async () => {
if (!isSupported) return;
setError("");

try {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
streamRef.current = stream;

const mimeType = MediaRecorder.isTypeSupported("audio/webm;codecs=opus")
? "audio/webm;codecs=opus"
: "audio/webm";

const recorder = new MediaRecorder(stream, { mimeType });
chunksRef.current = [];

recorder.ondataavailable = (e) => {
if (e.data.size > 0) chunksRef.current.push(e.data);
};
recorder.onerror = () => {
recorder.onstop = null;
recorder.ondataavailable = null;
stream.getTracks().forEach((track) => track.stop());
streamRef.current = null;
mediaRecorderRef.current = null;
chunksRef.current = [];
setIsListening(false);
setIsTranscribing(false);
setError("录音中断,请检查麦克风连接后重新录音。");
};

mediaRecorderRef.current = recorder;
recorder.start(250); // Collect data every 250ms
setIsListening(true);
} catch (err) {
console.error("Microphone access failed:", err);
streamRef.current?.getTracks().forEach((track) => track.stop());
streamRef.current = null;
mediaRecorderRef.current = null;
const name = err instanceof Error ? err.name : "";
setError(name === "NotAllowedError"
? "无法访问麦克风,请在系统设置中允许此应用使用麦克风后重试。"
: name === "NotFoundError"
? "未找到麦克风,请连接麦克风并检查系统默认输入设备。"
: name === "NotReadableError"
? "麦克风无法使用,请检查设备连接或关闭占用麦克风的应用后重试。"
: "录音启动失败,请检查麦克风和系统声音设置后重试。");
setIsListening(false);
}
}, [isSupported]);

const toggle = useCallback(() => {
if (isListening) {
stopListening();
} else if (!isTranscribing) {
startListening();
}
}, [isListening, isTranscribing, startListening, stopListening]);

// Cleanup on unmount
useEffect(() => {
refreshDevicesRef.current = () => { void refresh(); };
void refresh();
media?.addEventListener?.("devicechange", refresh);
return () => {
const recorder = mediaRecorderRef.current;
if (recorder) {
recorder.onstop = null;
recorder.ondataavailable = null;
recorder.onerror = null;
}
if (mediaRecorderRef.current?.state !== "inactive") {
try {
mediaRecorderRef.current?.stop();
} catch {
/* cleanup, safe to ignore */
}
}
streamRef.current?.getTracks().forEach((t) => t.stop());
mounted = false;
media?.removeEventListener?.("devicechange", refresh);
refreshDevicesRef.current = () => {};
sessionRef.current?.cancel();
sessionRef.current = null;
};
}, []);

return { isListening, isTranscribing, isSupported, error, toggle };
const cancel = useCallback(() => {
sessionRef.current?.cancel();
sessionRef.current = null;
setState(initialVoiceInputState);
}, []);

const toggle = useCallback(() => {
const active = sessionRef.current;
if (active?.state.status === "recording") { active.stop(); return; }
// A synchronous session state also guards rapid clicks before React renders.
if (active && active.state.status !== "idle") return;
if (!isSupported) return;
active?.cancel();
const session = new VoiceInputSession({
onChange(next) { if (sessionRef.current === session) setState(next); },
onResult(text) { if (sessionRef.current === session) onResultRef.current?.(text); },
onDeviceReady() { refreshDevicesRef.current(); },
});
sessionRef.current = session;
void session.start(microphoneId);
}, [isSupported, microphoneId]);

return {
...state, isSupported, microphones, microphoneId, setMicrophoneId, toggle, cancel,
isStarting: state.status === "starting",
isListening: state.status === "recording",
isTranscribing: state.status === "stopping" || state.status === "transcribing",
isBusy: state.status !== "idle",
};
}
Loading
Loading