微软TTS延迟因素分析

微软合成语音的过程中受多方面因素影响，具体每个因素有多大的影响，下面通过具体的测试，给出结论。

请求方式

通过微软的SDK有8种请求合成语音的API，可以分成Text or SSML，同步 or 异步，流式 or 非流式 3种形式。我们分别看每种方式合成语音的延迟情况。另外，还补充不同区域、是否预初始化SpeechSynthesizer、重复内容合成延迟的差异。

1. SpeakText

2. SpeakTextAsync

3. StartSpeakingText

4. StartSpeakingTextAsync

5. SpeakSsml

6. SpeakSsmlAsync

7. StartSpeakingSsml

8. StartSpeakingSsmlAsync

结论

语音合成的延迟影响最大因素是区域、是否流式。

测试配置

音色：zh-CN-XiaochenMultilingualNeural

语音：en-US

输出格式：SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3

测试内容：随机生成30个中文

Text or SSML

text方式：speechSynthesizerZH.SpeakText(text);

ssml方式：speechSynthesizerZH.SpeakSsml(ssml);

结论

两种方式相差不到，Text比SSML稍微快3~5%。

测试结果数据

测试代码

void textOrSsml() {
    SpeechConfig config = SpeechConfig.fromSubscription("{key}", "{region}");
    config.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer speechSynthesizerZH = new SpeechSynthesizer(config, null);
    
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    // ---- Init Link ---- //
    {
        var text = getRandomChinese(TEXT_LENGTH);
        long s = System.currentTimeMillis();
        speechSynthesizerZH.SpeakText(text);
 
        var ssmlText = getRandomChinese(TEXT_LENGTH);
        String ssml = buildSsml(ssmlText, "zh-CN-XiaochenMultilingualNeural", "en-US");
        speechSynthesizerZH.SpeakSsml(ssml);
        System.out.println("Init Link use times: " + (System.currentTimeMillis() - s));
    }
 
    var textTimes = new ArrayList<Long>();
    var ssmlTimes = new ArrayList<Long>();
 
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            var text = getRandomChinese(TEXT_LENGTH);
            String ssml = buildSsml(text, "zh-CN-XiaochenMultilingualNeural", "en-US");
            long s = System.currentTimeMillis();
            SpeechSynthesisResult speechSynthesisResult = speechSynthesizerZH.SpeakSsml(ssml);
            ssmlTimes.add(System.currentTimeMillis() - s);
        }
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            SpeechSynthesisResult speechSynthesisResult = speechSynthesizerZH.SpeakText(text);
            textTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tSpeakText\tSpeakSsml\tDifference\n");
    for (int i = 0; i < textTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(textTimes.get(i)).append("\t");
        report.append(ssmlTimes.get(i)).append("\t");
        report.append(textTimes.get(i) - ssmlTimes.get(i)).append("\n");
    }
    double textTimeAvg = textTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double ssmlTimeAvg = ssmlTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(textTimeAvg).append("\t");
    report.append(ssmlTimeAvg).append("\t");
    report.append(textTimeAvg - ssmlTimeAvg).append("\n");
    System.out.println(report);
}

同步 or 异步

同步方式：speechSynthesizerZH.SpeakText(text);

异步方式：speechSynthesizerZH.SpeakTextAsync(text).get();

结论

异步方式是使用线程池去调用SpeakText(text)方法，从单次请求看，延迟几乎没区别。SSML方式结果也是一样。

测试结果数据

测试代码

void syncOrAsync() throws ExecutionException, InterruptedException {
    SpeechConfig config = SpeechConfig.fromSubscription("{key}", "{region}");
    config.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer speechSynthesizerZH = new SpeechSynthesizer(config, null);
 
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    // ---- Init Link ---- //
    {
        var text1 = getRandomChinese(TEXT_LENGTH);
        var text2 = getRandomChinese(TEXT_LENGTH);
        long s = System.currentTimeMillis();
        speechSynthesizerZH.SpeakText(text1);
        speechSynthesizerZH.SpeakTextAsync(text2).get();
        System.out.println("Init Link use times: " + (System.currentTimeMillis() - s));
    }
 
    var textTimes = new ArrayList<Long>();
    var testAsyncTimes = new ArrayList<Long>();
 
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            speechSynthesizerZH.SpeakText(text);
            textTimes.add(System.currentTimeMillis() - s);
        }
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            speechSynthesizerZH.SpeakTextAsync(text).get();
            testAsyncTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tSpeakText\tSpeakTextAsync\tDifference\n");
    for (int i = 0; i < textTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(textTimes.get(i)).append("\t");
        report.append(testAsyncTimes.get(i)).append("\t");
        report.append(textTimes.get(i) - testAsyncTimes.get(i)).append("\n");
    }
    double textTimeAvg = textTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double ssmlTimeAvg = testAsyncTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(textTimeAvg).append("\t");
    report.append(ssmlTimeAvg).append("\t");
    report.append(textTimeAvg - ssmlTimeAvg).append("\n");
    System.out.println(report);
}

流式 or 非流式

流式方式：speechSynthesizerZH.StartSpeakingText(text);

非流式方式：speechSynthesizerZH.SpeakText(text);

结论

流式接口返回的首个音频数据，比非流式完整返回的延迟低了30%。在10个字的情况下，低了20%。文本越长，流式的延迟效果越好。

测试结果数据

测试代码

void streamOrNotStream() {
    SpeechConfig config = SpeechConfig.fromSubscription("{key}", "{region}");
    config.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer speechSynthesizerZH = new SpeechSynthesizer(config, null);
 
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    // ---- Init Link ---- //
    {
        var text1 = getRandomChinese(TEXT_LENGTH);
        var text2 = getRandomChinese(TEXT_LENGTH);
        long s = System.currentTimeMillis();
        speechSynthesizerZH.SpeakText(text1);
        SpeechSynthesisResult speechSynthesisResult = speechSynthesizerZH.StartSpeakingText(text2);
        AudioDataStream audioDataStream = AudioDataStream.fromResult(speechSynthesisResult);
        byte[] buffer = new byte[10000];
        long filledSize = audioDataStream.readData(buffer);
        System.out.println("Init Link use times: " + (System.currentTimeMillis() - s));
    }
 
    var notStreamTimes = new ArrayList<Long>();
    var streamTimes = new ArrayList<Long>();
 
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            speechSynthesizerZH.SpeakText(text);
            notStreamTimes.add(System.currentTimeMillis() - s);
        }
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            SpeechSynthesisResult speechSynthesisResult = speechSynthesizerZH.StartSpeakingText(text);
            AudioDataStream audioDataStream = AudioDataStream.fromResult(speechSynthesisResult);
            byte[] buffer = new byte[8000];
            long filledSize = audioDataStream.readData(buffer);
            streamTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tNotStream\tStream\tDifference\n");
    for (int i = 0; i < notStreamTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(notStreamTimes.get(i)).append("\t");
        report.append(streamTimes.get(i)).append("\t");
        report.append(notStreamTimes.get(i) - streamTimes.get(i)).append("\n");
    }
    double notStreamTimeAvg = notStreamTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double streamTimeAvg = streamTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(notStreamTimeAvg).append("\t");
    report.append(streamTimeAvg).append("\t");
    report.append(notStreamTimeAvg - streamTimeAvg).append("\n");
    System.out.println(report);
}

不同区域

测试区域：eastasia（东亚）、southeastasia（东南亚）、japanwest（日本西）、japaneast（日本东）

请求区域：东京

结论

japanwest（日本西：505ms）< japaneast（日本东：542）< eastasia（东亚：630）< southeastasia（东南亚：733）

从东亚换到日本西，延迟降低20%

测试结果数据

测试代码

static String diffRegion(int textLength, int loopTimes, Pair<String, String>... regions) {
    Map<String, SpeechSynthesizer> regionSpeechSynthesizer = new HashMap<>();
    Map<String, List<Long>> regionTimes = new HashMap<>();
    for (Pair<String, String> pair : regions) {
        val region = pair.getRight();
        SpeechConfig config = SpeechConfig.fromSubscription(pair.getLeft(), region);
        config.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
        config.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
        SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer(config, null);
 
        regionSpeechSynthesizer.put(region, speechSynthesizer);
        regionTimes.put(region, new ArrayList<>());
    }
 
    // ---- Init Link ---- //
    {
        long s = System.currentTimeMillis();
        for (Map.Entry<String, SpeechSynthesizer> entry : regionSpeechSynthesizer.entrySet()) {
            var text = getRandomChinese(textLength);
            SpeechSynthesisResult speechSynthesisResult = entry.getValue().SpeakText(text);
            if (speechSynthesisResult.getReason() == ResultReason.Canceled) {
                SpeechSynthesisCancellationDetails speechSynthesisCancellationDetails = SpeechSynthesisCancellationDetails.fromResult(speechSynthesisResult);
                System.out.println("failed: " + entry.getKey());
                System.out.println(speechSynthesisCancellationDetails.getErrorDetails());
                ;
            } else {
                System.out.println("ok：" + entry.getKey());
            }
        }
        System.out.println("Init Link use times: " + (System.currentTimeMillis() - s));
    }
 
    for (int i = 1; i <= loopTimes; i++) {
        System.out.println("loop NO." + i);
        for (Map.Entry<String, SpeechSynthesizer> entry : regionSpeechSynthesizer.entrySet()) {
            var text = getRandomChinese(textLength);
            long s = System.currentTimeMillis();
            entry.getValue().SpeakText(text);
            regionTimes.get(entry.getKey()).add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.");
    for (String region : regionTimes.keySet()) {
        report.append("\t" + region);
    }
    report.append("\n");
 
    for (int i = 0; i < loopTimes; i++) {
        report.append(i + 1).append("\t");
        for (List<Long> times : regionTimes.values()) {
            report.append(times.get(i)).append("\t");
        }
        report.append("\n");
    }
    // 统计平均时间
    report.append("avg").append("\t");
    for (List<Long> times : regionTimes.values()) {
        double avg = times.stream().mapToLong(Long::longValue).average().orElse(0);
        report.append(avg).append("\t");
    }
    report.append("\n");
    return report.toString();
}

预连接SpeechSynthesizer

预连接方式：

SpeechSynthesizer synthesizer = new SpeechSynthesizer(speechConfig, (AudioConfig) null);
Connection connection = Connection.fromSpeechSynthesizer(synthesizer);
connection.openConnection(true);

结论

合成语音延迟没有差别。

测试结果数据

测试代码

void preConnect() throws InterruptedException {
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    var reuseTimes = new ArrayList<Long>();
    var notReuseTimes = new ArrayList<Long>();
 
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            SpeechConfig config1 = SpeechConfig.fromSubscription("{key}", "{region}");
            config1.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
            config1.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
            SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer(config1, null);
            Connection connection = Connection.fromSpeechSynthesizer(speechSynthesizer);
            connection.openConnection(true);
 
            Thread.sleep(200);
 
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            speechSynthesizer.SpeakText(text);
            reuseTimes.add(System.currentTimeMillis() - s);
        }
 
        {
            SpeechConfig config2 = SpeechConfig.fromSubscription("{key}", "{region}");
            config2.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
            config2.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
            SpeechSynthesizer notPreConnectSpeechSynthesizer = new SpeechSynthesizer(config2, null);
 
            Thread.sleep(200);
 
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            notPreConnectSpeechSynthesizer.SpeakText(text);
            notReuseTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tPreConnect\tNotPreConnect\tDifference\n");
    for (int i = 0; i < reuseTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(reuseTimes.get(i)).append("\t");
        report.append(notReuseTimes.get(i)).append("\t");
        report.append(reuseTimes.get(i) - notReuseTimes.get(i)).append("\n");
    }
    double avg1 = reuseTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double avg2 = notReuseTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(avg1).append("\t");
    report.append(avg2).append("\t");
    report.append(avg1 - avg2).append("\n");
    System.out.println(report);
}

重用SpeechSynthesizer

重用：从池子里获取SpeechSynthesizer对象

不重用：每次请求重新new SpeechSynthesizer对象

结论

合成语音延迟没有差别。

测试结果数据

测试代码

void reuseOrNot() {
    SpeechConfig config1 = SpeechConfig.fromSubscription("{key}", "{region}");
    config1.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config1.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer reuseSpeechSynthesizer = new SpeechSynthesizer(config1, null);
 
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    // ---- Init Link ---- //
    {
        var text1 = getRandomChinese(TEXT_LENGTH);
        long s = System.currentTimeMillis();
        reuseSpeechSynthesizer.SpeakText(text1);
        System.out.println("Init Link use times: " + (System.currentTimeMillis() - s));
    }
 
    var reuseTimes = new ArrayList<Long>();
    var notReuseTimes = new ArrayList<Long>();
 
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            reuseSpeechSynthesizer.SpeakText(text);
            reuseTimes.add(System.currentTimeMillis() - s);
        }
        {
            long s = System.currentTimeMillis();
 
            SpeechConfig config2 = SpeechConfig.fromSubscription("{key}", "{region}");
            config2.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
            config2.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
            SpeechSynthesizer notPreInitSpeechSynthesizer = new SpeechSynthesizer(config2, null);
 
            var text = getRandomChinese(TEXT_LENGTH);
            notPreInitSpeechSynthesizer.SpeakText(text);
            notReuseTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tReuse\tNotReuse\tDifference\n");
    for (int i = 0; i < reuseTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(reuseTimes.get(i)).append("\t");
        report.append(notReuseTimes.get(i)).append("\t");
        report.append(reuseTimes.get(i) - notReuseTimes.get(i)).append("\n");
    }
    double reuseTimeAvg = reuseTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double notReuseTimeAvg = notReuseTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(reuseTimeAvg).append("\t");
    report.append(notReuseTimeAvg).append("\t");
    report.append(reuseTimeAvg - notReuseTimeAvg).append("\n");
    System.out.println(report);
}

重复内容合成

相同内容，重复多次调用。

结论

合成语音延迟没有差别。

测试结果数据

测试代码

void sameText() throws InterruptedException {
    SpeechConfig config1 = SpeechConfig.fromSubscription("{key}", "{region}");
    config1.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config1.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer sameSpeechSynthesizer = new SpeechSynthesizer(config1, null);
 
    SpeechConfig config2 = SpeechConfig.fromSubscription("{key}", "{region}");
    config2.setSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio24Khz48KBitRateMonoMp3);
    config2.setSpeechSynthesisVoiceName("zh-CN-XiaochenMultilingualNeural");
    SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer(config2, null);
 
    final int TEXT_LENGTH = 30;
    final int LOOP_TIMES = 50;
 
    var sameTextTimes = new ArrayList<Long>();
    var notSameTextTimes = new ArrayList<Long>();
 
    var sameText = getRandomChinese(TEXT_LENGTH);
    for (int i = 0; i <= LOOP_TIMES; i++) {
        {
            long s = System.currentTimeMillis();
            sameSpeechSynthesizer.SpeakText(sameText);
            sameTextTimes.add(System.currentTimeMillis() - s);
        }
 
        {
            var text = getRandomChinese(TEXT_LENGTH);
            long s = System.currentTimeMillis();
            speechSynthesizer.SpeakText(text);
            notSameTextTimes.add(System.currentTimeMillis() - s);
        }
    }
 
    // ---- 生成结果 ---- //
    StringBuilder report = new StringBuilder();
    report.append("NO.\tSameText\tNotSameText\tDifference\n");
    for (int i = 0; i < sameTextTimes.size(); i++) {
        report.append(i).append("\t");
        report.append(sameTextTimes.get(i)).append("\t");
        report.append(notSameTextTimes.get(i)).append("\t");
        report.append(sameTextTimes.get(i) - notSameTextTimes.get(i)).append("\n");
    }
    double avg1 = sameTextTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    double avg2 = notSameTextTimes.stream().mapToLong(Long::longValue).average().orElse(0);
    report.append("avg").append("\t");
    report.append(avg1).append("\t");
    report.append(avg2).append("\t");
    report.append(avg1 - avg2).append("\n");
    System.out.println(report);
}

公共方法

public static String getRandomChinese(int length) {
    StringBuilder sb = new StringBuilder();
    Random random = new Random();
 
    for (int i = 0; i < length; i++) {
        int codePoint = 0x4e00 + random.nextInt(0x9fa5 - 0x4e00 + 1);
        sb.append((char) codePoint);
    }
 
    return sb.toString();
}

private final static String SSML_TEMPLATE = """
            <speak version="1.0" xmlns="https://www.w3.org/2001/10/synthesis" xml:lang="{lang}">
              <voice name="{voiceName}">
                    {text}
              </voice>
            </speak>
            """;
    
public static String buildSsml(String text, String voiceName, String lang) {
    return SSML_TEMPLATE
            // 设置了<lang xml:lang="{lang}">标签里的多语言，就无法识别多语音
            .replaceAll("\\{lang\\}", lang)
            .replaceAll("\\{voiceName\\}", voiceName)
            .replaceAll("\\{text\\}", text);
}

引用

• 预连接SpeechSynthesizer：如何使用语音 SDK 降低语音合成延迟 - Azure AI services | Microsoft Learn

• 重用SpeechSynthesizer：如何使用语音 SDK 降低语音合成延迟 - Azure AI services | Microsoft Learn