Skip to content

Realtime Speech-to-Speech

XaiRealtimeClient is the SDK's WebSocket client for the xAI Speech-to-Speech API at wss://api.x.ai/v1/realtime. It supports bidirectional audio and text, server-side voice activity detection (VAD), function calling, built-in search tools, reconnects, and typed Grok Voice model selection.

See the xAI Speech-to-Speech guide and Voice API reference for service behavior and limits.

Quick Start

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
using Xai.Realtime;

        await using var client = new XaiRealtimeClient(apiKey);
        await client.ConnectAsync(
            model: VoiceModel.GrokVoiceThinkFast20,
            reasoningEffort: VoiceReasoningEffort.High,
            cancellationToken: cancellationToken);

        await client.SendSessionUpdateAsync(new SessionUpdatePayload
        {
            Session = new SessionConfig
            {
                Voice = "eve",
                Instructions = "Be helpful and concise.",
                Modalities = ["text", "audio"],
                TurnDetection = new TurnDetection
                {
                    Type = "server_vad",
                    Threshold = 0.85,
                    SilenceDurationMs = 500,
                    PrefixPaddingMs = 333,
                },
            },
        }, cancellationToken);

        await client.SendConversationItemCreateAsync(new ConversationItemCreatePayload
        {
            Item = new ConversationItem
            {
                Type = "message",
                Role = "user",
                Content =
                [
                    new ContentPart
                    {
                        Type = "input_text",
                        Text = "Introduce yourself in one sentence.",
                    },
                ],
            },
        }, cancellationToken);

        await client.SendResponseCreateAsync(new ResponseCreatePayload
        {
            Response = new ResponseConfig { Modalities = ["text", "audio"] },
        }, cancellationToken);

        await foreach (var serverEvent in client.ReceiveUpdatesAsync(cancellationToken))
        {
            if (serverEvent.IsResponseOutputAudioTranscriptDelta)
            {
                Console.Write(serverEvent.ResponseOutputAudioTranscriptDelta?.Delta);
            }
            else if (serverEvent.IsResponseDone)
            {
                break;
            }
            else if (serverEvent.IsError)
            {
                throw new InvalidOperationException(serverEvent.Error?.Error?.Message);
            }
        }

Authentication

Use the API-key constructor in server-side applications:

1
        await using var client = new XaiRealtimeClient(apiKey);

For a short-lived client secret, create the client without an API key and pass the token using the WebSocket subprotocol expected by xAI:

1
2
3
4
5
        await using var client = new XaiRealtimeClient();
        await client.ConnectAsync(
            model: VoiceModel.GrokVoiceLatest,
            additionalSubProtocols: [$"xai-client-secret.{clientSecret}"],
            cancellationToken: cancellationToken);

Do not expose a long-lived xAI API key in a browser or mobile client.

Models and Reasoning

The model and reasoning effort are selected during the WebSocket handshake:

1
2
3
4
        await client.ConnectAsync(
            model: VoiceModel.GrokVoiceThinkFast20,
            reasoningEffort: VoiceReasoningEffort.High,
            cancellationToken: cancellationToken);
SDK value Wire value Use
VoiceModel.GrokVoiceLatest grok-voice-latest Follow xAI's recommended model automatically
VoiceModel.GrokVoiceThinkFast20 grok-voice-think-fast-2.0 Pin Think Fast 2.0 for stable behavior
VoiceModel.GrokVoiceThinkFast10 grok-voice-think-fast-1.0 Remain on the previous model intentionally
VoiceReasoningEffort.High high Enable reasoning; this is the service default
VoiceReasoningEffort.None none Disable reasoning

The grok-voice-latest alias moves from Think Fast 1.0 to Think Fast 2.0 on August 5, 2026. Pin a versioned model when production behavior must remain stable.

If you supply the low-level uri override to ConnectAsync, include any required query parameters in that URI. The typed model and reasoningEffort values are used when the SDK builds the default endpoint URI.

Session Configuration

Send SessionUpdatePayload after connecting. Voice IDs are lowercase; use a built-in voice such as eve, ara, rex, sal, or leo, or a custom voice ID returned by the Custom Voices API.

Audio, Transcription, Resumption, and Pronunciation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
        var sessionUpdate = new SessionUpdatePayload
        {
            Session = new SessionConfig
            {
                Voice = "eve",
                Modalities = ["text", "audio"],
                TurnDetection = new TurnDetection
                {
                    Type = "server_vad",
                    Threshold = 0.85,
                    SilenceDurationMs = 500,
                    PrefixPaddingMs = 333,
                    IdleTimeoutMs = 10_000,
                },
                Audio = new AudioConfig
                {
                    Input = new AudioDirectionConfig
                    {
                        Format = new AudioFormatConfig
                        {
                            Type = "audio/pcm",
                            Rate = 24_000,
                        },
                        Transport = AudioTransport.Json,
                        Transcription = new AudioTranscriptionConfig
                        {
                            Model = "grok-transcribe",
                            LanguageHint = "en-US",
                            Keyterms = ["xAI", "Grok"],
                        },
                    },
                    Output = new AudioDirectionConfig
                    {
                        Format = new AudioFormatConfig
                        {
                            Type = "audio/pcm",
                            Rate = 24_000,
                        },
                        Transport = AudioTransport.Json,
                        Speed = 1.1,
                    },
                },
                Resumption = new ResumptionConfig { Enabled = true },
                Replace = new Dictionary<string, string>
                {
                    ["SQL"] = "sequel",
                    ["tryAGI"] = "try A G I",
                },
            },
        };

await client.SendSessionUpdateAsync(sessionUpdate, cancellationToken);

AudioTransport.Json carries base64 audio in JSON events. AudioTransport.Binary selects raw codec bytes in binary WebSocket messages. Use the raw byte SendAsync overload for binary input and ReceiveMessagesAsync for binary output. Input transcription accepts a BCP-47 language hint and up to 100 key terms. Output speed ranges from 0.7 to 1.5. Pronunciation replacements are case-insensitive and apply to whole words.

Session configuration is validated before serialization. Invalid VAD ranges, output speed, misplaced input/output options, and transcription keyterm limits throw ArgumentException before a message is sent. Call GetValidationErrors() when an application needs to display all problems without throwing.

Audio Formats

Format Type Supported sample rates
PCM16 little-endian audio/pcm 8000, 16000, 22050, 24000, 32000, 44100, 48000
G.711 μ-law audio/pcmu 8000
G.711 A-law audio/pcma 8000
Opus packets audio/opus 24000

For lowest integration overhead, capture and play 24 kHz PCM16 so the application does not need to resample.

Server VAD

With server VAD, xAI detects speech boundaries and creates responses automatically:

1
2
3
4
5
6
7
8
            TurnDetection = new TurnDetection
            {
                Type = "server_vad",
                Threshold = 0.85,
                SilenceDurationMs = 500,
                PrefixPaddingMs = 333,
                IdleTimeoutMs = 10_000,
            },

Higher thresholds require louder input before speech starts. Increase SilenceDurationMs when callers need longer pauses without ending their turn. IdleTimeoutMs lets the agent check whether an inactive caller is still present.

Manual Turn Control

Manual mode requires an explicit JSON null for turn_detection. UseManualTurnDetection() emits that null while preserving typed session configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
        var sessionUpdate = new SessionUpdatePayload
        {
            Session = new SessionConfig
            {
                Voice = "eve",
                Modalities = ["text", "audio"],
            }.UseManualTurnDetection(),
        };

await client.SendSessionUpdateAsync(sessionUpdate, cancellationToken);

In manual mode, append audio, commit it, and request a response yourself. Call UseServerTurnDetection(...) to switch the same configuration back to server VAD without emitting duplicate JSON properties.

Sending Audio

The byte overload accepts raw audio and handles base64 encoding for input_audio_buffer.append:

1
2
3
4
        ReadOnlyMemory<byte> audioChunk = await GetMicrophoneChunkAsync(cancellationToken);
        await client.SendInputAudioBufferAppendAsync(
            audio: audioChunk,
            cancellationToken: cancellationToken);

Send chunks continuously while the caller is speaking. Approximately 100 ms per chunk is a practical starting point.

When using manual turn detection, finish the turn explicitly:

1
2
3
4
5
6
7
8
        await client.SendInputAudioBufferCommitAsync(
            new InputAudioBufferCommitPayload(),
            cancellationToken);

        await client.SendResponseCreateAsync(new ResponseCreatePayload
        {
            Response = new ResponseConfig { Modalities = ["text", "audio"] },
        }, cancellationToken);

Receiving Audio and Transcripts

Audio deltas are base64 strings. Decode and enqueue each chunk for playback as soon as it arrives:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
        await foreach (var serverEvent in client.ReceiveUpdatesAsync(cancellationToken))
        {
            if (serverEvent.IsResponseOutputAudioDelta &&
                serverEvent.ResponseOutputAudioDelta?.Delta is { Length: > 0 } delta)
            {
                byte[] audioBytes = Convert.FromBase64String(delta);
                await playbackStream.WriteAsync(audioBytes, cancellationToken);
            }
            else if (serverEvent.IsResponseOutputAudioTranscriptDelta)
            {
                Console.Write(serverEvent.ResponseOutputAudioTranscriptDelta?.Delta);
            }
            else if (serverEvent.IsResponseOutputAudioDone)
            {
                await playbackStream.FlushAsync(cancellationToken);
            }
            else if (serverEvent.IsError)
            {
                var error = serverEvent.Error?.Error;
                throw new InvalidOperationException($"{error?.Code}: {error?.Message}");
            }
        }

Do not wait for response.done before starting playback; streaming each audio delta minimizes perceived latency.

Binary Output Audio

Set Audio.Output.Transport to AudioTransport.Binary, then use the combined receive stream. Binary messages contain raw bytes in the configured codec; JSON lifecycle events remain interleaved in wire order:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
        await foreach (var message in client.ReceiveMessagesAsync(cancellationToken))
        {
            if (message.IsBinaryAudio)
            {
                await playbackStream.WriteAsync(message.BinaryAudio, cancellationToken);
            }
            else if (message.Event is { } serverEvent && serverEvent.IsResponseDone)
            {
                break;
            }
        }

Do not enumerate ReceiveMessagesAsync and ReceiveUpdatesAsync concurrently on the same client.

Tools

Function Calling

Define a function with its JSON Schema parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
using System.Text.Json;

        using JsonDocument weatherSchema = JsonDocument.Parse(
            """
            {
              "type": "object",
              "properties": {
                "location": {
                  "type": "string",
                  "description": "City and country"
                }
              },
              "required": ["location"]
            }
            """);
        JsonElement weatherParameters = weatherSchema.RootElement.Clone();

        await client.SendSessionUpdateAsync(new SessionUpdatePayload
        {
            Session = new SessionConfig
            {
                Voice = "eve",
                Modalities = ["text", "audio"],
                Tools =
                [
                    new Tool
                    {
                        Type = "function",
                        Name = "get_weather",
                        Description = "Get the current weather for a location.",
                        Parameters = weatherParameters,
                    },
                ],
            },
        }, cancellationToken);

Execute the function when its arguments are complete, then add a function_call_output item:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
        if (serverEvent.IsResponseFunctionCallArgumentsDone)
        {
            var functionCall = serverEvent.ResponseFunctionCallArgumentsDone!;
            string outputJson = await ExecuteToolAsync(
                functionCall.Name!,
                functionCall.Arguments!,
                cancellationToken);

            await client.SendConversationItemCreateAsync(new ConversationItemCreatePayload
            {
                Item = new ConversationItem
                {
                    Type = "function_call_output",
                    CallId = functionCall.CallId,
                    Output = outputJson,
                },
            }, cancellationToken);

            // Wait for audio already buffered by the player to finish before requesting
            // the follow-up response, otherwise the two spoken turns can overlap.
            await WaitForPlaybackCompletionAsync(cancellationToken);
            await client.SendResponseCreateAsync(new ResponseCreatePayload(), cancellationToken);
        }

Built-in Search Tools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
        IList<Tool> tools =
        [
            new Tool { Type = "web_search" },
            new Tool
            {
                Type = "x_search",
                AllowedXHandles = ["grok", "xai"],
            },
            new Tool
            {
                Type = "file_search",
                VectorStoreIds = ["collection_abc123"],
                MaxNumResults = 5,
            },
        ];

The collection must already exist before it can be used by file_search.

Reconnection

Enable automatic reconnects before starting the receive loop:

1
2
3
4
5
        client.ReconnectOptions.Enabled = true;
        client.ReconnectOptions.MaxAttempts = 5;
        client.ReconnectOptions.InitialDelay = TimeSpan.FromSeconds(1);
        client.ReconnectOptions.MaxDelay = TimeSpan.FromSeconds(20);
        client.ReconnectOptions.BackoffMultiplier = 2;

The client remembers the URI and connection options used by the successful ConnectAsync call. Subscribe to Reconnecting, ExceptionOccurred, and Closed when the application needs connection telemetry.

Session Resumption

Enable Resumption.Enabled on the original session and save serverEvent.ConversationCreated?.Conversation?.Id. Conversation history expires after 30 minutes of inactivity. On a new client, the helper adds the encoded conversation_id handshake query and enables resumption on the new session update:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
        await client.ResumeConversationAsync(
            conversationId: conversationId,
            session: new SessionConfig
            {
                Voice = "eve",
                Modalities = ["text", "audio"],
            },
            model: VoiceModel.GrokVoiceThinkFast20,
            reasoningEffort: VoiceReasoningEffort.High,
            cancellationToken: cancellationToken);

Server Events

ServerEvent is a generated discriminated union. Check an Is* property before reading its matching payload.

Property Event type Meaning
IsSessionCreated session.created Session opened; includes the selected model
IsSessionUpdated session.updated Session configuration accepted
IsConversationCreated conversation.created Conversation opened
IsConversationItemAdded conversation.item.added Item added to conversation history
IsInputAudioBufferSpeechStarted input_audio_buffer.speech_started VAD detected speech
IsInputAudioBufferSpeechStopped input_audio_buffer.speech_stopped VAD detected silence
IsInputAudioBufferCommitted input_audio_buffer.committed Buffered audio committed
IsInputAudioTranscriptionCompleted input_audio_transcription.completed User audio transcription completed
IsResponseCreated response.created Assistant response started
IsResponseOutputItemAdded response.output_item.added Output item added
IsResponseOutputAudioTranscriptDelta response.output_audio_transcript.delta Incremental assistant transcript
IsResponseOutputAudioTranscriptDone response.output_audio_transcript.done Assistant transcript completed
IsResponseOutputAudioDelta response.output_audio.delta Incremental base64 audio
IsResponseOutputAudioDone response.output_audio.done Assistant audio completed
IsResponseFunctionCallArgumentsDone response.function_call_arguments.done Function arguments completed
IsMcpListToolsCompleted mcp_list_tools.completed MCP tool discovery completed
IsResponseMcpCallArgumentsDone response.mcp_call_arguments.done MCP call arguments completed
IsResponseMcpCallCompleted response.mcp_call.completed MCP call succeeded
IsResponseMcpCallFailed response.mcp_call.failed MCP call failed
IsResponseDone response.done Assistant response completed
IsError error Service error received

Unknown text messages raise the UnknownMessage event instead of being silently discarded.

Lifetime and Cancellation

Pass a cancellation token to connection, send, and receive methods. Prefer await using so an open WebSocket receives a normal close frame before disposal:

1
        await using var client = new XaiRealtimeClient(apiKey);

Keep audio capture and WebSocket connection startup parallel in latency-sensitive applications, buffer early microphone samples, and flush them after ConnectAsync completes.