Raw video and audio processing

In some scenarios, raw audio and video captured through the camera and microphone must be processed to achieve the desired functionality or to enhance the user experience. Video SDK enables you to pre-process and post-process the captured audio and video data for implementation of custom playback effects.

Understand the tech

You can use the raw data processing functionality in Video SDK to process the feed according to your particular scenario. This feature enables you to pre-process the captured signal before sending it to the encoder, or to post-process the decoded signal before playback. To implement processing of raw video and audio data in your app, take the following steps.

Set up video and audio frame observers.
Register video and audio frame observers before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw video and audio data.
Unregister the frame observers before you leave a channel.

The figure below shows the workflow you need to implement to process raw video and audio data in your app.

Process raw audio and video

Prerequisites

To follow this procedure you must have implemented Get Started with Interactive Live Streaming.

Project setup

To create the environment necessary to integrate processing of raw audio and video data in your app, open the Get Started with Interactive Live Streaming project you created previously.

Implement raw data processing

When a user captures or receives video and audio data, the data is available to the app for processing before it is played. This section shows how to retrieve this data and process it, step-by-step.

Implement the user interface

To enable or disable processing of captured raw video data, add a button to the user interface. In /app/res/layout/activity_main.xml add the following lines before </RelativeLayout>:

<Button
    android:id="@+id/ZoomButton"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_below="@+id/JoinButton"
    android:layout_alignEnd="@id/LeaveButton"
    android:layout_alignStart="@id/JoinButton"
    android:onClick="setZoom"
    android:text="Zoom In" />

Handle the system logic

This sections describes the steps required to use the relevant libraries, declare the necessary variables, and set up access to the UI elements.

Import the required Android and Agora libraries

To integrate Video SDK frame observer libraries into your app and access the button object, add the following statements after the last import statement in /app/java/com.example.<projectname>/MainActivity.

_5import io.agora.rtc2.video.IVideoFrameObserver; _5import io.agora.rtc2.IAudioFrameObserver; _5import io.agora.base.VideoFrame; _5import java.nio.ByteBuffer; _5import android.widget.Button;
Define a variable to manage video processing

In /app/java/com.example.<projectname>/MainActivity, add the following declaration to the MainActivity class:

_1private boolean isZoomed = false;

Implement processing of raw video and audio data

To register and use video and audio frame observers in your app, take the following steps:

Set up the audio frame observer

IAudioFrameObserver gives you access to each audio frame after it is captured or access to each audio frame before it is played back. To setup the IAudioFrameObserver, add the following lines to the MainActivity class after variable declarations:

_61private final IAudioFrameObserver iAudioFrameObserver = new IAudioFrameObserver() { _61 @Override _61 public boolean onRecordAudioFrame(String channelId, int type, int samplesPerChannel, _61 int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { _61 // Gets the captured audio frame. _61 // Add code here to process the recorded audio. _61 return false; _61 } _61 _61 @Override _61 public boolean onPlaybackAudioFrame(String channelId, int type, int samplesPerChannel, _61 int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { _61 // Gets the audio frame for playback. _61 // Add code here to process the playback audio. _61 return false; _61 } _61 _61 @Override _61 public boolean onMixedAudioFrame(String channelId, int type, int samplesPerChannel, _61 int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { _61 // Retrieves the mixed captured and playback audio frame. _61 return false; _61 } _61 _61 @Override _61 public boolean onEarMonitoringAudioFrame(int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { _61 return false; _61 } _61 _61 @Override _61 public boolean onPlaybackAudioFrameBeforeMixing(String channelId, int userId, int type, int samplesPerChannel, _61 int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { _61 // Retrieves the audio frame of a specified user before mixing. _61 return false; _61 } _61 _61 @Override _61 public int getObservedAudioFramePosition() { _61 return 0; _61 } _61 _61 @Override _61 public AudioParams getRecordAudioParams() { _61 return null; _61 } _61 _61 @Override _61 public AudioParams getPlaybackAudioParams() { _61 return null; _61 } _61 _61 @Override _61 public AudioParams getMixedAudioParams() { _61 return null; _61 } _61 _61 @Override _61 public AudioParams getEarMonitoringAudioParams() { _61 return null; _61 } _61};
Set up the video frame observer

IVideoFrameObserver gives you access to each local video frame after it is captured and access to each remote video frame before it is played back. In this example, your modify the captured video frame buffer to crop and scale the frame and play a zoomed-in version of the video. To set up IVideoFrameObserver, add the following lines to the MainActivity class after the variable declarations:

_65private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() { _65 @Override _65 public boolean onCaptureVideoFrame(VideoFrame videoFrame) { _65 if (isZoomed) { _65 VideoFrame.Buffer buffer = videoFrame.getBuffer(); _65 int w = buffer.getWidth(); _65 int h = buffer.getHeight(); _65 int cropX = (w - 320)/2, cropY = (h - 240)/2, cropWidth = 320, cropHeight = 240, scaleWidth = 320, scaleHeight = 240; _65 buffer = buffer.cropAndScale(cropX, cropY, cropWidth, cropHeight, scaleWidth, scaleHeight); _65 videoFrame.replaceBuffer(buffer, 270, videoFrame.getTimestampNs()); _65 } _65 return true; _65 } _65 _65 @Override _65 public boolean onPreEncodeVideoFrame(VideoFrame videoFrame) { _65 return false; _65 } _65 _65 @Override _65 public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) { _65 return false; _65 } _65 _65 @Override _65 public boolean onPreEncodeScreenVideoFrame(VideoFrame videoFrame) { _65 return false; _65 } _65 _65 @Override _65 public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) { _65 return false; _65 } _65 _65 @Override _65 public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) { _65 return false; _65 } _65 _65 @Override _65 public int getVideoFrameProcessMode() { _65 // The process mode of the video frame. 0 means read-only, and 1 means read-and-write. _65 return 1; _65 } _65 _65 @Override _65 public int getVideoFormatPreference() { _65 return 1; _65 } _65 _65 @Override _65 public boolean getRotationApplied() { _65 return false; _65 } _65 _65 @Override _65 public boolean getMirrorApplied() { _65 return false; _65 } _65 _65 @Override _65 public int getObservedFramePosition() { _65 return 0; _65 } _65};

Note that you must set the return value in getVideoFrameProcessMode to 1 in order for your raw data changes to take effect.
Register the video and audio frame observers

To receive callbacks declared in IVideoFrameObserver and IAudioFrameObserver, you must register the video and audio frame observers with the Agora Engine before joining a channel. To specify the format of audio frames captured by each IAudioFrameObserver callback, use the setRecordingAudioFrameParameters, setMixedAudioFrameParameters and setPlaybackAudioFrameParameters methods. To do this, add the following lines after if (checkSelfPermission()) { in the joinChannel method:

_11agoraEngine.registerVideoFrameObserver(iVideoFrameObserver); _11agoraEngine.registerAudioFrameObserver(iAudioFrameObserver); _11 _11// Set the format of the captured raw audio data. _11int SAMPLE_RATE = 16000, SAMPLE_NUM_OF_CHANNEL = 1, SAMPLES_PER_CALL = 1024; _11 _11agoraEngine.setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, _11 Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL); _11agoraEngine.setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, _11 Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL); _11agoraEngine.setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, SAMPLES_PER_CALL);
Unregister the video and audio observers when you leave a channel

When you leave a channel, you unregister the frame observers by calling the register frame observer method again with a null argument. To do this, add the following lines to the leaveChannel(View view) method before agoraEngine.leaveChannel();:

_2agoraEngine.registerVideoFrameObserver(null); _2agoraEngine.registerAudioFrameObserver(null);
Start and stop video processing

When a user presses the button, enable or disable video processing. To do this, add the following method to the MainActivity class:

_9public void setZoom (View view){ _9 isZoomed = !isZoomed; _9 _9 Button button = (Button) view; _9 if (isZoomed) _9 button.setText("Zoom Out"); _9 else _9 button.setText("Zoom In"); _9}

Test your implementation

To ensure that you have implemented raw data processing into your app:

Generate a temporary token in Agora Console.
In your browser, navigate to the Agora web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.
In Android Studio, open app/java/com.example.<projectname>/MainActivity, and update appId, channelName and token with the values for your temporary token.
Connect a physical Android device to your development device.
In Android Studio, click Run app. A moment later you see the project installed on your device.

If this is the first time you run the project, grant microphone and camera access to your app.
Press Join to see the video feed from the web app.
Test processing of raw video data.

Press Zoom In. You see that the local video captured by your device camera is cropped and scaled to give a zoom-in effect. The processed video is displayed both locally and remotely. Pressing the button again stops processing of raw video data and restores the original video.
Test processing of raw audio data.

Edit the iAudioFrameObserver definition by adding code that processes the raw audio data you receive in the following callbacks:
- onRecordAudioFrame: Gets the captured audio frame data
- onPlaybackAudioFrame: Gets the audio frame for playback

Reference

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.

Agora Developer Center