FaceUnity

This page explains how to integrate and use the FaceUnity AR Filter extension in your Android or iOS app.

To quickly integrate FaceUnity's AR filter capabilities, call the Video SDK setExtensionProperty method and pass the corresponding key and value.

Taking setExtensionProperty for example, the key is named after the FaceUnity API, and the value wraps certain or all the parameters of that API in JSON. When you call setExtensionProperty and pass in the pair of key and value, it is equivalent to calling the corresponding FaceUnity API. The case is the same for setExtensionPropertyWithVendor.

Currently, the extension encapsulates part of the APIs of the FaceUnity Nama SDK. For details, see the key overview.

The development environment requirements are as follows:

• Android Studio 4.1 or later.
• A physical device (not an emulator) running Android 5.0 or later.

• Implemented Get Started with Video Calling.

You need to activate the extension in Agora Console. Save the certificate file you obtain from activation. The certificate is used in subsequent integration.

1. In Extensions Marketplace page, download the package of FaceUnity AR Filter.
2. Unzip the package, and save all .aar files to the /app/libs path of your project folder.
3. Save the certificate file authpack.java to where the app module is located (For example, the package name in the sample code is io.agora.rte.extension.faceunity.example, so the certificate file should be saved to /app/src/main/java/io/agora/rte/extension/faceunity/example).
4. Contact Agora to get the resource package of the Faceunity extension. Save the model and prop files you need to the /app/src/main/assets path of the project folder. For details, see Resource package structure.
5. In the app/build.gradle file, add the following line in dependencies:
   
   

   _1

   implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

   
   
6. Import the required classes:
   
   

   _7

   import io.agora.rtc2.Constants;

   _7

   import io.agora.rtc2.IMediaExtensionObserver;

   _7

   import io.agora.rtc2.IRtcEngineEventHandler;

   _7

   import io.agora.rtc2.RtcEngine;

   _7

   import io.agora.rtc2.RtcEngineConfig;

   _7

   import io.agora.rtc2.video.VideoCanvas;

   _7

   import io.agora.rte.extension.faceunity.ExtensionManager;

   
   

This section describes the call sequence of using the extension. For a detailed parameter description, see the API Reference.

When initializing RtcEngine, call enableExtension to enable the extension.

_5

private void enableExtension(boolean enabled) {

_5

// Initialize ExtensionManager before calling enableExtension

_5

ExtensionManager.getInstance(mRtcEngine).initialize(this);

_5

mRtcEngine.enableExtension("FaceUnity", "Effect", enabled);

_5

}

To initialize the extension, call setExtensionProperty (Android), and pass in the corresponding keys and values:

1. Initialize the extension Set the key as fuSetup, and the value as the pointer to the certificate file.
2. Load the AI model Set the key as fuLoadAIModelFromPackage, and the value contains the path of the AI capability model file ai_xxx.bundle and the AI capability type.

For example:

_31

private void initExtension() {

_31

// Initialization

_31

try {

_31

JSONObject jsonObject = new JSONObject();

_31

JSONArray jsonArray = new JSONArray();

_31

for (byte it : authpack.A()) {

_31

jsonArray.put(it);

_31

}

_31

jsonObject.put("authdata", jsonArray);

_31

setExtensionProperty("fuSetup", jsonObject.toString());

_31

} catch (JSONException e) {

_31

Log.e(TAG, e.toString());

_31

}

_31

_31

// Load the AI model

_31

File modelDir = new File(getExternalFilesDir("assets"),

_31

"face_unity/model/ai_face_processor.bundle");

_31

try {

_31

JSONObject jsonObject = new JSONObject();

_31

jsonObject.put("data", modelDir.getAbsolutePath());

_31

jsonObject.put("type", 1 << 10);

_31

setExtensionProperty("fuLoadAIModelFromPackage", jsonObject.toString());

_31

} catch (JSONException e) {

_31

Log.e(TAG, e.toString());

_31

}

_31

}

_31

_31

// For the sake of calling setExtensionProperty multiple times

_31

private void setExtensionProperty(String key, String property) {

_31

mRtcEngine.setExtensionProperty("FaceUnity", "Effect", key, property);

_31

}

Call setExtensionProperty and pass in the corresponding keys and values.

You can implement the following functions:

• Load props and adjust beautification intensity
• Recognize and track human faces, gestures, and bodies

You can call the method as needed. For a full list of keys and values, see the API Reference.

​

The complete sample code and project is provided on GitHub:

• FaceUnity/android

​

1. Clone the repository:

_1

git clone https://github.com/AgoraIO-Community/AgoraMarketPlace.git

2. On the Extensions Marketplace Downloads page, download the Android package of FaceUnity AR Filter. Unzip the package, and save all .aar files to the FaceUnity/android/app/libs path.

3. Contact Agora to get the certificate file and resource package.

4. Save the certificate file authpack.java to FaceUnity/android/app/src/main/java/io/Agora/rte/extension/faceunity/example.

5. Save the required model and prop files from the resource package to FaceUnity/android/app/src/main/assets/face_unity under the project folder.

6. Open the sample project FaceUnity/android in Android Studio.

7. Sync the project with Gradle files.

8. Open the FaceUnity/android/app/src/main/java/io/Agora/rte/extension/FaceUnity/example/Config.java file, and replace <YOUR_APP_ID> with your App ID. To get an App ID, see Getting Started with Agora.

   
   

   _4

   public interface Config {

   _4

   String mAppId = "<YOUR_APP_ID>";

   _4

   String mToken = null;

   _4

   }

   
   

9. Connect an Android device (not an emulator), and run the project.

​

• For a full description of the Faceunity resource package, refer to:
• Android resource package
• iOS resource package

Video SDK

This section lists the APIs related to using extensions with the Agora SDK.

• addExtension in the RtcEngineConfig class
• enableExtension in the RtcEngineConfig class
• setExtensionProperty in the RtcEngineConfig class
• onEvent in the IMediaExtensionObserver class

​

The key corresponds to the name of the FaceUnity API, and the value corresponds to the parameters of the FaceUnity API. In this section, if the value is the same as the parameters of the FaceUnity API, the link leads to the FaceUnity documentation. If the value is different from the parameters of the FaceUnity API, the link leads to subsequent sections on this page.

Method keys refer to the keys you pass in when calling the setExtensionProperty/setExtensionPropertyWithVendor method of the Agora SDK.

Method keys	Description
fuSetup	Initialize the extension and authenticate the user. Must be executed before other keys. Otherwise, a crash can happen.
fuLoadAIModelFromPackage	Preload AI capabilities.
fuReleaseAIModel	Free up resources occupied by AI capabilities.

Method keys	Description
fuCreateItemFromPackage	Loads the prop package.
fuLoadTongueModel	Loads tongue detection data.
fuItemSetParam	Modifies or sets the value of a variable in the prop package.

Method keys	Description
fuDestroyItem	Destroys a specified item.
fuDestroyAllItems	Destroys all loaded items and releases all occupied resources.
fuOnDeviceLost	Resets the system's GL state. Use this key when the OpenGL context is released/destroyed by external resources.
fuDestroyLibData	Frees up the memory allocated to the face tracking module after calling fuSetup.

Method keys	Description
fuBindItems	Binds resource items to a target item.
fuUnbindItems	Unbinds the resource items from a target item.
fuIsTracking	Sets whether to get the number of faces being tracked.
fuSetMaxFaces	Sets the maximum number of tarcked faces.
fuSetDefaultRotationMode	Sets the default human face orientation.

Method keys	Description
fuFaceProcessorSetMinFaceRatio	Sets the distance of face detection.
fuSetTrackFaceAIType	Sets the fuTrackFace algorithm type.
fuSetFaceProcessorFov	Sets the fov (equivalent to focal length) of the FaceProcessor algorithm module.
fuHumanProcessorReset	Resets the state of the HumanProcessor algorithm module.
fuHumanProcessorSetMaxHumans	Sets the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorGetNumResults	Sets whether to get the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorSetFov	Sets the fov (equivalent to focal length) used by the HumanProcessor algorithm module to track 3D key points on human bodies.
fuHandDetectorGetResultNumHands	Sets whether to get the number of gestures tracked by the HandGesture algorithm module. Note that ai_gesture.bundle needs to be loaded.

​

Callback keys refer to the keys returned in the onEvent callback of the Agora SDK.

Callback keys	Description
fuIsTracking	Returns the number of faces being tracked.
fuHumanProcessorGetNumResults	Returns the number of human bodies tracked by the HumanProcessor algorithm module.
fuHandDetectorGetResultNumHands	Returns the number of gestures tracked by the HandGesture algorithm module.

The value contains the following parameters:

Value parameters	Description
authdata	The path to the certificate file.

The value contains the following parameters:

Value parameters	Description
data	String. The path of the AI capability model file ai_xxx.bundle. Such model files are located in the assets/AI_Model directory of the resource package.
type	Int. The AI capability type corresponding to the bundle file. Possible values are listed in enum FUAITYPE.

The value contains the following parameters:

Value parameters	Description
data	String. The path to the prop package you want to load. A prop package usually has a suffix *.bundle .

The value contains the following parameters:

Value parameters	Description
data	String. The path of tongue model data tongue.bundle.

The value contains the following parameters:

Value parameters	Description
obj_handle	String. The path of the prop package passed in when calling fuCreateItemFromPackage.
name``	String. The name of the variable to set in the prop package.
value	Object. The variable value to be set.

For details on the variable names and values in the prop package, refer to the FaceUnity documentation.

The value contains the following parameters:

Value parameters	Description
item	String. The path of the prop package passed in when calling fuCreateItemFromPackage.

The value contains the following parameters:

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array. The paths to the resource items you want to bind.

The value contains the following parameters:

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array, the paths to the resource items you want to unbind.

The value contains the following parameters:

Value parameters	Description
enable	Bool. Whether to get the number of faces being tracked. If set to true, you can receive the fuIsTracking callback.

The value contains the following parameters:

Value parameters	Description
enable	Bool. Whether to get the number of human bodies tracked by the HumanProcessor algorithm module. If set to true, you can receive the fuHumanProcessorGetNumResults callback.

The value contains the following parameters:

Value parameters	Description
enable	Bool. Whether to get the number of gestures tracked by the HandGesture algorithm module. If set to true, you can receive the fuHandDetectorGetResultNumHands callback.

​

The value contains the following parameters:

Value parameters	Description
faces	Int. The number of faces being tracked.

​

The value contains the following parameters:

Value parameters	Description
people	Int. The number of bodies being tracked.

​

The value contains the following parameters:

Value parameters	Description
hands	Int. The number of gestures being tracked.