Computer Vision Examples for Unity

This is the next step in the evolution of depth sensor examples (incl. "Azure Kinect and Femto Bolt Examples", "Kinect-v2 Examples", etc.). Instead of a depth sensor though this asset uses a plain web camera or video recording as input, and AI models to provide depth estimation, body tracking and other streams. The package contains over thirty demo scenes.

Web | Documentation | Twitter | LinkedIn

The avatar-demo scenes show how to utilize user-controlled avatars in your scenes, gesture demo – how to use discrete and continuous gestures in your projects, fitting room demos – how to overlay or blend the user’s body with virtual models, background removal demo – how to display user silhouettes on virtual background, etc. Short descriptions of all demo-scenes are available in the online documentation.

This package works with plain web cameras and video clips that can be played with Unity video player. It can be used with all versions of Unity – Free, Plus & Pro.

How to run the demo scenes:

1. Create a new Unity project (Unity 2023.2 or later, as required by Unity’s Sentis package).

2. Open the 'Package Manager'-window in Unity editor, click the '+'-button, select 'Install package by name' from the menu and enter 'com.unity.sentis'. Then hit Enter or the 'Install'-button. This will install 'Sentis' - the Unity package for AI model inference.

3. Import this package into the Unity project.

4. Open ‘File / Build settings’ and switch to 'PC, Mac & Linux Standalone'.

5. Check if 'Direct3D11' is the first option in the ‘Auto Graphics API for Windows’-list setting, in 'Player Settings / Other Settings / Rendering'.

6. Please first open and run a demo scene that contains body tracking from a subfolder of the 'ComputerVisionExamples/DemoScenes'-folder (e.g. AvatarDemo1-scene or OverlayDemo2-scene). Stand in front of the camera to calibrate. This is needed only once, to estimate the camera intrinsic parameters.

7. Open and run a demo scene of your choice from a subfolder of the 'ComputerVisionExamples/DemoScenes'-folder. Short descriptions of all demo-scenes are available in the online documentation.

Current limitations:

1. Because of the intensive GPU utilization, please avoid using the CVE-package on mobile platforms for now, if possible. The performance may not be good enough, and the device may overheat.

2. Currently, the CVE-package will not run on WebGL platform, because it doesn’t support compute shaders.

One request:

Please don't share this package or its demo scenes in source form with others, or as part of public repositories, without my explicit consent.

Troubleshooting:

* Please note, this is one of the first releases of the CVE-package. Don't expect everything to be perfect. If you get any issues or errors, please contact me to report your issues. Then have some patience, until all issues get resolved.

* If you get errors in console, like 'Texture2D' does not contain a definition for 'LoadImage' or 'Texture2D' does not contain a definition for 'EncodeToJPG', please open the Package Manager, select 'Built-in packages' and make sure 'Image conversion' and 'Physics 2D' packages are enabled.

* For other known issues, please look here.

Documentation:

* The basic documentation is available in the Readme-pdf file in the package.

* The online documentation is available here.

Third-Party Software:

This asset uses AI models for monocular depth estimation under MIT License, as well as AI models and scripts for detecting landmarks of human bodies in an image or video under Apache License Version 2.0. See 'Third-Party-Notices.txt' file in the package for details.