--- title: Background Removal with the Vonage Video API – WebRTC.ventures description: Taking WebRTC Video calling to the next step involves providing more functionality and custom behavior to our video conferencing applications. We've experimented with using the HTML canvas to add more customization to our video calls, and we've found that gives us a lot of power. In this post, we look at leveraging the Vonage Video API with the power of HTML canvas to apply virtual backgrounds to our video calls. image: https://blog.webrtc.ventures/hubfs/Imported_Blog_Media/Background-Removal-with-the-Vonage-Video-API.png --- [Skip to content](https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api#main-content) ![WebRTC.ventures](https://blog.webrtc.ventures/hubfs/logo-main-light.svg) - [WebRTC Services](https://webrtc.ventures/webrtc-development-services/) Show submenu for WebRTC Services - [Assess](https://webrtc.ventures/services/assess/) - [Build](https://webrtc.ventures/services/build/) - [Integrate](https://webrtc.ventures/services/integrate/) - [Test](https://webrtc.ventures/services/test/) - [Manage](https://webrtc.ventures/services/manage/) - [Migrating From Twilio Video](https://webrtc.ventures/twilio-video/) - Our Partners Show submenu for Our Partners - [Amazon Chime SDK](https://webrtc.ventures/partners/amazon-chime-sdk/) - Stories of Success Show submenu for Stories of Success - [Remote Video Interpreting Platform with Agent Routing](https://webrtc.ventures/successes/remote-video-translation-platform/) - [WebRTC to SIP Gateway for ASL Interpreters](https://webrtc.ventures/successes/webrtc-to-sip-gateway-for-asl-interpreters/) - [Audio Listening Device to Improve Surgical Outcomes](https://webrtc.ventures/successes/audio-listening-device-to-improve-surgical-outcomes/) - [Albania’s First Telemedicine Platform](https://webrtc.ventures/successes/albanias-first-telemedicine-platform/) - [Automated Video Storytelling with the Amazon Chime SDK](https://webrtc.ventures/successes/automated-video-storytelling-with-the-amazon-chime-sdk/) - [Large Scale Video Conferencing](https://webrtc.ventures/successes/large-scale-video-conferencing/) - [Broadcasting and Webcast Control Room](https://webrtc.ventures/successes/live-video-broadcasting-and-webcast-control-room/) - WebRTC Resources Show submenu for WebRTC Resources - [WebRTC Live](https://webrtc.ventures/webrtc-live/) - [WebRTC Tips](https://www.youtube.com/playlist?list=PLayYqdnyegt3QvxX-Rt6wbFIK8727Z5Oz) - [WebRTC.ventures Blog](https://webrtc.ventures/blog/) - [Migrating From Twilio Video](https://webrtc.ventures/twilio-video/) - [WebRTC.ventures Training Program](https://webrtc.ventures/webrtc-ventures-training-program/) - [About Us](https://webrtc.ventures/our-team/) Show submenu for About Us - [Blog](https://webrtc.ventures/blog/) - [Our Team](https://webrtc.ventures/our-team/) - [Jobs](https://webrtc.ventures/jobs/) Open main navigation Close main navigation - [WebRTC Services](https://webrtc.ventures/webrtc-development-services/) Show submenu for WebRTC Services - [Assess](https://webrtc.ventures/services/assess/) - [Build](https://webrtc.ventures/services/build/) - [Integrate](https://webrtc.ventures/services/integrate/) - [Test](https://webrtc.ventures/services/test/) - [Manage](https://webrtc.ventures/services/manage/) - [Migrating From Twilio Video](https://webrtc.ventures/twilio-video/) - Our Partners Show submenu for Our Partners - [Amazon Chime SDK](https://webrtc.ventures/partners/amazon-chime-sdk/) - Stories of Success Show submenu for Stories of Success - [Remote Video Interpreting Platform with Agent Routing](https://webrtc.ventures/successes/remote-video-translation-platform/) - [WebRTC to SIP Gateway for ASL Interpreters](https://webrtc.ventures/successes/webrtc-to-sip-gateway-for-asl-interpreters/) - [Audio Listening Device to Improve Surgical Outcomes](https://webrtc.ventures/successes/audio-listening-device-to-improve-surgical-outcomes/) - [Albania’s First Telemedicine Platform](https://webrtc.ventures/successes/albanias-first-telemedicine-platform/) - [Automated Video Storytelling with the Amazon Chime SDK](https://webrtc.ventures/successes/automated-video-storytelling-with-the-amazon-chime-sdk/) - [Large Scale Video Conferencing](https://webrtc.ventures/successes/large-scale-video-conferencing/) - [Broadcasting and Webcast Control Room](https://webrtc.ventures/successes/live-video-broadcasting-and-webcast-control-room/) - WebRTC Resources Show submenu for WebRTC Resources - [WebRTC Live](https://webrtc.ventures/webrtc-live/) - [WebRTC Tips](https://www.youtube.com/playlist?list=PLayYqdnyegt3QvxX-Rt6wbFIK8727Z5Oz) - [WebRTC.ventures Blog](https://webrtc.ventures/blog/) - [Migrating From Twilio Video](https://webrtc.ventures/twilio-video/) - [WebRTC.ventures Training Program](https://webrtc.ventures/webrtc-ventures-training-program/) - [About Us](https://webrtc.ventures/our-team/) Show submenu for About Us - [Blog](https://webrtc.ventures/blog/) - [Our Team](https://webrtc.ventures/our-team/) - [Jobs](https://webrtc.ventures/jobs/) - [Contact us](https://webrtc.ventures/contact/) [Contact us](https://webrtc.ventures/contact/) Jun 10, 2021 12:00:00 AM # Background Removal with the Vonage Video API ![Picture of Jen Oppenheimer](https://blog.webrtc.ventures/hs-fs/hubfs/Our-Team-Jen.jpg?width=50&name=Our-Team-Jen.jpg) [Jen Oppenheimer](https://blog.webrtc.ventures/author/jen-oppenheimer) Share: [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api) [X Twitter icon](https://twitter.com/intent/tweet?url=https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api) [envelope icon](mailto:?body=https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api) ## Introduction [In a previous blog post](https://webrtc.ventures/2020/01/ai-in-webrtc-background-removal-with-tensorflow-in-an-agora-video-chat-2/) we looked at how we could do background removal with Tensorflow in an [Agora](http://agora.io) Video Chat. We saw how semantic segmentation can be used to detect background pixels in a video stream, and how we could use that information to apply a virtual background. This time, we’ll take a look at integrating background removal with the [Vonage](http://www.vonage.com) Video API, as well as some optimizations that we found that improved performance and usability. ## Initial Approach As we saw last time, the [BodyPix model](https://github.com/tensorflow/tfjs-models/tree/master/body-pix) was used to detect background pixels from a video stream, and then use HTML canvas to paint a filtered out image with our replaced background. Let’s look at a high level overview of our integration: ![](https://blog.webrtc.ventures/hs-fs/hubfs/Imported_Blog_Media/justinblog1-848x1024-1.png?width=424&height=512&name=justinblog1-848x1024-1.png) ![](https://blog.webrtc.ventures/hs-fs/hubfs/Imported_Blog_Media/justinblog1-848x1024-1.png?width=424&height=512&name=justinblog1-848x1024-1.png) On each render cycle, a snapshot of what is currently in the <**video**> tag is taken, and then given to the bodyPix segmentation API. The background image is drawn onto the output ****. Then the background pixels from the segmentation are filtered out and drawn onto the ****. At this point, the **** now has the desired image of the user’s body and a custom background. A video stream is then taken from the **** and provided to the OpenTok publisher as the user’s video stream. This simple design makes it really clear how we are doing our rendering. Changing the rendering logic is pretty easy from this point of view – we just have to change what we are drawing onto the canvas. We could easily add in a conditional flag to use a background blur instead of a background image here. Here’s how we implement our design with some code: ``` private async startCanvasRendering() { const canvasContext = this.outputCanvas.getContext('2d') if (!canvasContext) { throw new Error('Could not get canvas context to start canvas rendering') } const net = await bodyPix.load({ architecture: 'ResNet50', outputStride: 16, quantBytes: 4 }) const canvasClone = (canvasContext.canvas.cloneNode(false) as HTMLCanvasElement) .getContext('2d') if (!canvasClone) { throw new Error('Unable to clone canvas context') } this.renderCanvas(canvasContext, canvasClone, net) } private async renderCanvas( context: CanvasRenderingContext2D, cloneContext: CanvasRenderingContext2D, net: BodyPix) { // Segment input video to identify background pixels vs body pixels const segmentation = await net.segmentPerson(this.localVideo, { flipHorizontal: true, internalResolution: 'high', segmentationThreshold: 0.4 }) // Draw the video to a canvas and use segmentation to remove background cloneContext.drawImage( this.localVideo, 0, 0, this.videoWidth, this.videoHeight) const imgData = cloneContext.getImageData( 0, 0, this.videoWidth, this.videoHeight) const pixels = imgData.data for (let i = 0; i < pixels.length; i += 4) { if (segmentation.data[i / 4] === 0) { pixels[i + 3] = 0 } } cloneContext.putImageData(imgData, 0, 0) context.drawImage( this.backgroundImage, 0, 0, this.videoWidth, this.videoHeight) context.drawImage( cloneContext.canvas, 0, 0, this.videoWidth, this.videoHeight) window.requestAnimationFrame(this.renderCanvas.bind(this, context)) } ``` JavaScript Copy The key here is to use the segmented output from the bodyPix API and make the pixels identified as the background, transparent. That way, when the filtered stream is drawn onto the output canvas, the background will be the custom image. While the canvas is now being rendered with a virtual background, we can capture a video stream from the canvas and provide that to the vonage API: ``` get canvasVideoTrack() { return this.outputCanvas.captureStream().getVideoTracks()[0] } initPublisher() { this.publisher = OT.initPublisher(this.publisherWrapper, { insertMode: 'append', width: '100%', height: '100%', videoSource: this.canvasVideoTrack }) this.session.publish(this.publisher) } ``` JavaScript Copy ## Analysis This process is quite resource intensive. Lower end devices may struggle to keep up with the processing. We can tune a few different areas to make a tradeoff between accuracy of the background removal versus the resource utilization. Reducing the video dimensions and frame rate can have a high impact on resource utilization. Lower end devices may be able to keep up with a resolution of 240p, while higher end devices might be able to run with a more desirable resolution of 480p. The BodyPix library also contains a few different models with different parameters that can be tuned to find a desirable balance. For instance, we could get some high accuracy with high resource utilization with the following model: ``` const net = await bodyPix.load({ architecture: 'ResNet50', outputStride: 16, quantBytes: 4 }) ``` JavaScript Copy For lower end devices we can have lower accuracy background removal with the following: ``` const net = await bodyPix.load({ architecture: 'MobileNetV1', outputStride: 16, multiplier: 0.5, quantBytes: 1 }) ``` JavaScript Copy Apart from tuning, we found one issue with our initial approach. As we can see from the code snippets above, we are recursively invoking **renderCanvas** in a callback from **window.requestAnimationFrame**.   An issue was found when a user was publishing their video from one chrome tab, but was also sharing another chrome tab to the video session. While the user had a separate tab active, their camera video (with background removal) was stuck, and then went away. Looking at some documentation for [window.requestAnimationFrame](https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame): > “requestAnimationFrame() calls are paused in most browsers when running in background tabs or hidden.” This API initially sounds like a really great tool for the job. However, this caveat makes it not so well suited to our use case. Video calling today has expanded way beyond just being a conversation involving a simple video stream. Many users today want to have presentations where they are sharing content. Having our application always being in the foreground is not an option. One might think to try using something like **window.setTimeout** instead, but we have found similar issues with this as well. [Turns out, this has been an issue for some time](https://stackoverflow.com/questions/5927284/how-can-i-make-setinterval-also-work-when-a-tab-is-inactive-in-chrome/5927432#12522580). ## Web Workers Web workers are a way to run javascript in background threads. These background threads do not get throttled the same way as with things like **setTimeout** or **requestAnimationFrame**. With this, we can move our canvas rendering into a background thread and resolve our issue. [Worker Timers](https://www.npmjs.com/package/worker-timers) is a library that provides **setInterval()** and **setTimeout()** APIs that will run the callbacks in on a web worker. This fits in really nicely with our existing code: ``` import * as workerTimers from 'worker-timers' ... private async renderCanvas( context: CanvasRenderingContext2D, cloneContext: CanvasRenderingContext2D, net: BodyPix) { // Segment input video to identify background pixels vs body pixels const segmentation = await net.segmentPerson(this.localVideo, { flipHorizontal: true, internalResolution: 'high', segmentationThreshold: 0.4 }) // Draw the video to a canvas and use segmentation to remove background ... workerTimers.setTimeout( this.renderCanvas.bind(this, context, cloneContext, net), 30) } ``` JavaScript Copy ## Results The Worker Timers library basically provided us with a drop in replacement to our initial approach. The library also was able to successfully render our canvas publisher in a background tab and not have the rendering process throttled by the browser. Higher end devices may be able to run this smoothly. [Testing](https://webrtc.ventures/webrtc-testing-services/) this on an i7 2019 Macbook Pro with 32GB of RAM ran ok for higher quality video with high accuracy settings on the BodyPix model. However this may not be as good for medium to lower end devices many users have today. Lower requirement configurations may be required along with some way to detect the specs of the user’s hardware, and choose and set the model appropriately. You can see the full POC code [here](https://github.com/justinIs/nuxt-background-removal). # Ready to build something new? **At WebRTC.ventures, we can build customizable video applications with features such as recordings, transcriptions, image processing, and more. We have an experienced team ready and happy to help you out. [Contact us today!](https://webrtc.ventures/contact/)** [CPaaS](https://blog.webrtc.ventures/tag/cpaas), [Managed Services](https://blog.webrtc.ventures/tag/managed-services), [Events](https://blog.webrtc.ventures/tag/events), [Video Conferencing](https://blog.webrtc.ventures/tag/video-conferencing), [Real Time Weekly](https://blog.webrtc.ventures/tag/real-time-weekly), [WebRTC Architecture](https://blog.webrtc.ventures/tag/webrtc-architecture), [Broadcasting](https://blog.webrtc.ventures/tag/broadcasting), [Medical](https://blog.webrtc.ventures/tag/medical), [WebRTC Live](https://blog.webrtc.ventures/tag/webrtc-live), [Showcase](https://blog.webrtc.ventures/tag/showcase), [Thoughts](https://blog.webrtc.ventures/tag/thoughts), [Vonage](https://blog.webrtc.ventures/tag/vonage), [Daily](https://blog.webrtc.ventures/tag/daily), [Jobs](https://blog.webrtc.ventures/tag/jobs), [Contact Centers](https://blog.webrtc.ventures/tag/contact-centers), [Telehealth](https://blog.webrtc.ventures/tag/telehealth), [Education](https://blog.webrtc.ventures/tag/education), [Technical](https://blog.webrtc.ventures/tag/technical), [General](https://blog.webrtc.ventures/tag/general), [Story of success](https://blog.webrtc.ventures/tag/story-of-success), [The WebRTC.ventures Blog](https://blog.webrtc.ventures/tag/the-webrtc-ventures-blog), [Testing](https://blog.webrtc.ventures/tag/testing), [Gaming](https://blog.webrtc.ventures/tag/gaming), [DevOps](https://blog.webrtc.ventures/tag/devops), [Symbl.ai](https://blog.webrtc.ventures/tag/symbl-ai) ## Related posts [![](https://blog.webrtc.ventures/hs-fs/hubfs/Imported_Blog_Media/Integrating-a-Slide-Deck-into-a-WebRTC-Conference.png?height=200&name=Integrating-a-Slide-Deck-into-a-WebRTC-Conference.png)](https://blog.webrtc.ventures/2022/06/webrtc-slidedeck) ## [Integrating a Slide Deck into a WebRTC Conference](https://blog.webrtc.ventures/2022/06/webrtc-slidedeck) ![Picture of Jen Oppenheimer](https://blog.webrtc.ventures/hs-fs/hubfs/Our-Team-Jen.jpg?width=50&name=Our-Team-Jen.jpg) [Jen Oppenheimer](https://blog.webrtc.ventures/author/jen-oppenheimer) Jun 22, 2022 12:00:00 AM Building an application with a CPaaS such as Vonage’s Video API (formerly known as TokBox), enables... [Read more](https://blog.webrtc.ventures/2022/06/webrtc-slidedeck) [![](https://blog.webrtc.ventures/hs-fs/hubfs/Imported_Blog_Media/Background-Removal-Using-Insertable-Streams-B.jpg?height=200&name=Background-Removal-Using-Insertable-Streams-B.jpg)](https://blog.webrtc.ventures/2023/02/background-removal-using-insertable-streams) ## [Background Removal Using Insertable Streams](https://blog.webrtc.ventures/2023/02/background-removal-using-insertable-streams) [Hector Zelaya](https://blog.webrtc.ventures/author/hector-zelaya) Feb 13, 2023 12:00:00 AM Background removal and the ability to set custom backgrounds has become a non-negotiable feature in... [Read more](https://blog.webrtc.ventures/2023/02/background-removal-using-insertable-streams) [![](https://blog.webrtc.ventures/hs-fs/hubfs/Imported_Blog_Media/Building-a-Video-Chat-with-the-Amazon-Chime-SDK-Marcell.png?height=200&name=Building-a-Video-Chat-with-the-Amazon-Chime-SDK-Marcell.png)](https://blog.webrtc.ventures/2022/10/building-a-video-chat-with-the-amazon-chime-sdk) ## [Building a Video Chat with the Amazon Chime SDK ](https://blog.webrtc.ventures/2022/10/building-a-video-chat-with-the-amazon-chime-sdk) [Marcell Silva](https://blog.webrtc.ventures/author/marcell-silva) Oct 5, 2022 12:00:00 AM The Amazon Chime SDK is an Amazon service that lets you create multimedia calls using... [Read more](https://blog.webrtc.ventures/2022/10/building-a-video-chat-with-the-amazon-chime-sdk) ### WebRTC.Ventures Services ... - Assess - Build - Integrate - Test - Deploy & Manage ### Contact Us ... **Video should be an opportunity, not a headache.** We’re here to build, integrate, assess and optimize, test, and even deploy and manage your live video application. [Contact us](https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api#contact) ### WebRTC Live ... View [past episodes](https://blog.webrtc.ventures/webrtc-live/) of our webinar interview series about the latest webRTC use cases and technical updates. Join our mailing list to learn about upcoming episodes! ![WebRTC Live](https://blog.webrtc.ventures/hs-fs/hubfs/Template%20Images/widged-bannet.webp?height=90&name=widged-bannet.webp) ![logo-main-light](https://blog.webrtc.ventures/hubfs/logo-main-light.svg "logo-main-light") We’re one of the few agencies in the world dedicated to WebRTC development. This dedication and experience is why so many people trust us to help bring live video application dreams to life.   ### Let's get started! ###### **[Contact us today](https://webrtc.ventures/contact/)** **[info@webrtc.ventures](mailto:info@webrtc.ventures)** ### Join our mailing list We don't e-mail often (a few times a month max), and it will be worth it when we do! Of course, we respect your privacy and will not share your contact information. © 2023 WebRTC.ventures, an [AgilityFeat](https://agilityfeat.com/) company / [Privacy Policy](https://webrtc.ventures/privacy-policy//) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Jen Oppenheimer", "url" : "https://blog.webrtc.ventures/author/jen-oppenheimer" }, "dateModified" : "2024-04-03T18:09:33.591Z", "datePublished" : "2021-06-10T04:00:00.000Z", "headline" : "Background Removal with the Vonage Video API – WebRTC.ventures", "image" : [ "https://blog.webrtc.ventures/hubfs/Imported_Blog_Media/Background-Removal-with-the-Vonage-Video-API.png" ], "mainEntityOfPage" : { "@id" : "https://blog.webrtc.ventures/2021/06/background-removal-with-the-vonage-video-api", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://blog.webrtc.ventures/hubfs/logo-main-light.svg" } } } ```