Discover the new easier way to develop Kurento video applications

Welcome to Kurento¶

Kurento is a WebRTC Media Server and a set of client APIs that simplify the development of advanced video applications for web and smartphone platforms. Its features include group communications, transcoding, recording, mixing, broadcasting and routing of audiovisual flows.

The code is open source, released under the terms of Apache License Version 2.0 and available on GitHub.

You can read more on our page About Kurento and WebRTC. Check now how to get started with Kurento and take a look at our tutorials, which showcase some demo applications.

The main documentation for the project is organized into different sections:

User Documentation
Feature Documentation
Project Documentation

Information about development of Kurento itself is also available:

Developer Documentation

About Kurento and WebRTC¶

Kurento is a WebRTC Media Server and a set of client APIs that simplify the development of advanced video applications for web and smartphone platforms. Its features include group communications, transcoding, recording, mixing, broadcasting and routing of audiovisual flows.

Kurento offers a multimedia framework that eases the task of building multimedia applications with the following features:

Dynamic WebRTC Media pipelines: Kurento allows custom media pipelines connected to WebRTC peers like web browsers and mobile apps. These media pipelines are based on composable elements such as players, recorders, mixers, etc. that can be mix-and-matched, activated, or deactivated at any point in time, even when the media is already flowing.
Client/Server Architecture: Apps developed with Kurento follow a client/server architecture. Kurento Media Server (KMS) is the server and offers a WebSocket interface implementing the Kurento Protocol, which allows Client Applications to define pipeline topologies.
Java and JavaScript Client Applications: The typical use case of a KMS deployment consists of a three-layer architecture, where the user’s browser interacts with the KMS server by means of an intermediate Client Application. There are several official Kurento Client Libraries, supporting the use of Java and JavaScript for the Client Applications. Clients for other languages can be easily implemented following the WebSocket protocol.
Third party Modules: The Kurento Media Server has an extensible architecture based on plugins, which allows third parties to implement modules that can be added to their media pipelines. This allows integration of media processing algorithms to any WebRTC application, like integrating Computer Vision, Augmented Reality, video indexing, and speech analysis. All is needed is to create a new Kurento element, and use it in any already existing media pipeline.

This document contains a high level explanation of how to become a KMS developer. Development of Kurento Client Applications is out of the scope for this document, and won’t be explained here.

The code of Kurento Media Server is open source, released under the terms of Apache License Version 2.0 and available on GitHub.

WebRTC media servers¶

WebRTC is a set of protocols, mechanisms and APIs that provide browsers and mobile applications with Real-Time Communications (RTC) capabilities over peer-to-peer connections. It has been conceived as a technology that allows browsers to communicate directly without the mediation of any kind of infrastructure. However, this model is only enough for creating basic web applications; features such as group communications, media stream recording, media broadcasting, or media transcoding are difficult to implement on top of it. For this reason, many applications end up requiring an intermediate media server.

Peer-to-peer WebRTC approach vs. WebRTC through a media server

Conceptually, a WebRTC media server is just a multimedia middleware where media traffic passes through when moving from source to destinations.

Media servers are capable of processing incoming media streams and offer different outcomes, such as:

Group Communications: Distributing among several receivers the media stream that one peer generates, i.e. acting as a Multi-Conference Unit (“MCU”).
Mixing: Transforming several incoming stream into one single composite stream.
Transcoding: On-the-fly adaptation of codecs and formats between incompatible clients.
Recording: Storing in a persistent way the media exchanged among peers.

Typical WebRTC Media Server capabilities

Kurento Media Server¶

Kurento’s main component is the Kurento Media Server (KMS), responsible for media transmission, processing, recording, and playback. KMS is built on top of the fantastic GStreamer multimedia library, and provides the following features:

Networked streaming protocols, including HTTP, RTP and WebRTC.
Group communications (MCU and SFU functionality) supporting both media mixing and media routing/dispatching.
Generic support for filters implementing Computer Vision and Augmented Reality algorithms.
Media storage that supports writing operations for WebM and MP4 and playing in all formats supported by GStreamer.
Automatic media transcoding between any of the codecs supported by GStreamer, including VP8, H.264, H.263, AMR, OPUS, Speex, G.711, and more.

Kurento Media Server capabilities

Kurento Design Principles¶

Kurento is designed based on the following main principles:

Separate Media and Signaling Planes

Signaling and Media are two separate planes and Kurento is designed so that applications can handle separately those facets of multimedia processing.

Distribution of Media and Application Services

Kurento Media Server and applications can be collocated, escalated or distributed among different machines.

A single application can invoke the services of more than one Kurento Media Server. The opposite also applies, that is, a Kurento Media Server can attend the requests of more than one application.

Suitable for the Cloud

Kurento is suitable to be integrated into cloud environments to act as a PaaS (Platform as a Service) component.

Media Pipelines

Chaining Media Elements via Media Pipelines is an intuitive approach to challenge the complexity of multimedia processing.

Application development

Developers do not need to be aware of internal Kurento Media Server complexities: all the applications can deployed in any technology or framework the developer likes, from client to server. From browsers to cloud services.

End-to-End Communication Capability

Kurento provides end-to-end communication capabilities so developers do not need to deal with the complexity of transporting, encoding/decoding and rendering media on client devices.

Fully Processable Media Streams

Kurento enables not only interactive interpersonal communications (e.g. Skype-like with conversational call push/reception capabilities), but also human-to-machine (e.g. Video on Demand through real-time streaming) and machine-to-machine (e.g. remote video recording, multisensory data exchange) communications.

Modular Processing of Media

Modularization achieved through media elements and pipelines allows defining the media processing functionality of an application through a “graph-oriented” language, where the application developer is able to create the desired logic by chaining the appropriate functionalities.

Auditable Processing

Kurento is able to generate rich and detailed information for QoS monitoring, billing and auditing.

Seamless IMS integration

Kurento is designed to support seamless integration into the IMS infrastructure of Telephony Carriers.

Transparent Media Adaptation Layer

Kurento provides a transparent media adaptation layer to make the convergence among different devices having different requirements in terms of screen size, power consumption, transmission rate, etc. possible.

About OpenVidu¶

OpenVidu is a new project from the same team that created Kurento. It acts as a wrapper around an installation of Kurento Media Server and encapsulates most of its features, in order to greatly simplify some of the most typical use cases of WebRTC, such as conference rooms.

An application developer using OpenVidu doesn’t need to worry about all the low-level technologies and protocols that form part of typical WebRTC communications. The main goal of this project is to provide a simple, effective, easy-to-use API; just include the OpenVidu client-side and OpenVidu Server for handling the media flows, and you’ll have a full-featured WebRTC-capable application.

For more advanced needs, or for applications that require more versatile management of media processing pipelines, Kurento is still the go-to solution; however, if you are planning on building a service which matches one of the simplified use cases covered by OpenVidu, we strongly suggest to check it out as it will be easier and cheaper to go that route.

Check out the OpenVidu project page, which contains all the relevant information: https://openvidu.io/

Getting Started¶

Generally speaking, these are the first steps that any user interested in Kurento should follow:

Know your use case

Choose between Kurento and OpenVidu.

Kurento Media Server has been designed as a general-purpose platform that can be used to create any kind of multimedia streaming applications. This makes KMS a powerful tool, however it also means that there is some unavoidable complexity that the developer must face.

WebRTC is a complex standard with a lot of moving parts, and you need to know about each one of these components and how they work together to achieve the multimedia communications that the standard strives to offer.

If your intended application consists of a complex setup with different kinds of sources and varied use cases, then Kurento is the best leverage you can use.

However, if you intend to solve a simpler use case, such as those of video conference applications, the OpenVidu project builds on top of Kurento to offer a simpler and easier to use solution that will save you time and development effort.
Install KMS

The installation guide explains different ways in which Kurento can be installed in your system. The fastest and easiest one is to use our pre-configured template for Amazon AWS.
Configure KMS

KMS is able to run as-is after a normal installation. However, there are several parameters that you might want to tune in the configuration files.
Write an Application

Write an application that queries the Kurento API to make use of the capabilities offered by KMS. The easiest way of doing this is to build on one of the provided Kurento Clients.

In general, you can use any programming language to write your application, as long as it speaks the Kurento Protocol.

Have a look at the features offered by Kurento, and follow any of the multiple tutorials that explain how to build basic applications.
Ask for help

If you face any issue with Kurento itself or have difficulties configuring the plethora of mechanisms that form part of WebRTC, don’t hesitate to ask for help to our community of users.

Still, there are times when the problems at hand require more specialized study. If you wish to get help from expert people with more inside knowledge of Kurento, get in contact with us to request Commercial Support.
Enjoy!

Kurento is a project that aims to bring the latest innovations closer to the people, and help connect them together. Make a great application with it, and let us know! We will be more than happy to find out about who is using Kurento and what is being built with it :-)

Installation Guide¶

Kurento Media Server (KMS) can be installed in multiple ways

Using an EC2 instance in the Amazon Web Services (AWS) cloud service. Using AWS is suggested to users who don’t want to worry about properly configuring a server and all software packages, because the provided setup does all this automatically.
Using the Docker images provided by the Kurento team. Docker images allow to run Kurento in any host machine, so for example it’s possible to run KMS on top of a Fedora or CentOS system. In theory it could even be possible to run under Windows, but so far that possibility hasn’t been explored, so you would be at your own risk.
A local installation with apt-get install, in any Ubuntu machine. This method allows to have total control of the installation process. Besides installing KMS, a common need is to also install a STUN or TURN server, especially if KMS or any of its clients are located behind a NAT firewall.

If you want to try nightly builds of KMS, then head to the section Installing Nightly Builds.

Amazon Web Services¶

The Kurento project provides an AWS CloudFormation template file. It can be used to create an EC2 instance that comes with everything needed and totally pre-configured to run KMS, including a Coturn server.

Note that this template is specifically tailored to be deployed on the default Amazon Virtual Private Cloud (Amazon VPC) network. You need an Amazon VPC to deploy this template.

Follow these steps to use it:

Access the AWS CloudFormation Console.
Click on Create Stack.
Look for the section Choose a template, and choose the option Specify an Amazon S3 template URL. Then, in the text field that gets enabled, paste this URL: https://s3-eu-west-1.amazonaws.com/aws.kurento.org/KMS-Coturn-cfn.yaml.
Follow through the steps of the configuration wizard:

4.1. Stack name: A descriptive name for your Stack.

4.2. InstanceType: Choose an appropriate size for your instance. Check the different ones.

4.3. KeyName: You need to create an RSA key beforehand in order to access the instance. Check AWS documentation on how to create one.

4.4. SSHLocation: For security reasons you may need to restrict SSH traffic to allow connections only from specific locations. For example, from your home or office.

4.5. TurnUser: User name for the TURN server.

4.6. TurnPassword: Password required to use the TURN server.

Note

The template file includes Coturn as a TURN server. The default user/password for this server is kurento/kurento. You can optionally change the username, but make sure to change the default password.
Finish the Stack creation process. Wait until the status of the newly created Stack reads CREATE_COMPLETE.
Select the Stack and then open the Outputs tab, where you’ll find the instance’s public IP address, and the Kurento Media Server endpoint URL that must be used by Application Servers.

Docker image¶

Kurento’s Docker Hub contains images built from each KMS release. Just head to the kurento-media-server Docker Hub page, and follow the instructions you’ll find there.

Local Installation¶

With this method, you will install KMS from the native Ubuntu package repositories made available by the Kurento project.

KMS has explicit support KMS has explicit support for two Long-Term Support (LTS) versions of Ubuntu: Ubuntu 16.04 (Xenial) and Ubuntu 18.04 (Bionic) (64-bits only). To install KMS, open a terminal and follow these steps:

Make sure that GnuPG is installed.

sudo apt-get update \
  && sudo apt-get install --no-install-recommends --yes \
     gnupg

Define what version of Ubuntu is installed in your system.

Run only one of these lines:

# Run ONLY ONE of these lines:
DISTRO="xenial"  # KMS for Ubuntu 16.04 (Xenial)
DISTRO="bionic"  # KMS for Ubuntu 18.04 (Bionic)

Add the Kurento repository to your system configuration.

Run these two commands in the same terminal you used in the previous step:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5AFA7A83

sudo tee "/etc/apt/sources.list.d/kurento.list" >/dev/null <<EOF
# Kurento Media Server - Release packages
deb [arch=amd64] http://ubuntu.openvidu.io/6.10.0 $DISTRO kms6
EOF

Install KMS:

sudo apt-get update \
  && sudo apt-get install --yes kurento-media-server

This will install the KMS release version that was specified in the previous commands.

The server includes service files which integrate with the Ubuntu init system, so you can use the following commands to start and stop it:

sudo service kurento-media-server start
sudo service kurento-media-server stop

Log messages from KMS will be available in /var/log/kurento-media-server/. For more details about KMS logs, check Debug Logging.

STUN and TURN servers¶

If Kurento Media Server, its Application Server, or any of the clients are located behind a NAT, you need to use a STUN or a TURN server in order to achieve NAT traversal. In most cases, STUN is effective in addressing the NAT issue with most consumer network devices (routers). However, it doesn’t work for many corporate networks, so a TURN server becomes necessary.

Apart from that, you need to open all UDP ports in your system configuration, as STUN will use any random port from the whole [0-65535] range.

Note

The features provided by TURN are a superset of those provided by STUN. This means that you don’t need to configure a STUN server if you are already using a TURN server.

For more information about why and when STUN/TURN is needed, check out the FAQ: When is STUN needed?

STUN server¶

To configure a STUN server in KMS, uncomment the following lines in the WebRtcEndpoint configuration file, located at /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini:

stunServerAddress=<serverIp>
stunServerPort=<serverPort>

Note

Be careful since comments inline (with ;) are not allowed for parameters in the configuration files. Thus, the following line is not correct:

stunServerAddress=<serverIp> ; Only IP addresses are supported

… and must be changed to something like this:

; Only IP addresses are supported
stunServerAddress=<serverIp>

The parameter serverIp should be the public IP address of the STUN server. It must be an IP address, not a domain name.

It should be easy to find some public STUN servers that are made available for free. For example:

194.66.127:19302
194.71.127:19302
125.200.127:19302
125.204.127:19302
194.72.127:19302
125.23.127:3478
72.174.163:3478
72.174.165:3478
72.174.167:3478
72.174.161:3478
97.25.20:3478
71.2.168:3478
227.67.194:3478
227.67.195:3478
23.150.92:3478
72.169.155:3478
72.169.156:3478
72.169.164:3478
72.169.166:3478
72.174.162:3478
72.174.164:3478
72.174.166:3478
72.174.160:3478
172.47.69:3478

TURN server¶

To configure a TURN server in KMS, uncomment the following lines in the WebRtcEndpoint configuration file, located at /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini:

turnURL=<user>:<password>@<serverIp>:<serverPort>

The parameter serverIp should be the public IP address of the TURN server. It must be an IP address, not a domain name.

See some examples of TURN configuration below:

turnURL=kurento:kurento@WWW.XXX.YYY.ZZZ:3478

… or using a free access Numb TURN/STUN server:

turnURL=user:password@66.228.45.110:3478

Note that it is somewhat easy to find free STUN servers available on the net, because their functionality is pretty limited and it is not costly to keep them working for free. However, this doesn’t happen with TURN servers, which act as a media proxy between peers and thus the cost of maintaining one is much higher.

It is rare to find a TURN server which works for free while offering good performance. Usually, each user opts to maintain their own private TURN server instances.

Coturn is an open source implementation of a TURN/STUN server. In the FAQ section there is a description about how to install and configure it.

Check your installation¶

To verify that KMS is up and running, use this command and look for the kurento-media-server process:

ps -ef | grep kurento-media-server

> nobody  1270  1  0 08:52 ?  00:01:00  /usr/bin/kurento-media-server

Unless configured otherwise, KMS will open the port 8888 to receive requests and send responses by means of the Kurento Protocol. Use this command to verify that this port is listening for incoming packets:

sudo netstat -tupan | grep kurento

> tcp6  0  0 :::8888  :::*  LISTEN  1270/kurento-media-server

Installing Nightly Builds¶

Some components of KMS are built nightly, with the code developed during that same day. Other components are built immediately when code is merged into the source repositories.

These builds end up being uploaded to Development repositories so they can be installed by anyone. Use these if you want to develop Kurento itself, or if you want to try the latest changes before they are officially released.

Warning

Nightly builds always represent the current state on the software development; 99% of the time this is stable code, very close to what will end up being released.

However, it’s also possible (although unlikely) that these builds might include undocumented changes, regressions, bugs or deprecations. It’s safer to be conservative and avoid using nightly builds in a production environment, unless you have a strong reason to do it.

Note

If you are looking to build KMS from the source code, then you should check the section aimed at development of KMS itself: Building from sources.

Kurento Media Server¶

Docker image¶

Kurento’s Docker Hub contains images built from each KMS nightly version. Just head to the kurento-media-server-dev Docker Hub page, and follow the instructions you’ll find there. The nightly images work exactly the same as Kurento Docker release images, but just using nightly builds instead of release ones.

Local Installation¶

The steps to install a nightly version of KMS are pretty much the same as those explained in Local Installation, with the only change of using a different package repository.

Make sure that GnuPG is installed.

sudo apt-get update \
  && sudo apt-get install --no-install-recommends --yes \
     gnupg

Define what version of Ubuntu is installed in your system.

Run only one of these lines:

# Run ONLY ONE of these lines:
DISTRO="xenial"  # KMS for Ubuntu 16.04 (Xenial)
DISTRO="bionic"  # KMS for Ubuntu 18.04 (Bionic)

Add the Kurento repository to your system configuration.

Run these two commands in the same terminal you used in the previous step:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5AFA7A83

sudo tee "/etc/apt/sources.list.d/kurento.list" >/dev/null <<EOF
# Kurento Media Server - Nightly packages
deb [arch=amd64] http://ubuntu.openvidu.io/dev $DISTRO kms6
EOF

Install KMS:

sudo apt-get update \
  && sudo apt-get install --yes kurento-media-server

This will install the latest KMS nightly version.

Kurento Java Client¶

The development builds of the Kurento Java Client are made available for Maven in https://maven.openvidu.io/ To use these, you need to add first this repository to your Maven configuration.

Adding a repository to Maven can be done at three scope levels:

Project level.

This will add access to development builds only for the project where the configuration is done. Open the project’s pom.xml and include this:

<project>
  ...
  <repositories>
    <repository>
      <id>kurento-snapshots</id>
      <name>Kurento Snapshots</name>
      <url>https://maven.openvidu.io/repository/snapshots/</url>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>kurento-snapshots</id>
      <name>Kurento Snapshots</name>
      <url>https://maven.openvidu.io/repository/snapshots/</url>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </pluginRepository>
  </pluginRepositories>
  ...
</project>

After this is included, there are two ways to use the updated versions:

In the same pom.xml, look for the desired <dependency> and change its version. For example:

<dependency>
  <groupId>org.kurento</groupId>
  <artifactId>kurento-client</artifactId>
  <version>6.10.0-SNAPSHOT</version>
</dependency>

If you have not specified a dependency version, use the -U switch in your next Maven run to force updating all dependencies.

User and System levels.

The file settings.xml provides configuration for all projects, but its contents have a different reach depending on where it is located:

At $HOME/.m2/settings.xml, it defines the settings that will be applied for a single user.
At /etc/maven/settings.xml, it defines the settings that will be applied for all Maven users on a machine.

To use this method, first edit the settings file at one of the mentioned locations, and include this:

<settings>
  ...
  <profiles>
    <profile>
      <id>kurento</id>
      <repositories>
        <repository>
          <id>kurento-snapshots</id>
          <name>Kurento Snapshots</name>
          <url>https://maven.openvidu.io/repository/snapshots/</url>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>kurento-snapshots</id>
          <name>Kurento Snapshots</name>
          <url>https://maven.openvidu.io/repository/snapshots/</url>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  ...
</settings>

After this is included, use the -Pkurento switch in your next Maven run to enable the new profile, so all artifacts get downloaded into you local repository. Once in your local repository, Maven can successfully resolve the dependencies and the profile no longer needs to be activated in future runs.

For more information about adding snapshot repositories to Maven, check their official documentation: Guide to Testing Development Versions of Plugins.

Kurento JavaScript Client¶

Node.js¶

If you are using the Kurento JavaScript Client from a Node.js application and want to use the latest development version of this library, you have to change the dependencies section in the application’s package.json. This way, NPM will point directly to the development repository:

"dependencies": {
    "kurento-client": "Kurento/kurento-client-js#master",
}

Browser JavaScript¶

If you are using the Kurento JavaScript Client from a browser application, with Bower to handle JS dependencies, and want to use the latest development version of this library, you have to change the dependencies section in the application’s bower.json. This way, Bower will point directly to the development repository:

"dependencies": {
    "kurento-client": "master",
    "kurento-utils": "master",
}

Alternatively, if your browser application is pointing directly to JavaScript libraries from HTML resources, then you have to change to development URLs:

<script type="text/javascript"
    src="http://builds.openvidu.io/dev/master/latest/js/kurento-client.min.js">
</script>

Kurento Tutorials¶

This section contains tutorials showing how to use Kurento framework to build different types of WebRTC and multimedia applications. These tutorials come in three flavors:

Java: Showing applications where clients interact with Spring Boot-based applications, that host the logic orchestrating the communication among clients and control Kurento Media Server capabilities.
Browser JavaScript: These show applications executing at the browser and communicating directly with the Kurento Media Server. In these tutorials all logic is directly hosted by the browser. Hence, no application server is necessary.
Node.js: In which clients interact with an application server made with Node.js technology. The application server holds the logic orchestrating the communication among the clients and controlling Kurento Media Server capabilities for them.

Note

The tutorials have been created with learning objectives. They are not intended to be used in production environments where different unmanaged error conditions may emerge. Use at your own risk!

Note

These tutorials require HTTPS in order to use WebRTC. Following instructions will provide further information about how to enable application security.

Hello World¶

This is one of the simplest WebRTC applications you can create with Kurento. It implements a WebRTC loopback (a WebRTC media stream going from client to Kurento Media Server and back to the client)

Kurento Java Tutorial - Hello World¶

This web application has been designed to introduce the principles of programming with Kurento for Java developers. It consists on a WebRTC video communication in mirror (loopback). This tutorial assumes you have basic knowledge of Java, JavaScript, HTML and WebRTC. We also recommend reading the section About Kurento and WebRTC before starting this tutorial.

Note

This application uses HTTPS. It will work fine is you run it in localhost and accept a security exception in the browser, but you should secure your application if running remotely. For more info, check Configure a Java server to use HTTPS.

Quick start¶

Follow these steps to run this demo application:

Install Kurento Media Server: Installation Guide.

Run these commands:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-hello-world
git checkout 6.10.0
mvn -U clean spring-boot:run -Dkms.url=ws://localhost:8888/kurento

Open the demo page with a WebRTC-compliant browser (Chrome, Firefox): https://localhost:8443/
Click on Start to begin the demo.
Grant access to your webcam.
As soon as the loopback connection is negotiated and established, you should see your webcam video in both the local and remote placeholders.
Click on Stop to finish the demo.

Understanding this example¶

Kurento provides developers a Kurento Java Client to control the Kurento Media Server. This client library can be used in any kind of Java application: Server Side Web, Desktop, Android, etc. It is compatible with any framework like Java EE, Spring, Play, Vert.x, Swing and JavaFX.

This Hello World demo is one of the simplest web applications you can create with Kurento. The following picture shows a screenshot of this demo running:

Kurento Hello World Screenshot: WebRTC in loopback

The interface of the application (an HTML web page) is composed by two HTML5 <video> tags: one showing the local stream (as captured by the device webcam) and the other showing the remote stream sent by the media server back to the client.

The logic of the application is quite simple: the local stream is sent to the Kurento Media Server, which sends it back to the client without modifications. To implement this behavior, we need to create a Media Pipeline composed by a single Media Element, i.e. a WebRtcEndpoint, which holds the capability of exchanging full-duplex (bidirectional) WebRTC media flows. This media element is connected to itself so that the media it receives (from browser) is sent back (to browser). This media pipeline is illustrated in the following picture:

Kurento Hello World Media Pipeline in context

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used:

A WebSocket is created between client and application server to implement a custom signaling protocol.
Another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server.

This communication takes place using the Kurento Protocol. For a detailed description, please read this section: Kurento Protocol.

The diagram below shows a complete sequence diagram, of the interactions with the application interface to: i) JavaScript logic; ii) Application server logic (which uses the Kurento Java Client); iii) Kurento Media Server.

Complete sequence diagram of Kurento Hello World (WebRTC in loopbak) demo

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

In the following, figure you can see a class diagram of the server side code:

Server-side class diagram of the HelloWorld app

The main class of this demo is HelloWorldApp.

As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost, listening in port 8888. If you reproduce this example, you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@SpringBootApplication
@EnableWebSocket
public class HelloWorldApp implements WebSocketConfigurer {
  @Bean
  public HelloWorldHandler handler() {
    return new HelloWorldHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/helloworld");
  }

  public static void main(String[] args) throws Exception {
    SpringApplication.run(HelloWorldApp.class, args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate with the application server, by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder that processes WebSocket requests in the path /helloworld.

The class HelloWorldHandler implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

public void handleTextMessage(WebSocketSession session, TextMessage message)
    throws Exception {
  [...]
  switch (messageId) {
    case "start":
      start(session, jsonMessage);
      break;
    case "stop": {
      stop(session);
      break;
    }
    case "onIceCandidate":
      onRemoteIceCandidate(session, jsonMessage);
      break;
    default:
      sendError(session, "Invalid message, ID: " + messageId);
      break;
  }
  [...]
}

The start() method performs the following actions:

Configure media processing logic. This is the part in which the application configures how Kurento has to process the media. In other words, the media pipeline is created here. To that aim, the object KurentoClient is used to create a MediaPipeline object. Using it, the media elements we need are created and connected. In this case, we only instantiate one WebRtcEndpoint for receiving the WebRTC stream and sending it back to the client.
```
final MediaPipeline pipeline = kurento.createMediaPipeline();

final WebRtcEndpoint webRtcEp =
    new WebRtcEndpoint.Builder(pipeline).build();

webRtcEp.connect(webRtcEp);
```

Create event listeners. All objects managed by Kurento have the ability to emit several types of events, as explained in Endpoint Events. Application Servers can listen for them in order to have more insight about what is going on inside the processing logic of the media server. It is a good practice to listen for all possible events, so the client application has as much information as possible.

// Common events for all objects that inherit from BaseRtpEndpoint
addErrorListener(
    new EventListener<ErrorEvent>() { ... });
addMediaFlowInStateChangeListener(
    new EventListener<MediaFlowInStateChangeEvent>() { ... });
addMediaFlowOutStateChangeListener(
    new EventListener<MediaFlowOutStateChangeEvent>() { ... });
addConnectionStateChangedListener(
    new EventListener<ConnectionStateChangedEvent>() { ... });
addMediaStateChangedListener(
    new EventListener<MediaStateChangedEvent>() { ... });
addMediaTranscodingStateChangeListener(
    new EventListener<MediaTranscodingStateChangeEvent>() { ... });

// Events specific to objects of class WebRtcEndpoint
addIceCandidateFoundListener(
    new EventListener<IceCandidateFoundEvent>() { ... });
addIceComponentStateChangeListener(
    new EventListener<IceComponentStateChangeEvent>() { ... });
addIceGatheringDoneListener(
    new EventListener<IceGatheringDoneEvent>() { ... });
addNewCandidatePairSelectedListener(
    new EventListener<NewCandidatePairSelectedEvent>() { ... });

WebRTC SDP negotiation. In WebRTC, the SDP Offer/Answer model is used to negotiate the audio or video tracks that will be exchanged between peers, together with a subset of common features that they support. This negotiation is done by generating an SDP Offer in one of the peers, sending it to the other peer, and bringing back the SDP Answer that will be generated in response.

In this particular case, the SDP Offer has been generated by the browser and is sent to Kurento, which then generates an SDP Answer that must be sent back to the browser as a response.
```
// 'webrtcSdpOffer' is the SDP Offer generated by the browser;
// send the SDP Offer to KMS, and get back its SDP Answer
String webrtcSdpAnswer = webRtcEp.processOffer(webrtcSdpOffer);
sendMessage(session, webrtcSdpAnswer);
```
Gather ICE candidates. While the SDP Offer/Answer negotiation is taking place, each one of the peers can start gathering the connectivity candidates that will be used for the ICE protocol. This process works very similarly to how a browser notifies its client code of each newly discovered candidate by emitting the event RTCPeerConnection.onicecandidate; likewise, Kurento’s WebRtcEndpoint will notify its client application for each gathered candidate via the event IceCandidateFound.
```
webRtcEp.gatherCandidates();
```

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences.

These libraries are brought to the project as Maven dependencies which download all required files from WebJars.org; they are loaded in the index.html page, and are used in the index.js file.

In the following snippet we can see the creation of the WebSocket in the path /helloworld. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in function start, the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('ws://' + location.host + '/helloworld');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error)
            return console.error('Error adding candidate: ' + error);
      });
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log('Starting video call ...');

   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   console.log('Creating WebRtcPeer and generating local sdp offer ...');

   var options = {
      localVideo : videoInput,
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function(error) {
            if (error)
               return console.error(error);
            webRtcPeer.generateOffer(onOffer);
         });
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

function onIceCandidate(candidate) {
   console.log('Local candidate' + JSON.stringify(candidate));

   var message = {
      id : 'onIceCandidate',
      candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(I_CAN_STOP);
   console.log('SDP answer received from server. Processing ...');

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   console.log('Stopping video call ...');
   setState(I_CAN_START);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
         id : 'stop'
      }
      sendMessage(message);
   }
   hideSpinner(videoInput, videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with WebJars.

JavaScript - Hello world¶

This web application has been designed to introduce the principles of programming with Kurento for JavaScript developers. It consists on a WebRTC video communication in mirror (loopback). This tutorial assumes you have basic knowledge of JavaScript, HTML and WebRTC. We also recommend reading the Introducing Kurento section before starting this tutorial.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

You’ll need to install Kurento Media Server before running this example. Read installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by a HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-hello-world
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the parameter ws_uri to the URL, as follows:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

Kurento provides developers a Kurento JavaScript Client to control Kurento Media Server. This client library can be used in any kind of JavaScript application including desktop and mobile browsers.

This hello world demo is one of the simplest web applications you can create with Kurento. The following picture shows an screenshot of this demo running:

Kurento Hello World Screenshot: WebRTC in loopback

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one showing the local stream (as captured by the device webcam) and the other showing the remote stream sent by the media server back to the client.

The logic of the application is quite simple: the local stream is sent to the Kurento Media Server, which sends it back to the client without modifications. To implement this behavior, we need to create a Media Pipeline composed by a single Media Element, i.e. a WebRtcEndpoint, which holds the capability of exchanging full-duplex (bidirectional) WebRTC media flows. This media element is connected to itself,, so that the media it receives (from browser) is send back (to browser). This media pipeline is illustrated in the following picture:

Kurento Hello World Media Pipeline in context

This is a web application, and therefore it follows a client-server architecture. Nevertheless, due to the fact that we are using the Kurento JavaScript client, there is not need to use an application server since all the application logic is held by the browser. The Kurento JavaScript Client is used directly to control Kurento Media Server by means of a WebSocket bidirectional connection:

Complete sequence diagram of Kurento Hello World (WebRTC in loopbak) demo

The following sections analyze in deep the client-side (JavaScript) code of this application, the dependencies, and how to run the demo. The complete source code can be found in GitHub.

JavaScript Logic¶

The Kurento hello-world demo follows a Single Page Application architecture (SPA). The interface is the following HTML page: index.html. This web page links two Kurento JavaScript libraries:

kurento-client.js : Implementation of the Kurento JavaScript Client.
kurento-utils.js : Kurento utility library aimed to simplify the WebRTC management in the browser.

In addition, these two JavaScript libraries are also required:

Bootstrap : Web framework for developing responsive web sites.
jquery.js : Cross-platform JavaScript library designed to simplify the client-side scripting of HTML.
adapter.js : WebRTC JavaScript utility library maintained by Google that abstracts away browser differences.
ekko-lightbox : Module for Bootstrap to open modal images, videos, and galleries.
demo-console : Custom JavaScript console.

The specific logic of the Hello World JavaScript demo is coded in the following JavaScript file: index.js. In this file, there is a function which is called when the green button labeled as Start in the GUI is clicked.

var startButton = document.getElementById("start");

startButton.addEventListener("click", function() {
   var options = {
     localVideo: videoInput,
     remoteVideo: videoOutput
   };

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)
      this.generateOffer(onOffer)
   });

   [...]
}

The function WebRtcPeer.WebRtcPeerSendrecv abstracts the WebRTC internal details (i.e. PeerConnection and getUserStream) and makes possible to start a full-duplex WebRTC communication, using the HTML video tag with id videoInput to show the video camera (local stream) and the video tag videoOutput to show the remote stream provided by the Kurento Media Server.

Inside this function, a call to generateOffer is performed. This function accepts a callback in which the SDP offer is received. In this callback we create an instance of the KurentoClient class that will manage communications with the Kurento Media Server. So, we need to provide the URI of its WebSocket endpoint. In this example, we assume it’s listening in port 8888 at the same host than the HTTP serving the application.

[...]

var args = getopts(location.search,
{
  default:
  {
    ws_uri: 'ws://' + location.hostname + ':8888/kurento',
    ice_servers: undefined
  }
});

[...]

kurentoClient(args.ws_uri, function(error, client){
  [...]
};

Once we have an instance of kurentoClient, we need to create a Media Pipeline, as follows:

client.create("MediaPipeline", function(error, _pipeline){
   [...]
});

If everything works correctly, we will have an instance of a media pipeline (variable _pipeline in this example). With it, we are able to create Media Elements. In this example we just need a single WebRtcEndpoint.

In WebRTC, SDP is used for negotiating media exchanges between applications. Such negotiation happens based on the SDP offer and answer exchange mechanism by gathering the ICE candidates as follows:

pipeline = _pipeline;

pipeline.create("WebRtcEndpoint", function(error, webRtc){
   if(error) return onError(error);

   setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

   webRtc.processOffer(sdpOffer, function(error, sdpAnswer){
     if(error) return onError(error);

     webRtcPeer.processAnswer(sdpAnswer, onError);
   });
   webRtc.gatherCandidates(onError);

   [...]
});

Finally, the WebRtcEndpoint is connected to itself (i.e., in loopback):

webRtc.connect(webRtc, function(error){
   if(error) return onError(error);

   console.log("Loopback established");
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

All dependencies of this demo can to be obtained using Bower. The list of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
}

To get these dependencies, just run the following shell command:

bower install

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at Bower.

Node.js - Hello world¶

This web application has been designed to introduce the principles of programming with Kurento for Node.js developers. It consists on a WebRTC video communication in mirror (loopback). This tutorial assumes you have basic knowledge of JavaScript, Node.js, HTML and WebRTC. We also recommend reading the Introducing Kurento section before starting this tutorial.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-hello-world
git checkout 6.10.0
npm install
npm start

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

Kurento provides developers a Kurento JavaScript Client to control Kurento Media Server. This client library can be used from compatible JavaScript engines including browsers and Node.js.

This hello world demo is one of the simplest web application you can create with Kurento. The following picture shows an screenshot of this demo running:

Kurento Hello World Screenshot: WebRTC in loopback

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one showing the local stream (as captured by the device webcam) and the other showing the remote stream sent by the media server back to the client.

The logic of the application is quite simple: the local stream is sent to the Kurento Media Server, which returns it back to the client without modifications. To implement this behavior we need to create a Media Pipeline composed by a single Media Element, i.e. a WebRtcEndpoint, which holds the capability of exchanging full-duplex (bidirectional) WebRTC media flows. This media element is connected to itself so that the media it receives (from browser) is send back (to browser). This media pipeline is illustrated in the following picture:

Kurento Hello World Media Pipeline in context

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side we use a Node.js application server consuming the Kurento JavaScript Client API to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Javascript Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The diagram below shows an complete sequence diagram from the interactions with the application interface to: i) JavaScript logic; ii) Application server logic (which uses the Kurento JavaScript Client); iii) Kurento Media Server.

Complete sequence diagram of Kurento Hello World (WebRTC in loopbak) demo

The following sections analyze in deep the server and client-side code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using the express framework for Node.js, but express is not a requirement for Kurento. The main script of this demo is server.js.

In order to communicate the JavaScript client and the Node application server a WebSocket is used. The incoming messages to this WebSocket (variable ws in the code) are conveniently handled to implemented the signaling protocol depicted in the figure before (i.e. messages start, stop, onIceCandidate).

var ws = require('ws');

[...]

var wss = new ws.Server({
    server : server,
    path : '/helloworld'
});

/*
 * Management of WebSocket messages
 */
wss.on('connection', function(ws) {
    var sessionId = null;
    var request = ws.upgradeReq;
    var response = {
        writeHead : {}
    };

    sessionHandler(request, response, function(err) {
        sessionId = request.session.id;
        console.log('Connection received with sessionId ' + sessionId);
    });

    ws.on('error', function(error) {
        console.log('Connection ' + sessionId + ' error');
        stop(sessionId);
    });

    ws.on('close', function() {
        console.log('Connection ' + sessionId + ' closed');
        stop(sessionId);
    });

    ws.on('message', function(_message) {
        var message = JSON.parse(_message);
        console.log('Connection ' + sessionId + ' received message ', message);

        switch (message.id) {
        case 'start':
            sessionId = request.session.id;
            start(sessionId, ws, message.sdpOffer, function(error, sdpAnswer) {
                if (error) {
                    return ws.send(JSON.stringify({
                        id : 'error',
                        message : error
                    }));
                }
                ws.send(JSON.stringify({
                    id : 'startResponse',
                    sdpAnswer : sdpAnswer
                }));
            });
            break;

        case 'stop':
            stop(sessionId);
            break;

        case 'onIceCandidate':
            onIceCandidate(sessionId, message.candidate);
            break;

        default:
            ws.send(JSON.stringify({
                id : 'error',
                message : 'Invalid message ' + message
            }));
            break;
        }

    });
});

In order to control the media capabilities provided by the Kurento Media Server, we need an instance of the KurentoClient in the Node application server. In order to create this instance, we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888.

var kurento = require('kurento-client');

var kurentoClient = null;

var argv = minimist(process.argv.slice(2), {
    default: {
        as_uri: 'https://localhost:8443/',
        ws_uri: 'ws://localhost:8888/kurento'
    }
});

[...]

function getKurentoClient(callback) {
    if (kurentoClient !== null) {
        return callback(null, kurentoClient);
    }

    kurento(argv.ws_uri, function(error, _kurentoClient) {
        if (error) {
            console.log("Could not find media server at address " + argv.ws_uri);
            return callback("Could not find media server at address" + argv.ws_uri
                    + ". Exiting with error " + error);
        }

        kurentoClient = _kurentoClient;
        callback(null, kurentoClient);
    });
}

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server. Our first operation is to create a Media Pipeline, then we need to create the Media Elements and connect them. In this example, we just need a single WebRtcEndpoint connected to itself (i.e. in loopback). These functions are called in the start function, which is fired when the start message is received:

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint
                        }
                        return callback(null, sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        return callback(null, webRtcEndpoint);
    });
}

 function connectMediaElements(webRtcEndpoint, callback) {
    webRtcEndpoint.connect(webRtcEndpoint, function(error) {
        if (error) {
            return callback(error);
        }
        return callback(null);
    });
}

As of Kurento Media Server 6.0, the WebRTC negotiation is done by exchanging ICE candidates between the WebRTC peers. To implement this protocol, the webRtcEndpoint receives candidates from the client in OnIceCandidate function. These candidates are stored in a queue when the webRtcEndpoint is not available yet. Then these candidates are added to the media element by calling to the addIceCandidate method.

var candidatesQueue = {};

[...]

function onIceCandidate(sessionId, _candidate) {
    var candidate = kurento.getComplexType('IceCandidate')(_candidate);

    if (sessions[sessionId]) {
        console.info('Sending candidate');
        var webRtcEndpoint = sessions[sessionId].webRtcEndpoint;
        webRtcEndpoint.addIceCandidate(candidate);
    }
    else {
        console.info('Queueing candidate');
        if (!candidatesQueue[sessionId]) {
            candidatesQueue[sessionId] = [];
        }
        candidatesQueue[sessionId].push(candidate);
    }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application. These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /helloworld. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication.

var ws = new WebSocket('ws://' + location.host + '/helloworld');
var webRtcPeer;

const I_CAN_START = 0;
const I_CAN_STOP = 1;
const I_AM_STARTING = 2;

[...]

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate)
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

In the function start the method WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to create the webRtcPeer object, which is used to handle the WebRTC communication.

videoInput = document.getElementById('videoInput');
videoOutput = document.getElementById('videoOutput');

[...]

function start() {
   console.log('Starting video call ...')

   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   console.log('Creating WebRtcPeer and generating local sdp offer ...');

    var options = {
      localVideo: videoInput,
      remoteVideo: videoOutput,
      onicecandidate : onIceCandidate
    }

    webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
        if(error) return onError(error);
        this.generateOffer(onOffer);
    });
}

function onIceCandidate(candidate) {
      console.log('Local candidate' + JSON.stringify(candidate));

      var message = {
         id : 'onIceCandidate',
         candidate : candidate
      };
      sendMessage(message);
}

function onOffer(error, offerSdp) {
   if(error) return onError(error);

   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

Dependencies¶

Server-side dependencies of this demo are managed using npm. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   [...]
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   [...]
   "kurento-utils" : "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at npm and Bower.

WebRTC Magic Mirror¶

This web application consists on a WebRTC loopback video communication, adding a funny hat over detected faces. This is an example of a Computer Vision and Augmented Reality filter.

Java - WebRTC magic mirror¶

This web application extends the Hello World Tutorial, adding media processing to the basic WebRTC loopback.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-magic-mirror
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application uses computer vision and augmented reality techniques to add a funny hat on top of faces. The following picture shows a screenshot of the demo running in a web browser:

Kurento Magic Mirror Screenshot: WebRTC with filter in loopback

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRtcEndpoint: Provides full-duplex (bidirectional) WebRTC capabilities.
FaceOverlay filter: Computer vision filter that detects faces in the video stream and puts an image on top of them. In this demo the filter is configured to put a Super Mario hat).

WebRTC with filter in loopback Media Pipeline

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

To communicate the client with the Java EE application server we have designed a simple signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server is as follows: i) Client starts the Magic Mirror. ii) Client stops the Magic Mirror.

If any exception happens, server sends an error message to the client. The detailed message sequence between client and application server is depicted in the following picture:

One to one video call signaling protocol

As you can see in the diagram, an SDP and ICE candidates needs to be exchanged between client and server to establish the WebRTC session between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer at the browser with the WebRtcEndpoint at the server. The complete source code of this demo can be found in GitHub.

Application Server Side¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

In the following figure you can see a class diagram of the server side code:

Server-side class diagram of the MagicMirror app

The main class of this demo is named MagicMirrorApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to your applications. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost, listening in port 8888. If you reproduce this tutorial, you’ll need to insert the specific location of your Kurento Media Server instance there.

@EnableWebSocket
@SpringBootApplication
public class MagicMirrorApp implements WebSocketConfigurer {

   final static String DEFAULT_KMS_WS_URI = "ws://localhost:8888/kurento";

   @Bean
   public MagicMirrorHandler handler() {
      return new MagicMirrorHandler();
   }

   @Bean
   public KurentoClient kurentoClient() {
      return KurentoClient.create();
   }

   @Override
   public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
      registry.addHandler(handler(), "/magicmirror");
   }

   public static void main(String[] args) throws Exception {
      new SpringApplication(MagicMirrorApp.class).run(args);
   }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /magicmirror.

MagicMirrorHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class MagicMirrorHandler extends TextWebSocketHandler {

   private final Logger log = LoggerFactory.getLogger(MagicMirrorHandler.class);
   private static final Gson gson = new GsonBuilder().create();

   private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<String, UserSession>();

   @Autowired
   private KurentoClient kurento;

   @Override
   public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
      JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

      log.debug("Incoming message: {}", jsonMessage);

      switch (jsonMessage.get("id").getAsString()) {
      case "start":
         start(session, jsonMessage);
         break;
      case "stop": {
         UserSession user = users.remove(session.getId());
         if (user != null) {
            user.release();
         }
         break;
      }
      case "onIceCandidate": {
         JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

         UserSession user = users.get(session.getId());
         if (user != null) {
            IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
                  jsonCandidate.get("sdpMid").getAsString(), jsonCandidate.get("sdpMLineIndex").getAsInt());
            user.addCandidate(candidate);
         }
         break;
      }
      default:
         sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
         break;
      }
   }

   private void start(WebSocketSession session, JsonObject jsonMessage) {
      ...
   }

   private void sendError(WebSocketSession session, String message) {
      ...
   }
}

In the following snippet, we can see the start method. It handles the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint and FaceOverlayFilter) and make the connections among them. A startResponse message is sent back to the client with the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // User session
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      users.put(session.getId(), user);

      // ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
         @Override
         public void onEvent(IceCandidateFoundEvent event) {
            JsonObject response = new JsonObject();
            response.addProperty("id", "iceCandidate");
            response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
            try {
               synchronized (session) {
                  session.sendMessage(new TextMessage(response.toString()));
               }
            } catch (IOException e) {
               log.debug(e.getMessage());
            }
         }
      });

      // Media logic
      FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build();

      String appServerUrl = System.getProperty("app.server.url", MagicMirrorApp.DEFAULT_APP_SERVER_URL);
      faceOverlayFilter.setOverlayedImage(appServerUrl + "/img/mario-wings.png", -0.35F, -1.2F, 1.6F, 1.6F);

      webRtcEndpoint.connect(faceOverlayFilter);
      faceOverlayFilter.connect(webRtcEndpoint);

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
         session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();

   } catch (Throwable t) {
      sendError(session, t.getMessage());
   }
}

Note

Notice the hat URL is provided by the application server and consumed by the KMS. This logic is assuming that the application server is hosted in local (localhost), and by the default the hat URL is https://localhost:8443/img/mario-wings.png. If your application server is hosted in a different host, it can be easily changed by means of the configuration parameter app.server.url, for example:

mvn -U clean spring-boot:run -Dapp.server.url=https://app_server_host:app_server_port

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /magicmirror. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('ws://' + location.host + '/magicmirror');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError("Error message from server: " + parsedMessage.message);
      break;
   case 'iceCandidate':
       webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
         if (error) {
            console.error("Error adding candidate: " + error);
            return;
         }
       });
       break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log("Starting video call ...")
   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   console.log("Creating WebRtcPeer and generating local sdp offer ...");

    var options = {
         localVideo: videoInput,
         remoteVideo: videoOutput,
         onicecandidate: onIceCandidate
       }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
      function (error) {
        if (error) {
           return console.error(error);
        }
        webRtcPeer.generateOffer(onOffer);
      });
}

function onOffer(offerSdp) {
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

function onIceCandidate(candidate) {
     console.log("Local candidate" + JSON.stringify(candidate));

     var message = {
       id: 'onIceCandidate',
       candidate: candidate
     };
     sendMessage(message);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript - Magic Mirror¶

This web application extends the Hello World Tutorial, adding media processing to the basic WebRTC loopback.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

You’ll need to install Kurento Media Server before running this example. Read installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-magic-mirror
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. Kurento Media Server must use WebSockets over SSL/TLS (WSS), so make sure you check this too. It is possible to locate the KMS in other machine simple adding the parameter ws_uri to the URL:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

This application uses computer vision and augmented reality techniques to add a funny hat on top of detected faces. The following picture shows a screenshot of the demo running in a web browser:

Kurento Magic Mirror Screenshot: WebRTC with filter in loopback

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to the Kurento Media Server, processed and then is returned to the client as a remote stream.

To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRtcEndpoint: Provides full-duplex (bidirectional) WebRTC capabilities.
FaceOverlay filter: Computer vision filter that detects faces in the video stream and puts an image on top of them. In this demo the filter is configured to put a Super Mario hat).

The media pipeline implemented is illustrated in the following picture:

WebRTC with filter in loopback Media Pipeline

The complete source code of this demo can be found in GitHub.

JavaScript Logic¶

This demo follows a Single Page Application architecture (SPA). The interface is the following HTML page: index.html. This web page links two Kurento JavaScript libraries:

kurento-client.js : Implementation of the Kurento JavaScript Client.
kurento-utils.js : Kurento utility library aimed to simplify the WebRTC management in the browser.

In addition, these two JavaScript libraries are also required:

Bootstrap : Web framework for developing responsive web sites.
jquery.js : Cross-platform JavaScript library designed to simplify the client-side scripting of HTML.
adapter.js : WebRTC JavaScript utility library maintained by Google that abstracts away browser differences.
ekko-lightbox : Module for Bootstrap to open modal images, videos, and galleries.
demo-console : Custom JavaScript console.

The specific logic of this demo is coded in the following JavaScript page: index.js. In this file, there is a function which is called when the green button labeled as Start in the GUI is clicked.

var startButton = document.getElementById("start");

startButton.addEventListener("click", function() {
   var options = {
     localVideo: videoInput,
     remoteVideo: videoOutput
   };

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)
      this.generateOffer(onOffer)
   });

   [...]
}

The function WebRtcPeer.WebRtcPeerSendrecv abstracts the WebRTC internal details (i.e. PeerConnection and getUserStream) and makes possible to start a full-duplex WebRTC communication, using the HTML video tag with id videoInput to show the video camera (local stream) and the video tag videoOutput to show the remote stream provided by the Kurento Media Server.

Inside this function, a call to generateOffer is performed. This function accepts a callback in which the SDP offer is received. In this callback we create an instance of the KurentoClient class that will manage communications with the Kurento Media Server. So, we need to provide the URI of its WebSocket endpoint. In this example, we assume it’s listening in port 8888 at the same host than the HTTP serving the application.

[...]

var args = getopts(location.search,
{
  default:
  {
    ws_uri: 'ws://' + location.hostname + ':8888/kurento',
    ice_servers: undefined
  }
});

[...]

kurentoClient(args.ws_uri, function(error, client){
  [...]
};

Once we have an instance of kurentoClient, the following step is to create a Media Pipeline, as follows:

client.create("MediaPipeline", function(error, _pipeline){
   [...]
});

If everything works correctly, we have an instance of a media pipeline (variable pipeline in this example). With this instance, we are able to create Media Elements. In this example we just need a WebRtcEndpoint and a FaceOverlayFilter. Then, these media elements are interconnected:

pipeline.create('WebRtcEndpoint', function(error, webRtcEp) {
  if (error) return onError(error);

  setIceCandidateCallbacks(webRtcPeer, webRtcEp, onError)

  webRtcEp.processOffer(sdpOffer, function(error, sdpAnswer) {
    if (error) return onError(error);

    webRtcPeer.processAnswer(sdpAnswer, onError);
  });
  webRtcEp.gatherCandidates(onError);

  pipeline.create('FaceOverlayFilter', function(error, filter) {
    if (error) return onError(error);

    filter.setOverlayedImage(args.hat_uri, -0.35, -1.2, 1.6, 1.6,
    function(error) {
      if (error) return onError(error);

    });

    client.connect(webRtcEp, filter, webRtcEp, function(error) {
      if (error) return onError(error);

      console.log("WebRtcEndpoint --> filter --> WebRtcEndpoint");
    });
  });
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

The dependencies of this demo has to be obtained using Bower. The definition of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at Bower.

Node.js - WebRTC magic mirror¶

This web application extends the Hello World Tutorial, adding media processing to the basic WebRTC loopback.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-magic-mirror
git checkout 6.10.0
npm install
npm start

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

This application uses computer vision and augmented reality techniques to add a funny hat on top of faces. The following picture shows a screenshot of the demo running in a web browser:

Kurento Magic Mirror Screenshot: WebRTC with filter in loopback

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRtcEndpoint: Provides full-duplex (bidirectional) WebRTC capabilities.
FaceOverlay filter: Computer vision filter that detects faces in the video stream and puts an image on top of them. In this demo the filter is configured to put a Super Mario hat).

WebRTC with filter in loopback Media Pipeline

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side we use a Node.js application server consuming the Kurento JavaScript Client API to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento JavaScript Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

To communicate the client with the Node.js application server we have designed a simple signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server is as follows: i) Client starts the Magic Mirror. ii) Client stops the Magic Mirror.

If any exception happens, server sends an error message to the client. The detailed message sequence between client and application server is depicted in the following picture:

One to one video call signaling protocol

As you can see in the diagram, an SDP and ICE candidates needs to be exchanged between client and server to establish the WebRTC session between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer at the browser with the WebRtcEndpoint at the server. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

This demo has been developed using the express framework for Node.js, but express is not a requirement for Kurento. The main script of this demo is server.js.

In order to communicate the JavaScript client and the Node application server a WebSocket is used. The incoming messages to this WebSocket (variable ws in the code) are conveniently handled to implemented the signaling protocol depicted in the figure before (i.e. messages start, stop, onIceCandidate).

var ws = require('ws');

[...]

var wss = new ws.Server({
    server : server,
    path : '/magicmirror'
});

/*
 * Management of WebSocket messages
 */
wss.on('connection', function(ws) {
    var sessionId = null;
    var request = ws.upgradeReq;
    var response = {
        writeHead : {}
    };

    sessionHandler(request, response, function(err) {
        sessionId = request.session.id;
        console.log('Connection received with sessionId ' + sessionId);
    });

    ws.on('error', function(error) {
        console.log('Connection ' + sessionId + ' error');
        stop(sessionId);
    });

    ws.on('close', function() {
        console.log('Connection ' + sessionId + ' closed');
        stop(sessionId);
    });

    ws.on('message', function(_message) {
        var message = JSON.parse(_message);
        console.log('Connection ' + sessionId + ' received message ', message);

        switch (message.id) {
        case 'start':
            sessionId = request.session.id;
            start(sessionId, ws, message.sdpOffer, function(error, sdpAnswer) {
                if (error) {
                    return ws.send(JSON.stringify({
                        id : 'error',
                        message : error
                    }));
                }
                ws.send(JSON.stringify({
                    id : 'startResponse',
                    sdpAnswer : sdpAnswer
                }));
            });
            break;

        case 'stop':
            stop(sessionId);
            break;

        case 'onIceCandidate':
            onIceCandidate(sessionId, message.candidate);
            break;

        default:
            ws.send(JSON.stringify({
                id : 'error',
                message : 'Invalid message ' + message
            }));
            break;
        }

    });
});

In order to control the media capabilities provided by the Kurento Media Server, we need an instance of the KurentoClient in the Node application server. In order to create this instance, we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888.

var kurento = require('kurento-client');

var kurentoClient = null;

var argv = minimist(process.argv.slice(2), {
    default: {
        as_uri: 'https://localhost:8443/',
        ws_uri: 'ws://localhost:8888/kurento'
    }
});

[...]

function getKurentoClient(callback) {
    if (kurentoClient !== null) {
        return callback(null, kurentoClient);
    }

    kurento(argv.ws_uri, function(error, _kurentoClient) {
        if (error) {
            console.log("Could not find media server at address " + argv.ws_uri);
            return callback("Could not find media server at address" + argv.ws_uri
                    + ". Exiting with error " + error);
        }

        kurentoClient = _kurentoClient;
        callback(null, kurentoClient);
    });
}

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server. Our first operation is to create a Media Pipeline, then we need to create the Media Elements and connect them. In this example, we need a WebRtcEndpoint connected to a FaceOverlayFilter, which is connected to the sink of the same WebRtcEndpoint. These functions are called in the start function, which is fired when the start message is received:

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, faceOverlayFilter, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint
                        }
                        return callback(null, sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        return callback(null, webRtcEndpoint);
    });
}

function connectMediaElements(webRtcEndpoint, faceOverlayFilter, callback) {
    webRtcEndpoint.connect(faceOverlayFilter, function(error) {
        if (error) {
            return callback(error);
        }

        faceOverlayFilter.connect(webRtcEndpoint, function(error) {
            if (error) {
                return callback(error);
            }

            return callback(null);
        });
    });
}

As of Kurento Media Server 6.0, the WebRTC negotiation is done by exchanging ICE candidates between the WebRTC peers. To implement this protocol, the webRtcEndpoint receives candidates from the client in OnIceCandidate function. These candidates are stored in a queue when the webRtcEndpoint is not available yet. Then these candidates are added to the media element by calling to the addIceCandidate method.

var candidatesQueue = {};

[...]

function onIceCandidate(sessionId, _candidate) {
    var candidate = kurento.getComplexType('IceCandidate')(_candidate);

    if (sessions[sessionId]) {
        console.info('Sending candidate');
        var webRtcEndpoint = sessions[sessionId].webRtcEndpoint;
        webRtcEndpoint.addIceCandidate(candidate);
    }
    else {
        console.info('Queueing candidate');
        if (!candidatesQueue[sessionId]) {
            candidatesQueue[sessionId] = [];
        }
        candidatesQueue[sessionId].push(candidate);
    }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application. These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /magicmirror. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication.

var ws = new WebSocket('ws://' + location.host + '/magicmirror');
var webRtcPeer;

const I_CAN_START = 0;
const I_CAN_STOP = 1;
const I_AM_STARTING = 2;

[...]

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate)
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

In the function start the method WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to create the webRtcPeer object, which is used to handle the WebRTC communication.

videoInput = document.getElementById('videoInput');
videoOutput = document.getElementById('videoOutput');

[...]

function start() {
   console.log('Starting video call ...')

   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   console.log('Creating WebRtcPeer and generating local sdp offer ...');

    var options = {
      localVideo: videoInput,
      remoteVideo: videoOutput,
      onicecandidate : onIceCandidate
    }

    webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
        if(error) return onError(error);
        this.generateOffer(onOffer);
    });
}

function onIceCandidate(candidate) {
      console.log('Local candidate' + JSON.stringify(candidate));

      var message = {
         id : 'onIceCandidate',
         candidate : candidate
      };
      sendMessage(message);
}

function onOffer(error, offerSdp) {
   if(error) return onError(error);

   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

Dependencies¶

Server-side dependencies of this demo are managed using npm. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   [...]
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   [...]
   "kurento-utils" : "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at npm and Bower.

RTP Receiver¶

This web application showcases reception of an incoming RTP or SRTP stream, and playback via a WebRTC connection.

Kurento Java Tutorial - RTP Receiver¶

This web application consists of a simple RTP stream pipeline: an RtpEndpoint is configured in KMS to listen for one incoming video stream. This stream must be generated by an external program. Visual feedback is provided in this page, by connecting the RtpEndpoint to a WebRtcEndpoint in receive-only mode.

The Java Application Server connects to all events emitted from KMS and prints log messages for each one, so this application is also a good reference to understand what are those events and their relationship with how KMS works. Check Endpoint Events for more information about events that can be emitted by KMS.

Note

This application uses HTTPS. It will work fine is you run it in localhost and accept a security exception in the browser, but you should secure your application if running remotely. For more info, check Configure a Java server to use HTTPS.

Quick start¶

Follow these steps to run this demo application:

Install and run Kurento Media Server: Installation Guide.

Install Java JDK and Maven:

sudo apt-get update
sudo apt-get install default-jdk maven

Run these commands:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-rtp-receiver/
git checkout 6.10.0
mvn -U clean spring-boot:run -Dkms.url=ws://localhost:8888/kurento

Open the demo page with a WebRTC-compliant browser (Chrome, Firefox): https://localhost:8443/
Click on Start to begin the demo.
Copy the KMS IP and Port values to the external streaming program.
As soon as the external streaming program starts sending RTP packets to the IP and Port where KMS is listening for incoming data, the video should appear in the page.
Click on Stop to finish the demo.

Understanding this example¶

To implement this behavior we have to create a Media Pipeline, composed of an RtpEndpoint and a WebRtcEndpoint. The former acts as an RTP receiver, and the later is used to show the video in the demo page.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier.

To communicate these entities, two WebSockets channels are used:

A WebSocket is created between the Application Server and the browser client, to implement a custom signaling protocol.
Another WebSocket is used to perform the communication between the Application Server and the Kurento Media Server. For this, the Application Server uses the Kurento Java Client library. This communication takes place using the Kurento Protocol (see Kurento Protocol).

The complete source code for this tutorial can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server side, based on the Spring Boot framework, which embeds a Tomcat web server within the resulting program, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

This graph shows the class diagram of the Application Server:

Server-side class diagram of the Application Server¶

Client-Side Logic¶

We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction between browser and application server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences.

These libraries are linked in the index.html page, and are used in the index.js file.

WebRTC One-To-Many broadcast¶

Video broadcasting for WebRTC. One peer transmits a video stream and N peers receive it.

Java - One to many video call¶

This web application consists on a one-to-many video call using WebRTC technology. In other words, it is an implementation of a video broadcasting web application.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-one2many-call
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

There will be two types of users in this application: 1 peer sending media (let’s call it Presenter) and N peers receiving the media from the Presenter (let’s call them Viewers). Thus, the Media Pipeline is composed by 1+N interconnected WebRtcEndpoints. The following picture shows an screenshot of the Presenter’s web GUI:

One to many video call screenshot

To implement this behavior we have to create a Media Pipeline composed by 1+N WebRtcEndpoints. The Presenter peer sends its stream to the rest of the Viewers. Viewers are configured in receive-only mode. The implemented media pipeline is illustrated in the following picture:

One to many video call Media Pipeline

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities two WebSockets are used. First, a WebSocket is created between client and server-side to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication is implemented by the Kurento Protocol. For further information, please see this page.

Client and application server communicate using a signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server is as follows:

1. A Presenter enters in the system. There must be one and only one Presenter at any time. For that, if a Presenter has already present, an error message is sent if another user tries to become Presenter.

2. N Viewers connect to the presenter. If no Presenter is present, then an error is sent to the corresponding Viewer.

Viewers can leave the communication at any time.

4. When the Presenter finishes the session each connected Viewer receives an stopCommunication message and also terminates its session.

We can draw the following sequence diagram with detailed messages between clients and server:

One to many video call signaling protocol

As you can see in the diagram, SDP and ICE candidates need to be exchanged between client and server to establish the WebRTC connection between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer in the browser with the WebRtcEndpoint in the server. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vertex, etc. We chose Spring Boot for convenience.

In the following, figure you can see a class diagram of the server side code:

Server-side class diagram of the One2Many app

The main class of this demo is named One2ManyCallApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to your applications. In this instantiation we see that a WebSocket is used to connect with Kurento Media Server, by default in the localhost and listening in the port 8888.

@EnableWebSocket
@SpringBootApplication
public class One2ManyCallApp implements WebSocketConfigurer {

   @Bean
   public CallHandler callHandler() {
      return new CallHandler();
   }

   @Bean
   public KurentoClient kurentoClient() {
      return KurentoClient.create();
   }

   public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
      registry.addHandler(callHandler(), "/call");
   }

   public static void main(String[] args) throws Exception {
      new SpringApplication(One2ManyCallApp.class).run(args);
   }

}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /call.

CallHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kind of incoming messages to the Server : presenter, viewer, stop, and onIceCandidate. These messages are treated in the switch clause, taking the proper steps in each case.

public class CallHandler extends TextWebSocketHandler {

   private static final Logger log = LoggerFactory.getLogger(CallHandler.class);
   private static final Gson gson = new GsonBuilder().create();

   private final ConcurrentHashMap<String, UserSession> viewers = new ConcurrentHashMap<String, UserSession>();

   @Autowired
   private KurentoClient kurento;

   private MediaPipeline pipeline;
   private UserSession presenterUserSession;

   @Override
   public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
      JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);
      log.debug("Incoming message from session '{}': {}", session.getId(), jsonMessage);

      switch (jsonMessage.get("id").getAsString()) {
      case "presenter":
         try {
            presenter(session, jsonMessage);
         } catch (Throwable t) {
            handleErrorResponse(t, session, "presenterResponse");
         }
         break;
      case "viewer":
         try {
            viewer(session, jsonMessage);
         } catch (Throwable t) {
            handleErrorResponse(t, session, "viewerResponse");
         }
         break;
      case "onIceCandidate": {
         JsonObject candidate = jsonMessage.get("candidate").getAsJsonObject();

         UserSession user = null;
         if (presenterUserSession != null) {
            if (presenterUserSession.getSession() == session) {
               user = presenterUserSession;
            } else {
               user = viewers.get(session.getId());
            }
         }
         if (user != null) {
            IceCandidate cand = new IceCandidate(candidate.get("candidate").getAsString(),
                  candidate.get("sdpMid").getAsString(), candidate.get("sdpMLineIndex").getAsInt());
            user.addCandidate(cand);
         }
         break;
      }
      case "stop":
         stop(session);
         break;
      default:
         break;
      }
   }

   private void handleErrorResponse(Throwable t, WebSocketSession session,
         String responseId) throws IOException {
      stop(session);
      log.error(t.getMessage(), t);
      JsonObject response = new JsonObject();
      response.addProperty("id", responseId);
      response.addProperty("response", "rejected");
      response.addProperty("message", t.getMessage());
      session.sendMessage(new TextMessage(response.toString()));
   }

   private synchronized void presenter(final WebSocketSession session, JsonObject jsonMessage) throws IOException {
      ...
   }

   private synchronized void viewer(final WebSocketSession session, JsonObject jsonMessage) throws IOException {
      ...
   }

   private synchronized void stop(WebSocketSession session) throws IOException {
      ...
   }

   @Override
   public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
      stop(session);
   }

}

In the following snippet, we can see the presenter method. It creates a Media Pipeline and the WebRtcEndpoint for presenter:

private synchronized void presenter(final WebSocketSession session, JsonObject jsonMessage) throws IOException {
   if (presenterUserSession == null) {
      presenterUserSession = new UserSession(session);

      pipeline = kurento.createMediaPipeline();
      presenterUserSession.setWebRtcEndpoint(new WebRtcEndpoint.Builder(pipeline).build());

      WebRtcEndpoint presenterWebRtc = presenterUserSession.getWebRtcEndpoint();

      presenterWebRtc.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

         @Override
         public void onEvent(IceCandidateFoundEvent event) {
            JsonObject response = new JsonObject();
            response.addProperty("id", "iceCandidate");
            response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
            try {
               synchronized (session) {
                  session.sendMessage(new TextMessage(response.toString()));
               }
            } catch (IOException e) {
               log.debug(e.getMessage());
            }
         }
      });

      String sdpOffer = jsonMessage.getAsJsonPrimitive("sdpOffer").getAsString();
      String sdpAnswer = presenterWebRtc.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "presenterResponse");
      response.addProperty("response", "accepted");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
         presenterUserSession.sendMessage(response);
      }
      presenterWebRtc.gatherCandidates();

   } else {
      JsonObject response = new JsonObject();
      response.addProperty("id", "presenterResponse");
      response.addProperty("response", "rejected");
      response.addProperty("message", "Another user is currently acting as sender. Try again later ...");
      session.sendMessage(new TextMessage(response.toString()));
   }
}

The viewer method is similar, but not he Presenter WebRtcEndpoint is connected to each of the viewers WebRtcEndpoints, otherwise an error is sent back to the client.

private synchronized void viewer(final WebSocketSession session, JsonObject jsonMessage) throws IOException {
   if (presenterUserSession == null || presenterUserSession.getWebRtcEndpoint() == null) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "viewerResponse");
      response.addProperty("response", "rejected");
      response.addProperty("message", "No active sender now. Become sender or . Try again later ...");
      session.sendMessage(new TextMessage(response.toString()));
   } else {
      if (viewers.containsKey(session.getId())) {
         JsonObject response = new JsonObject();
         response.addProperty("id", "viewerResponse");
         response.addProperty("response", "rejected");
         response.addProperty("message",
               "You are already viewing in this session. Use a different browser to add additional viewers.");
         session.sendMessage(new TextMessage(response.toString()));
         return;
      }
      UserSession viewer = new UserSession(session);
      viewers.put(session.getId(), viewer);

      String sdpOffer = jsonMessage.getAsJsonPrimitive("sdpOffer").getAsString();

      WebRtcEndpoint nextWebRtc = new WebRtcEndpoint.Builder(pipeline).build();

      nextWebRtc.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

         @Override
         public void onEvent(IceCandidateFoundEvent event) {
            JsonObject response = new JsonObject();
            response.addProperty("id", "iceCandidate");
            response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
            try {
               synchronized (session) {
                  session.sendMessage(new TextMessage(response.toString()));
               }
            } catch (IOException e) {
               log.debug(e.getMessage());
            }
         }
      });

      viewer.setWebRtcEndpoint(nextWebRtc);
      presenterUserSession.getWebRtcEndpoint().connect(nextWebRtc);
      String sdpAnswer = nextWebRtc.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "viewerResponse");
      response.addProperty("response", "accepted");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
         viewer.sendMessage(response);
      }
      nextWebRtc.gatherCandidates();
   }
}

Finally, the stop message finishes the communication. If this message is sent by the Presenter, a stopCommunication message is sent to each connected Viewer:

private synchronized void stop(WebSocketSession session) throws IOException {
   String sessionId = session.getId();
   if (presenterUserSession != null && presenterUserSession.getSession().getId().equals(sessionId)) {
      for (UserSession viewer : viewers.values()) {
         JsonObject response = new JsonObject();
         response.addProperty("id", "stopCommunication");
         viewer.sendMessage(response);
      }

      log.info("Releasing media pipeline");
      if (pipeline != null) {
         pipeline.release();
      }
      pipeline = null;
      presenterUserSession = null;
   } else if (viewers.containsKey(sessionId)) {
      if (viewers.get(sessionId).getWebRtcEndpoint() != null) {
         viewers.get(sessionId).getWebRtcEndpoint().release();
      }
      viewers.remove(sessionId);
   }
}

Client-Side¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /call. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are four incoming messages to client: presenterResponse, viewerResponse, iceCandidate, and stopCommunication. Convenient actions are taken to implement each step in the communication. For example, in the function presenter the function WebRtcPeer.WebRtcPeerSendonly of kurento-utils.js is used to start a WebRTC communication. Then, WebRtcPeer.WebRtcPeerRecvonly is used in the viewer function.

var ws = new WebSocket('ws://' + location.host + '/call');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'presenterResponse':
      presenterResponse(parsedMessage);
      break;
   case 'viewerResponse':
      viewerResponse(parsedMessage);
      break;
   case 'iceCandidate':
       webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
        if (!error) return;
         console.error("Error adding candidate: " + error);
       });
       break;
   case 'stopCommunication':
      dispose();
      break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

function presenter() {
   if (!webRtcPeer) {
      showSpinner(video);

      var options = {
               localVideo: video,
               onicecandidate: onIceCandidate
             }
      webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendonly(options,
         function (error) {
           if(error) {
              return console.error(error);
           }
           webRtcPeer.generateOffer(onOfferPresenter);
      });
   }
}

function viewer() {
   if (!webRtcPeer) {
      showSpinner(video);

      var options = {
               remoteVideo: video,
               onicecandidate: onIceCandidate
             }
      webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function (error) {
           if(error) {
              return console.error(error);
           }
          this.generateOffer(onOfferViewer);
      });
   }
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

Node.js - One to many video call¶

This web application consists on one-to-many video call using WebRTC technology. In other words, it is an implementation of a video broadcasting web application.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-one2many-call
git checkout 6.10.0
npm install
npm start

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

There will be two types of users in this application: 1 peer sending media (let’s call it Presenter) and N peers receiving the media from the Presenter (let’s call them Viewers). Thus, the Media Pipeline is composed by 1+N interconnected WebRtcEndpoints. The following picture shows an screenshot of the Presenter’s web GUI:

One to many video call screenshot

To implement this behavior we have to create a Media Pipeline composed by 1+N WebRtcEndpoints. The Presenter peer sends its stream to the rest of the Viewers. Viewers are configured in receive-only mode. The implemented media pipeline is illustrated in the following picture:

One to many video call Media Pipeline

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side we use the Kurento JavaScript Client in order to reach the Kurento Media Server. All in all, the high level architecture of this demo is three-tier. To communicate these entities two WebSockets are used. The first is created between the client browser and a Node.js application server to transport signaling messages. The second is used to communicate the Kurento JavaScript Client executing at Node.js and the Kurento Media Server. This communication is implemented by the Kurento Protocol. For further information, please see this page.

Client and application server communicate using a signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server is as follows:

1. A Presenter enters in the system. There must be one and only one Presenter at any time. For that, if a Presenter has already present, an error message is sent if another user tries to become Presenter.

2. N Viewers connect to the presenter. If no Presenter is present, then an error is sent to the corresponding Viewer.

Viewers can leave the communication at any time.

4. When the Presenter finishes the session each connected Viewer receives an stopCommunication message and also terminates its session.

We can draw the following sequence diagram with detailed messages between clients and server:

One to many video call signaling protocol

As you can see in the diagram, SDP and ICE candidates need to be exchanged between client and server to establish the WebRTC connection between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer in the browser with the WebRtcEndpoint in the server. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

This demo has been developed using the express framework for Node.js, but express is not a requirement for Kurento. The main script of this demo is server.js.

In order to communicate the JavaScript client and the Node application server a WebSocket is used. The incoming messages to this WebSocket (variable ws in the code) are conveniently handled to implemented the signaling protocol depicted in the figure before (i.e. messages presenter, viewer, stop, and onIceCandidate).

var ws = require('ws');

[...]

var wss = new ws.Server({
    server : server,
    path : '/one2many'
});

/*
 * Management of WebSocket messages
 */
wss.on('connection', function(ws) {

   var sessionId = nextUniqueId();
   console.log('Connection received with sessionId ' + sessionId);

    ws.on('error', function(error) {
        console.log('Connection ' + sessionId + ' error');
        stop(sessionId);
    });

    ws.on('close', function() {
        console.log('Connection ' + sessionId + ' closed');
        stop(sessionId);
    });

    ws.on('message', function(_message) {
        var message = JSON.parse(_message);
        console.log('Connection ' + sessionId + ' received message ', message);

        switch (message.id) {
        case 'presenter':
         startPresenter(sessionId, ws, message.sdpOffer, function(error, sdpAnswer) {
            if (error) {
               return ws.send(JSON.stringify({
                  id : 'presenterResponse',
                  response : 'rejected',
                  message : error
               }));
            }
            ws.send(JSON.stringify({
               id : 'presenterResponse',
               response : 'accepted',
               sdpAnswer : sdpAnswer
            }));
         });
         break;

        case 'viewer':
         startViewer(sessionId, ws, message.sdpOffer, function(error, sdpAnswer) {
            if (error) {
               return ws.send(JSON.stringify({
                  id : 'viewerResponse',
                  response : 'rejected',
                  message : error
               }));
            }

            ws.send(JSON.stringify({
               id : 'viewerResponse',
               response : 'accepted',
               sdpAnswer : sdpAnswer
            }));
         });
         break;

        case 'stop':
            stop(sessionId);
            break;

        case 'onIceCandidate':
            onIceCandidate(sessionId, message.candidate);
            break;

        default:
            ws.send(JSON.stringify({
                id : 'error',
                message : 'Invalid message ' + message
            }));
            break;
        }
    });
});

In order to control the media capabilities provided by the Kurento Media Server, we need an instance of the KurentoClient in the Node application server. In order to create this instance, we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888.

var kurento = require('kurento-client');

var kurentoClient = null;

var argv = minimist(process.argv.slice(2), {
    default: {
        as_uri: 'https://localhost:8443/',
        ws_uri: 'ws://localhost:8888/kurento'
    }
});

[...]

function getKurentoClient(callback) {
    if (kurentoClient !== null) {
        return callback(null, kurentoClient);
    }

    kurento(argv.ws_uri, function(error, _kurentoClient) {
        if (error) {
            console.log("Could not find media server at address " + argv.ws_uri);
            return callback("Could not find media server at address" + argv.ws_uri
                    + ". Exiting with error " + error);
        }

        kurentoClient = _kurentoClient;
        callback(null, kurentoClient);
    });
}

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server. Our first operation is to create a Media Pipeline, then we need to create the Media Elements and connect them. In this example, we need a WebRtcEndpoint (in send-only mode) for the presenter connected to N WebRtcEndpoint (in receive-only mode) for the viewers. These functions are called in the startPresenter and startViewer function, which is fired when the presenter and viewer message are received respectively:

function startPresenter(sessionId, ws, sdpOffer, callback) {
   clearCandidatesQueue(sessionId);

   if (presenter !== null) {
      stop(sessionId);
      return callback("Another user is currently acting as presenter. Try again later ...");
   }

   presenter = {
      id : sessionId,
      pipeline : null,
      webRtcEndpoint : null
   }

   getKurentoClient(function(error, kurentoClient) {
      if (error) {
         stop(sessionId);
         return callback(error);
      }

      if (presenter === null) {
         stop(sessionId);
         return callback(noPresenterMessage);
      }

      kurentoClient.create('MediaPipeline', function(error, pipeline) {
         if (error) {
            stop(sessionId);
            return callback(error);
         }

         if (presenter === null) {
            stop(sessionId);
            return callback(noPresenterMessage);
         }

         presenter.pipeline = pipeline;
         pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
            if (error) {
               stop(sessionId);
               return callback(error);
            }

            if (presenter === null) {
               stop(sessionId);
               return callback(noPresenterMessage);
            }

            presenter.webRtcEndpoint = webRtcEndpoint;

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                webRtcEndpoint.on('OnIceCandidate', function(event) {
                    var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                    ws.send(JSON.stringify({
                        id : 'iceCandidate',
                        candidate : candidate
                    }));
                });

            webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
               if (error) {
                  stop(sessionId);
                  return callback(error);
               }

               if (presenter === null) {
                  stop(sessionId);
                  return callback(noPresenterMessage);
               }

               callback(null, sdpAnswer);
            });

                webRtcEndpoint.gatherCandidates(function(error) {
                    if (error) {
                        stop(sessionId);
                        return callback(error);
                    }
                });
            });
        });
   });
}

function startViewer(sessionId, ws, sdpOffer, callback) {
   clearCandidatesQueue(sessionId);

   if (presenter === null) {
      stop(sessionId);
      return callback(noPresenterMessage);
   }

   presenter.pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
      if (error) {
         stop(sessionId);
         return callback(error);
      }
      viewers[sessionId] = {
         "webRtcEndpoint" : webRtcEndpoint,
         "ws" : ws
      }

      if (presenter === null) {
         stop(sessionId);
         return callback(noPresenterMessage);
      }

      if (candidatesQueue[sessionId]) {
         while(candidatesQueue[sessionId].length) {
            var candidate = candidatesQueue[sessionId].shift();
            webRtcEndpoint.addIceCandidate(candidate);
         }
      }

      webRtcEndpoint.on('OnIceCandidate', function(event) {
          var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
          ws.send(JSON.stringify({
              id : 'iceCandidate',
              candidate : candidate
          }));
      });

      webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
         if (error) {
            stop(sessionId);
            return callback(error);
         }
         if (presenter === null) {
            stop(sessionId);
            return callback(noPresenterMessage);
         }

         presenter.webRtcEndpoint.connect(webRtcEndpoint, function(error) {
            if (error) {
               stop(sessionId);
               return callback(error);
            }
            if (presenter === null) {
               stop(sessionId);
               return callback(noPresenterMessage);
            }

            callback(null, sdpAnswer);
              webRtcEndpoint.gatherCandidates(function(error) {
                  if (error) {
                     stop(sessionId);
                     return callback(error);
                  }
              });
          });
       });
   });
}

As of Kurento Media Server 6.0, the WebRTC negotiation is done by exchanging ICE candidates between the WebRTC peers. To implement this protocol, the webRtcEndpoint receives candidates from the client in OnIceCandidate function. These candidates are stored in a queue when the webRtcEndpoint is not available yet. Then these candidates are added to the media element by calling to the addIceCandidate method.

var candidatesQueue = {};

[...]

function onIceCandidate(sessionId, _candidate) {
    var candidate = kurento.getComplexType('IceCandidate')(_candidate);

    if (presenter && presenter.id === sessionId && presenter.webRtcEndpoint) {
        console.info('Sending presenter candidate');
        presenter.webRtcEndpoint.addIceCandidate(candidate);
    }
    else if (viewers[sessionId] && viewers[sessionId].webRtcEndpoint) {
        console.info('Sending viewer candidate');
        viewers[sessionId].webRtcEndpoint.addIceCandidate(candidate);
    }
    else {
        console.info('Queueing candidate');
        if (!candidatesQueue[sessionId]) {
            candidatesQueue[sessionId] = [];
        }
        candidatesQueue[sessionId].push(candidate);
    }
}

function clearCandidatesQueue(sessionId) {
   if (candidatesQueue[sessionId]) {
      delete candidatesQueue[sessionId];
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application. These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /one2many. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: presenterResponse, viewerResponse,``stopCommunication``, and iceCandidate. Convenient actions are taken to implement each step in the communication.

On the one hand, the function presenter uses the method WebRtcPeer.WebRtcPeerSendonly of kurento-utils.js to start a WebRTC communication in send-only mode. On the other hand, the function viewer uses the method WebRtcPeer.WebRtcPeerRecvonly of kurento-utils.js to start a WebRTC communication in receive-only mode.

var ws = new WebSocket('ws://' + location.host + '/one2many');
var webRtcPeer;

const I_CAN_START = 0;
const I_CAN_STOP = 1;
const I_AM_STARTING = 2;

[...]

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'presenterResponse':
      presenterResponse(parsedMessage);
      break;
   case 'viewerResponse':
      viewerResponse(parsedMessage);
      break;
   case 'stopCommunication':
      dispose();
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate)
      break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

function presenterResponse(message) {
   if (message.response != 'accepted') {
      var errorMsg = message.message ? message.message : 'Unknow error';
      console.warn('Call not accepted for the following reason: ' + errorMsg);
      dispose();
   } else {
      webRtcPeer.processAnswer(message.sdpAnswer);
   }
}

function viewerResponse(message) {
   if (message.response != 'accepted') {
      var errorMsg = message.message ? message.message : 'Unknow error';
      console.warn('Call not accepted for the following reason: ' + errorMsg);
      dispose();
   } else {
      webRtcPeer.processAnswer(message.sdpAnswer);
   }
}

On the one hand, the function presenter uses the method WebRtcPeer.WebRtcPeerSendonly of kurento-utils.js to start a WebRTC communication in send-only mode. On the other hand, the function viewer uses the method WebRtcPeer.WebRtcPeerRecvonly of kurento-utils.js to start a WebRTC communication in receive-only mode.

function presenter() {
   if (!webRtcPeer) {
      showSpinner(video);

      var options = {
         localVideo: video,
         onicecandidate : onIceCandidate
       }

      webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendonly(options, function(error) {
         if(error) return onError(error);

         this.generateOffer(onOfferPresenter);
      });
   }
}

function onOfferPresenter(error, offerSdp) {
   if (error) return onError(error);

   var message = {
      id : 'presenter',
      sdpOffer : offerSdp
   };
   sendMessage(message);
}

function viewer() {
   if (!webRtcPeer) {
      showSpinner(video);

      var options = {
         remoteVideo: video,
         onicecandidate : onIceCandidate
      }

      webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options, function(error) {
         if(error) return onError(error);

         this.generateOffer(onOfferViewer);
      });
   }
}

function onOfferViewer(error, offerSdp) {
   if (error) return onError(error)

   var message = {
      id : 'viewer',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

Dependencies¶

Server-side dependencies of this demo are managed using npm. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   [...]
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   [...]
   "kurento-utils" : "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at npm and Bower.

WebRTC One-To-One video call¶

This web application is a videophone (call one to one) based on WebRTC.

Java - One to one video call¶

This web application consists on a one-to-one video call using WebRTC technology. In other words, this application provides a simple video softphone.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-one2one-call
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

The following picture shows an screenshot of this demo running in a web browser:

One to one video call screenshot

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the local stream and other for the remote peer stream). If two users, A and B, are using the application, the media flow goes this way: The video camera stream of user A is sent to the Kurento Media Server, which sends it to user B. In the same way, B sends to Kurento Media Server, which forwards it to A. This means that KMS is providing a B2B (back-to-back) call service.

To implement this behavior, create sa Media Pipeline composed by two WebRtC endpoints connected in B2B. The implemented media pipeline is illustrated in the following picture:

One to one video call Media Pipeline

The client and the server communicate through a signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server would be as follows:

User A is registered in the server with his name
User B is registered in the server with her name
User A wants to call to User B
User B accepts the incoming call
The communication is established and media is flowing between User A and User B
One of the users finishes the video communication

The detailed message flow in a call are shown in the picture below:

One to many one call signaling protocol

As you can see in the diagram, SDP and ICE candidates need to be interchanged between client and server to establish the WebRTC connection between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer in the browser with the WebRtcEndpoint in the server.

The following sections describe in detail the server-side, the client-side, and how to run the demo. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vertex, etc. We have choose Spring Boot for convenience.

In the following figure you can see a class diagram of the server side code:

Server-side class diagram of the one to one video call app

The main class of this demo is named One2OneCallApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean.

@EnableWebSocket
@SpringBootApplication
public class One2OneCallApp implements WebSocketConfigurer {

   @Bean
   public CallHandler callHandler() {
      return new CallHandler();
   }

   @Bean
   public UserRegistry registry() {
      return new UserRegistry();
   }

   @Bean
   public KurentoClient kurentoClient() {
      return KurentoClient.create();
   }

   public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
      registry.addHandler(callHandler(), "/call");
   }

   public static void main(String[] args) throws Exception {
      new SpringApplication(One2OneCallApp.class).run(args);
   }

}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /call.

CallHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are five different kind of incoming messages to the application server: register, call, incomingCallResponse, onIceCandidate and stop. These messages are treated in the switch clause, taking the proper steps in each case.

public class CallHandler extends TextWebSocketHandler {

   private static final Logger log = LoggerFactory.getLogger(CallHandler.class);
   private static final Gson gson = new GsonBuilder().create();

   private final ConcurrentHashMap<String, CallMediaPipeline> pipelines = new ConcurrentHashMap<String, CallMediaPipeline>();

   @Autowired
   private KurentoClient kurento;

   @Autowired
   private UserRegistry registry;

   @Override
   public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
      JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);
      UserSession user = registry.getBySession(session);

      if (user != null) {
         log.debug("Incoming message from user '{}': {}", user.getName(), jsonMessage);
      } else {
         log.debug("Incoming message from new user: {}", jsonMessage);
      }

      switch (jsonMessage.get("id").getAsString()) {
      case "register":
         try {
            register(session, jsonMessage);
         } catch (Throwable t) {
            handleErrorResponse(t, session, "registerResponse");
         }
         break;
      case "call":
         try {
            call(user, jsonMessage);
         } catch (Throwable t) {
            handleErrorResponse(t, session, "callResponse");
         }
         break;
      case "incomingCallResponse":
         incomingCallResponse(user, jsonMessage);
         break;
      case "onIceCandidate": {
         JsonObject candidate = jsonMessage.get("candidate").getAsJsonObject();
         if (user != null) {
            IceCandidate cand = new IceCandidate(candidate.get("candidate").getAsString(),
                  candidate.get("sdpMid").getAsString(), candidate.get("sdpMLineIndex").getAsInt());
            user.addCandidate(cand);
         }
         break;
      }
      case "stop":
         stop(session);
         break;
      default:
         break;
      }
   }

   private void handleErrorResponse(Throwable t, WebSocketSession session,
         String responseId) throws IOException {
      stop(session);
      log.error(t.getMessage(), t);
      JsonObject response = new JsonObject();
      response.addProperty("id", responseId);
      response.addProperty("response", "rejected");
      response.addProperty("message", t.getMessage());
      session.sendMessage(new TextMessage(response.toString()));
   }

   private void register(WebSocketSession session, JsonObject jsonMessage) throws IOException {
      ...
   }

   private void call(UserSession caller, JsonObject jsonMessage) throws IOException {
      ...
   }

   private void incomingCallResponse(final UserSession callee, JsonObject jsonMessage) throws IOException {
      ...
   }

   public void stop(WebSocketSession session) throws IOException {
      ...
   }

   @Override
   public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
      stop(session);
      registry.removeBySession(session);
   }

}

In the following snippet, we can see the register method. Basically, it obtains the name attribute from register message and check if there are a registered user with that name. If not, the new user is registered and an acceptance message is sent to it.

private void register(WebSocketSession session, JsonObject jsonMessage) throws IOException {
   String name = jsonMessage.getAsJsonPrimitive("name").getAsString();

   UserSession caller = new UserSession(session, name);
   String responseMsg = "accepted";
   if (name.isEmpty()) {
      responseMsg = "rejected: empty user name";
   } else if (registry.exists(name)) {
      responseMsg = "rejected: user '" + name + "' already registered";
   } else {
      registry.register(caller);
   }

   JsonObject response = new JsonObject();
   response.addProperty("id", "resgisterResponse");
   response.addProperty("response", responseMsg);
   caller.sendMessage(response);
}

In the call method, the server checks if there is a registered user with the name specified in to message attribute, and sends an incomingCall message. If there is no user with that name, a callResponse message is sent to caller rejecting the call.

private void call(UserSession caller, JsonObject jsonMessage) throws IOException {
   String to = jsonMessage.get("to").getAsString();
   String from = jsonMessage.get("from").getAsString();
   JsonObject response = new JsonObject();

   if (registry.exists(to)) {
      UserSession callee = registry.getByName(to);
      caller.setSdpOffer(jsonMessage.getAsJsonPrimitive("sdpOffer").getAsString());
      caller.setCallingTo(to);

      response.addProperty("id", "incomingCall");
      response.addProperty("from", from);

      callee.sendMessage(response);
      callee.setCallingFrom(from);
   } else {
      response.addProperty("id", "callResponse");
      response.addProperty("response", "rejected: user '" + to + "' is not registered");

      caller.sendMessage(response);
   }
}

The stop method ends the video call. It can be called both by caller and callee in the communication. The result is that both peers release the Media Pipeline and ends the video communication:

public void stop(WebSocketSession session) throws IOException {
   String sessionId = session.getId();
   if (pipelines.containsKey(sessionId)) {
      pipelines.get(sessionId).release();
      CallMediaPipeline pipeline = pipelines.remove(sessionId);
      pipeline.release();

      // Both users can stop the communication. A 'stopCommunication'
      // message will be sent to the other peer.
      UserSession stopperUser = registry.getBySession(session);
      if (stopperUser != null) {
         UserSession stoppedUser = (stopperUser.getCallingFrom() != null)
               ? registry.getByName(stopperUser.getCallingFrom())
               : stopperUser.getCallingTo() != null
                     ? registry.getByName(stopperUser.getCallingTo())
                     : null;

         if (stoppedUser != null) {
            JsonObject message = new JsonObject();
            message.addProperty("id", "stopCommunication");
            stoppedUser.sendMessage(message);
            stoppedUser.clear();
         }
         stopperUser.clear();
      }

   }
}

In the incomingCallResponse method, if the callee user accepts the call, it is established and the media elements are created to connect the caller with the callee in a B2B manner. Basically, the server creates a CallMediaPipeline object, to encapsulate the media pipeline creation and management. Then, this object is used to negotiate media interchange with user’s browsers.

The negotiation between WebRTC peer in the browser and WebRtcEndpoint in Kurento Media Server is made by means of SDP generation at the client (offer) and SDP generation at the server (answer). The SDP answers are generated with the Kurento Java Client inside the class CallMediaPipeline (as we see in a moment). The methods used to generate SDP are generateSdpAnswerForCallee(calleeSdpOffer) and generateSdpAnswerForCaller(callerSdpOffer):

private void incomingCallResponse(final UserSession callee, JsonObject jsonMessage) throws IOException {
   String callResponse = jsonMessage.get("callResponse").getAsString();
   String from = jsonMessage.get("from").getAsString();
   final UserSession calleer = registry.getByName(from);
   String to = calleer.getCallingTo();

   if ("accept".equals(callResponse)) {
      log.debug("Accepted call from '{}' to '{}'", from, to);

      CallMediaPipeline pipeline = null;
      try {
         pipeline = new CallMediaPipeline(kurento);
         pipelines.put(calleer.getSessionId(), pipeline);
         pipelines.put(callee.getSessionId(), pipeline);

         String calleeSdpOffer = jsonMessage.get("sdpOffer").getAsString();
         callee.setWebRtcEndpoint(pipeline.getCalleeWebRtcEP());
         pipeline.getCalleeWebRtcEP().addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
            @Override
            public void onEvent(IceCandidateFoundEvent event) {
               JsonObject response = new JsonObject();
               response.addProperty("id", "iceCandidate");
               response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
               try {
                  synchronized (callee.getSession()) {
                     callee.getSession().sendMessage(new TextMessage(response.toString()));
                  }
               } catch (IOException e) {
                  log.debug(e.getMessage());
               }
            }
         });

         String calleeSdpAnswer = pipeline.generateSdpAnswerForCallee(calleeSdpOffer);
         String callerSdpOffer = registry.getByName(from).getSdpOffer();
         calleer.setWebRtcEndpoint(pipeline.getCallerWebRtcEP());
         pipeline.getCallerWebRtcEP().addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

            @Override
            public void onEvent(IceCandidateFoundEvent event) {
               JsonObject response = new JsonObject();
               response.addProperty("id", "iceCandidate");
               response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
               try {
                  synchronized (calleer.getSession()) {
                     calleer.getSession().sendMessage(new TextMessage(response.toString()));
                  }
               } catch (IOException e) {
                  log.debug(e.getMessage());
               }
            }
         });

         String callerSdpAnswer = pipeline.generateSdpAnswerForCaller(callerSdpOffer);

         JsonObject startCommunication = new JsonObject();
         startCommunication.addProperty("id", "startCommunication");
         startCommunication.addProperty("sdpAnswer", calleeSdpAnswer);

         synchronized (callee) {
            callee.sendMessage(startCommunication);
         }

         pipeline.getCalleeWebRtcEP().gatherCandidates();

         JsonObject response = new JsonObject();
         response.addProperty("id", "callResponse");
         response.addProperty("response", "accepted");
         response.addProperty("sdpAnswer", callerSdpAnswer);

         synchronized (calleer) {
            calleer.sendMessage(response);
         }

         pipeline.getCallerWebRtcEP().gatherCandidates();

      } catch (Throwable t) {
         log.error(t.getMessage(), t);

         if (pipeline != null) {
            pipeline.release();
         }

         pipelines.remove(calleer.getSessionId());
         pipelines.remove(callee.getSessionId());

         JsonObject response = new JsonObject();
         response.addProperty("id", "callResponse");
         response.addProperty("response", "rejected");
         calleer.sendMessage(response);

         response = new JsonObject();
         response.addProperty("id", "stopCommunication");
         callee.sendMessage(response);
      }

   } else {
      JsonObject response = new JsonObject();
      response.addProperty("id", "callResponse");
      response.addProperty("response", "rejected");
      calleer.sendMessage(response);
   }
}

The media logic in this demo is implemented in the class CallMediaPipeline. As you can see, the media pipeline of this demo is quite simple: two WebRtcEndpoint elements directly interconnected. Please take note that the WebRtcEndpoints need to be connected twice, one for each media direction.

public class CallMediaPipeline {

   private MediaPipeline pipeline;
   private WebRtcEndpoint callerWebRtcEP;
   private WebRtcEndpoint calleeWebRtcEP;

   public CallMediaPipeline(KurentoClient kurento) {
      try {
         this.pipeline = kurento.createMediaPipeline();
         this.callerWebRtcEP = new WebRtcEndpoint.Builder(pipeline).build();
         this.calleeWebRtcEP = new WebRtcEndpoint.Builder(pipeline).build();

         this.callerWebRtcEP.connect(this.calleeWebRtcEP);
         this.calleeWebRtcEP.connect(this.callerWebRtcEP);
      } catch (Throwable t) {
         if (this.pipeline != null) {
            pipeline.release();
         }
      }
   }

   public String generateSdpAnswerForCaller(String sdpOffer) {
      return callerWebRtcEP.processOffer(sdpOffer);
   }

   public String generateSdpAnswerForCallee(String sdpOffer) {
      return calleeWebRtcEP.processOffer(sdpOffer);
   }

   public void release() {
      if (pipeline != null) {
         pipeline.release();
      }
   }

   public WebRtcEndpoint getCallerWebRtcEP() {
      return callerWebRtcEP;
   }

   public WebRtcEndpoint getCalleeWebRtcEP() {
      return calleeWebRtcEP;
   }

}

In this class we can see the implementation of methods generateSdpAnswerForCaller and generateSdpAnswerForCallee. These methods delegate to WebRtc endpoints to create the appropriate answer.

Client-Side¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js.

In the following snippet we can see the creation of the WebSocket (variable ws) in the path /call. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are five incoming messages to client: resgisterResponse, callResponse, incomingCall, iceCandidate and startCommunication. Convenient actions are taken to implement each step in the communication. For example, in functions call and incomingCall (for caller and callee respectively), the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

 var ws = new WebSocket('ws://' + location.host + '/call');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'resgisterResponse':
      resgisterResponse(parsedMessage);
      break;
   case 'callResponse':
      callResponse(parsedMessage);
      break;
   case 'incomingCall':
      incomingCall(parsedMessage);
      break;
   case 'startCommunication':
      startCommunication(parsedMessage);
      break;
   case 'stopCommunication':
      console.info("Communication ended by remote peer");
      stop(true);
      break;
   case 'iceCandidate':
       webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
        if (!error) return;
         console.error("Error adding candidate: " + error);
       });
       break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

function incomingCall(message) {
   //If bussy just reject without disturbing user
   if (callState != NO_CALL) {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'bussy'
      };
      return sendMessage(response);
   }

   setCallState(PROCESSING_CALL);
   if (confirm('User ' + message.from
         + ' is calling you. Do you accept the call?')) {
      showSpinner(videoInput, videoOutput);

      from = message.from;
      var options = {
               localVideo: videoInput,
               remoteVideo: videoOutput,
               onicecandidate: onIceCandidate,
               onerror: onError
             }
       webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function (error) {
           if(error) {
              return console.error(error);
           }
           webRtcPeer.generateOffer (onOfferIncomingCall);
         });

   } else {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'user declined'
      };
      sendMessage(response);
      stop();
   }
}

function call() {
   if (document.getElementById('peer').value == '') {
      window.alert("You must specify the peer name");
      return;
   }
   setCallState(PROCESSING_CALL);
   showSpinner(videoInput, videoOutput);

   var options = {
            localVideo: videoInput,
            remoteVideo: videoOutput,
            onicecandidate: onIceCandidate,
            onerror: onError
          }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
      function (error) {
        if(error) {
           return console.error(error);
        }
        webRtcPeer.generateOffer (onOfferCall);
   });
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>draggabilly</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

Node.js - One to one video call¶

This web application consists on a one-to-one video call using WebRTC technology. In other words, this application provides a simple video softphone.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-one2one-call
git checkout 6.10.0
npm install
npm start

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

The following picture shows an screenshot of this demo running in a web browser:

One to one video call screenshot

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the local stream and other for the remote peer stream). If two users, A and B, are using the application, the media flow goes this way: The video camera stream of user A is sent to the Kurento Media Server, which sends it to user B. In the same way, B sends to Kurento Media Server, which forwards it to A. This means that KMS is providing a B2B (back-to-back) call service.

To implement this behavior create a Media Pipeline composed by two WebRtC endpoints connected in B2B. The implemented media pipeline is illustrated in the following picture:

One to one video call Media Pipeline

The client and the server communicate through a signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and application server logic is as follows:

User A is registered in the application server with his name
User B is registered in the application server with her name
User A issues a call to User B
User B accepts the incoming call
The communication is established and media flows between User A and User B
One of the users finishes the video communication

The detailed message flow in a call are shown in the picture below:

One to many one call signaling protocol

As you can see in the diagram, SDP and ICE candidates need to be exchanged between client and server to establish the WebRTC connection between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer in the browser with the WebRtcEndpoint in the server. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

This demo has been developed using the express framework for Node.js, but express is not a requirement for Kurento. The main script of this demo is server.js.

In order to communicate the JavaScript client and the Node application server a WebSocket is used. The incoming messages to this WebSocket (variable ws in the code) are conveniently handled to implemented the signaling protocol depicted in the figure before (i.e. messages register, call, incomingCallResponse, stop, and onIceCandidate).

var ws = require('ws');

[...]

var wss = new ws.Server({
    server : server,
    path : '/one2one'
});

wss.on('connection', function(ws) {
    var sessionId = nextUniqueId();
    console.log('Connection received with sessionId ' + sessionId);

    ws.on('error', function(error) {
        console.log('Connection ' + sessionId + ' error');
        stop(sessionId);
    });

    ws.on('close', function() {
        console.log('Connection ' + sessionId + ' closed');
        stop(sessionId);
        userRegistry.unregister(sessionId);
    });

    ws.on('message', function(_message) {
        var message = JSON.parse(_message);
        console.log('Connection ' + sessionId + ' received message ', message);

        switch (message.id) {
        case 'register':
            register(sessionId, message.name, ws);
            break;

        case 'call':
            call(sessionId, message.to, message.from, message.sdpOffer);
            break;

        case 'incomingCallResponse':
            incomingCallResponse(sessionId, message.from, message.callResponse, message.sdpOffer, ws);
            break;

        case 'stop':
            stop(sessionId);
            break;

        case 'onIceCandidate':
            onIceCandidate(sessionId, message.candidate);
            break;

        default:
            ws.send(JSON.stringify({
                id : 'error',
                message : 'Invalid message ' + message
            }));
            break;
        }

    });
});

In order to perform a call, each user (the caller and the callee) must be register in the system. For this reason, in the server-side there is a class named UserRegistry to store and locate users. Then, the register message fires the execution of the following function:

// Represents registrar of users
function UserRegistry() {
    this.usersById = {};
    this.usersByName = {};
}

UserRegistry.prototype.register = function(user) {
    this.usersById[user.id] = user;
    this.usersByName[user.name] = user;
}

UserRegistry.prototype.unregister = function(id) {
    var user = this.getById(id);
    if (user) delete this.usersById[id]
    if (user && this.getByName(user.name)) delete this.usersByName[user.name];
}

UserRegistry.prototype.getById = function(id) {
    return this.usersById[id];
}

UserRegistry.prototype.getByName = function(name) {
    return this.usersByName[name];
}

UserRegistry.prototype.removeById = function(id) {
    var userSession = this.usersById[id];
    if (!userSession) return;
    delete this.usersById[id];
    delete this.usersByName[userSession.name];
}

function register(id, name, ws, callback) {
    function onError(error) {
        ws.send(JSON.stringify({id:'registerResponse', response : 'rejected ', message: error}));
    }

    if (!name) {
        return onError("empty user name");
    }

    if (userRegistry.getByName(name)) {
        return onError("User " + name + " is already registered");
    }

    userRegistry.register(new UserSession(id, name, ws));
    try {
        ws.send(JSON.stringify({id: 'registerResponse', response: 'accepted'}));
    } catch(exception) {
        onError(exception);
    }
}

In order to control the media capabilities provided by the Kurento Media Server, we need an instance of the KurentoClient in the Node application server. In order to create this instance, we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888.

var kurento = require('kurento-client');

var kurentoClient = null;

var argv = minimist(process.argv.slice(2), {
    default: {
        as_uri: 'https://localhost:8443/',
        ws_uri: 'ws://localhost:8888/kurento'
    }
});

[...]

function getKurentoClient(callback) {
    if (kurentoClient !== null) {
        return callback(null, kurentoClient);
    }

    kurento(argv.ws_uri, function(error, _kurentoClient) {
        if (error) {
            console.log("Could not find media server at address " + argv.ws_uri);
            return callback("Could not find media server at address" + argv.ws_uri
                    + ". Exiting with error " + error);
        }

        kurentoClient = _kurentoClient;
        callback(null, kurentoClient);
    });
}

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server. Our first operation is to create a Media Pipeline, then we need to create the Media Elements and connect them. In this example, we need two WebRtcEndpoints, i.e. one peer caller and other one for the callee. This media logic is implemented in the class CallMediaPipeline. Note that the WebRtcEndpoints need to be connected twice, one for each media direction. This object is created in the function incomingCallResponse which is fired in the callee peer, after the caller executes the function call:

function call(callerId, to, from, sdpOffer) {
    clearCandidatesQueue(callerId);

    var caller = userRegistry.getById(callerId);
    var rejectCause = 'User ' + to + ' is not registered';
    if (userRegistry.getByName(to)) {
        var callee = userRegistry.getByName(to);
        caller.sdpOffer = sdpOffer
        callee.peer = from;
        caller.peer = to;
        var message = {
            id: 'incomingCall',
            from: from
        };
        try{
            return callee.sendMessage(message);
        } catch(exception) {
            rejectCause = "Error " + exception;
        }
    }
    var message  = {
        id: 'callResponse',
        response: 'rejected: ',
        message: rejectCause
    };
    caller.sendMessage(message);
}

function incomingCallResponse(calleeId, from, callResponse, calleeSdp, ws) {
    clearCandidatesQueue(calleeId);

    function onError(callerReason, calleeReason) {
        if (pipeline) pipeline.release();
        if (caller) {
            var callerMessage = {
                id: 'callResponse',
                response: 'rejected'
            }
            if (callerReason) callerMessage.message = callerReason;
            caller.sendMessage(callerMessage);
        }

        var calleeMessage = {
            id: 'stopCommunication'
        };
        if (calleeReason) calleeMessage.message = calleeReason;
        callee.sendMessage(calleeMessage);
    }

    var callee = userRegistry.getById(calleeId);
    if (!from || !userRegistry.getByName(from)) {
        return onError(null, 'unknown from = ' + from);
    }
    var caller = userRegistry.getByName(from);

    if (callResponse === 'accept') {
        var pipeline = new CallMediaPipeline();
        pipelines[caller.id] = pipeline;
        pipelines[callee.id] = pipeline;

        pipeline.createPipeline(caller.id, callee.id, ws, function(error) {
            if (error) {
                return onError(error, error);
            }

            pipeline.generateSdpAnswer(caller.id, caller.sdpOffer, function(error, callerSdpAnswer) {
                if (error) {
                    return onError(error, error);
                }

                pipeline.generateSdpAnswer(callee.id, calleeSdp, function(error, calleeSdpAnswer) {
                    if (error) {
                        return onError(error, error);
                    }

                    var message = {
                        id: 'startCommunication',
                        sdpAnswer: calleeSdpAnswer
                    };
                    callee.sendMessage(message);

                    message = {
                        id: 'callResponse',
                        response : 'accepted',
                        sdpAnswer: callerSdpAnswer
                    };
                    caller.sendMessage(message);
                });
            });
        });
    } else {
        var decline = {
            id: 'callResponse',
            response: 'rejected',
            message: 'user declined'
        };
        caller.sendMessage(decline);
    }
}

As of Kurento Media Server 6.0, the WebRTC negotiation is done by exchanging ICE candidates between the WebRTC peers. To implement this protocol, the webRtcEndpoint receives candidates from the client in OnIceCandidate function. These candidates are stored in a queue when the webRtcEndpoint is not available yet. Then these candidates are added to the media element by calling to the addIceCandidate method.

var candidatesQueue = {};

[...]

function onIceCandidate(sessionId, _candidate) {
    var candidate = kurento.getComplexType('IceCandidate')(_candidate);
    var user = userRegistry.getById(sessionId);

    if (pipelines[user.id] && pipelines[user.id].webRtcEndpoint && pipelines[user.id].webRtcEndpoint[user.id]) {
        var webRtcEndpoint = pipelines[user.id].webRtcEndpoint[user.id];
        webRtcEndpoint.addIceCandidate(candidate);
    }
    else {
        if (!candidatesQueue[user.id]) {
            candidatesQueue[user.id] = [];
        }
        candidatesQueue[sessionId].push(candidate);
    }
}

function clearCandidatesQueue(sessionId) {
    if (candidatesQueue[sessionId]) {
        delete candidatesQueue[sessionId];
    }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application. These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /one2one. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('ws://' + location.host + '/one2one');
var webRtcPeer;

[...]

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'registerResponse':
      resgisterResponse(parsedMessage);
      break;
   case 'callResponse':
      callResponse(parsedMessage);
      break;
   case 'incomingCall':
      incomingCall(parsedMessage);
      break;
   case 'startCommunication':
      startCommunication(parsedMessage);
      break;
   case 'stopCommunication':
      console.info("Communication ended by remote peer");
      stop(true);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate)
      break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

On the one hand, the function call is executed in the caller client-side, using the method WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js to start a WebRTC communication in duplex mode. On the other hand, the function incomingCall in the callee client-side uses also the method WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js to complete the WebRTC call.

function call() {
   if (document.getElementById('peer').value == '') {
      window.alert("You must specify the peer name");
      return;
   }

   setCallState(PROCESSING_CALL);

   showSpinner(videoInput, videoOutput);

   var options = {
      localVideo : videoInput,
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(
         error) {
      if (error) {
         console.error(error);
         setCallState(NO_CALL);
      }

      this.generateOffer(function(error, offerSdp) {
         if (error) {
            console.error(error);
            setCallState(NO_CALL);
         }
         var message = {
            id : 'call',
            from : document.getElementById('name').value,
            to : document.getElementById('peer').value,
            sdpOffer : offerSdp
         };
         sendMessage(message);
      });
   });
}

function incomingCall(message) {
   // If bussy just reject without disturbing user
   if (callState != NO_CALL) {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'bussy'

      };
      return sendMessage(response);
   }

   setCallState(PROCESSING_CALL);
   if (confirm('User ' + message.from
         + ' is calling you. Do you accept the call?')) {
      showSpinner(videoInput, videoOutput);

      var options = {
         localVideo : videoInput,
         remoteVideo : videoOutput,
         onicecandidate : onIceCandidate
      }

      webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
            function(error) {
               if (error) {
                  console.error(error);
                  setCallState(NO_CALL);
               }

               this.generateOffer(function(error, offerSdp) {
                  if (error) {
                     console.error(error);
                     setCallState(NO_CALL);
                  }
                  var response = {
                     id : 'incomingCallResponse',
                     from : message.from,
                     callResponse : 'accept',
                     sdpOffer : offerSdp
                  };
                  sendMessage(response);
               });
            });

   } else {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'user declined'
      };
      sendMessage(response);
      stop(true);
   }
}

Dependencies¶

Server-side dependencies of this demo are managed using npm. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   [...]
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   [...]
   "kurento-utils" : "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at npm and Bower.

WebRTC One-To-One video call with recording and filtering¶

This is an enhanced version of the the One-To-One application with video recording and Augmented Reality.

Java - Advanced One to one video call¶

This web application consists on an advanced one to one video call using WebRTC technology. It is an improved version of the one 2 one call tutorial).

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-one2one-call-advanced
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application incorporates the recording capability provided by the Kurento Media Server in a one to one video communication. In addition, a filter element (FaceOverlayFilter) is placed between the WebRtcEndpoints of the Media Pipeline. The following picture shows an screenshot of this demo running in a web browser:

Advanced one to one video call screenshot

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the local video camera stream (the caller stream, the smaller video in the picture) and other for the remote peer in the call (the callee stream, the bigger video in the picture). If two users, A and B, are using the application, the media flow goes this way: The video camera stream of user A is sent to the Kurento Media Server and sent again to the user B. On the other hand, user B sends its video camera stream to Kurento and then it is sent to user A.

This application is implemented by means of two Media Pipeline ‘s. First, the rich real-time WebRTC communication is performed two WebRtcEndpoints interconnected, and with a FaceOverlayFilter in between them. In addition and a RecorderEndpoint is used to store both streams in the file system of the Kurento Media Server. This media pipeline is illustrated in the following picture:

Advanced one to one video call media pipeline (1)

A second media pipeline is needed to play the previously recorded media. This pipeline is composed by a PlayerEndpoint which reads the files stored in the Kurento Media Server. This media element injects the media in a WebRtcEndpoint which is charge to transport the media to the HTML5 video tag in the browser:

Advanced one to one video call media pipeline (2)

Note

The playback of a static file can be done in several ways. In addition to this media pipeline (PlayerEndpoint -> WebRtcEndpoint) the recorded file could be served directly by an HTTP server.

To communicate the client with the server to manage calls we have designed a signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server would be as follows:

User A is registered in the server with his name
User B is registered in the server with her name
User A wants to call to User B
User B accepts the incoming call
The communication is established and media is flowing between User A and User B
One of the users finishes the video communication
One of the users play the recorded media

This is very simple protocol designed to show a simple one to one call application implemented with Kurento. In a professional application it can be improved, for example implementing seeking user, ordered finish, among other functions.

Assuming that User A is using Client A and User B is using Client B, we can draw the following sequence diagram with detailed messages between clients and server. The following diagram shows the two parts of the signaling protocol: first the enhanced real-time communication is performed, and then the playback of the recorded file is carried out.

Advanced one to one video call signaling protocol

As you can see in the diagram, SDP and ICE candidates need to be interchanged between client and server to establish the WebRTC connection between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer in the browser with the WebRtcEndpoint in the server.

The following sections describe in detail the server-side, the client-side, and how to run the demo. The complete source code of this demo can be found in GitHub.

Application Server Logic¶

As in the Magic Mirror tutorial, this demo has been developed using Java and Spring Boot.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vertex, etc. We have choose Spring Boot for convenience.

In the following figure you can see a class diagram of the server side code:

Server-side class diagram of the advanced one to one video call app

The main class of this demo is named One2OneCallAdvApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean.

@EnableWebSocket
@SpringBootApplication
public class One2OneCallAdvApp implements WebSocketConfigurer {

   final static String DEFAULT_APP_SERVER_URL = "https://localhost:8443";

   @Bean
   public CallHandler callHandler() {
      return new CallHandler();
   }

   @Bean
   public UserRegistry registry() {
      return new UserRegistry();
   }

   @Bean
   public KurentoClient kurentoClient() {
      return KurentoClient.create();
   }

   public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
      registry.addHandler(callHandler(), "/call");
   }

   public static void main(String[] args) throws Exception {
      new SpringApplication(One2OneCallAdvApp.class).run(args);
   }

}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /call.

CallHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are five different kind of incoming messages to the Server : register, call, incomingCallResponse, onIceCandidate and play. These messages are treated in the switch clause, taking the proper steps in each case.

public class CallHandler extends TextWebSocketHandler {

   private static final Logger log = LoggerFactory
         .getLogger(CallHandler.class);
   private static final Gson gson = new GsonBuilder().create();

   private final ConcurrentHashMap<String, MediaPipeline> pipelines = new ConcurrentHashMap<String, MediaPipeline>();

   @Autowired
   private KurentoClient kurento;

   @Autowired
   private UserRegistry registry;

   @Override
   public void handleTextMessage(WebSocketSession session, TextMessage message)
         throws Exception {
      JsonObject jsonMessage = gson.fromJson(message.getPayload(),
            JsonObject.class);
      UserSession user = registry.getBySession(session);

      if (user != null) {
         log.debug("Incoming message from user '{}': {}", user.getName(),
               jsonMessage);
      } else {
         log.debug("Incoming message from new user: {}", jsonMessage);
      }

      switch (jsonMessage.get("id").getAsString()) {
      case "register":
         register(session, jsonMessage);
         break;
      case "call":
         call(user, jsonMessage);
         break;
      case "incomingCallResponse":
         incomingCallResponse(user, jsonMessage);
         break;
      case "play":
         play(user, jsonMessage);
         break;
      case "onIceCandidate": {
         JsonObject candidate = jsonMessage.get("candidate")
               .getAsJsonObject();

         if (user != null) {
            IceCandidate cand = new IceCandidate(candidate.get("candidate")
                  .getAsString(), candidate.get("sdpMid").getAsString(),
                  candidate.get("sdpMLineIndex").getAsInt());
            user.addCandidate(cand);
         }
         break;
      }
      case "stop":
         stop(session);
         releasePipeline(user);
      case "stopPlay":
         releasePipeline(user);
      default:
         break;
      }
   }

   private void register(WebSocketSession session, JsonObject jsonMessage)
         throws IOException {
      ...
   }

   private void call(UserSession caller, JsonObject jsonMessage)
         throws IOException {
      ...
   }

   private void incomingCallResponse(final UserSession callee,
         JsonObject jsonMessage) throws IOException {
      ...
   }

   public void stop(WebSocketSession session) throws IOException {
      ...
   }

   public void releasePipeline(UserSession session) throws IOException {
      ...
   }

   private void play(final UserSession session, JsonObject jsonMessage)
         throws IOException {
      ...
   }

   @Override
   public void afterConnectionClosed(WebSocketSession session,
         CloseStatus status) throws Exception {
      stop(session);
      registry.removeBySession(session);
   }

}

In the following snippet, we can see the register method. Basically, it obtains the name attribute from register message and check if there are a registered user with that name. If not, the new user is registered and an acceptance message is sent to it.

private void register(WebSocketSession session, JsonObject jsonMessage)
      throws IOException {
   String name = jsonMessage.getAsJsonPrimitive("name").getAsString();

   UserSession caller = new UserSession(session, name);
   String responseMsg = "accepted";
   if (name.isEmpty()) {
      responseMsg = "rejected: empty user name";
   } else if (registry.exists(name)) {
      responseMsg = "rejected: user '" + name + "' already registered";
   } else {
      registry.register(caller);
   }

   JsonObject response = new JsonObject();
   response.addProperty("id", "resgisterResponse");
   response.addProperty("response", responseMsg);
   caller.sendMessage(response);
}

In the call method, the server checks if there are a registered user with the name specified in to message attribute and send an incomingCall message to it. Or, if there isn’t any user with that name, a callResponse message is sent to caller rejecting the call.

private void call(UserSession caller, JsonObject jsonMessage)
      throws IOException {
   String to = jsonMessage.get("to").getAsString();
   String from = jsonMessage.get("from").getAsString();
   JsonObject response = new JsonObject();

   if (registry.exists(to)) {
      UserSession callee = registry.getByName(to);
      caller.setSdpOffer(jsonMessage.getAsJsonPrimitive("sdpOffer")
            .getAsString());
      caller.setCallingTo(to);

      response.addProperty("id", "incomingCall");
      response.addProperty("from", from);

      callee.sendMessage(response);
      callee.setCallingFrom(from);
   } else {
      response.addProperty("id", "callResponse");
      response.addProperty("response", "rejected");
      response.addProperty("message", "user '" + to
            + "' is not registered");

      caller.sendMessage(response);
   }
}

In the incomingCallResponse method, if the callee user accepts the call, it is established and the media elements are created to connect the caller with the callee. Basically, the server creates a CallMediaPipeline object, to encapsulate the media pipeline creation and management. Then, this object is used to negotiate media interchange with user’s browsers.

As explained in the Magic Mirror tutorial, the negotiation between WebRTC peer in the browser and WebRtcEndpoint in Kurento Server is made by means of SDP generation at the client (offer) and SDP generation at the server (answer). The SDP answers are generated with the Kurento Java Client inside the class CallMediaPipeline (as we see in a moment). The methods used to generate SDP are generateSdpAnswerForCallee(calleeSdpOffer) and generateSdpAnswerForCaller(callerSdpOffer):

private void incomingCallResponse(final UserSession callee,
      JsonObject jsonMessage) throws IOException {
   String callResponse = jsonMessage.get("callResponse").getAsString();
   String from = jsonMessage.get("from").getAsString();
   final UserSession calleer = registry.getByName(from);
   String to = calleer.getCallingTo();

   if ("accept".equals(callResponse)) {
      log.debug("Accepted call from '{}' to '{}'", from, to);

      CallMediaPipeline callMediaPipeline = new CallMediaPipeline(
            kurento, from, to);
      pipelines.put(calleer.getSessionId(),
            callMediaPipeline.getPipeline());
      pipelines.put(callee.getSessionId(),
            callMediaPipeline.getPipeline());

      String calleeSdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String calleeSdpAnswer = callMediaPipeline
            .generateSdpAnswerForCallee(calleeSdpOffer);

      callee.setWebRtcEndpoint(callMediaPipeline.getCalleeWebRtcEP());
      callMediaPipeline.getCalleeWebRtcEP().addIceCandidateFoundListener(
            new EventListener<IceCandidateFoundEvent>() {

               @Override
               public void onEvent(IceCandidateFoundEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "iceCandidate");
                  response.add("candidate", JsonUtils
                        .toJsonObject(event.getCandidate()));
                  try {
                     synchronized (callee.getSession()) {
                        callee.getSession()
                              .sendMessage(
                                    new TextMessage(response
                                          .toString()));
                     }
                  } catch (IOException e) {
                     log.debug(e.getMessage());
                  }
               }
            });

      JsonObject startCommunication = new JsonObject();
      startCommunication.addProperty("id", "startCommunication");
      startCommunication.addProperty("sdpAnswer", calleeSdpAnswer);

      synchronized (callee) {
         callee.sendMessage(startCommunication);
      }

      callMediaPipeline.getCalleeWebRtcEP().gatherCandidates();

      String callerSdpOffer = registry.getByName(from).getSdpOffer();

      calleer.setWebRtcEndpoint(callMediaPipeline.getCallerWebRtcEP());
      callMediaPipeline.getCallerWebRtcEP().addIceCandidateFoundListener(
            new EventListener<IceCandidateFoundEvent>() {

               @Override
               public void onEvent(IceCandidateFoundEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "iceCandidate");
                  response.add("candidate", JsonUtils
                        .toJsonObject(event.getCandidate()));
                  try {
                     synchronized (calleer.getSession()) {
                        calleer.getSession()
                              .sendMessage(
                                    new TextMessage(response
                                          .toString()));
                     }
                  } catch (IOException e) {
                     log.debug(e.getMessage());
                  }
               }
            });

      String callerSdpAnswer = callMediaPipeline
            .generateSdpAnswerForCaller(callerSdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "callResponse");
      response.addProperty("response", "accepted");
      response.addProperty("sdpAnswer", callerSdpAnswer);

      synchronized (calleer) {
         calleer.sendMessage(response);
      }

      callMediaPipeline.getCallerWebRtcEP().gatherCandidates();

      callMediaPipeline.record();

   } else {
      JsonObject response = new JsonObject();
      response.addProperty("id", "callResponse");
      response.addProperty("response", "rejected");
      calleer.sendMessage(response);
   }
}

Finally, the play method instantiates a PlayMediaPipeline object, which is used to create Media Pipeline in charge of the playback of the recorded streams in the Kurento Media Server.

private void play(final UserSession session, JsonObject jsonMessage)
      throws IOException {
   String user = jsonMessage.get("user").getAsString();
   log.debug("Playing recorded call of user '{}'", user);

   JsonObject response = new JsonObject();
   response.addProperty("id", "playResponse");

   if (registry.getByName(user) != null
         && registry.getBySession(session.getSession()) != null) {
      final PlayMediaPipeline playMediaPipeline = new PlayMediaPipeline(
            kurento, user, session.getSession());
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();

      session.setPlayingWebRtcEndpoint(playMediaPipeline.getWebRtc());

      playMediaPipeline.getPlayer().addEndOfStreamListener(
            new EventListener<EndOfStreamEvent>() {
               @Override
               public void onEvent(EndOfStreamEvent event) {
                  UserSession user = registry
                        .getBySession(session.getSession());
                  releasePipeline(user);
                  playMediaPipeline.sendPlayEnd(session.getSession());
               }
            });

      playMediaPipeline.getWebRtc().addIceCandidateFoundListener(
            new EventListener<IceCandidateFoundEvent>() {

               @Override
               public void onEvent(IceCandidateFoundEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "iceCandidate");
                  response.add("candidate", JsonUtils
                        .toJsonObject(event.getCandidate()));
                  try {
                     synchronized (session) {
                        session.getSession()
                              .sendMessage(
                                    new TextMessage(response
                                          .toString()));
                     }
                  } catch (IOException e) {
                     log.debug(e.getMessage());
                  }
               }
            });

      String sdpAnswer = playMediaPipeline.generateSdpAnswer(sdpOffer);

      response.addProperty("response", "accepted");

      response.addProperty("sdpAnswer", sdpAnswer);

      playMediaPipeline.play();
      pipelines.put(session.getSessionId(),
            playMediaPipeline.getPipeline());
      synchronized (session.getSession()) {
         session.sendMessage(response);
      }

      playMediaPipeline.getWebRtc().gatherCandidates();

   } else {
      response.addProperty("response", "rejected");
      response.addProperty("error", "No recording for user '" + user
            + "'. Please type a correct user in the 'Peer' field.");
      session.getSession().sendMessage(
            new TextMessage(response.toString()));
   }
}

The media logic in this demo is implemented in the classes CallMediaPipeline and PlayMediaPipeline. The first media pipeline consists on two WebRtcEndpoint elements interconnected with a FaceOverlayFilter in between, and also with and RecorderEndpoint to carry out the recording of the WebRTC communication. Please take note that the WebRtc endpoints needs to be connected twice, one for each media direction. In this class we can see the implementation of methods generateSdpAnswerForCaller and generateSdpAnswerForCallee. These methods delegate to WebRtc endpoints to create the appropriate answer.

public class CallMediaPipeline {

   private static final SimpleDateFormat df = new SimpleDateFormat(
         "yyyy-MM-dd_HH-mm-ss-S");
   public static final String RECORDING_PATH = "file:///tmp/"
         + df.format(new Date()) + "-";
   public static final String RECORDING_EXT = ".webm";

   private final MediaPipeline pipeline;
   private final WebRtcEndpoint webRtcCaller;
   private final WebRtcEndpoint webRtcCallee;
   private final RecorderEndpoint recorderCaller;
   private final RecorderEndpoint recorderCallee;

   public CallMediaPipeline(KurentoClient kurento, String from, String to) {

      // Media pipeline
      pipeline = kurento.createMediaPipeline();

      // Media Elements (WebRtcEndpoint, RecorderEndpoint, FaceOverlayFilter)
      webRtcCaller = new WebRtcEndpoint.Builder(pipeline).build();
      webRtcCallee = new WebRtcEndpoint.Builder(pipeline).build();

      recorderCaller = new RecorderEndpoint.Builder(pipeline, RECORDING_PATH
            + from + RECORDING_EXT).build();
      recorderCallee = new RecorderEndpoint.Builder(pipeline, RECORDING_PATH
            + to + RECORDING_EXT).build();

      String appServerUrl = System.getProperty("app.server.url",
            One2OneCallAdvApp.DEFAULT_APP_SERVER_URL);
      FaceOverlayFilter faceOverlayFilterCaller = new FaceOverlayFilter.Builder(
            pipeline).build();
      faceOverlayFilterCaller.setOverlayedImage(appServerUrl
            + "/img/mario-wings.png", -0.35F, -1.2F, 1.6F, 1.6F);

      FaceOverlayFilter faceOverlayFilterCallee = new FaceOverlayFilter.Builder(
            pipeline).build();
      faceOverlayFilterCallee.setOverlayedImage(
            appServerUrl + "/img/Hat.png", -0.2F, -1.35F, 1.5F, 1.5F);

      // Connections
      webRtcCaller.connect(faceOverlayFilterCaller);
      faceOverlayFilterCaller.connect(webRtcCallee);
      faceOverlayFilterCaller.connect(recorderCaller);

      webRtcCallee.connect(faceOverlayFilterCallee);
      faceOverlayFilterCallee.connect(webRtcCaller);
      faceOverlayFilterCallee.connect(recorderCallee);
   }

   public void record() {
      recorderCaller.record();
      recorderCallee.record();
   }

   public String generateSdpAnswerForCaller(String sdpOffer) {
      return webRtcCaller.processOffer(sdpOffer);
   }

   public String generateSdpAnswerForCallee(String sdpOffer) {
      return webRtcCallee.processOffer(sdpOffer);
   }

   public MediaPipeline getPipeline() {
      return pipeline;
   }

   public WebRtcEndpoint getCallerWebRtcEP() {
      return webRtcCaller;
   }

   public WebRtcEndpoint getCalleeWebRtcEP() {
      return webRtcCallee;
   }
}

Note

Notice the hat URLs are provided by the application server and consumed by the KMS. This logic is assuming that the application server is hosted in local (localhost), and by the default the hat URLs are https://localhost:8443/img/mario-wings.png and https://localhost:8443/img/Hat.png. If your application server is hosted in a different host, it can be easily changed by means of the configuration parameter app.server.url, for example:

mvn -U clean spring-boot:run -Dapp.server.url=https://app_server_host:app_server_port

The second media pipeline consists on a PlayerEndpoint connected to a WebRtcEndpoint. The PlayerEndpoint reads the previously recorded media in the file system of the Kurento Media Server. The WebRtcEndpoint is used in receive-only mode.

public class PlayMediaPipeline {

   private static final Logger log = LoggerFactory
         .getLogger(PlayMediaPipeline.class);

   private WebRtcEndpoint webRtc;
   private PlayerEndpoint player;

   public PlayMediaPipeline(KurentoClient kurento, String user,
         final WebSocketSession session) {
      // Media pipeline
      MediaPipeline pipeline = kurento.createMediaPipeline();

      // Media Elements (WebRtcEndpoint, PlayerEndpoint)
      webRtc = new WebRtcEndpoint.Builder(pipeline).build();
      player = new PlayerEndpoint.Builder(pipeline, RECORDING_PATH + user
            + RECORDING_EXT).build();

      // Connection
      player.connect(webRtc);

      // Player listeners
      player.addErrorListener(new EventListener<ErrorEvent>() {
         @Override
         public void onEvent(ErrorEvent event) {
            log.info("ErrorEvent: {}", event.getDescription());
            sendPlayEnd(session);
         }
      });
   }

   public void sendPlayEnd(WebSocketSession session) {
      try {
         JsonObject response = new JsonObject();
         response.addProperty("id", "playEnd");
         session.sendMessage(new TextMessage(response.toString()));
      } catch (IOException e) {
         log.error("Error sending playEndOfStream message", e);
      }
   }

   public void play() {
      player.play();
   }

   public String generateSdpAnswer(String sdpOffer) {
      return webRtc.processOffer(sdpOffer);
   }

   public MediaPipeline getPipeline() {
      return pipeline;
   }

   public WebRtcEndpoint getWebRtc() {
      return webRtc;
   }

   public PlayerEndpoint getPlayer() {
      return player;
   }

}

Client-Side¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js.

In the following snippet we can see the creation of the WebSocket (variable ws) in the path /call. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are six incoming messages to client: resgisterResponse, callResponse, incomingCall, startCommunication, iceCandidate and play. Convenient actions are taken to implement each step in the communication. On the one hand, in functions call and incomingCall (for caller and callee respectively), the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication. On the other hand in the function play, the function WebRtcPeer.WebRtcPeerRecvonly is called since the WebRtcEndpoint is used in receive-only.

var ws = new WebSocket('ws://' + location.host + '/call');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'resgisterResponse':
      resgisterResponse(parsedMessage);
      break;
   case 'callResponse':
      callResponse(parsedMessage);
      break;
   case 'incomingCall':
      incomingCall(parsedMessage);
      break;
   case 'startCommunication':
      startCommunication(parsedMessage);
      break;
   case 'stopCommunication':
      console.info("Communication ended by remote peer");
      stop(true);
      break;
   case 'playResponse':
      playResponse(parsedMessage);
      break;
   case 'playEnd':
      playEnd();
      break;
   case 'iceCandidate':
       webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
        if (!error) return;
         console.error("Error adding candidate: " + error);
       });
       break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

function incomingCall(message) {
   // If bussy just reject without disturbing user
   if (callState != NO_CALL && callState != POST_CALL) {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'bussy'
      };
      return sendMessage(response);
   }

   setCallState(DISABLED);
   if (confirm('User ' + message.from
         + ' is calling you. Do you accept the call?')) {
      showSpinner(videoInput, videoOutput);

      from = message.from;
      var options = {
               localVideo: videoInput,
               remoteVideo: videoOutput,
               onicecandidate: onIceCandidate
             }
       webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function (error) {
           if(error) {
              return console.error(error);
           }
           this.generateOffer (onOfferIncomingCall);
         });
   } else {
      var response = {
         id : 'incomingCallResponse',
         from : message.from,
         callResponse : 'reject',
         message : 'user declined'
      };
      sendMessage(response);
      stop();
   }
}

function call() {
   if (document.getElementById('peer').value == '') {
      document.getElementById('peer').focus();
      window.alert("You must specify the peer name");
      return;
   }
   setCallState(DISABLED);
   showSpinner(videoInput, videoOutput);

   var options = {
            localVideo: videoInput,
            remoteVideo: videoOutput,
            onicecandidate: onIceCandidate
          }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
      function (error) {
        if(error) {
           return console.error(error);
        }
        this.generateOffer (onOfferCall);
   });
}

function play() {
   var peer = document.getElementById('peer').value;
   if (peer == '') {
      window.alert("You must insert the name of the user recording to be played (field 'Peer')");
      document.getElementById('peer').focus();
      return;
   }

   document.getElementById('videoSmall').style.display = 'none';
   setCallState(DISABLED);
   showSpinner(videoOutput);

   var options = {
            remoteVideo: videoOutput,
            onicecandidate: onIceCandidate
          }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
      function (error) {
        if(error) {
           return console.error(error);
        }
        this.generateOffer (onOfferPlay);
   });
}

function stop(message) {
   var stopMessageId = (callState == IN_CALL) ? 'stop' : 'stopPlay';
   setCallState(POST_CALL);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      if (!message) {
         var message = {
            id : stopMessageId
         }
         sendMessage(message);
      }
   }
   hideSpinner(videoInput, videoOutput);
   document.getElementById('videoSmall').style.display = 'block';
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>draggabilly</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

WebRTC Many-To-Many video call (Group Call)¶

This tutorial connects several participants to the same video conference. A group call will consist (in the media server side) in N*N WebRTC endpoints, where N is the number of clients connected to that conference.

Java - Group Call¶

This tutorial shows how to work wih the concept of rooms, allowing to connect several clients between them using WebRTC technology, creating a multiconference.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-group-call
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This tutorial shows how to work with the concept of rooms. Each room will create its own pipeline, being isolated from the other rooms. Clients connecting to a certain room, will only be able to exchange media with clients in the same room.

Each client will send its own media, and in turn will receive the media from all the other participants. This means that there will be a total of n*n webrtc endpoints in each room, where n is the number of clients.

When a new client enters the room, a new webrtc will be created and negotiated receive the media on the server. On the other hand, all participant will be informed that a new user has connected. Then, all participants will request the server to receive the new participant’s media.

The newcomer, in turn, gets a list of all connected participants, and requests the server to receive the media from all the present clients in the room.

When a client leaves the room, all clients are informed by the server. Then, the client-side code requests the server to cancel all media elements related to the client that left.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side with Spring Boot framework. This technology can be used to embed the Tomcat web server in the application and thus simplify the development process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is GroupCalldApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@EnableWebSocket
@SpringBootApplication
public class GroupCallApp implements WebSocketConfigurer {

  @Bean
  public UserRegistry registry() {
    return new UserRegistry();
  }

  @Bean
  public RoomManager roomManager() {
    return new RoomManager();
  }

  @Bean
  public CallHandler groupCallHandler() {
    return new CallHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  public static void main(String[] args) throws Exception {
    SpringApplication.run(GroupCallApp.class, args);
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(groupCallHandler(), "/groupcall");
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /groupcall.

CallHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are five different kind of incoming messages to the application server: joinRoom, receiveVideoFrom, leaveRoom and onIceCandidate. These messages are treated in the switch clause, taking the proper steps in each case.

public class CallHandler extends TextWebSocketHandler {

  private static final Logger log = LoggerFactory.getLogger(CallHandler.class);

  private static final Gson gson = new GsonBuilder().create();

  @Autowired
  private RoomManager roomManager;

  @Autowired
  private UserRegistry registry;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    final JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    final UserSession user = registry.getBySession(session);

    if (user != null) {
      log.debug("Incoming message from user '{}': {}", user.getName(), jsonMessage);
    } else {
      log.debug("Incoming message from new user: {}", jsonMessage);
    }

    switch (jsonMessage.get("id").getAsString()) {
      case "joinRoom":
        joinRoom(jsonMessage, session);
        break;
      case "receiveVideoFrom":
        final String senderName = jsonMessage.get("sender").getAsString();
        final UserSession sender = registry.getByName(senderName);
        final String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
        user.receiveVideoFrom(sender, sdpOffer);
        break;
      case "leaveRoom":
        leaveRoom(user);
        break;
      case "onIceCandidate":
        JsonObject candidate = jsonMessage.get("candidate").getAsJsonObject();

        if (user != null) {
          IceCandidate cand = new IceCandidate(candidate.get("candidate").getAsString(),
              candidate.get("sdpMid").getAsString(), candidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(cand, jsonMessage.get("name").getAsString());
        }
        break;
      default:
        break;
    }
  }

  @Override
  public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
      ...
  }

  private void joinRoom(JsonObject params, WebSocketSession session) throws IOException {
      ...
  }

  private void leaveRoom(UserSession user) throws IOException {
      ...
  }
}

In the following snippet, we can see the afterConnectionClosed method. Basically, it removes the userSession from registry and throws out the user from the room.

@Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
   UserSession user = registry.removeBySession(session);
   roomManager.getRoom(user.getRoomName()).leave(user);
}

In the joinRoom method, the server checks if there are a registered room with the name specified, add the user into this room and registries the user.

private void joinRoom(JsonObject params, WebSocketSession session) throws IOException {
   final String roomName = params.get("room").getAsString();
   final String name = params.get("name").getAsString();
   log.info("PARTICIPANT {}: trying to join room {}", name, roomName);

   Room room = roomManager.getRoom(roomName);
   final UserSession user = room.join(name, session);
   registry.register(user);
}

The leaveRoom method finish the video call from one user.

private void leaveRoom(UserSession user) throws IOException {
    final Room room = roomManager.getRoom(user.getRoomName());
    room.leave(user);
    if (room.getParticipants().isEmpty()) {
      roomManager.removeRoom(room);
    }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the conferenceroom.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /groupcall. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: existingParticipants, newParticipantArrived, participantLeft, receiveVideoAnswer and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/groupcall');
var participants = {};
var name;

window.onbeforeunload = function() {
   ws.close();
};

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'existingParticipants':
      onExistingParticipants(parsedMessage);
      break;
   case 'newParticipantArrived':
      onNewParticipant(parsedMessage);
      break;
   case 'participantLeft':
      onParticipantLeft(parsedMessage);
      break;
   case 'receiveVideoAnswer':
      receiveVideoResponse(parsedMessage);
      break;
   case 'iceCandidate':
      participants[parsedMessage.name].rtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
           if (error) {
            console.error("Error adding candidate: " + error);
            return;
           }
       });
       break;
   default:
      console.error('Unrecognized message', parsedMessage);
   }
}

function register() {
   name = document.getElementById('name').value;
   var room = document.getElementById('roomName').value;

   document.getElementById('room-header').innerText = 'ROOM ' + room;
   document.getElementById('join').style.display = 'none';
   document.getElementById('room').style.display = 'block';

   var message = {
      id : 'joinRoom',
      name : name,
      room : room,
   }
   sendMessage(message);
}

function onNewParticipant(request) {
   receiveVideo(request.name);
}

function receiveVideoResponse(result) {
   participants[result.name].rtcPeer.processAnswer (result.sdpAnswer, function (error) {
      if (error) return console.error (error);
   });
}

function callResponse(message) {
   if (message.response != 'accepted') {
      console.info('Call not accepted by peer. Closing call');
      stop();
   } else {
      webRtcPeer.processAnswer(message.sdpAnswer, function (error) {
         if (error) return console.error (error);
      });
   }
}

function onExistingParticipants(msg) {
   var constraints = {
      audio : true,
      video : {
         mandatory : {
            maxWidth : 320,
            maxFrameRate : 15,
            minFrameRate : 15
         }
      }
   };
   console.log(name + " registered in room " + room);
   var participant = new Participant(name);
   participants[name] = participant;
   var video = participant.getVideoElement();

   var options = {
         localVideo: video,
         mediaConstraints: constraints,
         onicecandidate: participant.onIceCandidate.bind(participant)
       }
   participant.rtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendonly(options,
      function (error) {
        if(error) {
           return console.error(error);
        }
        this.generateOffer (participant.offerToReceiveVideo.bind(participant));
   });

   msg.data.forEach(receiveVideo);
}

function leaveRoom() {
   sendMessage({
      id : 'leaveRoom'
   });

   for ( var key in participants) {
      participants[key].dispose();
   }

   document.getElementById('join').style.display = 'block';
   document.getElementById('room').style.display = 'none';

   ws.close();
}

function receiveVideo(sender) {
   var participant = new Participant(sender);
   participants[sender] = participant;
   var video = participant.getVideoElement();

   var options = {
      remoteVideo: video,
      onicecandidate: participant.onIceCandidate.bind(participant)
    }

   participant.rtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function (error) {
           if(error) {
              return console.error(error);
           }
           this.generateOffer (participant.offerToReceiveVideo.bind(participant));
   });;
}

function onParticipantLeft(request) {
   console.log('Participant ' + request.name + ' left');
   var participant = participants[request.name];
   participant.dispose();
   delete participants[request.name];
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

Media Elements metadata¶

This tutorial detects and draws faces present in the webcam video. It connects filters: KmsDetectFaces and the KmsShowFaces.

Java - Metadata¶

This tutorial detects and draws faces into the webcam video. The demo connects two filters, the KmsDetectFaces and the KmsShowFaces.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-metadata-example
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Note

This demo needs the kms-datachannelexample module installed in the media server. That module is available in the Kurento repositories, so it is possible to install it with:

sudo apt-get install kms-datachannelexample

Understanding this example¶

To implement this behavior we have to create a Media Pipeline composed by one WebRtcEndpoint and two filters KmsDetectFaces and KmsShowFaces. The first one detects faces into the image and it puts the info about the face (position and dimensions) into the buffer metadata. The second one reads the buffer metadata to find info about detected faces. If there is info about faces, the filter draws the faces into the image.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is MetadataApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost, listening in port 8888. If you reproduce this example, you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@EnableWebSocket
@SpringBootApplication
public class MetadataApp implements WebSocketConfigurer {

  static final String DEFAULT_APP_SERVER_URL = "https://localhost:8443";

  @Bean
  public MetadataHandler handler() {
    return new MetadataHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/metadata");
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(MetadataApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /metadata.

MetadataHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class MetadataHandler extends TextWebSocketHandler {

  private final Logger log = LoggerFactory.getLogger(MetadataHandler.class);
  private static final Gson gson = new GsonBuilder().create();

  private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<>();

  @Autowired
  private KurentoClient kurento;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    log.debug("Incoming message: {}", jsonMessage);

    switch (jsonMessage.get("id").getAsString()) {
      case "start":
        start(session, jsonMessage);
        break;
      case "stop": {
        UserSession user = users.remove(session.getId());
        if (user != null) {
          user.release();
        }
        break;
      }
      case "onIceCandidate": {
        JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

        UserSession user = users.get(session.getId());
        if (user != null) {
          IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
              jsonCandidate.get("sdpMid").getAsString(),
              jsonCandidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(candidate);
        }
        break;
      }
      default:
        sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
        break;
    }
  }

  private void start(final WebSocketSession session, JsonObject jsonMessage) {
    ...
  }

  private void sendError(WebSocketSession session, String message) {
   ...
  }
}

In the following snippet, we can see the start method. It handles the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint, KmsShowFaces and KmsDetectFaces) and make the connections among them. A startResponse message is sent back to the client with the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // User session
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      users.put(session.getId(), user);

      // ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.debug(e.getMessage());
          }
        }
      });

      // Media logic
      KmsShowFaces showFaces = new KmsShowFaces.Builder(pipeline).build();
      KmsDetectFaces detectFaces = new KmsDetectFaces.Builder(pipeline).build();

      webRtcEndpoint.connect(detectFaces);
      detectFaces.connect(showFaces);
      showFaces.connect(webRtcEndpoint);

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
        session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();

    } catch (Throwable t) {
      sendError(session, t.getMessage());
    }
}

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /metadata. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

 var ws = new WebSocket('wss://' + location.host + '/metadata');

 ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError("Error message from server: " + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error) {
            console.error("Error adding candidate: " + error);
            return;
         }
      });
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log("Starting video call ...")
   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   console.log("Creating WebRtcPeer and generating local sdp offer ...");

   var options = {
      localVideo : videoInput,
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function(error) {
            if (error) {
               return console.error(error);
            }
            webRtcPeer.generateOffer(onOffer);
         });
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error("Error generating the offer");
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log("Local candidate" + JSON.stringify(candidate));

   var message = {
      id : 'onIceCandidate',
      candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(I_CAN_STOP);
   console.log("SDP answer received from server. Processing ...");

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   console.log("Stopping video call ...");
   setState(I_CAN_START);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
         id : 'stop'
      }
      sendMessage(message);
   }
   hideSpinner(videoInput, videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

WebRTC Media Player¶

This tutorial reads a file from disk and plays the video to WebRTC.

Java - Player¶

This tutorial opens a URL and plays its content to WebRTC where it is possible to choose if it plays video and audio, only video or only audio.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-player
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

To implement this behavior we have to create a Media Pipeline composed by one PlayerEndpoint and one WebRtcEndpoint. The PlayerEnpdoint plays a video and WebRtcEndpoint shows it.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is PlayerApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@EnableWebSocket
@SpringBootApplication
public class PlayerApp implements WebSocketConfigurer {

  private static final String KMS_WS_URI_PROP = "kms.url";
  private static final String KMS_WS_URI_DEFAULT = "ws://localhost:8888/kurento";

  @Bean
  public PlayerHandler handler() {
    return new PlayerHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create(System.getProperty(KMS_WS_URI_PROP, KMS_WS_URI_DEFAULT));
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/player");
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(PlayerApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /player.

PlayerHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol, there are seven different kinds of incoming messages to the Server : start, stop, pause, resume, doSeek, getPosition and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class PlayerHandler extends TextWebSocketHandler {

  @Autowired
  private KurentoClient kurento;

  private final Logger log = LoggerFactory.getLogger(PlayerHandler.class);
  private final Gson gson = new GsonBuilder().create();
  private final ConcurrentHashMap<String, PlayerMediaPipeline> pipelines =
      new ConcurrentHashMap<>();

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);
    String sessionId = session.getId();
    log.debug("Incoming message {} from sessionId", jsonMessage, sessionId);

    try {
      switch (jsonMessage.get("id").getAsString()) {
        case "start":
          start(session, jsonMessage);
          break;
        case "stop":
          stop(sessionId);
          break;
        case "pause":
          pause(sessionId);
          break;
        case "resume":
          resume(session);
          break;
        case "doSeek":
          doSeek(session, jsonMessage);
          break;
        case "getPosition":
          getPosition(session);
          break;
        case "onIceCandidate":
          onIceCandidate(sessionId, jsonMessage);
          break;
        default:
          sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
          break;
      }
    } catch (Throwable t) {
      log.error("Exception handling message {} in sessionId {}", jsonMessage, sessionId, t);
      sendError(session, t.getMessage());
    }
  }


  private void start(final WebSocketSession session, JsonObject jsonMessage) {
    ...
  }

  private void pause(String sessionId) {
   ...
  }

  private void resume(final WebSocketSession session) {
  ...
  }

  private void doSeek(final WebSocketSession session, JsonObject jsonMessage) {
  ...
  }

  private void getPosition(final WebSocketSession session) {
  ...
  }

  private void stop(String sessionId) {
  ...
  }

  private void sendError(WebSocketSession session, String message) {
    ...
  }
}

In the following snippet, we can see the start method. It handles the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint and PlayerEndpoint) and makes the connections between them and plays the video. A startResponse message is sent back to the client with the SDP answer. When the MediaConnected event is received, info about the video is retrieved and sent back to the client in a videoInfo message.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
  final UserSession user = new UserSession(); MediaPipeline pipeline =
  kurento.createMediaPipeline(); user.setMediaPipeline(pipeline);
  WebRtcEndpoint webRtcEndpoint = new
  WebRtcEndpoint.Builder(pipeline).build();
  user.setWebRtcEndpoint(webRtcEndpoint); String videourl =
  jsonMessage.get("videourl").getAsString(); final PlayerEndpoint
  playerEndpoint = new PlayerEndpoint.Builder(pipeline, videourl).build();
  user.setPlayerEndpoint(playerEndpoint); users.put(session.getId(), user);

  playerEndpoint.connect(webRtcEndpoint);

  // 2. WebRtcEndpoint // ICE candidates
  webRtcEndpoint.addIceCandidateFoundListener(new
  EventListener<IceCandidateFoundEvent>() {
    @Override public void onEvent(IceCandidateFoundEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "iceCandidate"); response.add("candidate",
      JsonUtils.toJsonObject(event.getCandidate())); try {
        synchronized (session) {
          session.sendMessage(new
          TextMessage(response.toString()));
        }
      } catch (IOException e) {
        log.debug(e.getMessage());
      }
    }
  });

  String sdpOffer = jsonMessage.get("sdpOffer").getAsString(); String
  sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

  JsonObject response = new JsonObject(); response.addProperty("id",
  "startResponse"); response.addProperty("sdpAnswer", sdpAnswer);
  sendMessage(session, response.toString());

  webRtcEndpoint.addMediaStateChangedListener(new
  EventListener<MediaStateChangedEvent>() {
    @Override public void onEvent(MediaStateChangedEvent event) {

      if (event.getNewState() == MediaState.CONNECTED) {
        VideoInfo videoInfo = playerEndpoint.getVideoInfo();

        JsonObject response = new JsonObject();
        response.addProperty("id", "videoInfo");
        response.addProperty("isSeekable", videoInfo.getIsSeekable());
        response.addProperty("initSeekable", videoInfo.getSeekableInit());
        response.addProperty("endSeekable", videoInfo.getSeekableEnd());
        response.addProperty("videoDuration", videoInfo.getDuration());
        sendMessage(session, response.toString());
      }
    }
  });

  webRtcEndpoint.gatherCandidates();

  // 3. PlayEndpoint playerEndpoint.addErrorListener(new
  EventListener<ErrorEvent>() {
    @Override public void onEvent(ErrorEvent event) {
      log.info("ErrorEvent: {}", event.getDescription());
      sendPlayEnd(session);
    }
  });

  playerEndpoint.addEndOfStreamListener(new
  EventListener<EndOfStreamEvent>() {
    @Override public void onEvent(EndOfStreamEvent event) {
      log.info("EndOfStreamEvent: {}", event.getTimestamp());
      sendPlayEnd(session);
    }
  });

  playerEndpoint.play();
}

The pause method retrieves the user associated to the current session, and invokes the pause method on the PlayerEndpoint.

private void pause(String sessionId) {
  UserSession user = users.get(sessionId);

  if (user != null) {
    user.getPlayerEndpoint().pause();
  }
}

The resume method starts the PlayerEndpoint of the current user, sending back the information about the video, so the client side can refresh the stats.

private void resume(String sessionId) {
  UserSession user = users.get(session.getId());

  if (user != null) {
    user.getPlayerEndpoint().play(); VideoInfo videoInfo =
    user.getPlayerEndpoint().getVideoInfo();

    JsonObject response = new JsonObject(); response.addProperty("id",
    "videoInfo"); response.addProperty("isSeekable",
    videoInfo.getIsSeekable()); response.addProperty("initSeekable",
    videoInfo.getSeekableInit()); response.addProperty("endSeekable",
    videoInfo.getSeekableEnd()); response.addProperty("videoDuration",
    videoInfo.getDuration()); sendMessage(session, response.toString());
  }
}

The doSeek method gets the user by sessionId, and calls the method setPosition of the PlayerEndpoint with the new playing position. A seek message is sent back to the client if the seek fails.

private void doSeek(final WebSocketSession session, JsonObject jsonMessage) {
  UserSession user = users.get(session.getId());

  if (user != null) {
    try {
      user.getPlayerEndpoint().setPosition(jsonMessage.get("position").getAsLong());
    } catch (KurentoException e) {
      log.debug("The seek cannot be performed"); JsonObject response =
      new JsonObject(); response.addProperty("id", "seek");
      response.addProperty("message", "Seek failed"); sendMessage(session,
      response.toString());
    }
  }
}

The getPosition calls the method getPosition of the PlayerEndpoint of the current user. A position message is sent back to the client with the actual position of the video.

private void getPosition(final WebSocketSession session) {
  UserSession user = users.get(session.getId());

  if (user != null) {
    long position = user.getPlayerEndpoint().getPosition();

    JsonObject response = new JsonObject(); response.addProperty("id",
    "position"); response.addProperty("position", position);
    sendMessage(session, response.toString());
  }
}

The stop method is quite simple: it searches the user by sessionId and stops the PlayerEndpoint. Finally, it releases the media elements and removes the user from the list of active users.

private void stop(String sessionId) {
  UserSession user = users.remove(sessionId);

  if (user != null) {
    user.release();
  }
}

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
  try {
    JsonObject response = new JsonObject(); response.addProperty("id",
    "error"); response.addProperty("message", message);
    session.sendMessage(new TextMessage(response.toString()));
  } catch (IOException e) {
    log.error("Exception sending message", e);
  }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /player. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are seven incoming messages to client: startResponse, playEnd, error, videoInfo, seek, position and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/player');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'playEnd':
      playEnd();
      break;
     break;
   case 'videoInfo':
      showVideoData(parsedMessage);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error)
            return console.error('Error adding candidate: ' + error);
      });
      break;
   case 'seek':
      console.log (parsedMessage.message);
      break;
   case 'position':
      document.getElementById("videoPosition").value = parsedMessage.position;
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(video);

   var mode = $('input[name="mode"]:checked').val();
   console
         .log('Creating WebRtcPeer in " + mode + " mode and generating local sdp offer ...');

   // Video and audio by default
   var userMediaConstraints = {
      audio : true,
      video : true
   }

   if (mode == 'video-only') {
      userMediaConstraints.audio = false;
   } else if (mode == 'audio-only') {
      userMediaConstraints.video = false;
   }

   var options = {
      remoteVideo : video,
      mediaConstraints : userMediaConstraints,
      onicecandidate : onIceCandidate
   }

   console.info('User media constraints' + userMediaConstraints);

   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function(error) {
            if (error)
               return console.error(error);
            webRtcPeer.generateOffer(onOffer);
         });
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);

   var message = {
      id : 'start',
      sdpOffer : offerSdp,
      videourl : document.getElementById('videourl').value
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log('Local candidate' + JSON.stringify(candidate));

   var message = {
      id : 'onIceCandidate',
      candidate : candidate
   }
   sendMessage(message);
}

function startResponse(message) {
   setState(I_CAN_STOP);
   console.log('SDP answer received from server. Processing ...');

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function pause() {
   togglePause()
   console.log('Pausing video ...');
   var message = {
      id : 'pause'
   }
   sendMessage(message);
}

function resume() {
   togglePause()
   console.log('Resuming video ...');
   var message = {
      id : 'resume'
   }
   sendMessage(message);
}

function stop() {
   console.log('Stopping video ...');
   setState(I_CAN_START);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
         id : 'stop'
      }
      sendMessage(message);
   }
   hideSpinner(video);
}

function playEnd() {
   setState(I_CAN_START);
   hideSpinner(video);
}

function doSeek() {
   var message = {
     id : 'doSeek',
     position: document.getElementById("seekPosition").value
   }
   sendMessage(message);
 }

 function getPosition() {
   var message = {
     id : 'getPosition'
   }
   sendMessage(message);
 }

 function showVideoData(parsedMessage) {
   //Show video info
   isSeekable = parsedMessage.isSeekable;
   if (isSeekable) {
       document.getElementById('isSeekable').value = "true";
       enableButton('#doSeek', 'doSeek()');
     } else {
       document.getElementById('isSeekable').value = "false";
     }

     document.getElementById('initSeek').value = parsedMessage.initSeekable;
     document.getElementById('endSeek').value = parsedMessage.endSeekable;
     document.getElementById('duration').value = parsedMessage.videoDuration;

     enableButton('#getPosition', 'getPosition()');
 }

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

WebRTC outgoing Data Channels¶

This tutorial injects video into a QR filter and then sends the stream to WebRTC. QR detection events are delivered by means of WebRTC Data Channels, to be displayed in browser.

Java - Send DataChannel¶

This tutorial connects a player with a QR code detection filter and sends output to WebRTC. Code detection events are sent to browser using WebRTC datachannels.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-send-data-channel
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Note

This demo needs the kms-datachannelexample module installed in the media server. That module is available in the Kurento repositories, so it is possible to install it with:

sudo apt-get install kms-datachannelexample

Understanding this example¶

To implement this behavior we have to create a Media Pipeline composed by one PlayerEndpoint, one KmsSendData and one WebRtcEndpoint. The PlayerEnpdoint plays a video and it detects QR codes into the images. The info about detected codes is sent through data channels (KmsSendData) from the Kurento media server to the browser (WebRtcEndpoint). The browser shows the info in a text form.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is SendDataChannelApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@EnableWebSocket
@SpringBootApplication
public class SendDataChannelApp implements WebSocketConfigurer {

  static final String DEFAULT_APP_SERVER_URL = "https://localhost:8443";

  @Bean
  public SendDataChannelHandler handler() {
    return new SendDataChannelHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/senddatachannel");
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(SendDataChannelApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /senddatachannel.

SendDataChannelHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class SendDataChannelHandler extends TextWebSocketHandler {

  private final Logger log = LoggerFactory.getLogger(SendDataChannelHandler.class);
  private static final Gson gson = new GsonBuilder().create();

  private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<>();

  @Autowired
  private KurentoClient kurento;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    log.debug("Incoming message: {}", jsonMessage);

    switch (jsonMessage.get("id").getAsString()) {
      case "start":
        start(session, jsonMessage);
        break;
      case "stop": {
        UserSession user = users.remove(session.getId());
        if (user != null) {
          user.release();
        }
        break;
      }
      case "onIceCandidate": {
        JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

        UserSession user = users.get(session.getId());
        if (user != null) {
          IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
              jsonCandidate.get("sdpMid").getAsString(),
              jsonCandidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(candidate);
        }
        break;
      }
      default:
        sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
        break;
    }
  }

  private void start(final WebSocketSession session, JsonObject jsonMessage) {
    ...
  }

  private void sendError(WebSocketSession session, String message) {
    ...
  }
}

In the following snippet, we can see the start method. It handles the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint, KmsSendData and PlayerEndpoint) and make the connections among them. A startResponse message is sent back to the client with the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // User session
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).useDataChannels()
          .build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline,
          "http://files.openvidu.io/video/filter/barcodes.webm").build();
      user.setPlayer(player);
      users.put(session.getId(), user);

      // ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.debug(e.getMessage());
          }
        }
      });

      // Media logic
      KmsSendData kmsSendData = new KmsSendData.Builder(pipeline).build();

      player.connect(kmsSendData);
      kmsSendData.connect(webRtcEndpoint);

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
        session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();
      player.play();

    } catch (Throwable t) {
      sendError(session, t.getMessage());
    }
}

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /senddatachannel. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/senddatachannel');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError("Error message from server: " + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error) {
            console.error("Error adding candidate: " + error);
            return;
         }
      });
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log("Starting video call ...")
   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoOutput);

   var servers = null;
    var configuration = null;
    var peerConnection = new RTCPeerConnection(servers, configuration);

    console.log("Creating channel");
    var dataConstraints = null;

    channel = peerConnection.createDataChannel(getChannelName (), dataConstraints);

    channel.onmessage = onMessage;

    var dataChannelReceive = document.getElementById('dataChannelReceive');

    function onMessage (event) {
      console.log("Received data " + event["data"]);
      dataChannelReceive.value = event["data"];
    }

      console.log("Creating WebRtcPeer and generating local sdp offer ...");

   var options = {
      peerConnection: peerConnection,
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function(error) {
            if (error) {
               return console.error(error);
            }
            webRtcPeer.generateOffer(onOffer);
         });
}

function closeChannels(){

   if(channel){
     channel.close();
     $('#dataChannelSend').disabled = true;
     $('#send').attr('disabled', true);
     channel = null;
   }
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error("Error generating the offer");
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log("Local candidate" + JSON.stringify(candidate));

   var message = {
      id : 'onIceCandidate',
      candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(I_CAN_STOP);
   console.log("SDP answer received from server. Processing ...");

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   console.log("Stopping video call ...");
   setState(I_CAN_START);
   if (webRtcPeer) {
       closeChannels();

      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
         id : 'stop'
      }
      sendMessage(message);
   }
   hideSpinner(videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

WebRTC incoming Data Channel¶

This tutorial shows how text messages sent from browser can be delivered by Data Channels, to be displayed together with loopback video.

Java - Show DataChannel¶

This demo allows sending text from browser to the media server through data channels. That text will be shown in the loopback video.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-show-data-channel
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Note

This demo needs the kms-datachannelexample module installed in the media server. That module is available in the Kurento repositories, so it is possible to install it with:

sudo apt-get install kms-datachannelexample

Understanding this example¶

This tutorial creates a Media Pipeline consisting of media elements: WebRtcEndpoint and KmsSendData. Any text inserted in the textbox is sent from Kurento Media Server (KmsSendData) back to browser (WebRtcEndpoint) and shown with loopback video.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is ShowDataChannelApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it’s located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@EnableWebSocket
@SpringBootApplication
public class ShowDataChannelApp implements WebSocketConfigurer {

  static final String DEFAULT_APP_SERVER_URL = "https://localhost:8443";

  @Bean
  public ShowDataChannelHandler handler() {
    return new ShowDataChannelHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/showdatachannel");
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(ShowDataChannelApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /showdatachannel.

ShowDataChannelHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class ShowDataChannelHandler extends TextWebSocketHandler {

  private final Logger log = LoggerFactory.getLogger(ShowDataChannelHandler.class);
  private static final Gson gson = new GsonBuilder().create();

  private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<>();

  @Autowired
  private KurentoClient kurento;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    log.debug("Incoming message: {}", jsonMessage);

    switch (jsonMessage.get("id").getAsString()) {
      case "start":
        start(session, jsonMessage);
        break;
      case "stop": {
        UserSession user = users.remove(session.getId());
        if (user != null) {
          user.release();
        }
        break;
      }
      case "onIceCandidate": {
        JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

        UserSession user = users.get(session.getId());
        if (user != null) {
          IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
              jsonCandidate.get("sdpMid").getAsString(),
              jsonCandidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(candidate);
        }
        break;
      }
      default:
        sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
        break;
    }
  }

  private void start(final WebSocketSession session, JsonObject jsonMessage) {
    ...
  }

  private void sendError(WebSocketSession session, String message) {
    ...
  }
}

Following snippet shows method start, where ICE candidates are gathered and Media Pipeline and Media Elements (WebRtcEndpoint and KmsSendData) are created and connected. Message startResponse is sent back to client carrying the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // User session
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).useDataChannels()
          .build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      users.put(session.getId(), user);

      // ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.debug(e.getMessage());
          }
        }
      });

      // Media logic
      KmsShowData kmsShowData = new KmsShowData.Builder(pipeline).build();

      webRtcEndpoint.connect(kmsShowData);
      kmsShowData.connect(webRtcEndpoint);

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
        session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();

    } catch (Throwable t) {
      sendError(session, t.getMessage());
    }
 }

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /showdatachannel. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, error, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/showdatachannel');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'error':
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError("Error message from server: " + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error) {
            console.error("Error adding candidate: " + error);
            return;
         }
      });
      break;
   default:
      if (state == I_AM_STARTING) {
         setState(I_CAN_START);
      }
      onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log("Starting video call ...")
   // Disable start button
   setState(I_AM_STARTING);
   showSpinner(videoInput, videoOutput);

   var servers = null;
    var configuration = null;
    var peerConnection = new RTCPeerConnection(servers, configuration);

    console.log("Creating channel");
    var dataConstraints = null;

    channel = peerConnection.createDataChannel(getChannelName (), dataConstraints);

    channel.onopen = onSendChannelStateChange;
    channel.onclose = onSendChannelStateChange;

    function onSendChannelStateChange(){
        if(!channel) return;
        var readyState = channel.readyState;
        console.log("sencChannel state changed to " + readyState);
        if(readyState == 'open'){
          dataChannelSend.disabled = false;
          dataChannelSend.focus();
          $('#send').attr('disabled', false);
        } else {
          dataChannelSend.disabled = true;
          $('#send').attr('disabled', true);
        }
      }

    var sendButton = document.getElementById('send');
    var dataChannelSend = document.getElementById('dataChannelSend');

    sendButton.addEventListener("click", function(){
        var data = dataChannelSend.value;
        console.log("Send button pressed. Sending data " + data);
        channel.send(data);
        dataChannelSend.value = "";
      });

   console.log("Creating WebRtcPeer and generating local sdp offer ...");

   var options = {
      peerConnection: peerConnection,
      localVideo : videoInput,
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function(error) {
            if (error) {
               return console.error(error);
            }
            webRtcPeer.generateOffer(onOffer);
         });
}

function closeChannels(){

   if(channel){
     channel.close();
     $('#dataChannelSend').disabled = true;
     $('#send').attr('disabled', true);
     channel = null;
   }
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error("Error generating the offer");
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
      id : 'start',
      sdpOffer : offerSdp
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log("Local candidate" + JSON.stringify(candidate));

   var message = {
      id : 'onIceCandidate',
      candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(I_CAN_STOP);
   console.log("SDP answer received from server. Processing ...");

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   console.log("Stopping video call ...");
   setState(I_CAN_START);
   if (webRtcPeer) {
       closeChannels();

      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
         id : 'stop'
      }
      sendMessage(message);
   }
   hideSpinner(videoInput, videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript - Hello World with Data Channels¶

This web application extends the Hello World Tutorial, adding media processing to the basic WebRTC loopback and allowing send text from browser to the media server through data channels.

For the impatient: running this example¶

You’ll need to install Kurento Media Server before running this example. Read installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-hello-world-data-channel
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the parameter ws_uri to the URL, as follows:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Note

This demo needs the kms-datachannelexample module installed in the media server. That module is available in the Kurento repositories, so it is possible to install it with:

sudo apt-get install kms-datachannelexample

Understanding this example¶

The logic of the application is quite simple: the local stream is sent to Kurento Media Server, which returns it back to the client without modifications. To implement this behavior we need to create a Media Pipeline with a single Media Element, i.e. of type WebRtcEndpoint, which holds the capability of exchanging full-duplex (bidirectional) WebRTC media flows. It is important to set value of property useDataChannels to true during WebRtcEndpoint creation. This media element is connected to itself in order to deliver back received Media.

The application creates a channel between PeerConnection and WebRtcEndpoint used for message delivery.

Complete source code of this demo can be found in GitHub.

JavaScript Logic¶

This demo follows a Single Page Application architecture (SPA). The interface is the following HTML page: index.html. This web page links two Kurento JavaScript libraries:

kurento-client.js : Implementation of the Kurento JavaScript Client.
kurento-utils.js : Kurento utility library aimed to simplify the WebRTC management in the browser.

In addition, these two JavaScript libraries are also required:

Bootstrap : Web framework for developing responsive web sites.
jquery.js : Cross-platform JavaScript library designed to simplify the client-side scripting of HTML.
adapter.js : WebRTC JavaScript utility library maintained by Google that abstracts away browser differences.
ekko-lightbox : Module for Bootstrap to open modal images, videos, and galleries.
demo-console : Custom JavaScript console.

The specific logic of this demo is coded in the following JavaScript page: index.js. In this file, there is a function which is called when the green button labeled as Start in the GUI is clicked.

var startButton = document.getElementById("start");

startButton.addEventListener("click", function() {
   var options = {
      peerConnection: peerConnection,
      localVideo: videoInput,
      remoteVideo: videoOutput
    };

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)
      this.generateOffer(onOffer)
   });

   [...]
}

The function WebRtcPeer.WebRtcPeerSendrecv abstracts the WebRTC internal details (i.e. PeerConnection and getUserStream) and makes possible to start a full-duplex WebRTC communication, using the HTML video tag with id videoInput to show the video camera (local stream) and the video tag videoOutput to show the remote stream provided by the Kurento Media Server.

Inside this function, a call to generateOffer is performed. This function accepts a callback in which the SDP offer is received. In this callback we create an instance of the KurentoClient class that will manage communications with the Kurento Media Server. So, we need to provide the URI of its WebSocket endpoint. In this example, we assume it’s listening in port 8433 at the same host than the HTTP serving the application.

[...]

var args = getopts(location.search,
{
  default:
  {
    ws_uri: 'wss://' + location.hostname + ':8433/kurento',
    ice_servers: undefined
  }
});

[...]

kurentoClient(args.ws_uri, function(error, client){
  [...]
};

Once we have an instance of kurentoClient, the following step is to create a Media Pipeline, as follows:

client.create("MediaPipeline", function(error, _pipeline){
   [...]
});

If everything works correctly, we have an instance of a media pipeline (variable pipeline in this example). With this instance, we are able to create Media Elements. In this example we just need a WebRtcEndpoint with useDataChannels property as true. Then, this media elements is connected itself:

pipeline.create("WebRtcEndpoint", {useDataChannels: true}, function(error, webRtc){
   if(error) return onError(error);

   setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

   webRtc.processOffer(sdpOffer, function(error, sdpAnswer){
     if(error) return onError(error);

     webRtc.gatherCandidates(onError);

     webRtcPeer.processAnswer(sdpAnswer, onError);
   });

   webRtc.connect(webRtc, function(error){
     if(error) return onError(error);

     console.log("Loopback established");
   });
 });

In the following snippet, we can see how to create the channel and the send method of one channel.

var dataConstraints = null;
var channel = peerConnection.createDataChannel(getChannelName (), dataConstraints);
...

sendButton.addEventListener("click", function(){
    ...
    channel.send(data);
    ...
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

Demo dependencies are defined in file bower.json. They are managed using Bower.

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at Bower.

WebRTC recording¶

This tutorial has two parts:

A WebRTC loopback records the stream to disk.
The stream is played back.

Users can choose which type of media to send and record: audio, video or both.

Java - Recorder¶

This web application extends Hello World adding recording capabilities.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-hello-world-recording
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

In the first part of this tutorial, the local stream is sent to the media server, which in turn sends it back to the client, while recording it at the same time. In order to implement this behavior, we need to create a Media Pipeline consisting on a WebRtcEndpoint and a RecorderEnpoint.

The second part of this demo shows how to play recorded media. To achieve this, we need to create a Media Pipeline composed by a WebRtcEndpoint and a PlayerEndpoint. The uri property of the player is the uri of the recorded file.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in depth the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is HelloWorldRecApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@SpringBootApplication
@EnableWebSocket
public class HelloWorldRecApp implements WebSocketConfigurer {

  @Bean
  public HelloWorldRecHandler handler() {
    return new HelloWorldRecHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/recording");
  }

  @Bean
  public UserRegistry registry() {
    return new UserRegistry();
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(HelloWorldRecApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /recording.

HelloWorldRecHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop , play and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class HelloWorldRecHandler extends TextWebSocketHandler {

  private static final String RECORDER_FILE_PATH = "file:///tmp/HelloWorldRecorded.webm";

  private final Logger log = LoggerFactory.getLogger(HelloWorldRecHandler.class);
  private static final Gson gson = new GsonBuilder().create();

  @Autowired
  private UserRegistry registry;

  @Autowired
  private KurentoClient kurento;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    log.debug("Incoming message: {}", jsonMessage);

    UserSession user = registry.getBySession(session);
    if (user != null) {
      log.debug("Incoming message from user '{}': {}", user.getId(), jsonMessage);
    } else {
      log.debug("Incoming message from new user: {}", jsonMessage);
    }

    switch (jsonMessage.get("id").getAsString()) {
      case "start":
        start(session, jsonMessage);
        break;
      case "stop":
      case "stopPlay":
        if (user != null) {
          user.release();
        }
        break;
      case "play":
        play(user, session, jsonMessage);
        break;
      case "onIceCandidate": {
        JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

        if (user != null) {
          IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
              jsonCandidate.get("sdpMid").getAsString(),
              jsonCandidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(candidate);
        }
        break;
      }
      default:
        sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
        break;
    }
  }

  private void start(final WebSocketSession session, JsonObject jsonMessage) {
   ...
  }


  private void play(UserSession user, final WebSocketSession session, JsonObject jsonMessage) {
   ...
  }

  private void sendError(WebSocketSession session, String message) {
   ...
  }
}

In the following snippet, we can see the start method. It handles the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint and RecorderEndpoint) and make the connections among them. A startResponse message is sent back to the client with the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
    try {

      // 1. Media logic (webRtcEndpoint in loopback)
      MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      webRtcEndpoint.connect(webRtcEndpoint);

      MediaProfileSpecType profile = getMediaProfileFromMessage(jsonMessage);

      RecorderEndpoint recorder = new RecorderEndpoint.Builder(pipeline, RECORDER_FILE_PATH)
          .withMediaProfile(profile).build();

      connectAccordingToProfile(webRtcEndpoint, recorder, profile);

      // 2. Store user session
      UserSession user = new UserSession(session);
      user.setMediaPipeline(pipeline);
      user.setWebRtcEndpoint(webRtcEndpoint);
      registry.register(user);

      // 3. SDP negotiation
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      // 4. Gather ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.error(e.getMessage());
          }
        }
      });

      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (user) {
        session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();

      recorder.record();
    } catch (Throwable t) {
      log.error("Start error", t);
      sendError(session, t.getMessage());
    }
  }

The play method, creates a Media Pipeline with the Media Elements (WebRtcEndpoint and PlayerEndpoint) and make the connections among them. It will then send the recorded media to the client.

private void play(UserSession user, final WebSocketSession session, JsonObject jsonMessage) {
    try {

      // 1. Media logic
      final MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline, RECORDER_FILE_PATH).build();
      player.connect(webRtcEndpoint);

      // Player listeners
      player.addErrorListener(new EventListener<ErrorEvent>() {
        @Override
        public void onEvent(ErrorEvent event) {
          log.info("ErrorEvent for session '{}': {}", session.getId(), event.getDescription());
          sendPlayEnd(session, pipeline);
        }
      });
      player.addEndOfStreamListener(new EventListener<EndOfStreamEvent>() {
        @Override
        public void onEvent(EndOfStreamEvent event) {
          log.info("EndOfStreamEvent for session '{}'", session.getId());
          sendPlayEnd(session, pipeline);
        }
      });

      // 2. Store user session
      user.setMediaPipeline(pipeline);
      user.setWebRtcEndpoint(webRtcEndpoint);

      // 3. SDP negotiation
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "playResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      // 4. Gather ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.error(e.getMessage());
          }
        }
      });

      // 5. Play recorded stream
      player.play();

      synchronized (session) {
        session.sendMessage(new TextMessage(response.toString()));
      }

      webRtcEndpoint.gatherCandidates();
    } catch (Throwable t) {
      log.error("Play error", t);
      sendError(session, t.getMessage());
    }
  }

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /recording. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, playResponse, playEnd,``error``, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/recording');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'playResponse':
      playResponse(parsedMessage);
      break;
   case 'playEnd':
      playEnd();
      break;
   case 'error':
      setState(NO_CALL);
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error)
            return console.error('Error adding candidate: ' + error);
      });
      break;
   default:
      setState(NO_CALL);
   onError('Unrecognized message', parsedMessage);
   }
}

function start() {
   console.log('Starting video call ...');

   // Disable start button
   setState(DISABLED);
   showSpinner(videoInput, videoOutput);
   console.log('Creating WebRtcPeer and generating local sdp offer ...');

   var options = {
         localVideo : videoInput,
         remoteVideo : videoOutput,
         mediaConstraints : getConstraints(),
         onicecandidate : onIceCandidate
   }

   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
         function(error) {
      if (error)
         return console.error(error);
      webRtcPeer.generateOffer(onOffer);
   });
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
         id : 'start',
         sdpOffer : offerSdp,
         mode :  $('input[name="mode"]:checked').val()
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log('Local candidate' + JSON.stringify(candidate));

   var message = {
         id : 'onIceCandidate',
         candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(IN_CALL);
   console.log('SDP answer received from server. Processing ...');

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   var stopMessageId = (state == IN_CALL) ? 'stop' : 'stopPlay';
   console.log('Stopping video while in ' + state + '...');
   setState(POST_CALL);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
            id : stopMessageId
      }
      sendMessage(message);
   }
   hideSpinner(videoInput, videoOutput);
}

function play() {
   console.log("Starting to play recorded video...");

   // Disable start button
   setState(DISABLED);
   showSpinner(videoOutput);

   console.log('Creating WebRtcPeer and generating local sdp offer ...');

   var options = {
         remoteVideo : videoOutput,
         mediaConstraints : getConstraints(),
         onicecandidate : onIceCandidate
   }

   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function(error) {
      if (error)
         return console.error(error);
      webRtcPeer.generateOffer(onPlayOffer);
   });
}

function onPlayOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
         id : 'play',
         sdpOffer : offerSdp
   }
   sendMessage(message);
}

function getConstraints() {
   var mode = $('input[name="mode"]:checked').val();
   var constraints = {
         audio : true,
         video : true
   }

   if (mode == 'video-only') {
      constraints.audio = false;
   } else if (mode == 'audio-only') {
      constraints.video = false;
   }

   return constraints;
}


function playResponse(message) {
   setState(IN_PLAY);
   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function playEnd() {
   setState(POST_CALL);
   hideSpinner(videoInput, videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript - Recorder¶

This web application extends the Hello World Tutorial, adding recording capabilities.

For the impatient: running this example¶

You’ll need to install Kurento Media Server before running this example. Read installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You will need to download the source code form GitHub. There are two implementations of this tutorial, but they are functionally the same. It’s just the internal implementation that changes. After checking out the code, you can start the web server.

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-recorder
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-hello-world-recorder-generator
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the parameter ws_uri to the URL, as follows:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

In the first part of this demo, the local stream is sent to Kurento Media Server, which returns it back to the client and records to the same time. In order to implement this behavior we need to create a`Media Pipeline`:term: consisting of a WebRtcEndpoint and a RecorderEnpoint.

The second part of this demo shows how to play recorded media. To achieve this, we need to create a Media Pipeline composed by a WebRtcEndpoint and a PlayerEndpoint. The uri property of the player is the uri of the recorded file.

There are two implementations for this demo to be found in github:

Using callbacks.
Using yield.

Note

The snippets are based in demo with callbacks.

JavaScript Logic¶

This demo follows a Single Page Application architecture (SPA). The interface is the following HTML page: index.html. This web page links two Kurento JavaScript libraries:

kurento-client.js : Implementation of the Kurento JavaScript Client.
kurento-utils.js : Kurento utility library aimed to simplify the WebRTC management in the browser.

In addition, these two JavaScript libraries are also required:

Bootstrap : Web framework for developing responsive web sites.
jquery.js : Cross-platform JavaScript library designed to simplify the client-side scripting of HTML.
adapter.js : WebRTC JavaScript utility library maintained by Google that abstracts away browser differences.
ekko-lightbox : Module for Bootstrap to open modal images, videos, and galleries.
demo-console : Custom JavaScript console.

The specific logic of this demo is coded in the following JavaScript page: index.js. In this file, there is a function which is called when the green button, labeled as Start in the GUI, is clicked.

var startRecordButton = document.getElementById("start");

startRecordButton.addEventListener("click", startRecording);

function startRecording() {
   var options = {
     localVideo: videoInput,
     remoteVideo: videoOutput
   };

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)
      this.generateOffer(onOffer)
   });

   [...]
}

The function WebRtcPeer.WebRtcPeerSendrecv abstracts the WebRTC internal details (i.e. PeerConnection and getUserStream) and makes possible to start a full-duplex WebRTC communication, using the HTML video tag with id videoInput to show the video camera (local stream) and the video tag videoOutput to show the remote stream provided by the Kurento Media Server.

Inside this function, a call to generateOffer is performed. This function accepts a callback in which the SDP offer is received. In this callback we create an instance of the KurentoClient class that will manage communications with the Kurento Media Server. So, we need to provide the URI of its WebSocket endpoint. In this example, we assume it’s listening in port 8433 at the same host than the HTTP serving the application.

[...]

var args = getopts(location.search,
{
  default:
  {
    ws_uri: 'wss://' + location.hostname + ':8433/kurento',
    file_uri: 'file:///tmp/recorder_demo.webm', // file to be stored in media server
    ice_servers: undefined
  }
});

[...]

kurentoClient(args.ws_uri, function(error, client){
  [...]
};

Once we have an instance of kurentoClient, the following step is to create a Media Pipeline, as follows:

client.create("MediaPipeline", function(error, _pipeline){
   [...]
});

If everything works correctly, we have an instance of a media pipeline (variable pipeline in this example). With this instance, we are able to create Media Elements. In this example we just need a WebRtcEndpoint and a RecorderEndpoint. Then, these media elements are interconnected:

var elements =
   [
     {type: 'RecorderEndpoint', params: {uri : args.file_uri}},
     {type: 'WebRtcEndpoint', params: {}}
   ]

pipeline.create(elements, function(error, elements){
  if (error) return onError(error);

  var recorder = elements[0]
  var webRtc   = elements[1]

  setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

  webRtc.processOffer(offer, function(error, answer) {
    if (error) return onError(error);

    console.log("offer");

    webRtc.gatherCandidates(onError);
    webRtcPeer.processAnswer(answer);
  });

  client.connect(webRtc, webRtc, recorder, function(error) {
    if (error) return onError(error);

    console.log("Connected");

    recorder.record(function(error) {
      if (error) return onError(error);

      console.log("record");
    });
  });
});

When stop button is clicked, the recoder element stops to record, and all elements are released.

stopRecordButton.addEventListener("click", function(event){
    recorder.stop();
    pipeline.release();
    webRtcPeer.dispose();
    videoInput.src = "";
    videoOutput.src = "";

    hideSpinner(videoInput, videoOutput);

    var playButton = document.getElementById('play');
    playButton.addEventListener('click', startPlaying);
  })

In the second part, after play button is clicked, we have an instance of a media pipeline (variable pipeline in this example). With this instance, we are able to create Media Elements. In this example we just need a WebRtcEndpoint and a PlayerEndpoint with uri option like path where the media was recorded. Then, these media elements are interconnected:

var options = {uri : args.file_uri}

pipeline.create("PlayerEndpoint", options, function(error, player) {
  if (error) return onError(error);

  player.on('EndOfStream', function(event){
    pipeline.release();
    videoPlayer.src = "";

    hideSpinner(videoPlayer);
  });

  player.connect(webRtc, function(error) {
    if (error) return onError(error);

    player.play(function(error) {
      if (error) return onError(error);
      console.log("Playing ...");
    });
  });
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

Demo dependencies are located in file bower.json. Bower is used to collect them.

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at Bower.

WebRTC repository¶

This is similar to the recording tutorial, but using the repository to store metadata.

Java - Repository¶

This web application extends Hello World adding recording capabilities by means of the Kurento Repository.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

You need to have installed the Kurento Media Server before running this example. Read the installation guide for further information.

In addition, you also need the kurento-repository-server. This component is in charge of the storage and retrieval of the media. Please visit the Kurento Repository Server installation guide for further details.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-hello-world-repository/
git checkout 6.10.0
mvn -U clean spring-boot:run

Access the application connecting to the URL https://localhost:8443/ in a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. In addition, by default this demo is also suppossing that the Kurento Repository is up and running in the localhost. It can be changed by means of the property repository.uri. All in all, and due to the fact that we can use Maven to run the tutorial, you should execute the following command:

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento \
    -Drepository.uri=http://repository_host:repository_url

Understanding this example¶

On top of the recording capabilities from the base tutorial, this application creates a repository element to store media in that repository. Additionally, metadata about the recorded file can be also stored in the repository.

This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.

The following sections analyze in deep the server (Java) and client-side (JavaScript) code of this application. The complete source code can be found in GitHub.

Application Server Logic¶

This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.

Note

You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.

The main class of this demo is HelloWorldRecApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to the application. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost listening in port 8888. If you reproduce this example you’ll need to insert the specific location of your Kurento Media Server instance there.

Once the Kurento Client has been instantiated, you are ready for communicating with Kurento Media Server and controlling its multimedia capabilities.

@SpringBootApplication
@EnableWebSocket
public class HelloWorldRecApp implements WebSocketConfigurer {

  protected static final String DEFAULT_REPOSITORY_SERVER_URI = "http://localhost:7676";

  protected static final String REPOSITORY_SERVER_URI =
    System.getProperty("repository.uri", DEFAULT_REPOSITORY_SERVER_URI);

  @Bean
  public HelloWorldRecHandler handler() {
    return new HelloWorldRecHandler();
  }

  @Bean
  public KurentoClient kurentoClient() {
    return KurentoClient.create();
  }

  @Override
  public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
    registry.addHandler(handler(), "/repository");
  }

  @Bean
  public RepositoryClient repositoryServiceProvider() {
    return REPOSITORY_SERVER_URI.startsWith("file://") ? null
      : RepositoryClientProvider.create(REPOSITORY_SERVER_URI);
  }

  @Bean
  public UserRegistry registry() {
    return new UserRegistry();
  }

  public static void main(String[] args) throws Exception {
    new SpringApplication(HelloWorldRecApp.class).run(args);
  }
}

This web application follows a Single Page Application architecture (SPA), and uses a WebSocket to communicate client with application server by means of requests and responses. Specifically, the main app class implements the interface WebSocketConfigurer to register a WebSocketHanlder to process WebSocket requests in the path /repository.

HelloWorldRecHandler class implements TextWebSocketHandler to handle text WebSocket requests. The central piece of this class is the method handleTextMessage. This method implements the actions for requests, returning responses through the WebSocket. In other words, it implements the server part of the signaling protocol depicted in the previous sequence diagram.

In the designed protocol there are three different kinds of incoming messages to the Server : start, stop, stopPlay, play and onIceCandidates. These messages are treated in the switch clause, taking the proper steps in each case.

public class HelloWorldRecHandler extends TextWebSocketHandler {

  // slightly larger timeout
  private static final int REPOSITORY_DISCONNECT_TIMEOUT = 5500;

  private static final String RECORDING_EXT = ".webm";

  private final Logger log = LoggerFactory.getLogger(HelloWorldRecHandler.class);
  private static final SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss-S");
  private static final Gson gson = new GsonBuilder().create();

  @Autowired
  private UserRegistry registry;

  @Autowired
  private KurentoClient kurento;

  @Autowired
  private RepositoryClient repositoryClient;

  @Override
  public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);

    log.debug("Incoming message: {}", jsonMessage);

    UserSession user = registry.getBySession(session);
    if (user != null) {
      log.debug("Incoming message from user '{}': {}", user.getId(), jsonMessage);
    } else {
      log.debug("Incoming message from new user: {}", jsonMessage);
    }

    switch (jsonMessage.get("id").getAsString()) {
      case "start":
        start(session, jsonMessage);
        break;
      case "stop":
      case "stopPlay":
        if (user != null) {
          user.release();
        }
        break;
      case "play":
        play(user, session, jsonMessage);
        break;
      case "onIceCandidate": {
        JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();

        if (user != null) {
          IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
              jsonCandidate.get("sdpMid").getAsString(),
              jsonCandidate.get("sdpMLineIndex").getAsInt());
          user.addCandidate(candidate);
        }
        break;
      }
      default:
        sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
        break;
    }
  }

  private void start(final WebSocketSession session, JsonObject jsonMessage) {
   ...
  }

  private void play(UserSession user, final WebSocketSession session, JsonObject jsonMessage) {
    ...
  }

  private void sendError(WebSocketSession session, String message) {
    ...
  }
}

In the following snippet, we can see the start method. If a repository REST client or interface has been created, it will obtain a RepositoryItem from the remote service. This item contains an ID and a recording URI that will be used by the Kurento Media Server. The ID will be used after the recording ends in order to manage the stored media. If the client doesn’t exist, the recording will be performed to a local URI, on the same machine as the KMS. This method also deals with the ICE candidates gathering, creates a Media Pipeline, creates the Media Elements (WebRtcEndpoint and RecorderEndpoint) and makes the connections between them. A startResponse message is sent back to the client with the SDP answer.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // 0. Repository logic
      RepositoryItemRecorder repoItem = null;
      if (repositoryClient != null) {
        try {
          Map<String, String> metadata = Collections.emptyMap();
          repoItem = repositoryClient.createRepositoryItem(metadata);
        } catch (Exception e) {
          log.warn("Unable to create kurento repository items", e);
        }
      } else {
        String now = df.format(new Date());
        String filePath = HelloWorldRecApp.REPOSITORY_SERVER_URI + now + RECORDING_EXT;
        repoItem = new RepositoryItemRecorder();
        repoItem.setId(now);
        repoItem.setUrl(filePath);
      }
      log.info("Media will be recorded {}by KMS: id={} , url={}",
          (repositoryClient == null ? "locally" : ""), repoItem.getId(), repoItem.getUrl());

      // 1. Media logic (webRtcEndpoint in loopback)
      MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      webRtcEndpoint.connect(webRtcEndpoint);
      RecorderEndpoint recorder = new RecorderEndpoint.Builder(pipeline, repoItem.getUrl())
          .withMediaProfile(MediaProfileSpecType.WEBM).build();
      webRtcEndpoint.connect(recorder);

      // 2. Store user session
      UserSession user = new UserSession(session);
      user.setMediaPipeline(pipeline);
      user.setWebRtcEndpoint(webRtcEndpoint);
      user.setRepoItem(repoItem);
      registry.register(user);

      // 3. SDP negotiation
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      // 4. Gather ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.error(e.getMessage());
          }
        }
  }

The play method, creates a Media Pipeline with the Media Elements (WebRtcEndpoint and PlayerEndpoint) and make the connections between them. It will then send the recorded media to the client. The media can be served from the repository or directly from the disk. If the repository interface exists, it will try to connect to the remote service in order to obtain an URI from which the KMS will read the media streams. The inner workings of the repository restrict reading an item before it has been closed (after the upload finished). This will happen only when a certain number of seconds elapse after the last byte of media is uploaded by the KMS (safe-guard for gaps in the network communications).

private void play(UserSession user, final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // 0. Repository logic
      RepositoryItemPlayer itemPlayer = null;
      if (repositoryClient != null) {
        try {
          Date stopTimestamp = user.getStopTimestamp();
          if (stopTimestamp != null) {
            Date now = new Date();
            long diff = now.getTime() - stopTimestamp.getTime();
            if (diff >= 0 && diff < REPOSITORY_DISCONNECT_TIMEOUT) {
              log.info(
                  "Waiting for {}ms before requesting the repository read endpoint "
                      + "(requires {}ms before upload is considered terminated "
                      + "and only {}ms have passed)",
                  REPOSITORY_DISCONNECT_TIMEOUT - diff, REPOSITORY_DISCONNECT_TIMEOUT, diff);
              Thread.sleep(REPOSITORY_DISCONNECT_TIMEOUT - diff);
            }
          } else {
            log.warn("No stop timeout was found, repository endpoint might not be ready");
          }
          itemPlayer = repositoryClient.getReadEndpoint(user.getRepoItem().getId());
        } catch (Exception e) {
          log.warn("Unable to obtain kurento repository endpoint", e);
        }
      } else {
        itemPlayer = new RepositoryItemPlayer();
        itemPlayer.setId(user.getRepoItem().getId());
        itemPlayer.setUrl(user.getRepoItem().getUrl());
      }
      log.debug("Playing from {}: id={}, url={}",
          (repositoryClient == null ? "disk" : "repository"), itemPlayer.getId(),
          itemPlayer.getUrl());

      // 1. Media logic
      final MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline, itemPlayer.getUrl()).build();
      player.connect(webRtcEndpoint);

      // Player listeners
      player.addErrorListener(new EventListener<ErrorEvent>() {
        @Override
        public void onEvent(ErrorEvent event) {
          log.info("ErrorEvent for session '{}': {}", session.getId(), event.getDescription());
          sendPlayEnd(session, pipeline);
        }
      });
      player.addEndOfStreamListener(new EventListener<EndOfStreamEvent>() {
        @Override
        public void onEvent(EndOfStreamEvent event) {
          log.info("EndOfStreamEvent for session '{}'", session.getId());
          sendPlayEnd(session, pipeline);
        }
      });

      // 2. Store user session
      user.setMediaPipeline(pipeline);
      user.setWebRtcEndpoint(webRtcEndpoint);

      // 3. SDP negotiation
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      JsonObject response = new JsonObject();
      response.addProperty("id", "playResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      // 4. Gather ICE candidates
      webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
        @Override
        public void onEvent(IceCandidateFoundEvent event) {
          JsonObject response = new JsonObject();
          response.addProperty("id", "iceCandidate");
          response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
          try {
            synchronized (session) {
              session.sendMessage(new TextMessage(response.toString()));
            }
          } catch (IOException e) {
            log.error(e.getMessage());
          }
        }
  });

The sendError method is quite simple: it sends an error message to the client when an exception is caught in the server-side.

private void sendError(WebSocketSession session, String message) {
   try {
      JsonObject response = new JsonObject();
      response.addProperty("id", "error");
      response.addProperty("message", message);
      session.sendMessage(new TextMessage(response.toString()));
   } catch (IOException e) {
      log.error("Exception sending message", e);
   }
}

Client-Side Logic¶

Let’s move now to the client-side of the application. To call the previously created WebSocket service in the server-side, we use the JavaScript class WebSocket. We use a specific Kurento JavaScript library called kurento-utils.js to simplify the WebRTC interaction with the server. This library depends on adapter.js, which is a JavaScript WebRTC utility maintained by Google that abstracts away browser differences. Finally jquery.js is also needed in this application.

These libraries are linked in the index.html web page, and are used in the index.js. In the following snippet we can see the creation of the WebSocket (variable ws) in the path /repository. Then, the onmessage listener of the WebSocket is used to implement the JSON signaling protocol in the client-side. Notice that there are three incoming messages to client: startResponse, playResponse, playEnd,``error``, and iceCandidate. Convenient actions are taken to implement each step in the communication. For example, in functions start the function WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a WebRTC communication.

var ws = new WebSocket('wss://' + location.host + '/repository');

ws.onmessage = function(message) {
   var parsedMessage = JSON.parse(message.data);
   console.info('Received message: ' + message.data);

   switch (parsedMessage.id) {
   case 'startResponse':
      startResponse(parsedMessage);
      break;
   case 'playResponse':
      playResponse(parsedMessage);
      break;
   case 'playEnd':
      playEnd();
      break;
   case 'error':
      setState(NO_CALL);
      onError('Error message from server: ' + parsedMessage.message);
      break;
   case 'iceCandidate':
      webRtcPeer.addIceCandidate(parsedMessage.candidate, function(error) {
         if (error)
            return console.error('Error adding candidate: ' + error);
      });
      break;
   default:
      setState(NO_CALL);
   onError('Unrecognized message', parsedMessage);
   }
}

function start() {
console.log('Starting video call ...');

// Disable start button
setState(DISABLED);
showSpinner(videoInput, videoOutput);

console.log('Creating WebRtcPeer and generating local sdp offer ...');

var options = {
   localVideo : videoInput,
   remoteVideo : videoOutput,
   onicecandidate : onIceCandidate
}
webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
      function(error) {
         if (error)
            return console.error(error);
         webRtcPeer.generateOffer(onOffer);
      });
}

function onOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
         id : 'start',
         sdpOffer : offerSdp,
         mode :  $('input[name="mode"]:checked').val()
   }
   sendMessage(message);
}

function onError(error) {
   console.error(error);
}

function onIceCandidate(candidate) {
   console.log('Local candidate' + JSON.stringify(candidate));

   var message = {
         id : 'onIceCandidate',
         candidate : candidate
   };
   sendMessage(message);
}

function startResponse(message) {
   setState(IN_CALL);
   console.log('SDP answer received from server. Processing ...');

   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function stop() {
   var stopMessageId = (state == IN_CALL) ? 'stop' : 'stopPlay';
   console.log('Stopping video while in ' + state + '...');
   setState(POST_CALL);
   if (webRtcPeer) {
      webRtcPeer.dispose();
      webRtcPeer = null;

      var message = {
            id : stopMessageId
      }
      sendMessage(message);
   }
   hideSpinner(videoInput, videoOutput);
}

function play() {
   console.log("Starting to play recorded video...");

   // Disable start button
   setState(DISABLED);
   showSpinner(videoOutput);

   console.log('Creating WebRtcPeer and generating local sdp offer ...');

   var options = {
      remoteVideo : videoOutput,
      onicecandidate : onIceCandidate
   }
   webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerRecvonly(options,
         function(error) {
            if (error)
               return console.error(error);
            webRtcPeer.generateOffer(onPlayOffer);
         });
}

function onPlayOffer(error, offerSdp) {
   if (error)
      return console.error('Error generating the offer');
   console.info('Invoking SDP offer callback function ' + location.host);
   var message = {
         id : 'play',
         sdpOffer : offerSdp
   }
   sendMessage(message);
}

function playResponse(message) {
   setState(IN_PLAY);
   webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
         return console.error(error);
   });
}

function playEnd() {
   setState(POST_CALL);
   hideSpinner(videoInput, videoOutput);
}

function sendMessage(message) {
   var jsonMessage = JSON.stringify(message);
   console.log('Senging message: ' + jsonMessage);
   ws.send(jsonMessage);
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

WebRTC statistics¶

This tutorial implements a WebRTC loopback and shows how to collect WebRTC statistics.

JavaScript - Loopback stats¶

This web application extends the Hello World tutorial showing how statistics are collected.

For the impatient: running this example¶

You’ll need to install Kurento Media Server before running this example. Read installation guide for further information.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

Clone source code from GitHub and then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-loopback-stats
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Connect to URL https://localhost:8443/ using a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the parameter ws_uri to the URL, as follows:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

The logic of the application is quite simple: the local stream is sent to the Kurento Media Server, which returns it back to the client without modifications. To implement this behavior we need to create a Media Pipeline composed by the Media Element WebRtcEndpoint, which holds the capability of exchanging full-duplex (bidirectional) WebRTC media flows. This media element is connected to itself so any received media (from browser) is send back (to browser). Using method getStats the application shows all stats of element WebRtcEndpoint.

The complete source code of this demo can be found in GitHub.

JavaScript Logic¶

This demo follows a Single Page Application architecture (SPA). The interface is the following HTML page: index.html. This web page links two Kurento JavaScript libraries:

kurento-client.js : Implementation of the Kurento JavaScript Client.
kurento-utils.js : Kurento utility library aimed to simplify the WebRTC management in the browser.

In addition, these two JavaScript libraries are also required:

Bootstrap : Web framework for developing responsive web sites.
jquery.js : Cross-platform JavaScript library designed to simplify the client-side scripting of HTML.
adapter.js : WebRTC JavaScript utility library maintained by Google that abstracts away browser differences.
ekko-lightbox : Module for Bootstrap to open modal images, videos, and galleries.
demo-console : Custom JavaScript console.

The specific logic of this demo is coded in the following JavaScript page: index.js. In this file, there is a function which is called when the green button labeled as Start in the GUI is clicked.

var startButton = document.getElementById("start");

startButton.addEventListener("click", function() {
   var options = {
     localVideo: videoInput,
     remoteVideo: videoOutput
   };

   webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)
      this.generateOffer(onOffer)
   });

   [...]
}

The function WebRtcPeer.WebRtcPeerSendrecv hides internal details (i.e. PeerConnection and getUserStream) and makes possible to start a full-duplex WebRTC communication, using the HTML video tag with id videoInput to show the video camera (local stream) and the video tag videoOutput to show the remote stream provided by the Kurento Media Server.

Inside this function, a call to generateOffer is performed. This function accepts a callback in which the SDP offer is received. In this callback we create an instance of the KurentoClient class that will manage communications with the Kurento Media Server. So, we need to provide the URI of its WebSocket endpoint. In this example, we assume it’s listening in port 8433 at the same host than the HTTP serving the application.

[...]

var args = getopts(location.search,
{
  default:
  {
    ws_uri: 'wss://' + location.hostname + ':8433/kurento',
    ice_servers: undefined
  }
});

[...]

kurentoClient(args.ws_uri, function(error, client){
  [...]
};

Once we have an instance of kurentoClient, the following step is to create a Media Pipeline, as follows:

client.create("MediaPipeline", function(error, _pipeline){
   [...]
});

If everything works correctly, we have an instance of a media pipeline (variable pipeline in this example). With this instance, we are able to create Media Elements. In this example we just need a WebRtcEndpoint. Then, this media elements is connected itself:

pipeline.create("WebRtcEndpoint", function(error, webRtc) {
   if (error) return onError(error);

   webRtcEndpoint = webRtc;

   setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

   webRtc.processOffer(sdpOffer, function(error, sdpAnswer) {
     if (error) return onError(error);

     webRtc.gatherCandidates(onError);

     webRtcPeer.processAnswer(sdpAnswer, onError);
   });

   webRtc.connect(webRtc, function(error) {
     if (error) return onError(error);

     console.log("Loopback established");

     webRtcEndpoint.on('MediaStateChanged', function(event) {
       if (event.newState == "CONNECTED") {
         console.log("MediaState is CONNECTED ... printing stats...")
         activateStatsTimeout();
       }
     });
   });
});

In the following snippet, we can see getStats method. This method returns several statistic values of WebRtcEndpoint.

function getBrowserOutgoingVideoStats(webRtcPeer, callback) {
  var peerConnection = webRtcPeer.peerConnection;

  peerConnection.getStats(function(stats) {
    var results = stats.result();

    for (var i = 0; i < results.length; i++) {
      var res = results[i];
      if (res.type != 'ssrc') continue;

      //Publish it to be compliant with W3C stats draft
      var retVal = {
        timeStamp: res.timestamp,
        //StreamStats below
        associateStatsId: res.id,
        codecId: "--",
        firCount: res.stat('googFirsReceived'),
        isRemote: false,
        mediaTrackId: res.stat('googTrackId'),
        nackCount: res.stat('googNacksReceived'),
        pliCount: res.stat('googPlisReceived'),
        sliCount: 0,
        ssrc: res.stat('ssrc'),
        transportId: res.stat('transportId'),
        //Specific outbound below
        bytesSent: res.stat('bytesSent'),
        packetsSent: res.stat('packetsSent'),
        roundTripTime: res.stat('googRtt'),
        packetsLost: res.stat('packetsLost'),
        targetBitrate: "??",
        remb: "??"
      }
      return callback(null, retVal);
    }
    return callback("Error: could not find ssrc type on track stats", null);
  }, localVideoTrack);
}

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

Demo dependencies are located in file bower.json. Bower is used to collect them.

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
}

Note

We are in active development. You can find the latest version of Kurento JavaScript Client at Bower.

Features¶

This page summarizes the features that Kurento provides, with links to their own documentation page for the most important ones.

Kurento API, Clients, and Protocol¶

Kurento Media Server exposes all its functionality through an RPC API called Kurento API. This API can be queried directly by any kind of JSON-compatible client, but the recommended way to work with it is by means of a Kurento Client library; these are currently provided for Java, Browser Javascript, and Node.js.

If you prefer a different programming language, it’s possible to write a custom client library by following the specification of the Kurento Protocol, based on WebSocket and JSON-RPC.

The picture below shows how to use Kurento Clients in three scenarios:

Using the Kurento JavaScript Client directly in a compliant WebRTC browser.
Using the Kurento Java Client in a Java EE Application Server.
Using the Kurento JavaScript Client in a Node.js server.

Connection of Kurento Clients (Java and JavaScript) to Kuento Media Server

Complete examples for these three technologies is described in the Tutorials section.

The Kurento Client API is based on the concept of Media Elements. A Media Element holds a specific media capability. For example, the media element called WebRtcEndpoint holds the capability of sending and receiving WebRTC media streams; the media element called RecorderEndpoint has the capability of recording into the file system any media streams it receives; the FaceOverlayFilter detects faces on the exchanged video streams and adds a specific overlaid image on top of them, etc. Kurento exposes a rich toolbox of media elements as part of its APIs.

Some Media Elements provided out of the box by Kurento

To better understand theses concepts it is recommended to take a look to the sections Kurento API and Kurento Protocol. You can also take a look at the Reference Documentation of the API implementations that are currently provided: Kurento Client.

Kurento Modules¶

Kurento has been designed as a pluggable framework. Kurento Media Server uses several modules by default, named kms-core, kms-elements and kms-filters.

In addition, there are others built-in modules to enhance the capabilities provided by the Kurento Media Server. These modules are named kms-crowddetector, kms-pointerdetector, kms-chroma, and kms-platedetector.

Finally, Kurento Media Server can be expanded with new custom modules.

Kurento modules architecture. Kurento Media Server can be extended with built-it modules (crowddetector, pointerdetector, chroma, platedetector) and also with other custom modules.

For more information, read the section Kurento Modules.

RTP Streaming¶

Besides WebRTC connections, Kurento Media Server is able to manage standard RTP streams, allowing to connect an instance of KMS to a wide variety of devices.

There are two topics to note when dealing with RTP connections: the automatic congestion control algorithms that KMS implements (see Congestion Control / REMB), and the NAT traversal capabilities (see NAT Traversal).

Congestion Control / REMB¶

Kurento implements the Google Congestion Control algorithm, so it is able to generate and parse both abs-send-time RTP headers and REMB RTCP messages.

It is enabled by by passing the media-level attribute goog-remb in the SDP Offer. For example:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=video 5004 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=rtcp-fb:103 goog-remb
a=sendonly
a=ssrc:112233 cname:user@example.com

a=rtcp-fb is the RTCP Feedback capability attribute, as defined in RFC 4585.

KMS implements REMB propagation between the sender and receiver legs of a connection. This means that when KMS is used as a proxy between a video sender and one or more video receivers, the smallest REMB value from the receivers will be relayed to the sender. This allows the sender to choose a lower bitrate that will accommodate all of the receivers connected to KMS at the other side.

For more context about what is REMB and how it fits in the greater project of RMCAT, please read our Knowledge Base document: Congestion Control (RMCAT).

Configuration Guide¶

Kurento works by orchestrating a broad set of technologies that must be made to work together. Some of these technologies can accept different configuration parameters that Kurento makes available through several configuration files:

/etc/kurento/kurento.conf.json: The main configuration file. Provides settings for the behavior of Kurento Media Server itself.
/etc/kurento/modules/kurento/MediaElement.conf.ini: Generic parameters for all kinds of MediaElement.
/etc/kurento/modules/kurento/SdpEndpoint.conf.ini: Audio/video parameters for SdpEndpoint*s (i.e. *WebRtcEndpoint and RtpEndpoint).
/etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini: Specific parameters for WebRtcEndpoint.
/etc/kurento/modules/kurento/HttpEndpoint.conf.ini: Specific parameters for HttpEndpoint.
/etc/default/kurento-media-server: This file is loaded by the system’s service init files. Defines some environment variables, which have an effect on features such as the Debug Logging, or the Core Dump files that are generated when a crash happens.

Media Server¶

File: /etc/kurento/kurento.conf.json.

[TODO] Explain parameters.

MediaElement¶

File: /etc/kurento/modules/kurento/MediaElement.conf.ini.

[TODO] Explain parameters.

SdpEndpoint¶

File: /etc/kurento/modules/kurento/SdpEndpoint.conf.ini.

[TODO] Explain parameters.

WebRtcEndpoint¶

File: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.

[TODO] Explain parameters.

HttpEndpoint¶

File: /etc/kurento/modules/kurento/HttpEndpoint.conf.ini.

[TODO] Explain parameters.

Debug Logging¶

File: /etc/default/kurento-media-server.

Service Init¶

The package kurento-media-server provides a service file that integrates with the Ubuntu init system. This service file loads its user configuration from /etc/default/kurento-media-server, where the user is able to configure several features as needed.

Writing Kurento Applications ¶

Table of Contents

Writing Kurento Applications

Global Architecture ¶

Kurento can be used following the architectural principles of the web. That is, creating a multimedia application based on Kurento can be a similar experience to creating a web application using any of the popular web development frameworks.

At the highest abstraction level, web applications have an architecture comprised of three different layers:

Presentation layer (client side): Here we can find all the application code which is in charge of interacting with end users so that information is represented in a comprehensive way. This usually consists on HTML pages with JavaScript code.
Application logic (server side): This layer is in charge of implementing the specific functions executed by the application.
Service layer (server or Internet side): This layer provides capabilities used by the application logic such as databases, communications, security, etc. These services can be hosted in the same server as the application logic, or can be provided by external parties.

Following this parallelism, multimedia applications created using Kurento can also be implemented with the same architecture:

Presentation layer (client side): Is in charge of multimedia representation and multimedia capture. It is usually based on specific built-in capabilities of the client. For example, when creating a browser-based application, the presentation layer will use capabilities such as the <video> HTML tag or the WebRTC JavaScript APIs.
Application logic: This layer provides the specific multimedia logic. In other words, this layer is in charge of building the appropriate pipeline (by chaining the desired Media Elements) that the multimedia flows involved in the application will need to traverse.
Service layer: This layer provides the multimedia services that support the application logic such as media recording, media ciphering, etc. The Kurento Media Server (i.e. the specific Media Pipeline of Media Elements) is in charge of this layer.

The interesting aspect of this discussion is that, as happens with web development, Kurento applications can place the Presentation layer at the client side and the Service layer at the server side. However the Application logic, in both cases, can be located at either of the sides or even distributed between them. This idea is represented in the following picture:

Layered architecture of web and multimedia applications. Applications created using Kurento (right) can be similar to standard Web applications (left). Both types of applications may choose to place the application logic at the client or at the server code.

This means that Kurento developers can choose to include the code creating the specific media pipeline required by their applications at the client side (using a suitable Kurento Client or directly with Kurento Protocol) or can place it at the server side.

Both options are valid but each of them implies different development styles. Having said this, it is important to note that in the web developers usually tend to maintain client side code as simple as possible, bringing most of their application logic to the server. Reproducing this kind of development experience is the most usual way of using Kurento.

Note

In the following sections it is considered that all Kurento handling is done at the server side. Although this is the most common way of using Kurento, is important to note that all multimedia logic can be implemented at the client with the Kurento JavaScript Client.

Application Architecture ¶

Kurento, as most multimedia communication technologies out there, is built using two layers (called planes) to abstract key functions in all interactive communication systems:

Signaling Plane. The parts of the system in charge of the management of communications, that is, the modules that provides functions for media negotiation, QoS parametrization, call establishment, user registration, user presence, etc. are conceived as forming part of the Signaling Plane.
Media Plane. Functionalities such as media transport, media encoding/decoding and media processing make the Media Plane, which takes care of handling the media. The distinction comes from the telephony differentiation between the handling of voice and the handling of meta-information such as tone, billing, etc.

The following figure shows a conceptual representation of the high level architecture of Kurento:

Kurento Architecture. Kurento architecture follows the traditional separation between signaling and media planes.

The right side of the picture shows the application, which is in charge of the signaling plane and contains the business logic and connectors of the particular multimedia application being deployed. It can be build with any programming technology like Java, Node.js, PHP, Ruby, .NET, etc. The application can use mature technologies such as HTTP and SIP Servlets, Web Services, database connectors, messaging services, etc. Thanks to this, this plane provides access to the multimedia signaling protocols commonly used by end-clients such as SIP, RESTful and raw HTTP based formats, SOAP, RMI, CORBA or JMS. These signaling protocols are used by client side of applications to command the creation of media sessions and to negotiate their desired characteristics on their behalf. Hence, this is the part of the architecture, which is in contact with application developers and, for this reason, it needs to be designed pursuing simplicity and flexibility.

On the left side, we have the Kurento Media Server, which implements the media plane capabilities providing access to the low-level media features: media transport, media encoding/decoding, media transcoding, media mixing, media processing, etc. The Kurento Media Server must be capable of managing the multimedia streams with minimal latency and maximum throughput. Hence the Kurento Media Server must be optimized for efficiency.

Communicating client, server and Kurento ¶

As can be observed in the figure below, a Kurento application involves interactions among three main modules:

Client Application: Involves the native multimedia capabilities of the client platform plus the specific client-side application logic. It can use Kurento Clients designed for client platforms (for example, Kurento JavaScript Client).
Application Server: Involves an application server and the server-side application logic. It can use Kurento Clients designed to server platforms (for example, Kurento Java Client for Java EE and Kurento JavaScript Client for Node.js).
Kurento Media Server: Receives commands to create specific multimedia capabilities (i.e. specific pipelines adapted to the needs of the application).

The interactions maintained among these modules depend on the specifics of each application. However, in general, for most applications can be reduced to the following conceptual scheme:

Main interactions between architectural modules. These occur in two phases: negotiation and media exchange. Remark that the color of the different arrows and boxes is aligned with the architectural figures presented above. For example, orange arrows show exchanges belonging to the signaling plane, blue arrows show exchanges belonging to the Kurento Protocol, red boxes are associated to the Kurento Media Server, and green boxes with the application.

1. Media negotiation phase (signaling)¶

At a first stage, a client (a browser in a computer, a mobile application, etc.) issues a message to the application requesting some kind of multimedia capability. This message can be implemented with any protocol (HTTP, WebSocket, SIP, etc.). For instance, that request could ask for the visualization of a given video clip.

When the application receives the request, if appropriate, it will carry out the specific server side application logic, which can include Authentication, Authorization and Accounting (AAA), CDR generation, consuming some type of web service, etc.

After that, the application processes the request and, according to the specific instructions programmed by the developer, commands Kurento Media Server to instantiate the suitable Media Elements and to chain them in an appropriate Media Pipeline. Once the pipeline has been created successfully, Kurento Media Server responds accordingly and the application forwards the successful response to the client, showing it how and where the media service can be reached.

During the above mentioned steps no media data is really exchanged. All the interactions have the objective of negotiating the whats, hows, wheres and whens of the media exchange. For this reason, we call it the negotiation phase. Clearly, during this phase only signaling protocols are involved.

2. Media exchange phase ¶

After the signaling part, a new phase starts with the aim to produce the actual media exchange. The client addresses a request for the media to the Kurento Media Server using the information gathered during the negotiation phase.

Following with the video-clip visualization example mentioned above, the browser will send a GET request to the IP address and port of the Kurento Media Server where the clip can be obtained and, as a result, an HTTP reponse containing the media will be received.

Following the discussion with that simple example, one may wonder why such a complex scheme for just playing a video, when in most usual scenarios clients just send the request to the appropriate URL of the video without requiring any negotiation. The answer is straightforward. Kurento is designed for media applications involving complex media processing. For this reason, we need to establish a two-phase mechanism enabling a negotiation before the media exchange. The price to pay is that simple applications, such as one just downloading a video, also need to get through these phases. However, the advantage is that when creating more advanced services the same simple philosophy will hold. For example, if we want to add Augmented Reality or Computer Vision features to that video-clip, we just need to create the appropriate pipeline holding the desired Media Elements during the negotiation phase. After that, from the client perspective, the processed clip will be received as any other video.

Real time WebRTC applications with Kurento ¶

The client communicates its desired media capabilities through an SDP Offer/Answer negotiation. Hence, Kurento is able to instantiate the appropriate WebRTC endpoint, and to require it to generate an SDP Answer based on its own capabilities and on the SDP Offer. When the SDP Answer is obtained, it is given back to the client and the media exchange can be started. The interactions among the different modules are summarized in the following picture:

Interactions in a WebRTC session. During the negotiation phase, an SDP Offer is sent to KMS, requesting the capabilities of the client. As a result, Kurento Media Server generates an SDP Answer that can be used by the client for establishing the media exchange.

The application developer is able to create the desired pipeline during the negotiation phase, so that the real-time multimedia stream is processed accordingly to the application needs.

As an example, imagine that you want to create a WebRTC application recording the media received from the client and augmenting it so that if a human face is found, a hat will be rendered on top of it. This pipeline is schematically shown in the figure below, where we assume that the Filter element is capable of detecting the face and adding the hat to it.

Example pipeline for a WebRTC session. A WebRtcEndpoint is connected to a RecorderEndpoint storing the received media stream and to an Augmented Reality filter, which feeds its output media stream back to the client. As a result, the end user will receive its own image filtered (e.g. with a hat added onto her head) and the stream will be recorded and made available for further recovery into a repository (e.g. a file).

Media Plane ¶

From the application developer perspective, Media Elements are like Lego pieces: you just need to take the elements needed for an application and connect them, following the desired topology. In Kurento jargon, a graph of connected media elements is called a Media Pipeline. Hence, when creating a pipeline, developers need to determine the capabilities they want to use (the Media Elements) and the topology determining which Media Element provides media to which other Media Elements (the connectivity).

Simple Example of a Media Pipeline

The connectivity is controlled through the connect primitive, exposed on all Kurento Client APIs.

This primitive is always invoked in the element acting as source and takes as argument the sink element following this scheme:

sourceMediaElement.connect(sinkMediaElement)

For example, if you want to create an application recording WebRTC streams into the file system, you’ll need two media elements: WebRtcEndpoint and RecorderEndpoint. When a client connects to the application, you will need to instantiate these media elements making the stream received by the WebRtcEndpoint (which is capable of receiving WebRTC streams) to be fed to the RecorderEndpoint (which is capable of recording media streams into the file system). Finally you will need to connect them so that the stream received by the former is transferred into the later:

WebRtcEndpoint.connect(RecorderEndpoint)

To simplify the handling of WebRTC streams in the client-side, Kurento provides an utility called WebRtcPeer. Nevertheless, the standard WebRTC API (getUserMedia, RTCPeerConnection, and so on) can also be used to connect to WebRtcEndpoints. For further information please visit the Tutorials section.

Writing Kurento Modules ¶

Table of Contents

Writing Kurento Modules

[TODO REVIEW]

You can expand the Kurento Media Server developing your own modules. There are two flavors of Kurento modules:

Modules based on OpenCV. This kind of modules are recommended if you would like to develop a filter providing Computer Vision or Augmented Reality features.
Modules based on GStreamer. This kind of modules provide a generic entry point for media processing with the GStreamer framework. Such modules are more powerful but also they are more difficult to develop. Skills in GStreamer development are necessary.

The starting point to develop a filter is to create the filter structure. For this task, you can use the kurento-module-scaffold tool. This tool is distributed with the kurento-media-server-dev package. To install this tool run this command:

sudo apt-get install kurento-media-server-dev

The tool usage is different depending on the chosen flavor:

OpenCV module:

kurento-module-scaffold.sh <module_name> <output_directory> opencv_filter

Gstreamer module:

kurento-module-scaffold.sh <module_name> <output_directory>

The tool generates the folder tree, all the needed CmakeLists.txt files, and example files of Kurento module descriptor files (.kmd). These files contain the description of the modules, the constructor, the methods, the properties, the events and the complex types defined by the developer.

Once kmd files are completed it is time to generate the corresponding code. The tool kurento-module-creator generates glue code to server-side. Run this from the root directory:

cd build
cmake ..

The following sections detail how to create your module depending on the filter type you chose (OpenCV or GStreamer).

OpenCV module ¶

We have four files in src/server/implementation/:

ModuleNameImpl.cpp
ModuleNameImpl.hpp
ModuleNameOpenCVImpl.cpp
ModuleNameOpenCVImpl.hpp

The first two files should not be modified. The last two files will contain the logic of your module.

The file ModuleNameOpenCVImpl.cpp contains functions to deal with the methods and the parameters (you must implement the logic). Also, this file contains a function called process. This function will be called with each new frame, thus you must implement the logic of your filter inside it.

GStreamer module ¶

In this case, we have two directories inside the src/ folder:

The gst-plugins/ folder contains the implementation of your GStreamer Element (the kurento-module-scaffold generates a dummy filter).
Inside the server/objects/ folder you have two files:
```
ModuleNameImpl.cpp
ModuleNameImpl.hpp
```

In the file ModuleNameImpl.cpp you have to invoke the methods of your GStreamer element. The module logic will be implemented in the GStreamer Element.

For both kind of modules ¶

If you need extra compilation dependencies you can add compilation rules to the kurento-module-creator using the function generate_code in the CmakeLists.txt file, located in src/server/.

The following parameters are available:

SERVER_STUB_DESTINATION (required) The generated code that you may need to modify will be generated on the folder indicated by this parameter.
MODELS (required) This parameter receives the folders where the models (.kmd files) are located.
INTERFACE_LIB_EXTRA_SOURCES INTERFACE_LIB_EXTRA_HEADERS INTERFACE_LIB_EXTRA_INCLUDE_DIRS INTERFACE_LIB_EXTRA_LIBRARIES These parameters allow to add additional source code to the static library. Files included in INTERFACE_LIB_EXTRA_HEADERS will be installed in the system as headers for this library. All the parameters accept a list as input.
SERVER_IMPL_LIB_EXTRA_SOURCES SERVER_IMPL_LIB_EXTRA_HEADERS SERVER_IMPL_LIB_EXTRA_INCLUDE_DIRS SERVER_IMPL_LIB_EXTRA_LIBRARIES These parameters allow to add additional source code to the interface library. Files included in SERVER_IMPL_LIB_EXTRA_HEADERS will be installed in the system as headers for this library. All the parameters accept a list as input.
MODULE_EXTRA_INCLUDE_DIRS MODULE_EXTRA_LIBRARIES These parameters allow to add extra include directories and libraries to the module.
SERVER_IMPL_LIB_FIND_CMAKE_EXTRA_LIBRARIES This parameter receives a list of strings. Each string has this format: libname[ libversion range] (possible ranges can use symbols AND OR < <= > >= ^ and ~).
- ^ indicates a version compatible using Semantic Versioning.
- ~ Indicates a version similar, that can change just last indicated version character.

Once the module logic is implemented and the compilation process is finished, you need to install your module in your system. You can follow two different ways:

1. You can generate the Debian package (debuild -us -uc) and install it (dpkg -i). 2. You can define the following environment variables in the file /etc/default/kurento:

KURENTO_MODULES_PATH=<module_path>/build/src
GST_PLUGIN_PATH=<module_path>/build/src

Now, you need to generate code for Java or JavaScript to use your module from the client-side.

For Java, from the build directory you have to execute cmake .. -DGENERATE_JAVA_CLIENT_PROJECT=TRUE command, that generates a Java folder with client code. You can run make java_install and your module will be installed in your Maven local repository. To use the module in your Maven project, you have to add the dependency to the pom.xml file:
```
<dependency>
  <groupId>org.kurento.module</groupId>
  <artifactId>modulename</artifactId>
  <version>moduleversion</version>
</dependency>
```
For JavaScript, you should run cmake .. -DGENERATE_JS_CLIENT_PROJECT=TRUE. This command generates a js/ folder with client code. Now you can manually add the JavaScript library to use your module in your application. Alternatively, you can use Bower (for Browser JavaScript) or NPM (for Node.js). To do that, you should add your JavaScript module as a dependency in your bower.json or package.json file respectively, as follows:
```
"dependencies": {
  "modulename": "moduleversion"
}
```

Examples ¶

Simple examples for both kind of modules are available in GitHub:

There are a lot of examples showking how to define methods, parameters or events in all our public built-in modules:

Moreover, all our modules are developed using this methodology. For that reason you can take a look to our main modules:

Frequently Asked Questions¶

About NAT, ICE, STUN, TURN¶

What is NAT?¶

Network Address Translation (NAT) is a mechanism that hides from the public access the private IP addresses of machines inside a network. This NAT mechanism is typically found in all types of network devices, ranging from home routers to full-fledged corporate firewalls. In all cases the effect is the same: machines inside the NAT cannot be freely accessed from outside.

The effects of a NAT is very negative for WebRTC communications: machines inside the network will be able to send data to the outside, but they won’t be able to receive data from remote endpoints that are sitting outside the network. In order to allow for this need, NAT devices typically allow to configure NAT bindings to let data come in from the outside part of the network; creating these NAT bindings is what is called NAT traversal, also commonly referred as “opening ports”.

What is ICE?¶

Interactive Connectivity Establishment (ICE) is a protocol used for NAT traversal. It defines a technique that allows communication between two endpoints when one is inside a NAT and the other is outside of it. The net effect of the ICE process is that the NAT will be left with all needed ports open for communication, and both endpoints will have complete information about the IP address and ports where the other endpoint can be contacted.

ICE doesn’t work standalone: it needs to use a helper protocol called STUN.

What are STUN and TURN?¶

Session Traversal Utilities for NAT (STUN) is a protocol that complements ICE in the task of solving the NAT traversal issue. It can be used by any endpoints to determine the IP address and port allocated to it by a NAT. It can also be used to check connectivity between two endpoints, and as a keep-alive protocol to maintain NAT bindings. STUN works with many existing NATs, and does not require any special behavior from them.

Traversal Using Relays around NAT (TURN) is an extension of STUN, used where the NAT security policies are too strict and the needed NAT bindings cannot be successfully created. In these situations, it is necessary for the host to use the services of an intermediate node that acts as a communication relay.

Note

TURN is an extension of STUN. This means that you don’t need to configure a STUN server if you are already using a TURN server.

When is STUN needed?¶

STUN is needed for every endpoint behind a NAT. All NAT-ed peers need to open their own NAT ports, doing NAT traversal by using a STUN server that is outside of the NAT.

If you are installing Kurento in a NAT environment (eg. in any cloud provider), you need to use a STUN or TURN server, and configure KMS appropriately in /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini. Apart from that, you need to open all UDP ports in your security group, as STUN/TURN will use any port available from the whole 0-65535 range.

Similarly, all browser endpoints that are behind a NAT need to configure the STUN and/or TURN server details with the iceServers field of the RTCPeerConnection constructor.

Let’s see this with an example: The typical installation scenario for Kurento Media Server is to have a strict separation between Application Server and client. KMS and Application Server are running in a cloud machine without any NAT or port restriction on incoming connections, while a browser client runs from any (possibly restricted) network that forbids incoming connections on any port that hasn’t been “opened” in advance. The client may communicate with the Application Server for signaling purposes, but at the end of the day the bulk of the communication is done between the WebRTC engines of the browser and KMS.

In scenarios such as this one, the client is able to send data to KMS because its NAT will allow outgoing packets. However, KMS will not be able to send data to the client, because the client’s NAT is closed for incoming packets. This is solved by configuring the client to use some STUN server, then opening the appropriate ports in the NAT by using the STUN protocol. After this operation, the client is now able to receive audio/video streams from KMS:

This procedure is called ICE.

Note that you can also deploy KMS behind a NAT firewall, as long as KMS itself is also configured to open its own NAT ports by following the same procedure (again, with a STUN server that is outside of the NAT).

Note

TURN is an extension of STUN. This means that you don’t need to configure a STUN server if you are already using a TURN server.

How to install Coturn?¶

Coturn is a STUN server and (optionally) a TURN relay, supporting all features required for the ICE protocol and allowing to establish WebRTC connections between hosts that sit behind a NAT.

Coturn can be installed directly from the Ubuntu package repositories:

sudo apt-get install coturn

Edit the file /etc/turnserver.conf and configure the server according to your needs.

This basic configuration is a good first step; it will work for using Coturn with Kurento Media Server for WebRTC streams:
```
# TURN server public address, if Coturn is behind NAT.
# It must be an IP address, not a domain name.
external-ip=<CoturnPublicIpAddress>

# TURN server lower and upper bounds of the UDP relay endpoints.
# Default: 49152, 65535.
#min-port=49152
#max-port=65535

# Uncomment to run server in 'normal' 'moderate' verbose mode.
# By default the verbose mode is off.
#verbose

# Use fingerprints in the TURN messages.
fingerprint

# Use long-term credential mechanism.
lt-cred-mech

# 'Static' user accounts for long-term credentials mechanism.
user=<TurnUser>:<TurnPassword>

# Realm used for the long-term credentials mechanism.
realm=kurento.org

# Set the log file name.
# The log file can be reset sending a SIGHUP signal to the turnserver process.
log-file=/var/log/turnserver/turnserver.log

# Disable log file rollover and use log file name as-is.
simple-log
```
- The external-ip is necessary in cloud providers which use internal NATs, such as Amazon EC2. Write in <CoturnPublicIpAddress> your server’s public IPv4 address, such as 111.222.333.444. It must be an IP address, not a domain name.
- The options fingerprint, lt-cred-mech, and realm are needed for WebRTC.
- The user parameter is the most basic form of authorization to use the TURN relay capabilities. Write your desired user name and password in the fields <TurnUser> and <TurnPassword>.
- Other parameters can be tuned as needed. For more information, check the Coturn help pages:
  - https://github.com/coturn/coturn/wiki/turnserver
  - https://github.com/coturn/coturn/wiki/CoturnConfig
  - A fully commented example configuration file: https://raw.githubusercontent.com/coturn/coturn/master/examples/etc/turnserver.conf
Edit the file /etc/default/coturn and set
```
TURNSERVER_ENABLED=1
```
so the server starts automatically as a system service daemon.
Configure KMS and point it to where the server is listening for connections. Edit the file /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini and set either the STUN or the TURN parameters:
```
stunServerAddress=<CoturnPublicIpAddress>
stunServerPort=3478
```
```
turnURL=<TurnUser>:<TurnPassword>@<CoturnPublicIpAddress>:3478
```
If you only configure the STUN parameters in KMS, then the TURN relay capability of Coturn won’t be used. Of course, if you instead configure the whole TURN URL, then KMS will be able to use the Coturn server as a TURN relay when it needs to. Note that TURN is an extension of STUN, so if you configure TURN then there is no need to also configure the STUN details in KMS.

The following ports should be open in the firewall or your cloud machine’s Security Groups:
- 3478 TCP & UDP.
- 49152-65535 UDP: As per RFC 5766, these are the ports that the TURN server will use to exchange media. These ports can be changed using Coturn’s min-port and max-port parameters.
Note

Plain STUN doesn’t constrain the range of ports that might be used, so by default you should open all UDP ports. You can, however, restrict which ports will be used by Kurento Media Server, by editing the file /etc/kurento/modules/kurento/BaseRtpEndpoint.conf.ini. That allows to have a reduced set of ports open in your server.

Lastly, start the Coturn server and the media server:

sudo service coturn start
sudo service kurento-media-server restart

Note

Make sure to check your installation using this test application:

https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/

How To …¶

Know how many Media Pipelines do I need for my Application?¶

Media Elements can only communicate with each other when they are part of the same pipeline. Different MediaPipelines in the server are independent do not share audio, video, data or events.

A good heuristic is that you will need one pipeline per each set of communicating partners in a channel, and one Endpoint in this pipeline per audio/video streams reaching a partner.

Know how many Endpoints do I need?¶

Your application will need to create an Endpoint for each media stream flowing to (or from) the pipeline. As we said in the previous answer, each set of communicating partners in a channel will be in the same Media Pipeline, and each of them will use one or more Endpoints. They could use more than one if they are recording or reproducing several streams.

Know to what client a given WebRtcEndPoint belongs or where is it coming from?¶

Kurento API currently offers no way to get application attributes stored in a Media Element. However, the application developer can maintain a hashmap or equivalent data structure mapping the WebRtcEndpoint internal Id (which is a string) to whatever application information is desired.

Why do I get the error …¶

“Cannot create gstreamer element”?¶

This is a typical error which happens when you update Kurento Media Server from version 4 to 5. The problem is related to the GStreamer dependency version. The solution is the following:

sudo apt-get remove kurento*
sudo apt-get autoremove
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install kurento-media-server

Troubleshooting Issues ¶

If you are facing an issue with Kurento Media Server, follow this basic check list:

Step 1. Test with the latest version of Kurento Media Server: 6.10.0. Follow the installation instructions here: Installation Guide.
Step 2: If the problem still happens in the latest version, and the Kurento developers are already tracking progress for a solution in a bug report or a support contract, you may test the latest (unreleased) changes by installing a nightly version of KMS: Installing Nightly Builds.
Step 3: When your issue exists in both the latest and nightly versions, try resorting to the Open-Source Community. Kurento users might be having the same issue and maybe you can find help in there.
Step 4: If you want full attention from the Kurento team, get in contact with us to request Commercial Support.

This document will try to outline several bits of knowledge that can prove very useful when studying a failure or error in KMS.

Table of Contents

Media Server ¶

Media Server crashed ¶

If the Media Server crashes, it will write an stack trace into the file /var/log/kurento-media-server/errors.log. Also, in typical Ubuntu systems, the Linux Kernel will generate a crash core dump in /var/crash/ (although your system might be configured to generate them into a different directory).

However, these files won’t contain much useful information if the relevant debug symbols are not installed. Before filing a bug report, make sure to run your breaking test case with all debugging packages already installed:

PACKAGES=(
  # Third-party libraries
  libglib2.0-0-dbg
  libssl1.0.0-dbg

  # Kurento forked libraries
  gstreamer1.5-plugins-base-dbg
  gstreamer1.5-plugins-good-dbg
  gstreamer1.5-plugins-ugly-dbg
  gstreamer1.5-plugins-bad-dbg
  gstreamer1.5-libav-dbg
  libgstreamer1.5-0-dbg
  libnice10-dbgsym
  libsrtp1-dbg
  openwebrtc-gst-plugins-dbg
  kmsjsoncpp-dbg

  # KMS main components
  kms-jsonrpc-dbg
  kms-core-dbg
  kms-elements-dbg
  kms-filters-dbg
  kurento-media-server-dbg

  # KMS extra modules
  kms-chroma-dbg
  kms-crowddetector-dbg
  kms-platedetector-dbg
  kms-pointerdetector-dbg
)

sudo apt-get update
sudo apt-get install "${PACKAGES[@]}"

For example, see the difference between the same stack trace, as generated before installing the debug symbols, and after installing them. Don’t send a stack trace that looks like the first one in this example:

# ==== NOT USEFUL: WITHOUT debugging symbols ====
$ cat /var/log/kurento-media-server/errors.log
Segmentation fault (thread 139667051341568, pid 14132)
Stack trace:
[kurento::MediaElementImpl::mediaFlowInStateChange(int, char*, KmsElementPadType)]
/usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0x1025E0
[g_signal_emit]
/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0:0x2B08F
[check_if_flow_media]
/usr/lib/x86_64-linux-gnu/libkmsgstcommons.so.6:0x1F9E4
[g_hook_list_marshal]
/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x3A904

# ==== USEFUL: WITH debugging symbols ====
$ cat /var/log/kurento-media-server/errors.log
Segmentation fault (thread 140672899761920, pid 15217)
Stack trace:
[kurento::MediaElementImpl::mediaFlowInStateChange(int, char*, KmsElementPadType)]
/home/kurento/kms-omni-build/kms-core/src/server/implementation/objects/MediaElementImpl.cpp:479
[g_signal_emit]
/build/glib2.0-prJhLS/glib2.0-2.48.2/./gobject/gsignal.c:3443
[cb_buffer_received]
/home/kurento/kms-omni-build/kms-core/src/gst-plugins/commons/kmselement.c:578
[g_hook_list_marshal]
/build/glib2.0-prJhLS/glib2.0-2.48.2/./glib/ghook.c:673

The second stack trace is much more helpful, because it indicates the exact file names and line numbers where the crash happened. With these, a developer will at least have a starting point where to start looking for any potential bug.

It’s important to note that stack traces, while helpful, are not a replacement for actually running the software under a debugger. Most crashes like this will need further investigation before they can be fixed.

CPU usage grows too high ¶

Kurento Media Pipelines can get pretty complex if your use case requires so, which would mean more processing power is required to run them; however, even for the simplest cases it’s possible that you find out unexpected spikes in CPU usage, which in extreme cases could end up crashing the server due to resource exhaustion in the machine.

Check these points in an attempt to find possible causes for the high CPU usage:

Currently Kurento has performance issues with source videos bigger or equal to 720p. 1080p is not recommended (although it might work but the Kurento team hasn’t done any factual analysis to prove it).
Source and destination video codecs must be compatible. This has always been a source of performance problems in WebRTC communications.
- For example, if some participants are using Firefox and talking in a room, they will probably negotiate VP8 codec with Kurento; then later someone enters with Safari, CPU usage explodes due to transcoding is now suddenly required, because Safari only supports H.264 (VP8 support was added only since Desktop Safari v68).
- Another example is you have some VP8 streams running nicely but then stream recording is enabled with the MP4 recording profile, which uses H.264. Same story: video needs to be converted, and that uses a lot of CPU.
Also check if other processes are running in the same machine and using the CPU. For example, if Coturn is running and using a lot of resources because too many users end up connecting via Relay (TURN).

Of these, video transcoding is the main user of CPU cycles, because encoding video is a computationally expensive operation. As mentioned earlier, keep an eye on the TRANSCODING events sent from Kurento to your Application Server, or alternatively look for TRANSCODING ACTIVE messages in the media server logs.

If you see that transcoding is active at some point, you may get a bit more information about why, by enabling this line:

export GST_DEBUG="${GST_DEBUG:-3},Kurento*:5,agnosticbin*:5"

in your daemon settings file, /etc/default/kurento-media-server.

Then look for these messages in the media server log output:

Upstream provided caps: (caps)
Downstream wanted caps: (caps)
Find TreeBin with wanted caps: (caps)

Which will end up with either of these sets of messages:

If source codec is compatible with destination:
- TreeBin found! Use it for (audio|video)
- TRANSCODING INACTIVE for (audio|video)
If source codec is not compatible with destination:
- TreeBin not found! Transcoding required for (audio|video)
- TRANSCODING ACTIVE for (audio|video)

These messages can help understand what codec settings are being received by Kurento (“Upstream provided caps”) and what is being expected at the other side by the stream receiver (“Downstream wanted caps”).

Service init doesn’t work ¶

The package kurento-media-server provides a service file that integrates with the Ubuntu init system. This service file loads its user configuration from /etc/default/kurento-media-server, where the user is able to configure several features as needed.

In Ubuntu, log messages from init scripts are managed by systemd, and can be checked in to ways:

/var/log/syslog contains a copy of all init service messages. You can open it to see past messages, or follow it in real time with this command:
```
tail -f /var/log/syslog
```
You can query the status of the kurento-media-server service with this command:
```
systemctl status kurento-media-server.service
```

OpenH264 not found ¶

Problem: Installing and running KMS on a clean Ubuntu installation shows this message:

(gst-plugin-scanner:15): GStreamer-WARNING **: Failed to load plugin
'/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopenh264.so': libopenh264.so.0:
cannot open shared object file: No such file or directory

Also these conditions apply:

Packages openh264-gst-plugins-bad-1.5 and openh264 are already installed.
The file /usr/lib/x86_64-linux-gnu/libopenh264.so is a broken link to the non-existing file /usr/lib/x86_64-linux-gnu/libopenh264.so.0.

Reason: The package openh264 didn’t install correctly. This package is just a wrapper that needs Internet connectivity during its installation stage, to download a binary blob file from this URL: http://ciscobinary.openh264.org/libopenh264-1.4.0-linux64.so.bz2

If the machine is disconnected during the actual installation of this package, the download will fail silently with some error messages printed on the standard output, but the installation will succeed.

Solution: Ensure that the machine has access to the required URL, and try reinstalling the package:

sudo apt-get install --reinstall openh264

Missing audio or video streams ¶

If the Kurento Tutorials are showing an spinner, or your application is missing media streams, that’s a strong indication that the network topology requires using either a STUN or TURN server, to traverse through the NAT firewall of intermediate routers. Check STUN and TURN servers.

There are some KMS log messages that could indicate a bad configuration of STUN or TURN; these are useful to look for:

STUN server Port not found in config; using default value: 3478
STUN server IP address not found in config; NAT traversal requires either STUN or TURN server
TURN server IP address not found in config; NAT traversal requires either STUN or TURN server

If you see these messages, it’s a clear indication that STUN or TURN are not properly configured in KMS.

Low video quality ¶

You have several ways to override the default settings for variable bitrate:

Methods in org.kurento.client.BaseRtpEndpoint:
- setMinVideoRecvBandwidth() / setMaxVideoRecvBandwidth()
- setMinVideoSendBandwidth() / setMaxVideoSendBandwidth()
Methods in org.kurento.client.MediaElement:
- setMinOutputBitrate() / setMaxOutputBitrate()
  
  This setting is also configurable in /etc/kurento/modules/kurento/MediaElement.conf.ini

Application Server ¶

These are some common errors found to affect Kurento Application Servers:

KMS is not running ¶

Usually, the Kurento Client library is directed to connect with an instance of KMS that the developer expects will be running in some remote server. If there is no instance of KMS running at the provided URL, the Kurento Client library will raise an exception which the Application Server should catch and handle accordingly.

This is a sample of what the console output will look like, with the logging level set to DEBUG:

$ mvn -U clean spring-boot:run -Dkms.url=ws://localhost:8888/kurento
INFO org.kurento.tutorial.player.Application  : Starting Application on TEST with PID 16448
DEBUG o.kurento.client.internal.KmsUrlLoader  : Executing getKmsUrlLoad(b843d6f6-02dd-49b4-96b6-f2fd2e8b1c8d) in KmsUrlLoader
DEBUG o.kurento.client.internal.KmsUrlLoader  : Obtaining kmsUrl=ws://localhost:8888/kurento from config file or system property
DEBUG org.kurento.client.KurentoClient        : Connecting to kms in ws://localhost:8888/kurento
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket     : Creating JsonRPC NETTY Websocket client
DEBUG o.kurento.jsonrpc.client.JsonRpcClient  : Enabling heartbeat with an interval of 240000 ms
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket  : [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
WARN o.kurento.jsonrpc.client.JsonRpcClient   : [KurentoClient]  Error sending heartbeat to server. Exception: [KurentoClient]  Exception connecting to WebSocket server ws://localhost:8888/kurento
WARN o.kurento.jsonrpc.client.JsonRpcClient   : [KurentoClient]  Stopping heartbeat and closing client: failure during heartbeat mechanism
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket  : [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
DEBUG o.k.jsonrpc.internal.ws.PendingRequests : Sending error to all pending requests
WARN o.k.j.c.JsonRpcClientNettyWebSocket      : [KurentoClient]  Trying to close a JsonRpcClientNettyWebSocket with channel == null
WARN ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: Factory method 'kurentoClient' threw exception; nested exception is org.kurento.commons.exception.KurentoException: Exception connecting to KMS
ERROR o.s.boot.SpringApplication              : Application startup failed

As opposed to that, the console output for when a connection is successfully done with an instance of KMS should look similar to this sample:

$ mvn -U clean spring-boot:run -Dkms.url=ws://localhost:8888/kurento
INFO org.kurento.tutorial.player.Application : Starting Application on TEST with PID 21617
DEBUG o.kurento.client.internal.KmsUrlLoader : Executing getKmsUrlLoad(af479feb-dc49-4a45-8b1c-eedf8325c482) in KmsUrlLoader
DEBUG o.kurento.client.internal.KmsUrlLoader : Obtaining kmsUrl=ws://localhost:8888/kurento from config file or system property
DEBUG org.kurento.client.KurentoClient       : Connecting to kms in ws://localhost:8888/kurento
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : Creating JsonRPC NETTY Websocket client
DEBUG o.kurento.jsonrpc.client.JsonRpcClient : Enabling heartbeat with an interval of 240000 ms
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Connecting native client
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Creating new NioEventLoopGroup
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Initiating new Netty channel. Will create new handler too!
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : [KurentoClient]  channel active
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : [KurentoClient]  WebSocket Client connected!
INFO org.kurento.tutorial.player.Application : Started Application in 1.841 seconds (JVM running for 4.547)

KMS became unresponsive (due to network error or crash)¶

The Kurento Client library is programmed to start a retry-connect process whenever the other side of the RPC channel -ie. the KMS instance- becomes unresponsive. An error exception will raise, which again the Application Server should handle, and then the library will automatically start trying to reconnect with KMS.

This is how this process would look like. In this example, KMS was restarted so the Kurento Client library lost connectivity with KMS for a moment, but then it was able con reconnect and continue working normally:

INFO org.kurento.tutorial.player.Application  : Started Application in 1.841 seconds (JVM running for 4.547)

(... Application is running normally at this point)
(... Now, KMS becomes unresponsive)

INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  channel closed
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  JsonRpcWsClient disconnected from ws://localhost:8888/kurento because Channel closed.
DEBUG o.kurento.jsonrpc.client.JsonRpcClient : Disabling heartbeat. Interrupt if running is false
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  JsonRpcWsClient reconnecting to ws://localhost:8888/kurento.
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Connecting native client
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Closing previously existing channel when connecting native client
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : [KurentoClient]  Closing client
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Initiating new Netty channel. Will create new handler too!
WARN o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Trying to close a JsonRpcClientNettyWebSocket with channel == null
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : TryReconnectingForever=true
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : TryReconnectingMaxTime=0
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : maxTimeReconnecting=9223372036854775807
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : currentTime=1510773733903
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : Parar de reconectar=false
WARN o.k.j.c.AbstractJsonRpcClientWebSocket  : [KurentoClient]  Exception trying to reconnect to server ws://localhost:8888/kurento. Retrying in 5000 millis

org.kurento.jsonrpc.JsonRpcException: [KurentoClient]  Exception connecting to WebSocket server ws://localhost:8888/kurento
   at (...)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8888
   at (...)

(... Now, KMS becomes responsive again)

DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  JsonRpcWsClient reconnecting to ws://localhost:8888/kurento.
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Connecting native client
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Creating new NioEventLoopGroup
INFO o.k.j.c.JsonRpcClientNettyWebSocket     : [KurentoClient]  Initiating new Netty channel. Will create new handler too!
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : [KurentoClient]  channel active
DEBUG o.k.j.c.JsonRpcClientNettyWebSocket    : [KurentoClient]  WebSocket Client connected!
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  Req-> {"id":2,"method":"connect","jsonrpc":"2.0"}
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  <-Res {"id":2,"result":{"serverId":"1a3b4912-9f2e-45da-87d3-430fef44720f","sessionId":"f2fd16b7-07f6-44bd-960b-dd1eb84d9952"},"jsonrpc":"2.0"}
DEBUG o.k.j.c.AbstractJsonRpcClientWebSocket : [KurentoClient]  Reconnected to the same session in server ws://localhost:8888/kurento

(... At this point, the Kurento Client is connected again to KMS)

Node.js / NPM failures ¶

Kurento Client does not currently support Node v10 (LTS), you will have to use Node v8 or below.

“Expects at least 4 fields”¶

This message can manifest in multiple variations of what is essentially the same error:

DOMException: Failed to parse SessionDescription: m=video 0 UDP/TLS/RTP/SAVPF Expects at least 4 fields

OperationError (DOM Exception 34): Expects at least 4 fields

The reason for this is that Kurento hasn’t enabled support for the video codec H.264, but it needs to communicate with another peer which only supports H.264, such as the Safari browser. Thus, the SDP Offer/Answer negotiation rejects usage of the corresponding media stream, which is what is meant by m=video 0.

The solution is to ensure that both peers are able to find a match in their supported codecs. To enable H.264 support in Kurento, check these points:

The package openh264-gst-plugins-bad-1.5 must be installed in the system.
The package openh264 must be correctly installed. Specifically, the post-install script of this package requires Internet connectivity, because it downloads a codec binary blob from the Cisco servers. See OpenH264 not found.
The H.264 codec must be enabled in the corresponding Kurento settings file: /etc/kurento/modules/kurento/SdpEndpoint.conf.json. Ensure that the entry corresponding to this codec does exist and is not commented out. For example:
```
"videoCodecs": [
  { "name": "VP8/90000" },
  { "name": "H264/90000" }
]
```

Networking issues ¶

WebRTC connection is not established ¶

There is a multitude of possible reasons for a failed WebRTC connection, so you can start by following this checklist:

Deploy a properly configured STUN or TURN server. Coturn tends to work fine for this, and Kurento has some documentation about how to install and configure it: https://doc-kurento.readthedocs.io/en/latest/user/faq.html#install-coturn-turn-stun-server
Use this WebRTC sample page to test that your STUN/TURN server is working properly: https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
Configure your STUN/TURN server in Kurento, as explained here: https://doc-kurento.readthedocs.io/en/latest/user/installation.html#stun-and-turn-servers

Note

The features provided by TURN are a superset of those provided by STUN. This means that you don’t need to configure a STUN server if you are already using a TURN server.
Make sure your Kurento settings syntax is correct. For STUN servers, this would be:
```
stunServerAddress=<serverAddress>
stunServerPort=<serverPort>
```
For TURN servers, the correct line is like this:
```
turnURL=username:password@address:port
```
Check the debug logs of the STUN/TURN server. Maybe the server is failing and some useful error messages are being printed there.

Check the debug logs of KMS. In case of an incorrect configuration, you’ll find these messages:

INFO  STUN server Port not found in config; using default value: 3478
INFO  STUN server IP address not found in config; NAT traversal requires either STUN or TURN server
INFO  TURN server IP address not found in config; NAT traversal requires either STUN or TURN server

In case of having correctly configured a STUN server in KMS, the log messages will read like this:

INFO  Using STUN reflexive server IP: <IpAddress>
INFO  Using STUN reflexive server Port: <Port>

And in case of a TURN server:

INFO  Using TURN relay server: <user:password>@<IpAddress>:<Port>
INFO  TURN server info set: <user:password>@<IpAddress>:<Port>

Check that any SDP mangling you might be doing in your Application Server is being done correctly.

This is one of the most hard to catch examples we’ve seen in our mailing list:

> The problem was that our Socket.IO client did not correctly URL-Encode its JSON payload when xhr-polling, which resulted in all “plus” signs (‘+’) being changed into spaces (‘ ‘) on the server. This meant that the ufrag in the client’s SDP was invalid if it contained a plus sign! Only some of the connections failed because not all ufrags contain plus signs.

ICE fails with timeouts ¶

Problem:

You have configured a STUN/TURN server in a different machine than Kurento Media Server.
The ICE connection tests fail due to timeout on trying pairs.

Solution:

Make sure that all required UDP ports for media content are open on the sever; otherwise, not only the ICE process will fail, but also the video or audio streams themselves won’t be able to reach either server.

Multicast fails in Docker ¶

Problem:

Your Kurento Media Server is running in a Docker container.
MULTICAST streams playback fail with an error such as this one:
```
DEBUG rtspsrc gstrtspsrc.c:7553:gst_rtspsrc_handle_message:<source> timeout on UDP port
```
Note that in this example, to see this message you would need to enable DEBUG log level for the rtspsrc category; see Logging levels and components.

Solution:

For Multicast streaming to work properly, you need to disable Docker’s network namespacing and use --net host. Note that this gives the container direct access to the host interfaces, and you’ll need to connect through published ports to access others containers.

This is a limitation of Docker; you can follow the current status with this issue: https://github.com/moby/moby/issues/23659

If using Docker Compose, use network_mode: host such as this:

version: "3.7"
services:
  kms:
    image: kurento/kurento-media-server:6.9.0
    container_name: kms
    restart: always
    network_mode: host
    environment:
      - GST_DEBUG=2,Kurento*:5

References:

Element-specific info ¶

PlayerEndpoint ¶

RTSP broken video ¶

Some users have reported huge macro-blocks or straight out broken video frames when using a PlayerEndpoint to receive an RTSP stream containing H.264 video. A possible solution to fix this issue is to fine-tune the PlayerEndpoint’s networkCache parameter. It basically sets the buffer size (in milliseconds) that the underlying GStreamer decoding element will use to cache the stream.

There’s no science for that parameter, though. The perfect value depends on your network topology and efficiency, so you should proceed in a trial-and-error approach. For some situations, values lower than 100ms have worked fine; some users have reported that 10ms was required to make their specific camera work, others have seen good results with setting this parameter to 0ms.

RTSP Video stuttering ¶

The GStreamer element in charge of RTSP reception is rtspsrc, and this element contains an rtpjitterbuffer.

This jitter buffer gets full when network packets arrive faster than what Kurento is able to process. If this happens, then PlayerEndpoint will start dropping packets, which will show up as video stuttering on the output streams, while triggering a warning in Kurento logs:

WARNING  kmsutils  discont_detection_probe() <kmsagnosticbin0:sink>  Stream discontinuity detected on non-keyframe

You can check if this problem is affecting you by running with DEBUG logging level enabled for the rtpjitterbuffer component, and searching for a specific message:

export GST_DEBUG="${GST_DEBUG:-3},rtpjitterbuffer:5"
/usr/bin/kurento-media-server 2>&1 | grep -P 'rtpjitterbuffer.*(Received packet|Queue full)'

With this command, a new line will get printed for each single Received packet, plus an extra line will appear informing about Queue full whenever a packet is dropped.

There is not much you can fine tune in KMS to solve this problem; the most practical solution is to reduce the amount of data, mostly by decreasing either video resolution or video bitrate.

Kurento Media Server is known to work well receiving videos of up to 720p resolution (1280x720) at 30fps and around 2Mbps. If you are using values beyond those, there is a chance that KMS will be unable to process all incoming data on time, and this will cause that buffers fill up and frames get dropped. Try reducing the resolution of your input videos to see if this helps solving the issue.

RecorderEndpoint ¶

Zero-size video files ¶

If you are trying to generate a video recording, keep in mind that the endpoint will wait until all tracks (audio, video) start arriving.

Quoting from the Client documentation:

It is recommended to start recording only after media arrives, either to the endpoint that is the source of the media connected to the recorder, to the recorder itself, or both. Users may use the MediaFlowIn and MediaFlowOut events, and synchronize the recording with the moment media comes in. In any case, nothing will be stored in the file until the first media packets arrive.

Follow this checklist to see if any of these problems is preventing the RecorderEndpoint from working correctly:

The RecorderEndpoint is configured for both audio and video, but only video (or only audio) is being provided by the application.
Availability of audio/video devices at recorder client initialization, and just before starting the recording.
User is disconnecting existing hardware, or maybe connecting new hardware (usb webcams, mic, etc).
User is clicking “Deny” when asked to allow access to microphone/camera by the browser.
User is sleeping/hibernating the computer, and then possibly waking it up, while recording.
Check the browser information about the required media tracks, e.g. track.readyState.
Track user agents, ICE candidates, etc.

Browser ¶

Safari doesn’t work ¶

Apple Safari is a browser that follows some policies that are much more restrictive than those of other common browsers such as Google Chrome or Mozilla Firefox.

For some tips about how to ensure the best compatibility with Safari, check Apple Safari.

Support¶

If you are facing an issue with Kurento Media Server, follow this basic check list:

Step 1. Test with the latest version of Kurento Media Server: 6.10.0. Follow the installation instructions here: Installation Guide.
Step 2: If the problem still happens in the latest version, and the Kurento developers are already tracking progress for a solution in a bug report or a support contract, you may test the latest (unreleased) changes by installing a nightly version of KMS: Installing Nightly Builds.
Step 3: When your issue exists in both the latest and nightly versions, try resorting to the Open-Source Community. Kurento users might be having the same issue and maybe you can find help in there.
Step 4: If you want full attention from the Kurento team, get in contact with us to request Commercial Support.

Community Support¶

Kurento is a project mainly supported by its Open-Source Community. All people answering your questions are doing it with their own time, so please be kind and provide as much information as possible.

If you have questions about configuration, infrastructure, or general usage of Kurento Media Server, Stack Overflow is the best place to ask. Tag questions with kurento so other folks can find them easily.

Another option is to contact the Community through the Kurento Public Mailing List, which is a discussion forum.

Good questions to ask would be:

What is the best Media Pipeline to use for <place your use case here>?
How do I check that the ICE connectivity checks are working properly for WebRTC?
Which audio/video codec combination should I use to ensure no transcoding needs to take place?

Bugs & Support Issues¶

You can file bug reports on our Issue Tracker, and they will be addressed as soon as possible.

Support is a volunteered effort, and there is no guaranteed response time. If you need answers quickly, you can get in contact with us to request Commercial Support.

Reporting Issues¶

When reporting a bug, please include as much information as possible, this will help us solve the problem. Also, try to follow these guidelines as closely as possible, because they make it easier for people to work on the issue, and that means more chances that the issue gets fixed:

Be proactive. If you are working with an old version of Kurento, please check with newer versions, specially with the nightly version. We can’t emphasize this enough: it’s the first thing that we are going to ask.
Be curious. Has it been asked before? Is it really a bug? Everybody hates duplicated reports. The Search tool is your friend!
Be precise. Don’t wander around your situation and go straight to the point, unless the context around it is technically required to understand what is going on. Describe as precisely as possible what you are doing and what is happening but you think that shouldn’t happen.
Be specific. Explain how to reproduce the problem, being very systematic about it: step by step, so others can reproduce the bug. Also, report only one problem per opened issue.

If you definitely think you have hit a bug, try to include these in your bug report:

A description of the problem (e.g. what type of abnormal effect you are seeing).
A detailed specification of what you were executing (e.g. a specific code snippet firing the bug).
A detailed description of the execution environment (e.g. browser, operating system, KMS version, etc).
The relevant log generated by KMS, browser and, eventually, application server. Make sure to honor the relevant part: providing a 50MB log where only 10 lines are of interest is not providing a relevant log.
A proof-of-concept is of great help. This may need a bit of upfront work on your side, to isolate the actual component that presents issues from the rest of your application logic. Doing this will hugely increase the chances that developers of Kurento start working right away on the issue, if they are able to reproduce the problem without needing to have your whole system in place.

Commercial Support¶

Kurento is formed by a small team of people. This means that our task pipeline is quite restricted, and most feature or support requests end up being stored in the backlog for a long time. We advance as fast as we can, but time and resources are limited and at the end of the day there is so much that we can do.

If you have some needs that require urgent attention, or want to help with funding development on the Kurento project, we offer consultancy and support services on demand.

Please contact us at openvidu@gmail.com and let us know about your project!

Kurento API ¶

Table of Contents

Kurento API

Kurento Media Server can be controlled through the API it exposes, so application developers can use high level languages to interact with it. The Kurento project already provides Kurento Client implementations of this API for several platforms.

If you prefer a programming language different from the supported ones, you can implement your own Kurento Client by using the Kurento Protocol, which is based on WebSocket and JSON-RPC.

In the following sections we will describe the Kurento API from a high-level point of view, showing the media capabilities exposed by Kurento Media Server to clients. If you want to see working demos using Kurento, please refer to the Tutorials section.

Media Elements and Media Pipelines ¶

Kurento is based on two concepts that act as building blocks for application developers:

Media Elements. A Media Element is a functional unit performing a specific action on a media stream. Media Elements are a way of every capability is represented as a self-contained “black box” (the Media Element) to the application developer, who does not need to understand the low-level details of the element for using it. Media Elements are capable of receiving media from other elements (through media sources) and of sending media to other elements (through media sinks). Depending on their function, Media Elements can be split into different groups:
- Input Endpoints: Media Elements capable of receiving media and injecting it into a pipeline. There are several types of input endpoints. File input endpoints take the media from a file, Network input endpoints take the media from the network, and Capture input endpoints are capable of capturing the media stream directly from a camera or other kind of hardware resource.
- Filters: Media Elements in charge of transforming or analyzing media. Hence there are filters for performing operations such as mixing, muxing, analyzing, augmenting, etc.
- Hubs: Media Objects in charge of managing multiple media flows in a pipeline. A Hub contains a different HubPort for each one of the Media Elements that are connected. Depending on the Hub type, there are different ways to control the media. For example, there is a Hub called Composite that merges all input video streams in a unique output video stream, with all inputs arranged in a grid.
- Output Endpoints: Media Elements capable of taking a media stream out of the Media Pipeline. Again, there are several types of output endpoints, specialized in files, network, screen, etc.
Media Pipeline: A Media Pipeline is a chain of Media Elements, where the output stream generated by a source element is fed into one or more sink elements. Hence, the pipeline represents a “pipe” capable of performing a sequence of operations over a stream.

Example of a Media Pipeline implementing an interactive multimedia application receiving media from a WebRtcEndpoint, overlaying and image on the detected faces and sending back the resulting stream

The Kurento API is Object-Oriented. This means that it is based on Classes that can be instantiated in the form of Objects; these Objects provide properties that are a representation of the internal state of the Kurento server, and methods that expose the operations that can be performed by the server.

The following class diagram shows some of the relationships of the main classes in the Kurento API:

Class diagram of main classes in Kurento API¶

Endpoints ¶

A WebRtcEndpoint is an input/output endpoint that provides media streaming for Real Time Communications (RTC) through the web. It implements WebRTC technology to communicate with browsers.

An RtpEndpoint is an input/output endpoint that provides bidirectional content delivery capabilities with remote networked peers, through the RTP protocol. It uses SDP for media negotiation.

An HttpPostEndpoint is an input endpoint that accepts media using HTTP POST requests like HTTP file upload function.

A PlayerEndpoint is an input endpoint that retrieves content from file system, HTTP URL or RTSP URL and injects it into the Media Pipeline.

A RecorderEndpoint is an output endpoint that provides function to store contents in reliable mode (doesn’t discard data). It contains Media Sink pads for audio and video.

The following class diagram shows the relationships of the main endpoint classes:

digraph endpoints {
bgcolor = "transparent";
fontname = "Bitstream Vera Sans";
fontsize = 8;
size = "12,8";

node [
fillcolor = "#E7F2FA";
fontname = "Bitstream Vera Sans";
fontsize = 8;
shape = "record";
style = "filled";
]

edge [
arrowtail = "empty";
dir = "back";
fontname = "Bitstream Vera Sans";
fontsize = 8;
]

"MediaElement" -> "Endpoint";
"Endpoint" -> "SessionEndpoint";
"Endpoint" -> "UriEndpoint";

"SessionEndpoint" -> "HttpEndpoint";
"SessionEndpoint" -> "SdpEndpoint";

"HttpEndpoint" -> "HttpPostEndpoint";

"SdpEndpoint" -> "RtpEndpoint";
"SdpEndpoint" -> "WebRtcEndpoint";

"UriEndpoint" -> "PlayerEndpoint";
"UriEndpoint" -> "RecorderEndpoint";
}

Class diagram of main Endpoints in Kurento API¶

Filters ¶

Filters are MediaElements that perform media processing, Computer Vision, Augmented Reality, and so on.

The ZBarFilter filter detects QR and bar codes in a video stream. When a code is found, the filter raises a CodeFoundEvent. Clients can add a listener to this event to execute some action.

The FaceOverlayFilter filter detects faces in a video stream and overlaid it with a configurable image.

GStreamerFilter is a generic filter interface that allows injecting any GStreamer element into a Kurento Media Pipeline. Note however that the current implementation of GStreamerFilter only allows single elements to be injected; one cannot indicate more than one at the same time; use several GStreamerFilters if you need to inject more than one element at the same time.

The following class diagram shows the relationships of the main filter classes:

digraph filters {
bgcolor = "transparent";
fontname = "Bitstream Vera Sans";
fontsize = 8;
size = "12,8";

node [
fillcolor = "#E7F2FA";
fontname = "Bitstream Vera Sans";
fontsize = 8;
shape = "record";
style = "filled";
]

edge [
arrowtail = "empty";
dir = "back";
fontname = "Bitstream Vera Sans";
fontsize = 8;
]

"MediaElement" -> "Filter";
"Filter" -> "ZBarFilter";
"Filter" -> "FaceOverlayFilter";
"Filter" -> "GStreamerFilter";
}

Class diagram of main Filters in Kurento API¶

Hubs ¶

Hubs are media objects in charge of managing multiple media flows in a pipeline. A Hub has several hub ports where other Media Elements are connected.

Composite is a hub that mixes the audio stream of its connected inputs and constructs a grid with the video streams of them.

DispatcherOneToMany is a Hub that sends a given input to all the connected output HubPorts.

Dispatcher is a hub that allows routing between arbitrary input-output HubPort pairs.

The following class diagram shows the relationships of the hubs:

digraph hubs {
bgcolor = "transparent";
fontname = "Bitstream Vera Sans";
fontsize = 8;
size = "12,8";

node [
fillcolor = "#E7F2FA";
fontname = "Bitstream Vera Sans";
fontsize = 8;
shape = "record";
style = "filled";
]

edge [
arrowtail = "empty";
dir = "back";
fontname = "Bitstream Vera Sans";
fontsize = 8;
]

"MediaObject" -> "Hub";
"MediaObject" -> "MediaElement";

"Hub" -> "HubPort" [headlabel = "*", constraint = false, dir = normal, arrowhead = "vee", labelangle = 60];

"MediaElement" -> "HubPort";

"Hub" -> "Composite";
"Hub" -> "Dispatcher";
"Hub" -> "DispatcherOneToMany";
}

Class diagram of main Hubs in Kurento API¶

Kurento Client¶

Currently, the Kurento project provides implementations of the Kurento API for two programming languages: Java and JavaScript.

In the future, additional Kurento Clients can be created, exposing the same kind of modularity in other languages such as Python, C/C++, PHP, etc.

Kurento Java Client¶

Kurento Java Client is a Java SE layer which consumes the Kurento API and exposes its capabilities through a simple-to-use interface based on Java POJOs representing Media Elements and Media Pipelines.

This API is abstract in the sense that all the non-intuitive inherent complexities of the internal Kurento Protocol workings are abstracted and developers do not need to deal with them when creating applications. Using the Kurento Java Client only requires adding the appropriate dependency to a Maven project or to download the corresponding jar into the application’s Java Classpath.

It is important to remark that the Kurento Java Client is a media-plane control API. In other words, its objective is to expose the capability of managing media objects, but it does not provide any signaling plane capabilities.

Kurento JavaScript Client¶

Kurento JavaScript Client is a JavaScript layer which consumes the Kurento API and exposes its capabilities to JavaScript developers. It allow to build Node.js and browser based applications.

Reference Documentation¶

Kurento Client JavaDoc
Kurento Client JsDoc
Kurento Js Utils: a JavaScript utility library aimed to simplify the development of WebRTC applications.

Kurento Protocol ¶

Table of Contents

Kurento Protocol

Kurento Media Server is controlled by means of an RPC API, implemented in terms of the Kurento Protocol specification as described in this document, based on WebSocket and JSON-RPC.

JSON-RPC message format ¶

Kurento Protocol uses the JSON-RPC 2.0 Specification to encode its API messages. The following subsections describe the contents of the JSON messages that follow this spec.

Request ¶

An RPC call is represented by sending a request message to a server. The request message has the following members:

jsonrpc: A string specifying the version of the JSON-RPC protocol. It must be 2.0.
id: A unique identifier established by the client that contains a string or number. The server must reply with the same value in the response message. This member is used to correlate the context between both messages.
method: A string containing the name of the method to be invoked.
params: A structured value that holds the parameter values to be used during the invocation of the method.

The following JSON shows a sample request for the creation of a PlayerEndpoint Media Element:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "create",
  "params": {
    "type": "PlayerEndpoint",
    "constructorParams": {
      "pipeline": "6829986",
      "uri": "http://host/app/video.mp4"
    },
    "sessionId": "c93e5bf0-4fd0-4888-9411-765ff5d89b93"
  }
}

Successful Response ¶

When an RPC call is made, the server replies with a response message. In case of a successful response, the response message will contain the following members:

jsonrpc: A string specifying the version of the JSON-RPC protocol. It must be 2.0.
id: Must match the value of the id member in the request message.
result: Its value is determined by the method invoked on the server.
In case the connection is rejected, the response includes a message with a rejected attribute containing an message with a code and message attributes with the reason why the session was not accepted, and no sessionId is defined.

The following example shows a typical successful response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "value": "442352747",
    "sessionId": "c93e5bf0-4fd0-4888-9411-765ff5d89b93"
  }
}

Error Response ¶

When an RPC call is made, the server replies with a response message. In case of an error response, the response message will contain the following members:

jsonrpc: A string specifying the version of the JSON-RPC protocol. It must be 2.0.
id: Must match the value of the id member in the request message. If there was an error in detecting the id in the request message (e.g. Parse Error/Invalid Request), id is null.
error: A message describing the error through the following members:
- code: An integer number that indicates the error type that occurred.
- message: A string providing a short description of the error.
- data: A primitive or structured value that contains additional information about the error. It may be omitted. The value of this member is defined by the server.

The following example shows a typical error response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": "33",
    "message": "Invalid parameter format"
  }
}

Kurento API over JSON-RPC ¶

As explained in the Kurento API section, Kurento Media Server exposes a full fledged API to let applications process media in several ways.

To allow this rich API, Kurento Clients require full-duplex communications between client and server. For this reason, the Kurento Protocol is based on the WebSocket transport.

Before issuing commands, the Kurento Client requires establishing a WebSocket connection with Kurento Media Server to this URL: ws://hostname:port/kurento.

Once the WebSocket has been established, the Kurento Protocol offers different types of request/response messages:

ping: Keep-alive method between client and Kurento Media Server.
create: Instantiates a new media object, that is, a pipeline or media element.
invoke: Calls a method of an existing media object.
subscribe: Subscribes to some specific event, to receive notifications when it gets emitted by a media object.
unsubscribe: Removes an existing subscription to an event.
release: Deletes the object and release resources used by it.

The Kurento Protocol allows that Kurento Media Server sends requests to clients:

onEvent: This request is sent from Kurento Media server to subscribed clients when an event occurs.

Ping ¶

In order to warrant the WebSocket connectivity between the client and the Kurento Media Server, a keep-alive method is implemented. This method is based on a ping method sent by the client, which must be replied with a pong message from the server. If no response is obtained in a time interval, the client will assume that the connectivity with the media server has been lost. The parameter interval is the time available to receive the pong message from the server, in milliseconds. By default this value is 240000 (4 minutes).

This is an example of a ping request:

{
  "id": 1,
  "method": "ping",
  "params": {
    "interval": 240000
  },
  "jsonrpc": "2.0"
}

The response to a ping request must contain a result object with the parameter value: pong. The following snippet shows the pong response to the previous ping request:

{
  "id": 1,
  "result": {
    "value": "pong"
  },
  "jsonrpc": "2.0"
}

Create ¶

This message requests the creation of an object from the Kurento API (Media Pipelines and Media Elements). The parameter type specifies the type of the object to be created. The parameter constructorParams contains all the information needed to create the object. Each message needs different constructorParams to create the object. These parameters are defined in the Kurento API section.

Media Elements have to be contained in a previously created Media Pipeline. Therefore, before creating Media Elements, a Media Pipeline must exist. The response of the creation of a Media Pipeline contains a parameter called sessionId, which must be included in the next create requests for Media Elements.

The following example shows a request message for the creation of an object of the type MediaPipeline:

{
  "id": 2,
  "method": "create",
  "params": {
    "type": "MediaPipeline",
    "constructorParams": {},
    "properties": {}
  },
  "jsonrpc": "2.0"
}

The response to this request message is as follows. Notice that the parameter value identifies the created Media Pipelines, and sessionId is the identifier of the current session:

{
  "id": 2,
  "result": {
    "value": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline",
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

The response message contains the identifier of the new object in the field value. As usual, the field id must match the value of the id member in the request message. The sessionId is also returned in each response.

The following example shows a request message for the creation of an object of the type WebRtcEndpoint within an existing Media Pipeline (identified by the parameter mediaPipeline). Notice that in this request, the sessionId is already present, while in the previous example it was not (since at that point it was unknown for the client):

{
  "id": 3,
  "method": "create",
  "params": {
    "type": "WebRtcEndpoint",
    "constructorParams": {
      "mediaPipeline": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline"
    },
    "properties": {},
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

The response to this request message is as follows:

{
  "id": 3,
  "result": {
    "value": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline/087b7777-aab5-4787-816f-f0de19e5b1d9_kurento.WebRtcEndpoint",
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

Invoke ¶

This message requests the invocation of an operation in the specified object. The parameter object indicates the id of the object in which the operation will be invoked. The parameter operation carries the name of the operation to be executed. Finally, the parameter operationParams has the parameters needed to execute the operation.

The following example shows a request message for the invocation of the operation connect on a PlayerEndpoint connected to a WebRtcEndpoint:

{
  "id": 5,
  "method": "invoke",
  "params": {
    "object": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline/76dcb8d7-5655-445b-8cb7-cf5dc91643bc_kurento.PlayerEndpoint",
    "operation": "connect",
    "operationParams": {
      "sink": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline/087b7777-aab5-4787-816f-f0de19e5b1d9_kurento.WebRtcEndpoint"
    },
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

The response message contains the value returned while executing the operation invoked in the object, or nothing if the operation doesn’t return any value.

This is the typical response while invoking the operation connect (that doesn’t return anything):

{
  "id": 5,
  "result": {
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

Release ¶

This message requests releasing the resources of the specified object. The parameter object indicates the id of the object to be released:

{
  "id": 36,
  "method": "release",
  "params": {
    "object": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline",
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

The response message only contains the sessionId:

{
  "id": 36,
  "result": {
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

Unsubscribe ¶

This message requests the cancellation of a previous event subscription. The parameter subscription contains the subscription id received from the server when the subscription was created.

The following example shows a request message requesting the cancellation of the subscription 353be312-b7f1-4768-9117-5c2f5a087429 for a given object:

{
  "id": 38,
  "method": "unsubscribe",
  "params": {
    "subscription": "052061c1-0d87-4fbd-9cc9-66b57c3e1280",
    "object": "6ba9067f-cdcf-4ea6-a6ee-d74519585acd_kurento.MediaPipeline/76dcb8d7-5655-445b-8cb7-cf5dc91643bc_kurento.PlayerEndpoint",
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

The response message only contains the sessionId:

{
  "id": 38,
  "result": {
    "sessionId": "bd4d6227-0463-4d52-b1c3-c71f0be68466"
  },
  "jsonrpc": "2.0"
}

OnEvent ¶

When a client is subscribed to some events from an object, the server sends an onEvent request each time an event of that type is fired in the object. This is possible because the Kurento Protocol is implemented with WebSockets and there is a full duplex channel between client and server. The request that server sends to client has all the information about the event:

source: The object source of the event.
type: The type of the event.
timestamp: [DEPRECATED: Use timestampMillis] The timestamp associated with this event: Seconds elapsed since the UNIX Epoch (Jan 1, 1970, UTC).
timestampMillis: The timestamp associated with this event: Milliseconds elapsed since the UNIX Epoch (Jan 1, 1970, UTC).
tags: Media elements can be labeled using the methods setSendTagsInEvents and addTag, present in each element. These tags are key-value metadata that can be used by developers for custom purposes. Tags are returned with each event by the media server in this field.

The following example shows a notification sent from server to client, notifying of an event EndOfStream for a PlayerEndpoint object:

{
  "jsonrpc":"2.0",
  "method":"onEvent",
  "params":{
    "value":{
      "data":{
        "source":"681f1bc8-2d13-4189-a82a-2e2b92248a21_kurento.MediaPipeline/e983997e-ac19-4f4b-9575-3709af8c01be_kurento.PlayerEndpoint",
        "tags":[],
        "timestamp":"1441277150",
        "timestampMillis":"1441277150433",
        "type":"EndOfStream"
      },
      "object":"681f1bc8-2d13-4189-a82a-2e2b92248a21_kurento.MediaPipeline/e983997e-ac19-4f4b-9575-3709af8c01be_kurento.PlayerEndpoint",
      "type":"EndOfStream"
    }
  }
}

Notice that this message has no id field due to the fact that no response is required.

Network issues ¶

Resources handled by KMS are high-consuming. For this reason, KMS implements a garbage collector.

A Media Element is collected when the client is disconnected longer than 4 minutes. After that time, these media elements are disposed automatically. Therefore, the WebSocket connection between client and KMS shoul be active at all times. In case of temporary network disconnection, KMS implements a mechanism that allows the client to reconnect.

For this, there is a special kind of message with the format shown below. This message allows a client to reconnect to the same KMS instance to which it was previously connected:

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "connect",
  "params": {
    "sessionId":"4f5255d5-5695-4e1c-aa2b-722e82db5260"
  }
}

If KMS replies as follows:

{
  "jsonrpc": "2.0",
  "id": 7,
  "result": {
    "sessionId":"4f5255d5-5695-4e1c-aa2b-722e82db5260"
  }
}

… this means that the client was able to reconnect to the same KMS instance. In case of reconnection to a different KMS instance, the message is the following:

{
  "jsonrpc":"2.0",
  "id": 7,
  "error":{
    "code":40007,
    "message":"Invalid session",
    "data":{
      "type":"INVALID_SESSION"
    }
  }
}

In this case, the client is supposed to invoke the connect primitive once again in order to get a new sessionId:

{
  "jsonrpc":"2.0",
  "id": 7,
  "method":"connect"
}

Example: WebRTC in loopback ¶

This section describes an example of the messages exchanged between a Kurento Client and the Kurento Media Server, in order to create a WebRTC in loopback. This example is fully depicted in the Tutorials section. The steps are the following:

Client sends a request message in order to create a Media Pipeline:

{
  "id":1,
  "method":"create",
  "params":{
    "type":"MediaPipeline",
    "constructorParams":{},
    "properties":{}
  },
  "jsonrpc":"2.0"
}

2. KMS sends a response message with the identifier for the Media Pipeline and the Media Session:

{
  "id":1,
  "result":{
    "value":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline",
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

Client sends a request to create a WebRtcEndpoint:

{
  "id":2,
  "method":"create",
  "params":{
    "type":"WebRtcEndpoint",
    "constructorParams":{
      "mediaPipeline":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline"
    },
    "properties": {},
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

KMS creates the WebRtcEndpoint and sends back to the client the Media Element identifier:

{
  "id":2,
  "result":{
    "value":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline/e72a1ff5-e416-48ff-99ef-02f7fadabaf7_WebRtcEndpoint",
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

Client invokes the connect primitive in the WebRtcEndpoint in order to create a loopback:

{
  "id":3,
  "method":"invoke",
  "params":{
    "object":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline/e72a1ff5-e416-48ff-99ef-02f7fadabaf7_WebRtcEndpoint",
    "operation":"connect",
    "operationParams":{
      "sink":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline/e72a1ff5-e416-48ff-99ef-02f7fadabaf7_WebRtcEndpoint"
    },
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

KMS carries out the connection and acknowledges the operation:

{
  "id":3,
  "result":{
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

Client invokes the processOffer primitive in the WebRtcEndpoint in order to start the SDP Offer/Answer negotiation for WebRTC:

{
  "id":4,
  "method":"invoke",
  "params":{
    "object":"c4a84b47-1acd-4930-9f6d-008c10782dfe_MediaPipeline/e72a1ff5-e416-48ff-99ef-02f7fadabaf7_WebRtcEndpoint",
    "operation":"processOffer",
    "operationParams":{
      "offer":"SDP"
    },
    "sessionId":"ba4be2a1-2b09-444e-a368-f81825a6168c"
  },
  "jsonrpc":"2.0"
}

KMS carries out the SDP negotiation and returns the SDP Answer:

{
  "id":4,
  "result":{
    "value":"SDP"
  },
  "jsonrpc":"2.0"
}

Creating a custom Kurento Client ¶

In order to implement a Kurento Client you need to follow the reference documentation. The best way to know all details is to take a look to the IDL file that defines the interface of the Kurento elements.

We have defined a custom IDL format based on JSON. From it, we automatically generate the client code for the Kurento Client libraries:

Kurento Module Creator ¶

Kurento Clients contain code that is automatically generated from the IDL interface files, using a tool named Kurento Module Creator. This tool can be also used to create custom clients in other languages.

Kurento Module Creator can be installed in an Ubuntu machine using the following command:

sudo apt-get install kurento-module-creator

The aim of this tool is to generate the client code and also the glue code needed in the server-side. For code generation it uses Freemarker as the template engine. The typical way to use Kurento Module Creator is by running a command like this:

kurento-module-creator -c <CODEGEN_DIR> -r <ROM_FILE> -r <TEMPLATES_DIR>

Where:

CODEGEN_DIR: Destination directory for generated files.
ROM_FILE: A space-separated list of Kurento Media Element Description (kmd files), or folders containing these files. For example, you can take a look to the kmd files within the Kurento Media Server source code.
TEMPLATES_DIR: Directory that contains template files. As an example, you can take a look to the internal Java templates and JavaScript templates directories.

Kurento Modules¶

Kurento is a pluggable framework. Each plugin in Kurento is called a module.

If you are interested in writing our own modules, please read the section about Writing Kurento Modules.

We classify Kurento modules into three groups, namely:

Main modules. Incorporated out of the box with Kurento Media Server:
- kms-core: Main components of Kurento Media Server.
- kms-elements: Implementation of Kurento Media Elements (WebRtcEndpoint, PlayerEndpoint, etc.)
- kms-filters: Implementation of Kurento Filters (FaceOverlayFilter, ZBarFilter, etc.)
Built-in modules. Extra modules developed by the Kurento team to enhance the basic capabilities of Kurento Media Server. So far, there are four built-in modules, namely:
- kms-pointerdetector: Filter that detects pointers in video streams, based on color tracking. Install command:
```
sudo apt-get install kms-pointerdetector
```
- kms-chroma: Filter that takes a color range in the top layer and makes it transparent, revealing another image behind. Install command:
```
sudo apt-get install kms-chroma
```
- kms-crowddetector: Filter that detects people agglomeration in video streams. Install command:
```
sudo apt-get install kms-crowddetector
```
- kms-platedetector: Filter that detects vehicle plates in video streams. Install command:
```
sudo apt-get install kms-platedetector
```
  Warning
  
  The plate detector module is a prototype and its results are not always accurate. Consider this if you are planning to use this module in a production environment.
Custom modules. Extensions to Kurento Media Server which provides new media capabilities.

The following picture shows an schematic view of the Kurento Media Server with its different modules:

Kurento modules architecture Kurento Media Server can be extended with built-it modules (crowddetector, pointerdetector, chroma, platedetector) and also with other custom modules.

Taking into account the built-in modules, the Kurento toolbox is extended as follows:

Extended Kurento Toolbox The basic Kurento toolbox (left side of the picture) is extended with more Computer Vision and Augmented Reality filters (right side of the picture) provided by the built-in modules.

The remainder of this page is structured in four sections in which the built-in modules (kms-pointerdetector, kms-chroma, kms-crowddetector, kms-platedetector) are used to develop simple applications (tutorials) aimed to show how to use them.

Module Tutorial - Pointer Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a pointer-tracking filter element.

Java Module - Pointer Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a pointer tracking filter element.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-pointerdetector should be also installed:

sudo apt-get install kms-pointerdetector

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-pointerdetector
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a pointer in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with PointerDetector filter in loopback Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PointerDetector instead of FaceOverlay filter.

In order to perform pointer detection, there must be a calibration stage, in which the color of the pointer is registered by the filter. To accomplish this step, the pointer should be placed in a square visible in the upper left corner of the video after going thorugh the filter, as follows:

Pointer calibration stage

When the desired color to track is filling that box, a calibration message is sent from the client to the server. This is done by clicking on the Calibrate blue button of the GUI.

After that, the color of the pointer is tracked in real time by Kurento Media Server. PointerDetectorFilter can also define regions in the screen called windows in which some actions are performed when the pointer is detected when the pointer enters (WindowInEvent event) and exits (WindowOutEvent event) the windows. This is implemented in the server-side logic as follows:

// Media Logic (Media Pipeline and Elements)
UserSession user = new UserSession();
MediaPipeline pipeline = kurento.createMediaPipeline();
user.setMediaPipeline(pipeline);
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline)
      .build();
user.setWebRtcEndpoint(webRtcEndpoint);
users.put(session.getId(), user);

webRtcEndpoint
      .addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

   @Override
   public void onEvent(IceCandidateFoundEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "iceCandidate");
      response.add("candidate", JsonUtils
      .toJsonObject(event.getCandidate()));
      try {
         synchronized (session) {
      session.sendMessage(new TextMessage(
            response.toString()));
         }
      } catch (IOException e) {
         log.debug(e.getMessage());
      }
   }
      });

pointerDetectorFilter = new PointerDetectorFilter.Builder(pipeline,
      new WindowParam(5, 5, 30, 30)).build();

pointerDetectorFilter
      .addWindow(new PointerDetectorWindowMediaParam("window0",
      50, 50, 500, 150));

pointerDetectorFilter
      .addWindow(new PointerDetectorWindowMediaParam("window1",
      50, 50, 500, 250));

webRtcEndpoint.connect(pointerDetectorFilter);
pointerDetectorFilter.connect(webRtcEndpoint);

pointerDetectorFilter
      .addWindowInListener(new EventListener<WindowInEvent>() {
   @Override
   public void onEvent(WindowInEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "windowIn");
      response.addProperty("roiId", event.getWindowId());
      try {
         session.sendMessage(new TextMessage(response
         .toString()));
      } catch (Throwable t) {
         sendError(session, t.getMessage());
      }
   }
      });

pointerDetectorFilter
      .addWindowOutListener(new EventListener<WindowOutEvent>() {

   @Override
   public void onEvent(WindowOutEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "windowOut");
      response.addProperty("roiId", event.getWindowId());
      try {
         session.sendMessage(new TextMessage(response
         .toString()));
      } catch (Throwable t) {
         sendError(session, t.getMessage());
      }
   }
      });

// SDP negotiation (offer and answer)
String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

// Sending response back to client
JsonObject response = new JsonObject();
response.addProperty("id", "startResponse");
response.addProperty("sdpAnswer", sdpAnswer);
synchronized (session) {
   session.sendMessage(new TextMessage(response.toString()));
}

webRtcEndpoint.gatherCandidates();

The following picture illustrates the pointer tracking in one of the defined windows:

Pointer tracking over a window

In order to send the calibration message from the client side, this function is used in the JavaScript side of this demo:

function calibrate() {
   console.log("Calibrate color");

   var message = {
         id : 'calibrate'
      }
   sendMessage(message);
}

When this message is received in the application server side, this code is execute to carry out the calibration:

private void calibrate(WebSocketSession session, JsonObject jsonMessage) {
   if (pointerDetectorFilter != null) {
      pointerDetectorFilter.trackColorFromCalibrationRegion();
   }
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript Module - Pointer Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a pointer tracking filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-pointerdetector should be also installed:

sudo apt-get install kms-pointerdetector

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-pointerdetector
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. Kurento Media Server must use WebSockets over SSL/TLS (WSS), so make sure you check this too. It is possible to locate the KMS in other machine simple adding the parameter ws_uri to the URL:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a pointer in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with PointerDetector filter in loopback Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PointerDetector instead of FaceOverlay filter.

In order to perform pointer detection, there must be a calibration stage, in which the color of the pointer is registered by the filter. To accomplish this step, the pointer should be placed in a square in the upper left corner of the video, as follows:

Pointer calibration stage

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

In that precise moment, a calibration operation should be carried out. This is done by clicking on the Calibrate blue button of the GUI.

After that, the color of the pointer is tracked in real time by Kurento Media Server. PointerDetectorFilter can also define regions in the screen called windows in which some actions are performed when the pointer is detected when the pointer enters (WindowIn event) and exits (WindowOut event) the windows. This is implemented in the JavaScript logic as follows:

...
kurentoClient.register('kurento-module-pointerdetector')
const PointerDetectorWindowMediaParam = kurentoClient.getComplexType('pointerdetector.PointerDetectorWindowMediaParam')
const WindowParam                     = kurentoClient.getComplexType('pointerdetector.WindowParam')
...

kurentoClient(args.ws_uri, function(error, client) {
  if (error) return onError(error);

  client.create('MediaPipeline', function(error, _pipeline) {
    if (error) return onError(error);

    pipeline = _pipeline;

    console.log("Got MediaPipeline");

    pipeline.create('WebRtcEndpoint', function(error, webRtc) {
      if (error) return onError(error);

      console.log("Got WebRtcEndpoint");

      setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

      webRtc.processOffer(sdpOffer, function(error, sdpAnswer) {
        if (error) return onError(error);

        console.log("SDP answer obtained. Processing ...");

        webRtc.gatherCandidates(onError);
        webRtcPeer.processAnswer(sdpAnswer);
      });

      var options =
      {
        calibrationRegion: WindowParam({
          topRightCornerX: 5,
          topRightCornerY:5,
          width:30,
          height: 30
        })
      };

      pipeline.create('pointerdetector.PointerDetectorFilter', options, function(error, _filter) {
        if (error) return onError(error);

        filter = _filter;

        var options = PointerDetectorWindowMediaParam({
          id: 'window0',
          height: 50,
          width:50,
          upperRightX: 500,
          upperRightY: 150
        });

        filter.addWindow(options, onError);

        var options = PointerDetectorWindowMediaParam({
          id: 'window1',
          height: 50,
          width:50,
          upperRightX: 500,
          upperRightY: 250
        });

        filter.addWindow(options, onError);

        filter.on ('WindowIn', function (data){
          console.log ("Event window in detected in window " + data.windowId);
        });

        filter.on ('WindowOut', function (data){
          console.log ("Event window out detected in window " + data.windowId);
        });

        console.log("Connecting ...");
        client.connect(webRtc, filter, webRtc, function(error) {
          if (error) return onError(error);

          console.log("WebRtcEndpoint --> Filter --> WebRtcEndpoint");
        });
      });
    });
  });
});

The following picture illustrates the pointer tracking in one of the defined windows:

Pointer tracking over a window

In order to carry out the calibration process, this JavaScript function is used:

function calibrate() {
  if(filter) filter.trackColorFromCalibrationRegion(onError);
}

function onError(error) {
  if(error) console.error(error);
}

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

The dependencies of this demo has to be obtained using Bower. The definition of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
   "kurento-module-pointerdetector": "6.10.0"
}

To get these dependencies, just run the following shell command:

bower install

Note

We are in active development. You can find the latest versions at Bower.

Node.js Module - Pointer Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a pointer tracking filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-pointerdetector should be also installed:

sudo apt-get install kms-pointerdetector

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-pointerdetector
git checkout 6.10.0
npm install

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a pointer in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with PointerDetector filter in loopback Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PointerDetector instead of FaceOverlay filter.

In order to perform pointer detection, there must be a calibration stage, in which the color of the pointer is registered by the filter. To accomplish this step, the pointer should be placed in a square in the upper left corner of the video, as follows:

Pointer calibration stage

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

In that precise moment, a calibration operation should be carried out. This is done by clicking on the Calibrate blue button of the GUI.

After that, the color of the pointer is tracked in real time by Kurento Media Server. PointerDetectorFilter can also define regions in the screen called windows in which some actions are performed when the pointer is detected when the pointer enters (WindowIn event) and exits (WindowOut event) the windows. This is implemented in the JavaScript logic as follows:

...
kurento.register('kurento-module-pointerdetector');
const PointerDetectorWindowMediaParam = kurento.getComplexType('pointerdetector.PointerDetectorWindowMediaParam');
const WindowParam                     = kurento.getComplexType('pointerdetector.WindowParam');
...

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint, filter) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, filter, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    filter.on('WindowIn', function (_data) {
                        return callback(null, 'WindowIn', _data);
                    });

                    filter.on('WindowOut', function (_data) {
                        return callback(null, 'WindowOut', _data);
                    });

                    var options1 = PointerDetectorWindowMediaParam({
                        id: 'window0',
                        height: 50,
                        width: 50,
                        upperRightX: 500,
                        upperRightY: 150
                    });
                    filter.addWindow(options1, function(error) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }
                    });

                    var options2 = PointerDetectorWindowMediaParam({
                        id: 'window1',
                        height: 50,
                        width:50,
                        upperRightX: 500,
                        upperRightY: 250
                    });
                    filter.addWindow(options2, function(error) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint,
                            'pointerDetector' : filter
                        }
                        return callback(null, 'sdpAnswer', sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        var options = {
            calibrationRegion: WindowParam({
                topRightCornerX: 5,
                topRightCornerY:5,
                width:30,
                height: 30
            })
        };

        pipeline.create('pointerdetector.PointerDetectorFilter', options, function(error, filter) {
            if (error) {
                return callback(error);
            }

            return callback(null, webRtcEndpoint, filter);
        });
    });
}

The following picture illustrates the pointer tracking in one of the defined windows:

Pointer tracking over a window

In order to carry out the calibration process, this JavaScript function is used:

function calibrate() {
   if (webRtcPeer) {
      console.log("Calibrating...");
      var message = {
         id : 'calibrate'
      }
      sendMessage(message);
   }
}

Dependencies¶

Dependencies of this demo are managed using NPM. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   "kurento-utils" : "6.10.0",
   "kurento-module-pointerdetector": "6.10.0"
}

Note

We are in active development. You can find the latest versions at npm and Bower.

Module Tutorial - Chroma Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a chroma filter element.

Java Module - Chroma Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a chroma filter element.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-chroma should be also installed:

sudo apt-get install kms-chroma

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-chroma
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a chroma in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with Chroma filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a Chroma instead of FaceOverlay filter.

In order to perform chroma detection, there must be a color calibration stage. To accomplish this step, at the beginning of the demo, a little square appears in upper left of the video, as follows:

Chroma calibration stage

In the first second of the demo, a calibration process is done, by detecting the color inside that square. When the calibration is finished, the square disappears and the chroma is substituted with the configured image. Take into account that this process requires good lighting condition. Otherwise the chroma substitution will not be perfect. This behavior can be seen in the upper right corner of the following screenshot:

Chroma filter in action

The media pipeline of this demo is is implemented in the server-side logic as follows:

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // Media Logic (Media Pipeline and Elements)
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline)
            .build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      users.put(session.getId(), user);

      webRtcEndpoint
            .addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

               @Override
               public void onEvent(IceCandidateFoundEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "iceCandidate");
                  response.add("candidate", JsonUtils
                        .toJsonObject(event.getCandidate()));
                  try {
                     synchronized (session) {
                        session.sendMessage(new TextMessage(
                              response.toString()));
                     }
                  } catch (IOException e) {
                     log.debug(e.getMessage());
                  }
               }
            });

      ChromaFilter chromaFilter = new ChromaFilter.Builder(pipeline,
            new WindowParam(5, 5, 40, 40)).build();
      String appServerUrl = System.getProperty("app.server.url",
            ChromaApp.DEFAULT_APP_SERVER_URL);
      chromaFilter.setBackground(appServerUrl + "/img/mario.jpg");

      webRtcEndpoint.connect(chromaFilter);
      chromaFilter.connect(webRtcEndpoint);

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      // Sending response back to client
      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
         session.sendMessage(new TextMessage(response.toString()));
      }
      webRtcEndpoint.gatherCandidates();

   } catch (Throwable t) {
      sendError(session, t.getMessage());
   }
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript Module - Chroma Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a chroma filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-chroma should be also installed:

sudo apt-get install kms-chroma

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-chroma
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. Kurento Media Server must use WebSockets over SSL/TLS (WSS), so make sure you check this too. It is possible to locate the KMS in other machine simple adding the parameter ws_uri to the URL:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a chroma in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with Chroma filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a Chroma instead of FaceOverlay filter.

In order to perform chroma detection, there must be a color calibration stage. To accomplish this step, at the beginning of the demo, a little square appears in upper left of the video, as follows:

Chroma calibration stage

In the first second of the demo, a calibration process is done, by detecting the color inside that square. When the calibration is finished, the square disappears and the chroma is substituted with the configured image. Take into account that this process requires lighting condition. Otherwise the chroma substitution will not be perfect. This behavior can be seen in the upper right corner of the following screenshot:

Chroma filter in action

Note

Modules can have options. For configure these options, you need get the constructor to them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

The media pipeline of this demo is is implemented in the JavaScript logic as follows:

...
kurentoClient.register('kurento-module-chroma')
const WindowParam = kurentoClient.getComplexType('chroma.WindowParam')
...

kurentoClient(args.ws_uri, function(error, client) {
  if (error) return onError(error);

  client.create('MediaPipeline', function(error, _pipeline) {
    if (error) return onError(error);

    pipeline = _pipeline;

    console.log("Got MediaPipeline");

    pipeline.create('WebRtcEndpoint', function(error, webRtc) {
      if (error) return onError(error);

      setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

      webRtc.processOffer(sdpOffer, function(error, sdpAnswer) {
        if (error) return onError(error);

        console.log("SDP answer obtained. Processing...");

        webRtc.gatherCandidates(onError);
        webRtcPeer.processAnswer(sdpAnswer);
      });

      console.log("Got WebRtcEndpoint");

      var options =
      {
        window: WindowParam({
          topRightCornerX: 5,
          topRightCornerY: 5,
          width: 30,
          height: 30
        })
      }

      pipeline.create('chroma.ChromaFilter', options, function(error, filter) {
        if (error) return onError(error);

        console.log("Got Filter");

        filter.setBackground(args.bg_uri, function(error) {
          if (error) return onError(error);

          console.log("Set Image");
        });

        client.connect(webRtc, filter, webRtc, function(error) {
          if (error) return onError(error);

          console.log("WebRtcEndpoint --> filter --> WebRtcEndpoint");
        });
      });
    });
  });
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

The dependencies of this demo has to be obtained using Bower. The definition of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
   "kurento-module-pointerdetector": "6.10.0"
}

To get these dependencies, just run the following shell command:

bower install

Note

We are in active development. You can find the latest versions at Bower.

Node.js Module - Chroma Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a chroma filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-chroma should be also installed:

sudo apt-get install kms-chroma

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-chroma
git checkout 6.10.0
npm install

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a chroma in a WebRTC stream based on color tracking.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with Chroma filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a Chroma instead of FaceOverlay filter.

In order to perform chroma detection, there must be a color calibration stage. To accomplish this step, at the beginning of the demo, a little square appears in upper left of the video, as follows:

Chroma calibration stage

In the first second of the demo, a calibration process is done, by detecting the color inside that square. When the calibration is finished, the square disappears and the chroma is substituted with the configured image. Take into account that this process requires lighting condition. Otherwise the chroma substitution will not be perfect. This behavior can be seen in the upper right corner of the following screenshot:

Chroma filter in action

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

The media pipeline of this demo is is implemented in the JavaScript logic as follows:

...
kurento.register('kurento-module-chroma');
...

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint, filter) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, filter, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint
                        }
                        return callback(null, sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        var options = {
            window: kurento.getComplexType('chroma.WindowParam')({
                topRightCornerX: 5,
                topRightCornerY: 5,
                width: 30,
                height: 30
            })
        }
        pipeline.create('chroma.ChromaFilter', options, function(error, filter) {
            if (error) {
                return callback(error);
            }

            return callback(null, webRtcEndpoint, filter);
        });
    });
}

function connectMediaElements(webRtcEndpoint, filter, callback) {
    webRtcEndpoint.connect(filter, function(error) {
        if (error) {
            return callback(error);
        }

        filter.setBackground(url.format(asUrl) + 'img/mario.jpg', function(error) {
            if (error) {
                return callback(error);
            }

            filter.connect(webRtcEndpoint, function(error) {
                if (error) {
                    return callback(error);
                }

                return callback(null);
            });
        });
    });
}

Dependencies¶

Dependencies of this demo are managed using NPM. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   "kurento-utils" : "6.10.0",
   "kurento-module-pointerdetector": "6.10.0"
}

Note

We are in active development. You can find the latest versions at npm and Bower.

Module Tutorial - Crowd Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a crowd detector filter. This filter detects people agglomeration in video streams.

Java Module - Crowd Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a crowd detector filter. This filter detects clusters of people in video streams.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-crowddetector should be also installed:

sudo apt-get install kms-crowddetector

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-crowddetector
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a crowd in a WebRTC stream.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with crowdDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a CrowdDetector instead of FaceOverlay filter.

To setup a CrowdDetectorFilter, first we need to define one or more regions of interest (ROIs). A ROI determines the zone within the video stream, which are going to be monitored and analised by the filter. To define a ROI, we need to configure at least three points. These points are defined in relative terms (0 to 1) to the video width and height.

CrowdDetectorFilter performs two actions in the defined ROIs. On one hand, the detected crowds are colored over the stream. On the other hand, different events are raised to the client.

To understand crowd coloring, we can take a look to an screenshot of a running example of CrowdDetectorFilter. In the picture below, we can see that there are two ROIs (bounded with white lines in the video). On these ROIs, we can see two different colors over the original video stream: red zones are drawn over detected static crowds (or moving slowly). Blue zones are drawn over the detected crowds moving fast.

Crowd detection sample

Regarding crowd events, there are three types of events, namely:

CrowdDetectorFluidityEvent. Event raised when a certain level of fluidity is detected in a ROI. Fluidity can be seen as the level of general movement in a crowd.
CrowdDetectorOccupancyEvent. Event raised when a level of occupancy is detected in a ROI. Occupancy can be seen as the level of agglomeration in stream.
CrowdDetectorDirectionEvent. Event raised when a movement direction is detected in a ROI by a crowd.

Both fluidity as occupancy are quantified in a relative metric from 0 to 100%. Then, both attributes are qualified into three categories: i) Minimum (min); ii) Medium (med); iii) Maximum (max).

Regarding direction, it is quantified as an angle (0-360º), where 0 is the direction from the central point of the video to the top (i.e., north), 90 correspond to the direction to the right (east), 180 is the south, and finally 270 is the west.

With all these concepts, now we can check out the Java server-side code of this demo. As depicted in the snippet below, we create a ROI by adding RelativePoint instances to a list. Each ROI is then stored into a list of RegionOfInterest instances.

Then, each ROI should be configured. To do that, we have the following methods:

setFluidityLevelMin: Fluidity level (0-100%) for the category minimum.

setFluidityLevelMed: Fluidity level (0-100%) for the category medium.

setFluidityLevelMax: Fluidity level (0-100%) for the category maximum.

setFluidityNumFramesToEvent: Number of consecutive frames detecting a fluidity level to rise a event.

setOccupancyLevelMin: Occupancy level (0-100%) for the category minimum.

setOccupancyLevelMed: Occupancy level (0-100%) for the category medium.

setOccupancyLevelMax: Occupancy level (0-100%) for the category maximum.

setOccupancyNumFramesToEvent: Number of consecutive frames detecting a occupancy level to rise a event.

setSendOpticalFlowEvent: Boolean value that indicates whether or not directions events are going to be tracked by the filter. Be careful with this feature, since it is very demanding in terms of resource usage (CPU, memory) in the media server. Set to true this parameter only when you are going to need directions events in your client-side.

setOpticalFlowNumFramesToEvent: Number of consecutive frames detecting a direction level to rise a event.

setOpticalFlowNumFramesToReset: Number of consecutive frames detecting a occupancy level in which the counter is reset.

setOpticalFlowAngleOffset: Counterclockwise offset of the angle. This parameters is useful to move the default axis for directions (0º=north, 90º=east, 180º=south, 270º=west).

All in all, the media pipeline of this demo is implemented as follows:

// Media Logic (Media Pipeline and Elements)
MediaPipeline pipeline = kurento.createMediaPipeline();
pipelines.put(session.getId(), pipeline);

WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline)
      .build();
webRtcEndpoint
   .addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
      @Override
      public void onEvent(IceCandidateFoundEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "iceCandidate");
      response.add("candidate",
         JsonUtils.toJsonObject(event.getCandidate()));
      try {
         synchronized (session) {
         session.sendMessage(new TextMessage(response
            .toString()));
         }
      } catch (IOException e) {
         log.debug(e.getMessage());
      }
      }
   });

List<RegionOfInterest> rois = new ArrayList<>();
List<RelativePoint> points = new ArrayList<RelativePoint>();

points.add(new RelativePoint(0, 0));
points.add(new RelativePoint(0.5F, 0));
points.add(new RelativePoint(0.5F, 0.5F));
points.add(new RelativePoint(0, 0.5F));

RegionOfInterestConfig config = new RegionOfInterestConfig();

config.setFluidityLevelMin(10);
config.setFluidityLevelMed(35);
config.setFluidityLevelMax(65);
config.setFluidityNumFramesToEvent(5);
config.setOccupancyLevelMin(10);
config.setOccupancyLevelMed(35);
config.setOccupancyLevelMax(65);
config.setOccupancyNumFramesToEvent(5);
config.setSendOpticalFlowEvent(false);
config.setOpticalFlowNumFramesToEvent(3);
config.setOpticalFlowNumFramesToReset(3);
config.setOpticalFlowAngleOffset(0);

rois.add(new RegionOfInterest(points, config, "roi0"));

CrowdDetectorFilter crowdDetectorFilter = new CrowdDetectorFilter.Builder(
      pipeline, rois).build();

webRtcEndpoint.connect(crowdDetectorFilter);
crowdDetectorFilter.connect(webRtcEndpoint);

// addEventListener to crowddetector
crowdDetectorFilter.addCrowdDetectorDirectionListener(
   new EventListener<CrowdDetectorDirectionEvent>() {
   @Override
   public void onEvent(CrowdDetectorDirectionEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "directionEvent");
      response.addProperty("roiId", event.getRoiID());
      response.addProperty("angle",
      event.getDirectionAngle());
      try {
         session.sendMessage(new TextMessage(response
         .toString()));
      } catch (Throwable t) {
         sendError(session, t.getMessage());
      }
   }
      });

crowdDetectorFilter.addCrowdDetectorFluidityListener(
   new EventListener<CrowdDetectorFluidityEvent>() {
   @Override
   public void onEvent(CrowdDetectorFluidityEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "fluidityEvent");
      response.addProperty("roiId", event.getRoiID());
      response.addProperty("level",
      event.getFluidityLevel());
      response.addProperty("percentage",
      event.getFluidityPercentage());
      try {
         session.sendMessage(new TextMessage(response
         .toString()));
      } catch (Throwable t) {
         sendError(session, t.getMessage());
      }
   }
      });

crowdDetectorFilter.addCrowdDetectorOccupancyListener(
   new EventListener<CrowdDetectorOccupancyEvent>() {
   @Override
   public void onEvent(CrowdDetectorOccupancyEvent event) {
      JsonObject response = new JsonObject();
      response.addProperty("id", "occupancyEvent");
      response.addProperty("roiId", event.getRoiID());
      response.addProperty("level",
      event.getOccupancyLevel());
      response.addProperty("percentage",
      event.getOccupancyPercentage());
      try {
         session.sendMessage(new TextMessage(response
         .toString()));
      } catch (Throwable t) {
         sendError(session, t.getMessage());
      }
   }
      });

// SDP negotiation (offer and answer)
String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

// Sending response back to client
JsonObject response = new JsonObject();
response.addProperty("id", "startResponse");
response.addProperty("sdpAnswer", sdpAnswer);
session.sendMessage(new TextMessage(response.toString()));

webRtcEndpoint.gatherCandidates();

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript Module - Crowd Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a crowd detector filter. This filter detects people agglomeration in video streams.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-crowddetector should be also installed:

sudo apt-get install kms-crowddetector

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-crowddetector
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. Kurento Media Server must use WebSockets over SSL/TLS (WSS), so make sure you check this too. It is possible to locate the KMS in other machine simple adding the parameter ws_uri to the URL:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a crowd in a WebRTC stream.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with crowdDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a CrowdDetector instead of FaceOverlay filter.

To setup a CrowdDetectorFilter, first we need to define one or more region of interests (ROIs). A ROI delimits the zone within the video stream in which crowd are going to be tracked. To define a ROI, we need to configure at least three points. These points are defined in relative terms (0 to 1) to the video width and height.

CrowdDetectorFilter performs two actions in the defined ROIs. On the one hand, the detected crowd are colored over the stream. On the other hand, different events are raised to the client.

To understand crowd coloring, we can take a look to an screenshot of a running example of CrowdDetectorFilter. In the picture below, we can see that there are two ROIs (bounded with white lines in the video). On these ROIs, we can see two different colors over the original video stream: red zones are drawn over detected static crowds (or moving slowly). Blue zones are drawn over the detected crowds moving fast.

Crowd detection sample

Regarding crowd events, there are three types of events, namely:

CrowdDetectorFluidityEvent. Event raised when a certain level of fluidity is detected in a ROI. Fluidity can be seen as the level of general movement in a crowd.
CrowdDetectorOccupancyEvent. Event raised when a level of occupancy is detected in a ROI. Occupancy can be seen as the level of agglomeration in stream.
CrowdDetectorDirectionEvent. Event raised when a movement direction is detected in a ROI by a crowd.

Both fluidity as occupancy are quantified in a relative metric from 0 to 100%. Then, both attributes are qualified into three categories: i) Minimum (min); ii) Medium (med); iii) Maximum (max).

Regarding direction, it is quantified as an angle (0-360º), where 0 is the direction from the central point of the video to the top (i.e., north), 90 correspond to the direction to the right (east), 180 is the south, and finally 270 is the west.

With all these concepts, now we can check out the Java server-side code of this demo. As depicted in the snippet below, we create a ROI by adding RelativePoint instances to a list. Each ROI is then stored into a list of RegionOfInterest instances.

Then, each ROI should be configured. To do that, we have the following methods:

fluidityLevelMin: Fluidity level (0-100%) for the category minimum.

fluidityLevelMed: Fluidity level (0-100%) for the category medium.

fluidityLevelMax: Fluidity level (0-100%) for the category maximum.

fluidityNumFramesToEvent: Number of consecutive frames detecting a fluidity level to rise a event.

occupancyLevelMin: Occupancy level (0-100%) for the category minimum.

occupancyLevelMed: Occupancy level (0-100%) for the category medium.

occupancyLevelMax: Occupancy level (0-100%) for the category maximum.

occupancyNumFramesToEvent: Number of consecutive frames detecting a occupancy level to rise a event.

sendOpticalFlowEvent: Boolean value that indicates whether or not directions events are going to be tracked by the filter. Be careful with this feature, since it is very demanding in terms of resource usage (CPU, memory) in the media server. Set to true this parameter only when you are going to need directions events in your client-side.

opticalFlowNumFramesToEvent: Number of consecutive frames detecting a direction level to rise a event.

opticalFlowNumFramesToReset: Number of consecutive frames detecting a occupancy level in which the counter is reset.

opticalFlowAngleOffset: Counterclockwise offset of the angle. This parameters is useful to move the default axis for directions (0º=north, 90º=east, 180º=south, 270º=west).

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

All in all, the media pipeline of this demo is is implemented as follows:

...
kurentoClient.register('kurento-module-crowddetector')
const RegionOfInterest       = kurentoClient.getComplexType('crowddetector.RegionOfInterest')
const RegionOfInterestConfig = kurentoClient.getComplexType('crowddetector.RegionOfInterestConfig')
const RelativePoint          = kurentoClient.getComplexType('crowddetector.RelativePoint')
...

kurentoClient(args.ws_uri, function(error, client) {
  if (error) return onError(error);

  client.create('MediaPipeline', function(error, p) {
    if (error) return onError(error);

    pipeline = p;

    console.log("Got MediaPipeline");

    pipeline.create('WebRtcEndpoint', function(error, webRtc) {
      if (error) return onError(error);

      console.log("Got WebRtcEndpoint");

      setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

      webRtc.processOffer(sdpOffer, function(error, sdpAnswer) {
        if (error) return onError(error);

        console.log("SDP answer obtained. Processing ...");

        webRtc.gatherCandidates(onError);

        webRtcPeer.processAnswer(sdpAnswer);
      });

      var options =
      {
        rois:
        [
          RegionOfInterest({
            id: 'roi1',
            points:
            [
              RelativePoint({x: 0,   y: 0}),
              RelativePoint({x: 0.5, y: 0}),
              RelativePoint({x: 0.5, y: 0.5}),
              RelativePoint({x: 0,   y: 0.5})
            ],
            regionOfInterestConfig: RegionOfInterestConfig({
              occupancyLevelMin: 10,
              occupancyLevelMed: 35,
              occupancyLevelMax: 65,
              occupancyNumFramesToEvent: 5,
              fluidityLevelMin: 10,
              fluidityLevelMed: 35,
              fluidityLevelMax: 65,
              fluidityNumFramesToEvent: 5,
              sendOpticalFlowEvent: false,
              opticalFlowNumFramesToEvent: 3,
              opticalFlowNumFramesToReset: 3,
              opticalFlowAngleOffset: 0
            })
          })
        ]
      }

      pipeline.create('crowddetector.CrowdDetectorFilter', options, function(error, filter)
      {
        if (error) return onError(error);

        console.log("Connecting...");

        filter.on('CrowdDetectorDirection', function (data){
          console.log("Direction event received in roi " + data.roiID +
             " with direction " + data.directionAngle);
        });

        filter.on('CrowdDetectorFluidity', function (data){
          console.log("Fluidity event received in roi " + data.roiID +
           ". Fluidity level " + data.fluidityPercentage +
           " and fluidity percentage " + data.fluidityLevel);
        });

        filter.on('CrowdDetectorOccupancy', function (data){
          console.log("Occupancy event received in roi " + data.roiID +
           ". Occupancy level " + data.occupancyPercentage +
           " and occupancy percentage " + data.occupancyLevel);
        });

        client.connect(webRtc, filter, webRtc, function(error){
          if (error) return onError(error);

          console.log("WebRtcEndpoint --> Filter --> WebRtcEndpoint");
        });
      });
    });
  });
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

The dependencies of this demo has to be obtained using Bower. The definition of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
   "kurento-module-pointerdetector": "6.10.0"
}

To get these dependencies, just run the following shell command:

bower install

Note

We are in active development. You can find the latest versions at Bower.

Node.js Module - Crowd Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a crowd detector filter. This filter detects people agglomeration in video streams.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-crowddetector should be also installed:

sudo apt-get install kms-crowddetector

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-crowddetector
git checkout 6.10.0
npm install

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a crowd in a WebRTC stream.

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with crowdDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a CrowdDetector instead of FaceOverlay filter.

To setup a CrowdDetectorFilter, first we need to define one or more region of interests (ROIs). A ROI delimits the zone within the video stream in which crowd are going to be tracked. To define a ROI, we need to configure at least three points. These points are defined in relative terms (0 to 1) to the video width and height.

CrowdDetectorFilter performs two actions in the defined ROIs. On the one hand, the detected crowd are colored over the stream. On the other hand, different events are raised to the client.

To understand crowd coloring, we can take a look to an screenshot of a running example of CrowdDetectorFilter. In the picture below, we can see that there are two ROIs (bounded with white lines in the video). On these ROIs, we can see two different colors over the original video stream: red zones are drawn over detected static crowds (or moving slowly). Blue zones are drawn over the detected crowds moving fast.

Crowd detection sample

Regarding crowd events, there are three types of events, namely:

CrowdDetectorFluidityEvent. Event raised when a certain level of fluidity is detected in a ROI. Fluidity can be seen as the level of general movement in a crowd.
CrowdDetectorOccupancyEvent. Event raised when a level of occupancy is detected in a ROI. Occupancy can be seen as the level of agglomeration in stream.
CrowdDetectorDirectionEvent. Event raised when a movement direction is detected in a ROI by a crowd.

Both fluidity as occupancy are quantified in a relative metric from 0 to 100%. Then, both attributes are qualified into three categories: i) Minimum (min); ii) Medium (med); iii) Maximum (max).

Regarding direction, it is quantified as an angle (0-360º), where 0 is the direction from the central point of the video to the top (i.e., north), 90 correspond to the direction to the right (east), 180 is the south, and finally 270 is the west.

With all these concepts, now we can check out the Java server-side code of this demo. As depicted in the snippet below, we create a ROI by adding RelativePoint instances to a list. Each ROI is then stored into a list of RegionOfInterest instances.

Then, each ROI should be configured. To do that, we have the following methods:

fluidityLevelMin: Fluidity level (0-100%) for the category minimum.

fluidityLevelMed: Fluidity level (0-100%) for the category medium.

fluidityLevelMax: Fluidity level (0-100%) for the category maximum.

fluidityNumFramesToEvent: Number of consecutive frames detecting a fluidity level to rise a event.

occupancyLevelMin: Occupancy level (0-100%) for the category minimum.

occupancyLevelMed: Occupancy level (0-100%) for the category medium.

occupancyLevelMax: Occupancy level (0-100%) for the category maximum.

occupancyNumFramesToEvent: Number of consecutive frames detecting a occupancy level to rise a event.

sendOpticalFlowEvent: Boolean value that indicates whether or not directions events are going to be tracked by the filter. Be careful with this feature, since it is very demanding in terms of resource usage (CPU, memory) in the media server. Set to true this parameter only when you are going to need directions events in your client-side.

opticalFlowNumFramesToEvent: Number of consecutive frames detecting a direction level to rise a event.

opticalFlowNumFramesToReset: Number of consecutive frames detecting a occupancy level in which the counter is reset.

opticalFlowAngleOffset: Counterclockwise offset of the angle. This parameters is useful to move the default axis for directions (0º=north, 90º=east, 180º=south, 270º=west).

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

All in all, the media pipeline of this demo is is implemented as follows:

...
kurento.register('kurento-module-crowddetector');
const RegionOfInterest       = kurento.getComplexType('crowddetector.RegionOfInterest');
const RegionOfInterestConfig = kurento.getComplexType('crowddetector.RegionOfInterestConfig');
const RelativePoint          = kurento.getComplexType('crowddetector.RelativePoint');
...

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint, filter) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, filter, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    filter.on('CrowdDetectorDirection', function (_data){
                        return callback(null, 'crowdDetectorDirection', _data);
                    });

                    filter.on('CrowdDetectorFluidity', function (_data){
                        return callback(null, 'crowdDetectorFluidity', _data);
                    });

                    filter.on('CrowdDetectorOccupancy', function (_data){
                        return callback(null, 'crowdDetectorOccupancy', _data);
                    });

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint
                        }
                        return callback(null, 'sdpAnswer', sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        var options = {
          rois: [
            RegionOfInterest({
              id: 'roi1',
              points: [
                RelativePoint({x: 0  , y: 0  }),
                RelativePoint({x: 0.5, y: 0  }),
                RelativePoint({x: 0.5, y: 0.5}),
                RelativePoint({x: 0  , y: 0.5})
              ],
              regionOfInterestConfig: RegionOfInterestConfig({
                occupancyLevelMin: 10,
                occupancyLevelMed: 35,
                occupancyLevelMax: 65,
                occupancyNumFramesToEvent: 5,
                fluidityLevelMin: 10,
                fluidityLevelMed: 35,
                fluidityLevelMax: 65,
                fluidityNumFramesToEvent: 5,
                sendOpticalFlowEvent: false,
                opticalFlowNumFramesToEvent: 3,
                opticalFlowNumFramesToReset: 3,
                opticalFlowAngleOffset: 0
              })
            })
          ]
        }
        pipeline.create('crowddetector.CrowdDetectorFilter', options, function(error, filter) {
            if (error) {
                return callback(error);
            }

            return callback(null, webRtcEndpoint, filter);
        });
    });
}

Dependencies¶

Dependencies of this demo are managed using NPM. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   "kurento-utils" : "6.10.0",
   "kurento-module-pointerdetector": "6.10.0"
}

Note

We are in active development. You can find the latest versions at npm and Bower.

Module Tutorial - Plate Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a plate detector filter element.

Java Module - Plate Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a plate detector filter element.

Note

This tutorial has been configured to use https. Follow the instructions to secure your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-platedetector should be also installed:

sudo apt-get install kms-platedetector

Warning

Plate detector module is a prototype and its results is not always accurate. Consider this if you are planning to use this module in a production environment.

To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:

git clone https://github.com/Kurento/kurento-tutorial-java.git
cd kurento-tutorial-java/kurento-platedetector
git checkout 6.10.0
mvn -U clean spring-boot:run

The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute the following command

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a plate in a WebRTC stream on optical character recognition (OCR).

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with plateDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PlateDetector instead of FaceOverlay filter. A screenshot of the running example is shown in the following picture:

Plate detector demo in action

The following snippet shows how the media pipeline is implemented in the Java server-side code of the demo. An important issue in this code is that a listener is added to the PlateDetectorFilter object (addPlateDetectedListener). This way, each time a plate is detected in the stream, a message is sent to the client side. As shown in the screenshot below, this event is printed in the console of the GUI.

private void start(final WebSocketSession session, JsonObject jsonMessage) {
   try {
      // Media Logic (Media Pipeline and Elements)
      UserSession user = new UserSession();
      MediaPipeline pipeline = kurento.createMediaPipeline();
      user.setMediaPipeline(pipeline);
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline)
            .build();
      user.setWebRtcEndpoint(webRtcEndpoint);
      users.put(session.getId(), user);

      webRtcEndpoint
            .addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {

               @Override
               public void onEvent(IceCandidateFoundEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "iceCandidate");
                  response.add("candidate", JsonUtils
                        .toJsonObject(event.getCandidate()));
                  try {
                     synchronized (session) {
                        session.sendMessage(new TextMessage(
                              response.toString()));
                     }
                  } catch (IOException e) {
                     log.debug(e.getMessage());
                  }
               }
            });

      PlateDetectorFilter plateDetectorFilter = new PlateDetectorFilter.Builder(
            pipeline).build();

      webRtcEndpoint.connect(plateDetectorFilter);
      plateDetectorFilter.connect(webRtcEndpoint);

      plateDetectorFilter
            .addPlateDetectedListener(new EventListener<PlateDetectedEvent>() {
               @Override
               public void onEvent(PlateDetectedEvent event) {
                  JsonObject response = new JsonObject();
                  response.addProperty("id", "plateDetected");
                  response.addProperty("plate", event.getPlate());
                  try {
                     session.sendMessage(new TextMessage(response
                           .toString()));
                  } catch (Throwable t) {
                     sendError(session, t.getMessage());
                  }
               }
            });

      // SDP negotiation (offer and answer)
      String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
      String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);

      // Sending response back to client
      JsonObject response = new JsonObject();
      response.addProperty("id", "startResponse");
      response.addProperty("sdpAnswer", sdpAnswer);

      synchronized (session) {
         session.sendMessage(new TextMessage(response.toString()));
      }
      webRtcEndpoint.gatherCandidates();
   } catch (Throwable t) {
      sendError(session, t.getMessage());
   }
}

Dependencies¶

This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:

<dependencies>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-client</artifactId>
   </dependency>
   <dependency>
      <groupId>org.kurento</groupId>
      <artifactId>kurento-utils-js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars</groupId>
      <artifactId>webjars-locator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>bootstrap</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>demo-console</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>adapter.js</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>jquery</artifactId>
   </dependency>
   <dependency>
      <groupId>org.webjars.bower</groupId>
      <artifactId>ekko-lightbox</artifactId>
   </dependency>
</dependencies>

Note

We are in active development. You can find the latest version of Kurento Java Client at Maven Central.

Kurento Java Client has a minimum requirement of Java 7. Hence, you need to include the following properties in your pom:

<maven.compiler.target>1.7</maven.compiler.target>
<maven.compiler.source>1.7</maven.compiler.source>

JavaScript Module - Plate Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a plate detector filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-platedetector should be also installed:

sudo apt-get install kms-platedetector

Warning

Plate detector module is a prototype and its results is not always accurate. Consider this if you are planning to use this module in a production environment.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

Due to Same-origin policy, this demo has to be served by an HTTP server. A very simple way of doing this is by means of an HTTP Node.js server which can be installed using npm :

sudo npm install http-server -g

You also need the source code of this demo. You can clone it from GitHub. Then start the HTTP server:

git clone https://github.com/Kurento/kurento-tutorial-js.git
cd kurento-tutorial-js/kurento-platedetector
git checkout 6.10.0
bower install
http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. Kurento Media Server must use WebSockets over SSL/TLS (WSS), so make sure you check this too. It is possible to locate the KMS in other machine simple adding the parameter ws_uri to the URL:

https://localhost:8443/index.html?ws_uri=wss://kms_host:kms_port/kurento

Notice that the Kurento Media Server must connected using a Secure WebSocket (i.e., the KMS URI starts with wss://). For this reason, the support for secure WebSocket must be enabled in the Kurento Media Server you are using to run this tutorial. For further information about securing applications, please visit the following page.

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a plate in a WebRTC stream on optical character recognition (OCR).

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with plateDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PlateDetector instead of FaceOverlay filter. An screenshot of the running example is shown in the following picture:

Plate detector demo in action

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

The following snippet shows how the media pipeline is implemented in the Java server-side code of the demo. An important issue in this code is that a listener is added to the PlateDetectorFilter object (addPlateDetectedListener). This way, each time a plate is detected in the stream, a message is sent to the client side. As shown in the screenshot below, this event is printed in the console of the GUI.

...
kurentoClient.register('kurento-module-platedetector')
...

kurentoClient(args.ws_uri, function(error, client) {
  if (error) return onError(error);

  client.create('MediaPipeline', function(error, _pipeline) {
    if (error) return onError(error);

    pipeline = _pipeline;

    console.log("Got MediaPipeline");

    pipeline.create('WebRtcEndpoint', function(error, webRtc) {
      if (error) return onError(error);

      console.log("Got WebRtcEndpoint");

      setIceCandidateCallbacks(webRtcPeer, webRtc, onError)

      webRtc.processOffer(sdpOffer, function(error, sdpAnswer) {
        if (error) return onError(error);

        console.log("SDP answer obtained. Processing...");

        webRtc.gatherCandidates(onError);
        webRtcPeer.processAnswer(sdpAnswer);
      });

      pipeline.create('platedetector.PlateDetectorFilter', function(error, filter) {
        if (error) return onError(error);

        console.log("Got Filter");

        filter.on('PlateDetected', function (data){
          console.log("License plate detected " + data.plate);
        });

        client.connect(webRtc, filter, webRtc, function(error) {
          if (error) return onError(error);

          console.log("WebRtcEndpoint --> filter --> WebRtcEndpoint");
        });
      });
    });
  });
});

Note

The TURN and STUN servers to be used can be configured simple adding the parameter ice_servers to the application URL, as follows:

https://localhost:8443/index.html?ice_servers=[{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
https://localhost:8443/index.html?ice_servers=[{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}]

Dependencies¶

The dependencies of this demo has to be obtained using Bower. The definition of these dependencies are defined in the bower.json file, as follows:

"dependencies": {
   "kurento-client": "6.10.0",
   "kurento-utils": "6.10.0"
   "kurento-module-pointerdetector": "6.10.0"
}

To get these dependencies, just run the following shell command:

bower install

Note

We are in active development. You can find the latest versions at Bower.

Node.js Module - Plate Detector Filter¶

This web application consists on a WebRTC video communication in mirror (loopback) with a plate detector filter element.

Note

This tutorial has been configurated for using https. Follow these instructions for securing your application.

For the impatient: running this example¶

First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information. In addition, the built-in module kms-platedetector should be also installed:

sudo apt-get install kms-platedetector

Warning

Plate detector module is a prototype and its results is not always accurate. Consider this if you are planning to use this module in a production environment.

Be sure to have installed Node.js and Bower in your system. In an Ubuntu machine, you can install both as follows:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To launch the application, you need to clone the GitHub project where this demo is hosted, install it and run it:

git clone https://github.com/Kurento/kurento-tutorial-node.git
cd kurento-tutorial-node/kurento-platedetector
git checkout 6.10.0
npm install

If you have problems installing any of the dependencies, please remove them and clean the npm cache, and try to install them again:

rm -r node_modules
npm cache clean

Finally, access the application connecting to the URL https://localhost:8443/ through a WebRTC capable browser (Chrome, Firefox).

Note

These instructions work only if Kurento Media Server is up and running in the same machine as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding the argument ws_uri to the npm execution command, as follows:

npm start -- --ws_uri=ws://kms_host:kms_port/kurento

In this case you need to use npm version 2. To update it you can use this command:

sudo npm install npm -g

Understanding this example¶

This application uses computer vision and augmented reality techniques to detect a plate in a WebRTC stream on optical character recognition (OCR).

The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:

WebRTC with plateDetector filter Media Pipeline

The complete source code of this demo can be found in GitHub.

This example is a modified version of the Magic Mirror tutorial. In this case, this demo uses a PlateDetector instead of FaceOverlay filter. An screenshot of the running example is shown in the following picture:

Plate detector demo in action

Note

Modules can have options. For configuring these options, you’ll need to get the constructor for them. In Javascript and Node, you have to use kurentoClient.getComplexType(‘qualifiedName’) . There is an example in the code.

The following snippet shows how the media pipeline is implemented in the Java server-side code of the demo. An important issue in this code is that a listener is added to the PlateDetectorFilter object (addPlateDetectedListener). This way, each time a plate is detected in the stream, a message is sent to the client side. As shown in the screenshot below, this event is printed in the console of the GUI.

...
kurento.register('kurento-module-platedetector');
...

function start(sessionId, ws, sdpOffer, callback) {
    if (!sessionId) {
        return callback('Cannot use undefined sessionId');
    }

    getKurentoClient(function(error, kurentoClient) {
        if (error) {
            return callback(error);
        }

        kurentoClient.create('MediaPipeline', function(error, pipeline) {
            if (error) {
                return callback(error);
            }

            createMediaElements(pipeline, ws, function(error, webRtcEndpoint, filter) {
                if (error) {
                    pipeline.release();
                    return callback(error);
                }

                if (candidatesQueue[sessionId]) {
                    while(candidatesQueue[sessionId].length) {
                        var candidate = candidatesQueue[sessionId].shift();
                        webRtcEndpoint.addIceCandidate(candidate);
                    }
                }

                connectMediaElements(webRtcEndpoint, filter, function(error) {
                    if (error) {
                        pipeline.release();
                        return callback(error);
                    }

                    webRtcEndpoint.on('OnIceCandidate', function(event) {
                        var candidate = kurento.getComplexType('IceCandidate')(event.candidate);
                        ws.send(JSON.stringify({
                            id : 'iceCandidate',
                            candidate : candidate
                        }));
                    });

                    filter.on('PlateDetected', function (data){
                        return callback(null, 'plateDetected', data);
                    });

                    webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
                        if (error) {
                            pipeline.release();
                            return callback(error);
                        }

                        sessions[sessionId] = {
                            'pipeline' : pipeline,
                            'webRtcEndpoint' : webRtcEndpoint
                        }
                        return callback(null, 'sdpAnswer', sdpAnswer);
                    });

                    webRtcEndpoint.gatherCandidates(function(error) {
                        if (error) {
                            return callback(error);
                        }
                    });
                });
            });
        });
    });
}

function createMediaElements(pipeline, ws, callback) {
    pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
        if (error) {
            return callback(error);
        }

        pipeline.create('platedetector.PlateDetectorFilter', function(error, filter) {
            if (error) {
                return callback(error);
            }

            return callback(null, webRtcEndpoint, filter);
        });
    });
}

Dependencies¶

Dependencies of this demo are managed using NPM. Our main dependency is the Kurento Client JavaScript (kurento-client). The relevant part of the package.json file for managing this dependency is:

"dependencies": {
   "kurento-client" : "6.10.0"
}

At the client side, dependencies are managed using Bower. Take a look to the bower.json file and pay attention to the following section:

"dependencies": {
   "kurento-utils" : "6.10.0",
   "kurento-module-pointerdetector": "6.10.0"
}

Note

We are in active development. You can find the latest versions at npm and Bower.

Kurento Utils JS¶

[TODO full review]

Overview¶

Kurento Utils is a wrapper object of an RTCPeerConnection. This object is aimed to simplify the development of WebRTC-based applications.

The source code of this project can be cloned from the GitHub repository.

How to use it¶

Minified file - Download the file from here.

NPM - Install and use library in your NodeJS files.

npm install kurento-utils

var utils = require('kurento-utils');

Bower - Generate the bundled script file

bower install kurento-utils

Import the library in your html page

<script
src="bower_components/kurento-utils/js/kurento-utils.js"></script>

Examples¶

There are several tutorials that show kurento-utils used in complete WebRTC applications developed on Java, Node and JavaScript. These tutorials are in GitHub, and you can download and run them at any time.

Java - https://github.com/Kurento/kurento-tutorial-java
Node - https://github.com/Kurento/kurento-tutorial-node
JavaScript - https://github.com/Kurento/kurento-tutorial-js

In the following lines we will show how to use the library to create an RTCPeerConnection, and how to negotiate the connection with another peer. The library offers a WebRtcPeer object, which is a wrapper of the browser’s RTCPeerConnection API. Peer connections can be of different types: unidirectional (send or receive only) or bidirectional (send and receive). The following code shows how to create the latter, in order to be able to send and receive media (audio and video). The code assumes that there are two video tags in the page that loads the script. These tags will be used to show the video as captured by your own client browser, and the media received from the other peer. The constructor receives a property that holds all the information needed for the configuration.

 var videoInput = document.getElementById('videoInput');
 var videoOutput = document.getElementById('videoOutput');

 var constraints = {
     audio: true,
     video: {
       width: 640,
       framerate: 15
     }
 };

 var options = {
   localVideo: videoInput,
   remoteVideo: videoOutput,
   onicecandidate : onIceCandidate,
   mediaConstraints: constraints
 };


var webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, function(error) {
      if(error) return onError(error)

      this.generateOffer(onOffer)
   });

With this little code, the library takes care of creating the RTCPeerConnection, and invoking getUserMedia in the browser if needed. The constraints in the property are used in the invocation, and in this case both microphone and webcam will be used. However, this does not create the connection. This is only achieved after completing the SDP negotiation between peers. This process implies exchanging SDPs offer and answer and, since Trickle ICE is used, a number of candidates describing the capabilities of each peer. How the negotiation works is out of the scope of this document. More info can be found in this link.

In the previous piece of code, when the webRtcPeer object gets created, the SDP offer is generated with this.generateOffer(onOffer). The only argument passed is a function, that will be invoked one the browser’s peer connection has generated that offer. The onOffer callback method is responsible for sending this offer to the other peer, by any means devised in your application. Since that is part of the signaling plane and business logic of each particular application, it won’t be covered in this document.

Assuming that the SDP offer has been received by the remote peer, it must have generated an SDP answer, that should be received in return. This answer must be processed by the webRtcEndpoint, in order to fulfill the negotiation. This could be the implementation of the onOffer callback function. We’ve assumed that there’s a function somewhere in the scope, that allows sending the SDP to the remote peer.

function onOffer(error, sdpOffer) {
  if (error) return onError(error);

  // We've made this function up sendOfferToRemotePeer(sdpOffer,
  function(sdpAnswer) {
    webRtcPeer.processAnswer(sdpAnswer);
  });
}

As we’ve commented before, the library assumes the use of Trickle ICE to complete the connection between both peers. In the configuration of the webRtcPeer, there is a reference to a onIceCandidate callback function. The library will use this function to send ICE candidates to the remote peer. Since this is particular to each application, we will just show the signature

function onIceCandidate(candidate) {
  // Send the candidate to the remote peer
}

In turn, our client application must be able to receive ICE candidates from the remote peer. Assuming the signaling takes care of receiving those candidates, it is enough to invoke the following method in the webRtcPeer to consider the ICE candidate

webRtcPeer.addIceCandidate(candidate);

Following the previous steps, we have:

Sent and SDP offer to a remote peer
Received an SDP answer from the remote peer, and have the webRtcPeer process that answer.
Exchanged ICE candidates between both peer, by sending the ones generated in the browser, and processing the candidates received by the remote peer.

This should complete the negotiation process, and should leave us with a working bidirectional WebRTC media exchange between both peers.

Using data channels¶

WebRTC data channels lets you send text or binary data over an active WebRTC connection. The WebRtcPeer object can provide access to this functionality by using the RTCDataChannel form the wrapped RTCPeerConnection object. This allows you to inject into and consume data from the pipeline. This data can be treated by each endpoint differently. For instance, a WebRtcPeer object in the browser, will have the same behavior as the RTCDataChannel (you can see a description here). Other endpoints could make use of this channel to send information: a filter that detects QR codes in a video stream, could send the detected code to the clients through a data channel. This special behavior should be specified in the filter.

The use of data channels in the WebRtcPeer object is indicated by passing the dataChannels flag in the options bag, along with the desired options.

 var options = {
     localVideo : videoInput,
     remoteVideo : videoOutput,
     dataChannels : true,
     dataChannelConfig: {
       id : getChannelName(),
       onmessage : onMessage,
       onopen : onOpen,
       onclose : onClosed,
       onbufferedamountlow : onbufferedamountlow,
       onerror : onerror
     },
     onicecandidate : onIceCandidate
 }

 webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, onWebRtcPeerCreated);

The values in dataChannelConfig are all optional. Once the webRtcPeer object is created, and after the connection has been successfully negotiated, users can send data through the data channel

webRtcPeer.send('your data stream here');

The format of the data you are sending, is determined by your application, and the definition of the endpoints that you are using.

The lifecycle of the underlying RTCDataChannel, is tied to that of the webRtcPeer: when the webRtcPeer.dispose() method is invoked, the data channel will be closed and released too.

Reference documentation¶

WebRtcPeer¶

The constructor for WebRtcPeer is WebRtcPeer(mode, options, callback) where:

mode: Mode in which the PeerConnection will be configured. Valid values are
- recv: receive only media.
- send: send only media.
- sendRecv: send and receive media.
options : It is a group of parameters and they are optional. It is a json object.
- localVideo: Video tag in the application for the local stream.
- remoteVideo: Video tag in the application for the remote stream.
- videoStream: Provides an already available video stream that will be used instead of using the media stream from the local webcam.
- audioStreams: Provides an already available audio stream that will be used instead of using the media stream from the local microphone.
- mediaConstraints: Defined the quality for the video and audio
- connectionConstraints: Defined the connection constraint according with browser like googIPv6, DtlsSrtpKeyAgreement…
- peerConnection: Use a peerConnection which was created before
- sendSource: Which source will be used
  webcam
  
  screen
  
  window
- onstreamended: Method that will be invoked when stream ended event happens
- onicecandidate: Method that will be invoked when ice candidate event happens
- oncandidategatheringdone: Method that will be invoked when all candidates have been harvested
- dataChannels: Flag for enabling the use of data channels. If true, then a data channel will be created in the RTCPeerConnection object.
- dataChannelConfig: It is a JSON object with the configuration passed to the DataChannel when created. It supports the following keys:
  id: Specifies the id of the data channel. If none specified, the same id of the WebRtcPeer object will be used.
  
  options: Options object passed to the data channel constructor.
  
  onopen: Function invoked in the onopen event of the data channel, fired when the channel is open.
  
  onclose: Function invoked in the onclose event of the data channel, fired when the data channel is closed.
  
  onmessage: Function invoked in the onmessage event of the data channel. This event is fired every time a message is received.
  
  onbufferedamountlow: Is the event handler called when the bufferedamountlow event is received. Such an event is sent when RTCDataChannel.bufferedAmount drops to less than or equal to the amount specified by the RTCDataChannel.bufferedAmountLowThreshold property.
  
  onerror: Callback function onviked when an error in the data channel is produced. If none is provided, an error trace message will be logged in the browser console.
- simulcast: Indicates whether simulcast is going to be used. Value is true|false
- configuration: It is a JSON object where ICE Servers are defined using
  iceServers: The format for this variable is like:
  
  [{"urls":"turn:turn.example.org","username":"user","credential":"myPassword"}] [{"urls":"stun:stun1.example.net"},{"urls":"stun:stun2.example.net"}]
callback: It is a callback function which indicate, if all worked right or not

Also there are 3 specific methods for creating WebRtcPeer objects without using mode parameter:

WebRtcPeerRecvonly(options, callback): Create a WebRtcPeer as receive only.

WebRtcPeerSendonly(options, callback): Create a WebRtcPeer as send only.

WebRtcPeerSendrecv(options, callback): Create a WebRtcPeer as send and receive.

MediaConstraints¶

Constraints provide a general control surface that allows applications to both select an appropriate source for a track and, once selected, to influence how a source operates. getUserMedia() uses constraints to help select an appropriate source for a track and configure it. For more information about media constraints and its values, you can check here.

By default, if the mediaConstraints is undefined, this constraints are used when getUserMedia is called:

{
  audio: true,
  video: {
    width: 640,
    framerate: 15
  }
}

If mediaConstraints has any value, the library uses this value for the invocation of getUserMedia. It is up to the browser whether those constraints are accepted or not.

In the examples section, there is one example about the use of media constraints.

Methods¶

getPeerConnection¶

Using this method the user can get the peerConnection and use it directly.

showLocalVideo¶

Use this method for showing the local video.

getLocalStream¶

Using this method the user can get the local stream. You can use muted property to silence the audio, if this property is true.

getRemoteStream¶

Using this method the user can get the remote stream.

getCurrentFrame¶

Using this method the user can get the current frame and get a canvas with an image of the current frame.

processAnswer¶

Callback function invoked when a SDP answer is received. Developers are expected to invoke this function in order to complete the SDP negotiation. This method has two parameters:

sdpAnswer: Description of sdpAnswer
callback: It is a function with error like parameter. It is called when the remote description has been set successfully.

processOffer¶

Callback function invoked when a SDP offer is received. Developers are expected to invoke this function in order to complete the SDP negotiation. This method has two parameters:

sdpOffer: Description of sdpOffer
callback: It is a function with error and sdpAnswer like parameters. It is called when the remote description has been set successfully.

dispose¶

This method frees the resources used by WebRtcPeer.

addIceCandidate¶

Callback function invoked when an ICE candidate is received. Developers are expected to invoke this function in order to complete the SDP negotiation. This method has two parameters:

iceCandidate: Literal object with the ICE candidate description
callback: It is a function with error like parameter. It is called when the ICE candidate has been added.

getLocalSessionDescriptor¶

Using this method the user can get peerconnection’s local session descriptor.

getRemoteSessionDescriptor¶

Using this method the user can get peerconnection’s remote session descriptor.

generateOffer¶

Creates an offer that is a request to find a remote peer with a specific configuration.

Souce code¶

The code is at github.

Be sure to have Node.js and Bower installed in your system:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g bower

To install the library, it is recommended to do that from the NPM repository:

npm install kurento-utils

Alternatively, you can download the code using Git and install manually its dependencies:

git clone https://github.com/Kurento/kurento-utils
cd kurento-utils
npm install

Build for browser¶

After you download the project, to build the browser version of the library you’ll only need to execute the grunt task runner. The file needed will be generated on the dist folder. Alternatively, if you don’t have it globally installed, you can run a local copy by executing:

cd kurento-utils
node_modules/.bin/grunt

Endpoint Events ¶

This is a list of all events that can be emitted by an instance of WebRtcEndpoint. This class belongs to a chain of inherited classes, so this list includes events from all of them, starting from the topmost class in the inheritance tree:

Table of Contents

MediaObject events ¶

This is the base interface used to manage capabilities common to all Kurento elements, including both MediaElement and MediaPipeline.

Error ¶

Some error has occurred. Check the event parameters (such as description, errorCode, and type) to get information about what happened.

MediaElement events ¶

These events indicate some low level information about the state of GStreamer, the underlying multimedia framework.

ElementConnected ¶

[TODO - add contents]

ElementDisconnected ¶

[TODO - add contents]

MediaFlowInStateChange ¶

State = Flowing: Data is arriving from the KMS Pipeline, and flowing into the Element. Technically, this means that there are GStreamer Buffers flowing from the Pipeline to the Element’s sink pad. For example, with a Recorder element this event would fire when media arrives from the Pipeline to be written to disk.
State = NotFlowing: The Element is not receiving any input data from the Pipeline.

MediaFlowOutStateChange ¶

State = Flowing: There is data flowing out from the Element towards the KMS Pipeline. Technically, this means that there are GStreamer Buffers flowing from the Element’s src pad to the Pipeline. For example, with a Player element this event would fire when media is read from disk and is pushed to the Pipeline.
State = NotFlowing: The Element is not sending any output data to the Pipeline.

MediaTranscodingStateChange ¶

All Endpoint objects in Kurento Media Server embed a custom-made GStreamer element called agnosticbin. This element is used to provide seamless interconnection of components in the MediaPipeline, regardless of the format and codec configuration of the input and output media streams.

When media starts flowing through any MediaElement-derived object, an internal dynamic configuration is automatically done in order to match the incoming and outgoing media formats. If both input and output formats are compatible (at the codec level), then the media can be transferred directly without any extra processing. However, if the input and output media formats don’t match, the internal transcoding module will get enabled to convert between them.

For example: If a WebRtcEndpoint receives a VP8 video stream from a Chrome browser, and has to send it to a Safari browser (which only supports the H.264 codec), then the media needs to be transcoded. The WebRtcEndpoint will automatically do it.

State = Transcoding: The MediaElement will transcode the incoming media, because its format is not compatible with the requested output.
State = NotTranscoding: The MediaElement will not transcode the incoming media, because its format is compatible with the requested output.

BaseRtpEndpoint events ¶

These events provide information about the state of the RTP connection for each stream in the WebRTC call.

Note that the MediaStateChanged event is not 100% reliable to check if a RTP connection is active: RTCP packets do not usually flow at a constant rate. For example, minimizing a browser window with an RTCPeerConnection might affect this interval.

ConnectionStateChanged ¶

State = Connected: All of the KmsIRtpConnection objects have been created [TODO: explain what this means].
State = Disconnected: At least one of the KmsIRtpConnection objects is not created yet.

Call sequence:

signal KmsIRtpConnection::"connected"
  -> signal KmsSdpSession::"connection-state-changed"
    -> signal KmsBaseRtpEndpoint::"connection-state-changed"
      -> BaseRtpEndpointImpl::updateConnectionState

MediaStateChanged ¶

State = Connected: At least one of the audio or video RTP streams in the session is still alive (sending or receiving RTCP packets). Equivalent to the signal GstRtpBin::”on-ssrc-active”, which gets triggered whenever the GstRtpBin receives an RTCP Sender Report (RTCP SR) or RTCP Receiver Report (RTCP RR).
State = Disconnected: None of the RTP streams belonging to the session is alive (ie. no RTCP packets are sent or received for any of them).

These signals from GstRtpBin will trigger the MediaStateChanged event:

GstRtpBin::"on-bye-ssrc": State = Disconnected.
GstRtpBin::"on-bye-timeout": State = Disconnected.
GstRtpBin::"on-timeout": State = Disconnected.
GstRtpBin::"on-ssrc-active": State = Connected.

Call sequence:

signal GstRtpBin::"on-bye-ssrc"
|| signal GstRtpBin::"on-bye-timeout"
|| signal GstRtpBin::"on-timeout"
|| signal GstRtpBin::"on-ssrc-active"
  -> signal KmsBaseRtpEndpoint::"media-state-changed"
    -> BaseRtpEndpointImpl::updateMediaState

Note

MediaStateChanged (State = Connected) will happen after these other events have been emitted:

NewCandidatePairSelected.
IceComponentStateChanged (State: Connected).
MediaFlowOutStateChange (State: Flowing).

WebRtcEndpoint events ¶

These events provide information about the state of libnice, the underlying library in charge of the ICE Gathering process. The ICE Gathering is typically done before attempting any WebRTC call.

For further reference, see the libnice’s Agent documentation and source code.

DataChannelClose ¶

[TODO - add contents]

DataChannelOpen ¶

[TODO - add contents]

IceCandidateFound ¶

A new local candidate has been found, after the ICE Gathering process was started. Equivalent to the signal NiceAgent::”new-candidate-full”.

IceComponentStateChange ¶

This event carries the state values from the signal NiceAgent::”component-state-changed”.

State = Disconnected: There is no active connection, and the ICE process is idle.

NiceAgent state: NICE_COMPONENT_STATE_DISCONNECTED, “No activity scheduled”.
State = Gathering: The Endpoint has started finding all possible local candidates, which will be notified through the event IceCandidateFound.

NiceAgent state: NICE_COMPONENT_STATE_GATHERING, “Gathering local candidates”.
State = Connecting: The Endpoint has started the connectivity checks between at least one pair of local and remote candidates. These checks will always start as soon as possible (i.e. whenever the very first remote candidates arrive), so don’t assume that the candidate gathering has already finished, because it will probably still be running in parallel; some (possibly better) candidates might still be waiting to be found and gathered.

NiceAgent state: NICE_COMPONENT_STATE_CONNECTING, “Establishing connectivity”.
State = Connected: At least one candidate pair resulted in a successful connection. This happens right after the event NewCandidatePairSelected. When this event triggers, the effective communication between peers can start, and usually this means that media will start flowing between them. However, the candidate gathering hasn’t really finished yet, which means that some (possibly better) candidates might still be waiting to be found, gathered, checked for connectivity, and if that completes successfully, selected as new candidate pair.

NiceAgent state: NICE_COMPONENT_STATE_CONNECTED, “At least one working candidate pair”.
State = Ready: All local candidates have been gathered, all pairs of local and remote candidates have been tested for connectivity, and a successful connection was established.

NiceAgent state: NICE_COMPONENT_STATE_READY, “ICE concluded, candidate pair selection is now final”.
State = Failed: All local candidates have been gathered, all pairs of local and remote candidates have been tested for connectivity, but still none of the connection checks was successful, so no connectivity was reached to the remote peer.

NiceAgent state: NICE_COMPONENT_STATE_FAILED, “Connectivity checks have been completed, but connectivity was not established”.

This graph shows the possible state changes (source):

$// libnice state transition diagram for NiceComponentState digraph NiceComponentState { bgcolor = "transparent"; rankdir = "TB"; node [shape = "doublecircle"]; "DISCONNECTED"; node [shape = "circle"]; // Successful path -> green edge [color = "chartreuse3"]; "DISCONNECTED" -> "GATHERING" [label = "nice_agent_gather_candidates"]; "GATHERING" -> "CONNECTING" [label = "nice_agent_set_remote_candidates"]; "CONNECTING" -> "CONNECTED" [label = "At least one candidate pair succeeds"]; "CONNECTED" -> "READY" [label = "All candidate pairs checks finished"]; // Normal control flow -> black edge [color = "black"]; "READY" -> "CONNECTED" [label = "Selected candidate pair fails"]; "FAILED" -> "CONNECTING" [xlabel = "nice_agent_set_remote_candidates"]; "DISCONNECTED" -> "CONNECTING" [label = "nice_agent_set_remote_candidates"]; // Failure paths -> gray edge [xlabel = "Failure", color = "gray"]; "DISCONNECTED" -> "FAILED"; "GATHERING" -> "FAILED"; "CONNECTING" -> "FAILED"; "CONNECTED" -> "FAILED"; "READY" -> "FAILED"; }$

libnice state transition diagram for NiceComponentState¶

Note

The states Ready and Failed indicate that the ICE transport has completed gathering and is currently idle. However, since events such as adding a new interface or a new TURN server will cause the state to go back, Ready and Failed are not terminal states.

IceGatheringDone ¶

All local candidates have been found, so the gathering process is finished for this peer. Note this doesn’t imply that the remote peer has finished its own gathering, so more remote candidates might still arrive. Equivalent to the signal NiceAgent::”candidate-gathering-done”.

NewCandidatePairSelected ¶

During the connectivity checks one of the pairs happened to provide a successful connection, and the pair had a higher preference than the previously selected one (or there was no previously selected pair yet). Equivalent to the signal NiceAgent::”new-selected-pair”.

Sample sequence of events: WebRtcEndpoint ¶

Once an instance of WebRtcEndpoint is created inside a Media Pipeline, an event handler should be added for each one of the events that can be emitted by the endpoint. Later, the endpoint should be instructed to do one of either:

Generate an SDP Offer, when KMS is the caller. Later, the remote peer will generate an SDP Answer as a reply, which must be provided to the endpoint.
Process an SDP Offer generated by the remote peer, when KMS is the callee. This will in turn generate an SDP Answer, which should be provided to the remote peer.

As a last step, the WebRtcEndpoint should be instructed to start the ICE Gathering process.

You can see a working example of this in Kurento Java Tutorial - Hello World. This example code shows the typical usage for the WebRtcEndpoint:

KurentoClient kurento;
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEp = new WebRtcEndpoint.Builder(pipeline).build();
webRtcEp.addIceCandidateFoundListener(...);
webRtcEp.addIceComponentStateChangedListener(...);
webRtcEp.addIceGatheringDoneListener(...);
webRtcEp.addNewCandidatePairSelectedListener(...);

// Receive an SDP Offer, via the application's custom signaling mechanism
String sdpOffer = recvMessage();

// Process the SDP Offer, generating an SDP Answer
String sdpAnswer = webRtcEp.processOffer(sdpOffer);

// Send the SDP Answer, via the application's custom signaling mechanism
sendMessage(sdpAnswer);

// Start gathering candidates for ICE
webRtcEp.gatherCandidates();

The application’s custom signaling mechanism could be as simple as some ad-hoc messaging protocol built upon WebSocket endpoints.

When a WebRtcEndpoint instance has been created, and all event handlers have been added, starting the ICE process will generate a sequence of events very similar to this one:

IceCandidateFound
IceComponentStateChanged (Gathering)
AddIceCandidate
IceComponentStateChanged (Connecting)
AddIceCandidate
IceCandidateFound
NewCandidatePairSelected
IceComponentStateChanged (Connected)
NewCandidatePairSelected
IceGatheringDone
IceComponentStateChanged: (Ready)

IceCandidateFound

Repeated multiple times; tipically, candidates of type host (corresponding to the LAN, local network) are almost immediately found after starting the ICE gathering, and this event can arrive even before the event IceComponentStateChanged is emitted.
IceComponentStateChanged (state: Gathering)

At this point, the local peer is gathering more candidates, and it is also waiting for the candidates gathered by the remote peer, which could start arriving at any time.
AddIceCandidate

Repeated multiple times; the remote peer found some initial candidates, and started sending them. Typically, the first candidate received is of type host, because those are found the fastest.
IceComponentStateChanged (state: Connecting)

After receiving the very first of the remote candidates, the ICE Agent starts with the connectivity checks.
AddIceCandidate

Repeated multiple times; the remote peer will continue sending its own gathered candidates, of any type: host, srflx (STUN), relay (TURN).
IceCandidateFound

Repeated multiple times; the local peer will also continue finding more of the available local candidates.
NewCandidatePairSelected

The ICE Agent makes local and remote candidate pairs. If one of those pairs pass the connectivity checks, it is selected for the WebRTC connection.
IceComponentStateChanged (state: Connected)

After selecting a candidate pair, the connection is established. At this point, the media stream(s) can start flowing.
NewCandidatePairSelected

Typically, better candidate pairs will be found over time. The old pair will be abandoned in favor of the new one.
IceGatheringDone

When all candidate pairs have been tested, no more work is left to do for the ICE Agent. The gathering process is finished.
IceComponentStateChanged (state: Ready)

As a consequence of finishing the ICE gathering, the component state gets updated.

NAT Traversal¶

NAT Traversal, also known as Hole Punching, is the procedure of opening an inbound port in the NAT tables of the routers which implement this technology (which are the vast majority of home and corporate routers).

There are different types of NAT, depending on how they behave: Full Cone, Address-Restricted Cone, Port-Restricted Cone, and Symmetric. For a comprehensive explanation of NAT and the different types that exist, please read our Knowledge Base document: NAT Types and NAT Traversal.

WebRTC with ICE¶

ICE is the standard method used by WebRTC to solve the issue of NAT Traversal. Kurento supports ICE by means of a 3rd-party library: libnice, The GLib ICE implementation.

Refer to the logging documentation if you need to enable the debug logging for this library.

RTP without ICE¶

KMS is able to automatically infer what is the public IP and port of any remote peer which is communicating with it through an RTP connection. This removes the need to use ICE in some specific situations, where that complicated mechanism is not desired. This new automatic port discovery was inspired by the Connection-Oriented Media Transport (COMEDIA) as presented by the early Drafts of what finally would become the RFC 4145.

TCP-Based Media Transport in the Session Description Protocol (SDP) (IETF RFC 4145) defines an SDP extension which adds TCP connections and procedures, such as how a passive machine would wait for connections from a remote active machine and be able to obtain connection information from the active one, upon reception of an initial connection.

Early Drafts of RFC 4145 (up to Draft 05) also contemplated the usage of this same concept of “Connection-Oriented Media Transport in SDP” with UDP connections, as a way of aiding NAT traversal. This is what has been used as a basis for the implementation of automatic port discovery in KMS.

It works as follows:

The machine behind a NAT router acts as the active peer. It sends an SDP Offer to the other machine, the passive peer.
1. Sending an SDP Offer from behind a NAT means that the IP and port specified in the SDP message are actually just the private IP and port of that machine, instead of the public ones. The passive peer won’t be able to use these to communicate back to the active peer. Due to this, the SDP Offer states the port 9 (Discard port) instead of whatever port the active machine will be using.
2. The SDP Offer includes the media-level attribute a=direction:active, so the passive peer is able to acknowledge that the Connection-Oriented Media Transport is being used for that media, and it writes a=direction:passive in its SDP Answer.
The passive peer receives the SDP Offer and answers it as usual, indicating the public IP and port where it will be listening for incoming packets. Besides that, it must ignore the IP and port indicated in the received SDP Offer. Instead, it must enter a wait state, until the active peer starts sending some packets.
When the active peer sends the first RTP/RTCP packets to the IP and port specified in the SDP Answer, the passive peer will be able to analyze them on reception and extract the public IP and reception port of the active peer.
The passive peer is now able to send RTP/RTCP packets to the discovered IP and port values of the active peer.

This mechanism has the following requisites and/or limitations:

Only the active peer can be behind a NAT router. The passive peer must have a publicly accessible IP and port for RTP.
The active peer must be able to receive RTP/RTCP packets at the same ports that are used to send RTP/RTCP packets. In other words, the active peer must be compatible with Symmetric RTP and RTCP as defined in IETF RFC 4961.
The active peer must actually do send some RTP/RTCP packets before the passive peer is able to send any data back. In other words, it is not possible to establish a one-way stream where only the passive peer sends data to the active peer.

This is how to enable the Connection-Oriented Media Transport mode:

The SDP Offer must be sent from the active peer to the passive peer.
The IP stated in the SDP Offer can be anything (as it will be ignored), so 0.0.0.0 can be used.
The Port stated in the SDP Offer should be 9 (Discard port).
The active peer must include the media-level attribute a=direction:active in the SDP Offer, for each media that requires automatic port discovery.
The passive peer must acknowledge that it supports the automatic port discovery mode, by including the media-level attribute a=direction:passive in its SDP Answer. As per normal rules of the SDP Offer/Answer Model (IETF RFC 3264), if this attribute is not present in the SDP Answer, then the active peer must assume that the passive peer is not compatible with this functionality and should react to this fact as whatever is deemed appropriate by the application developer.

Example¶

This is a minimal example of an SDP Offer/Answer negotiation that a machine would perform with KMS from behind a NAT router. The highlighted lines are those relevant to NAT Traversal:

SDP Offer¶

v=0
o=- 0 0 IN IP4 0.0.0.0
s=Example sender
c=IN IP4 0.0.0.0
t=0 0
m=audio 9 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=sendonly
a=direction:active
a=ssrc:111111 cname:active@example.com
m=video 9 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=sendonly
a=direction:active
a=ssrc:222222 cname:active@example.com

This is what KMS would answer:

SDP Answer¶

v=0
o=- 3696336115 3696336115 IN IP4 80.28.30.32
s=Kurento Media Server
c=IN IP4 80.28.30.32
t=0 0
m=audio 56740 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=recvonly
a=direction:passive
a=ssrc:4061617641 cname:user885892801@host-b546a6e8
m=video 37616 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=recvonly
a=direction:passive
a=ssrc:1363449382 cname:user885892801@host-b546a6e8

In this particular example, KMS is installed in a server with the public IP 80.28.30.32; also, it won’t be sending media to the active peer, only receiving it (as requested by the application with a=sendonly, and acknowledged by KMS with a=recvonly).

Note that even in this case, KMS still needs to know on what port the sender is listening for RTCP feedback packets, which are a mandatory part of the RTP protocol. So, in this example, KMS will learn the public IP and port of the active machine, and will use those to send the Receiver Report RTCP packets to the sender.

Securing Kurento Applications¶

[TODO full review]

Starting with Chrome 47, WebRTC is only allowed from SECURE ORIGINS (HTTPS or localhost). Check their release notes for further information about this issue.

Note

Keep in mind that serving your application through HTTPS, forces you to use WebSockets Secure (WSS) if you are using websockets to control your application server.

Securing Application Servers¶

Configure a Java server to use HTTPS¶

The application needs a certificate in order to enable HTTPS:
- Request a certificate from a local certification authority.
- Create an self-signed certificate.
  keytool -genkey -keyalg RSA -alias selfsigned -keystore \ keystore.jks -storepass password -validity 360 -keysize 2048
Use the certificate in your application:
- Include a valid keystore in the jar file:
  File keystore.jks must be in the project’s root path, and a file named application.properties must exist in src/main/resources/, with the following content:
  
  server.port: 8443 server.ssl.key-store: keystore.jks server.ssl.key-store-password: yourPassword server.ssl.keyStoreType: JKS server.ssl.keyAlias: yourKeyAlias
You can also specify the location of the properties file. When launching your Spring-Boot based app, issue the flag -Dspring.config.location=<path-to-properties> .
Start application

mvn -U clean spring-boot:run -Dkms.url=ws://kms_host:kms_port/kurento

Note

If you plan on using a webserver as proxy, like Nginx or Apache, you’ll need to setAllowedOrigins when registering the handler. Please read the official Spring documentation entry for more info.

Configure a Node server to use HTTPS¶

The application requires a valid SSL certificate in order to enable HTTPS:
- Request a certificate from a local certification authority.
- Create your own self-signed certificate as explained here. This will show you how to create the required files: server.crt, server.key and server.csr.

Add the following changes to server.js in order to enable HTTPS:

...
var express = require('express');
var ws = require('ws');
var fs    = require('fs');
var https = require('https');
...

var options =
{
  key:  fs.readFileSync('key/server.key'),
  cert: fs.readFileSync('keys/server.crt')
};

var app = express();

var server = https.createServer(options, app).listen(port, function() {
...
});
...

var wss = new ws.Server({
 server : server,
 path : '/'
});

wss.on('connection', function(ws) {

....

Start application

npm start

Configure Javascript applications to use HTTPS¶

You’ll need to provide a valid SSL certificate in order to enable HTTPS:
- Request a certificate from a local certification authority.
- Create your own self-signed certificate as explained here. This will show you how to create the required files: server.crt, server.key and server.csr.
Start the application using the certificates:

http-server -p 8443 -S -C keys/server.crt -K keys/server.key

Securing application servers¶

Configure Kurento Media Server to use Secure WebSocket (WSS)¶

First, you need to change the configuration file of Kurento Media Server, i.e. /etc/kurento/kurento.conf.json, uncommenting the following lines:

"secure": {
  "port": 8433,
  "certificate": "defaultCertificate.pem",
  "password": ""
},

If this PEM certificate is a signed certificate (by a Certificate Authority such as Verisign), then you are done. If you are going to use a self-signed certificate (suitable for development), then there is still more work to do.

You can generate a self signed certificate by doing this:

certtool --generate-privkey --outfile defaultCertificate.pem
echo 'organization = your organization name' > certtool.tmpl
certtool --generate-self-signed --load-privkey defaultCertificate.pem \
   --template certtool.tmpl >> defaultCertificate.pem
sudo chown kurento defaultCertificate.pem

Due to the fact that the certificate is self-signed, applications will reject it by default. For this reason, you’ll need to force them to accept it.

Browser applications: You’ll need to manually accept the certificate as trusted one before secure WebSocket connections can be established. By default, this can be done by connecting to connecting to https://localhost:8433/kurento and accepting the certificate in the browser.
Java applications, follow the instructions of this link (get InstallCert.java from here). You’ll need to instruct the KurentoClient needs to be configured to allow the use of certificates. For this purpose, we need to create our own JsonRpcClient:

SslContextFactory sec = new SslContextFactory(true);
sec.setValidateCerts(false);
JsonRpcClientWebSocket rpcClient = new JsonRpcClientWebSocket(uri, sec);
KurentoClient kuretoClient = KurentoClient.createFromJsonRpcClient(rpcClient);

Node applications, please take a look to this page.

Second, you have to change the WebSocket URI in your application logic. For instance, in the hello-world application within the tutorials, this would be done as follows:

Java: Changing this line in HelloWorldApp.java:

final static String DEFAULT_KMS_WS_URI = "wss://localhost:8433/kurento";

Browser JavaScript: Changing this line in index.js:

const ws_uri = 'wss://' + location.hostname + ':8433/kurento';

Node.js: Changing this line in server.js:

const ws_uri = "wss://localhost:8433/kurento";

WebRTC Statistics¶

[TODO full review]

Introduction¶

WebRTC streams (audio, video, or data) can be lost, and experience varying amounts of network delay. In order to assess the performance of WebRTC applications, it could be required to be able to monitor the WebRTC features of the underlying network and media pipeline.

To that aim, Kurento provides WebRTC statistics gathering for the server-side (Kurento Media Server, KMS). The implementation of this capability follows the guidelines provided in the W3C WebRTC’s Statistics API. Therefore, the statistics gathered in the KMS can be divided into two groups:

inboundrtp: statistics on the stream received in the KMS.
outboundrtp: statistics on the stream sent by KMS.

API description¶

As usual, WebRTC statistics gathering capability is provided by the KMS and is consumed by means of the different Kurento client implementations (Java, JavaScript clients are provided out of the box). To read these statistics, first it should be enabled using the method setLatencyStats of a Media Pipeline object. Using the Kurento Java client this is done as follows:

String kmsWsUri = "ws://localhost:8888/kurento";
KurentoClient kurentoClient = KurentoClient.create(kmsWsUri);
MediaPipeline mediaPipeline = kurentoClient.createMediaPipeline();
mediaPipeline.setLatencyStats(true);

// ...

… and using the JavaScript client:

var kmsWsUri = "ws://localhost:8888/kurento";
kurentoClient(kmsWsUri, function(error, kurentoClient) {
   kurentoClient.create("MediaPipeline", function(error, mediaPipeline) {
      mediaPipeline.setLatencyStats(true, function(error){

         // ...

      });
   });
});

Once WebRTC statistics are enabled, the second step is reading the statistics values using the method getStats of a Media Element, For example, to read the statistics of a WebRtcEndpoint object in Java:

WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(mediaPipeline).build();
MediaType mediaType = ... // it can be MediaType.VIDEO, MediaType.AUDIO, or MediaType.DATA
Map<String, Stats> statsMap = webRtcEndpoint.getStats(mediaType);

// ...

… and in JavaScript:

mediaPipeline.create("WebRtcEndpoint", function(error, webRtcEndpoint) {
   var mediaType = ... // it can be 'VIDEO', 'AUDIO', or 'DATA'
   webRtcEndpoint.getStats(mediaType, function(error, statsMap) {

      // ...

   });
});

Notice that the WebRTC statistics are read as a map. Therefore, each entry of this collection has a key and a value, in which the key is the specific statistic, with a given value at the reading time. Take into account that these values make reference to real-time properties, and so these values vary in time depending on multiple factors (for instance network performance, KMS load, and so on). The complete description of the statistics are defined in the KMD interface description. The most relevant statistics are listed below:

ssrc: The synchronized source (SSRC).
firCount: Count the total number of Full Intra Request (FIR) packets received by the sender. This metric is only valid for video and is sent by receiver.
pliCount: Count the total number of Packet Loss Indication (PLI) packets received by the sender and is sent by receiver.
nackCount: Count the total number of Negative ACKnowledgement (NACK) packets received by the sender and is sent by receiver.
sliCount: Count the total number of Slice Loss Indication (SLI) packets received by the sender. This metric is only valid for video and is sent by receiver.
remb: The Receiver Estimated Maximum Bitrate (REMB). This metric is only valid for video.
packetsLost: Total number of RTP packets lost for this SSRC.
packetsReceived: Total number of RTP packets received for this SSRC.
bytesReceived: Total number of bytes received for this SSRC.
jitter: Packet Jitter measured in seconds for this SSRC.
packetsSent: Total number of RTP packets sent for this SSRC.
bytesSent: Total number of bytes sent for this SSRC.
targetBitrate: Presently configured bitrate target of this SSRC, in bits per second.
roundTripTime: Estimated round trip time (seconds) for this SSRC based on the RTCP timestamp.
audioE2ELatency: End-to-end audio latency measured in nano seconds.
videoE2ELatency: End-to-end video latency measured in nano seconds.

All in all, the process for gathering WebRTC statistics in the KMS can be summarized in two steps: 1) Enable WebRTC statistics; 2) Read WebRTC. This process is illustrated in the following picture. This diagram also describes the JSON-RPC messages exchanged between Kurento client and KMS following the Kurento Protocol:

Sequence diagram for gathering WebRTC statistics in KMS

Example¶

There is a running tutorial which uses the WebRTC gathering as described before. This demo has been implemented using the JavaScript client and it is available on GitHub: kurento-loopback-stats.

From a the Media Pipeline point of view, this demo application consists in a WebRtcEndpoint in loopback. Once the demo is up and running, WebRTC are enabled and gathered with a rate of 1 second.

In addition to the KMS WebRTC statistics, the client-side (i.e. browser WebRtc peer) are also gathered by the application. This is done using the standard method provided by the peerConnection object, i.e using its method getStats. Please check out the JavaScript logic located in the index.js file for implementation details.

Both kinds of WebRTC statistics values (i.e. browser and KMS side) are updated and shown each second in the application GUI, as follows:

Statistics results in the kurento-loopback-stats demo GUI

Debug Logging¶

When running Kurento Media Server manually with /usr/bin/kurento-media-server, all logging messages are by default printed to standard out (stdout).

The KMS native packages modify this behavior to ensure logging information is placed in a more conventional location for the platform. By default logs should be made available in /var/log/kurento-media-server/, unless customized in the service settings file, /etc/default/kurento-media-server (for Debian/Ubuntu packages). Log files are named as follows:

{DateTime}.{LogNumber}.pid{PID}.log

{DateTime}: Logging file creation date and time, in ISO 8601 Extended Notation for the date, and Basic Notation for the time. For example: 2018-12-31T235959.
{LogNumber}: Log file number. A new one will be created whenever the maximum size limit is reached (100 MB by default).
{PID}: Process Identifier of kurento-media-sever.

When the KMS service starts correctly, a log file such as this one will be created:

2018-06-14T194426.00000.pid13006.log

Besides normal log files, an errors.log file stores error messages and stack traces, in case KMS crashes.

Note

Log files in this folder are rotated, and old files will get eventually deleted when new ones are created. This helps with preventing that all available disk space ends up filled with logs.

Each line in a log file has a fixed structure:

{DateTime} {PID} {ThreadID} {Level} {Component} {FileLine} {Function} {Object}? {Message}

{DateTime}: Date and time of the logging message, in ISO 8601 Extended Notation, with six decimal places for the seconds fraction. For example: 2018-12-31T23:59:59,123456.
{PID}: Process Identifier of kurento-media-sever.
{ThreadID}: Thread ID from which the message was issued. For example: 0x0000111122223333.
{Level}: Logging level. This value will typically be INFO or DEBUG. If unexpected error situations happen, the WARNING and ERROR levels will contain information about the problem.
{Component}: Name of the component that generated the log line. For example: KurentoModuleManager, webrtcendpoint, qtmux, etc.
{FileLine}: File name and line number, separated by a colon. For example: main.cpp:255.
{Function}: Name of the function in which the log message was generated. For example: main(), loadModule(), kms_webrtc_endpoint_gather_candidates(), etc.
{Object}: [Optional] Name of the object that issued the message, if one was specified for the log message. For example: <kmswebrtcendpoint0>, <fakesink1>, <audiotestsrc0:src>, etc.
{Message}: The actual log message.

For example, when KMS starts correctly, a message like this will be printed:

2018-06-14T19:44:26,918243  13006  0x00007f59401f5880  info  KurentoMediaServer  main.cpp:255  main()  Kurento Media Server started

Logging levels and components¶

Each different {Component} of KMS is able to generate its own logging messages. Besides that, each individual logging message has a severity {Level}, which defines how critical (or superfluous) the message is.

These are the different message levels, as defined by the GStreamer logging library:

(1) ERROR: Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error.
(2) WARNING: Logs all warnings. Typically these are non-fatal, but user-visible problems that are expected to happen.
(3) FIXME: Logs all “fixme” messages. Fixme messages are messages that indicate that something in the executed code path is not fully implemented or handled yet. The purpose of this message is to make it easier to spot incomplete/unfinished pieces of code when reading the debug log.
(4) INFO: Logs all informational messages. These are typically used for events in the system that happen only once, or are important and rare enough to be logged at this level.
(5) DEBUG: Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object’s lifetime; these include setup, teardown, change of parameters, etc.
(6) LOG: Logs all log messages. These are messages for events that happen repeatedly during an object’s lifetime; these include streaming and steady-state conditions.
(7) TRACE: Logs all trace messages. These messages for events that happen repeatedly during an object’s lifetime such as the ref/unref cycles.
(8) MEMDUMP: Log all memory dump messages. Memory dump messages are used to log (small) chunks of data as memory dumps in the log. They will be displayed as hexdump with ASCII characters.

Logging categories and levels can be filtered by two methods:

Use a command-line argument if you are manually running KMS. For example, run:

/usr/bin/kurento-media-server \
  --gst-debug-level=3 \
  --gst-debug="Kurento*:4,kms*:4"

You can also replace the command-line arguments with the environment variable GST_DEBUG. For example, run:
```
export GST_DEBUG="3,Kurento*:4,kms*:4"
/usr/bin/kurento-media-server
```

If you are using the native packages (installing KMS with apt-get) and running KMS as a system service, then you can also configure the GST_DEBUG variable in the KMS service settings file, /etc/default/kurento-media-server:

# Logging level.
export GST_DEBUG="3,Kurento*:4,kms*:4"

Logs will be colored by default, but colors can be explicitly disabled in the same two ways: either with --gst-debug-no-color or with export GST_DEBUG_NO_COLOR=1. When running KMS as a system service, this option is enabled in order to generate clean logs without strange terminal ANSI color escape sequences.

Suggested levels¶

Here are some tips on what logging components and levels could be most useful depending on what is the issue to be analyzed. They are given in the environment variable form, so they can be copied directly into the KMS KMS service settings file, /etc/default/kurento-media-server:

The default suggested level is what KMS sets automatically when it is started as a system service from the init scripts:

export GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4"

From that baseline, one can add any other values to extend the amount of information that gets logged:

Event MediaFlow{In,Out} state changes

export GST_DEBUG="${GST_DEBUG:-3},KurentoMediaElementImpl:5"

ICE candidate gathering
```
export GST_DEBUG="${GST_DEBUG:-3},kmsiceniceagent:5,kmswebrtcsession:5,webrtcendpoint:4"
```
Note
- See also libnice to enable advanced logging.
- kmsiceniceagent shows messages from the Nice Agent (low-level handling of candidates).
- kmswebrtcsession shows messages from the KMS WebRtcSession class (broarder decision logic).
- webrtcendpoint shows messages from the WebRtcEndpoint (very basic logging).

Player

export GST_DEBUG="${GST_DEBUG:-3},playerendpoint:5"

Recorder

export GST_DEBUG="${GST_DEBUG:-3},KurentoRecorderEndpointImpl:4,recorderendpoint:5,qtmux:5"

REMB congestion control
```
export GST_DEBUG="${GST_DEBUG:-3},kmsremb:5"
```
Note
- kmsremb:5 (debug level 5) shows only effective REMB send/recv values.
- kmsremb:6 (debug level 6) shows full (very verbose) handling of all source SSRCs.

RPC calls

export GST_DEBUG="${GST_DEBUG:-3},KurentoWebSocket*:5"

RTP Sync

export GST_DEBUG="${GST_DEBUG:-3},kmsutils:5,rtpsynchronizer:5,rtpsynccontext:5,basertpendpoint:5"

SDP processing

export GST_DEBUG="${GST_DEBUG:-3},kmssdpsession:5"

Transcoding of media

export GST_DEBUG="${GST_DEBUG:-3},Kurento*:5,agnosticbin*:5"

Unit tests

export GST_DEBUG="${GST_DEBUG:-3},check:5"

3rd-Party libraries¶

libnice¶

libnice is the GLib implementation of ICE, the standard method used by WebRTC to solve the issue of NAT Traversal.

This library uses the standard GLib logging functions, which comes disabled by default but can be enabled very easily. This can prove useful in situations where a developer is studying an issue with the ICE process. However, the debug output of libnice is very verbose, so it makes sense that it is left disabled by default for production systems.

To enable debug logging on libnice, set the environment variable G_MESSAGES_DEBUG with one or more of these values (separated by commas):

libnice
libnice-stun
libnice-tests
libnice-socket
libnice-pseudotcp
libnice-pseudotcp-verbose
all

After doing this, GLib messages themselves must be enabled in the Kurento logging system, by setting an appropriate level for the glib component.

Example:

export G_MESSAGES_DEBUG="libnice,libnice-stun"
export GST_DEBUG="${GST_DEBUG:-3},glib:5"
/usr/bin/kurento-media-server

You can also set this configuration in the Kurento service settings file, which gets installed at /etc/default/kurento-media-server.

libsoup¶

libsoup is the GNOME HTTP client/server library. It is used to perform HTTP requests, and currently this is used in Kurento by the KmsImageOverlay and the KmsLogoOverlay filters.

It is possible to enable detailed debug logging of the HTTP request/response headers, by defining the environment variable SOUP_DEBUG=1 before running KMS:

export SOUP_DEBUG=1
/usr/bin/kurento-media-server

Kurento Team¶

The Kurento development team is formed under the CodeURJC Research Group, which belongs to the spanish Rey Juan Carlos University, located in Madrid. This team is financed by the University and by Naeva Tec.

Micael Gallego is the current lead of the Kurento project. He is involved mainly in the development of the Kurento Java client, Java tutorials, testing infrastructure and Kurento Module Creator.
Boni García has worked to keep all out integration tests up to date, and he also wrote tutorials and test documentation.
Pablo Fuente works in Kurento Room, OpenVidu, and also in Kurento’s Java and JavaScript client libraries.
Juan Navarro is our Kurento Media Server guy. He loves working close to the metal with C/C++ and GStreamer. Follow him on Twitter to get updates on new Kurento releases and the low-level stuff he is working on.
Fede Díaz is our DevOps, helping the project with CI infrastructure, performance testing, and so on.
Patxi Gortázar is the original DevOps of Kurento platform and he’s still working across several areas of the project.

Contribution Guide¶

You can contribute to the Kurento project through bug reports or bug fixes in the Issue Tracker, code for new features, or improvements to the documentation. To contribute, write in the the Kurento Public Mailing List providing full information about your contribution and its value.

You must comply with the following contribution guidelines:

Specify the contents of your contribution either through a detailed bug description, through a Pull Request, or through a patch.
Specify the licensing restrictions of the code you contribute.
For newly created code to be incorporated in the Kurento codebase, you must accept Kurento to own the code copyright, so that its open source nature is guaranteed.
Justify appropriately the need and value of your contribution. The Kurento project has no obligations in relation to accepting contributions from third parties.

The Kurento project leaders have the right of asking for further explanations, tests or validations of any code contributed to the community before it being incorporated into the Kurento codebase. You must be ready to address all these kind of concerns before having your code approved.

Code of Conduct¶

Open Source Software communities are complex structures where different interests, expectations and visions need to converge and find some kind of equilibrium. In this process, all stakeholders need to understand and comply with a minimal set of rules that guarantee that things happen to the benefit of the community as a whole and that efforts are invested in the most optimal way for that to happen.

Of course, the Kurento team would be happy to have the appropriate resources that allowed providing full and detailed answers to all issues that may arise. Unfortunately this is not the case, and as happens in most OSS projects out there, we need to optimize how efforts are invested and think on the benefit of the community as a whole, instead of ending up satisfying the specific needs of a specific user.

Having said this, it is also clear that complying with a minimum set of netiquette rules is a plus for having questions and issues answered. Most of these rules are common sense, but it may be worthy to state them in a more explicit way so that Kurento users are able to check if they are doing their best to have their issues and questions addressed. Here they go:

Be courteous. Any kind of insult, threat or undervaluation of other people’s efforts will only contribute to having your request ignored.
Make your homework. Asking questions such as “I want to create a system like Skype, please explain me the steps” may require very extensive answers and you’ll probably find that nobody in the community is willing to invest the time to write them, save the case that someone happens to be writing a book on the topic. In general, don’t ask others to make your work.
Follow the reporting guidelines. When creating a new bug report, following these guidelines will greatly help others to study your issue and look for solutions, which in the end is a positive net for you.
Read the documentation first. Requesting help on issues that are clearly addressed in the documentation is, in general, a bad practice.
Check the Community Support. Things like opening a new discussion thread on the mailing list dealing with a problem that has already been discussed in another thread, will be probably perceived as slackness by the rest of the community. Avoid this and remember that Google is your friend.
Beware of cross-posting. In general, cross-posting is not considered as a good practice. If for some reason you need to send the same request to different mailing lists, inform in all of them about that providing links to the corresponding threads in the other lists so that the rest of users can check where answers finally arrived.
Be constructive. Claims of the kind “this design is bad” or “you are doing it wrong” are not particularly useful. If you don’t like something, provide specific suggestions (or better code) showing how things should be improved.
Maintain the focus. Kurento Community Support places have the objective of discussing Kurento-related issues. If you want to have information related to other different projects or to WebRTC in general, contact the corresponding community. Of course, spam shall be punished with immediate banning from the mailing lists.

Complying with these rules will contribute to improve the quality of the Kurento Community as a whole, making it the most helpful source of help and support.

Bests, and have a nice coding time.

Release Notes¶

6.10.0 (Apr 2019)¶

Kurento Media Server 6.10 is seeing the light with some important news!

To install it: Installation Guide.

Hello Ubuntu Bionic¶

Preliminary support for Ubuntu 18.04 LTS (Bionic Beaver) has landed in Kurento, and all the CI machinery is already prepared to compile and generate Debian packages into a new repository.

To install KMS on this version of Ubuntu, just follow the usual installation instructions:

DISTRO="bionic"  # KMS for Ubuntu 18.04 (Bionic)

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5AFA7A83

sudo tee "/etc/apt/sources.list.d/kurento.list" >/dev/null <<EOF
# Kurento Media Server - Release packages
deb [arch=amd64] http://ubuntu.openvidu.io/6.10.0 $DISTRO kms6
EOF

sudo apt-get update \
  && sudo apt-get install --yes kurento-media-server

Mostly everything is already ported to Bionic and working properly, but the port is not 100% finished yet. You can track progress in this board: https://github.com/orgs/Kurento/projects/1

The two biggest omissions so far are:

None of the extra modules have been ported yet; i.e. the example plugins that are provided for demonstration purposes, such as kms-chroma, kms-crowddetector, kms-platedetector, kms-pointerdetector.
OpenCV plugins in Kurento still uses the old C API. This still worked out fine in Ubuntu 16.04, but it doesn’t any more in 18.04 for plugins that use external training resouces, such as the HAAR filters.

Plugins that need to load external OpenCV training data files won’t work. For now, the only plugin affected by this limitation in KMS seems to be facedetector because it won’t be able to load the newer training data sets provided by OpenCV 3.2.0 on Ubuntu 18.04. Consequently, other plugins that depend on this one, such as the faceoverlay filter and its FaceOverlayFilter API, won’t work either.

Not much time has been invested in these two plugins, given that they are just simple demonstrations and no production application should be built upon them. Possibly the only way to solve this problem would be to re-write these plugins from scratch, using OpenCV’s newer C++ API, which has support for the newer training data files provided in recent versions of OpenCV.

This issue probably affects others of the extra modules mentioned earlier. Those haven’t even started to be ported.

Hello Chrome 74¶

Google is moving forward the security of WebRTC world by dropping support for the old DTLS 1.0 protocol. Starting from Chrome 74, DTLS 1.2 will be required for all WebRTC connections, and all endpoints that want to keep compatibility must be updated to use this version of the protocol.

Our current target operating system, Ubuntu 16.04 (Xenial), provides the library OpenSSL version 1.0.2g, which already offers support for DTLS 1.2 [1]. So, the only change that was needed to bring Kurento up to date in compatibility with Chrome was to actually make use of this newer version of the DTLS protocol.

Bye Ubuntu Trusty¶

Our resources are pretty limited here and the simpler is our CI pipeline, and the less work we need to dedicate making sure KMS works as expected in all Operating Systems, the best effort we’ll be able to make improving the stability and features of the server.

Our old friend Ubuntu 14.04 LTS (Trusty Tahr) is reaching its deprecation and End Of Life date in April 2019 (source: https://wiki.ubuntu.com/Releases) so this seems like the best time to drop support for this version of Ubuntu.

Canonical is not the only one setting an end to Trusty with their lifecycle schedules; Google is also doing so because by requiring DTLS 1.2 support for WebRTC connections, Trusty is left out of the game, given that it only provides OpenSSL 1.0.1f which doesn’t support DTLS 1.2 [1].

Reducing forks to a minimum¶

Moving on to Ubuntu 18.04 and dropping support of Ubuntu 14.04 are efforts which pave the road for a longer-term target of dropping custom-built tools and libraries, while standarizing the dependencies that are used in the Kurento project.

These are some of the objectives that we’d like to approach:

Using standard tools to create Ubuntu packages.

We’re dropping the custom compile_project.py tool and instead will be pushing the use of git-buildpackage. Our newer kurento-buildpackage.sh uses git-buildpackage for the actual creation of Debian packages from Git repositories.
Dropping as many forks as possible. As shown in Code repositories, Kurento uses a good number of forked libraries that are packaged and distributed as part of the usual releases, via the Kurento package repositories.

Keeping all these forks alive is a tedious task, very error-prone most of the times. Some are definitely needed, such as openh264 or usrsctp because those are not distributed by Ubuntu itself so we need to do it.

Some others, such as the fork of libsrtp, have already been dropped and we’re back to using the official versions provided by Ubuntu (yay!)

Lastly, the big elephant in the room is all the GStreamer forks, which are stuck in an old version of GStreamer (1.8) and would probably benefit hugely from moving to newer releases.

We hope that moving to Ubuntu 18.04 can ease the transition from our forks of each library to the officially provided versions.
Ultimately, a big purpose we’re striving for is to have Kurento packages included among the official ones in Ubuntu, although that seems like a bit far away for now.

Clearer Transcoding log messages¶

Codec transcoding is always a controversial feature, because it is needed for some cases which cannot be resolved in any other way, but it is undesired because it will consume a lot of CPU power.

All debug log messages related to transcoding have been reviewed to make them as clear as possible, and the section Troubleshooting Issues has been updated accordingly.

If you see that transcoding is active at some point, you may get a bit more information about why, by enabling this line:

export GST_DEBUG="${GST_DEBUG:-3},Kurento*:5,agnosticbin*:5"

in your daemon settings file, /etc/default/kurento-media-server.

Then look for these messages in the media server log output:

Upstream provided caps: (caps)
Downstream wanted caps: (caps)
Find TreeBin with wanted caps: (caps)

Which will end up with either of these sets of messages:

If source codec is compatible with destination:
- TreeBin found! Use it for (audio|video)
- TRANSCODING INACTIVE for (audio|video)
If source codec is not compatible with destination:
- TreeBin not found! Transcoding required for (audio|video)
- TRANSCODING ACTIVE for (audio|video)

These messages can help understand what codec settings are being received by Kurento (“Upstream provided caps”) and what is being expected at the other side by the stream receiver (“Downstream wanted caps”).

Recording with Matroska¶

It’s now possible, thanks to a user contribution, to configure the RecorderEndpoint to use the Matroska multimedia container (MKV), using the H.264 codec for video.

This has big implications for the robustness of the recording, because with the MP4 container format it was possible to lose the whole file if the recorder process crashed for any reason. MP4 stores its metadata only at the end of the file, so if the file gets truncated it means that it won’t be playable. Matroska improves the situation here, and a truncated file will still be readable.

For more information about the issues of the MP4 container, have a look a then new knowledge section: H.264 video codec.

New JSON settings parser¶

Kurento uses the JSON parser that comes with the Boost C++ library; this parser accepted comments in JSON files, so we could comment out some lines when needed. The most common example of this was to force using only VP8 or H.264 video codecs in the Kurento settings file, /etc/kurento/modules/kurento/SdpEndpoint.conf.json:

"videoCodecs" : [
  {
    "name" : "VP8/90000"
  },
  {
    "name" : "H264/90000"
  }
]

This is the default form of the mentioned file, allowing Kurento to use either VP8 or H.264, as needed. To disable VP8, this would change as follows:

"videoCodecs" : [
//  {
//    "name" : "VP8/90000"
//  },
  {
    "name" : "H264/90000"
  }
]

And it worked fine. The Boost JSON parser would ignore all lines starting with //, disregarding them as comments.

However, starting from Boost version 1.59.0, the Boost JSON parser gained the great ability of not allowing comments; it was rewritten without any consideration for backwards-compatibility (yeah, it wouldn’t hurt the Boost devs if they practiced a bit of “Do NOT Break Users” philosophy from Linus Torvalds, or at least followed Semantic Versioning…)

The devised workaround has been to allow inline comment characters inside the JSON attribute fields, so the former comment can now be done like this:

"videoCodecs": [
    { "//name": "VP8/90000" },
    { "name": "H264/90000" }
]

Whenever you want to comment out some line in a JSON settings file, just append the // characters to the beginning of the field name.

Code Sanitizers and Valgrind¶

If you are developing Kurento, you’ll probably benefit from running AddressSanitizer, ThreadSanitizer, and other related tools that help finding memory, threading, and other kinds of bugs.

kms-cmake-utils includes now the arsenm/sanitizers-cmake tool in order to integrate the CMake build system with the mentioned compiler utilities. You’ll also find some useful suppressions for these tools in the kms-omni-build dir.

Similarly, if you want to run KMS under Valgrind, kms-omni-build contains some utility scripts that can prove to be very handy.

Special Thanks¶

A great community is a key part of what makes any open source project special. From bug fixes, patches, and features, to those that help new users in the forum / mailing list and GitHub issues, we’d like to say: Thanks!

Additionally, special thanks to these awesome community members for their contributions:

@prlanzarin (Paulo Lanzarin) for:
- Add API support for MKV profile for recordings Kurento/kms-core#14, Kurento/kms-elements#13.
- Fixed config-interval prop type checking in basertpendpoint and rtppaytreebin Kurento/kms-core#15 and @leetal (Alexander Widerberg) for reporting #321.
- rtph26[45]depay: Don’t handle NALs inside STAP units twice (cherry-picked from upstream) Kurento/gst-plugins-good#2.
@tioperez (Luis Alfredo Perez Medina) for reporting #349 and sharing his results with RTSP and Docker.
@goroya for Kurento/kurento-media-server#10.

[1]	(1, 2) DTLS 1.2 was added in OpenSSL 1.0.2: Major changes between OpenSSL 1.0.1l and OpenSSL 1.0.2 [22 Jan 2015].

6.9.0 (Dec 2018)¶

(lib)Nicer and performant¶

Howdy!

A new release of Kurento Media Server brings huge stability and performance improvements.

Until recently, some Kurento users had been suffering a fatal bug which would cause crashes and segmentation faults in some specific configurations, related to abruptly interrupted TCP connections. Thanks to the invaluable help of Olivier Crête, who is the main maintainer and developer of the libnice library, this bug has been fixed and all projects that make use of this library are now able to benefit from the fix.

Kurento is of course one of these projects, making use of the libnice library to provide WebRTC compatibility. Starting from version 6.9.0, the latest improvements made to the libnice library have been integrated, and now all related crashes have stopped happening. Once again, we want to send a huge thank you to Olivier for his great work!

Another relevant topic for this release is all the effort directed at creating proper test infrastructure to perform load and stress testing with Kurento Media Server. We pushed forward to unify some work that had been laying around in different branches, and another high-impact patch ended up merged into mainline libnice. This time, removing a global lock that caused a performance cap.

Our intention has been to raise the number of simultaneous streams that the media server is able to sustain, as a direct response to the lackluster results that Kurento obtained in a recent benchmark / comparison between WebRTC implementations, published in webrtcH4cKS a couple of months ago. For this purpose, one of the hard tasks was to create a custom stress test that could be used as proof for Kurento capabilities; at first we were able to reproduce the reduced performance shown in the published benchmark, but as of latest weeks, Kurento has been more on par with other solutions:

Kurento was able to support a maximum of 133 simultaneously connected users (browsers): 19 conference rooms with 7 users each.
Due to the SFU architecture, this means that KMS sustained 133 inbound streams, and 798 outbound streams.
The test machine was an 8-core, 16GB RAM instance on AWS.
Test videos are 540×360 in size.
All participants were simulated with Chrome browsers, so the video formats were always compatible between senders and receivers. This is a very important detail because it means that video transcoding (“on-the-fly adaptation of codecs and formats between incompatible clients”) was not necessary and thus it was inactive.

During these stress tests, CPU usage grew with number of participants, and RAM usage stayed at around 25% (4GB):

Next is a summary of all additions, changes and/or fixes that are included in this release:

Added¶

[kms-elements] The port range that PlayerEndpoint uses for incoming RTP streams can now be configured with the rtspClientPortRange property in /etc/kurento/modules/kurento/PlayerEndpoint.conf.ini. Thanks Mislav Čakarić for the Pull Request #14!
[kms-omni-build] The bin/ folder now contains a set of scripts that help developers run Valgrind to find memory leaks and access errors.
[doc-kurento] Our documentation has received a big bump in the project management section, this time related to how our End-to-End tests are designed and how Kurento developers can use them. This has little to no relevance to end users, but it’s a first step into the long-term objective of letting developers who use Kurento have better knowledge about our testing software.

If you want to have a look, check out the Testing section. Thanks Boni García for all the work!

Changed¶

[kms-core] The default port range for incoming RTP streams (applicable to both RtpEndpoint and WebRtcEndpoint) was [0, 65535]. This changed to exclude all privileged ports, which are system-protected and require root permissions to be used; now the port range is [1024, 65535].

Of course, you are still able to configure a different range of ports by setting the minPort and/or maxPort properties in /etc/kurento/modules/kurento/BaseRtpEndpoint.conf.ini.
[kurento-media-server] The daemon settings file in /etc/default/kurento-media-server now disables colored output by default. This is useful to produce a clean error log file in /var/log/kurento-media-server/errors.log.
[kurento-media-server] There was some confusion surrounding how to configure the System Limits that Kurento needs for working, specifically the Open File Descriptors limit. These are configured by the daemon files during the service startup, but there was no error control and these limits could end up being silently left with their default values (which is a bad thing and a sure way to have performance issues with the media server). There are now controls in place that will set up some valid limits.
[kurento-media-server] Error messages in /var/log/kurento-media-server/errors.log are now separated between executions of Kurento Media Server, with this line:
```
<TIMESTAMP> -- New execution
```
Before this change, all errors were appended without separation, which could become confusing and users had problems identifying where older executions finished and where the latest one started.

Fixed¶

[kurento-media-server] The daemon would crash if the debug log files couldn’t be created for any reason (e.g. because of permission problems). After this fix, if log files cannot be created or accessed, file logging will be disabled and logging will revert back to being printed to stdout.

6.8.1 (Oct 2018)¶

Release 6.8.0 contained a critical bug that was immediately patched into 6.8.1, so these Release Notes include changes from both versions.

This release goes hand-in-hand with a new version of OpenVidu, Kurento’s sister project. Check OpenVidu 2.5.0 Release Notes if you are interested in building any of the common use cases that are covered by that project, such as conference calls and video chat rooms.

Added¶

Log messages that come from GLib-based libraries are now integrated into the general Kurento logging system.

Previous to this addition, the only way to obtain debug logs from the libnice library was to run KMS directly on console; even after enabling debug logging, the relevant messages would not appear in the Kurento logs because libnice was just printing its messages in the standard output.

Starting from KMS 6.8.0, all messages will be redirected to Kurento logs, located at /var/log/kurento-media-server/. Remember that specific 3rd-party libraries such as libnice still require that their logging functions are explicitly enabled; check libnice for more details.
Hub and HubPort elements now support for DATA streams. This means that a WebRTC DataChannels stream can be processed through a Hub, for example a Composite, and the DATA stream will be available for the element to process.
Thanks to the previous addition, Composite element has now support for merging multiple DataChannel streams.
GStreamerFilter is now able to set its inner element’s properties “on the fly” during runtime. For example, if you used a coloreffects filter, before this addition you would need to configure the video parameters beforehand, and they would stay the same during the whole execution of the Kurento pipeline. Now, it is possible to change the filter’s properties at any time, during the execution of the pipeline.

The OpenVidu project is using this capability to offer real-time audio/video filtering during WebRTC calls; check Voice and video filters for more details.

Changed¶

Output logs now use standard format ISO 8601 for all timestamps. This affects both log files names, and their contents:

Log files will now be named such as this:
```
2018-06-14T194426.00000.pid13006.log
```
And each individual log message will contain timestamps of this form:
```
2018-06-14T19:44:26,918243
```
disableRequestCache is now exposed in settings file (kurento.conf.json). This can be used to disable the RPC Request Cache for troubleshooting or debugging purposes.
Clearer log messages about what is going on when the maximum resource usage threshold is reached.
System service configuration file /etc/default/kurento-media-server now contains more useful examples and explanations for each option.
libnice has been updated from the old version 0.1.13 to the newer 0.1.15 (snapshot - not officially released yet). This should help with fixing a crash issue that KMS has been suffering lately. For more details, see “Known Issues” below.

Fixed¶

System service init files will now append to the error log file /var/log/kurento-media-server/errors.log, instead of truncating it on each restart.

Known Issues¶

libnice is a 3rd-party library used by Kurento, which in the last months has been suffering of a serious bug that caused crashes in some specific conditions. There are several places where this issue has been discussed, such as the issue #247 in Kurento, the issue #33, and the mail topic in the Kurento List.

In our release 6.8.1 we have upgraded libnice to the development/snapshot build that will at some point become the official libnice 0.1.15; this seems to work much better than the older version 0.1.13 that we had been using so far in all previous releases of Kurento Media Server.

We haven’t been able to repeat the crash with this latest version of libnice, however we’ll wait until this new version has been tested more extensively, and only then will consider this issue as closed.

If you are interested or have been affected by this issue, note that in the days previous to writing these Release Notes there has been some movement in the issue #33 from the libnice project. Hopefully we are closer to a definite fix for this!

6.7.2 (May 2018)¶

Added¶

WebRtcEndpoint now allows to specify the desired direction for generated SDP Offers.

The most common use case for WebRTC calls is to generate an SDP Offer in the Browser and then pass it over to process in Kurento (which in turn generates an SDP Answer). However the opposite workflow is also possible: to have Kurento generating the SDP Offer, with the browser being the one which gets to process it.

This feature has been part of Kurento for a while, but there was a small bit missing in the picture: the SDP Offers generated by Kurento had always the direction attribute set for a two-way media stream, i.e. both receiving and sending media. Now, it is possible to configure the WebRtcEndpoint to generate SDP Offers with all three possible directions: a=sendrecv, a=sendonly, a=recvonly, for send-receive, send-only, and receive-only, respectively.

Changed¶

WebRtcEndpoint now doesn’t use reduced minimum interval for RTCP packets when the REMB network congestion control algorithm is not in use.

RFC 3550 section 6.2 (RTCP Transmission Interval) recommends an initial value of 5 seconds for the interval between sending of RTCP packets. However, Kurento configures a much lower value, and RTCP packets are sent every 500 milliseconds. The reasoning for this is that the congestion control algorithm (Congestion Control / REMB) should be able to react as soon as possible when congestion occurs, and this is helped by a faster RTCP interval.

The use case for this change is audio-only calls; Kurento doesn’t support using the network congestion algorithm for audio streams, so it doesn’t make sense to force sending high-frequency RTCP packets in this case. This has the secondary effect of greatly reducing consumed bandwidth when an audio track is muted via one of the browser’s RTCPeerConnection methods.

Fixed¶

Send mandatory attribute a=setup:actpass in SDP Offers. RFC 3550 section 5 (Establishing a Secure Channel) states:
```
The endpoint that is the offerer MUST use the setup attribute
value of setup:actpass
```
This was not the case in Kurento; Chrome was permissive about this mistake, but Firefox was complaining about the lack of this attribute in SDP Offers generated by the WebRtcEndpoint.

Developer Guide ¶

This section is a comprehensive guide for development of Kurento itself. The intended reader of this text is any person who wants to get involved in writing code for the Kurento project, or to understand how the source code of this project is structured.

If you are looking to write applications that make use of Kurento, then you should read Writing Kurento Applications.

Table of Contents

Introduction ¶

This is an overview of the tools and technologies used by KMS:

The code is written in C and C++ languages.
The code style is heavily influenced by that of Gtk and GStreamer projects.
CMake is the build tool of choice, and is used to build all modules.
Source code is versioned in several GitHub repositories.
The officially supported platforms are Long-Term Support (LTS) versions of Ubuntu: Ubuntu 16.04 (Xenial) and Ubuntu 18.04 (Bionic) (64-bits only).
The GStreamer multimedia framework sits at the heart of Kurento Media Server.
In addition to GStreamer, KMS uses other libraries like boost, jsoncpp, libnice, etc.

Code repositories ¶

Kurento source code is stored in several GitHub repositories at https://github.com/Kurento. Each one of these repositories has a specific purpose and usually contains the code required to build a shared library of the same name.

An overview of the relationships between all repos forming the Kurento Media Server:

$digraph dependencies_all { bgcolor = "transparent"; fontname = "Bitstream Vera Sans"; fontsize = 8; size = "12,8"; rankdir = "RL"; // Kurento external libraries "jsoncpp"; "libsrtp"; "openh264"; "usrsctp"; "gstreamer"; "gst-plugins-base" -> "gstreamer"; "gst-plugins-good" -> "gst-plugins-base"; "gst-plugins-bad" -> {"gst-plugins-base" "libsrtp" "openh264"}; "gst-plugins-ugly" -> "gst-plugins-base"; "gst-libav" -> "gst-plugins-base"; "openwebrtc-gst-plugins" -> {"gstreamer" "gst-plugins-base" "usrsctp"}; "libnice" -> "gstreamer"; // KMS main components "kurento-module-creator"; "kms-cmake-utils"; "kms-jsonrpc" -> {"jsoncpp" "kms-cmake-utils"}; "kms-core" -> {"gstreamer" "gst-plugins-base" "gst-plugins-good" "gst-plugins-bad" "gst-plugins-ugly" "gst-libav" "kurento-module-creator" "kms-cmake-utils" "kms-jsonrpc"}; "kms-elements" -> {"openwebrtc-gst-plugins" "libnice" "kms-core"}; "kms-filters" -> "kms-elements"; "kurento-media-server" -> {"kms-core" "kms-elements" "kms-filters"}; // KMS extra modules "kms-chroma" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-crowddetector" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-datachannelexample" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-platedetector" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-pointerdetector" -> {"kms-core" "kms-elements" "kms-filters"}; }$

All dependency relationships¶

As the dependency graph is not strictly linear, there are multiple possible ways to order all modules into a linear dependency list; this section provides one possible ordered list, which will be consistently used through all Kurento documents.

Fork repositories:

KMS depends on several open source libraries, the main one being GStreamer. Sometimes these libraries show specific behaviors that need to be tweaked in order to be useful for KMS; other times there are bugs that have been fixed but the patch is not accepted at the upstream source for whatever reason. In these situations, while the official path of feature requests and/or patch submit is still tried, we have created a fork of the affected libraries.

jsoncpp
libsrtp
openh264
usrsctp
gstreamer (produces libgstreamer1.5)
gst-plugins-base
gst-plugins-good
gst-plugins-bad
gst-plugins-ugly
gst-libav
openwebrtc-gst-plugins
libnice (produces gstreamer1.0-nice, gstreamer1.5-nice)

Main repositories

kurento-module-creator: It is a code generation tool for generating code scaffolding for plugins. This code includes KMS code and Kurento client code. It has mainly Java code.
kms-cmake-utils: Contains a set of utilities for building KMS with CMake.
kms-jsonrpc: Kurento protocol is based on JsonRpc, and makes use of a JsonRpc library contained in this repository. It has C++ code.
kms-core: Contains the core GStreamer code. This is the base library that is needed for other libraries. It has 80% C code and a 20% C++ code.
kms-elements: Contains the main elements offering pipeline capabilities like WebRtc, Rtp, Player, Recorder, etc. It has 80% C code and a 20% C++ code.
kms-filters: Contains the basic video filters included in KMS. It has 65% C code and a 35% C++ code.
kurento-media-server: Contains the main entry point of KMS. That is, the main() function for the server executable code. This program depends on libraries located in the above repositories. It has mainly C++ code.

Extra repositories

KMS is distributed with some basic GStreamer pipeline elements, but other elements are available in form of modules. These modules are demos of what third party modules could be written and integrated into Kurento. These are just for instructional purposes, and shouldn’t be used in production servers.

Omni-Build repository

This repository is an special project because it is designed to build all KMS Main repositories from a single entry point. This repo brings the other KMS Main repositories as Git submodules: it makes KMS development easier because if you build this project, you don’t need to manually install the libraries of the other KMS Main repositories. However, all other development and support libraries must still be installed manually.

kms-omni-build

Client repositories

Application Servers can be developed in Java, JavaScript with Node.js, or JavaScript directly in the browser. Each of these languages have their support tools made available in their respective repositories.

kurento-client-js (Node.js Application Servers, browser JavaScript)
kurento-java (Java Application Servers)

Tutorial or demo repositories

There are several repositories that contain sample code for developers that use Kurento or want to develop a custom Kurento module. Currently these are:

A KMS developer must know how to work with KMS Fork and Main repositories and understand that each of these have a different development life cycle. The majority of development for KMS will occur at theK MS Main repositories, while it’s unusual to make changes in Fork repositories except for updating their upstream versions.

Development 101 ¶

KMS is a C/C++ project developed with an Ubuntu system as main target, which means that its dependency management and distribution is based on the Debian package system.

Libraries ¶

It is not a trivial task to configure the compiler to use a set of libraries because a library can be composed of several .so and .h files. To make this task easier, pkg-config is used when compiling programs and libraries. In short: when a library is installed in a system, it registers itself in the pkg-config database with all its required files, which allows to later query those values in order to compile with the library in question.

For example, if you want to compile a C program which depends on GLib 2.0, you can run:

gcc -o program program.c $(pkg-config --libs --cflags glib-2.0)

Debian packages ¶

In a Debian/Ubuntu system, development libraries are distributed as Debian packages which are made available in public package repositories. When a C or C++ project is developed in these systems, it is usual to distribute it also in Debian packages. It is then possible to install them with the command apt-get install, which will handle automatically all the package’s dependencies.

When a library is packaged, the result usually consists of several packages. These are some pointers on the most common naming conventions for packages, although they are not always strictly enforced by Debian or Ubuntu maintainers:

bin package: Package containing the binary files for the library itself. Programs are linked against them during development, and they are also loaded in production. The package name starts with lib, followed by the name of the library.
dev package: Contains files needed to link with the library during development. The package name starts with lib and ends with -dev. For example: libboost-dev or libglib2.0-dev.
dbg package: Contains debug symbols to ease error debugging during development. The package name starts with lib and ends with -dbg. For example: libboost-dbg.
doc package: Contains documentation for the library. Used in development. The package name starts with lib and ends with -doc. For example: libboost-doc.
src package: Package containing the source code for the library. It uses the same package name as the bin version, but it is accessed with the command apt-get source instead of apt-get install.

Build tools ¶

There are several tools for building C/C++ projects: Autotools, Make, CMake, Gradle, etc. The most prominent tool for building projects is the Makefile, and all the other tools tend to be simply wrappers around this one. KMS uses CMake, which generates native Makefiles to build and package the project. There are some IDEs that recognize CMake projects directly, such as JetBrains CLion or Qt Creator.

A CMake projects consists of several CMakeLists.txt files, which define how to compile and package native code into binaries and shared libraries. These files also contain a list of the libraries (dependencies) needed to build the code.

To specify a dependency it is necessary to know how to configure this library in the compiler. The already mentioned pkg-config tool is the standard de-facto for this task, so CMake comes with the ability to use pkg-config under the hood. There are also some libraries built with CMake that use some specific CMake-only utilities.

Building from sources ¶

To work directly with KMS source code, or to just build KMS from sources, the easiest way is using the module kms-omni-build. Just follow these steps:

Add the Kurento repository to your system configuration.
Clone kms-omni-build.
Install build dependencies: tools like GCC, CMake, etc., and KMS development libraries.
Build with CMake and Make.
Run the newly compiled KMS.
Run KMS tests.

Install required tools ¶

This command will install the basic set of tools that are needed for the next steps:

sudo apt-get update \
  && sudo apt-get install --no-install-recommends --yes \
     git gnupg devscripts equivs

Add Kurento repository ¶

These commands will add the Kurento repository to be accessed by apt-get. Run all inside the same terminal:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5AFA7A83

# Run *ONLY ONE* of these lines:
DISTRO="xenial"  # KMS for Ubuntu 16.04 (Xenial)
DISTRO="bionic"  # KMS for Ubuntu 18.04 (Bionic)

sudo tee "/etc/apt/sources.list.d/kurento.list" >/dev/null <<EOF
# Kurento Media Server - Nightly packages
deb [arch=amd64] http://ubuntu.openvidu.io/dev $DISTRO kms6
EOF

apt-get update

Download KMS ¶

Run:

git clone https://github.com/Kurento/kms-omni-build.git
cd kms-omni-build
git submodule update --init --recursive
git submodule update --remote

Note

--recursive and --remote are not used together, because each individual submodule may have some other submodules that are intended to be loaded in some specific commit, and we don’t want to update those with upstream’s latest changes.

OPTIONAL: Change to the master branch of each submodule, if you will be developing on each one of those:

REF=master
for d in $(find . -maxdepth 1 -mindepth 1 -type d)
do pushd $d ; git checkout "$REF" ; popd ; done

You can also set REF to any other branch or tag, such as REF=6.7.1. This will bring the code to the state it had in that version.

Install build dependencies ¶

Run:

DIRS=(
    kurento-module-creator
    kms-cmake-utils
    kms-jsonrpc
    kms-core
    kms-elements
    kms-filters
    kurento-media-server
)

for DIR in "${DIRS[@]}"; do
    mk-build-deps --install --remove \
        --tool='apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends --yes' \
        "${DIR}/debian/control"
done

Build KMS ¶

Run:

TYPE=Debug
mkdir build-$TYPE
cd build-$TYPE
cmake -DCMAKE_BUILD_TYPE=$TYPE ..

export MAKEFLAGS="-j$(nproc)"
make

CMake accepts the following build types: Debug, Release, RelWithDebInfo. So, for a Release build, you would run TYPE=Release instead of TYPE=Debug.

Note

If your cmake command fails, make sure you don’t have multiple build directories below kms-omni-build or any of its subdirectories. We have seen that having multiple build dirs can cause issues, so it’s better to only have one.

If you want to work with multiple build dirs at the same time, it’s better to just work on a separate Git clone, outside the kms-omni-build directory.

It is also possible to enable GCC’s AddressSanitizer or ThreadSanitizer with these flags:

-DENABLE_ANALYZER_ASAN=ON  # Enable the AddressSanitizer (aka ASan) memory error detector. Implies ``CMAKE_BUILD_TYPE=Release``.

-DSANITIZE_ADDRESS=ON
-DSANITIZE_MEMORY=ON
-DSANITIZE_THREAD=ON
-DSANITIZE_UNDEFINED=ON
-DSANITIZE_LINK_STATIC=ON

[TODO: integration with these tools is not really finished]

ASan: Launch with LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/5/libasan.so kurento-media-server

Verbose mode can be enabled too:

-DCMAKE_VERBOSE_MAKEFILE=ON

Lastly, it’s possible to run either Unit tests or Valgrind tests, by using different make targets:

make check
make valgrind

Launch KMS ¶

Run:

export GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4"

kurento-media-server/server/kurento-media-server \
    --modules-path=. \
    --modules-config-path=./config \
    --conf-file=./config/kurento.conf.json \
    --gst-plugin-path=.

You can set the logging level of specific categories with the option --gst-debug, which can be used multiple times, once for each category. Besides that, the global logging level is specified with --gst-debug-level. These values can also be defined in the environment variable GST_DEBUG (see Debug Logging).

Other launch options that could be useful:

--logs-path, -d <Path> : Path where rotating log files will be stored
--log-file-size, -s <Number> : Maximum file size for log files, in MB
--number-log-files, -n <Number> : Maximum number of log files to keep

More launch options, handled by GStreamer: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html

KMS Unit Tests ¶

KMS uses the Check unit testing framework for C (https://libcheck.github.io/check/). To build and run all tests, change the last one of the build commands from make to make check. All available tests will run, and a summary report will be shown at the end.

Note

It is recommended to first disable GStreamer log colors, that way the resulting log files won’t contain extraneous escape sequences such as ^[[31;01m ^[[00m. Also, it could be useful to specify a higher logging level than the default; set the environment variable GST_DEBUG, as explained in Logging levels and components.

The complete command would look like this:

export GST_DEBUG_NO_COLOR=1
export GST_DEBUG="3,check:5"
make check

The log output of the whole test suite will get saved into the file ./Testing/Temporary/LastTest.log. To find the starting point of each individual test inside this log file, search for the words “test start”. For the start of a specific test, search for “{TestName}: test start”. For example:

webrtcendpoint.c:1848:test_vp8_sendrecv: test start

To build and run one specific test, use make {TestName}.check. For example:

make test_agnosticbin.check

If you want to analyze memory usage with Valgrind, use make {TestName}.valgrind. For example:

make test_agnosticbin.valgrind

Clean up your system ¶

To leave the system in a clean state, remove all KMS packages and related development libraries. Run this command and, for each prompted question, visualize the packages that are going to be uninstalled and press Enter if you agree. This command is used on a daily basis by the development team at Kurento with the option --yes (which makes the process automatic and unattended), so it should be fairly safe to use. However we don’t know what is the configuration of your particular system, and running in manual mode is the safest bet in order to avoid uninstalling any unexpected package.

Run:

PACKAGES=(
  # KMS main components + extra modules
  '^(kms|kurento).*'

  # Kurento external libraries
  ffmpeg
  '^gir1.2-gst.*1.5'
  gir1.2-nice-0.1
  '^(lib)?gstreamer.*1.5.*'
  '^lib(nice|s3-2|srtp|usrsctp).*'
  '^srtp-.*'
  '^openh264(-gst-plugins-bad-1.5)?'
  '^openwebrtc-gst-plugins.*'

  # System development libraries
  '^libboost-?(filesystem|log|program-options|regex|system|test|thread)?-dev'
  '^lib(glib2.0|glibmm-2.4|opencv|sigc++-2.0|soup2.4|ssl|tesseract|vpx)-dev'
  uuid-dev
)

# Run a loop over all package names and uninstall them.
for PACKAGE in "${PACKAGES[@]}"; do
  sudo apt-get purge --auto-remove "$PACKAGE" || { echo "Skip unexisting"; }
done

Working on a forked library ¶

These are the two typical workflows used to work with fork libraries:

Full cycle ¶

This workflow has the easiest and fastest setup, however it also is the slowest one. To make a change, you would edit the code in the library, then build it, generate Debian packages, and lastly install those packages over the ones already installed in your system. It would then be possible to run KMS and see the effect of the changes in the library.

This is of course an extremely cumbersome process to follow during anything more complex than a couple of edits in the library code.

In-place linking ¶

The other work method consists on changing the system library path so it points to the working copy where the fork library is being modified. Typically, this involves building the fork with its specific tool (which often is Automake), changing the environment variable LD_LIBRARY_PATH, and running KMS with such configuration that any required shared libraries will load the modified version instead of the one installed in the system.

This allows for the fastest development cycle, however the specific instructions to do this are very project-dependent. For example, when working on the GStreamer fork, maybe you want to run GStreamer without using any of the libraries installed in the system (see https://cgit.freedesktop.org/gstreamer/gstreamer/tree/scripts/gst-uninstalled).

[TODO: Add concrete instructions for every forked library]

Debian packaging ¶

You can easily create Debian packages for KMS itself and for any of the forked libraries. Packages are generated by a Python script called compile_project.py, which can be found in the adm-scripts repository, and you can use it to generate Debian packages locally in your machine. Versions number of all packages are timestamped, so a developer is able to know explicitly which version of each package has been installed at any given time.

Follow these steps to generate Debian packages from any of the Kurento repositories:

(Optional) Make sure the system is in a clean state. The section Clean up your system explains how to do this.
(Optional) Add Kurento Packages repository. The section about Dependency resolution explains what is the effect of adding the repo, and the section Add Kurento repository explains how to do this.
Install system tools and Python modules. Run:
```
PACKAGES=(
    build-essential
    debhelper
    curl
    fakeroot
    flex
    git openssh-client
    libcommons-validator-java
    python
    python-apt
    python-debian
    python-git
    python-requests
    python-yaml
    subversion
    wget
)

sudo apt-get update \
  && sudo apt-get install --no-install-recommends --yes \
     "${PACKAGES[@]}"
```
Note
- flex will be automatically installed by GStreamer, but for now a bug in package version detection prevents that.
- libcommons-validator-java seems to be required to build gstreamer (it failed with lots of errors from jade, when building documentation files).
- subversion (svn) is used in the Python build script (compile_project.py) due to GitHub’s lack of support for git-archive protocol (see https://github.com/isaacs/github/issues/554).
Download the Kurento CI scripts and the desired module (change kms-core to the name of the module you want to build). Run:
```
git clone https://github.com/Kurento/adm-scripts.git
git clone https://github.com/Kurento/kms-core.git
```
Build packages for the desired module. Run:
```
sudo -s
export PYTHONUNBUFFERED=1
export PATH="$PWD/adm-scripts:$PWD/adm-scripts/kms:$PATH"

cd kms-core
compile_project.py --base_url https://github.com/Kurento compile
```
Another variable you can export is DEB_BUILD_OPTIONS, in order to disable any of unit testing, doc generation (which at the Debian level is mostly nothing, this doesn’t refer to the whole Kurento project documentation site), and binary stripping. For example:
```
export DEB_BUILD_OPTIONS="nocheck nodoc nostrip"
```

Dependency resolution: to repo or not to repo ¶

The script compile_project.py is able to resolve all dependencies for any given module. For each dependency, the following process will happen:

If the dependency is already available to apt-get from the Kurento Packages repository, it will get downloaded and installed. This means that the dependency will not get built locally.
If the dependency is not available to apt-get, its corresponding project will be cloned from the Git repo, built, and packaged itself. This triggers a recursive call to compile_project.py, which in turn will try to satisfy all the dependencies corresponding to that sub-project.

It is very important to keep in mind the dependency resolution mechanism that happens in the Python script, because it can affect which packages get built in the development machine. If the Kurento Packages repository has been configured for ``apt-get``, then all dependencies for a given module will be downloaded and installed from the repo, instead of being built. On the other hand, if the Kurento repo has not been configured, then all dependencies will be built from source.

This can have a very big impact on the amount of modules that need to be built to satisfy the dependencies of a given project. The most prominent example is kurento-media-server: it basically depends on everything else. If the Kurento repo is available to apt-get, then all of KMS libraries will be downloaded and installed. If the repo is not available, then all source code of KMS will get downloaded and built, including the whole GStreamer libraries and other forked libraries.

Package generation script ¶

This is the full procedure followed by the compile_project.py script:

Check if all development dependencies for the given module are installed in the system. This check is done by parsing the file debian/control of the project.
If some dependencies are not installed, apt-get tries to install them.
For each dependency defined in the file .build.yaml, the script checks if it got installed during the previous step. If it wasn’t, then the script checks if these dependencies can be found in the source code repository given as argument. The script then proceeds to find this dependency’s real name and requirements by checking its online copy of the debian/control file.
Every dependency with source repository, as found in the previous step, is cloned and the script is run recursively with that module.
When all development dependencies are installed (either from package repositories or compiling from source code), the initially requested module is built, and its Debian packages are generated and installed.

How-To ¶

How to add or update external libraries ¶

Add or change it in these files:

debian/control.
CMakeLists.txt.

How to add new fork libraries ¶

Fork the repository.
Create a .build.yaml file in this repository, listing its project dependencies (if any).
Add dependency to debian/control in the project that uses it.
Add dependency to CMakeLists.txt in the project that uses it.

How to work with API changes ¶

What to do when you are developing a new feature that spans across KMS and the public API? This is a summary of the actions done in CI by adm-scripts/kurento_generate_java_module.sh and adm-scripts/kurento_maven_deploy.sh:

Work on your changes, which may include changing the KMS files where the Kurento API is defined.

Generate client SDK dependencies:

cd <module>  # E.g. kms-filters
rm -rf build
mkdir build && cd build
cmake .. -DGENERATE_JAVA_CLIENT_PROJECT=TRUE -DDISABLE_LIBRARIES_GENERATION=TRUE
cd java
mvn clean install

Generate client SDK:
```
cd kurento-java
mvn clean install
```
At this point, the new Java packages have been generated and installed in the local repository. Your Java application can now make use of any changes that were introduced in the API.

Known problems ¶

Sometimes the GStreamer fork doesn’t compile correctly. Try again.
Some unit tests can fail, especially if the storage server (which contains some required input files) is having connectivity issues. If tests fail, packages are not generated. To skip tests, edit the file debian/rules and change -DGENERATE_TESTS=TRUE to -DGENERATE_TESTS=FALSE -DDISABLE_TESTS=TRUE.

Continuous Integration¶

We have two types of repositories containing Debian packages: either Release or Nightly builds of the KMS Main Repositories and KMS Fork Repositories.

Each of these types of repos are made available for two Long-Term Support (LTS) versions of Ubuntu: Ubuntu 16.04 (Xenial) and Ubuntu 18.04 (Bionic) (64-bits only).

After some exploration of the different options we had, in the end we settled on the option to have self-contained repos, where all Kurento packages are stored and no dependencies with additional repositories are needed.

There is an independent repository for each released version of Kurento, and one single repository for nightly builds:

Release: deb http://ubuntu.openvidu.io/<KmsVersion> <UbuntuCodename> kms6
Nightly: deb http://ubuntu.openvidu.io/dev <UbuntuCodename> kms6

Here, <KmsVersion> is any of the released versions of KMS (e.g. 6.7.2, 6.8.1, 6.9.0, etc.) and <UbuntuCodename> is the name of each supported Ubuntu version (e.g. xenial, bionic, etc. if more added in the future.)

We also have several Continuous-Integration (CI) jobs such that new nightly packages can be built from each Git repository’s master branch, to be then uploaded to the nightly repositories.

All scripts used by CI are stored in the Git repo adm-scripts.

Release Procedures ¶

Table of Contents

Release Procedures

Introduction ¶

Kurento as a project spans across a multitude of different technologies and languages, each of them with their sets of conventions and best practices. This document aims to summarize all release procedures that apply to each one of the modules that compose the Kurento project. The main form of categorization is by technology type: C/C++ based modules, Java modules, JavaScript modules, and others.

General considerations ¶

Lists of projects in this document are sorted according to the repository lists given in Code repositories.
Kurento projects to be released have supposedly been under development, and will have development version numbers:
- In Java (Maven) projects, development versions are indicated by the suffix -SNAPSHOT after the version number. Example: 6.9.1-SNAPSHOT.
- In C/C++ (CMake) projects, development versions are indicated by the suffix -dev after the version number. Example: 6.9.1-dev.
These suffixes must be removed for release, and then recovered again to resume development.
All dependencies to development versions will be changed to a release version during the release procedure. Concerning people will be asked to choose an appropriate release version for each development dependency.
Tags are named with the version number of the release. Example: 6.9.0.
Contrary to the project version, the Debian package versions don’t contain development suffixes, and should always be of the form 1.2.3-0kurento1:
- The first part (1.2.3) is the project’s base version number.
- The second part (0kurento1) is the Debian package revision. The first number (0) means that the package only exists in Kurento (not in Debian or Ubuntu); this is typically the case for the projects forked by Kurento. The rest (kurento1) means that this is the first package released by Kurento for the corresponding base version.
Please check the Debian Policy Manual and this Ask Ubuntu answer for more information about the package versions.

Note

Most Kurento fork packages have a Debian package revision that starts with 1 instead of 0. This was due to a mistake, but changing it back to 0 would cause more problems than solutions so we’re maintaining those until the projects get updated to a newer base version.
Kurento uses Semantic Versioning. Whenever you need to decide what is going to be the definitive release version for a new release, try to follow the SemVer guidelines:
```
Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards-compatible manner, and
3. PATCH version when you make backwards-compatible bug fixes.
```
Please refer to https://semver.org/ for more information.

Example

If the last Kurento release was 6.8.2 (with e.g. Debian package version 6.8.2-0kurento3, because it had been repackaged 3 times) then after release the project versions should have been left as 6.8.3-dev (or 6.8.3-SNAPSHOT for Java components).

If the next release of Kurento only includes patches, then the next version number 6.8.3 is already good. However, maybe our release includes new functionality, which according to Semantic Versioning should be accompanied with a bump in the minor version number, so the next release version number should be 6.9.0. The Debian package version is reset accordignly, so the full version is 6.9.0-0kurento1.

If you are repackaging an already released version (for example, because maybe after release you found out that the packages fail to install) then just increment the Debian package version: 0kurento2.

Note

Made a mistake?

Don’t be afraid of applying some Git magic to solve mistakes during the release process. Here are some which can be useful:

How to remove a release tag?
- Remove the local tag:
```
git tag --delete <TagName>
```
- Remove the remote tag:
```
git push --delete origin <TagName>
```
How to push just a local tag?
```
git push origin <TagName>
```
How to ammend a commit and push it again?

See: https://www.atlassian.com/git/tutorials/rewriting-history#git-commit–amend
```
# <Remove Tag>
# <Amend>
# <Create Tag>
git push --force --follow-tags
```

Warning

As of this writing, there is a mix of methods in the CI scripts (adm-scripts) when it comes to handle the release versions. The instructions in this document favor creating and pushing git tags manually in the developer computer, however some projects also make use of the script kurento_check_version.sh, which tries to detect when a project’s version is not a development snapshot, then creates and pushes a git tag automatically. However if the tag alreeady exists (created manually by the developer), then the git tag command fails, and this script prints a warning message before continuing with its work.

We’ve been toying with different methodologies between handling the tags automatically in CI or handling them manually by the developer before releasing new versions; both of these methods have pros and cons. For example, if tags are handled manually by the developer, solving mistakes in the release process becomes simpler because there are no surprises from CI creating tags inadvertently; on the other hand, leaving them to be created by CI seems to simplify a bit the release process, but not really by a big margin.

Fork Repositories ¶

This graph shows the dependencies between forked projects used by Kurento:

$digraph dependencies_forks { bgcolor = "transparent"; fontname = "Bitstream Vera Sans"; fontsize = 8; size = "12,8"; rankdir = "RL"; // Kurento external libraries "jsoncpp"; "libsrtp"; "openh264"; "usrsctp"; "gstreamer"; "gst-plugins-base" -> "gstreamer"; "gst-plugins-good" -> "gst-plugins-base"; "gst-plugins-bad" -> {"gst-plugins-base" "libsrtp" "openh264"}; "gst-plugins-ugly" -> "gst-plugins-base"; "gst-libav" -> "gst-plugins-base"; "openwebrtc-gst-plugins" -> {"gstreamer" "gst-plugins-base" "usrsctp"}; "libnice" -> "gstreamer"; }$

Projects forked by Kurento¶

Release order:

jsoncpp
libsrtp
openh264
usrsctp
gstreamer
gst-plugins-base
gst-plugins-good
gst-plugins-bad
gst-plugins-ugly
gst-libav
openwebrtc-gst-plugins
libnice

For each project above:

Prepare release.
Push a new tag to Git.
Move to next development version.

Release steps ¶

Set the Debian package version, commit the results, and create a tag.

cd gst-plugins-bad

# Edit these
NEW_VERSION="0.1.15"
NEW_DEBIAN="1kurento3"

PACKAGE_VERSION="${NEW_VERSION}-${NEW_DEBIAN}"
COMMIT_MSG="Prepare release $PACKAGE_VERSION"

gbp dch \
    --ignore-branch \
    --git-author \
    --spawn-editor=never \
    --new-version="$PACKAGE_VERSION" \
    ./debian/

SNAPSHOT_ENTRY="* UNRELEASED"
RELEASE_ENTRY="* $COMMIT_MSG"

# First appearance of 'UNRELEASED': Put our commit message
sed --in-place --expression="0,/${SNAPSHOT_ENTRY}/{s/${SNAPSHOT_ENTRY}/${RELEASE_ENTRY}/}" \
    ./debian/changelog

# Remaining appearances of 'UNRELEASED' (if any): Delete line
sed --in-place --expression="/${SNAPSHOT_ENTRY}/d" \
    ./debian/changelog

git add debian/changelog
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$PACKAGE_VERSION"
git push --follow-tags

Follow on with releasing Kurento Media Server.

Kurento Media Server ¶

All KMS projects:

$digraph dependencies_kms { bgcolor = "transparent"; fontname = "Bitstream Vera Sans"; fontsize = 8; size = "12,8"; rankdir = "RL"; // KMS main components { rank = "same"; "kurento-module-creator"; "kms-cmake-utils"; } "kms-jsonrpc" -> "kms-cmake-utils"; "kms-core" -> {"kurento-module-creator" "kms-cmake-utils" "kms-jsonrpc"}; "kms-elements" -> "kms-core"; "kms-filters" -> "kms-elements"; "kurento-media-server" -> {"kms-core" "kms-elements" "kms-filters"}; // KMS extra modules "kms-chroma" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-crowddetector" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-datachannelexample" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-platedetector" -> {"kms-core" "kms-elements" "kms-filters"}; "kms-pointerdetector" -> {"kms-core" "kms-elements" "kms-filters"}; }$

Projects that are part of Kurento Media Server¶

Release order:

kurento-module-creator
kms-cmake-utils
kms-jsonrpc
kms-core
kms-elements
kms-filters
kurento-media-server
kms-chroma
kms-crowddetector
kms-platedetector
kms-pointerdetector

For each project above:

Prepare release.
Push a new tag to Git.
Move to next development version.

Preparation: kurento-module-creator ¶

If kurento-maven-plugin is going to get also a new release, then edit the file kurento-module-creator/src/main/templates/maven/model_pom_xml.ftl to update the plugin version in the auto-generation template:

   <groupId>org.kurento</groupId>
   <artifactId>kurento-maven-plugin</artifactId>
-  <version>6.8.2</version>
+  <version>6.9.0</version>
   <executions>

Preparation: KMS API Java modules ¶

Test the KMS API Java module generation (local check).

apt-get install --yes \
    kurento-module-creator \
    kms-cmake-utils \
    kms-jsonrpc-dev \
    kms-core-dev \
    kms-elements-dev \
    kms-filters-dev

cd kms-omni-build

for DIR in kms-core kms-elements kms-filters; do
    pushd "$DIR"
    mkdir build && cd build/
    cmake .. -DGENERATE_JAVA_CLIENT_PROJECT=TRUE -DDISABLE_LIBRARIES_GENERATION=TRUE
    cd java/
    mvn --batch-mode clean install -Dmaven.test.skip=true
    popd
done

Release steps ¶

Review changes in all KMS sub-projects, and edit kurento-media-server/CHANGELOG.md to add them.

You can use this command to get a list of commit messages since last release:
```
git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
```
Decide what is going to be the definitive release version. For this, follow the SemVer guidelines, as explained above in General considerations.
Set the definitive release version in all projects. Use the helper script kms-omni-build/bin/set-versions.sh to set version numbers, commit the results, and create a tag.
```
cd kms-omni-build
./bin/set-versions.sh <ReleaseVersion> --debian <DebianVersion> \
    --release --commit --tag
```
- Example
  
  To set all project versions to 6.9.0, all Debian package versions to 6.9.0-0kurento1, commit everything, and create the tag 6.9.0, run this:
```
cd kms-omni-build
./bin/set-versions.sh 6.9.0 --debian 0kurento1 \
    --release --commit --tag
```
Now push changes:
```
git submodule foreach 'git push --follow-tags'
```

It’s also nice to update the git-submodule references of the all-in-one repo kms-omni-build, and create a tag just like in all the other repos.

NEW_VERSION="6.9.0"
COMMIT_MSG="Prepare release $NEW_VERSION"

cd kms-omni-build
git add .
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
git push --follow-tags

Start the KMS CI job with the parameters JOB_RELEASE ENABLED and JOB_ONLY_KMS DISABLED.
Wait until all packages get created and published correctly. Fix any issues that appear.

The KMS CI job is a Jenkins MultiJob Project. If it fails at any stage, after fixing the cause of the error it’s not needed to start the job again from the beginning; instead, it is possible to resume the build from the point it was before the failure. For this, just open the latest build number that failed (with a red marker in the Build History panel at the left of the job page); in the description of the build, the action Resume build is available on the left side.
Check that the Auto-Generated API Client JavaScript repos have been updated (which should happen as part of the CI jobs for all Kurento Media Server modules that contain API Definition files (.KMD).
- kms-core -> kurento-client-core-js
- kms-elements -> kurento-client-elements-js
- kms-filters -> kurento-client-filters-js
- kms-chroma -> kurento-module-chroma-js
- kms-crowddetector -> kurento-module-crowddetector-js
- kms-platedetector -> kurento-module-platedetector-js
- kms-pointerdetector -> kurento-module-pointerdetector-js
When all repos have been released, and CI jobs have finished successfully,
- Open the Nexus Sonatype Staging Repositories section.
- Select kurento repository.
- Inspect contents to ensure they are as expected:
  - kurento-module-creator (if it was released)
  - kms-api-core
  - kms-api-elements
  - kms-api-filters
- Close repository.
- Wait a bit.
- Refresh.
- Release repository.
- Maven artifacts will be available after 10 minutes.
AFTER THE WHOLE RELEASE HAS BEEN COMPLETED: Set the next development version in all projects. To choose the next version number, increment the patch number. Use the helper script kms-omni-build/bin/set-versions.sh to set version numbers and commit.
```
cd kms-omni-build
./bin/set-versions.sh <NextVersion> --debian <DebianVersion> \
    --commit
```
- Example
  
  If the last release has been 6.9.0, then the next development version number should be 6.9.1:
```
cd kms-omni-build
./bin/set-versions.sh 6.9.1 --debian 0kurento1 \
    --commit
```
Now push changes:
```
git submodule foreach 'git push'
```

Kurento JavaScript client ¶

Release order:

kurento-jsonrpc-js
kurento-utils-js
kurento-client-js
kurento-tutorial-js
kurento-tutorial-node

For each project above:

Prepare release.
Push a new tag to Git.
Move to next development version.

Release steps ¶

Decide what is going to be the definitive release version. For this, follow the SemVer guidelines, as explained above in General considerations.
Set the definitive release version in all projects. This operation varies between projects.
- kurento-jsonrpc-js, kurento-utils-js, kurento-client-js: in file package.json.
- kurento-tutorial-js: in each one of the bower.json files.
- kurento-tutorial-node: in each one of the package.json files.
Review all dependencies to remove -dev versions.

This command can be used to search for all -dev versions:
```
grep -Fr -- '-dev"'
```

Test the build.

PROJECTS=(
    kurento-jsonrpc-js
    kurento-utils-js
    kurento-client-js
)

for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    npm install
    node_modules/.bin/grunt
    node_modules/.bin/grunt sync:bower
    popd  # $PROJECT
done

If the beautifier step fails, use Grunt to run the beautifier and fix all files that need it.

npm install
node_modules/.bin/grunt jsbeautifier::file:<FilePath>.js

Some times it’s needed to run Grunt a couple of times until it ends without errors.

All-In-One script.

(Note: Always use mvn --batch-mode if you copy this to an actual script!)

NEW_VERSION="6.9.0"
COMMIT_MSG="Prepare release $NEW_VERSION"

PROJECTS=(
    kurento-jsonrpc-js
    kurento-utils-js
    kurento-client-js
    kurento-tutorial-js
    kurento-tutorial-node
)

for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    git pull --rebase
    for FILE in $(find . -name *.json); do
        TEMP="$(mktemp)"
        jq "if has(\"version\") then .version = \"$NEW_VERSION\" else . end" \
            "$FILE" > "$TEMP" && mv "$TEMP" "$FILE"
        git add "$FILE"
    done
    git commit -m "$COMMIT_MSG"
    git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
    git push --follow-tags
    popd  # $PROJECT
done

AFTER THE WHOLE RELEASE HAS BEEN COMPLETED: Set the next development version in all projects. To choose the next version number, increment the patch number and add “-dev”.

All-In-One script.

NEW_VERSION="6.9.1-dev"
COMMIT_MSG="Prepare for next development iteration"

PROJECTS=(
    kurento-jsonrpc-js
    kurento-utils-js
    kurento-client-js
    kurento-tutorial-js
    kurento-tutorial-node
)

for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    for FILE in $(find . -name *.json); do
        TEMP="$(mktemp)"
        jq "if has(\"version\") then .version = \"$NEW_VERSION\" else . end" \
            "$FILE" > "$TEMP" && mv "$TEMP" "$FILE"
        git add "$FILE"
    done
    git commit -m "$COMMIT_MSG"
    git push
    popd  # $PROJECT
done

Kurento Maven plugin ¶

Review changes in all KMS sub-projects, and edit changelog to add them.

You can use this command to get a list of commit messages since last release:
```
git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
```
Decide what is going to be the definitive release version. For this, follow the SemVer guidelines, as explained above in General considerations.

Set the definitive release version in pom.xml. Remove “-SNAPSHOT”.

   <groupId>org.kurento</groupId>
   <artifactId>kurento-maven-plugin</artifactId>
-  <version>1.2.3-SNAPSHOT</version>
+  <version>1.2.3</version>
   <packaging>maven-plugin</packaging>

Git add, commit, tag, and push.

NEW_VERSION="1.2.3"
COMMIT_MSG="Prepare release $NEW_VERSION"
git add pom.xml changelog
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
git push --follow-tags

The CI jobs should start automatically; some tests are run as a result of this commit, so you should wait for their completion.

AFTER THE WHOLE RELEASE HAS BEEN COMPLETED: Set the next development version in all projects. To choose the next version number, increment the patch number and add “-SNAPSHOT”.

   <groupId>org.kurento</groupId>
   <artifactId>kurento-maven-plugin</artifactId>
-  <version>1.2.3</version>
+  <version>1.2.4-SNAPSHOT</version>
   <packaging>maven-plugin</packaging>

Git add, commit, and push.

COMMIT_MSG="Prepare for next development iteration"
git add pom.xml
git commit -m "$COMMIT_MSG"
git push

Kurento Java client ¶

Release order:

kurento-qa-pom
kurento-java
kurento-tutorial-java
kurento-tutorial-test

For each project above:

Prepare release.
Push a new tag to Git.
Move to next development version.

Preparation: kurento-java ¶

If there have been changes in the API of Kurento Media Server modules (in the .KMD JSON files), update the corresponding versions in kurento-parent-pom/pom.xml:

    <properties>
-   <version.kms-api-core>6.8.2</version.kms-api-core>
-   <version.kms-api-elements>6.8.2</version.kms-api-elements>
-   <version.kms-api-filters>6.8.2</version.kms-api-filters>
+   <version.kms-api-core>6.9.0</version.kms-api-core>
+   <version.kms-api-elements>6.9.0</version.kms-api-elements>
+   <version.kms-api-filters>6.9.0</version.kms-api-filters>

Doing this ensures that the Java client gets generated according to the latest versions of the API definitions.

Similarly, update the version numbers of any other Kurento project that has been updated:

<version.kurento-utils-js>6.7.0</version.kurento-utils-js>
<version.kurento-maven-plugin>6.7.0</version.kurento-maven-plugin>

<version.kurento-chroma>6.6.0</version.kurento-chroma>
<version.kurento-crowddetector>6.6.0</version.kurento-crowddetector>
<version.kurento-platedetector>6.6.0</version.kurento-platedetector>
<version.kurento-pointerdetector>6.6.0</version.kurento-pointerdetector>

Release steps ¶

Decide what is going to be the definitive release version. For this, follow the SemVer guidelines, as explained above in General considerations.
Set the definitive release version in all projects. This operation varies between projects.

Note

kurento-tutorial-java and kurento-tutorial-test require that kurento-java has been installed locally before being able to change their version numbers programmatically with Maven.
Review all dependencies to remove SNAPSHOT versions.

Note

In project kurento-java, all dependencies are defined as properties in the file kurento-parent-pom/pom.xml.

This command can be used to search for all SNAPSHOT versions:
```
grep -Fr -- '-SNAPSHOT'
```

Test the build.

Use the profile ‘kurento-release’ to enforce no SNAPSHOT dependencies are present.

pushd kurento-qa-pom
mvn -U clean install -Dmaven.test.skip=true -Pkurento-release
popd  # kurento-qa-pom

pushd kurento-java
mvn -U clean install -Dmaven.test.skip=true -Pkurento-release
popd  # kurento-java

PROJECTS=(kurento-tutorial-java kurento-tutorial-test)
for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    mvn -U clean install -Dmaven.test.skip=true -Pkurento-release
    popd  # $PROJECT
done

All-In-One script:

(Note: Always use mvn --batch-mode if you copy this to an actual script!)

NEW_VERSION="6.9.0"
COMMIT_MSG="Prepare release $NEW_VERSION"

pushd kurento-qa-pom
git pull --rebase
mvn versions:set -DgenerateBackupPoms=false \
    -DnewVersion="$NEW_VERSION"
git ls-files --modified | grep 'pom.xml' | xargs -r git add
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
git push --follow-tags
popd  # kurento-qa-pom

pushd kurento-java
git pull --rebase
mvn versions:set -DgenerateBackupPoms=false \
    -DnewVersion="$NEW_VERSION" \
    --file kurento-parent-pom/pom.xml
mvn -U clean install -Dmaven.test.skip=true \
    -Pkurento-release
git clean -xdf  # Delete build files
git ls-files --modified | grep 'pom.xml' | xargs -r git add
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
git push --follow-tags
popd  # kurento-java

PROJECTS=(kurento-tutorial-java kurento-tutorial-test)
for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    git pull --rebase
    mvn versions:update-parent -DgenerateBackupPoms=false \
        -DallowSnapshots=false \
        -DparentVersion="[${NEW_VERSION}]"
    mvn -N versions:update-child-modules -DgenerateBackupPoms=false \
        -DallowSnapshots=false
    git clean -xdf  # Delete build files
    git ls-files --modified | grep 'pom.xml' | xargs -r git add
    git commit -m "$COMMIT_MSG"
    git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
    git push --follow-tags
    popd  # $PROJECT
done

When all repos have been released, and CI jobs have finished successfully:
- Open the Nexus Sonatype Staging Repositories section.
- Select kurento repositories.
- Inspect contents to ensure they are as expected: kurento-java, etc.
- Close repositories.
- Wait a bit.
- Refresh.
- Release repositories.
- Maven artifacts will be available after 10 minutes.
- Open the Nexus Sonatype Staging Repositories section.
- Select kurento repository.
- Inspect contents to ensure they are as expected:
  - kurento-client
  - kurento-commons
  - kurento-integration-tests
  - kurento-java
  - kurento-jsonrpc
  - kurento-jsonrpc-client
  - kurento-jsonrpc-client-jetty
  - kurento-jsonrpc-server
  - kurento-parent-pom
  - kurento-repository (ABANDONED PROJECT)
  - kurento-repository-client (ABANDONED)
  - kurento-repository-internal (ABANDONED)
  - kurento-test
- Close repository.
- Wait a bit.
- Refresh.
- Release repository.
- Maven artifacts will be available after 10 minutes.

AFTER THE WHOLE RELEASE HAS BEEN COMPLETED: Set the next development version in all projects. To choose the next version number, increment the patch number and add “-SNAPSHOT”.

Note

Maven can do this automatically with the Maven Versions Plugin.

All-In-One script:

(Note: Always use mvn --batch-mode if you copy this to an actual script!)

COMMIT_MSG="Prepare for next development iteration"

pushd kurento-qa-pom
mvn versions:set -DgenerateBackupPoms=false \
    -DnextSnapshot=true
git ls-files --modified | grep 'pom.xml' | xargs -r git add
git commit -m "$COMMIT_MSG"
git push
popd  # kurento-qa-pom

pushd kurento-java
mvn versions:set -DgenerateBackupPoms=false \
    -DnextSnapshot=true \
    --file kurento-parent-pom/pom.xml
mvn -U clean install -Dmaven.test.skip=true
git clean -xdf  # Delete build files
git ls-files --modified | grep 'pom.xml' | xargs -r git add
git commit -m "$COMMIT_MSG"
git push
popd  # kurento-java

PROJECTS=(kurento-tutorial-java kurento-tutorial-test)
for PROJECT in "${PROJECTS[@]}"; do
    pushd "$PROJECT"
    mvn versions:update-parent -DgenerateBackupPoms=false \
        -DallowSnapshots=true
    mvn -N versions:update-child-modules -DgenerateBackupPoms=false \
        -DallowSnapshots=true
    git clean -xdf  # Delete build files
    git ls-files --modified | grep 'pom.xml' | xargs -r git add
    git commit -m "$COMMIT_MSG"
    git push
    popd  # $PROJECT
done

Docker images ¶

A new set of development images is deployed to Kurento Docker Hub on each nightly build. Besides, a release version will be published as part of the CI jobs chain when the KMS CI job is triggered.

Kurento documentation ¶

The documentation scripts will download both Java and JavaScript clients, generate HTML Javadoc / Jsdoc pages from them, and embed everything into a static section.

For this reason, the documentation must be built only after all the other modules have been released.

Ensure that the whole nightly CI chain works:

Job doc-kurento -> job doc-kurento-readthedocs -> New build at ReadTheDocs.
Edit VERSIONS.conf.sh to set all relevant version numbers: version of the documentation itself, and all referred modules and client libraries. These numbers can be different because not all of the Kurento projects are necessarily released with the same frequency.
Test the build locally, check everything works.
```
make html
```
Note that the JavaDoc and JsDoc pages won’t be generated locally if you don’t have your system prepared to do so; also there are some Sphinx constructs or plugins that might fail if you don’t have them ready to use, but the ReadTheDocs servers have them so they should end up working fine.

In any case, always check the final result of the intermediate documentation builds at https://doc-kurento.readthedocs.io/en/latest/, to have an idea of how the final release build will end up looking like.

Git add, commit, tag, and push:

NEW_VERSION="6.9.0"
COMMIT_MSG="Prepare release $NEW_VERSION"

git add VERSIONS.conf.sh
git commit -m "$COMMIT_MSG"
git tag -a -m "$COMMIT_MSG" "$NEW_VERSION"
git push --follow-tags

Run the doc-kurento CI job with the parameter JOB_RELEASE ENABLED.
CI automatically tags Release versions in the ReadTheDocs source repo, doc-kurento-readthedocs, so the release will show up as “stable” in ReadTheDocs.
Open the ReadTheDocs Versions dashboard and in the Default Version ComboBox select the latest version available.

Note

We don’t set the Default Version field to “stable”, because we want that the actual version number gets shown in the upper part of the side panel (below the Kurento logo, above the search box) when users open the documentation. If “stable” was selected here, then users would just see the word “stable” in the mentioned panel.

Security Hardening¶

Hardening is a set of mechanisms that can be activated by turning on several compiler flags, and are commonly used to protect resulting programs against memory corruption attacks. These mechanisms have been standard practice since at least 2011, when the Debian project set on the goal of releasing all their packages with the security hardening build flags enabled [1]. Ubuntu has also followed the same policy regarding their own build procedures [2].

Kurento Media Server had been lagging in this respect, and old releases only implemented the standard Debian hardening options that are applied by the dpkg-buildflags tool by default:

Format string checks (-Wformat -Werror=format-security) [3]. These options instruct the compiler to make sure that the arguments supplied to string functions such as printf and scanf have types appropriate to the format string specified, and that the conversions specified in the format string make sense.
Fortify Source (-D_FORTIFY_SOURCE=2) [4]. When this macro is defined at compilation time, several compile-time and run-time protections are enabled around unsafe use of some glibc string and memory functions such as memcpy and strcpy, with get replaced by their safer counterparts. This feature can prevent some buffer overflow attacks, but it requires optimization level -O1 or higher so it is not enabled in Debug builds (which use -O0).
Stack protector (-fstack-protector-strong) [5]. This compiler option provides a randomized stack canary that protects against stack smashing attacks that could lead to buffer overflows, and reduces the chances of arbitrary code execution via controlling return address destinations.
Read-Only Relocations (RELRO) (-Wl,-z,relro). This linker option marks any regions of the relocation table as “read-only” if they were resolved before execution begins. This reduces the possible areas of memory in a program that can be used by an attacker that performs a successful GOT-overwrite memory corruption exploit. This option works best with the linker’s Immediate Binding mode, which forces all regions of the relocation table to be resolved before execution begins. However, immediate binding is disabled by default.

Starting from version 6.7, KMS also implements these extra hardening measurements:

Position Independent Code (-fPIC) / Position Independent Executable (-fPIE -pie) [6]. Allows taking advantage of the Address Space Layout Randomization (ASLR) protection offered by the Kernel. This protects against Return-Oriented Programming (ROP) attacks and generally frustrates memory corruption attacks. This option was initially made the default in Ubuntu 16.10 for some selected architectures, and in Ubuntu 17.10 was finally enabled by default across all architectures supported by Ubuntu.

Note

The PIC/PIE feature adds a very valuable protection against attacks, but has one important requisite: all shared objects must be compiled as position-independent code. If your shared library has stopped linking with KMS, or your plugin stopped loading at run-time, try recompiling your code with the -fPIC option.
Immediate Binding (-Wl,-z,now). This linker option improves upon the Read-Only Relocations (RELRO) option, by forcing that all dynamic symbols get resolved at start-up (instead of on-demand). Combined with the RELRO flag, this means that the GOT can be made entirely read-only, which prevents even more types of GOT-overwrite memory corruption attacks.

Hardening validation¶

Debian-based distributions provide the hardening-check tool (package hardening-includes), which can be used to check if a binary file (either an executable or a shared library) was properly hardened:

$ hardening-check /usr/sbin/sshd
/usr/sbin/sshd:
Position Independent Executable: yes
Stack protected: yes
Fortify Source functions: yes
Read-only relocations: yes
Immediate binding: yes

Hardening in Kurento¶

Since version 6.7, Kurento Media Server is built with all the mentioned hardening measurements. All required flags are added in the Debian package generation step, by setting the environment variable DEB_BUILD_MAINT_OPTIONS to hardening=+all, as described by Debian hardening options. This variable is injected into the build environment by the CMake module kms-cmake-utils/CMake/CommonBuildFlags.cmake, which is included by all modules of KMS.

PIC/PIE in GCC¶

This section explains how the Position Independent Code (PIC) and Position Independent Executable (PIE) features are intended to be used (in GCC). Proper use of these is required to achieve correct application of ASLR by the Kernel.

PIC must be enabled in the compiler for compilation units that will end up linked into a shared library. Note that this includes objects that get packed as a static library before being linked into a shared library.
PIC must be enabled in the linker for shared libraries.
PIE or PIC (the former being the recommended one) must be enabled in the compiler for compilation units that will end up linked into an executable file. Note that this includes objects that get packed as a static library before being linked into the executable.
PIE must be enabled in the linker for executable files.

Now follows some examples of applying these rules into an hypothetical project composed of one shared library and one executable file:

The shared library (SHARED.so) is composed of 4 source files:
- A.c and B.c are compiled first into a static library: AB.a. GCC flags: -fPIC.
- C.c and D.c are compiled into object files C.o and D.o. GCC flags: -fPIC.
- AB.a, C.o, and D.o are linked into a shared library: SHARED.so. GCC flags: -shared -fPIC.
The executable file (PROGRAM) is composed of 4 source files:
- E.c and F.c are compiled first into a static library: EF.a. GCC flags: -fPIE (*).
- G.c and H.c are compiled into object files G.o and H.o. GCC flags: -fPIE (*).
- EF.a, G.o, and H.o are linked into an executable file: PROGRAM. GCC flags: -pie -fPIE (… -lSHARED).

(*): In these cases, it is also possible to compile these files with -fPIC, although -fPIE is recommended. It is also possible to mix both; for example E.c and F.c can be compiled with -fPIC, while G.c and H.c are compiled with -fPIE (empirically tested, it works fine).

See also

Options for Code Generation Conventions: See -fPIC, -fPIE.
Options for Linking: See -shared, -pie.
dpkg-buildflags: See FEATURE AREAS > hardening > pie.

PIC/PIE in CMake¶

CMake has partial native support to enable PIC/PIE in a project, via the POSITION_INDEPENDENT_CODE and CMAKE_POSITION_INDEPENDENT_CODE variables. We consider it “partial” because these variables add the corresponding flags for the compilation steps, but the flag -pie is not automatically added to the linker step.

We raised awareness about this issue in their bug tracker: POSITION_INDEPENDENT_CODE does not add -pie.

The effect of setting POSITION_INDEPENDENT_CODE to ON for a CMake target (or setting CMAKE_POSITION_INDEPENDENT_CODE for the whole project), is the following:

If the target is a library, the flag -fPIC is added by CMake to the compilation and linker steps.
If the target is an executable, the flag -fPIE is added by CMake to the compilation and linker steps.

However, CMake is lacking that it does not add the flag -pie to the linker step of executable targets, so final executable programs are not properly hardened for ASLR protection by the Kernel.

Kurento Media Server works around this limitation of CMake by doing this in the CMake configuration:

# Use "-fPIC" / "-fPIE" for all targets by default, including static libs
set(CMAKE_POSITION_INDEPENDENT_CODE ON)

# CMake doesn't add "-pie" by default for executables (CMake issue #14983)
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pie")

It would be nice if CMake took over the whole process of generating valid PIC/PIE libraries and executables, by ensuring that all needed flags are added in the correct places. It’s actually very close to that, by only missing the -pie flag while linking executable programs.

Footnotes

[1]	https://wiki.debian.org/Hardening#Notes_on_Memory_Corruption_Mitigation_Methods

[2]	https://wiki.ubuntu.com/Security/Features#Userspace_Hardening

[3]	https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html

[4]	http://man7.org/linux/man-pages/man7/feature_test_macros.7.html

[5]	https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html

[6]	https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html

Writing this documentation¶

Although each of the Kurento repositories contains a README file, the main source of truth for up-to-date information about Kurento is this documentation you are reading right now, hosted at Read The Docs.

The final deliverable form of the documentation is obtained by following a 3-step process:

The source files are written in a markup language called reStructuredText (reST). This format is less known that the popular Markdown, but it is much more powerful and adequate for long-form documentation writing.
The source files, written in reST format, are processed and converted to other deliverable formats by Sphinx, a documentation processing tool which adds some layers of useful syntax to the reST baseline, and also takes care of generating all the documents in their final form.

Finally, the generated HTML files are hosted in Read The Docs, the service of choice for lots of open-source projects. Actually Read The Docs is not only the hosting, but they also perform the Sphinx generation step itself: they provide a Continuous Integration system that watches a Git repository and triggers a new documentation build each time it detects changes.

Kurento documentation files are written using both the basic features of reST, and the extra features that are provided by Sphinx. The reST language itself can be learned by checking any reference documents such as the reStructuredText quick reference and the reStructuredText Primer.

Sphinx adds its own set of useful markup elements, to make reST even more useful for writing documentation. To learn more about this, the most relevant section in their documentation is Sphinx Markup Constructs.

Besides the extra markup added by Sphinx, there is also the possibility to use Sphinx Extensions, which each one does in turn add its own markup to extend even more the capabilities of the language. For example, as of this writing we are using sphinx.ext.graphviz and sphinx-ext-wikipedia extensions, to easily insert links to Wikipedia articles and embedded diagrams in the documents.

Building locally¶

If you are writing documentation for Kurento, there is no need to commit every change and push to see the result only to discover that an image doesn’t show up as you expected, or some small mistake breaks a table layout.

First of all, it’s a very good idea to use a text editor that provides spell checking and live-preview visualization of reST files; this alone will help catching most grammatical and syntactic mistakes. Visual Studio Code is a great option, it provides extensions for both of these things.

To install VS Code and the two mentioned functions, use their installer, then run these commands:
```
code --install-extension streetsidesoftware.code-spell-checker
code --install-extension lextudio.restructuredtext
```

Secondly, you can build the documentation locally. Just install the required dependencies:

# Ensure that old versions of Sphinx are not installed
sudo apt-get purge --auto-remove \
    python-sphinx \
    sphinx-common \
    python-sphinx-rtd-theme \
    sphinx-rtd-theme-common

# Ensure that Sphinx is not installed with Python 3
sudo pip3 uninstall sphinx
sudo pip3 uninstall sphinx_rtd_theme
sudo pip3 uninstall sphinxcontrib_websupport

# Install Sphinx with Python 2
sudo apt-get install python-pip python-setuptools
sudo pip2 install --upgrade sphinx sphinx_rtd_theme

And then just run make html to build and open your new shiny files:

cd doc-kurento/
make html
firefox build/html/index.html

Kurento documentation Style Guide¶

Paragraph conventions¶

Line breaks: Don’t break the lines. The documentation is a prose text, and not source code, so the typical restrictions of line length don’t apply here. Use automatic line breaks in your editor, if you want. The overall flow of the text should be dictated by the width of the screen where the text is being presented, and not by some arbitrary line length limit.

Inline markup¶

Paths, file names, package names, acronyms, and in general human-oriented words are emphasized with single asterisks (as in *word*). Sample phrases:

This document talks about Kurento Media Server (*KMS*).
All dependency targets are defined in the *CMakeLists.txt* file.
You need to install *libboost-dev* for development.

Code, commands, arguments, environment variables, commit hashes, and in general machine-oriented keywords are emphasized with double back quotes (as in ``word``). Sample phrases:

Use ``apt-get install`` to set up all required packages.
Set ``CMAKE_BUILD_TYPE=Debug`` to build with debugging symbols.
The argument ``--gst-debug`` can be used to control the logging level.

There is no difference between using single asterisks (*word*), and single back quotes (`word`); they get rendered as italic text. So, always use asterisks when wanting to emphasize some text.
As opposed to Markdown, underscores (as in _word_) don’t get rendered, so don’t use them to emphasize text.

Header conventions¶

Header separation: Always separate each header from the preceding paragraph, by using 3 empty lines. The only exception to this rule is when two headers come together (e.g. a document title followed by a section title); in that case, they are separated by just 1 empty line.
Header shape: reST allows to express section headers with any kind of characters that form an underline shape below the section title. We follow these conventions for Kurento documentation files:
1. Level 1 (Document title). Use = above and below:
```
=======
Level 1
=======
```
1. Level 2. Use = below:
```
Level 2
=======
```
1. Level 3. Use -:
```
Level 3
-------
```
1. Level 4. Use ~:
```
Level 4
~~~~~~~
```
1. Level 5. Use ":
```
Level 5
"""""""
```

Sphinx documentation generator¶

Our Sphinx-based project is hosted in the doc-kurento repository. Here, the main entry point for running Sphinx is the Makefile, based on the template that is provided for new projects by Sphinx itself. This Makefile is customized to attend our particular needs, and implements several targets:

init-workdir. This target constitutes the first step to be run before most other targets. Our documentation source files contain substitution keywords in some parts, in the form | KEYWORD |, which is expected to be substituted by some actual value during the generation process. Currently, the only keyword in use is VERSION, which must be expanded to the actual version of the documentation being built.

For example, here is the VERSION_KMS keyword when substituted with its final value: 6.10.0.

Yes, Sphinx does already include a substitutions feature by itself, and the keyword VERSION is precisely one of the supported substitutions. Sadly, this feature of Sphinx is very unreliable. For example, it won’t work if the keyword is located inside a literal code block, or inside an URL. So, we must resort to performing the substitutions by ourselves if we want reliable results.

The source folder is copied into the build directory, and then the substitutions take place over this copy.
langdoc. This target creates the automatically generated reference documentation for each Kurento Client. Currently, this means the Javadoc and Jsdoc documentations for Java and Js clients, respectively. The Kurento client repositories are checked out in the same version as specified by the documentation version file, or in the master branch if no such version tag exists. Then, the client stubs of the Kurento API are automatically generated, and from the resulting source files, the appropriate documentation is automatically generated too.

The langdoc target is usually run before the html target, in order to end up with a complete set of HTML documents that include all the reST documentation with the Javadoc/Jsdoc sections.
dist. This target is a convenience shortcut to generate the documentation in the most commonly requested formats: HTML, PDF and EPUB. All required sub-targets will be run and the resulting files will be left as a compressed package in the dist/ subfolder.
ci-readthedocs. This is a special target that is meant to be called exclusively by our Continuous Integration system. The purpose of this job is to manipulate all the documentation into a state that is a valid input for the Read The Docs CI system. Check the next section for more details.

Read The Docs builds¶

It would be great if Read The Docs worked by simply calling the command make html, as then we would be able to craft a Makefile that would build the complete documentation in one single step (by making the Sphinx’s html target dependent on our init-workdir and langdoc). But alas, they don’t work like this; instead, they run Sphinx directly from their Python environment, rendering our Makefile as useless in their CI.

In order to overcome this limitation, we opted for the simple solution of handling RTD a specifically-crafted Git repository, with the contents that they expect to find. This works as follows:

Read The Docs has been configured to watch for changes in the doc-kurento-readthedocs repo, instead of doc-kurento.
The init-workdir and langdoc targets run locally from our doc-kurento repo.
The resulting files from those targets are copied as-is to the doc-kurento-readthedocs repository.
Everything is then committed and pushed to this later repo, thus triggering a new RTD build.

Testing¶

Software testing is a broad term within software engineering encompassing a wide spectrum of different concepts. Depending on the size of the System Under Test (SUT) and the scenario in which it is exercised, testing can be carried out at different levels. There is no universal classification for all the different testing levels. Nevertheless, the following levels are broadly accepted:

Unit: individual program units are tested. Unit tests typically focus on the functionality of individual objects or methods.
Integration: units are combined to create composite components. Integration tests focus on the interaction of different units.
System: all of the components are integrated and the system is tested as a whole.

There is a special type of system tests called end-to-end (E2E). In E2E tests, the final user is typically impersonated, i.e., simulated using automation techniques. The main benefit of E2E tests is the simulation of real user scenarios in an automated fashion. As described in the rest of this document, a rich variety of E2E has been implemented to assess Kurento.

E2E Tests¶

This section introduces the different types of E2E implemented to assess different parts of Kurento, namely functional, stability, tutorials, and API.

Functional¶

Functional tests are aimed to evaluate a given capability provided by Kurento. These tests have created in Java. You can find the source code in the repository kurento-test within kurento-java. In order to run functional tests, Maven should be used as follows:

git clone https://github.com/Kurento/kurento-java
cd kurento-java
mvn verify -pl kurento-integration-tests/kurento-test -Pintegration -Dgroups=org.kurento.commons.testing.SystemFunctionalTests

By default, these tests required a local Kurento Media Server installed in the machine running the tests. In addition, Chrome and Firefox browsers are also required. For further information about running these tests, please read next section.

The main types of functional tests for Kurento are the following:

WebRTC. Real-time media in the web is one of the core Kurento capabilities, and therefore, a rich test suite to assess the use of WebRTC in Kurento has been implemented. Moreover, two special WebRTC features are also tested:
- Datachannels. A WebRTC data channel allows to send custom data over an active connection to a peer. Tests using Chrome and Firefox has been implemented to check WebRTC datachannels.
- ICE. In order to create media communication between peers avoiding NAT traversal problems, ICE (Interactive Connectivity Establishment) negotiation is used in WebRTC. Kurento ICE tests check this connectivity using different network setups (NATs, reflexive, bridge).
Recorder. Another important capability provided by Kurento is the media archiving. Recorder tests use RecorderEndpoint media element by ensuring that the recorded media is as expected.
Player. KMS’s PlayerEndpoint allows to inject media from seekable or non-seekable sources to a media pipeline. A suite of tests have been implemented to assess this feature.
Composite/Dispatcher. KMS allows to mix media using different media elements (Composite and Dispatcher). These tests are aimed to asses the result of this media mixing.

Stability¶

Stability tests verifies Kurento capabilities in different scenarios:

Running media pipelines in large amount of time.
Using a lot of resources (CPU, memory) of a KMS instance.

Stability tests have been also created using Java, and they are contained in the project kurento-test. Again, we use Maven to execute stability tests against a local KMS and using also local browsers (Chrome, Firefox):

git clone https://github.com/Kurento/kurento-java
cd kurento-java
mvn verify -pl kurento-integration-tests/kurento-test -Pintegration -Dgroups=org.kurento.commons.testing.SystemStabilityTests

Tutorials¶

The documentation of Kurento includes a number of tutorials tutorials which allows to understand Kurento capabilities using ready to be used simple applications. Kurento tutorials have been developed for three technologies: Java, JavaScript, and Node.js. Moreover, for some of the Java tutorials, different E2E tests have been created. These tests are available in the project kurento-tutorial-test. In order to run these tests, Maven should be used:

git clone https://github.com/Kurento/kurento-tutorial-test
cd kurento-tutorial-test
mvn verify

API¶

The Kurento API is available in two languages: Java and JavaScript. For both of them, a test suite has been created to verify the correctness of the Kurento API against a running instance of KMS. In you want to run API tests for Java, as usual for Kurento tests, Maven is required, as follows:

git clone https://github.com/Kurento/kurento-java
cd kurento-java
mvn verify -pl kurento-integration-tests/kurento-client-test -Pintegration -Dgroups=org.kurento.commons.testing.KurentoClientTests

In order to run JavaScript API tests against a running instance of local KMS, the command to be used is the following:

git clone https://github.com/Kurento/kurento-client-js
cd kurento-client-js
npm install
rm -f node_modules/kurento-client && ln -s .. node_modules/kurento-client
npm test

Running Java tests¶

Functional, stability, and Java API tests for Kurento have been created using a custom Java library called Kurento Testing Framework (KTF). For more details about this framework, please take a look to the next section. If you are interested only in running a group of functional or stability E2E tests in order to assess Kurento, please keep reading this section.

Maven is the the way which E2E Kurento are executed. Therefore, in order to run E2E tests, first we need in have Java and Maven installed. The next step is cloning the GitHub repository which contains the test sources. Most of them are located in the kurento-test project, located inside of kurento-java. Inside this project, we need to invoke Maven to execute tests, for example as follows:

git clone https://github.com/Kurento/kurento-java
cd kurento-java
mvn verify -pl kurento-integration-tests/kurento-test -Pintegration -Dgroups=org.kurento.commons.testing.IntegrationTests -Dtest=WebRtcOneLoopbackTest

Let’s take a closer look to the Maven command:

mvn verify: Command to execute the verify goal in Maven, which involves the execution of the unit and integration tests of a Maven project.
-pl kurento-integration-tests/kurento-test: Maven option to select a single project for the goal, in this case kurento-test.
-Dgroups=org.kurento.commons.testing.IntegrationTests: The Kurento E2E test suite is divided into different JUnit 4’s categories. This option allows to select different types of IntegrationTests. The most used values for this group are:
- IntegrationTests: Parent category for all Kurento E2E tests.
- SystemFunctionalTests: To run functional tests (as defined in section before).
- SystemStabilityTests: To run stability tests (as defined in section before).
- KurentoClientTests: To run Java API tests (as defined in section before). If this option is used, the project should be also changed using -pl kurento-integration-tests/kurento-client-test
-Dtest=WebRtcOneLoopbackTest: Although not mandatory, it is highly recommended to select a test or group of test using the parameter -Dtest of Maven. Using this command we can select a test using the Java class name. Moreover, the wildcard * can be used. Kurento tests follow a fixed notation for test naming, and so, this can be used to select a group of tests, as follows:
- -Dtest=WebRtc*: Used to execute all the functional Kurento tests for WebRTC.
- -Dtest=Player*: Used to execute all the functional Kurento tests for player.
- -Dtest=Recorder*: Used to execute all the functional Kurento tests for recorder.
- -Dtest=Composite*: Used to execute all the functional Kurento tests for composite.
- -Dtest=Dispatcher*: Used to execute all the functional Kurento tests for dispatcher.

An HTML report summarizing the results of a test suite executed with KTF is automatically created for Kurento tests. This report is called report.html and it is located by default on the target folder when tests are executed with Maven. The following picture shows an example of the content of this report.

Kurento Test Framework report sample

Kurento tests are highly configurable. This configuration is done simply adding extra JVM parameters (i.e. -Dparameter=value) to the previous Maven command. The following sections summarizes the main test parameters and its default values organized in different categories.

Kurento Media Server¶

Kurento Media Server (KMS) is the heart of Kurento and therefore it must be properly configured in E2E tests. The following table summarizes the main options to setup KMS in these tests:

Parameter	Description	Default value
`test.kms.scope`	Specifies how to start KMS when is internally managed by test: `local`: Try to use local KMS installation. Test will fail is no local KMS is found. `remote`: KMS is a remote host (use `kms.login` and `kms.key` to access using SSH to the remote machine). `docker`: Request the docker daemon to start a KMS container based in the image specified by `test.kms.docker.image.name`. Test will fail if daemon is unable to start KMS container. In order to use this scope, a Docker server should be installed in the machine running tests. In addition, the Docker REST should be available for Docker client (used in test).	`local`
`test.kms.autostart`	Specifies if tests must start KMS (for all possible values of `test.kms.scope`) or an external KMS service is used: `false`: Test must use an external KMS service whose URL is provided by property `kms.ws.uri` `test`: KMS instance is started for before each test execution. KMS is destroyed after test execution. `testsuite`: KMS service is started at the beginning of test suite execution. A test suite is the group of tests to be executed (e.g. all functional tests). KMS service is stopped after test suite execution.	`test`
`kms.ws.uri`	URL of a KMS service. This property is mandatory when service is externally managed (`-Dtest.kms.autostart=false`) and ignored otherwise. Notice this URL must be reachable from Selenium nodes as well as from tests.	`ws://localhost:8888/kurento`
`test.kms.docker.image.name`	KMS docker image used to start a new docker container when KMS service is internally managed by test (`-Dtest.kms.autostart=test` or `testsuite`) with docker scope (`-Dtest.kms.scope=docker`). Ignored if `test.kms.autostart=false`. See available Docker images for KMS in Docker Hub.	`kurento/kurento-media-server-dev:latest`
`test.kms.docker.image.forcepulling`	Force pulling for Docker image for KMS	`true`
`test.kms.debug`	Debug options used to start KMS service when is internally managed by test (`-Dtest.kms.autostart=test` or `testsuite`). Ignored if `test.kms.autostart=false`.	`2,media_server:5,Kurento:5,KurentoMediaServerServiceHandler:7`
`kms.log.folder`	Path of KMS log folder	`/var/log/kurento-media-server`
`kms.command`	Shell command to start KMS	`/usr/bin/kurento-media-server`
`kms.login`	Username to login by SSH in the machine hosting KMS	none
`kms.key`	Certificate path to login by SSH in the machine hosting KMS	none
`kms.gst.plugins`	GST plugins to be used in KMS	none
`test.print.log`	Print KMS logs at the end of a failed test	`true`

For example, in order to run the complete WebRTC functional test suite against a local instance KMS, the Maven would be as follows:

mvn verify -pl kurento-integration-tests/kurento-test -Pintegration -Dgroups=org.kurento.commons.testing.SystemFunctionalTests -Dtest.kms.autostart=false -Dtest=WebRtc*

In this case, an instance of KMS should be available in the machine running the tests. Concretely, KMS should be accessible in the URL ws://localhost:8888/kurento (which is the default value for kms.ws.uri).

Browsers¶

In order to test automatically the web application under test using Kurento, web browsers (typically Chrome or Firefox, which allow to use WebRTC) are required. The options to configure these browsers are summarized in the following table:

Parameter	Description	Default value
`test.selenium.scope`	Specifies the scope used for browsers in Selenium test scenarios: `local`: browser installed in the local machine. `docker`: browser in Docker container (Chrome or Firefox). `saucelabs`: browser in SauceLabs cloud.	`local`
`docker.node.chrome.image`	Docker image identifier for Chrome when browser scope is `docker`.	`elastestbrowsers/chrome:latest`
`docker.node.firefox.image`	Docker image identifier for Firefox when browser scope is `docker`.	`elastestbrowsers/firefox:latest`
`test.selenium.record`	Allow recording the browser while executing a test, and generate a video with the completely test. This feature can be activated (`true`) only if the scope for browsers is `docker`.	`false`
`test.config.file`	Path of a JSON based file with configuration keys (test scenario, see “KTF explained” section for further details). Its content is transparently managed by test infrastructure and passed to tests for configuration purposes.	`test.conf.json`
`test.timezone`	Time zone for dates in browser log traces. This feature is interesting when using Saucelabs browsers, in order to match dates from browsers with KMS. Accepted values are `GMT`, `CET`, etc.	none
`saucelab.user`	User for SauceLabs	none
`saucelab.key`	Key path for SauceLabs	none
`saucelab.idle.timeout`	Idle time in seconds for SauceLabs requests	`120`
`saucelab.command.timeout`	Command timeout for SauceLabs requests	`300`
`saucelab.max.duration`	Maximum duration for a given SauceLabs session (in seconds)	1800

For example, in order to run the complete WebRTC functional test suite using dockerized browsers and recordings, the command would be as follows:

mvn verify -pl kurento-integration-tests/kurento-test -Pintegration -Dgroups=org.kurento.commons.testing.SystemFunctionalTests -Dtest.selenium.scope=docker -Dtest.selenium.record=true -Dtest=WebRtc*

In order to avoid wasting to much space disks, recording are deleted at the end of the test if the test is succeeded. For failed tests, recordings will be available by the default on the path target/surefire-reports/ (this can be change using the property -Dtest.project.path).

Web server¶

Kurento is typically consumed using a web application. E2E tests follow this architecture, and so, a web application up and running in a web server is required. Kurento-test provides a sample web application out-of-the-box aimed to assess main Kurento features. Also, a custom web application for tests can be specified using its URL. The following table summarizes the configuration options for the test web applications.

Parameter	Description	Default value
`test.app.autostart`	Specifies whether test application where Selenium browsers connect must be started by test or if it is externally managed: `false` : Test application is externally managed and not started by test. This is required when the web under test is already online. In this case, the URL where Selenium browsers connects is specified by the properties: `test.host`, `test.port`, `test.path` and `test.protocol`. `test` : test application is started before each test execution. `testsuite`: Test application is started at the beginning of test execution.	`testsuite`
`test.host`	IP address or host name of the URL where Selenium browsers will connect when test application is externally managed (`-Dtest.app.autostart=false`). Notice this address must be reachable by Selenium browsers and hence network topology between browser and test application must be taken into account.	`127.0.0.1`
`test.port`	Specifies port number where test application must bind in order to listen for browser requests.	`7779`
`test.path`	Path of the URL where Selenium connects when test application is externally managed (`-Dtest.app.autostart=false`).	`/`
`test.protocol`	Protocol of the URL where Selenium browsers will connect when test application is externally managed (`-Dtest.app.autostart=false`).	`http`
`test.url.timeout`	Timeout (in seconds) to wait that web under test is available.	`500`

Fake clients¶

In some tests (typically in performance or stability tests), another instance of KMS is used to generate what we call fake clients, which are WebRTC peers which are connected in a WebRTC one to many communication. The KMS used for this features (referred as fake KMS) is controlled with the parameters summarized in the following table:

Parameter	Description	Default value
`fake.kms.scope`	This property is similar to `-Dtest.kms.scope`, except that it affects the KMS used by fake client sessions.	`local`
`fake.kms.ws.uri`	URL of a KMS service used by WebRTC clients. This property is used when service is externally managed (`-Dfake.kms.autostart=false`) and ignored otherwise. If not specified, `kms.ws.uri` is first looked at before using the default value.	`ws://localhost:8888/kurento`
`fake.kms.autostart`	Specifies if tests must start KMS or an external KMS service must be used for fake clients (sessions that use KMS media pipelines instead of the WebRTC stack provided by a web browser): `false`: Test must use an external KMS service whose URL is provided by the property `fake.kms.ws.uri` (with `kms.ws.uri` as fallback). Test will fail if neither properties are provided. `test`: KMS instance is started for before each test execution. KMS is destroyed after test execution. `testsuite`: KMS service is started at the beginning of test suite execution. KMS service is stopped after test suite execution. Following properties are honored when KMS is managed by test: `fake.kms.scope`, `test.kms.docker.image.name`, `test.kms.debug`	`false`

Although available in KTF, the fake clients feature is not very used in the current tests. You can see an example in the stability test LongStabilityCheckMemoryTest.

Other test features¶

Kurento tests can be configured in many different ways. The following table summarizes these miscellaneous features for tests.

Parameter	Description	Default value
`test.num.retries`	Number of retries for failed tests	`1`
`test.report`	Path for HTML report	`target/report.html`
`test.project.path`	Path for test file output (e.g. recordings).	`target/surefire-reports/`
`test.workspace`	Absolute path of working directory used by tests as temporary storage. Make sure test user has full access to this folder.	`/tmp`
`test.workspace.host`	Absolute path, seen by docker agent, where directory `test.workspace` is mounted. Mandatory when scope is set to docker, as it is used by test infrastructure to share config files. This property is ignored when scope is different from docker.	`none`
`test.files.url`	Path where the files will be for playing	`http://files.openvidu.io`
`test.files.disk`	Absolute path where test files (videos) are located.	`/var/lib/jenkins/test-files`
`test.files.http`	Web server where test files (videos) are located.	`files.openvidu.io`
`project.path`	In Maven reactor projects this is the absolute path of the module where tests are located. This parameter is used by test infrastructure to place test attachments. Notice this parameter must not include a trailing `/`.	`.`
`kms.generate.rtp.pts.stats`	Path where rtp/pst statistics will be stored	`file://WORKSPACE/testClassName`
`bower.kurentoclient.tag`	Tag used by Bower to download kurento-client	none
`bower.kurentoutils.tag`	Tag used by Bower to download kurento-utils.	none
`bower.release.url`	URL from where JavaScript binaries (kurento-client and kurento-utils) will be downloaded. Dependencies will be gathered from Bower if this parameter is not provided.	none
`test.seek.repetitions`	Number of times the tests with seek feature will be executed	`100`
`test.num.sessions`	Number of total sessions executed in stability tests	`50`
`test.screenshare.title`	Title of the window to be shared automatically from tests	`Screen 1`

Kurento Testing Framework explained¶

In order to assess properly Kurento from a final user perspective, a rich suite of E2E tests has been designed and implemented. To that aim, the Kurento Testing Framework (KTF) has been created. KTF is a part of the Kurento project aimed to carry out end-to-end (E2E) tests for Kurento. KTF has been implemented on the top of two well-known open-source testing frameworks: JUnit and Selenium.

KTF provides high level capabilities to perform advanced automated testing for Kurento-based applications. KTF has been implemented in Java, and as usual it is hosted on GitHub, in the project kurento-test. KTF has been designed on the top of JUnit 4, providing a rich hierarchy of classes which are going to act as parent for JUnit 4 tests cases. This hierarchy is the following:

Kurento Testing Framework class hierarchy

The most important classes of this diagram are the following:

KurentoTest: Top class of the KTF. It provides different features out-of-the-box for tests extending this class, namely:
- Improved test lifecycle: KTF enhances the lyfecycle of JUnit 4 test cases, watching the result of tests (passed, failed). Moreover, KTF provides extra annotations to be used in different parts of the test lifecycle, such as FailedTest, FinishedTest, FinishedTestClass, StartedTest, StartedTestClass, or SucceededTest.
- Reporting: As introduced before, an HTML report summarizing the results of a test suite executed with KTF is automatically created for Kurento tests (report.html, located by default on the target folder when tests are executed with Maven).
- Retries mechanism: In order to detect flaky tests, a retries mechanism is present in KTF. This mechanism allows to repeat a failed test a configurable number of times.
KurentoClientTest: It provides an instance of Kurento Media Server (KMS) together with a instance of a Kurento Java Client to control KMS. There are three options to run this KMS (see parameter test.kms.scope):
- Local KMS. To use this option, it is a pre-requisite to have KMS installed in the machine running this type of tests.
- Remote KMS. To use this option, it is a pre-requisite that KMS is installed in a remote host. If this KMS is going to be started by tests, then it is also required to have SSH access to the remote host in which KMS is installed (using parameters kms.login and kms.key).
- KMS in a Docker container. To use this option, it is a pre-requisite to have Docker installed in the machine running this type of tests.
BrowserTest: This class provides wrappers of Selenium WebDriver instances aimed to control a group of web browsers for tests. By default, KTF allows to use Chrome or Firefox as browsers. The scope of these browsers can be configured to use:
- Local browser, i.e. installed in the local machine.
- Remote browser, i.e. installed in the remote machines (using Selenium Grid).
- Docker browsers, i.e. executed in Docker containers.
- Saucelabs browsers. Saucelabs is a cloud solution for web testing. It provides a big number of browsers to be used in Selenium tests. KTF provides seamless integration with Saucelabs.
Test scenario can be configured in BrowserTest tests in two different ways:
- Programmatically using Java. Test scenario uses JUnit 4’s parameterized feature. The Java class TestScenario is used by KTF to configure the scenario, for example as follows:
@Parameters(name = "{index}: {0}") public static Collection<Object[]> data() { TestScenario test = new TestScenario(); test.addBrowser(BrowserConfig.BROWSER, new Browser.Builder().browserType(BrowserType.CHROME) .scope(BrowserScope.LOCAL).webPageType(webPageType).build()); return Arrays.asList(new Object[][] { { test } }); }
- Using a JSON file. KTF allows to setup tests scenarios based on a custom customizable JSON notation. In these JSON files, several test executions can be setup. For each execution, the browser scope can be chosen. For example, the following example shows a test scenario in which two executions are defined. First execution defines two local browsers (identified as peer1 and peer2), Chrome and Firefox respectively. The second execution defines also two browsers, but this time browsers are located in the cloud infrastructure provided by Saucelabs.
{ "executions":[ { "peer1":{ "scope":"local", "browser":"chrome" }, "peer2":{ "scope":"local", "browser":"firefox" } }, { "peer1":{ "scope":"saucelabs", "browser":"explorer", "version":"11" }, "peer2":{ "scope":"saucelabs", "browser":"safari", "version":"36" } } ] }
KurentoClientBrowserTest: This class can be seen as a mixed of the previous ones, since it provides the capability to use KMS (local or dockerized) together with a group of browser test using a test scenario. Moreover, it provides a web server started with each test for testing purposed, with a custom web page available to test WebRTC in Kurento in a easy manner. As can be seen in the diagram before, this class is the parent of a rich variety of different classes. In short, these classes are used to distinguish among different types of tests. See next section for more information.

Test scenario in JSON¶

Test scenario consist of a list of executions, where each execution describes how many browsers must be available and their characteristics. Each browser has an unique identifier (can be any string) meaningful for the test. The following keys can be specified in a JSON test scenario in order to customize individual instances:

scope: Specifies what type of browser infrastructure has to be used by the test execution. This value can be overridden by command line property -Dtest.selenium.scope.
- local: Start the browser as a local process in the same CPU where test is executed.
- docker: Start browser as a docker container.
- saucelabs: Start browser in SauceLabs.
host: IP address or host name of URL used by the browser to execute tests. This value can be overridden by command line property -Dtest.host
port: Port number of the URL used by the browser to execute the test. This value can be overridden by command line property -Dtest.port
path: Path of the URL used by browser to execute the test. This value can be overridden by command line property -Dtest.path
protocol: Protocol of the URL used by browser to execute the test. This value can be overridden by command line property -Dtest.protocol
browser: Specifies the browser platform to be used by the test execution. Test will fail if required browser is not found.
saucelabsUser : SauceLabs user. This property is mandatory for SauceLabs scope and ignored otherwise. Its value can be overridden by command line property -Dsaucelab.user
saucelabsKey: SauceLabs key. This property is mandatory for SauceLabs scope and ignored otherwise. Its value can be overridden by command line property -Dsaucelab.key
version: Version of browser to be used when test is executed in SauceLabs infrastructure. Test will fail if requested version is not found.

Congestion Control (RMCAT)¶

RTP Media Congestion Avoidance Techniques (RMCAT) is an IETF Working Group that aims to develop new protocols which can manage network congestion in the context of RTP streaming. The goals for any congestion control algorithm are:

Preventing network collapse due to congestion.
Allowing multiple flows to share the network fairly.

A good introduction to RMCAT, its history and its context can be found in this blog post by Mozilla: What is RMCAT congestion control, and how will it affect WebRTC?.

As a result of this Working Group, several algorithms have been proposed so far by different vendors:

By Cisco: NADA, A Unified Congestion Control Scheme for Real-Time Media.
By Ericsson: SCReAM, Self-Clocked Rate Adaptation for Multimedia.
By Google: GCC, Google Congestion Control Algorithm for Real-Time Communication.

Google Congestion Control¶

Google’s GCC is the RMCAT algorithm of choice for WebRTC, and it’s used by WebRTC-compatible web browsers. In GCC, both sender and receiver of an RTP session collaborate to find out a realistic estimation of the actual network bandwidth that is available:

The RTP sender generates special timestamps called abs-send-time, and sends them as part of the RTP packet’s Header Extension.
The RTP receiver generates a new type of RTCP Feedback message called Receiver Estimated Maximum Bitrate (REMB). These messages are appended to normal RTCP packets.

Meaning of REMB¶

There has been some misconceptions about what is the exact meaning of the value that is carried by REMB messages. In short, REMB is used for reporting the aggregate bandwidth estimates of the receiver, across all media streams that are sharing the same RTP session.

Sources:

In [rmcat] comments on draft-alvestrand-rmcat-remb-02, this question is responded:

> - the maximum bitrate, is it defined as the maximum bitrate for
>   a particular stream, or the maximum bitrate for the whole "Session" ?

As per GCC, REMB can be used for reporting the sum of the receiver
estimate across all media streams that share the same end-to-end path.
I supposed the SSRCs of the multiple media streams that make up the
aggregate estimate are reported in the block.

In [rtcweb] REMB with many ssrc entries, a similar question is answered, talking about an explicit example:

> If Alice is sending Bob an audio stream (SSRC 1111) and a video stream
> (SSRC 2222) and Bob sends a REMB feedback message with:
>   - bitrate: 100000 (100kbits/s)
>   - ssrcs: 1111, 2222
>
> Does it mean that Alice should limit the sum of her sending audio and
> video bitrates to 100kbits/s? or does it mean that Alice can send
> 100kbits/s of audio and 100kbits/s of video (total = 200)?

The way it was originally designed, it meant that the total should be
100 Kbits/sec. REMB did not take a position on how the sender chose to
allocate bits between the SSRCs, only the total amount of bits sent.

H.264 video codec ¶

Table of Contents

H.264 video codec

NAL Units (NALU)¶

Sources:

An H.264 video is organized into Network Abstraction Layer Units (“NAL units” or “NALU”) that help transporting it with optimal performance depending on whether the transport is stream-oriented or packet-oriented:

For stream-oriented transports: the Byte-Stream Format. The NAL units are delivered as a continuous stream of bytes, which means that some boundary mechanism will be needed for the receiver to detect when one unit ends and the next one starts. This is done with the typical method of adding a unique byte pattern before each unit: The receiver will scan the incoming stream, searching for this 3-byte start code prefix, and finding one will mark the beginning of a new unit.
For packet-oriented transports: the Packet-Transport Format. This is the preferred method for the Advanced Video Coding (AVC) standard, and it builds upon the fact that all data is carried in packets that are already framed by the system transport protocol (such as RTP/UDP), so start code prefix patterns are not needed for identification of the boundaries of NAL units.

The NAL units can contain either actual data video (VCL units) or codec metadata (non-VCL units). Some of the most important types of NAL units are:

Sequence Parameter Set (SPS): This non-VCL NALU contains information required to configure the decoder such as profile, level, resolution, frame rate.
Picture Parameter Set (PPS): Similar to the SPS, this non-VCL NALU contains information on entropy coding mode, slice groups, motion prediction, quantization parameters (QP), and deblocking filters.
Instantaneous Decoder Refresh (IDR): This VCL NALU is a self contained image slice. That is, an IDR can be decoded and displayed without referencing any other NALU save SPS and PPS.
Access Unit Delimiter (AUD): An AUD is an optional NALU that can be use to delimit frames in an elementary stream. It is not required (unless otherwise stated by the container/protocol, like TS), and is often not included in order to save space, but it can be useful to finds the start of a frame without having to fully parse each NALU.

SPS, PPS ¶

A large number of NAL units are combined to form a single video frame; the metadata of such frame would be transmitted in a Picture Parameter Set (PPS). Likewise, a set of PPS would form an actual video sequence, and the metadata for it would be transmitted in a Sequence Parameter Set (SPS). Both PPS and SPS can be sent well ahead of the actual units that will refer to them; then, each individual unit will just contain an index pointing to the corresponding parameter set, so the receiver is able to successfully decode the video.

Details about the exact contents of PPS and SPS packets can be found in the specification document for ISO/IEC 14496-10 (also known as H.264 or MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC)), sections “Sequence parameter set data syntax” and “Picture parameter set RBSP syntax”.

Parameter sets can be sent in-band with the actual video, or sent out-of-band via some channel which might be more reliable than the transport of the video itself. This second option makes sense for transports where it is possible that some corruption or information loss might happen; losing a PPS could prevent decoding of a whole frame, and losing an SPS would be worse as it could render a whole chunk of the video impossible to decode, so it is important to transmit these parameter sets via the most reliable channel.

GStreamer caps ¶

Whenever using H.264 as the video codec in Kurento, we’ll see log messages such as this one:

caps: video/x-h264, stream-format=(string)avc, alignment=(string)au,
codec_data=(buffer)0142c01fffe1000e6742c01f8c8d40501e900f08846a01000468ce3c80,
level=(string)3.1, profile=(string)constrained-baseline, width=(int)640,
height=(int)480, framerate=(fraction)0/1, interlace-mode=(string)progressive,
chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8,
parsed=(boolean)true

This describes in detail all aspects of the encoded video; some of them are universal properties of any video (such as the width, height and framerate), while others are highly specific to the H.264 encoding.

stream-format: Indicates if the H.264 video is stream-oriented (stream-format = byte-stream) or packet-oriented (stream-format = avc). For byte-stream videos the required parameter sets will be sent in-band with the video, but for avc the video metadata is conveyed via an additional caps field named codec_data.
codec_data: Only present when the video is packet oriented (stream-format = avc), this value represents an AVCDecoderConfigurationRecord struct.
Other information such as level, profile, width, height, framerate, interlace-mode, and the various chroma and luma settings, are just duplicated values that were extracted from the codec_data by an H.264 parser (namely the h264parse GStreamer element). This is also indicated by means of setting the field parsed=true.

GStreamer `codec_data`¶

GStreamer passes a codec_data field in its caps when the H.264 video is using the avc stream format. This field is printed in debug logs as a long hexadecimal sequence, but in reality it is an instance of an AVCDecoderConfigurationRecord, defined in the standard ISO/IEC 14496-15 (aka. MPEG-4) as follows:

aligned(8) class AVCDecoderConfigurationRecord {
    unsigned int(8) configurationVersion = 1;
    unsigned int(8) AVCProfileIndication;
    unsigned int(8) profile_compatibility;
    unsigned int(8) AVCLevelIndication;
    bit(6) reserved = ‘111111’b;
    unsigned int(2) lengthSizeMinusOne;
    bit(3) reserved = ‘111’b;
    unsigned int(5) numOfSequenceParameterSets;
    for (i=0; i< numOfSequenceParameterSets; i++) {
        unsigned int(16) sequenceParameterSetLength ;
        bit(8*sequenceParameterSetLength) sequenceParameterSetNALUnit;
    }
    unsigned int(8) numOfPictureParameterSets;
    for (i=0; i< numOfPictureParameterSets; i++) {
        unsigned int(16) pictureParameterSetLength;
        bit(8*pictureParameterSetLength) pictureParameterSetNALUnit;
    }
    if( profile_idc  ==  100  ||  profile_idc  ==  110  ||
        profile_idc  ==  122  ||  profile_idc  ==  144 )
    {
        bit(6) reserved = ‘111111’b;
        unsigned int(2) chroma_format;
        bit(5) reserved = ‘11111’b;
        unsigned int(3) bit_depth_luma_minus8;
        bit(5) reserved = ‘11111’b;
        unsigned int(3) bit_depth_chroma_minus8;
        unsigned int(8) numOfSequenceParameterSetExt;
        for (i=0; i< numOfSequenceParameterSetExt; i++) {
            unsigned int(16) sequenceParameterSetExtLength;
            bit(8*sequenceParameterSetExtLength) sequenceParameterSetExtNALUnit;
        }
    }
}

AVCProfileIndication: profile code as defined in ISO/IEC 14496-10. (profile_idc)
profile_compatibility: byte which occurs between the profile_idc and level_idc in a sequence parameter set (SPS), as defined in ISO/IEC 14496-10. (constraint_setx_flag)
AVCLevelIndication: level code as defined in ISO/IEC 14496-10. (level_idc)
lengthSizeMinusOne: length in bytes of the NALUnitLength field in an AVC video sample or AVC parameter set sample of the associated stream minus one. For example, a size of one byte is indicated with a value of 0. The value of this field shall be one of 0, 1, or 3 corresponding to a length encoded with 1, 2, or 4 bytes, respectively.
numOfSequenceParameterSets: number of SPSs that are used as the initial set of SPSs for decoding the AVC elementary stream.
sequenceParameterSetLength: length in bytes of the SPS NAL unit as defined in ISO/IEC 14496-10.
sequenceParameterSetNALUnit: a SPS NAL unit, as specified in ISO/IEC 14496-10. SPSs shall occur in order of ascending parameter set identifier with gaps being allowed.
numOfPictureParameterSets: number of picture parameter sets (PPSs) that are used as the initial set of PPSs for decoding the AVC elementary stream.
pictureParameterSetLength: length in bytes of the PPS NAL unit as defined in ISO/IEC 14496-10.
pictureParameterSetNALUnit: a PPS NAL unit, as specified in ISO/IEC 14496-10. PPSs shall occur in order of ascending parameter set identifier with gaps being allowed.
chroma_format: chroma_format indicator as defined by the chroma_format_idc parameter in ISO/IEC 14496-10.
bit_depth_luma_minus8: bit depth of the samples in the Luma arrays. For example, a bit depth of 8 is indicated with a value of zero (bit depth = 8 + bit_depth_luma_minus8). The value of this field shall be in the range of 0 to 4, inclusive.
bit_depth_chroma_minus8: bit depth of the samples in the Chroma arrays. For example, a bit depth of 8 is indicated with a value of zero (bit depth = 8 + bit_depth_chroma_minus8). The value of this field shall be in the range of 0 to 4, inclusive.
numOfSequenceParameterSetExt: number of Sequence Parameter Set Extensions that are used for decoding the AVC elementary stream.
sequenceParameterSetExtLength: length in bytes of the SPS Extension NAL unit as defined in ISO/IEC 14496-10.
sequenceParameterSetExtNALUnit: a SPS Extension NAL unit, as specified in ISO/IEC 14496-10.

Example mapping ¶

Let’s “translate” a sample codec_data into its components, to show the meaning of each field:

codec_data=(buffer)0142c01fffe1000e6742c01f8c8d40501e900f08846a01000468ce3c80

This would map to an AVCDecoderConfigurationRecord struct as follows:

0142c01fffe1000e6742c01f8c8d40501e900f08846a01000468ce3c80
01                                                         -> configurationVersion = 0x01 = 1
  42                                                       -> AVCProfileIndication = 0x42 = 66
    c0                                                     -> profile_compatibility = 0xC0
      1f                                                   -> AVCLevelIndication = 0x1F = 31
        ff                                                 -> lengthSizeMinusOne = 0b11 = 3
          e1                                               -> numOfSequenceParameterSets = 0b00001 = 1
            000e                                           -> sequenceParameterSetLength = 0x000E = 14
                6742c01f8c8d40501e900f08846a               -> 1x14 bytes sequenceParameterSetNALUnit
                                            01             -> numOfPictureParameterSets = 0x01 = 1
                                              0004         -> pictureParameterSetLength = 0x0004 = 4
                                                  68ce3c80 -> 1x4 bytes pictureParameterSetNALUnit

This is the mapping for the first bytes of the Sequence Parameter Set NAL Unit:

6742c01f8c8d40501e900f08846a
67                           -> Header = 0x67 = 0b0110_0111
                                forbidden_zero_bit = 0b0 = 0
                                nal_ref_idc = 0b11 = 3
                                nal_unit_type = 0b00111 = 7
  42                         -> profile_idc = 0x42 = 66
    c0                       -> constraint_setx_flag = 0xC0 = 0b1100_0000
                                constraint_set0_flag = 1
                                constraint_set1_flag = 1
      1f                     -> level_idc = 0x1F = 31
        ...                  -> Chroma, luma, scaling and more information

Note how the fields profile_idc, constraint_setx_flag, and level_idc get duplicated outside of this structure, in the codec_data’s AVCProfileIndication, profile_compatibility, and AVCLevelIndication, respectively.

In this example case, according to the definitions from ISO/IEC 14496-10 (Annex A.2.1), a profile_idc of 66 with constraint_set0_flag and constraint_set1_flag = 1 corresponds to the H.264 Constrained Baseline profile; and level_idc = 31 which means Level 3.1.

MP4 recording format¶

While AAC and H.264 are the most common audio and video encodings used in the industry, MP4 is the de-facto standard file format to store them. MP4 acts as a “container”, as its job is to contain one or more audio or video streams, together with any additional data such as subtitles or chapter markers.

When you configure a WebRTC stream over the network, the actual information being sent or received are the individual audio and video streams, possibly encoded as the mentioned AAC and H.264 codecs. However, if you use Kurento’s RecorderEndpoint, then you need to choose a container format in order to store these streams into an actual file. Here is when the most important decision must be done, as different container formats have different advantages and drawbacks.

MP4 metadata issues¶

An MP4 file breaks down into several units of data, called atoms. Some of these atoms will contain actual audio samples and video frames; some contain accompanying data such as captions, chapter index, title, poster, etc; and lastly some contain so-called metadata which is information about the video itself, required by the player or receiving decoder to do its job adequately.

In an MP4 container, every movie contains a video metadata atom or “moov atom”, describing the timescale, duration, and display characteristics of the movie. It acts as an index of the video data, and the file will not start to play until the player can access this index.

By default, MP4 stores the moov atom at the end of the file. This is fine for local playback, since the entire file is available for playback right away. However, writing the moov atom only at the end is a very bad idea for cases such as progressive download or streaming as it would force the download of the entire file first before it will start playback. Similarly, live recording in this format can be problematic if the recorder is abruptly stopped for whatever reason (think an unexpected crash), rendering an unreadable file.

The solution for these special cases is to move the moov atom to the beginning of the file. This ensures that the required movie information is downloaded (or recorded) first, enabling playback to start right away. The placement of the moov atom is specified in various software packages through settings such as “progressive download,” “fast start,” “streaming mode,” or similar options.

MP4 Fast-Start in Kurento¶

An easy solution to the interrupted recording problem is to move the moov atom metadata to the beginning of the file. This means that if the recording crashes, all required metadata will have already been written out to the file, so the file will still be readable (at least to some point). Using this technique is called MP4 Fast-Start.

MP4 Fast-Start has been tested for recording in Kurento, however the results were not very satisfying. In case of a crash, the files were effectively playable, however this would come at a great cost in terms of resulting file size. MP4 videos with Fast-Start mode enabled would grow in size much faster than those recorded without Fast-Start enabled.

Due to this observation, MP4 container format is just considered not the best choice for live recording with Kurento, with WEBM being the better option. Also the popular MKV (Matroska container format) has been made available to use for recording since Kurento Media Server release 6.10. This format is much more robust than MP4 and provides great results for live recording.

NAT Types and NAT Traversal ¶

Table of Contents

NAT Types and NAT Traversal

Sources:

Basic Concepts ¶

IP connection ¶

An IP connection is uniquely identified by a connection “quadruplet” consisting of two tuples:

(Source IP address, source port number).
(Destination IP address, destination port number).

Visualization:

(SRC_ADDR, SRC_PORT) -> (DST_ADDR, DST_PORT)

Note

-> denotes the direction of the connection.
(ADDR, PORT) denotes an (IP address, IP port) tuple.
(SRC_ADDR, SRC_PORT) is the IP tuple of the machine that makes the connection.
(DST_ADDR, DST_PORT) is the IP tuple of the machine that receives the connection.

Inbound NAT connection ¶

A NAT maintains inbound rules which are used to translate IP tuples between the NAT’s external and internal sides. Usually, the external side of the NAT is facing the public internet (WAN), while the internal side of the NAT is facing the local network (LAN). These inbound rules have the form of a hash table (or map); for each given key, there is a resulting value:

The key of the table is an IP quadruplet, formed by a NAT’s external IP tuple and a remote machine’s IP tuple.
The value of the table is a NAT’s internal IP tuple.

In other words, a NAT creates an equivalence between external combinations of IP addresses and ports, and internal IP addresses and ports.

Typically, these NAT rules are created automatically during an outbound connection from within the LAN to some remote machine: it’s at that moment when the NAT creates a new entry into its table (this is step 1 in the following visualizations). Later, this entry in the NAT table is used to decide which local machine needs to receive the response that the remote machine might send (this is step 2 in the visualizations). Rules created in this way are called “dynamic rules”; on the other hand, “static rules” can be explicitly created by an administrator, in order to set up a fixed NAT table for a given local machine.

Visualization:

    {NAT internal side}  |    {NAT external side}  |  {Remote machine}
                         |                         |
1. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, EXT_PORT) -> (REM_ADDR, REM_PORT) ]
2. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, EXT_PORT) <- (REM_ADDR, REM_PORT) ]

Note

-> and <- denote the direction of the connection in each step.
=> denotes the creation of a new rule (key) in the NAT table.
[ (ADDR, PORT), (ADDR, PORT) ] denotes the key (IP quadruplet) used to access the NAT table.
<= denotes the resolution of the NAT mapping.
(INT_ADDR, INT_PORT) is the source IP tuple on the internal side of the NAT for a local machine making a connection during step 1, and it is the destination IP tuple for the same machine receiving the connection during step 2.
(EXT_ADDR, EXT_PORT) is the source IP tuple on the external side of the NAT from where a connection is originated during step 1, and it is the destination IP tuple for the connection being received during step 2.
(REM_ADDR, REM_PORT) is the destination IP tuple of some remote machine receiving a connection during step 1, and it is the source IP tuple of a remote machine that makes a connection during step 2.

Types of NAT ¶

There are two categories of NAT behavior, namely Cone and Symmetric NAT. The crucial difference between them is that the former will use the same port numbers for internal and external IP tuples, while the later will always use different numbers for each side of the NAT. This will be explained later in more detail.

Besides, there are 3 types of Cone NATs, with varying degrees of restrictions regarding the allowed sources of incoming connections. To connect with a local machine which is behind a Cone NAT, it’s first required that the local machine performs an outbound connection to a remote one. This way, a dynamic rule will be created for the destination IP tuple, allowing the remote machine to connect back. The only exception is the Full Cone NAT, where a static rule can be created beforehand by an administrator, thanks to the fact that this kind of NAT ignores what is the source IP tuple of the remote machine that is connecting.

Full Cone NAT ¶

This type of NAT allows inbound connections from any source IP address and any source port, as long as the destination tuple exists in a previously created rule.

Typically, these rules are statically created beforehand by an administrator. These are the kind of rules that are used to configure Port Forwarding (aka. “opening the ports”) in most consumer-grade routers. Of course, as it is the case for all NAT types, it is also possible to create dynamic rules by first performing an outbound connection.

Visualization:

    {NAT internal side}  |    {NAT external side}  |  {Remote machine}
                         |                         |
1. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, INT_PORT) -> (REM_ADDR, REM_PORT) ]
2. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, INT_PORT) <- (   *    ,    *    ) ]

Note

* means that any value could be used: a remote machine can connect from any IP address and port.
The source IP address (REM_ADDR) in step 2 can be different from the destination IP address that was used in step 1.
The source IP port (REM_PORT) in step 2 can be different from the destination IP port that was used in step 1.
The same port (INT_PORT) is used in the internal and the external sides of the NAT. This is the most common case for all Cone NATs, only being different for Symmetric NATs.

(Address-)Restricted Cone NAT ¶

This type of NAT allows inbound connections from a specific source IP address but allowing for any source port. Typically, an inbound rule of this type was previously created dynamically, when the local machine initiated an outbound connection to a remote one.

Visualization:

    {NAT internal side}  |    {NAT external side}  |  {Remote machine}
                         |                         |
1. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, INT_PORT) -> (REM_ADDR, REM_PORT) ]
2. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, INT_PORT) <- (REM_ADDR,    *    ) ]

Note

The source IP address (REM_ADDR) in step 2 must be the same as the destination IP address that was used in step 1.
The source IP port (REM_PORT) in step 2 can be different from the destination IP port that was used in step 1.
The same port (INT_PORT) is used in the internal and the external sides of the NAT.

Port-Restricted Cone NAT ¶

This is the most restrictive type of Cone NAT: it only allows inbound connections from a specific source IP address and a specific source port. Again, an inbound rule of this type was previously created dynamically, when the local machine initiated an outbound connection to a remote one.

Visualization:

    {NAT internal side}  |    {NAT external side}  |  {Remote machine}
                         |                         |
1. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, INT_PORT) -> (REM_ADDR, REM_PORT) ]
2. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, INT_PORT) <- (REM_ADDR, REM_PORT) ]

Note

The source IP address (REM_ADDR) in step 2 must be the same as the destination IP address that was used in step 1.
The source IP port (REM_PORT) in step 2 must be the same as the destination IP port that was used in step 1.
The same port (INT_PORT) is used in the internal and the external sides of the NAT.

Symmetric NAT ¶

This type of NAT behaves in the same way of a Port-Restricted Cone NAT, with an important difference: for each outbound connection to a different remote IP tuple (i.e. to a different remote machine), the NAT assigns a new random source port on the external side. This means that two consecutive connections from the same local port to two different remote machines will have two different external source ports, even if the internal source IP tuple is the same for both of them.

This is also the only case where the ICE connectivity protocol will find Peer Reflexive candidates which differ from the Server Reflexive ones, due to the differing ports between the connection to the STUN server and the direct connection between peers.

Visualization:

    {NAT internal side}  |    {NAT external side}  |  {Remote machine}
                         |                         |
1. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, EXT_PORT1) -> (REM_ADDR, REM_PORT1) ]
2. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, EXT_PORT1) <- (REM_ADDR, REM_PORT1) ]
...
3. (INT_ADDR, INT_PORT) => [ (EXT_ADDR, EXT_PORT2) -> (REM_ADDR, REM_PORT2) ]
4. (INT_ADDR, INT_PORT) <= [ (EXT_ADDR, EXT_PORT2) <- (REM_ADDR, REM_PORT2) ]

Note

When the outbound connection is done in step 1, EXT_PORT1 gets defined as a new random port number, assigned for the new remote IP tuple (REM_ADDR, REM_PORT1).
Later, another outbound connection is done in step 3, from the same local address and port to the same remote machine but at a different port. EXT_PORT2 is a new random port number, assigned for the new remote IP tuple (REM_ADDR, REM_PORT2).

Types of NAT in the Real World ¶

Quoting from Wikipedia:

This terminology has been the source of much confusion, as it has proven inadequate at describing real-life NAT behavior. Many NAT implementations combine these types, and it is, therefore, better to refer to specific individual NAT behaviors instead of using the Cone/Symmetric terminology. RFC 4787 attempts to alleviate this issue by introducing standardized terminology for observed behaviors. For the first bullet in each row of the above table, the RFC would characterize Full-Cone, Restricted-Cone, and Port-Restricted Cone NATs as having an Endpoint-Independent Mapping, whereas it would characterize a Symmetric NAT as having an Address- and Port-Dependent Mapping. For the second bullet in each row of the above table, RFC 4787 would also label Full-Cone NAT as having an Endpoint-Independent Filtering, Restricted-Cone NAT as having an Address-Dependent Filtering, Port-Restricted Cone NAT as having an Address and Port-Dependent Filtering, and Symmetric NAT as having either an Address-Dependent Filtering or Address and Port-Dependent Filtering. There are other classifications of NAT behavior mentioned, such as whether they preserve ports, when and how mappings are refreshed, whether external mappings can be used by internal hosts (i.e., its Hairpinning behavior), and the level of determinism NATs exhibit when applying all these rules.[2]

Especially, most NATs combine symmetric NAT for outgoing connections with static port mapping, where incoming packets addressed to the external address and port are redirected to a specific internal address and port. Some products can redirect packets to several internal hosts, e.g., to divide the load between a few servers. However, this introduces problems with more sophisticated communications that have many interconnected packets, and thus is rarely used.

NAT Traversal ¶

The NAT mechanism is implemented in a vast majority of home and corporate routers, and it completely prevents the possibility of running any kind of server software in a local machine that sits behind these kinds of devices. NAT Traversal, also known as Hole Punching, is the procedure of opening an inbound port in the NAT tables of these routers.

To connect with a local machine which is behind any type of NAT, it’s first required that the local machine performs an outbound connection to the remote one. This way, a dynamic rule will be created for the destination IP tuple, allowing the remote machine to connect back.

In order to tell one machine when it has to perform an outbound connection to another one, and the destination IP tuple it must use, the typical solution is to use a helper service such as STUN. This is usually managed by a third machine, a server sitting on a public internet address. It retrieves the external IP and port of each peer, and gives that information to the other peers that want to communicate.

STUN/TURN requirement:

Symmetric to Symmetric: TURN.
Symmetric to Port-Restricted Cone: TURN.
Symmetric to Address-Restricted Cone: STUN (but probably not reliable).
Symmetric to Full Cone: STUN.
Everything else: STUN.

Do-It-Yourself hole punching ¶

It is very easy to test the NAT capabilities in a local network. To do this, you need access to two machines:

One siting behind a NAT. We’ll call this the host A.
One directly connected to the internet, with no firewall. This is host B.

Set some helper variables: the public IP address of each host, and their listening ports:

A_IP="11.11.11.11"  # Public IP address of the NAT which hides the host A
A_PORT="1111"       # Listening port on the host A
B_IP="22.22.22.22"  # Public IP address of the host B
B_PORT="2222"       # Listening port of the host B

A starts listening for data. Leave this running in A:
```
nc -4nul "$A_PORT"
```
B tries to send data, but the NAT in front of A will discard the packets. Run in B:
```
echo "TEST" | nc -4nu -q 1 -p "$B_PORT" "$A_IP" "$A_PORT"
```

A performs a hole punch, forcing its NAT to create a new inbound rule. B awaits for the UDP packet, for verification purposes.

Run in B:

sudo tcpdump -n -i eth0 "src host $A_IP and udp dst port $B_PORT"

Run in A:

sudo hping3 --count 1 --udp --baseport "$A_PORT" --keep --destport "$B_PORT" "$B_IP"

B tries to send data again. Run in B:

echo "TEST" | nc -4nu -q 1 -p "$B_PORT" "$A_IP" "$A_PORT"

Note

The difference between a Cone NAT and a Symmetric NAT can be detected during step 3. If the tcpdump command on B shows a source port equal to $A_PORT, then the NAT is respecting the source port chosen by the application, which means that it is one of the Cone NAT types. However, if tcpdump shows that the source port is different from $A_PORT, then the NAT is changing the source port during outbound mapping, which means that it is a Symmetric NAT.
In the case of a Cone NAT, the data sent from B should arrive correctly at A after step 4.
In the case of a Symmetric NAT, the data sent from B won’t arrive at A after step 4, because $A_PORT is the wrong destination port. If you write the correct port (as discovered in step 3) instead of $A_PORT, then the data should arrive to A.

PySTUN ¶

PySTUN is a tool that uses STUN servers in order to try and detect what is the type of the NAT, when ran from a machine behind it.

Currently it has been best updated in one of its forks, so we suggest using that instead of the version from the original creator. To install and run:

git clone https://github.com/konradkonrad/pystun.git pystun-konrad
cd pystun-konrad/
git checkout research
mv README.md README.rst
sudo python setup.py install
pystun

RTP Streaming Commands ¶

Table of Contents

In this document you will find several examples of command-line programs that can be used to generate RTP and SRTP streams. These streams can then be used to feed any general (S)RTP receiver, although the intention here is to use them to connect an RtpEndpoint from a Kurento Media Server pipeline.

The tool used for all these programs is gst-launch, part of the GStreamer multimedia library.

These examples start from the simplest and then build on each other to end up with a full featured RTP generator. Of course, as more features are added, the command grows in complexity. A very good understanding of gst-launch and of GStreamer is recommended.

To run these examples, follow these initial steps:

Install required packages:

sudo apt-get update
sudo apt-get install --no-install-recommends \
    gstreamer1.0-{tools,libav} \
    gstreamer1.0-plugins-{base,good,bad,ugly}

[Optional] Enable debug logging:
```
export GST_DEBUG=3
```
Copy & paste these commands into a console.

Write a Kurento application that does this:

sdpAnswer = rtpEndpoint.processOffer(sdpOffer);
log.info("SDP Answer: {}", sdpAnswer);

Start pipeline (e.g. in the RTP Receiver tutorial, push “Start”)
From the logs: get the KMS port from the SDP Answer (in the RTP Receiver tutorial, this appears in the web page)
Set PEER_V in the gst-launch commands to the KMS port.

RTP sender examples ¶

Simplest RTP sender ¶

Features:

Video RTP sender

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
bash -c 'gst-launch-1.0 -e \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000" \
    ! udpsink host=$PEER_IP port=$PEER_V'

Example 2 ¶

Features:

Video RTP sender
Video RTCP receiver

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
SELF_V=5004 SELF_VSSRC=112233 \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! udpsink host=$PEER_IP port=$PEER_V \
  udpsrc port=$((SELF_V+1)) ! r.recv_rtcp_sink'

Example 3 ¶

Features:

Video RTP sender
Video RTCP receiver console dump

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
SELF_V=5004 SELF_VSSRC=112233 \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! udpsink host=$PEER_IP port=$PEER_V \
  udpsrc port=$((SELF_V+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink \
    t. ! queue ! fakesink dump=true async=false'

Example 4 ¶

Features:

Video RTP & RTCP sender
Video RTCP receiver console dump

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
SELF_V=5004 SELF_VSSRC=112233 \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! udpsink host=$PEER_IP port=$PEER_V \
  r.send_rtcp_src ! udpsink host=$PEER_IP port=$((PEER_V+1)) sync=false async=false \
  udpsrc port=$((SELF_V+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink \
    t. ! queue ! fakesink dump=true async=false'

Example 5 ¶

Features:

Video RTP & RTCP sender
Video RTCP receiver console dump
Symmetrical ports (for autodiscovery)

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
SELF_V=5004 SELF_VSSRC=112233 \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! udpsink host=$PEER_IP port=$PEER_V bind-port=$SELF_V \
  r.send_rtcp_src ! udpsink host=$PEER_IP port=$((PEER_V+1)) bind-port=$((SELF_V+1)) sync=false async=false \
  udpsrc port=$((SELF_V+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink \
    t. ! queue ! fakesink dump=true async=false'

Example 6 ¶

Features:

Audio RTP & RTCP sender
Video RTCP receiver console dump
Symmetrical ports (for autodiscovery)

PEER_A=9006 PEER_IP=127.0.0.1 \
SELF_A=5006 SELF_ASSRC=445566 \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  audiotestsrc volume=0.5 ! audioconvert ! opusenc \
    ! rtpopuspay ! "application/x-rtp,payload=(int)96,clock-rate=(int)48000,ssrc=(uint)$SELF_ASSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! udpsink host=$PEER_IP port=$PEER_A bind-port=$SELF_A \
  r.send_rtcp_src ! udpsink host=$PEER_IP port=$((PEER_A+1)) bind-port=$((SELF_A+1)) sync=false async=false \
  udpsrc port=$((SELF_A+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink \
    t. ! queue ! fakesink dump=true async=false'

Full-featured RTP sender ¶

Features:

Audio & Video RTP & RTCP sender
Audio & Video RTCP receiver
Video RTCP receiver console dump
Symmetrical ports (for autodiscovery)

PEER_A=9006 PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="video.mp4" \
SELF_A=5006 SELF_ASSRC=445566 \
SELF_V=5004 SELF_VSSRC=112233 \
bash -c 'gst-launch-1.0 -e \
  rtpbin name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  filesrc location="$SELF_PATH" ! decodebin name=d \
  d. ! queue ! audioconvert ! opusenc \
    ! rtpopuspay ! "application/x-rtp,payload=(int)96,clock-rate=(int)48000,ssrc=(uint)$SELF_ASSRC" \
    ! r.send_rtp_sink_0 \
  d. ! queue ! videoconvert ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink_1 \
  r.send_rtp_src_0 ! udpsink host=$PEER_IP port=$PEER_A bind-port=$SELF_A \
  r.send_rtcp_src_0 ! udpsink host=$PEER_IP port=$((PEER_A+1)) bind-port=$((SELF_A+1)) sync=false async=false \
  udpsrc port=$((SELF_A+1)) ! r.recv_rtcp_sink_0 \
  r.send_rtp_src_1 ! udpsink host=$PEER_IP port=$PEER_V bind-port=$SELF_V \
  r.send_rtcp_src_1 ! udpsink host=$PEER_IP port=$((PEER_V+1)) bind-port=$((SELF_V+1)) sync=false async=false \
  udpsrc port=$((SELF_V+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink_1 \
    t. ! queue ! fakesink dump=true async=false'

RTP receiver examples ¶

Example 1 ¶

Features:

Video RTP & RTCP receiver
RTCP sender

PEER_V=5004 PEER_IP=127.0.0.1 \
SELF_V=9004 \
CAPS_V="media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)103" \
bash -c 'gst-launch-1.0 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  udpsrc port=$SELF_V ! "application/x-rtp,$CAPS_V" ! r.recv_rtp_sink \
    r.recv_rtp_src ! rtph264depay ! decodebin ! autovideosink \
  udpsrc port=$((SELF_V+1)) ! r.recv_rtcp_sink \
  r.send_rtcp_src ! udpsink host=$PEER_IP port=$((PEER_V+1)) sync=false async=false'

Note

RtpSession is used to handle RTCP, and it needs explicit video caps.

Example 2 ¶

Features:

Audio & Video RTP & RTCP receiver
Video RTCP receiver console dump
Audio & Video RTCP sender
Symmetrical ports (for autodiscovery)

PEER_A=5006 PEER_ASSRC=445566 PEER_V=5004 PEER_VSSRC=112233 PEER_IP=127.0.0.1 \
SELF_A=9006 SELF_V=9004 \
CAPS_A="media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS,payload=(int)96" \
CAPS_V="media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)103" \
bash -c 'gst-launch-1.0 -e \
  rtpbin name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  udpsrc port=$SELF_A ! "application/x-rtp,$CAPS_A" ! r.recv_rtp_sink_0 \
    r.recv_rtp_src_0_${PEER_ASSRC}_96 ! rtpopusdepay ! decodebin ! autoaudiosink \
  udpsrc port=$((SELF_A+1)) ! r.recv_rtcp_sink_0 \
  r.send_rtcp_src_0 ! udpsink host=$PEER_IP port=$((PEER_A+1)) bind-port=$((SELF_A+1)) sync=false async=false \
  udpsrc port=$SELF_V ! "application/x-rtp,$CAPS_V" ! r.recv_rtp_sink_1 \
    r.recv_rtp_src_1_${PEER_VSSRC}_103 ! rtph264depay ! decodebin ! autovideosink \
  udpsrc port=$((SELF_V+1)) ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink_1 \
    t. ! queue ! fakesink dump=true async=false \
  r.send_rtcp_src_1 ! udpsink host=$PEER_IP port=$((PEER_V+1)) bind-port=$((SELF_V+1)) sync=false async=false'

SRTP examples ¶

For the SRTP examples, you need to install the Kurento’s fork of GStreamer:

sudo apt-get update
sudo apt-get install --no-install-recommends \
    gstreamer1.5-{tools,libav} \
    gstreamer1.5-plugins-{base,good,bad,ugly}

SRTP simple sender ¶

Features:

Video SRTP sender

PEER_V=9004 PEER_IP=127.0.0.1 \
SELF_PATH="$HOME/videos/bbb" \
SELF_VSSRC=112233 \
SELF_KEY="4142434445464748494A4B4C4D4E4F505152535455565758595A31323334" \
bash -c 'gst-launch-1.5 -e \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,ssrc=(uint)$SELF_VSSRC" \
    ! srtpenc key="$SELF_KEY" \
      rtp-cipher="aes-128-icm" rtp-auth="hmac-sha1-80" \
      rtcp-cipher="aes-128-icm" rtcp-auth="hmac-sha1-80" \
    ! udpsink host=$PEER_IP port=$PEER_V'

SRTP simple receiver ¶

Features:

Video SRTP receiver

PEER_VSSRC=112233 \
PEER_KEY="4142434445464748494A4B4C4D4E4F505152535455565758595A31323334" \
SELF_V=9004 \
SRTP_CAPS="payload=(int)103,ssrc=(uint)$PEER_VSSRC,roc=(uint)0, \
    srtp-key=(buffer)$PEER_KEY, \
    srtp-cipher=(string)aes-128-icm,srtp-auth=(string)hmac-sha1-80, \
    srtcp-cipher=(string)aes-128-icm,srtcp-auth=(string)hmac-sha1-80" \
bash -c 'gst-launch-1.5 -e \
  udpsrc port=$SELF_V ! "application/x-srtp,$SRTP_CAPS" ! srtpdec \
  ! rtph264depay ! decodebin ! autovideosink'

Note

No RtpSession is used to handle RTCP, so no need for explicit video caps.

SRTP complete sender ¶

Features:

Video SRTP & SRTCP sender
SRTCP receiver console dump

PEER_V=9004 PEER_VSSRC=332211 PEER_IP=127.0.0.1 \
PEER_KEY="343332315A595857565554535251504F4E4D4C4B4A494847464544434241" \
SELF_PATH="$HOME/videos/bbb" \
SELF_V=5004 SELF_VSSRC=112233 \
SELF_KEY="4142434445464748494A4B4C4D4E4F505152535455565758595A31323334" \
SRTP_CAPS="payload=(int)103,ssrc=(uint)$PEER_VSSRC,roc=(uint)0, \
    srtp-key=(buffer)$PEER_KEY, \
    srtp-cipher=(string)aes-128-icm,srtp-auth=(string)hmac-sha1-80, \
    srtcp-cipher=(string)aes-128-icm,srtcp-auth=(string)hmac-sha1-80" \
bash -c 'gst-launch-1.5 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \
  srtpenc name=e key="$SELF_KEY" \
    rtp-cipher="aes-128-icm" rtp-auth="hmac-sha1-80" \
    rtcp-cipher="aes-128-icm" rtcp-auth="hmac-sha1-80" \
  srtpdec name=d \
  filesrc location="$SELF_PATH" ! decodebin \
    ! x264enc tune=zerolatency \
    ! rtph264pay ! "application/x-rtp,payload=(int)103,ssrc=(uint)$SELF_VSSRC" \
    ! r.send_rtp_sink \
  r.send_rtp_src ! e.rtp_sink_0 \
    e.rtp_src_0 ! udpsink host=$PEER_IP port=$PEER_V \
  r.send_rtcp_src ! e.rtcp_sink_0 \
    e.rtcp_src_0 ! udpsink host=$PEER_IP port=$((PEER_V+1)) sync=false async=false \
  udpsrc port=$((SELF_V+1)) ! "application/x-srtcp,$SRTP_CAPS" ! d.rtcp_sink \
    d.rtcp_src ! tee name=t \
    t. ! queue ! r.recv_rtcp_sink \
    t. ! queue ! fakesink dump=true async=false'

SRTP complete receiver ¶

Features:

Video SRTP & SRTCP receiver
SRTCP sender

PEER_V=5004 PEER_VSSRC=112233 PEER_IP=127.0.0.1 \
PEER_KEY="4142434445464748494A4B4C4D4E4F505152535455565758595A31323334" \
SELF_V=9004 SELF_VSSRC=332211 \
SELF_KEY="343332315A595857565554535251504F4E4D4C4B4A494847464544434241" \
SRTP_CAPS="payload=(int)103,ssrc=(uint)$PEER_VSSRC,roc=(uint)0, \
    srtp-key=(buffer)$PEER_KEY, \
    srtp-cipher=(string)aes-128-icm,srtp-auth=(string)hmac-sha1-80, \
    srtcp-cipher=(string)aes-128-icm,srtcp-auth=(string)hmac-sha1-80" \
CAPS_V="media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)103" \
bash -c 'gst-launch-1.5 -e \
  rtpsession name=r sdes="application/x-rtp-source-sdes,cname=(string)\"recv\@example.com\"" \
  srtpenc name=e key="$SELF_KEY" \
    rtp-cipher="aes-128-icm" rtp-auth="hmac-sha1-80" \
    rtcp-cipher="aes-128-icm" rtcp-auth="hmac-sha1-80" \
  srtpdec name=d \
  udpsrc port=$SELF_V ! "application/x-srtp,$SRTP_CAPS" ! d.rtp_sink \
    d.rtp_src ! "application/x-rtp,$CAPS_V" ! r.recv_rtp_sink \
    r.recv_rtp_src ! rtph264depay ! decodebin ! autovideosink \
  udpsrc port=$((SELF_V+1)) ! "application/x-srtcp,$SRTP_CAPS" ! d.rtcp_sink \
    d.rtcp_src ! r.recv_rtcp_sink \
  fakesrc num-buffers=-1 sizetype=2 \
    ! "application/x-rtp,payload=(int)103,ssrc=(uint)$SELF_VSSRC" ! r.send_rtp_sink \
    r.send_rtp_src ! fakesink async=false \
  r.send_rtcp_src ! e.rtcp_sink_0 \
    e.rtcp_src_0 ! udpsink host=$PEER_IP port=$((PEER_V+1)) sync=false async=false'

Note

fakesrc is used to force rtpsession to use the desired SSRC.

Additional Notes ¶

These are some random and unstructured notes that don’t have the same level of detail as the previous section. They are here just as a way of taking note of alternative methods or useful bits of information, but don’t expect that any command from this section works at all.

About ‘sync=false’¶

https://gstreamer.freedesktop.org/documentation/design/latency.html

Pipeline initialization is done with 3 state changes: - NULL→READY: Underlying devices are probed to ensure they can be accessed. - READY→PAUSED: Preroll is done, which means that an initial frame is brought from the sources and set into the sinks of the pipeline. - PAUSED→PLAYING: Sources start generating frames, and sinks start receiving and processing them.

The “sync” property indicates whether the element is Live (sync=true) or Non-Live (sync=false). - Live elements are synchronized against the clock, and only process data according to the established rate. The timestamps of the incoming buffers will be used to schedule the exact render time of its contents. - Non-Live elements do not synchronize with any clock, and process data as fast as possible. The pipeline will ignore the timestamps of the video frames and it will play them as they arrive, ignoring all timing information. Note that setting “sync=false” is almost never a solution when timing-related problems occur.

The “async” property enables (async=true) or disables (async=false) the Preroll feature. - Live sources cannot produce an initial frame until they are set to PLAYING state, so Preroll cannot be done with them on PAUSE state. If Prerolling is enabled in a Live sink, it will be set on hold waiting for that initial frame to arrive, and only then they will be able to complete the Preroll and start playing. - Non-Live sources should be able to produce an initial frame before reaching the PLAYING state, allowing their downstream sinks to Preroll as soon as the PAUSED state is set.

For example, a video camera or an output window/screen would be Live elements; a local file would be a Non-Live element.

Since RTCP packets from the sender should be sent as soon as possible and do not participate in preroll, sync=false and async=false are configured on udpsink.

See: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-rtpbin.html

About the SRTP Master Key ¶

The SRTP Master Key is the concatenation of (key, salt). With AES_CM_128 + HMAC_SHA1_80, Master Key is 30 bytes: 16 bytes key + 14 bytes salt.

Key formats:

GStreamer (gst-launch): Hexadecimal.
Kurento (RtpEndpoint): Plain text.
SDP Offer/Answer: Base64.

Use this website to convert between formats: http://tomeko.net/online_tools/hex_to_base64.php

Encryption key used by the sender examples:

ASCII: ABCDEFGHIJKLMNOPQRSTUVWXYZ1234.
In Hex: 4142434445464748494A4B4C4D4E4F505152535455565758595A31323334.
In Base64: QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVoxMjM0.

Encryption key used by the receiver examples:

ASCII: 4321ZYXWVUTSRQPONMLKJIHGFEDCBA.
In Hex: 343332315A595857565554535251504F4E4D4C4B4A494847464544434241.
In Base64: NDMyMVpZWFdWVVRTUlFQT05NTEtKSUhHRkVEQ0JB.

Using FFmpeg ¶

It should be possible to use FFmpeg to send or receive RTP streams; just make sure that all stream details match between the SDP negotiation and the actual encoded stream. For example: reception ports, Payload Type, encoding settings, etc.

This command is a good starting point to send RTP:

ffmpeg -re -i "video.mp4" -c:v libx264 -tune zerolatency -payload_type 103 \
  -an -f rtp rtp://IP:PORT

Note that Payload Type is 103 in these and all other examples, because that’s the number used in the SDP Offer sent to the RtpEndpoint in Kurento. You could use any other number, just make sure that it gets used consistently in both SDP Offer and RTP sender program.

SDP Offer examples ¶

Some examples of the SDP Offer that should be sent to Kurento’s RtpEndpoint to configure it with needed parameters for the RTP sender examples shown in this page:

Audio & Video RTP & RTCP sender

A basic SDP message that describes a simple Audio + Video RTP stream.

v=0
o=- 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=audio 5006 RTP/AVP 96
a=rtpmap:96 opus/48000/2
a=sendonly
a=ssrc:445566 cname:user@example.com
m=video 5004 RTP/AVP 103
a=rtpmap:103 H264/90000
a=sendonly
a=ssrc:112233 cname:user@example.com

Some modifications that would be done for KMS:

Add support for REMB Congestion Control.
Add symmetrical ports (for Port Autodiscovery).

v=0
o=- 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=audio 5006 RTP/AVP 96
a=rtpmap:96 opus/48000/2
a=sendonly
a=direction:active
a=ssrc:445566 cname:user@example.com
m=video 5004 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=rtcp-fb:103 goog-remb
a=sendonly
a=direction:active
a=ssrc:112233 cname:user@example.com

Apple Safari¶

There are two main implementations of the Safari browser: the Desktop edition which can be found in Mac OS workstations and laptops, and the iOS edition which comes installed as part of the iOS Operating System in mobile devices such as iPhone or iPad.

Codec issues¶

Safari (both Desktop and iOS editions) included a half-baked implementation of the WebRTC standard, at the least with regards to the codecs compatibility. The WebRTC specs state that both VP8 and H.264 video codecs MUST be implemented in all WebRTC endpoints [*], but Apple only added VP8 support starting from Safari Release 68. Older versions of the browser won’t be able to decode VP8 video, so if the source video isn’t already in H.264 format, Kurento Media Server will need to transcode the input video so they can be received by Safari.

[*]

RFC 7742, Section 5. Mandatory-to-Implement Video Codec: | WebRTC Browsers MUST implement the VP8 video codec as described in | [RFC6386] and H.264 Constrained Baseline as described in [H264]. | | WebRTC Non-Browsers that support transmitting and/or receiving video | MUST implement the VP8 video codec as described in [RFC6386] and | H.264 Constrained Baseline as described in [H264].

In order to ensure compatibility with Safari browsers, also caring to not trigger on-the-fly transcoding between video codecs, it is important to make sure that Kurento has been configured with support for H.264, and it is also important to check that the SDP negotiations are actually choosing this as the preferred codec.

If you are targeting Safari version 68+, then this won’t pose any problem, as now both H.264 and VP8 can be used for WebRTC.

HTML policies for video playback¶

Until recently, this has been the recommended way of inserting a video element in any HTML document:

<video id="myVideo" autoplay></video>

All Kurento tutorials are written to follow this example. As a general rule, most browsers honor the autoplay attribute, Desktop Safari included; however, we have recently found that iOS Safari is an exception to this rule, because it implements a more restrictive set of rules that must be followed in order to allow playing video from a <video> HTML tag.

There are two things to consider in order to make an HTML document that is compatible with iOS Safari:

Call the play() method, instead of using the autoplay attribute of the <video> tag.
Add the playsinline attribute to the <video> tag.

Sources for this section:

<video autoplay>¶

The autoplay attribute is honored by all browsers, and it makes the <video> tag to automatically start playing as soon as the source stream is available. In other words: the method video.play() gets implicitly called as soon as a source video stream becomes available and is set with video.srcObject = stream:

<video id="myVideo" autoplay></video>

However, in iOS Safari (version >= 10), the autoplay attribute is only available for videos that have no sound, are muted, or have a disabled audio track. In any other case, the autoplay attribute will be ignored, and the video won’t start playing automatically when a new stream is set.

The currently recommended solution for this issue is to avoid using the autoplay attribute. Instead, manually call the play() method as a result of some user interaction. For example, when a user clicks a button. The safest way is to call the video.play() method from inside a button’s onclick event handler.

<video playsinline>¶

Most browsers assume that a video should be played from inside the specific area that the <video> element occupies. So, for example, a tag such as this one:

<video id="myVideo"></video>

will play the video in an area that is 480x360 pixels.

That is not the case for iOS Safari: all videos play full screen by default: whenever a video starts playing, the browser will maximize its area to fill all the available space in the screen. This can be avoided by adding the playsinline attribute to the <video> tag:

<video id="myVideo" playsinline></video>

With this, videos will play in iOS Safari as they do in any other browser, effectively as inline videos inside their corresponding area.

Glossary¶

This is a glossary of terms that often appear in discussion about multimedia transmissions. Some of the terms are specific to GStreamer or Kurento, and most of them are described and linked to their RFC, W3C or Wikipedia documents.

Agnostic media

One of the big problems of media is that the number of variants of video and audio codecs, formats and variants quickly creates high complexity in heterogeneous applications. So Kurento developed the concept of an automatic converter of media formats that enables development of agnostic elements. Whenever a media element’s source is connected to another media element’s sink, the Kurento framework verifies if media adaption and transcoding is necessary and, if needed, it transparently incorporates the appropriate transformations making possible the chaining of the two elements into the resulting Pipeline.

AVI

Audio Video Interleaved, known by its initials AVI, is a multimedia container format introduced by Microsoft in November 1992 as part of its Video for Windows technology. AVI files can contain both audio and video data in a file container that allows synchronous audio-with-video playback. AVI is a derivative of the Resource Interchange File Format (RIFF).

See also

Audio Video Interleave

Resource Interchange File Format

Bower

Bower is a package manager for the web. It offers a generic solution to the problem of front-end package management, while exposing the package dependency model via an API that can be consumed by a build stack.

Builder Pattern

The builder pattern is an object creation software design pattern whose intention is to find a solution to the telescoping constructor anti-pattern. The telescoping constructor anti-pattern occurs when the increase of object constructor parameter combination leads to an exponential list of constructors. Instead of using numerous constructors, the builder pattern uses another object, a builder, that receives each initialization parameter step by step and then returns the resulting constructed object at once.

See also

Builder pattern

CORS

Cross-origin resource sharing is a mechanism that allows JavaScript code on a web page to make XMLHttpRequests to different domains than the one the JavaScript originated from. It works by adding new HTTP headers that allow servers to serve resources to permitted origin domains. Browsers support these headers and enforce the restrictions they establish.

See also

Cross-origin resource sharing

enable-cors.org: Information on the relevance of CORS and how and when to enable it.

DOM

Document Object Model is a cross-platform and language-independent convention for representing and interacting with objects in HTML, XHTML and XML documents.

EOS

End Of Stream is an event that occurs when playback of some media source has finished. In Kurento, some elements will raise an EndOfStream event.

GStreamer

GStreamer is a pipeline-based multimedia framework written in the C programming language.

H.264

A Video Compression Format. The H.264 standard can be viewed as a “family of standards” composed of a number of profiles. Each specific decoder deals with at least one such profiles, but not necessarily all.

See also

H.264/MPEG-4 AVC

RFC 6184: RTP Payload Format for H.264 Video (This RFC obsoletes RFC 3984)

HTTP

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

See also

Hypertext Transfer Protocol

RFC 2616: Hypertext Transfer Protocol – HTTP/1.1

ICE

Interactive Connectivity Establishment (ICE) is a technique used to achieve NAT Traversal. ICE makes use of the STUN protocol and its extension, TURN. ICE can be used by any aplication that makes use of the SDP Offer/Answer model..

See also

Interactive Connectivity Establishment

RFC 5245: Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols

IMS

IP Multimedia Subsystem (IMS) is the 3GPP’s Mobile Architectural Framework for delivering IP Multimedia Services in 3G (and beyond) Mobile Networks.

See also

IP Multimedia Subsystem

3GPP

RFC 3574: Transition Scenarios for 3GPP Networks

jQuery

jQuery is a cross-platform JavaScript library designed to simplify the client-side scripting of HTML.

JSON

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is designed to be easy to understand and write for humans and easy to parse for machines.

JSON-RPC

JSON-RPC is a simple remote procedure call protocol encoded in JSON. JSON-RPC allows for notifications and for multiple calls to be sent to the server which may be answered out of order.

Kurento

Kurento is a platform for the development of multimedia-enabled applications. Kurento is the Esperanto term for the English word ‘stream’. We chose this name because we believe the Esperanto principles are inspiring for what the multimedia community needs: simplicity, openness and universality. Some components of Kurento are the Kurento Media Server, the Kurento API, the Kurento Protocol, and the Kurento Client.

Kurento API

An object oriented API to create media pipelines to control media. It can be seen as and interface to Kurento Media Server. It can be used from the Kurento Protocol or from Kurento Clients.

Kurento Client

A programming library (Java or JavaScript) used to control an instance of Kurento Media Server from an application. For example, with this library, any developer can create a web application that uses Kurento Media Server to receive audio and video from the user web browser, process it and send it back again over Internet. The Kurento Client libraries expose the Kurento API to application developers.

Kurento Protocol

Communication between KMS and clients by means of JSON-RPC messages. It is based on WebSocket that uses JSON-RPC v2.0 messages for making requests and sending responses.

KMS

Kurento Media Server

Kurento Media Server is the core element of Kurento since it responsible for media transmission, processing, loading and recording.

Maven

Maven is a build automation tool used primarily for Java projects.

Media Element

A Media Element is a module that encapsulates a specific media capability. For example RecorderEndpoint, PlayerEndpoint, etc.

Media Pipeline

A Media Pipeline is a chain of media elements, where the output stream generated by one element (source) is fed into one or more other elements input streams (sinks). Hence, the pipeline represents a “machine” capable of performing a sequence of operations over a stream.

Media Plane

In a traditional IP Multimedia Subsystem, the handling of media is conceptually splitted in two layers. The layer that handles the media itself -with functionalities such as media transport, encoding/decoding, and processing- is called Media Plane.

See also

IP Multimedia Subsystem

Signaling Plane

MP4

MPEG-4 Part 14 or MP4 is a digital multimedia format most commonly used to store video and audio, but can also be used to store other data such as subtitles and still images.

See also

MPEG-4 Part 14

Multimedia

Multimedia is concerned with the computer controlled integration of text, graphics, video, animation, audio, and any other media where information can be represented, stored, transmitted and processed digitally. There is a temporal relationship between many forms of media, for instance audio, video and animations. There 2 are forms of problems involved in

Sequencing within the media, i.e. playing frames in correct order or time frame.

Synchronization, i.e. inter-media scheduling. For example, keeping video and audio synchronized or displaying captions or subtitles in the required intervals.

See also

Multimedia

Multimedia container format

Container or wrapper formats are meta-file formats whose specification describes how different data elements and metadata coexist in a computer file. Simpler multimedia container formats can contain different types of audio formats, while more advanced container formats can support multiple audio and video streams, subtitles, chapter-information, and meta-data, along with the synchronization information needed to play back the various streams together. In most cases, the file header, most of the metadata and the synchro chunks are specified by the container format.

See also

Multimedia container format

NAT

Network Address Translation

Network address translation (NAT) is the technique of modifying network address information in Internet Protocol (IP) datagram packet headers while they are in transit across a traffic routing device for the purpose of remapping one IP address space into another.

See also

Network address translation

NAT-T

NAT Traversal

NAT traversal (sometimes abbreviated as NAT-T) is a general term for techniques that establish and maintain Internet protocol connections traversing network address translation (NAT) gateways, which break end-to-end connectivity. Intercepting and modifying traffic can only be performed transparently in the absence of secure encryption and authentication.

See also

NAT Types and NAT Traversal: Entry in our Knowledge Base.

Node.js

Node.js is a cross-platform runtime environment for server-side and networking applications. Node.js applications are written in JavaScript, and can be run within the Node.js runtime on OS X, Microsoft Windows and Linux with no changes.

npm

npm is the official package manager for Node.js.

OpenCV

OpenCV (Open Source Computer Vision Library) is a BSD-licensed open source computer vision and machine learning software library. OpenCV aims to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception.

Pad, Media

A Media Pad is is an element’s interface with the outside world. Data streams from the MediaSource pad to another element’s MediaSink pad.

See also

GStreamer Pad: Definition of the Pad structure in GStreamer

QR

QR code (Quick Response Code) is a type of two-dimensional barcode. that became popular in the mobile phone industry due to its fast readability and greater storage capacity compared to standard UPC barcodes.

See also

QR code

REMB

Receiver Estimated Maximum Bitrate (REMB) is a type of RTCP feedback message that a RTP receiver can use to inform the sender about what is the estimated reception bandwidth currently available for the stream itself. Upon reception of this message, the RTP sender will be able to adjust its own video bitrate to the conditions of the network. This message is a crucial part of the Google Congestion Control (GCC) algorithm, which provides any RTP session with the ability to adapt in cases of network congestion.

The GCC algorithm is one of several proposed algorithms that have been proposed by an IETF Working Group named RTP Media Congestion Avoidance Techniques (RMCAT).

draft-alvestrand-rmcat-remb: RTCP message for Receiver Estimated Maximum Bitrate
draft-ietf-rmcat-gcc: A Google Congestion Control Algorithm for Real-Time Communication

REST

Representational state transfer (REST) is an architectural style consisting of a coordinated set of constraints applied to components, connectors, and data elements, within a distributed hypermedia system. The term representational state transfer was introduced and defined in 2000 by Roy Fielding in his doctoral dissertation.

See also

Representational state transfer

RTCP

The RTP Control Protocol (RTCP) is a sister protocol of the RTP, that provides out-of-band statistics and control information for an RTP flow.

See also

RTP Control Protocol

RFC 3605: Real Time Control Protocol (RTCP) attribute in Session Description Protocol (SDP)

RTP

Real-time Transport Protocol (RTP) is a standard packet format designed for transmitting audio and video streams on IP networks. It is used in conjunction with the RTP Control Protocol. Transmissions using the RTP audio/video profile (RTP/AVP) typically use SDP to describe the technical parameters of the media streams.

See also

Real-time Transport Protocol

RTP audio video profile

RFC 3550: RTP: A Transport Protocol for Real-Time Applications

Same-origin policy

The “same-origin policy” is a web application security model. The policy permits scripts running on pages originating from the same domain to access each other’s DOM with no specific restrictions, but prevents access to DOM on different domains.

See also

Same-origin policy

SDP

Session Description Protocol

SDP Offer/Answer

The Session Description Protocol (SDP) is a text document that describes the parameters of a streaming media session. It is commonly used to describe the characteristics of RTP streams (and related protocols such as RTSP).

The SDP Offer/Answer model is a negotiation between two peers of a unicast stream, by which the sender and the receiver share the set of media streams and codecs they wish to use, along with the IP addresses and ports they would like to use to receive the media.

This is an example SDP Offer/Answer negotiation. First, there must be a peer that wishes to initiate the negotiation; we’ll call it the offerer. It composes the following SDP Offer and sends it to the other peer -which we’ll call the answerer-:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=Example sender
c=IN IP4 127.0.0.1
t=0 0
m=audio 5006 RTP/AVP 96
a=rtpmap:96 opus/48000/2
a=sendonly
m=video 5004 RTP/AVP 103
a=rtpmap:103 H264/90000
a=sendonly

Upon receiving that Offer, the answerer studies the parameters requested by the offerer, decides if they can be satisfied, and composes an appropriate SDP Answer that is sent back to the offerer:

v=0
o=- 3696336115 3696336115 IN IP4 192.168.56.1
s=Example receiver
c=IN IP4 192.168.56.1
t=0 0
m=audio 0 RTP/AVP 96
a=rtpmap:96 opus/48000/2
a=recvonly
m=video 31278 RTP/AVP 103
a=rtpmap:103 H264/90000
a=recvonly

The SDP Answer is the final step of the SDP Offer/Answer Model. With it, the answerer agrees to some of the parameter requested by the offerer, but not all.

In this example, audio 0 means that the answerer rejects the audio stream that the offerer intended to send; also, it accepts the video stream, and the a=recvonly acknowledges that the answerer will exclusively act as a receiver, and won’t send any stream back to the other peer.

See also

Session Description Protocol

Anatomy of a WebRTC SDP

RFC 4566: SDP: Session Description Protocol
RFC 4568: Session Description Protocol (SDP) Security Descriptions for Media Streams

Semantic Versioning

Semantic Versioning is a formal convention for specifying compatibility using a three-part version number: major version; minor version; and patch.

Signaling Plane

It is the layer of a media system in charge of the information exchanges concerning the establishment and control of the different media circuits and the management of the network, in contrast to the transfer of media, done by the Signaling Plane. Functions such as media negotiation, QoS parametrization, call establishment, user registration, user presence, etc. as managed in this plane.

See also

Media Plane

Sink, Media

A Media Sink is a MediaPad that outputs a Media Stream. Data streams from a MediaSource pad to another element’s MediaSink pad.

SIP

Session Initiation Protocol (SIP) is a signaling plane protocol widely used for controlling multimedia communication sessions such as voice and video calls over Internet Protocol (IP) networks. SIP works in conjunction with several other application layer protocols:

SDP for media identification and negotiation.
RTP, SRTP or WebRTC for the transmission of media streams.
A TLS layer may be used for secure transmission of SIP messages.

See also

Session Initiation Protocol

Source, Media

A Media Source is a Media Pad that generates a Media Stream.

SPA

Single-Page Application

A single-page application is a web application that fits on a single web page with the goal of providing a more fluid user experience akin to a desktop application.

Sphinx

Sphinx is a documentation generation system. Text is first written using reStructuredText markup language, which then is transformed by Sphinx into different formats such as PDF or HTML. This is the documentation tool of choice for the Kurento project.

Spring Boot

Spring Boot is Spring’s convention-over-configuration solution for creating stand-alone, production-grade Spring based applications that can you can “just run”. It embeds Tomcat or Jetty directly and so there is no need to deploy WAR files in order to run web applications.

SRTCP

SRTCP provides the same security-related features to RTCP, as the ones provided by SRTP to RTP. Encryption, message authentication and integrity, and replay protection are the features added by SRTCP to RTCP.

See also

SRTP

SRTP

Secure RTP is a profile of RTP (Real-time Transport Protocol), intended to provide encryption, message authentication and integrity, and replay protection to the RTP data in both unicast and multicast applications. Similarly to how RTP has a sister RTCP protocol, SRTP also has a sister protocol, called Secure RTCP (or SRTCP).

See also

Secure Real-time Transport Protocol

RFC 3711: The Secure Real-time Transport Protocol (SRTP)

SSL

Secure Socket Layer. See TLS.

STUN

STUN stands for Session Traversal Utilities for NAT. It is a standard protocol (IETF RFC 5389) used by NAT traversal algorithms to assist hosts in the discovery of their public network information. If the routers between peers use full cone, address-restricted, or port-restricted NAT, then a direct link can be discovered with STUN alone. If either one of the routers use symmetric NAT, then a link can be discovered with STUN packets only if the other router does not use symmetric or port-restricted NAT. In this later case, the only alternative left is to discover a relayed path through the use of TURN.

Trickle ICE

Extension to the ICE protocol that allows ICE agents to send and receive candidates incrementally rather than exchanging complete lists. With such incremental provisioning, ICE agents can begin connectivity checks while they are still gathering candidates and considerably shorten the time necessary for ICE processing to complete.

See also

draft-ietf-ice-trickle: Trickle ICE: Incremental Provisioning of Candidates for the Interactive Connectivity Establishment (ICE) Protocol

TLS

Transport Layer Security (TLS) and its predecessor Secure Socket Layer (SSL).

See also

Transport Layer Security

RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2

TURN

TURN stands for Traversal Using Relays around NAT. Like STUN, it is a network protocol (IETF RFC 5766) used to assist in the discovery of paths between peers on the Internet. It differs from STUN in that it uses a public intermediary relay to act as a proxy for packets between peers. It is used when no other option is available since it consumes server resources and has an increased latency. The only time when TURN is necessary is when one of the peers is behind a symmetric NAT and the other peer is behind either a symmetric NAT or a port-restricted NAT.

VP8

VP8 is a video compression format created by On2 Technologies as a successor to VP7. Its patents rights are owned by Google, who made an irrevocable patent promise on its patents for implementing it and released a specification under the Creative Commons Attribution 3.0 license.

See also

VP8

RFC 6386: VP8 Data Format and Decoding Guide

WebM

WebM is an open media file format designed for the web. WebM files consist of video streams compressed with the VP8 video codec and audio streams compressed with the Vorbis audio codec. The WebM file structure is based on the Matroska media container.

WebRTC

WebRTC is a set of protocols, mechanisms and APIs that provide browsers and mobile applications with Real-Time Communications (RTC) capabilities over peer-to-peer connections.

See also

WebRTC Working Draft

WebSocket

WebSocket specification (developed as part of the HTML5 initiative) defines a full-duplex single socket connection over which messages can be sent between client and server.