Java - WebRTC magic mirror
This web application extends the Hello World Tutorial, adding media processing to the basic WebRTC loopback.
Note
Web browsers require using HTTPS to enable WebRTC, so the web server must use SSL and a certificate file. For instructions, check Configure a Java server to use HTTPS.
For convenience, this tutorial already provides dummy self-signed certificates (which will cause a security warning in the browser).
For the impatient: running this example
First of all, you should install Kurento Media Server to run this demo. Please visit the installation guide for further information.
To launch the application, you need to clone the GitHub project where this demo is hosted, and then run the main class:
git clone https://github.com/Kurento/kurento.git
cd kurento/tutorials/java/magic-mirror/
git checkout 7.2.0
mvn -U clean spring-boot:run
The web application starts on port 8443 in the localhost by default. Therefore, open the URL https://localhost:8443/ in a WebRTC compliant browser (Chrome, Firefox).
Note
These instructions work only if Kurento Media Server is up and running in the same machine
as the tutorial. However, it is possible to connect to a remote KMS in other machine, simply adding
the flag kms.url to the JVM executing the demo. As we’ll be using maven, you should execute
the following command
mvn -U clean spring-boot:run \
-Dspring-boot.run.jvmArguments="-Dkms.url=ws://{KMS_HOST}:8888/kurento"
Understanding this example
This application uses computer vision and augmented reality techniques to add a funny hat on top of faces. The following picture shows a screenshot of the demo running in a web browser:
Kurento Magic Mirror Screenshot: WebRTC with filter in loopback
The interface of the application (an HTML web page) is composed by two HTML5 video tags: one for the video camera stream (the local client-side stream) and other for the mirror (the remote stream). The video camera stream is sent to Kurento Media Server, which processes and sends it back to the client as a remote stream. To implement this, we need to create a Media Pipeline composed by the following Media Element s:
WebRtcEndpoint: Provides full-duplex (bidirectional) WebRTC capabilities.
FaceOverlay filter: Computer vision filter that detects faces in the video stream and puts an image on top of them. In this demo the filter is configured to put a Super Mario hat).
WebRTC with filter in loopback Media Pipeline
This is a web application, and therefore it follows a client-server architecture. At the client-side, the logic is implemented in JavaScript. At the server-side, we use a Spring-Boot based application server consuming the Kurento Java Client API, to control Kurento Media Server capabilities. All in all, the high level architecture of this demo is three-tier. To communicate these entities, two WebSockets are used. First, a WebSocket is created between client and application server to implement a custom signaling protocol. Second, another WebSocket is used to perform the communication between the Kurento Java Client and the Kurento Media Server. This communication takes place using the Kurento Protocol. For further information on it, please see this page of the documentation.
To communicate the client with the Java EE application server we have designed a simple signaling protocol based on JSON messages over WebSocket ‘s. The normal sequence between client and server is as follows: i) Client starts the Magic Mirror. ii) Client stops the Magic Mirror.
If any exception happens, server sends an error message to the client. The detailed message sequence between client and application server is depicted in the following picture:
One to one video call signaling protocol
As you can see in the diagram, an SDP and ICE candidates needs to be exchanged between client and server to establish the WebRTC session between the Kurento client and server. Specifically, the SDP negotiation connects the WebRtcPeer at the browser with the WebRtcEndpoint at the server. The complete source code of this demo can be found in GitHub.
Application Server Side
This demo has been developed using Java in the server-side, based on the Spring Boot framework, which embeds a Tomcat web server within the generated maven artifact, and thus simplifies the development and deployment process.
Note
You can use whatever Java server side technology you prefer to build web applications with Kurento. For example, a pure Java EE application, SIP Servlets, Play, Vert.x, etc. Here we chose Spring Boot for convenience.
In the following figure you can see a class diagram of the server side code:
Server-side class diagram of the MagicMirror app
The main class of this demo is named MagicMirrorApp. As you can see, the KurentoClient is instantiated in this class as a Spring Bean. This bean is used to create Kurento Media Pipelines, which are used to add media capabilities to your applications. In this instantiation we see that we need to specify to the client library the location of the Kurento Media Server. In this example, we assume it is located at localhost, listening in port TCP 8888. If you reproduce this tutorial, you’ll need to insert the specific location of your Kurento Media Server instance there.
@EnableWebSocket
@SpringBootApplication
public class MagicMirrorApp implements WebSocketConfigurer {
final static String DEFAULT_KMS_WS_URI = "ws://localhost:8888/kurento";
@Bean
public MagicMirrorHandler handler() {
return new MagicMirrorHandler();
}
@Bean
public KurentoClient kurentoClient() {
return KurentoClient.create();
}
@Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(handler(), "/magicmirror");
}
public static void main(String[] args) throws Exception {
new SpringApplication(MagicMirrorApp.class).run(args);
}
}
This web application follows a Single Page Application architecture
(SPA), and uses a WebSocket to communicate client with
application server by means of requests and responses. Specifically, the main
app class implements the interface WebSocketConfigurer to register a
WebSocketHandler to process WebSocket requests in the path /magicmirror.
MagicMirrorHandler
class implements TextWebSocketHandler to handle text WebSocket requests.
The central piece of this class is the method handleTextMessage. This
method implements the actions for requests, returning responses through the
WebSocket. In other words, it implements the server part of the signaling
protocol depicted in the previous sequence diagram.
In the designed protocol there are three different kinds of incoming messages to
the Server : start, stop and onIceCandidates. These messages are
treated in the switch clause, taking the proper steps in each case.
public class MagicMirrorHandler extends TextWebSocketHandler {
private final Logger log = LoggerFactory.getLogger(MagicMirrorHandler.class);
private static final Gson gson = new GsonBuilder().create();
private final ConcurrentHashMap<String, UserSession> users = new ConcurrentHashMap<String, UserSession>();
@Autowired
private KurentoClient kurento;
@Override
public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);
log.debug("Incoming message: {}", jsonMessage);
switch (jsonMessage.get("id").getAsString()) {
case "start":
start(session, jsonMessage);
break;
case "stop": {
UserSession user = users.remove(session.getId());
if (user != null) {
user.release();
}
break;
}
case "onIceCandidate": {
JsonObject jsonCandidate = jsonMessage.get("candidate").getAsJsonObject();
UserSession user = users.get(session.getId());
if (user != null) {
IceCandidate candidate = new IceCandidate(jsonCandidate.get("candidate").getAsString(),
jsonCandidate.get("sdpMid").getAsString(), jsonCandidate.get("sdpMLineIndex").getAsInt());
user.addCandidate(candidate);
}
break;
}
default:
sendError(session, "Invalid message with id " + jsonMessage.get("id").getAsString());
break;
}
}
private void start(WebSocketSession session, JsonObject jsonMessage) {
...
}
private void sendError(WebSocketSession session, String message) {
...
}
}
In the following snippet, we can see the start method. It handles the ICE
candidates gathering, creates a Media Pipeline, creates the Media Elements
(WebRtcEndpoint and FaceOverlayFilter) and make the connections among
them. A startResponse message is sent back to the client with the SDP
answer.
private void start(final WebSocketSession session, JsonObject jsonMessage) {
try {
// User session
UserSession user = new UserSession();
MediaPipeline pipeline = kurento.createMediaPipeline();
user.setMediaPipeline(pipeline);
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
user.setWebRtcEndpoint(webRtcEndpoint);
users.put(session.getId(), user);
// ICE candidates
webRtcEndpoint.addIceCandidateFoundListener(new EventListener<IceCandidateFoundEvent>() {
@Override
public void onEvent(IceCandidateFoundEvent event) {
JsonObject response = new JsonObject();
response.addProperty("id", "iceCandidate");
response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
try {
synchronized (session) {
session.sendMessage(new TextMessage(response.toString()));
}
} catch (IOException e) {
log.debug(e.getMessage());
}
}
});
// Media logic
FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build();
String appServerUrl = System.getProperty("app.server.url", MagicMirrorApp.DEFAULT_APP_SERVER_URL);
faceOverlayFilter.setOverlayedImage(appServerUrl + "/img/mario-wings.png", -0.35F, -1.2F, 1.6F, 1.6F);
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
// SDP negotiation (offer and answer)
String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
String sdpAnswer = webRtcEndpoint.processOffer(sdpOffer);
JsonObject response = new JsonObject();
response.addProperty("id", "startResponse");
response.addProperty("sdpAnswer", sdpAnswer);
synchronized (session) {
session.sendMessage(new TextMessage(response.toString()));
}
webRtcEndpoint.gatherCandidates();
} catch (Throwable t) {
sendError(session, t.getMessage());
}
}
Note
Notice the hat URL is provided by the application server and consumed by the KMS. This logic is assuming
that the application server is hosted in local (localhost), and by the default the hat URL is
https://localhost:8443/img/mario-wings.png. If your application server is hosted in a different host, it
can be easily changed by means of the configuration parameter app.server.url, for example:
mvn -U clean spring-boot:run -Dapp.server.url=https://app_server_host:app_server_port
The sendError method is quite simple: it sends an error message to the
client when an exception is caught in the server-side.
private void sendError(WebSocketSession session, String message) {
try {
JsonObject response = new JsonObject();
response.addProperty("id", "error");
response.addProperty("message", message);
session.sendMessage(new TextMessage(response.toString()));
} catch (IOException e) {
log.error("Exception sending message", e);
}
}
Client-Side
Let’s move now to the client-side of the application. To call the previously
created WebSocket service in the server-side, we use the JavaScript class
WebSocket. We use a specific Kurento JavaScript library called
kurento-utils.js to simplify the WebRTC interaction with the server. This
library depends on adapter.js, which is a JavaScript WebRTC utility
maintained by Google that abstracts away browser differences. Finally
jquery.js is also needed in this application.
These libraries are linked in the
index.html
web page, and are used in the
index.js.
In the following snippet we can see the creation of the WebSocket (variable
ws) in the path /magicmirror. Then, the onmessage listener of the
WebSocket is used to implement the JSON signaling protocol in the client-side.
Notice that there are three incoming messages to client: startResponse,
error, and iceCandidate. Convenient actions are taken to implement each
step in the communication. For example, in functions start the function
WebRtcPeer.WebRtcPeerSendrecv of kurento-utils.js is used to start a
WebRTC communication.
var ws = new WebSocket('ws://' + location.host + '/magicmirror');
ws.onmessage = function(message) {
var parsedMessage = JSON.parse(message.data);
console.info('Received message: ' + message.data);
switch (parsedMessage.id) {
case 'startResponse':
startResponse(parsedMessage);
break;
case 'error':
if (state == I_AM_STARTING) {
setState(I_CAN_START);
}
onError("Error message from server: " + parsedMessage.message);
break;
case 'iceCandidate':
webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
if (error) {
console.error("Error adding candidate: " + error);
return;
}
});
break;
default:
if (state == I_AM_STARTING) {
setState(I_CAN_START);
}
onError('Unrecognized message', parsedMessage);
}
}
function start() {
console.log("Starting video call ...")
// Disable start button
setState(I_AM_STARTING);
showSpinner(videoInput, videoOutput);
console.log("Creating WebRtcPeer and generating local sdp offer ...");
var options = {
localVideo: videoInput,
remoteVideo: videoOutput,
onicecandidate: onIceCandidate
}
webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,
function (error) {
if (error) {
return console.error(error);
}
webRtcPeer.generateOffer(onOffer);
});
}
function onOffer(offerSdp) {
console.info('Invoking SDP offer callback function ' + location.host);
var message = {
id : 'start',
sdpOffer : offerSdp
}
sendMessage(message);
}
function onIceCandidate(candidate) {
console.log("Local candidate" + JSON.stringify(candidate));
var message = {
id: 'onIceCandidate',
candidate: candidate
};
sendMessage(message);
}
Dependencies
This Java Spring application is implemented using Maven. The relevant part of the pom.xml is where Kurento dependencies are declared. As the following snippet shows, we need two dependencies: the Kurento Client Java dependency (kurento-client) and the JavaScript Kurento utility library (kurento-utils) for the client-side. Other client libraries are managed with webjars:
<dependencies>
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-client</artifactId>
</dependency>
<dependency>
<groupId>org.kurento</groupId>
<artifactId>kurento-utils-js</artifactId>
</dependency>
<dependency>
<groupId>org.webjars</groupId>
<artifactId>webjars-locator</artifactId>
</dependency>
<dependency>
<groupId>org.webjars.bower</groupId>
<artifactId>bootstrap</artifactId>
</dependency>
<dependency>
<groupId>org.webjars.bower</groupId>
<artifactId>demo-console</artifactId>
</dependency>
<dependency>
<groupId>org.webjars.bower</groupId>
<artifactId>adapter.js</artifactId>
</dependency>
<dependency>
<groupId>org.webjars.bower</groupId>
<artifactId>jquery</artifactId>
</dependency>
<dependency>
<groupId>org.webjars.bower</groupId>
<artifactId>ekko-lightbox</artifactId>
</dependency>
</dependencies>
Note
You can find the latest version of Kurento Java Client at Maven Central.