Warning

Kurento is a low-level platform to create WebRTC applications from scratch. You will be responsible of managing STUN/TURN servers, networking, scalability, etc. If you are new to WebRTC, we recommend using OpenVidu instead.

OpenVidu is an easier to use, higher-level, Open Source platform based on Kurento.

Troubleshooting Issues

If you are facing an issue with Kurento Media Server, follow this basic check list:

For more information about how to request support, and how to submit bug reports and commercial enquiries, have a look at the Support page.

My Kurento Media Server doesn’t work, what should I do?

This document outlines several bits of knowledge that can prove very useful when studying a failure or error in KMS:

Media Server Crashes

We want Kurento to be as stable as possible! When you notice a server crash, it’s a good time to report a bug so we can know about the issue. But before that, you need to check a couple things:

  1. Make sure that you are running the latest version of Kurento Media Server: 7.0.0.

  2. Have debug symbols installed. Otherwise, your bug report won’t be useful.

Then, please provide us with information about the crash:

  • Kurento tries to write an execution stack trace in the file /var/log/kurento-media-server/errors.log. Open the errors.log file and look for a line similar to this one:

    2019-09-19T13:44:48+02:00 -- New execution
    

    Then, see if you can find the stack trace that matches with the time when the crash occurred. Attach that stack trace to your bug report.

  • If you installed Kurento with apt-get install, and Apport is installed, then Ubuntu generates a crash report that you will find in /var/crash/_usr_bin_kurento-media-server.<PID>.crash. This contains information that can be used to inspect KMS with a debugger, so it can be very useful to find the cause of the crash. Upload it somewhere, or attach it to your bug report.

    Note

    The .crash report file is already compressed, so you can go ahead and upload it to some file transfer service to share it with us.

    Note

    The .crash report file must be deleted afterwards. If an old crash report exists, new ones will not be generated. So if you are experiencing crashes, make sure that the crash report file is always deleted after having shared it with us, so future crashes will also generate new report files.

  • Otherwise, you can manually enable the generation of a core dump whenever KMS crashes. For this, edit the file /etc/default/kurento-media-server and uncomment the setting DAEMON_CORE_PATTERN, which by default will tell the Linux Kernel to generate core dumps in /tmp/.

    Note

    The core dump file is NOT compressed, so before uploading you should compress it, for a typically huge file size reduction, before uploading it to some file transfer service and sharing it with us.

  • As a last resort, if no crash report can be obtained by any means, you may need to run KMS with a debugger. To do so, please follow the instructions here Run and debug with GDB, to get a backtrace when the crash happens.

  • Finally, if a developer suspects that the crash might be due to a memory corruption error, we could ask you to run with a special build of Kurento that comes bundled with support for AddressSanitizer, a memory access error detector.

    To do this, you’ll need to run a Kurento Docker image with AddressSanitizer. If we ask for it, you would have to provide the Docker logs from running this image.

    For this reason (and also for better test repeatability), it’s a very good idea that you have your services planned in a way that it’s possible to run Kurento Media Server from Docker, at any time, regardless of what is your normal / usual method of deploying Kurento.

Corrupted Video

Problem

  • Video image seems fine, but playback suffers from a lot of stuttering (i.e. it is not smooth, constantly “jumps” around). See here: RTSP Video stuttering.

  • Video playback is smooth (no color issues, no macroblocks, no excessive stuttering), but the perceived quality of the details is very poor.

  • Video contains green or pink patches in some areas:

    Green patches covering part of the picture

    Green patches covering part of the picture.

  • Video contains huge blocks (aka. “macroblocks”) that are dragged around through the video:

    Macroblocks distorting the picture

    Macroblocks distorting the picture.

  • KMS logs contain lots of these messages (in bursts of several per second):

    WARN rtpsource [...] duplicate or reordered packet (seqnr 32462, expected 32464)
    
    WARN kmsutils [...] GAP of 3 ms at PTS=0:01:54.187106448 (packet loss?); will request a new keyframe
    
    WARN kmsutils [...] DISCONTINUITY at non-keyframe; will drop until keyframe
    

Reason

  • Network congestion or an otherwise weak network link is causing a high rate of packet loss and, in the case of WebRTC, an automatic degradation of video quality on the sender side. Most web browsers will automatically reduce their video output quality if they detect that the network is congested.

  • Too much data is sent to Kurento’s PlayerEndpoint, which is not able to process it all on time, causing it to drop parts of the video.

  • A badly configured H.264 encoder in the sender side, especially when using a PlayerEndpoint to consume the video stream of an IP camera.

Solution

  • For decoding errors (color issues, macroblocks) the most effective change you can do is to reduce the video resolution and/or quality (bitrate, framerate) at the sender. This will make the video smaller, helping it to travel through congested networks.

  • Getting a stronger network link on both sender and receiver sides will always help. For example, moving closer to the Wifi access points, using Ethernet cables when possible, or moving to a better data coverage area.

  • Configure your sender encoder with correct parameters. See the next section about video encoding.

  • When the network link is not an issue, remember to change the default maximum bitrate of 500 Kbps that Kurento uses to send WebRTC.

    See also:

About sender video encoding

The maximum bitrate for WebRTC video (used by web browsers such as Chrome) is 2 Mbps for perfect conditions, so you should probably avoid pushing more than that in your application.

Regarding the video encoder at the sender side, keep in mind that the most compatible H.264 setting is the Constrained Baseline Profile, Level 3.1.

Lastly, note that Chrome not only adapts its own video sending bitrate according to network conditions, but also the resolution of the video. If you see a much lower resolution than expected, you should check the sender WebRTC stats, to see if it isn’t because of Chrome deciding to do so.

See also:

WebRTC will detect the bandwidth available on the network, and will adapt the video bitrate on the fly (and, in some cases such as the Chrome web browser, the resolution will change too). This adaptation is influenced by some properties of the network, such as jitter, latency, and packet loss. If your WebRTC video plays back smoothly but with a very poor quality, this mostly means that the network link itself is poor.

See also:

About H.264 & VP8 color encoding

The H.264 and VP8 video codecs use a color encoding system called YCbCr (sometimes also written as YCrCb), which the decoder has to convert into the well known RGB (”Red-Green-Blue”) model that is used by computer screens.

When there is data loss, the decoder will assume that all missing values are 0 (zero). It just turns out that a YCbCr value of (0,0,0) is equivalent to the green color in RGB.

Kurento detects that packets have been lost in the network, and sends retransmission requests to the source of the WebRTC or RTP stream. However, if packet loses are too high due to a weak or congested network, enough losses will build up until the video decoding gets negatively affected.

Cisco has also a nice paragraph covering this in their Knowledge Base: Pink and green patches in a video stream (archive):

Why do I see pink or green patches in my video stream […]?

Pink and green patches or lines seen in decoded video are often the result of packet loss or incorrect data in the video stream. Many video codecs (including H.261, H.263 and H.264) use the Y’CbCr system to represent color space, where Y’ is the ‘luma’ (brightness) component and Cb and Cr are the blue and red chroma components respectively. For many Y’CbCr values there is no equivalent RGB value and the colour seen on the display depends on the details of the algorithm used. A Y’CbCr value of (0,0,0) is often converted into the green color while a Y’CbCr value of (255,255,255) leads to a pink color.

If you encounter the symptoms described above, follow normal packet loss and network troubleshooting procedures.

Other Media Server issues

Reached limit / Resource temporarily unavailable

If the server is malfunctioning or crashing and you can see a message similar to this one in the logs:

Reached KMS files limit: 819 (system max: 1024)

or

Error creating thread: Resource temporarily unavailable

then KMS is hitting resource limits imposed by the Kernel. The 2 most common reasons for this issue are:

  1. You might have a custom service or Kurento plugin that is acquiring resources without releasing them afterwards. You should profile and debug your code to make sure that it is not leaking resources (such as open file descriptors, threads, sockets, etc) and exhausting the limits enforced by the Linux Kernel.

  2. Congratulations! Your service is growing, time to deal with resource & concurrency issues.

    Check the running stats of your operating system, to see if the KMS process is being limited to the default 1024 file/device handles (ulimit), and increase that number.

    For local installations (with apt-get install), you can edit the file /etc/default/kurento-media-server to raise either of the DAEMON_MAX_FILES and/or DAEMON_MAX_THREADS and see if this solves the issue. For other installation methods such as Docker, you will need to use tool-specific mechanisms to change resource limits, like running with docker run --ulimit.

    If systemd is configured, it might also be applying its own limit on process resources; for example you can check how many threads are being used by Kurento and what is the maximum, with these commands:

    systemctl status kurento-media-server | grep Tasks
    systemctl show kurento-media-server | grep TasksMax
    

    In systemd it is possible to configure limits with parameters such as DefaultTasksMax in either /etc/systemd/system.conf (for the default system instance) or /etc/systemd/user.conf (for the custom user instance, if you have configured one in your machine). The current effective value of DefaultTasksMax can be queried with this command:

    systemctl show --property DefaultTasksMax
    

    If left to its default setting, DefaultTasksMax will be 15% of the absolute maximum limit from the Kernel, that you can get or set through the file /proc/sys/kernel/pid_max. If you change that, don’t forget to run systemctl daemon-reexec to have systemd load the new value.

    Note

    You need to use systemctl daemon-reexec because systemctl daemon-reload has a bug in systemd <= v242 (so this affects Ubuntu versions earlier than 20.04 “Focal”).

GStreamer-CRITICAL messages in the log

GLib and GStreamer use a lot of internal assert() calls, to catch bugs in their own source code. When an assertion fails, a warning message is printed to the logs and the program continues running. Not crashing is good, of course, but Kurento hitting bugs of an underlying library will cause problems, sooner or later.

So, it’s in our best interest to watch out for these warnings. Report them to us if you see any in your logs! ;-)

Here are a couple examples of such messages:

(kurento-media-server:4619): GStreamer-CRITICAL **:
    gst_element_query: assertion 'GST_IS_ELEMENT (element)' failed
(kurento-media-server:15636): GLib-CRITICAL **:
    g_error_free: assertion 'error != NULL' failed

These warnings don’t really provide much debug information. To find out more, we’ll need you to run KMS under a debug session. Please, follow the instructions here Run and debug with GDB, to get a backtrace from the “GStreamer-CRITICAL” error.

CPU usage grows too high

Kurento Media Pipelines can get pretty complex if your use case requires so, which would mean more processing power is required to run them; however, even for the simplest cases it’s possible that you find out unexpected spikes in CPU usage, which in extreme cases could end up crashing the server due to resource exhaustion in the machine.

Check these points in an attempt to find possible causes for the high CPU usage:

  • Kurento Media Server is known to work well with videos of up to 720p resolution (1280x720) at 30fps and around 2Mbps. Using values beyond those might work fine, but the Kurento team hasn’t done any factual analysis to prove it. With heavier data loads there is a chance that KMS will be unable to process all incoming data on time, and this will cause that buffers fill up and frames get dropped. Try reducing the resolution of your input videos if you see video stuttering.

  • Source and destination video codecs must be compatible. This has always been a source of performance problems in WebRTC communications.

    • For example, if some participants are using Firefox and talking in a room, they will probably negotiate VP8 codec with Kurento; then later someone enters with Safari, CPU usage explodes due to transcoding is now suddenly required, because Safari only supports H.264 (VP8 support was added only since Desktop Safari v68).

    • Another example is you have some VP8 streams running nicely but then stream recording is enabled with the MP4 recording profile, which uses H.264. Same story: video needs to be converted, and that uses a lot of CPU.

  • Also check if other processes are running in the same machine and using the CPU. For example, if Coturn is running and using a lot of resources because too many users end up connecting via Relay (TURN).

Of these, video transcoding is the main user of CPU cycles, because encoding video is a computationally expensive operation. As mentioned earlier, keep an eye on the TRANSCODING events sent from Kurento to your Application Server, or alternatively look for TRANSCODING ACTIVE messages in the media server logs.

If you see that transcoding is active at some point, you may get a bit more information about why, by enabling this line:

export GST_DEBUG="${GST_DEBUG:-2},Kurento*:5,agnosticbin*:5"

in your daemon settings file, /etc/default/kurento-media-server.

Then look for these messages in the media server log output:

  • Upstream provided caps: (caps)

  • Downstream wanted caps: (caps)

  • Find TreeBin with wanted caps: (caps)

Which will end up with either of these sets of messages:

  • If source codec is compatible with destination:

    • TreeBin found! Use it for (audio|video)

    • TRANSCODING INACTIVE for (audio|video)

  • If source codec is not compatible with destination:

    • TreeBin not found! Transcoding required for (audio|video)

    • TRANSCODING ACTIVE for (audio|video)

These messages can help understand what codec settings are being received by Kurento (”Upstream provided caps”) and what is being expected at the other side by the stream receiver (”Downstream wanted caps”).

Memory usage grows too high

Problem

Each new Session consumes some memory, but later the memory is not freed back to the system after the Kurento Session is closed.

Reason

The most common cause for increasingly growing memory usage is not a memory leak, but Memory Fragmentation.

Solution

Try using an alternative memory allocator to see if it solves the issue of memory fragmentation. Please have a look at Using Jemalloc.

If you still think there might be a memory leak in KMS, keep reading:

  • Neither top nor ps are the right tool for the job to establish whether Kurento Media Server has a memory leak; Valgrind is.

  • Tools like top or ps show memory usage as seen by the Operating System, not by the process of the media server. Even after freeing memory, there is no guarantee that the memory will get returned to the Operating System. Typically, it won’t! Memory allocator implementations do not return free’d memory : it is marked as available for use by the same program, but not by others. So top or ps won’t be able to “see” the memory after KMS frees it.

    See: free() in C doesn’t reduce memory usage.

To run Kurento Media Server with Valgrind and find memory leaks, the process is just a matter of following the steps outlined in Build from sources, but with an extra argument:

bin/build-run.sh --valgrind-memcheck

Also, please have a look at the information shown in Media Server Crashes about our special Docker image based on AddressSanitizer. Running KMS with this image might help finding memory-related issues.

Service init doesn’t work

The package kurento-media-server provides a service file that integrates with the Ubuntu init system. This service file loads its user configuration from /etc/default/kurento-media-server, where the user is able to configure several features as needed.

In Ubuntu, log messages from init scripts are managed by systemd, and can be checked in to ways:

  • /var/log/syslog contains a copy of all init service messages. You can open it to see past messages, or follow it in real time with this command:

    tail -f /var/log/syslog
    
  • You can query the status of the kurento-media-server service with this command:

    systemctl status kurento-media-server.service
    

OpenH264 not found

Problem:

Installing and running KMS on a clean Ubuntu installation shows this message:

(gst-plugin-scanner:15): GStreamer-WARNING **: Failed to load plugin
'/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstopenh264.so': libopenh264.so.0:
cannot open shared object file: No such file or directory

Also these conditions apply:

  • Packages openh264-gst-plugins-bad-1.0 and openh264 are already installed.

  • The file /usr/lib/x86_64-linux-gnu/libopenh264.so is a broken link to the non-existing file /usr/lib/x86_64-linux-gnu/libopenh264.so.0.

Reason

The package openh264 didn’t install correctly. This package is just a wrapper that needs Internet connectivity during its installation stage, to download a binary blob file from this URL: http://ciscobinary.openh264.org/libopenh264-1.4.0-linux64.so.bz2

If the machine is disconnected during the actual installation of this package, the download will fail silently with some error messages printed on the standard output, but the installation will succeed.

Solution

Ensure that the machine has access to the required URL, and try reinstalling the package:

sudo apt-get update ; sudo apt-get install --reinstall openh264

Missing audio or video streams

If the Kurento Tutorials are showing an spinner, or your application is missing media streams, that’s a strong indication that the network topology requires using either a STUN server or a TURN relay, to traverse through the NAT of intermediate routers. Check the section about installing a STUN/TURN server.

If your application is expected to work with audio-only or video-only streams, make sure that Kurento Pipeline elements are not connected with the default connect(MediaElement) method (Java, JavaScript):

  • Use the connect(MediaElement, MediaType) method (Java, JavaScript).

  • Monitor the MediaFlowInStateChanged and MediaFlowOutStateChanged events from all MediaElements.

  • Make sure that the element providing media (the source) is firing a MediaFlowOut event, and that the receiver (the sink) is firing a corresponding MediaFlowIn event.

Application Server

These are some common errors found to affect Kurento Application Servers:

KMS is not running

Usually, the Kurento Client library is directed to connect with an instance of KMS that the developer expects will be running in some remote server. If there is no instance of KMS running at the provided URL, the Kurento Client library will raise an exception which the Application Server should catch and handle accordingly.

This is a sample of what the console output will look like, with the logging level set to DEBUG:

$ mvn -U clean spring-boot:run \
      -Dspring-boot.run.jvmArguments="-Dkms.url=ws://localhost:8888/kurento"
INFO  [...] Starting Application on TEST with PID 16448
DEBUG [...] Executing getKmsUrlLoad(b843d6f6-02dd-49b4-96b6-f2fd2e8b1c8d) in KmsUrlLoader
DEBUG [...] Obtaining kmsUrl=ws://localhost:8888/kurento from config file or system property
DEBUG [...] Connecting to kms in ws://localhost:8888/kurento
DEBUG [...] Creating JsonRPC NETTY Websocket client
DEBUG [...] Enabling heartbeat with an interval of 240000 ms
DEBUG [...] [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
WARN  [...] [KurentoClient]  Error sending heartbeat to server. Exception: [KurentoClient]  Exception connecting to WebSocket server ws://localhost:8888/kurento
WARN  [...] [KurentoClient]  Stopping heartbeat and closing client: failure during heartbeat mechanism
DEBUG [...] [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
DEBUG [...] Sending error to all pending requests
WARN  [...] [KurentoClient]  Trying to close a JsonRpcClientNettyWebSocket with channel == null
WARN  [...] Exception encountered during context initialization - cancelling refresh attempt: Factory method 'kurentoClient' threw exception; nested exception is org.kurento.commons.exception.KurentoException: Exception connecting to KMS
ERROR [...] Application startup failed

As opposed to that, the console output for when a connection is successfully done with an instance of KMS should look similar to this sample:

$ mvn -U clean spring-boot:run \
      -Dspring-boot.run.jvmArguments="-Dkms.url=ws://localhost:8888/kurento"
INFO  [...] Starting Application on TEST with PID 21617
DEBUG [...] Executing getKmsUrlLoad(af479feb-dc49-4a45-8b1c-eedf8325c482) in KmsUrlLoader
DEBUG [...] Obtaining kmsUrl=ws://localhost:8888/kurento from config file or system property
DEBUG [...] Connecting to kms in ws://localhost:8888/kurento
DEBUG [...] Creating JsonRPC NETTY Websocket client
DEBUG [...] Enabling heartbeat with an interval of 240000 ms
DEBUG [...] [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO  [...] [KurentoClient]  Connecting native client
INFO  [...] [KurentoClient]  Creating new NioEventLoopGroup
INFO  [...] [KurentoClient]  Initiating new Netty channel. Will create new handler too!
DEBUG [...] [KurentoClient]  channel active
DEBUG [...] [KurentoClient]  WebSocket Client connected!
INFO  [...] Started Application in 1.841 seconds (JVM running for 4.547)

KMS became unresponsive (due to network error or crash)

The Kurento Client library is programmed to start a retry-connect process whenever the other side of the RPC channel -ie. the KMS instance- becomes unresponsive. An error exception will raise, which again the Application Server should handle, and then the library will automatically start trying to reconnect with KMS.

This is how this process would look like. In this example, KMS was restarted so the Kurento Client library lost connectivity with KMS for a moment, but then it was able con reconnect and continue working normally:

INFO [...] Started Application in 1.841 seconds (JVM running for 4.547)

(... Application is running normally at this point)
(... Now, KMS becomes unresponsive)

INFO  [...] [KurentoClient]  channel closed
DEBUG [...] [KurentoClient]  JsonRpcWsClient disconnected from ws://localhost:8888/kurento because Channel closed.
DEBUG [...] Disabling heartbeat. Interrupt if running is false
DEBUG [...] [KurentoClient]  JsonRpcWsClient reconnecting to ws://localhost:8888/kurento.
DEBUG [...] [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO  [...] [KurentoClient]  Connecting native client
INFO  [...] [KurentoClient]  Closing previously existing channel when connecting native client
DEBUG [...] [KurentoClient]  Closing client
INFO  [...] [KurentoClient]  Initiating new Netty channel. Will create new handler too!
WARN  [...] [KurentoClient]  Trying to close a JsonRpcClientNettyWebSocket with channel == null
DEBUG [...] tryReconnectingForever = true
DEBUG [...] tryReconnectingMaxTime = 0
DEBUG [...] maxTimeReconnecting = 9223372036854775807
DEBUG [...] currentTime = 1510773733903
DEBUG [...] Stop connection retries: false
WARN  [...] [KurentoClient]  Exception trying to reconnect to server ws://localhost:8888/kurento. Retrying in 5000 ms

org.kurento.jsonrpc.JsonRpcException: [KurentoClient]  Exception connecting to WebSocket server ws://localhost:8888/kurento
   at (...)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8888
   at (...)

(... Now, KMS becomes responsive again)

DEBUG [...] [KurentoClient]  JsonRpcWsClient reconnecting to ws://localhost:8888/kurento.
DEBUG [...] [KurentoClient]  Connecting webSocket client to server ws://localhost:8888/kurento
INFO  [...] [KurentoClient]  Connecting native client
INFO  [...] [KurentoClient]  Creating new NioEventLoopGroup
INFO  [...] [KurentoClient]  Initiating new Netty channel. Will create new handler too!
DEBUG [...] [KurentoClient]  channel active
DEBUG [...] [KurentoClient]  WebSocket Client connected!
DEBUG [...] [KurentoClient]  Req-> {"id":2,"method":"connect","jsonrpc":"2.0"}
DEBUG [...] [KurentoClient]  <-Res {"id":2,"result":{"serverId":"1a3b4912-9f2e-45da-87d3-430fef44720f","sessionId":"f2fd16b7-07f6-44bd-960b-dd1eb84d9952"},"jsonrpc":"2.0"}
DEBUG [...] [KurentoClient]  Reconnected to the same session in server ws://localhost:8888/kurento

(... At this point, the Kurento Client is connected again to KMS)

Node.js / NPM failures

Kurento Client does not currently support Node.js v10 (LTS), you will have to use Node.js v8 or below.

Connection ends exactly after 60 seconds

This is typically caused by an intermediate proxy, which is prematurely ending the WebSocket session from the Application Server, and thus making the media server believe that all resources should be released.

For example, if Nginx Reverse Proxy is used, the default value of proxy_read_timeout is 60 seconds, but the default Kurento Ping/Pong keep-alive mechanism works in intervals of 240 seconds.

This issue can also manifest itself with this (misleading) error message in the browser’s JavaScript console:

WebRTC: ICE failed, add a TURN server and see about:webrtc for more details

The solution is to increase the timeout value in your proxy settings.

“Expects at least 4 fields”

This message can manifest in multiple variations of what is essentially the same error:

DOMException: Failed to parse SessionDescription: m=video 0 UDP/TLS/RTP/SAVPF Expects at least 4 fields

OperationError (DOM Exception 34): Expects at least 4 fields

The reason for this is that Kurento hasn’t enabled support for the video codec H.264, but it needs to communicate with another peer which only supports H.264, such as the Safari browser. Thus, the SDP Offer/Answer negotiation rejects usage of the corresponding media stream, which is what is meant by m=video 0.

The solution is to ensure that both peers are able to find a match in their supported codecs. To enable H.264 support in Kurento, check these points:

  • The package openh264-gst-plugins-bad-1.0 must be installed in the system.

  • The package openh264 must be correctly installed. Specifically, the post-install script of this package requires Internet connectivity, because it downloads a codec binary blob from the Cisco servers. See OpenH264 not found.

  • The H.264 codec must be enabled in the corresponding Kurento settings file: /etc/kurento/modules/kurento/SdpEndpoint.conf.json. Ensure that the entry corresponding to this codec does exist and is not commented out. For example:

    "videoCodecs": [
      { "name": "VP8/90000" },
      { "name": "H264/90000" }
    ]
    

“Error: ‘operationParams’ is required”

This issue is commonly caused by setting an invalid ID to any of the client method calls. The usual solution is to provide a null identifier, forcing the server to generate a new one for the object.

For example, a Node.js application wanting to use the ImageOverlayFilter API (Java, JavaScript) might mistakenly try to provide an invalid ID in the addImage() call:

const filter = await pipeline.create("ImageOverlayFilter");
await filter.addImage("IMAGE_ID", "https://IMAGE_URL", 0.5, 0.5, 0.5, 0.5, true, true);
await webRtcEndpoint.connect(filter);
await filter.connect(webRtcEndpoint);

This will fail, causing a MARSHALL_ERROR in the media server, and showing the following stack trace in the client side:

Trace: { Error: 'operationParams' is required
    at node_modules/kurento-client/lib/KurentoClient.js:373:24
    at Object.dispatchCallback [as callback] (node_modules/kurento-jsonrpc/lib/index.js:546:9)
    at processResponse (node_modules/kurento-jsonrpc/lib/index.js:667:15)
    [...]
    at WebsocketStream.onMessage (node_modules/websocket-stream/index.js:45:15) code: 40001, data: { type: 'MARSHALL_ERROR' } }

The solution is to simply use null for the first argument of the method:

await filter.addImage(null, "https://IMAGE_URL", 0.5, 0.5, 0.5, 0.5, true, true);

WebRTC failures

There is a multitude of possible reasons for a failed WebRTC connection, so you can start by following this checklist:

  • Deploy a STUN/TURN server (such as Coturn), to make remote WebRTC connections possible: How to install Coturn?.

  • Test if your STUN/TURN server is working correctly: How to test my STUN/TURN server?.

  • Configure your STUN/TURN server in Kurento Media Server: STUN/TURN Server.

  • Check the debug logs of your STUN/TURN server. Maybe the server is failing and some useful error messages are being printed in there.

  • Check the debug logs of Kurento Media Server. Look for messages that confirm a correct configuration:

    INFO [...] Using STUN reflexive server IP: <IpAddress>
    INFO [...] Using STUN reflexive server Port: <Port>
    
    INFO [...] Using TURN relay server: <user:password>@<IpAddress>:<Port>
    INFO [...] TURN server info set: <user:password>@<IpAddress>:<Port>
    
  • Check that any SDP mangling you (or any of your third-party libraries) might be doing in your Application Server is being done correctly.

    This is one of the most hard to catch examples we’ve seen in our mailing list:

    > The problem was that our Socket.IO client did not correctly URL-Encode its JSON payload when xhr-polling, which resulted in all “plus” signs (‘+’) being changed into spaces (’ ‘) on the server. This meant that the ufrag in the client’s SDP was invalid if it contained a plus sign! Only some of the connections failed because not all ufrag contain plus signs.

  • If WebRTC seems to disconnect exactly after some amount of time, every single time, watch out for proxy timeouts. Sometimes you have to extend the timeout for the site that is being hit with the problem. See also: Connection ends exactly after 60 seconds.

  • Have a look at these articles about troubleshooting WebRTC:

ICE connection problems

If your application receives an IceComponentStateChanged event with state FAILED from Kurento Media Server, it means that the WebRTC ICE connectivity has been abruptly interrupted. In general terms, this implies that there is some network connectivity issue between KMS and the remote peer (typically, a web browser), but the exact reason can fall into a myriad possible causes. You will need to investigate what happened on the user’s and the server’s network when the failure happened.

Here are some tips to keep in mind:

  • Check that you have correctly configured a STUN server or TURN relay, both in Kurento Media Server (file WebRtcEndpoint.conf.ini), and in the client browsers (through the RTCPeerConnection’s iceServers setting).

  • Check that the TURN credentials are correct, by using the Trickle ICE test page to test your STUN/TURN server, as explained here: How to test my STUN/TURN server?.

  • It is always a good idea to work out the correlation between ICE failures on KMS with ICE failures on the client browser. The combined logs of both sides might shed some light into what caused the disconnection.

  • Analyze all NewCandidatePairSelected events emitted by Kurento. A lot of ICE candidates are tested for connectivity during the WebRTC session establishment, but only the actual working ones are reported with the NewCandidatePairSelected event. A careful examination of all selected local and remote candidates might reveal useful information about the kind of connectivity issues that clients might be having.

    For example, maybe you see that most or all of the selected local or remote candidates are of typ relay, i.e. using a TURN relay as a proxy for the audio/video streams. This would mean two things:

    1. That the TURN relay will be under high server load, possibly saturating the machine’s resources.

    2. That direct peer-to-peer WebRTC connections are not being established, giving you a good starting point to investigate why this is happening. Usually, when you see usage of the TURN relay, this is caused by overzealous hardware or software firewalls, or the presence of Symmetric NAT modem/routers somewhere in the network path.

  • If you see messages about ICE connection tests failing due to timeout on trying pairs, make sure that all required UDP ports for media content are open on the sever; otherwise, not only the ICE process will fail, but also the video or audio streams themselves won’t be able to reach each WebRTC peer.

mDNS ICE candidate fails: Name or service not known

Problem

When the browser conceals the local IP address behind an mDNS candidate, these errors appear in Kurento logs:

kmsicecandidate  [...] Error code 0: 'Error resolving '2da1b2bb-a601-44e8-b672-dc70e3493bc4.local': Name or service not known'
kmsiceniceagent  [...] Cannot parse remote candidate: 'candidate:2382557538 1 udp 2113937151 2da1b2bb-a601-44e8-b672-dc70e3493bc4.local 50635 typ host generation 0 ufrag /Og/ network-cost 999'
kmswebrtcsession [...] Adding remote candidate to ICE Agent: Agent failed, stream_id: '1'

Solution

mDNS name resolution must be enabled in the system. Check out the contents of /etc/nsswitch.conf, you should see something similar to this:

hosts: files mdns4_minimal [NOTFOUND=return] dns

If not, try fully reinstalling the package libnss-mdns:

sudo apt-get purge libnss-mdns
sudo apt-get update ; sudo apt-get install libnss-mdns

Installing this package does automatically edit the config file in an appropriate way. Now the mdns4_minimal module should appear listed in the hosts line.

Caveat: mDNS does not work from within Docker

See mDNS and Crossbar.io Fabric (Docker) #21:

Docker does not play well with mDNS/zeroconf/Bonjour: resolving .local hostnames from inside containers does not work (easily). […] The reasons run deep into how Docker configures DNS inside a container.

So if you are running a Docker image, .local names won’t be correctly resolved even if you install the required packages. This happens with Kurento or whatever other software; it seems to be a Docker configuration problem / bug.

Disabling mDNS in Chrome

Chrome allows disabling mDNS, which is something that could be useful during development. However when development is finished, don’t forget to test your application with default settings, including with this option enabled!

To disable mDNS, open this URL: chrome://flags/#enable-webrtc-hide-local-ips-with-mdns and change the setting to “Disabled”.

Docker issues

Publishing Docker ports eats memory

Docker will consume a lot of memory when publishing big enough port ranges. As of this writing, there is no quick and easy solution to this issue.

You should not expose a large port range in your Docker containers; instead, prefer using Host Networking (--network host). To elaborate a bit more, as mentioned here:

the problem is that - given the current state of Docker - it seems you should NOT even be trying to expose large numbers of ports. You are advised to use the host network anyway, due to the overhead involved with large port ranges. (it adds both latency, as well as consumes significant resources - e.g. see https://www.percona.com/blog/2016/02/05/measuring-docker-cpu-network-overhead/)

If you are looking for a more official source, there is still (for years) an open issue in Docker about this: moby/moby#11185 (comment)

Multicast fails in Docker

Problem

  • Your Kurento Media Server is running in a Docker container.

  • MULTICAST streams playback fail with an error such as this one:

    DEBUG rtspsrc [...] timeout on UDP port
    

    Note that in this example, to see this message you would need to enable DEBUG log level for the rtspsrc category; see Logging levels and components.

Solution

For Multicast streaming to work properly, you need to disable Docker network isolation and use --network host. Note that this gives the container direct access to the host interfaces, and you’ll need to connect through published ports to access others containers.

This is a limitation of Docker; you can follow the current status with this issue: #23659 Cannot receive external multicast inside container.

If using Docker Compose, use network_mode: host such as this:

version: "3.7"
services:
  kms:
    image: kurento/kurento-media-server:7.0.0
    container_name: kms
    restart: always
    network_mode: host
    environment:
      - GST_DEBUG=2,Kurento*:5

References:

Element-specific info

PlayerEndpoint

RTSP broken audio

If you have your own RTSP tool generating OPUS encoded audio to be consumed in Kurento with a PlayerEndpoint (Java, JavaScript), and the resulting audio is very choppy and robotic, you should start by verifying that your encoding process is configured correctly for the OPUS frame size used in WebRTC.

This was the case for a user who later shared with us the reasons for the bad quality audio they were perceiving:

Bad audio quality

> There was a mismatch between the incoming raw audio frame size and the opus encoding frame size, which resulted in a bad encoding cadence causing irregular encoded frame intervals.

> We remedied this by ensuring that the incoming audio frame size and the opus encoding frame size are the same — or the incoming frame size is a divisor of the encoding frame size.

RTSP broken video

Some users have reported huge macro-blocks or straight out broken video frames when using a PlayerEndpoint to receive an RTSP stream containing H.264 video. A possible solution to fix this issue is to fine-tune the PlayerEndpoint’s networkCache parameter. It basically sets the buffer size (in milliseconds) that the underlying GStreamer decoding element will use to cache the stream.

There’s no science for that parameter, though. The perfect value depends on your network topology and efficiency, so you should proceed in a trial-and-error approach. For some situations, values lower than 100ms have worked fine; some users have reported that 10ms was required to make their specific camera work, others have seen good results with setting this parameter to 0ms. However, these are outlier cases and normally a higher networkCache is needed.

In principle, networkCache = 0 would mean that all RTP packets must be exactly on point at the expected times in the RTSP stream, or else they will be dropped. So even a slight amount of jitter or delay in the network might cause packets to be dropped when they arrive to the PlayerEndpoint.

networkCache translates directly to the latency property of GStreamer’s rtspsrc element, which in turn is passed to the rtpbin and ultimately the rtpjitterbuffer inside it.

RTSP Video stuttering

Problem

PlayerEndpoint is used to consume an RTSP stream from some source (typically, an IP camera). However, the resulting video (e.g. after recording with RecorderEndpoint, or after relaying video to WebRTC viewers with WebRtcEndpoint) shows stuttering (i.e. the video playback is not smooth, it constantly “jumps” around).

Reason

The source video is too heavy and KMS is not able to process it on time, so it lags behind and ends up losing parts of it.

Solution

The most effective change you can do is to reduce the video resolution and/or quality (bitrate, framerate) at the sender.

Kurento Media Server is known to work well receiving videos of up to 720p resolution (1280x720) at 30fps and bitrate around 2Mbps. If you are using values beyond those, there is a chance that KMS will be unable to process all incoming data on time, and this will cause buffers filling up and frames getting dropped. Try reducing the resolution of your input videos to see if this helps solving the issue.

See also:

Background

The GStreamer element in charge of RTSP reception is rtspsrc, and this element contains an rtpjitterbuffer.

This jitter buffer gets full when network packets arrive faster than what Kurento is able to process. If this happens, then PlayerEndpoint will start dropping packets, showing up as video stuttering on the output.

You can check if this problem is affecting you by running with DEBUG logging level enabled for the rtpjitterbuffer component, and searching for a specific message:

export GST_DEBUG="${GST_DEBUG:-2},rtpjitterbuffer:5"
/usr/bin/kurento-media-server 2>&1 | grep -P 'rtpjitterbuffer.*(Received packet|Queue full)'

With this command, a new line will get printed for each single Received packet, plus an extra line will appear informing about Queue full whenever a packet is dropped.

RecorderEndpoint

Zero-size video files

Remember that the client documentation contains lots of important information about usage of the RecorderEndpoint.

Follow this checklist to make sure none of these are the cause of your issue:

  • The RecorderEndpoint was created with a mediaProfile type that assumes both audio and video. If you intend to record audio-only or video-only media, select the appropriate _AUDIO_ONLY or _VIDEO_ONLY profile when creating the recorder instance. For example, to record a WebRTC screen capture (as obtained from a web browser’s call to MediaDevices.getDisplayMedia()), choose WEBM_VIDEO_ONLY instead of just WEBM.

  • The RecorderEndpoint was connected with the default connect(MediaElement) method (Java, JavaScript) (which assumes both audio and video), but the stream is audio-only or video-only.

    • Monitor the MediaFlowInStateChanged and MediaFlowOutStateChanged events from all MediaElements.

    • Make sure that the element providing media (the source) is firing a MediaFlowOut event, and that the RecorderEndpoint is firing a corresponding MediaFlowIn event.

    • If your recording should be only-audio or only-video, use the connect(MediaElement, MediaType) method (Java, JavaScript).

  • Check the availability of audio/video devices at recorder client initialization, and just before starting the recording.

  • User is disconnecting existing hardware, or maybe connecting new hardware (usb webcams, mic, etc).

  • User is clicking “Deny” when asked to allow access to microphone/camera by the browser.

  • User is sleeping/hibernating the computer, and then possibly waking it up, while recording.

  • Check the browser information about the required media tracks, e.g. track.readyState.

  • Track user agents, ICE candidates, etc.

Smaller or low quality video files

Kurento will just record whatever arrives as input, so if your recordings have less quality or lower resolution than expected, this is because the source video was already sent like that.

In most situations, the real cause of this issue is the web browser encoding and sending a low bitrate or a low resolution video. Keep in mind that some browsers (Chrome, as of this writing) are able to dynamically adjust the output resolution; this means that the real size of the video coming out from Chrome will vary over time. Normally it starts small, and after some time it improves, when the browser detects that the available network bandwidth allows for it.

Check this section to get some advice about how to investigate low quality issues: Corrupted Video.

Browser

Safari doesn’t work

Apple Safari is a browser that follows some policies that are much more restrictive than those of other common browsers such as Google Chrome or Mozilla Firefox.

For some tips about how to ensure the best compatibility with Safari, check Apple Safari.