302 lines
12 KiB
Markdown
302 lines
12 KiB
Markdown
# Yet another camera streamer
|
|
|
|
There's a number of great projects doing an UVC/CSI camera streaming
|
|
on SBC (like Raspberry PI's).
|
|
|
|
This is yet another camera streamer project that is primarly focused
|
|
on supporting a fully hardware accelerated streaming of MJPEG streams
|
|
and H264 video streams for minimal latency.
|
|
|
|
This supports well CSI cameras that provide 10-bit Bayer packed format
|
|
from sensor, by using a dedicated ISP of Raspberry PI's.
|
|
|
|
Take into account that this is a draft project, and is nowhere as complete
|
|
and well supported as (awesome ustreamer)[https://github.com/pikvm/ustreamer].
|
|
This project was inspired by mentioned ustreamer.
|
|
|
|
## Validate your system
|
|
|
|
**This streamer does only use hardware, and does not support any software decoding or encoding**.
|
|
It does require system to provide:
|
|
|
|
1. ISP (`/dev/video13`, `/dev/video14`, `/dev/video15`)
|
|
2. JPEG encoder (`/dev/video31`)
|
|
3. H264 encoder (`/dev/video11`)
|
|
4. JPEG/H264 decoder (for UVC cameras, `/dev/video10`)
|
|
5. At least LTS kernel (5.15) for Raspberry PIs
|
|
|
|
You can validate the presence of all those devices with:
|
|
|
|
```bash
|
|
uname -a
|
|
v4l2-ctl --list-devices
|
|
```
|
|
|
|
The `5.15 kernel` is easy to get since this is LTS kernel for Raspberry PI OS:
|
|
|
|
```bash
|
|
apt-get update
|
|
apt-get dist-upgrade
|
|
reboot
|
|
```
|
|
|
|
Ensure that your `/boot/config.txt` has enough of GPU memory (required for JPEG re-encoding):
|
|
|
|
```
|
|
# Example for IMX519
|
|
dtoverlay=vc4-kms-v3d,cma-128
|
|
gpu_mem=128 # preferred 160 or 256MB
|
|
dtoverlay=imx519
|
|
|
|
# Example for Arducam 64MP
|
|
gpu_mem=128
|
|
dtoverlay=arducam_64mp,media-controller=1
|
|
|
|
# Example for USB cam
|
|
gpu_mem=128
|
|
```
|
|
|
|
## Compile
|
|
|
|
```bash
|
|
git clone https://github.com/ayufan-research/camera-streamer.git --recursive
|
|
apt-get -y install libavformat-dev libavutil-dev libavcodec-dev libcamera-dev liblivemedia-dev v4l-utils pkg-config xxd build-essential cmake libssl-dev
|
|
|
|
cd camera-streamer/
|
|
make
|
|
sudo make install
|
|
```
|
|
|
|
## Use it
|
|
|
|
There are three modes of operation implemented offering different
|
|
compatibility to performance.
|
|
|
|
### Use a preconfigured systemd services
|
|
|
|
The simplest is to use preconfigured `service/camera-streamer*.service`.
|
|
Those can be used as an example, and can be configured to fine tune parameters.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
systemctl enable $PWD/service/camera-streamer-arducam-16MP.service
|
|
systemctl start camera-streamer-arducam-16MP
|
|
```
|
|
|
|
If everything was OK, there will be web-server at `http://<IP>:8080/`.
|
|
|
|
Error messages can be read `journalctl -xef -u camera-streamer-arducam-16MP`.
|
|
|
|
### High-compatibility via `libcamera`
|
|
|
|
This script uses `libcamera` to access camera and provide
|
|
manual and automatic brightness and exposure controls.
|
|
The settings for those are configurable via described below controls.
|
|
|
|
This due to extra overhead has worse latency than direct decoding
|
|
via ISP.
|
|
|
|
```bash
|
|
tools/libcamera_camera.sh -help
|
|
tools/libcamera_camera.sh -camera-format=YUYV
|
|
```
|
|
|
|
### High-performance mode via ISP for CSI
|
|
|
|
This script uses high-performance implementation for directly
|
|
accessing sensor feeds and passing it via DMA into bcm2385 ISP.
|
|
As such it does not implement brightness control similarly to `libcamera`.
|
|
|
|
```bash
|
|
# This script uses dumped IMX519 parametrs that are feed into bcm2385 ISP module
|
|
# This does not provide automatic brightness control
|
|
# Other sensors can be supported the same way as long as ISP parameters are adapted
|
|
tools/imx519_camera.sh -help
|
|
tools/imx519_camera.sh -camera-format=RG10 ...
|
|
```
|
|
|
|
This mode allows to provide significantly better performance for camera sensors
|
|
than described in specs. For example for Arducam 16MP (using IMX519) it is possible to achieve:
|
|
|
|
1. 120fps on 1280x720, with latency of about 50ms, where-as documentation says 1280x720x60p
|
|
1. 60fps on 1920x1080, with latency of about 90ms, where-as documentation says 1920x1080x30p
|
|
1. 30fps on 2328x1748, with latency of about 140ms, where-as documentation says 2328x1748x15p
|
|
|
|
The above was tested with camera connected to Raspberry PI Zero 2W, streaming `MJPEG` over WiFi
|
|
with camera recording https://www.youtube.com/watch?v=e8ZtPSIfWPc from desktop monitor. So, it as
|
|
well measured also a desktop rendering latency.
|
|
|
|
### High-performance mode via direct decoding for USB
|
|
|
|
This script uses direct decoding or passthrough of MJPEG or H264 streams from UVC camera into
|
|
re-encoded stream provided over web-interface.
|
|
|
|
```bash
|
|
tools/usb_camera.sh -help
|
|
tools/usb_camera.sh -camera-format=H264 ...
|
|
tools/usb_camera.sh -camera-format=MJPEG ...
|
|
```
|
|
|
|
## HTTP web server
|
|
|
|
All streams are exposed over very simple HTTP server, providing different streams for different purposes:
|
|
|
|
- `http://<ip>:8080/` - index page
|
|
- `http://<ip>:8080/snapshot` - provide JPEG snapshot (works well everywhere)
|
|
- `http://<ip>:8080/stream` - provide M-JPEG stream (works well everywhere)
|
|
- `http://<ip>:8080/video` - provide H264 in-browser muxed video output (using https://github.com/samirkumardas/jmuxer/, works only in Desktop Chrome)
|
|
- `http://<ip>:8080/video.mp4` or `http://<ip>:8080/video.mkv` - provide remuxed `mkv` or `mp4` stream (uses `ffmpeg` to remux, works as of now only in Desktop Chrome and Safari)
|
|
|
|
### Resolution
|
|
|
|
Camera capture and resolution exposed is controlled by threee parameters:
|
|
|
|
- `-camera-width` and `-camera-height` define capture resolution
|
|
- `-camera-video.height` - define height for an aspect ratio scaled resolution for `/video` and `/webrtc` (H264) output - this might require rescaller and might not always work
|
|
- `-camera-stream.height` - define height for an aspect ratio scaled resolution for `/stream` (MJPEG) output - this might require rescaller and might not always work
|
|
- `-camera-snapshot.height` - define height for an aspect ratio scaled resolution for `/snapshot` (JPEG) output - this might require rescaller and might not always work
|
|
|
|
Any `video`, `stream` and `snapshot` might not work as this requires usually decoding, scaling, and encoding to achieve the desired resolution.
|
|
|
|
This works ONLY BEST when using `libcamera`, the support for `USB` will varry and might require configuring `/boot/config.txt` to set enough of GPU memory to be able to re-encode JPEG.
|
|
|
|
## RTSP server
|
|
|
|
The camera-streamer implements RTSP server via `live555`. Enable it with:
|
|
|
|
- adding `-rtsp-port`: will enable RTSP server on 8554
|
|
- adding `-rtsp-port=1111`: will enable RTSP server on custom port
|
|
|
|
The camera-streamer will expose two stream (if low res mode is enabled):
|
|
|
|
- `rtsp://<ip>:8554/stream.h264` - high resolution stream (always enabled if H264 stream is available directly or via encoding)
|
|
- `rtsp://<ip>:8554/stream_low_res.h264` - low resolution stream if low res mode is configured via `-camera-low_res_factor`
|
|
|
|
## List all available controls
|
|
|
|
You can view all available configuration parameters by adding `-log-verbose`
|
|
to one of the above commands, like:
|
|
|
|
```bash
|
|
tools/libcamera_camera.sh -log-verbose ...
|
|
```
|
|
|
|
Depending on control they have to be used for camera, ISP, or JPEG or H264 codec:
|
|
|
|
```bash
|
|
# specify camera option
|
|
-camera-options=brightness=1000
|
|
|
|
# specify ISP option
|
|
-camera-isp.options=digital_gain=1000
|
|
|
|
# specify H264 option
|
|
-camera-h264.options=bitrate=10000000
|
|
|
|
# specify JPEG option
|
|
-camera-jpeg.options=compression_quality=60
|
|
```
|
|
|
|
## List all available formats and use proper one
|
|
|
|
You might list all available capture formats for your camera:
|
|
|
|
```bash
|
|
v4l2-ctl -d /dev/video0 --list-formats-ext
|
|
```
|
|
|
|
Some of them might be specified to streamer:
|
|
|
|
```bash
|
|
tools/*_camera.sh -camera-format=RG10 # Bayer 10 packet
|
|
tools/*_camera.sh -camera-format=YUYV
|
|
tools/*_camera.sh -camera-format=MJPEG
|
|
tools/*_camera.sh -camera-format=H264 # This is unstable due to h264 key frames support
|
|
```
|
|
|
|
## Camera support
|
|
|
|
### Arducam 16MP
|
|
|
|
The 16MP sensor is supported by default in Raspberry PI OS after adding to `/boot/config.txt`.
|
|
However it will not support auto-focus nor manual focus due to lack of `ak7535` compiled
|
|
and enabled in `imx519`. Focus can be manually controlled via `i2c-tools`:
|
|
|
|
```shell
|
|
# /boot/config.txt
|
|
dtoverlay=imx519,media-controller=0
|
|
gpu_mem=160 # at least 128
|
|
|
|
# /etc/modules-load.d/modules.conf
|
|
i2c-dev
|
|
|
|
# after starting camera execute to control the focus with `0xXX`, any value between `0x00` to `0xff`
|
|
# RPI02W (and possible 2+, 3+):
|
|
i2ctransfer -y 22 w4@0x0c 0x0 0x85 0x00 0x00
|
|
|
|
# RPI4:
|
|
i2ctransfer -y 11 w4@0x0c 0x0 0xXX 0x00 0x00
|
|
```
|
|
|
|
Latency according to my tests is due to the way how buffers are enqueued and processing delay, this is for triple buffering in a case where sensor is able to deliver frames quick enough. Depending on how you queue (on which slope) and when you enqueue buffer you might achieve significantly better latency as shown in the ISP-mode. The `libcamera` can still achieve 120fps, it is just slightly slower :)
|
|
|
|
#### 2328x1748@30fps
|
|
|
|
```shell
|
|
# libcamera
|
|
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=120 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=63/0, processing_ms=101.1, frame_ms=33.1
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=64/0, processing_ms=99.2, frame_ms=31.9
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=65/0, processing_ms=99.6, frame_ms=34.8
|
|
|
|
# direct ISP-mode
|
|
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=30 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=32/0, processing_ms=49.7, frame_ms=33.3
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=33/0, processing_ms=49.7, frame_ms=33.3
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=34/0, processing_ms=49.7, frame_ms=33.4
|
|
```
|
|
|
|
#### 2328x1748@10fps
|
|
|
|
```shell
|
|
# libcamera
|
|
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=10 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=585/0, processing_ms=155.3, frame_ms=100.0
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=586/0, processing_ms=155.5, frame_ms=100.2
|
|
|
|
# direct ISP-mode
|
|
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=10 -camera-width=2328 -camera-height=1748 -camera-high_res_factor=1.5 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=260/0, processing_ms=57.5, frame_ms=99.7
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=261/0, processing_ms=57.6, frame_ms=100.0
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=262/0, processing_ms=58.0, frame_ms=100.4
|
|
```
|
|
|
|
#### 1280x720@120fps for Arducam 16MP
|
|
|
|
```shell
|
|
# libcamera
|
|
$ ./camera_streamer -camera-path=/base/soc/i2c0mux/i2c@1/imx519@1a -camera-type=libcamera -camera-format=YUYV -camera-fps=120 -camera-width=1280 -camera-height=720 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=139/0, processing_ms=20.1, frame_ms=7.9
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=140/0, processing_ms=20.6, frame_ms=8.8
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=141/0, processing_ms=19.8, frame_ms=8.1
|
|
|
|
# direct ISP-mode
|
|
$ ./camera_streamer -camera-path=/dev/video0 -camera-format=RG10 -camera-fps=120 -camera-width=1280 -camera-height=720 -log-filter=buffer_lock
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf0 (refs=2), frame=157/0, processing_ms=18.5, frame_ms=8.4
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf1 (refs=2), frame=158/0, processing_ms=18.5, frame_ms=8.3
|
|
device/buffer_lock.c: http_jpeg: Captured buffer JPEG:capture:mplane:buf2 (refs=2), frame=159/0, processing_ms=18.5, frame_ms=8.3
|
|
```
|
|
|
|
## WebRTC support
|
|
|
|
The WebRTC is accessible via `http://<ip>:8080/video` by default and is available when there's H264 output generated.
|
|
|
|
WebRTC support is implemented using awesome [libdatachannel](https://github.com/paullouisageneau/libdatachannel/) library.
|
|
|
|
The support will be compiled by default when doing `make`.
|
|
|
|
## License
|
|
|
|
GNU General Public License v3.0
|