Video Streaming Update 2020 – Robotic Vision

Since my old video streaming tutorial get a lot of view I wanted to add some new info and an introduction. If you are building a robot and need to stream video this is for you!

 

Why android to desktop

-Android

The android device I use is an old old v21 phone. I can buy them for $5 a piece or just ask around for an old phone in someones junk drawer. I waned something cheap and available. Lots of sensors too. Compare that to a naked raspberry pi at $35 with no camera and much weaker software.

-Desktop

If you are using neural nets then you need some processing power. Sending video is a good option to avoid weighing down your bot.

Streaming from android

The older posts that you linked from used the basic camera api. DO NOT use the basic camera api.

The second thing I found is bigflake. Its useful but its complicated and I don’t think its that fast although it was difficult to benchmark.

Every app that uses an android is using the webrtc framework; google video, snapchat, signal etc. However webrtc has a very limiting api an is designed for video calls. However I found this library on github. It opens the api and allows you to extract video frame by frame. My remote computers start video command has a 740 ms turnaround before recieiving its first nal from the remote camera on my home network.

video rec

encoder

These two txt files are the classes that extract video from my android. I draw them on a texture but you dont have to if you want a blank screen. It takes 16ms to encode and 5 ms to packetize and send using the packetizer from the older tutorial.

 

FFMPEG

FFmpegframegrabber is used to grabframes and display them.

My first hang up was installing.  You need to install the latest release as library.

https://github.com/bytedeco/javacv/releases

Instantiation

final FrameGrabber grabber;

...


grabber = new FFmpegFrameGrabber(inputStream, 0);

Although I still cannot benchmark it properly I have got it working in “real time” . I had to download the snapshot,  change a variable by hand, build it and then paste that jar to replace the javacv.jar that came in the release files you downloaded above/

line 926  change it from 0 to 1 to eliminate the latency

// Enable multithreading when available
video_c.thread_count(1);

Heres my stack overflow question.

https://stackoverflow.com/questions/59185079/ffmpeg-javacv-latency-issue

The older section goes into detail and the packetization is still required. The other parts are worth a read too.

Leave a Reply

Your email address will not be published. Required fields are marked *