Stream Video From Android Part 1 – Why & How

2020 update  Where I go over better methods on android and give some quick ffmpeg tips

 

 

I’m not sure how you got here, but if you need to stream or dissect video this might be your lucky day. Whenever I write a new article my biggest joy is choosing an image to represent the experience a user should expect when tackling this project. I couldn’t make up my mind and both are appropriate.

 

Over the last several weeks I have been falling down the rabbit hole of streaming from an android device to another computer. Despite the ubiquity of streaming apps right now there is not a lot of easy solutions  on how to stream from android and I found more unanswered questions than answers.

In this post I will tell you step by step how I streamed from android to a pc. But be warned this is for average to ninja developers. If its your first android, desktop of web app you might not make it through this explanation.

Secondly, I am no expert in video or streaming services. I’m posting this article as fresh as I can from learning in order to maintain my beginner point of view. This article is a great stepping stone for someone like me trying to understand the basics. The code works on my device. Its not production code, video artifacts remain and I have a lot of optimization and polishing that will be done. But I think its much easier to understand my rough code with all its notes etc that to see a bunch of polished methods

Posts

PART 1 – WHY & HOW

Part 2 – Getting Camera Data

Part 3- Understanding h264 mp4

PART 4 – PARSE BOXES AND SPS OR PPS

PART 5 – PARSE NALUS

PART 6 – PACKETIZE RTP

PART 7 – DEPACKETIZE AND DISPLAY

PART 8 – TIPS, TRICKS AND TESTS

Prologue-

There are several ways to come at streaming lets consider them all and I tell you why i did what i did.

— Using Someone Else’s Frame Work —

The first is relying on an outside streaming service. There are several services that will gladly provide a simple api and charge you to stream data from the phone to a server somewhere.

The second is a free library that will give you a framework. Webrtc is an example that can allow you to video chat and provides an api for establishing communications. The second is libstream which was one of my greatest learning tools in this project.

Third are headless api’s such as ffmpeg and gstreamer. Ffmpeg is a little tough to get started with but it provided some great data for me to examine.

If you need to video chat or stream entertainment content these are great solutions.

–Doing The dirty Work Yourself —

There are two pitfalls with the above methods. Either in the form of money or constraints based on the api. I wanted to really understand how the video  data was being routed. I wanted to route everything myself and control almost every detail.

My intention was to eventually use this android video and audio data for AI processing at a remote location. I didn’t want to let someone else’s API decide the bandwidth usage or whether FPS or image compression would be used to compensate for such issues of reduced connection.  I also wanted to control how when connection are established. Truthfully I had no idea how the above things would be handled in an api. But I didn’t want to find out after  I committed and decided I needed to learn a bit so the journey was worth it for me.

Inside android there are two methods for getting video data. MediaRecorder and MediaCodec. I chose MediaRecorder out of simplicity although I suspect MediaCodec has further advantages. If you follow this guide I suspect you can switch later when you are a bit smarter and have had some time to breath.

In this example I’m using android studio and sending the data to a javafx app on windows desktop

Below is the basic outline of the steps:

This is the very very broad strokes.

Camera2 ->Basic video class or recording with camera2 and a few tweaks

Data pipe -> This gets your data out of the camera2 class

Packetizer -> Bundle that data and send it somewhere

Depacketizer -> Get that video back out andinto a decoder

Why is it so hard?

Looking at the three steps above you may wonder, if you have already been trying a bit, why is it so tough? Think about it. The first successful streaming service was skype in 2003. Google duo only came out in 2016 and apple video chat a few years earlier.  Even more remarkable is that these video services share many of the similar software libraries and video streaming in any quality relies on the heavily patented h264 codec. So don’t stress out too much. Its tough and there’s a lot to know.

This is a multi part post so please keep moving forward.