Want a little bit more? You got it.

A bit of backstory on my experience.  I wrote these blog posts because it seemed there are reference manuals written by experts for experts with no bridge for the beginner to cross relating to these subjects. This article plus the resources I mentioned in the first post and in this page is everything I used. If you read this blog please leave a comment and say hi. I will get great satisfaction knowing it helped you out!

Even still, when your bit shifting and copying buffers you may make a small error. With 150k byte nalus coming in every 32ms you might get a little overwhelmed if your just starting.

You might start feeling like this

https://youtu.be/f9EKv-LM1nI

FFmeg

FFmpeg is what a lot of people are using. It comes wrapped up in javacv so I touched on it a bit already. But the documentation sucks so let me show you how to use it real fast.

  1. Download the build for your machine. I’m using windows so I download a compressed file and extract it to my program files folder. Then I must set my system path so I can use command line. Search internet on command line installs if you don’t understand as that’s a whole other topic in itself.

2. Navigate cmd line to folder with video and use this command to send a udp stream.

ffmpeg -re -i jjj.mp4 -vcodec libx264 -acodec copy -f h264 “udp://192.168.99.1:8550”

If you have a udp socket open you can see hex output or play the video. If you use a saved video from you android device this is a great way to make sure your receiving code is correct. For fun replace h264 with mpegts and write it to hex. Or change the libx264 to copy.

 

Writing to hex

Creating useful data from your test streams

 

[java]
public class FFStream {

private final String TAG = "FFStream";

File file;
DatagramSocket socket;
boolean shouldListen = false;
BufferedWriter writer;

public FFStream()
{
// get the file to send debug info to
FileChooser fileChooser = new FileChooser();

file = fileChooser.showOpenDialog(null);

if (file == null){
return;
}

createSocket();
try{
writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), StandardCharsets.UTF_8));
}catch (IOException ioe){
System.err.println(TAG + " cons " + ioe.toString());
}

}

private void createSocket()
{
System.out.println(TAG + " createsocket ");

try{
socket = new DatagramSocket(8550);
}catch (IOException ioe){
System.err.println(TAG + " createsocket " + ioe.toString());
}

shouldListen = true;

Executors.newSingleThreadExecutor().execute(new Runnable() {
@Override
public void run() {

byte[] inbuffer ;
String ip = "failed";
try{
ip = InetAddress.getLocalHost().getHostAddress();
}catch (UnknownHostException e){
System.err.println(TAG + " createsocket " + e.toString());
}

String port = String.valueOf(socket.getLocalPort());
System.out.println(TAG + " connect info " + ip + ":" + port);
while(shouldListen)

{
inbuffer = new byte[1500];

DatagramPacket packet = new DatagramPacket(inbuffer, inbuffer.length);

try {
System.out.println(TAG + " waiting on data");
socket.receive(packet); //blocking

byte[] data = new byte[packet.getLength()];

System.arraycopy(packet.getData(), packet.getOffset(),data,0,packet.getLength());

write(data);

} catch (IOException ioe) {
System.err.println(TAG + " createsocket " + ioe.toString());
}

}

}
});

}

private void write(byte[] incoming) throws IOException
{
System.out.println(TAG + " write " + String.valueOf(incoming.length) );

int count = 0;

for (byte b :
incoming) {
count++;
//writer.write("0x");
writer.write(String.format("%02X", b));

writer.write(" ");

if ((count % 16) == 0 ){
writer.newLine();

}

}

}

public void setShouldListen(boolean shouldListen) {
this.shouldListen = shouldListen;
}
}
[/java]

Debugging

I used these methods to great advantage to check what was being written in different methods

[java]
public static void debugHex(String call, byte[] arr, int length)
{

StringBuilder sb = new StringBuilder();
int count = 0;
for (byte b :
arr) {
sb.append(String.format("%02X", b));
sb.append(" ");
count++;
if (length == count){
break;
}
}

System.out.println(TAG + call + sb.toString());

}

public static void deBugHexTrailing(String call, byte[] arr, int length)
{
StringBuilder sb = new StringBuilder();
int count = 0;

for (int i = arr.length-1; i >= (arr.length – length) && i >= 0; i–) {

sb.append(String.format("%02X", arr[i]));
sb.append(" ");

}

System.out.println(TAG + call + sb.toString());
}

[/java]

I also used this to check on empty spaces in my nalus and caught a byte[] buffer that was padding data! Whoops!

 

[java]
public static void fillCompleteNalData(byte[] out, int entryPos, int exitPos)
{
int m = (exitPos – entryPos) /2;

StringBuilder sb = new StringBuilder();

sb
.append(" entry ").append(String.valueOf(entryPos))
.append(" exit ").append(String.valueOf(exitPos))
.append(" bstart ").append(String.format("%02X", out[entryPos]))
.append(" bmid ").append(String.format("%02X", out[entryPos + m]))
.append(" blast ").append(String.format("%02X", out[entryPos]));

System.err.println(sb.toString());

}
[/java]

Comparing reassembled nalus

When I was done I also watched the debugger for my android and my pc to compare the length of the nalu I parsed at the encoder to the one I give to my receiving decoder. After using the above test code though they were an exact match.

 

 

Join the Conversation

8 Comments

  1. Although I work with libav in C/C++, I enjoyed reading your article becuse it showed me that the mystery of H264 streaming and decoding is to be handled if you break it down into small pieces.

    Also nice to read that I’m not the only one having trouble with the documentation of ffmpeg/libav… 😉

    There is no git (or whatever) repository of your complete code for this project?!?

    Thank you very much for publishing your experience!

    1. Thanks for saying hi I’m so glad it helped! There is no git as as I have just posted the text files in each page for now. I have already refined this code much further and will continue to do so as the project develops. This is part of a library for making remote connections to a variety of robots. Maybe I can get that on to git when its done. Right now some obvious issues with the code published here are threading and consumer producer patterns which are missing and make it run a lot smoother. Also, this code grabs the video before its saved and it should be grabbed from hardware direct in order to speed up the whole process.

    2. I am in the same boat with you 🙂 ffmpeg/libav is a nightmare for me (I am a Java developer, which makes it 5 times harder)

    3. We are in the same boat. ffmpeg/libav is a nightmare (And Im not even a c++ dev, but java)

  2. Hey, I am trying to get the image from the IP camera (via RTSP) on an Android device to be as close to realtime as possible. Enjoyed reading your post, helped me with understanding of the internals for the rfc 6184 and H264.

    1. Glad it helped. Its far from inclusive because as you notice there are other layers of abstraction which require the deeper dive to get closer to hardware and reduce those delays you are noticing. But its a good start!

Leave a comment

Leave a Reply to Maik Cancel reply

Your email address will not be published. Required fields are marked *