Getting to what you really read all this stuff for.
-> Get the transfer h264.java file referenced in this article <-
Why do we need to parse nalus? Because nalues are 100k bytes and we cant send files that size over the internet. But your clever, you’ll just want to send them over TCP and let the java socket class do the work . Not so fast!
When we are sending data we should consider 2 things, one packetizing that data efficiently and timing how fast those nalus are coming out so we can play them at the right speed on the other side. Remember, normally a video file has all those boxes and header that tell it when to play each frame. But when we are streaming we simply have nalus pouring out of the buffer.
Remember that our android myvideoclass does two things. Records a shot video to get the sps pps. Then it restarts in stream mode and passes the streaming data to th transferH264 class. The transferh264 class does two things for me.
- It reads the incoming stream and sorts out the nalus
- It packetizes those nalus to be sent to wherever the hell
Remember, we passed the transferh264 object a pipe/inputstream and also a reference to a udp socket if you didn’t notice. This is all done on a separate thread. Here is our repeating loop. Notice we create our sps and start by searching for a nalu with picture data. Then we keep recording each nalu and timing them so we can rebuild them with correct timing on the other side.
[java]
@Override
public void run()
{
/*
try{
streamTCP();
}catch (IOException ioe){
Log.e(TAG, "run: ", ioe );
}
*/
confirmUDPWorks();
try {
// find the mdat box?
//build our description
buildSPSPPS();
//find th first nalu
syncWithNalu();
while (notEOF)
{
duration = System.nanoTime() – start;
buildNalu();
start = System.nanoTime();
readNextHeader();
}
in.close();
rtpSocket.close();
} catch (IOException e) {
Log.e(getClass().getSimpleName(),
"Exception transferring file", e);
}
}
[/java]
Now in our situation we know that we need to find Avcc style header but I will show how to search for both avcc and for annex b in a byte stream.
Here is the method I used in my actual code. If you wanted to search for annex b instead of this
[java]
naluHeader[0] = naluHeader[1];
naluHeader[1] = naluHeader[2];
naluHeader[2] = naluHeader[3];
naluHeader[3] = naluHeader[4];
naluHeader[4] = (byte) in.read();
type = naluHeader[4]&0x1F;
if (type == 5 || type == 1)
{
naluLength = (naluHeader[3]&0xFF | (naluHeader[2]&0xFF)<<8 | (naluHeader[1]&0xFF)<<16 | (naluHeader[0]&0xFF)<<24) – 1; //minus type for header!!! if (naluLength > 0 && naluLength < 200000)
{
//Log.d(TAG, "naluSearch: found length = " + String.valueOf(naluLength) + " of type: " + String.valueOf(type) + " try req: " + String.valueOf(reqLoops));
break;
}
[/java]
try this
[java]
if (naluHeader[3] == 0x00 && naluHeader[2] == 0x00 && naluHeader[1] == 0x00 && naluHeader[0] == 0x01)
{
//we found it
}
[/java]
[java]
private void syncWithNalu() throws IOException
{
//Log.d(TAG, "syncWithNalu: started – we have no position! invalid data is length = " + String.valueOf(naluLength) + " type: " + String.valueOf(type));
byte save = naluHeader[0];
boolean firstPass = true;
int reqLoops = 0;
while (true){
reqLoops++;
naluHeader[0] = naluHeader[1];
naluHeader[1] = naluHeader[2];
naluHeader[2] = naluHeader[3];
naluHeader[3] = naluHeader[4];
naluHeader[4] = (byte) in.read();
type = naluHeader[4]&0x1F;
if (type == 5 || type == 1)
{
naluLength = (naluHeader[3]&0xFF | (naluHeader[2]&0xFF)<<8 | (naluHeader[1]&0xFF)<<16 | (naluHeader[0]&0xFF)<<24) – 1; //minus type for header!!! if (naluLength > 0 && naluLength < 200000)
{
//Log.d(TAG, "naluSearch: found length = " + String.valueOf(naluLength) + " of type: " + String.valueOf(type) + " try req: " + String.valueOf(reqLoops));
break;
}
if (naluLength==0)
{
Log.d(TAG, "naluSearch: null nalu");
}
}else if (firstPass) {
firstPass = false;
int testtype = (naluHeader[2] &0xFF | (naluHeader[1]&0xFF)<<8 | (naluHeader[0]&0xFF)<<16 | (save &0xFF)<<24) – 1; //minus type for header!!!
//DEBUG BAD NALUS HERE
String tt = String.valueOf(testtype);
byte[] b = new byte[512];
//Debug.debugHex("syncwithnalu " + tt, test, test.length);
b[0] = save;
b[1] = naluHeader[0];
b[2] = naluHeader[1];
b[3] = naluHeader[2];
b[4] = naluHeader[3];
b[5] = naluHeader[4];
in.read(b, 6, b.length-6);
Debug.debugHex("syncwithnalu " , b, 30);
}
}
}
[/java]
Once we sync our nalu stream we know exactly how long our next nalu should be. So lets fill it in with the buildnalu method. Simply copying into a buffer. Well get to packetization in the next section so don’t worry about that part yet.
[java]
//build next data which should be video payload
private void buildNalu() throws IOException
{
naluBuffer = new byte[naluLength+5];
//here we recombine our original header to our nalu data to be sent
naluBuffer[0] = naluHeader[0];
naluBuffer[1] = naluHeader[1];
naluBuffer[2] = naluHeader[2];
naluBuffer[3] = naluHeader[3];
naluBuffer[4] = naluHeader[4];
in.read(naluBuffer, 5, naluLength);
naluLength = naluBuffer.length;
/*
test[0] = naluBuffer[naluBuffer.length-4];
test[1] = naluBuffer[naluBuffer.length-3];
test[2] = naluBuffer[naluBuffer.length-2];
test[3] = naluBuffer[naluBuffer.length-1];
test[4] = naluBuffer[naluBuffer.length-4];
test[5] = naluBuffer[naluBuffer.length-3];
test[6] = naluBuffer[naluBuffer.length-2];
test[7] = naluBuffer[naluBuffer.length-1];
*/
timeStampCalulations(); //here we calc the time between reading each nalu. each nalu must have different time stamp
// String s1 = String.format("%8s", Integer.toBinaryString(naluHeader[4] & 0xFF)).replace(‘ ‘, ‘0’);
//Log.d(TAG, "packetizeNalu: expected raw " + s1);
//debugPackets("buildnalu ", naluBuffer);
packetizeNalu();
}
[/java]
So we have our first nalu loaded and sent. Our next nalu should be right behind it. No need to search. We test and make sure its right. If it is we go ahead and let it load the data. If the nalu is bad we go back to syncing method.
[java]
/read next header into header fields. expects to be dropped into correct position or it will perform a sync
private void readNextHeader() throws IOException
{
in.read(naluHeader, 0, 5);
type = naluHeader[4]&0x1F;
// String s1 = String.format("%8s", Integer.toBinaryString(naluHeader[4] & 0xFF)).replace(‘ ‘, ‘0’);
// Log.d(TAG, "packetizeNalu: handing type " + s1 + " " + String.valueOf(type));
naluLength = (naluHeader[3]&0xFF | (naluHeader[2]&0xFF)<<8 | (naluHeader[1]&0xFF)<<16 | (naluHeader[0]&0xFF)<<24)- 1; //minus 1 for header!!! if (naluLength >= 200000 || naluLength < 0){
syncWithNalu();
}else{
Log.d(TAG, "readNextHeader success type " + String.valueOf(type) + " length " + String.valueOf(naluLength));
}
// IDR is a stand alone picture. sending spspps will ake it readable even in a live stream format without session description protocal
if (type == 5){
packetizeDecsription();
}
}
[/java]
Notice how we check if the header is type 5. Type 5 slice is an IDR picture which means its the full readable image. It is followed by type 1 for example which tells us what to change on the original type 5 slice. So before each type 5 slice we send the sps and pps so that the decoder receiving the images has all the data it need to decode the pictures.
Below is my debugger output of the first 10 characters of each nalu without the 4 bytes of length. There was probably about 10 more type 41 and then it repeated again and again. This is the pattern we are trying to feed to our packetizing method.
VideoDecoderaddPacket type: 24
Debug transfer 67 80 80 1E E9 01 68 22 FD C0
Debug transfer 68 06 06 E2
Debug transfer 65 B8 20 00 9F 80 78 00 12 8A
Debug transfer 41 E2 20 09 F0 1E 40 7B 0C E0
Debug transfer 41 E4 40 09 F0 29 30 D6 00 AE
Debug transfer 41 E6 60 09 F1 48 31 80 99 40 Now that we have our data. Lets chop it up and send it in the next part.