Stream Video From Android Part 6 – Packetize RTP

I know, you deserve a nap but please hang in there.

The file I’m referencing is in the last post if you need it. Also MAJOR WARNING HERE!!! I did not test my rtp packet code against another software. There may be errors because I simply wrote what seemed to make sense and then wrote a javafx program to open it on the other side. But this still should get you pretty dang close.

In the previous post we had our sps, pps and different nalus being fed into a packetizer to be sent over the internet. As you are well aware in java we can use a TCP or UDP connection. Either is fine but I will focus on UDP for this post. Lets talk about that process.

RTP is a defined format that can send all kinds of data including video streams. A udp packet contains an rtp packet which contain a piece of data. We choose our packet size based on maximum transmission unit which is the number of bytes we can send at a time. In our example we set our mtu to 1500 and we limit our payload to 1300 so we have space for the enclosing packet headers as well.

Let look at the spspps packet. Its smaller and can be sent in a single rtp packet. All rtp packets must comply with the rfc guidelines. Search rfc 6184  to see what I’m talking about. It describes packetizing different data in different ways. Remember our buildspspps method? It needs to be organized according to these protocols.  Below we build the payload that will be inserted into our rtp packet. This is done only once.

// get from myvideo / build sps and pps data
private void buildSPSPPS()
{
    //without this stream is worthless
    if (sps == null || pps == null){
        notEOF = false;
        Log.d(TAG, "buildSPSPPS: no sps or pps data");
        return;
    }

    if (description == null){

        description = new byte[sps.length + pps.length + pref.length ];


        description[0] = 24;
        //rtp header trpe 24 = Single-time aggregation packet     5.7.1

        // Write NALU 1 size into the array (NALU 1 is the SPS).
        description[1] = (byte) (sps.length >> 8);
        description[2] = (byte) (sps.length & 0xFF);

        // Write NALU 2 size into the array (NALU 2 is the PPS).
        description[sps.length + 3] = (byte) (pps.length >> 8);
        description[sps.length + 4] = (byte) (pps.length & 0xFF);

        //write prefix
        //System.arraycopy(pref, 0, description, description.length-6, pref.length);

        // Write NALU 1 into the array, then write NALU 2 into the array.
        System.arraycopy(sps, 0, description, 3, sps.length);
        System.arraycopy(pps, 0, description, 5 + sps.length, pps.length);

        Debug.debugFull(" build spspps ", description);

    }

Then before every idr picture we send this data via an rtp packet.

private void packetizeDecsription()
{
    buildRTPPacket(24, timeStamp, description, description.length);
}

Its important to note that we need to keep track of each packet sent so that or depacketizer can determine order and timing on the other side. That means we can have only a single buildRTPPacket() method and that it must be accessed from a single thread or synchronized. There are plenty of diagrams within the source code file but as you can see I build a header and combine the payload with the header.  All rtp packets have the same info and are sent through this method.

public void buildRTPPacket(int payloadType, int timeStamp, byte[] payload, int payloadLength)
{
    //Log.d(TAG, "buildRTPPacket: " + String.valueOf(payloadType));
    //this is the actual packet being sent
    byte[] rtpPacket = new byte[HEADER_SIZE + payloadLength];
    sequenceNumber++;                                   //keep our packet stream linear.splt nalus with same timestamp are ordered by this number


    rtpHeader = new byte[HEADER_SIZE];

    rtpHeader[0] = (byte) 0b10000000;                  //(byte) (VERSION << 6 | PADDING << 5 | EXTENSION << 4 | CSRC_COUNT);
    rtpHeader[1] = (byte) payloadType;                 //ignore market bit //The first byte of a NAL unit co-serves as the RTP payload header   ->   https://tools.ietf.org/html/rfc6184   5.6
    rtpHeader[2] = (byte) (sequenceNumber >> 8);       //sequence move bits 8-16 right into the 8 bit buffer
    rtpHeader[3] = (byte) (sequenceNumber & 0xff);     //sequence only keep the the last 8 bits by masking
    rtpHeader[4] = (byte) (timeStamp >> 24);           //time stamp
    rtpHeader[5] = (byte) (timeStamp >> 18);           //time stamp
    rtpHeader[6] = (byte) (timeStamp >> 8);            //time stamp
    rtpHeader[7] = (byte) (timeStamp & 0xFF);          //time stamp
    rtpHeader[8] =  (byte) (SSRC >> 24);               //ssrc
    rtpHeader[9] =  (byte) (SSRC >> 16);               //ssrc
    rtpHeader[10] = (byte) (SSRC >> 8);                //ssrc
    rtpHeader[11] = (byte) (SSRC & 0xff);              //ssrc



    // here we load the header into the first 12 bytes and the payload after that
    System.arraycopy(rtpHeader, 0, rtpPacket,0,HEADER_SIZE);
    System.arraycopy(payload,0,rtpPacket,HEADER_SIZE, rtpPacket.length-HEADER_SIZE);

    debugPackets();

    //send as soon as its built  with true or false for console debugging
    send(rtpPacket, false);



}

Packetizing the nalu is easier than unpacking it. Most nalus, but possibly not all, are much much bigger than our mtu so they need to be broken down. I’m using the FU-A format because the libstream library I was studying used it as well. It requires a two part header on top of each nalu piece. Each nalu piece shares the same timestamp as you can see but they each have an incremental sequence number provided by the rtp packet. Without the correct sequence number and timestamp your nalus cannot be reassembled. Below I loop until my entire nalu is sent.

//called by build nalu to send asap after building it will packetize nalu split it up or whatever
// single nalu (section 5.6) or fu-a    see ->    https://tools.ietf.org/html/rfc6184   section 5.4
private void packetizeNalu()
{

    //Debug.checkNaluDataV2(naluBuffer);
    //Log.d(TAG, "packetizeNalu: " + String.valueOf(naluBuffer.length));

    //error checking
    int nalLengthChecker = (naluBuffer[3]&0xFF | (naluBuffer[2]&0xFF)<<8 | (naluBuffer[1]&0xFF)<<16 | (naluBuffer[0]&0xFF)<<24); //original nal length
    nalLengthChecker += 4; //here we add for the header 0-3... remember length includes nal header[4]type already
    int writtenLength = 0;
    int bytesAdded = 2; //here is our fua header



    byte[] buffer;

    //single nalu
    if (naluLength+1 <= MAX_SIZE )
    {
        buffer = new byte[naluLength+1];
        buffer[0] = naluHeader[4];
        System.arraycopy(naluBuffer, 0, buffer, 1, naluLength);

        buildRTPPacket(type, timeStamp, naluBuffer, naluLength);

    }
    // nalu is split like fu-a type
    else{
        /*
             FU-indicator      FU header
           +---------------+ +---------------+ +---------------+
           |0|1|2|3|4|5|6|7| |0|1|2|3|4|5|6|7|
           +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+     FU Payload
           |F|NRI| TypeofFU| |S|E|R|  Type   |
           +---------------+ +---------------+ +---------------+
             See rfc 6184 5.8 figure 15-ish
             See rfc 6184 5.3 "the value of NRI to 11"

                FU-A is type 28

         */
        byte[] fuaHeader = new byte[2];

        fuaHeader[0] = 0b01111100; //set indicator with "11" and type decimal 28 = FU-A -> 01111100
        fuaHeader[1] = (byte) (naluHeader[4] & 0x1F); //set header 3-7


        int tally = 0;
        int tocopy;
        boolean secondloop = false;

        while(tally < naluLength)
        {
            tocopy = (naluLength - tally);              //see whats left to write
            if (tocopy >= MAX_SIZE-2)                       // we minus 2 to make space for both header bytes
            {
                tocopy = MAX_SIZE-2;                    //fit into max allowable packet size
                buffer = new byte[MAX_SIZE];
            }else{
                buffer = new byte[tocopy+2];            //or shrink buffer to whats left plus header
            }

            if (secondloop)         //turn ser to double zero on second loop
            {
                 fuaHeader[1] = (byte) (fuaHeader[1]^(1 << 7));
                secondloop = false;
                //String s1 = String.format("%8s", Integer.toBinaryString(fuaHeader[1] & 0xFF)).replace(' ', '0');
                //Log.d(TAG, "packetizeNalu: center header " + s1);

            }

            if (tally == 0)                                 //first nalu in multi part. set SER...see above
            {
                fuaHeader[1] += 0x80;
                secondloop = true;
                //String s1 = String.format("%8s", Integer.toBinaryString(fuaHeader[1] & 0xFF)).replace(' ', '0');
                //Log.d(TAG, "packetizeNalu: adjusted header " + s1);

            }

            System.arraycopy(naluBuffer, tally, buffer, 2, tocopy); //copy to buffer skipping first 2 bytes
            tally += tocopy;

            if (tally >= naluLength)                        //weve copied all the data, set ser to last on multipart
            {
                fuaHeader[1] += 0x40;
                //String s1 = String.format("%8s", Integer.toBinaryString(fuaHeader[1] & 0xFF)).replace(' ', '0');
                //Log.d(TAG, "packetizeNalu: re-adjut header " + s1);
            }

            buffer[0] = fuaHeader[0];
            buffer[1] = fuaHeader[1];

            buildRTPPacket(fuaHeader[0], timeStamp, buffer, buffer.length);

            writtenLength += (buffer.length - bytesAdded); //here we count how many bytes were sent to packetizer to compare to our starting amount
        }

        if (writtenLength != nalLengthChecker){
            Log.e(TAG, "packetizeNalu: Mismatched Size orig: " + String.valueOf(nalLengthChecker) + " written " + String.valueOf(writtenLength), null );
        }


    }

    //Debug.checkNaluData(naluBuffer);

}

Then we send them to there destination. That’s it! Your data is on its way!!!

 

Leave a Reply

Your email address will not be published. Required fields are marked *