Getting those packets onto a screen.
There will be one more post after this talking about some extra classes and techniques I used to get this done. So if I gloss over something here make sure and check there to get your codes straight.
-> Get the videodecoder code here <-
-> Get the imagedecoder code here <-
In order to decode the video we need a decoder that can understand what the nalu data has inside it. I’m using javafx with the javacv library to create the imagedecoder class. It paints an image onto an imageview with each frame it gets.
Here’s how I call it. Notice I tested it with a saved and emailed video from my android device first to make sure it was working.
[java]
public void createImageDecoder()
{
System.out.println(TAG + " image decoder");
/*
FileChooser fileChooser = new FileChooser();
File file = fileChooser.showOpenDialog(null);
if (file == null){
return;
}
*/
ImageView imageView = mController.getImageView_Images();
Executors.newSingleThreadExecutor().execute( mediaDecode = new Runnable() {
@Override
public void run() {
ImageDecoder.streamImageToImageView(
imageView,
pipedInputStream,
//file,
"h264",
10,
10000000,
"medium",
0
);
}
});
}
[/java]
The thing is, the imagedecoder class needs pristine annexb style nalus to work. I had all kinds of trouble getting it to play. Finally I downloaded ffmpeg to my windows machine and sent a stream of a video I had saved on my computer already to test the player. It worked. But I was even more clever, I also recorded the bytes sent in that stream so I could compare to what I was sending in myself. Muhahaha!!
Here’s the first section of what ffmpeg streamed. Notice anything? It send a nalu type 0x06 after the sps and pps. I also found out that it sent a nalu type 0x01 as well. I still am not sure what these are as I am writing this blog moments after completing my stream.
00 00 00 01 67 64 00 16 AC D9 40 88 16 FB F0 11 00 00 03 00 01 00 00 03 00 14 0F 16 2D 96 00 00 00 01 68 EB E3 CB 22 C0 00 00 01 06 05 FF FF AA DC 45 E9 BD E6 D9 48 B7 96 2C D8 20 D9 23 EE EF 78 32 36 34 20 2D 20 63 6F 72 65 20 31 35 35 20 72 32 39 30 31 20 37 64 30 66 66 32 32 20 2D 20 48 2E 32 36 34 2F 4D 50 45 47 2D 34 20 41 56 43 20 63 6F 64 65 63 20 2D 20 43 6F 70 79 6C 65 66 74 20 32 30 30 33 2D 32 30 31 38 20 2D 20 68 74 74 70 3A 2F 2F 77 77 77 2E 76 69 64 65 6F 6C 61 6E 2E 6F 72 67 2F 78 32 36 34 2E 68 74 6D 6C 20 2D 20 6F 70 74 69 6F 6E 73 3A 20 63 61 62 61 63 3D 31 20 72 65 66 3D 33 20 64 65 62 6C 6F 63 6B 3D 31 3A 30 3A 30 20 61 6E 61 6C 79 73 65 3D 30 78 33 3A 30 78 31 31 33 20 6D 65 3D 68 65 78 20 73 75 62 6D 65 3D 37 20 70 73 79 3D 31 20 70 73 79 5F 72 64 3D 31 2E 30 30 3A 30 2E 30 30 20 6D 69 78 65 64 5F 72 65 66 3D 31 20 6D 65 5F 72 61 6E 67 65 3D 31 36 20 63 68 72 6F 6D 61 5F 6D 65 3D 31 20 74 72 65 6C 6C 69 73 3D 31 20 38 78 38
Here is the stream we will be sending. Notice is goes from sps pps straight to type 65 which is an idr slice. Also notice I had an error (bytes[1]&[2] are same) in my sps pps I was not aware of at the time. Very frustrating. After I fixed these errors this stream pattern plays!
00 00 00 01 67 80 80 1F E9 01 68 22 FD C0 36 85 09 A8 00 00 00 01 68 06 06 E2 00 00 00 01 65 B8 40 0B E4 2F F9 FF 12 00 02 1A B8 48 F0 FF 36 5D 07 1E 52 C3 1F F3 FA A5 77 44 70 91 04 48 6A 59 C9 AE D3 B9 AA 18 C2 15 82 B4 30 92 2E C5 2D 26 C5 B0 A7 EE CD 9B 7E 99 D0 BE 8A 3E AF 69 18 DC 40 5D 40 3F 77 5C 98 49 C6 6D 4E ED 16 ED FB 7A 0A 04 AF D0 90 61 75 02 CE 3B 04 D3 69 A3 19 8E A6 AD 20 9B 69 A7 6C 88 AC 6E 5F F3 1A 2E 86 30 8D C0 15 74 C5 BC 5B 4E D7 F4 62 02 A8 B2 DA DA 08 31 80 48 F5 F7 5E 39 CC A6 5D E9 0B 62 DF B4 DE 1B 70 6E 8E 4D 40 B1 FC B6 68 C9 80 BA 82 1F F8 D7 68 E6 B3 6B 5B 4D 53 14 05 60 AB 9A 7D 5E D3 24 C3 41 75 16 4E 35 5F FE DA 76 DB 1F 18 36 11 CD 74 8C 62 DD 0B A8 74 4F 00 82 F6 E9 27 A4 6D 8E 24 92 2F F2 F0 BA 83 58 04 B6 9A 4E D6 DD AC 71 78 15 34 97 CF 50 C6 32 3C 9B 8B 69 E9 A6 D9 B3 D7 13 22 A1 54 D7 A6 82 0A 64 08 07 4D 3D 34 FA 76 FE 85 D4 6C 8F F4 D3
In order to create this beautiful array of data we need our videodecoder class to sort through packets and serve up complete nalus in order. My example is still missing timing info an optimization so the picture is choppy and has artifacts. But this is getting you in the door. Which is a hell of lot better than where you started!
You have my code but the overall strategy is pretty basic.
- incoming udp packets are written into my video decoder addpacket method
- Packets are sorted by type all my packets were type 24(spspps) and type 28(nalu chunks)
- packets are reworked and sent to the video decoder
Reworking sps pps
Thisvis easy simply split them up, add your 0x00 0x00 0x00 0x01 start code and send em through.
Reworking type 28 fua nalu chunks
To rebuild my nalus I kept a list
private Map<Integer, NaluBuffer> assemblyLine = new HashMap<>();
If a nalu has 100 pieces each piece shares the same timestamp. So I created a synchronized method to check if my list has already started building the nalu or if I need to start a new buffer. As below…
[java]
// Unpack either any split up nalu – This will get 99.999999 of nalus
synchronized private void unpackType28(byte[] twentyEight)
{
//Debug.deBugHexTrailing("unpack 28 ", twentyEight, 20 );
int ts = (twentyEight[4] << 24 | twentyEight[5] << 16 | twentyEight[6] << 8 | twentyEight[7] & 0XFF); //each nalu has a unique timestamp
//int seqN = (twentyEight[2] << 8 | twentyEight[3] & 0xFF); //each part of that nalu is numbered in order.
// numbers are from every packet ever. not this nalu. no zero or 1 start
//check if already building this nalu
if (assemblyLine.containsKey(ts)){
assemblyLine.get(ts).addPiece(twentyEight);
}
//add a new nalu
else
{
assemblyLine.put(ts, new NaluBuffer(ts, twentyEight));
}
}
[/java]
As each piece is loaded into a buffer a few things happen. We record how long its been waiting (nalus that aren’t completed in under a second are worthless), we strip out the rtp headers etc, and we count sequence numbers to rebuild each piece one after another checking if the nalu is complete each time. Once complete its sent through to the video decoder.
My current code needs serious optimization. So you will notice major artifacts due to missing or late nalus and timing? forget about it. But its pretty simple to understand and you can build those features yourself.
One More post to go!