Parrot ARDrone 2.0 Video Streaming through OpenCV in Linux

Parrot ARDrone 2.0

Recently, I got the Parrot ARDrone 2.0. Enjoying taking control of electronics and software, I wanted to write my own code that will stream the video from the quadcopter to my computer and give me the ability to process the individual video frames with OpenCV. There are lots of examples how to do this for ARDrone 1.x, but virtually nothing for ARDrone 2.0, which completely changed the way it’s streaming the video. After many hours of research and putting different pieces of code together, I got it to work on ARDrone 2.0. In this post, I will show you how to do it.

This code relies heavily on the official Linux SDK for ARDrone 2.0. You can download the SDK from here:

ARDrone 2.0 SDK

The Linux example in the SDK is way too bloated for someone to quickly fudge a working video demo from it. Luckily, good people are out there, and folks shared a stripped down version of a simple video output code. Download it here:

ARDrone 2.0 Video Output Example

(Credit for this code goes to those who posted it here: https://projects.ardrone.org/boards/1/topics/show/5022)

Next, install OpenCV. If you’re on Ubuntu as I am, you can just install it using the Synaptic manager. Don’t forget to install libcv and libcv-dev. In fact, install dev packages for all the OpenCV packages you’ll install. The minimum should include these:

libopencv-dev
libopencv-core
libopencv-highgui
libopencv-highui-dev
libcv
libcv-dev

I installed them all, but I think only the above listed ones suffice for this project.

Now extract the downloaded video example code to the folder ARDrone_SDK_2_0/Examples/Linux/, where ARDrone_SDK_2_0 is the directory containing your extracted ARDrone 2.0 SDK. So after you extract the example into that location, you will have a folder called sdk_demo under ARDrone_SDK_2_0/Examples/Linux/.

Now we have to modify the Make file to include OpenCV libraries. Credit for this and big help for tackling this problem comes from this site, which is tailored for ARDrone 1.x series: http://gauth.fr/2011/09/use-opencv-with-the-ar-drone-sdk. Moving on…

Open the file ARDrone_SDK_2_0/Examples/Linux/video_demo/Build/Makefile in your text editor and add the following under the line 44 (line containing GENERIC_INCLUDES:=$(addprefix -I,$(GENERIC_INCLUDES)):

GENERIC_INCLUDES+=`pkg-config --cflags opencv`

This will include the OpenCV header files in the compilation process.

Below this line, there is a line saying:

GENERIC_LIBS=-lpc_ardrone -lrt -lgtk-x11-2.0 -lcairo

Rewrite it to contain this:

GENERIC_LIBS=-lpc_ardrone -lrt -lgtk-x11-2.0 -lcairo `pkg-config --libs opencv`

This will include the OpenCV libraries during linking process.

Next we need to make the modifications to the example source code to make it work with OpenCV. First, the example uses PIX_FMT_RGB565 video encoding format. We need to switch to one that OpenCV works well with, which is PIX_FMT_RGB24. To do so, open ARDrone_SDK_2_0/Examples/Linux/video_demo/Sources/ardrone_testing_tools.c and change line 164 from

out_picture->format = PIX_FMT_RGB565;

to

out_picture->format = PIX_FMT_RGB24;

This will effectively break the video output as written by this example, so now the only way to display the video is using OpenCV. That’s what we want, so it’s okay to disable the original video output capabilities.

Next edit this source code file: ARDrone_SDK_2_0/Examples/Linux/video_demo/Sources/Video/display_stage.c. First, add the the OpenCV header files to the top of the file like so:

#include <opencv/cv.h>
#include <opencv/highgui.h>

Then add the following function to convert the video frames to OpenCV image objects:

IplImage *ipl_image_from_data(uint8_t* data, int reduced_image, int width, int height)
{
  IplImage *currframe;
  IplImage *dst;

  currframe = cvCreateImage(cvSize(width,height), IPL_DEPTH_8U, 3);
  dst = cvCreateImage(cvSize(width,height), IPL_DEPTH_8U, 3);

  currframe->imageData = data;
  cvCvtColor(currframe, dst, CV_BGR2RGB);
  cvReleaseImage(&currframe);

  return dst;
}

Then edit the following function in the code like so:

C_RESULT display_stage_transform (display_stage_cfg_t *cfg, vp_api_io_data_t *in, vp_api_io_data_t *out)
{
    uint32_t width = 0, height = 0;
    getPicSizeFromBufferSize (in->size, &width, &height);

    IplImage *img = ipl_image_from_data((uint8_t*)in->buffers[0], 1, 640, 360);
    cvNamedWindow("video", CV_WINDOW_AUTOSIZE);
    cvShowImage("video", img);
    cvWaitKey(1);
    cvReleaseImage(&img);

    /*
    // We don't need any of the below because OpenCV will be displaying the image, not GTK.

    // Process only if we are using RGB565
    if (FALSE == cfg->paramsOK)
    {
        return C_OK;
    }
    // Realloc frameBuffer if needed
    if (in->size != cfg->fbSize)
    {
        cfg->frameBuffer = vp_os_realloc (cfg->frameBuffer, in->size);
        cfg->fbSize = in->size;
    }

    // Copy last frame to frameBuffer
    vp_os_memcpy (cfg->frameBuffer, in->buffers[in->indexBuffer], cfg->fbSize);

    // Ask GTK to redraw the window
    uint32_t width = 0, height = 0;
    getPicSizeFromBufferSize (in->size, &width, &height);
    if (TRUE == gtkRunning)
    {
        gtk_widget_queue_draw_area (cfg->widget, 0, 0, width, height);
    }

    // Tell the pipeline that we don't have any output
    out->size = 0;
    */

    return C_OK;
}

I commented out all the old code in the function because we won’t be displaying the video output using that code anymore. It wouldn’t work with our new video format anyway.

Now, we’re basically done. Go to ARDrone_SDK_2_0/Examples/Linux/video_demo/Build and make the source code by typing

make

in terminal in that folder. The built executable will be in ARDrone_SDK_2_0/Examples/Linux/Build/Release. Just run it with

./linux_video_demo

and you should get a Parrot ARDrone 2.0 video output on your screen (of course, make sure you’re wifi connected to the drone first). Enjoy!

Bookmark the permalink.
  • pkout

    Hi! The file ardrone_testing_tools.c is not in the AR Drone SDK, but in the video example code that you need to download above (see my links above). Once you download it, the compressed file will contain a folder video_demo. You need to extract that folder and copy it into the official SDK so that it’s in this location: ARDrone_SDK_2_0\Examples\Linux\video_demo. Then follow my steps above. I hope this helps.

  • emk

    Hi,

    I compile your code and all ok, but when I execute it shows “unsoported codec” and “segment fault”. When I compile without Opencv (–cflag and –libs) and without code in display_stage the execution is correct…

    any idea?

    Thank you.

    • pkout

      Hi,

      Hmm…not sure here. What version of OpenCV do you have on your system? Did you compile it yourself or installed it using apt-get or a similar tool?

      • emk

        Hi pkout,

        with OpenCV 2.4.2, and I have compiled it. The compilation process is ok, but on rutime the console shows those messages….
        and shows the “segment fault” and exit.

        Then, when I compile it without –cflags –libs opencv and delete the opencv-lines of codes, the original demo works well.

        Best regards,

        • pkout

          Hi emk,

          I didn’t compile my OpenCV myself. I installed it from repositories using apt-get on my ubuntu. I think my OpenCV is version 2.3, but it shouldn’t matter. So my feeling is that there might be some compilation dependency – some of the OpenCV compilation flags might need to be turned on. But that sounds strange because we are using only the bare core of OpenCV. If you’re on Ubuntu or any other distro that has OpenCV in their repositories, I would recommend uninstalling your compiled version and installing the official repository version. Chances are something went wrong with your compilation. Sorry I can’t give you a better hint. These sorts of errors give almost no clue as to what is wrong. In general, I prefer using repository packages from compiling. I’ve had segmentation fault problems with libraries that I compiled myself before. Best luck! Petr.

          • emk

            Hi again,

            I solved it !!!, I compiled all opencv, ffmpeg, etc.. libraries from the beginning, and works fine. Maybe the “segment fault” error is showed when any pointer has an incorrect address, ….

            We use Opencv 2.4.2, the last version.

            Thanks for all.

  • pkout

    Hi emk,

    Congratulations! This is strange. I didn’t have to compile any of these libraries. It just worked using the ones installed from the repositories. Well, obviously something wasn’t right in the setup on your computer. I am glad to see you worked it out! Enjoy! Best, Petr.

  • L10

    Hi Petr,

    I got your example to work but I am confused as to how to use it from this point forward. How am I suppose to process the images with OpenCV? At what files am I suppose to make those functions? Let’s say I want to make a function to track a ball with the Drone’s camera, where and how should I start?

    Thanks.

    • pkout

      Hi,

      Congrats to getting the example to work! As for OpenCV, if you have no experience with it, you should first learn the basics of it. Then you will know what to do. My code gives you an OpenCV IplImage object for each frame. You have to take it from there by yourself. What you do totally depends on what you want to achieve. A good place to learn the basics of OpenCV is this blog: http://www.aishack.in/2010/07/tracking-colored-objects-in-opencv/

      Good luck!
      Petr

      • L10

        Hey, thanks a lot pkout. I took a look at your link and it is really helping me understand OpenCV. I actually tinkered around and got your code working for me. It is a simple test. I can share if you like. Thanks again! Your guide is very helpful!

        • pkout

          It’s a pleasure to read that it’s working for you! Good luck with your further development!

  • Lizbeth

    HI, my name is Sabrina Lizbeth Vega (lizbethvegam@gmail.com)

    I am running out of time with my project and still stuck with the iOS vs ARDrone SDK 2.0 and OpenCV.
    I installed ARDrone 2.0 SDK and openCV library. It’s ok. I am working with ios 7

    Could you tell me how I be can able to implement this code on ios?
    Please help me. I apologize for mi poor english

    • pkout

      Hi Sabrina. I am sorry, but I have no experience whatsoever working on iOS platform, so I can’t help you there.

      • Lizbeth

        ok, no problem, thank you anyway

        🙁

        • kariuki githua

          After following your steps am unable to make. Errors occur while compiling

          kariukigithua@ubuntu:~/Downloads/ARDrone_SDK_2_0_1/Examples/Linux/video_demo/Build$ make
          Libs already extracted
          Building target static
          Architecture i686 is already built
          Creating universal static lib file from architectures i686
          Build done.
          Checking required Ubuntu packages …
          ok.
          Building ARDroneTool/Lib
          Building ARDroneTool/Lib
          dep ardrone_testing_tool.c
          cc ardrone_testing_tool.c
          ld ardrone_testing_tool
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o: In function `ardrone_tool_init_custom’:
          ardrone_testing_tool.c:(.text+0x4e): undefined reference to `ardroneVersion’
          ardrone_testing_tool.c:(.text+0x6a): undefined reference to `ardrone_application_default_config’
          ardrone_testing_tool.c:(.text+0x75): undefined reference to `ardrone_application_default_config’
          ardrone_testing_tool.c:(.text+0x7f): undefined reference to `ardrone_application_default_config’
          ardrone_testing_tool.c:(.text+0x89): undefined reference to `ardrone_application_default_config’
          ardrone_testing_tool.c:(.text+0x92): undefined reference to `ardrone_application_default_config’
          ardrone_testing_tool.c:(.text+0x296): undefined reference to `video_stage_init’
          ardrone_testing_tool.c:(.text+0x29c): undefined reference to `ardroneVersion’
          ardrone_testing_tool.c:(.text+0x2d0): undefined reference to `video_recorder_init’
          ardrone_testing_tool.c:(.text+0x2d5): undefined reference to `video_stage_resume_thread’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o: In function `ardrone_tool_shutdown_custom’:
          ardrone_testing_tool.c:(.text+0x2f4): undefined reference to `video_stage_resume_thread’
          ardrone_testing_tool.c:(.text+0x31e): undefined reference to `ardroneVersion’
          ardrone_testing_tool.c:(.text+0x326): undefined reference to `video_recorder_resume_thread’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x0): undefined reference to `ardrone_navdata_control_init’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x4): undefined reference to `ardrone_navdata_control_process’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x8): undefined reference to `ardrone_navdata_control_release’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x10): undefined reference to `ardrone_general_navdata_init’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x14): undefined reference to `ardrone_general_navdata_process’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x18): undefined reference to `ardrone_general_navdata_release’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x20): undefined reference to `video_navdata_handler_init’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x24): undefined reference to `video_navdata_handler_process’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x28): undefined reference to `video_navdata_handler_release’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x30): undefined reference to `ardrone_academy_navdata_init’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x34): undefined reference to `ardrone_academy_navdata_process’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x38): undefined reference to `ardrone_academy_navdata_release’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x74): undefined reference to `thread_video_stage’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0x94): undefined reference to `thread_video_recorder’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0xb4): undefined reference to `thread_navdata_update’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o:(.data+0xd4): undefined reference to `thread_ardrone_control’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/ardrone_testing_tool.o: In function `main’:
          ardrone_testing_tool.c:(.text.startup+0xb0): undefined reference to `gtk_init’
          ardrone_testing_tool.c:(.text.startup+0xc4): undefined reference to `ardrone_tool_main’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/Video/display_stage.o: In function `on_expose_event’:
          display_stage.c:(.text+0x111): undefined reference to `gtk_window_get_type’
          display_stage.c:(.text+0x121): undefined reference to `g_type_check_instance_cast’
          display_stage.c:(.text+0x135): undefined reference to `gtk_window_resize’
          display_stage.c:(.text+0x144): undefined reference to `gdk_cairo_create’
          display_stage.c:(.text+0x169): undefined reference to `cairo_image_surface_create_for_data’
          display_stage.c:(.text+0x181): undefined reference to `cairo_set_source_surface’
          display_stage.c:(.text+0x189): undefined reference to `cairo_paint’
          display_stage.c:(.text+0x191): undefined reference to `cairo_surface_destroy’
          display_stage.c:(.text+0x199): undefined reference to `cairo_destroy’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/Video/display_stage.o: In function `display_stage_transform’:
          display_stage.c:(.text+0x22e): undefined reference to `cvCreateImage’
          display_stage.c:(.text+0x256): undefined reference to `cvCreateImage’
          display_stage.c:(.text+0x277): undefined reference to `cvCvtColor’
          display_stage.c:(.text+0x27f): undefined reference to `cvReleaseImage’
          display_stage.c:(.text+0x297): undefined reference to `cvNamedWindow’
          display_stage.c:(.text+0x2ab): undefined reference to `cvShowImage’
          display_stage.c:(.text+0x2b7): undefined reference to `cvWaitKey’
          display_stage.c:(.text+0x2bf): undefined reference to `cvReleaseImage’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/Video/display_stage.o: In function `ipl_image_from_data’:
          display_stage.c:(.text+0x2f5): undefined reference to `cvCreateImage’
          display_stage.c:(.text+0x315): undefined reference to `cvCreateImage’
          display_stage.c:(.text+0x336): undefined reference to `cvCvtColor’
          display_stage.c:(.text+0x342): undefined reference to `cvReleaseImage’
          ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/Video/display_stage.o: In function `thread_gtk’:
          display_stage.c:(.text+0x391): undefined reference to `gtk_window_new’
          display_stage.c:(.text+0x3c2): undefined reference to `g_signal_connect_data’
          display_stage.c:(.text+0x3e2): undefined reference to `gtk_main_quit’
          display_stage.c:(.text+0x3f2): undefined reference to `g_signal_connect_data’
          display_stage.c:(.text+0x3f7): undefined reference to `gtk_window_get_type’
          display_stage.c:(.text+0x405): undefined reference to `g_type_check_instance_cast’
          display_stage.c:(.text+0x415): undefined reference to `gtk_window_set_position’
          display_stage.c:(.text+0x421): undefined reference to `g_type_check_instance_cast’
          display_stage.c:(.text+0x439): undefined reference to `gtk_window_set_default_size’
          display_stage.c:(.text+0x449): undefined reference to `gtk_widget_set_app_paintable’
          display_stage.c:(.text+0x459): undefined reference to `gtk_widget_set_double_buffered’
          display_stage.c:(.text+0x461): undefined reference to `gtk_widget_show_all’
          display_stage.c:(.text+0x470): undefined reference to `gtk_main’
          collect2: error: ld returned 1 exit status
          make[3]: *** [/home/kariukigithua/Downloads/ARDrone_SDK_2_0_1/Examples/Linux/video_demo/Build/../../Build/Release/ardrone_testing_tool] Error 1
          make[2]: *** [all] Error 2
          make[1]: *** [build_app] Error 2
          make: *** [linux_video_demo] Error 2

          • Miguel Rocha Jr

            I am having this exact problem. Did you fin a solution for it?

          • IvoStinghen

            any solution Miguel?

          • IvoStinghen

            @miguel_rocha_jr:disqus

      • kariuki githua

        please help pkout

        kariukigithua@ubuntu:~/Downloads/ARDrone_SDK_2_0_1/Examples/Linux/video_demo/Build$ make
        Libs already extracted
        Building target static
        Architecture i686 is already built
        Creating universal static lib file from architectures i686
        Build done.
        Checking required Ubuntu packages …
        ok.
        Building ARDroneTool/Lib
        Building ARDroneTool/Lib
        ld ardrone_testing_tool
        /usr/bin/ld: ../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.13.0-32-generic_GNU_Linux_usrbingcc_4.8.2/Video/display_stage.o: undefined reference to symbol ‘gdk_cairo_create’
        //usr/lib/i386-linux-gnu/libgdk-x11-2.0.so.0: error adding symbols: DSO missing from command line
        collect2: error: ld returned 1 exit status
        make[3]: *** [/home/kariukigithua/Downloads/ARDrone_SDK_2_0_1/Examples/Linux/video_demo/Build/../../Build/Release/ardrone_testing_tool] Error 1
        make[2]: *** [all] Error 2
        make[1]: *** [build_app] Error 2
        make: *** [linux_video_demo] Error 2

        • Ed Carlos Schneider

          Please, see my post on top.

  • Alfred Junior Owona She

    Hello, i am a IT student and i want to stream ar drone i follow your tutorial but when do make on the terminal it display :
    generic.makefile:217: recipe for target ‘../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.16.0-23-generic_GNU_Linux_usrbingcc_4.9.1/Video/display_stage.o’ failed
    make[3]: *** [../../Soft/Build/targets_versions/linux_video_demo_PROD_MODE_vlib_Linux_3.16.0-23-generic_GNU_Linux_usrbingcc_4.9.1/Video/display_stage.o] Error 1
    app.makefile:33: recipe for target ‘all’ failed
    make[2]: *** [all] Error 2
    Makefile:163: recipe for target ‘build_app’ failed
    make[1]: *** [build_app] Error 2
    Makefile:73: recipe for target ‘linux_video_demo’ failed
    make: *** [linux_video_demo] Error 2

    Thanks in advance.

    • Ed Carlos Schneider

      Hi, i have the same error. I resolved this problem with adjust in a “ARDrone_SDK_2_0/Examples/Linux/video_demo/Build/Makefile” file, on replace the line “GENERIC_LIBS=-lpc_ardrone -lrt -lgtk-x11-2.0 -lcairo” i see this is different on my MakeFile. At this point, i copy the original line and add “pkg-config…” option, so i have this: “GENERIC_LIBS=-lpc_ardrone -lrt -lgtk-x11-2.0 -lcairo -lgdk-x11-2.0 -lgobject-2.0 -lm `pkg-config –libs opencv`” and now it compile with success and not show error!

  • sunnywust

    Hello,i can’t open the website of https://projects.ardrone.org/boards/1/topics/show/5022,could you sent the “ARDrone 2.0 Video Output Example” code to my email which is m13387558496@163.com,thank you!

  • sunnywust

    who can send me the code of “ARDrone 2.0 Video Output Example”,my email is m13387558496@163.com,
    Thanks in advance.