Aleksandar Krstikj.com

Tracking a ball and rotating camera with OpenCV and Arduino

by on May.20, 2013, under Code, Hardware, IT

I recently started playing around with the OpenCV library and one of the first tutorials I did dealt with tracking a ball with the camera. I expanded on that concept a bit by having the camera rotate and continue tracking the ball even if it goes off screen. I did this by connecting the camera to a small servo controlled by an Arduino Uno, which receives commands from my PC via a serial connection.


The base project I used can be found here. I decided not to reinvent the wheel and used that code as a starting point so props to the author. However I did modify the code slightly and also added some more things to make it work for my project. The modification refers to changing the color model from RGB to HSV as I found out that it’s much easier to specify a tracking color by using the Hue value rather then finding a good RGB value. The additions refer to the code for serial connectivity and the Arduino instructions send via the serial connection itself.

You can see how the project ended up looking and working in this short YouTube video I made:


You can download the OpenCV (C++) code here, and the Arduino code here.

I hope you enjoy this little project and feel free to post any questions or comments you might have regarding it.

:, , , , , ,

67 Comments for this entry

  • Andrei Cosofret

    Hello !
    I had an issue when i was trying to repeat this project.
    I installed MS Visual C++ 2010 Express and openCV 2.2 .
    I ran the c++ program in MS Visual C++ using as webcam my integrated laptop webcam and it worked. The two windows (original and processed) along with the cmd prompt showed up, but the laptop webcam being too weak it was difficult for it to distinguish the colors and to track them and impossible to connect to a servo motor :).
    So I tried a webcam (Microsoft LifeCAM VX-800) but when a ran the program the original window is grey and I don’t know what should I do to make the MS webcam working. Please help me ! my email adress : andreicosofret@gmail.com

    • Ethan Hunt

      Did you connect the MS webcam to the same computer (laptop) which already has an integrated webcam on it? If so, you may want to try and change the webcam that is being used by OpenCV.

      Look for the line that says: “p_capWebcam = cvCaptureFromCAM(0);” The number zero basically refers to the first webcam connected to the system. Try changing this number to 1 or something even higher. Based on how your OS detects the webcams they don’t always end up at the zero spot. You can also try it with -1, which means that OpenCV will use any camera it finds.

      Try it out and let me know 🙂

      • MaxonTan

        Based on Andrei Cosofret problem and the solution given by you. I had face this king of problem too abd i had tried the value from 0 til 100 (as well as negative) and yet the screen still appear in gray colour. Bro, is that got any coding can replace with “p_capWebcam = cvCaptureFromCAM(0);”? Reply me ASAP thanks bro

  • doel

    0 down vote favorite

    I’m using OpenCV in Visual Studio 2010 to detect sleepiness eye of the camera, and I try to send a value to the Arduino to activate buzzer (alarm) and the dc motor will stop after the eye drowsiness is detected (covered) by the camera. I am using the Arduino Uno. I have completed C + + code, but I have trouble sending this data to the Arduino. I am currently using the RS-232 cable for this. I use a type-B USB cable to program my Arduino and RS-232 to try to send data from Visual Studio for Arduino. How can I do it …which for opencv and arduino code

    • Ethan Hunt

      Take a look at the C++ code I used in my project. It contains the code needed to send commands to the Arduino via RS-232. You can see how to open the serial connection and how to send data. Notice the “WriteFile()” method I’m using to do this.

      Also, you can take a look at my other project located here: http://aleksandarkrstikj.com/mm-sorter-using-opencv-and-arduino/ . It also utilizes the “ReadFile()” method if you want to send commands from the Arduino back to the PC.

      In case you communication via the RS-232 cable doesn’t go through, try communicating with the Arduino directly through the virtual serial port it creates when you connect it via USB to your PC. That’s how I was using it. Just make sure to note which is the serial port that’s opened on your PC. My code expects the serial port at COM3 but yours may be different, so adjust accordingly.

      Let me know if that helps you. All the best.

  • pi

    how do the instructions for sending serial commands from pc (OpenCV + +) to the Arduino.?

    I made the driver’s drowsiness detection devices with camera

    that at pc = if you have found sleepiness (“alarm active”) then ..

    that in Arduino (hardware)
    1 = bazer will live
    2 = the wheels initially inactive to die (dc motors)
    3 = posts lcd “danger”

    how i can do it .. please help me

    • Ethan Hunt

      Please read the response I gave to the user ‘doel’ above.

      It will point you in the right direction as far as serial communication between a PC and an Arduino is concerned.

      You can take a look at the code I have posted in projects here to see how I managed to send commands to and from the Arduino.

  • GPS

    Hi! We’re using Visual Studio and OpenCV in our project to detect the red ball with the laser attached to the servo motor. We can’t seem to transmit the data from Visual Studio to Arduino Uno to control the servo motor via USB cable. We also tried to try the code you’ve posted but still, it doesn’t control our servo motor. What do you think can be the possible problem?

    • Ethan Hunt

      Can you successfully send the Arduino commands over the serial console of the Arduino IDE? I would try that first just to make sure something else isn’t causing the issues.

      Other than that, make sure that the COM port that the Arduino is attached to matches the COM port in the serial connection in Visual Studio. I had it setup to work for me but yours might be different.

      I can’t really say much more without some additional information like source code.

      • GPS

        Good day,

        finally i was able to get your codes working. it was just a matter of using the com port recognized by the pc.

        Moreover there is one more thing that i would like to achieve and may I ask for your opinion 😀

        It’s basically about tracking a red ball, however the camera shouldn’t be moving and is placed still only in one position. There will be a low cost laser pointer that will be attached to the servo motor instead which will then be controlled to track the target object.

        In addition to that, we are planning to make use of the extracted y coordinate from the program, so what happens now is there will be actually 2 servo motors, one for handling x values and the other to handle the y values which are working concurrently as the ball is being tracked.

        How are we able to point the laser pointer in such a way it is precisely matched to that of the x-y positions calculated from the image frame?

        Any insights from you will be highly appreciated 😀

        Thanks and more power! 🙂

        • Ethan Hunt

          Sound like an interesting project. To be honest, I’m quite busy these days so I haven’t given this too much thought but here’s what I’m thinking:

          You’ll get the x and y coordinates from OpenCV. You’ll need a way to calibrate the lasers, that is the servos to the coordinates. In my mind, this would be a trial and error thing.

          Since 1px on screen isn’t equal to 1 servo step you’ll need to find some ratio that matches. For example 1 servo step for every 10px.

          You can then take let’s say the x coordinate, and divide it by 10 (or whatever number you end up with) and end up with the number of steps you need to move.

          You’ll just have to make sure that the servo starting mid position is the camera mid position (both x and y) so that you can equally move left and right. Otherwise you might end up in a situation where the servo is at its end but the x coordinate is still to the left or right. Of course, if your servo can move 360 degrees forget I said that (my servos aren’t continuous) 🙂

          Same thing goes for the y coordinate. However since you’ll need to move the whole ‘y-assembly’ along when you’re moving your ‘x’ make sure that the whole thing is light enough for the servos to handle. Again, I’m thinking of my servos but if yours are some state of the art mega servos don’t worry about it 🙂

          Hope it helps!

          • GPS

            thanks for the insightful suggestions! we’re now actually working on trial and error calibration. at the same time figuring out the most optimal ratio of pixel to degrees for the tracking part.

            thanks again and more power to you! we might be back for more inquiries 😀

  • Harry

    Hi, thanks for sharing. I’m also planning to implement something similar with multi cameras and your work puts me on track.

  • Sloppy

    First of all let me say thank you. Thank you for the time and effort you put into this.
    From a noob in openCV standpoint this really helps understanding how to start working with openCV.

    I am using VS2010. I downloaded OpenFrameworks that include openCV.
    However, I cannot combine your code with the openCV libraries.

    I dont understand how all of this should be together for it to build correctly.
    as much as I understand, I should create a new win32 project. then in the folder created I should place the opencv and opencv2 folders.

    however, the 3 include statements say they cannot find the source file.

    what am I doing wrong?

    thank you very much again! 🙂

  • Fadil

    i cant open up a serial port…cn u pls help???o provide me a code or some article???i had tried tserial by Thierry.but no code seems to work..

    • Ethan Hunt

      To be perfectly honest, I also had serial port issues when I started doing this project.

      What I found out is that different operating systems behave differently as far as opening serial ports is concerned. So doing this on Win XP may not be the same as doing it on Win 7 or another OS. The reason behind this is the availability of libraries to open a serial connection.

      I would suggest that you do a search tailored for your OS and programming language. While I was searching, I found several different libraries that could manage a serial connection. I ended up going with this one because that worked for me, but there were a bunch of options out there.

      Some of them were outdated though, working only on older OSs, going as far back as Windows 98, so take that into consideration. Make sure that what you end up using is confirmed to work with your setup. After you get precise search results, it’s relatively simple to find examples and tutorials to set it up properly.

  • abdul

    hi..where I found the reference face tracking with OpenCV template matching method..can you see link for me..

  • keithpang

    A really nice job due!
    By the way, is it possible to generate the tracking signal to robotics? Like ABB and FANUC robot. I got a robot and it is programmable, I’m trying to apply this to the robot but it is new for me. Do you have any advise? Again, really great job bro! Keep in touch:)

    • Ethan Hunt

      Sorry, but my work with robotics usually boils down to some home-made flying-by-the-seat-of-your-pants robots built using home made materials. I haven’t worked with production-grade programmable robots such as the ones you mentioned, so I wouldn’t be able to offer any concrete advice.

  • Pedro

    Hi, If I have a stepper motor instead of a servo motor. What changes will I need make in the project?

  • Fadil

    hey Alex!
    can u pls tell me step by step how i can communicate with arduino????i am struggling a lot
    thank you!

  • Jomar

    sir! good day! do u know how to write a program that lets opencv compute and detect the location of an object and send commands to the motors via arduino to move the robot on that location? thx in advance! 🙂

    • Ethan Hunt

      Sorry, I don’t really have any code written that could help you with your project. This is very different than the project I’ve done here, so I don’t think you can reuse something other than some small code snippets related to hardware/software communication.

  • Gagan

    Hi, Your project is quite interesting. I would like to know if a program can be written in Arduino using Open CV libraries ? Is it possible to burn the program onto a Arduino controller thus the camera not needing to interface with a computer ?

    • Ethan Hunt

      Hmmm, interesting question. I have to admit I haven’t really looked into this myself, probably cuz I already had the USB webcam interfaced to my PC and didn’t want to bother. But now that you mentioned it, I got a few ideas myself which involve a camera directly connected to an Arduino Microcontroller.

      Well, obviously a regular USB webcam wouldn’t be able to interface with an Arduino directly due to drivers and stuff (at least not to my knowledge). It got me thinking that perhaps there’s a camera shield for the Arduino or something like that. A quick search revealed the following websites: http://www.arducam.com/tutorial/ and http://learn.adafruit.com/ttl-serial-camera/overview . It seems there are cameras which can connect to the Arduino or to a shield like I mentioned before.

      However, now there’s the question of using OpenCV libraries to do the processing. Personally, I was using the C++ flavor of OpenCV to do my processing which isn’t compatible with the Arduino. However, I believe OpenCV exists in a C variant as well, so perhaps that will work. I haven’t tried it myself, and I doubt it will work on my Arduino Uno due to the slow processor, and small amount of memory the micro-controller has. I just don’t think you’ll be able to fit an OpenCV program on it, yet alone have the power to process an image, not to mention a video stream. Perhaps some of the more powerful Arduino models might be able to do something. I really can’t say.

      It’s a very interesting question though, so I hope another reader jumps in to shed some light on the matter. Feel free to share if you find something else yourself. I might implement something like that in a future project.

      Long story short, my current thoughts are that I need a PC in order to process the complex video stream fast enough, and then send simple commands to the Arduino running the small simple program for servo control.

  • jairam naik

    hi,
    without serial communication can we track a object with arducam shield? I am looking code for object recognition in arducam shield and without using serial communication. if possible give reply to jairamsms2@yahoo.com

    • Ethan Hunt

      As I mentioned in the post just above yours, I haven’t done this myself, and I don’t really believe it can be done due to the limitations of the Arduino platform.

      I think that the micro-controller just doesn’t have the power to process video in real time. Additionally, most of the code I’ve seen for the arducam shield and Arduino cameras in general seem to only capture low resolution images, so this kind of confirms my original thoughts.

      Perhaps it might be able to capture one still image and do some low-level slow processing on it (using OpenCV in C), but real time video seems out of scope.

      Again, I haven’t experimented with this too much, so I can’t say anything for sure.

      • mafaz wali

        Sir ,

        your project is really interesting . What i want to do is , i want to detect and track and object same as you did .. i further want to make my robot follow the detected object as well(it should be able to go towards the detected object). The robot has two wheels and a free wheel. i am using two dc motors for the wheels.. plz guide me. thnx.

  • Anthony Stark

    Hello …am trying to do something similar to what u have done but using the x and y position. i am kinda of confuse cause am new to this environment of programming. did u only use the x coordinate in your code? how did u allocate it to the servo or was the x coordinates allocated to the “servoPosition” in ur codes? if i was to use the y co ordinates, do i edit the “servoOrientation”?

    • Ethan Hunt

      Awesome, Iron Man is reading my blog 🙂

      What I did was to get the x coordinate as a pixel position on the image. I then “translate” the pixel position into servo motor degrees. So if 90 degrees orientation is pixel 100, then if I move to pixel 110, I’ll move my servo to let’s say 95 or 100 degrees. Same goes for the other direction.

      Now, I’m not really sure what you’re doing that requires x and y. Could you explain a bit more, so maybe I could help you? It doesn’t make sense to me to apply x and y to one servo. Perhaps you have two servos, one for horizontal and another for vertical movement?

      Try to explain your setup as best as you can, and we’ll go from there 🙂

      • Anthony Stark

        sorry about that, i forgot to mention i was going to use 2 servos for the horizontal and vertical axis. i want to one servo to collect data for x and the other for the y. hope that explains better cause am finding a bit difficult trying to accomplish that. thanks
        FYI am no iron man lol

  • Sandeep Yadav

    Hi there,Hope you all doing well. Im doing that project for my final year project. Im doing electrical engineering so that is bit hard for me as it’s more relatively to electronics and computer science engineering. Can anybody help me please…how servo motors are communicating with Camera and Arduino board? Can we use other microcontroller for same application? why we choosed Arduino as a controller? is there anything that is related to electrical in this project. Thanks in advance…hope to hear from someone soon.

  • Suraj Nair

    Hey there! Thank you so much! Am working on something similar. My tracking code works fine. I’ve been trying to use your code for Serial communication. My arduino is connected via USB to my computer, and the port created is COM15. However, your code returns that the port does not exist. Any suggestions?

    • Ethan Hunt

      My code, as posted here, tries to connect to “COM3” as that’s what I was using. You can see this in the second line of the main method. Change the port to whatever works in your case. Let me know if it works out.

  • Nauman

    Where is Circuit diagram of this project ?

  • Masoud Salehi

    hi
    how can i determined blue HSV range for scalar() in, inRange() function?

    please help me .
    thanks for your favor

  • Mr.K

    Hi Ethan, thanks for sharing this nice article. I indeed wanna implement it, to improve our existing embedded application. Due to time constraint and my poor knowledge in hardware matters, I hope no hardware issue occur when implementing this. Can you please let me know which webcam and servo type you use (product’s web link)?, thus I would have identical environment as you did.

    One last question, do we need breadboard to wire servo to Arduino?

    Thanks in advance.

    • Ethan Hunt

      Hey there! The webcam I used is a Logitech C525. Descent performance, 720p and stuff, and still cheap enough. You can find it HERE (and you can leave me an Amazon kickback if you do get it :)) The servos are really ‘basic’ as far as performance goes, and I see them used in almost every Arduino project out there. The model is “TowerPro SG90 9G Mini Servo”. Take a look HERE. Finally, no, you don’t need a breadboard. The servo has 3 pins, and you just need to route them to VCC, GND, and data/signal. It’s easier with a breadboard as the servo connector has all 3 pins next to each other so you can’t directly connect it to an Arduino. But you can just cut off the connector and plug in the wires separately. Hope that helps! And thanks in advance in case you do end up using my Amazon links 🙂

  • Bushra

    Sir, I tried this amazing project of yours using the codes provided by you, but its not being able t detect my ball until its being brought very close to the cam , moreover its detecting multiple objects..
    Kindly help..

    • Ethan Hunt

      Hey there. Glad you found my project useful. The code itself is setup to detect all ball-like objects that appear on screen. However, when using it, I make sure to only have just one.

      I may be wrong, but poor detection is usually a result of either a poor (low quality/low resolution) camera, or poor lighting. What kind of camera are you using?

      You could try increasing the contrast between your ball object and your background. For instance, you could make sure to have a bright background (single-color works best) and bright clothes on (white shirt?), and have a dark ball. Also, try to have as much light as possible. Natural light works better compared to artificial (at least in my experience).

      The ball-finding algorithm looks for these edges in contrast, so do whatever you can to make the object easy to detect, and to avoid having any sort of multicolored backgrounds.

      Of course, the primary thing to check out is the camera itself. As I said before, if the lens quality and/or resolution (and even FPS) is not that great, you can’t really expect excellent results in any computer vision project.

      Hope that helps!
      Regards.

  • Sovatna

    Dear Ethan Hunt,

    Thanks for your kind sharing. I’m a new user so that I do not know well of even how to connect with all specification. May you please help make a schematic in detail. I’m really appreciate for your kindness.

    Sovatna,

  • Gunbir Singh

    Hi does our code is only for detecting the circles.The source code you have attached is for detecting which color.

  • ashish

    i want to send x,y cordinate of the ball to the arduino .
    how i can do serial communication between them to send position x,and position y , values ? can u provide me that program of serial communication for writing integer value on the serial poart.

  • deepthi

    plsss help me in installing opencv 2.4.10 using MS visual basic 2010 express in windows 8 32 bit.
    i couldnt debug any of the program.

  • ahmed

    Hey awesome work! Do you think it would be possible to minipulate the code to track and follow a person? I think it would be very useful for a project I am working on.

  • abdullah

    hey great work , I was wondering if I can use the same concept to track the sun in a solar cell tracking system using the ttl serial camera as my sensor.

    • Ethan Hunt

      I guess in theory you could. Although I imagine you would need a few camera filters to dim the sun’s effect on the camera. If you could filter out the sun so that it appears as a proper “ball” without bleeding light all over your camera sensor than the same code would be applicable. You would just need to modify the tracked color to whatever filtered color you’re actually seeing (purple, orange, yellow – depends on the filter).

    • Jai

      use photo resistors for tracking the sun no filter needed

  • James Bond

    I work on that project for my study and when I launch the code on Open Cv ( on my netbook) they said that the port are not oppened, the webcam are well plug to the USB port, do you have a solution ? please

  • sergey

    mobile versionis trecker (open cv).necessary to assemble such devices based on android and IOIO. help. http://www.youtube.com/watch?v=wwD0kkeY1uI

  • Berkay

    I LOVE YOU MAN!!!

  • Jai

    ethan i was on you channel and i posted but i have not heard back from you but hey where can i get this ?
    #include
    #include

    this is what i got from VS2013
    errors
    ERRORS: Error 4 error LNK2019: unresolved external symbol cvReleaseMemStorage referenced in function main

    Error 2 error LNK2019: unresolved external symbol cvInRangeS referenced in function main

    Error 5 error LNK2019: unresolved external symbol cvGetSeqElem referenced in function main

    Error 3 error LNK2019: unresolved external symbol cvCreateMemStorage referenced in function main

    Error 1 error LNK2019: unresolved external symbol cvCreateImage referenced in function main

    Error 6 error LNK2019: unresolved external symbol
    cvCircle referenced in function main

    Error 7 error LNK1120: 6 unresolved externals 1>Source.obj : error LNK2019: unresolved external symbol
    cvCreateMemStorage referenced in function main 1>Source.obj : error LNK2019: unresolved external symbol
    cvReleaseMemStorage referenced in function main 1>Source.obj : error LNK2019: unresolved external symbol cvGetSeqElem referenced in function main 1>Source.obj : error LNK2019: unresolved external symbol cvCircle referenced in function main

  • Tinashe

    Hi,

    I came across your site when searching for ways to move a camera in the x y directions in tracking human eyes (the focus is on designing an automated realtime Iris recognition system), i would say great work there. In my project, tthe openCV will be running on BeagleBone Black, a system that will combine the PC and Aduino you used. To control the two servo motors for x and y directions, i need to utilise the PyBBIO library together with using openCV. For me to achieve that, i intend to use Python for openCV and the PyBBIO library. The BeagleBone Black is running Debian.

    My question is on the feasibility of the project and can you assist me on that?

    Thank you

  • Mekaci

    Hi friend, I tried your code and it seems working fine and I believe that I set up and done everything correctly too; but my problem is ” I cannot see even myself from that screens ( from my webcam ) ” I get black screen only with it says ” port opened ” and working fine but no image on the screen. This is the screenshot :
    http://oi63.tinypic.com/w9fxqe.jpg

    would you help me please how to fix this issue ? thank you.

  • niran

    Sir i tried to build your code but i am getting a gray image in the camera output window and in procced output window ,please guide me

  • Joe

    hi
    when debug
    Original screen show gray color no video
    Processed screen black color
    http://imgur.com/qoO5JDY
    I don’t have any ideal
    Please help me
    thanks.

  • Den

    Sir, I also tried the code but same output, gray image in the camera window and black in the process window, please help us

  • sa

    Hi
    I’m A MSc Student, I need some advice from you
    I want to began with my project, but I don’t know how to began! any way, my project is very near from your camera tracking, my idea is about tracking a mobile target by 4 camera (each 1 located on the corner), but when this target move toward some direction, the far 2 camera should inter in safe mode (sleep or something else) but if the mobile target change it’s direction, these 2 camera shout locate it and record the direction and the speed as input data to my computer, and of course the 2 camera in the other side should inter to the sleep mode.
    awaiting for your reply

    s_chalabi@yahoo.com

    Yours
    Sa Cha

  • INDRIYA GALIH PRAYOGI

    Hi,
    Mr can you help me setting color to orange?

  • trebla

    hi…. can show a step by step of how to connect arduino to visual studio C++… quick response plz… i need for my thesis project…

  • Amit

    Sir, I have seen your project and I wanted a big help from you as I am also making a project in which the camera will process the image of a sign board and go to the particular direction provided by that sign board…….
    as you have given the codes for open cv and aurdino so sir how will I peform the task and what all software I need to download for this project

1 Trackback or Pingback for this entry

Leave a Reply