lookdown video analysis code

Having seen that the ardone is using a lookdown camera for stability, I though I would put together a simple bit of code to look at a video stream and find point of interest. They seem to have a grid of 25 points that they lock on to points of interest but I have just picked any 100 points. It is this high because each frame is processed separately with no attempt to find a tracking point in frame n +1 close to where a point was in frame n.

In my sample vid ( from micrsoft) it finds points OK and it can be seen that there is tracking , i.e. a point on the skier is followed across several frames. Video came from here http://support.microsoft.com/kb/119383 .

I have seen examples of using cameras for horizon estimation but I think the look down camera has great potential for drone stabilisation.

There are some built in functions in opencv that calculate so calledd optical flow and I think they combine feature detection across sequential frames. That might be the way to go

Any suggestions, alternative code, sample videos, etc would be welcome.

To run my code you need python2.6 and opencv ( I am on windows). You could change the code to use a usb camera but my laptop goes so slooooowwwww when I try to use the camera.

Views: 245

Attachments:

▶ Reply to This

Replies to This Discussion

Permalink Reply by OpticalFlow on January 10, 2010 at 7:29pm
Plenty of research has been done on using optical flow for stabilization of UAVs, google for some of Mandyam Srinivasan's publications to get initial pointers.

If you're using OpenCV: have a look at the fback (Farneback 2-frame dense optical flow) sample applications included in OpenCV 2.0. These are fairly fast and give dense optical flow. Another option is the Lukas Kanade tracker (lkdemo) but you likely won't have the resources to be running this on your UAV platform.

Also, there is a commercial product: Robbe's helicommand that uses a downward looking camera to hold position on RC helicopters. I wonder what sort of tracking algorithm they're using. There is also a cheaper, plagiarised version from China: the KDS Flymentor.
Permalink Reply by Diarmuid Wrenne on January 11, 2010 at 9:57am
Hi,

The KDS Flymentor looks to be a gyro based solution (from what I can see)

I had a look at some other opencv optical flow stuff. Internally they seem to use the cvGoodFeaturestotrack which i used in code. The bit I can't see is what assumption is made by opencv about the relationship between points . How does it know that a point at 15,50 in frame n is the same physical point as the pointy found at 155,55 in frame n+1. That's a discussion for the opnecv forum, is suppose.

Did you see the ar.drone tracking vid on youtube. Any idea what they are using for that?

http://www.youtube.com/watch?v=LFBtRPjnauo

Thanks

Diarmuid
Permalink Reply by Diarmuid Wrenne on January 11, 2010 at 12:02pm
just read this link.

http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/

I think this is closer to what ar.drone are doing. The link takes a grid applied to filter images . It then uses patch-matching to look for similiar areas in a section of the second image centered on grid point. It can be seen in the video that the point is only tracked by a particular grid point for a few frames even if the point is in the video for longer. It is taken over by a another grid point after a few frames. I can see why this is better. The cvGoodfeaturestotrack seemed very arbitary and inconsistent. I would prefer it to find a single very stroing point to track in a subsection of the image.
Permalink Reply by Alby on January 12, 2010 at 1:49pm
I wonder if you could use the same approach a CMOS/Laser mouse uses. I figure the smarts are on the mouse and it merely returns movement information?
Permalink Reply by Diarmuid Wrenne on January 13, 2010 at 4:05am
The VB optical mouse project is still a great hack.

http://spritesmods.com/?art=mouseeye

here is a more up to date one

http://www.planet-source-code.com/vb/scripts/ShowCode.asp?txtCodeId...

There are some arduino links too

http://www.bidouille.org/hack/mousecam/
Permalink Reply by Simon Wood on January 13, 2010 at 8:53am
Permalink Reply by Diarmuid Wrenne on January 13, 2010 at 9:43am
That solution looks very effective. It's not clear where the new lens was sourced from but I think that solution could be used as the basis for an IMU for small heli's with the proviso that the target has to be well lit.
Permalink Reply by Diarmuid Wrenne on January 11, 2010 at 10:28am
With a name like OpticalFlow I assume you know what you are talking about.!!!

Just thinking about the quadropter application, one of the features is that it can rotate on its Center of mass. I wonder how the optical flow would look. I assume that some sort of average of all the vectors is taken to determine actual movement. How would that look for a rotation .
Permalink Reply by OpticalFlow on January 11, 2010 at 4:52pm
There is a vast amount of literature out there on optical flow patterns for different motion and on things such as determining ego-motion of the camera from optical flow.

There is some very cool research on how insects use optical flow for flight control, in particular Mandyam Srinivasan's groundbreaking work on bees. Srini has also demonstrated hovering a helicopter with optical flow only. Locusts use the expanding pattern of optical flow vectors that you see when moving towards an object to calculate time to impact/collision with an obstacle. This has also been used in Micro-UAVs.
You may want to check out http://www.centeye.com/pages/techres/opticflow.html .


Helicommand (and KDS Flymentor) use gyros/acc for attitude stabilization but they have a downward-looking camera to do the position hold. I'm not sure whether they calculate optical flow or just an overall-displacement for the whole image. As to what the Parrot.AR drones uses I can only guess. From the video it looks like they're trying to track feature points at fixed positions (on a grid, to get optical flow information evenly throughout the image) plus some other good features to track elsewehere in the image (maybe Harris corners or something more sophisticated like SiFT, SURF or Colander features although the hardware constraints would be limiting the sophistication.)

ModeratorPermalink Reply by Chris Anderson on January 10, 2010 at 9:16pm
The AR.Drone combines the look-down optical flow with altitude data from the ultrasonic sensors, so it can scale the detected movement appropriately depending on height.
Permalink Reply by Andrew Dunlop on January 11, 2010 at 12:03am
Permalink Reply by Diarmuid Wrenne on January 11, 2010 at 4:03am
Thanks

Those links look great. Here is an academic one that looks good.

http://www.mil.ufl.edu/~nechyba/mav/

They do the processing on the GS.

RSS

Contests

Season Two of the Trust Time Trial (T3) Contest has now begun. The third round was a reliablilty/aerial photography round for both planes and copters, which is now closed. Stay tuned for the next round, beginning soon.

A list of all T3 contests is here. 

 

Top Content 

1 What do you think the APM should do on fail safe

What do you think the APM should do on fail safe
2 ArduCopter 2.5 released!

ArduCopter 2.5 released!
3 Brainstorming Specifications.

Brainstorming Specifications.
4 Housekeeping note: Kevin Finisterre

Housekeeping note: Kevin Finisterre
5 Hexa-copter found after failed attempt of delivering cell phones to inmates of a Brazilian penitentiary

Hexa-copter found after failed attempt of delivering cell phones to inmates of a Brazilian penitentiary

Groups

© 2012   Created by Chris Anderson.

Badges  |  Report an Issue  |  Terms of Service