Vision / optic flow sensors for drones- Can you give me a wish list?
Thank You everyone who came here after reading my blog post from today (and Thank You to Chris Anderson for providing a forum like this one!).
At Centeye, we've been developing optical flow / vision sensors and integrating them on RC aircraft test beds for years now. Check out our website for details if you wish. In 2001 we obtained altitude hold on a fixed-wing aircraft (a foam Wingo, remember those?) using an early "Ladybug" sensor, and later in 2003 basic obstacle avoidance using a scratch-built aircraft. Some of this work was described in a Wired Online article way back in 2004. Ever since way back then many people have expressed an interest in these sensors.
Now, partially due to our participation in the Harvard RoboBee project and partially from our own funds, we are in a position to develop a portfolio of academic/experimentalist/consumer class sensors. The current batch of silicon vision sensors (yes- we design our own silicon!) is currently in fabrication, and should be finished sometime in February.
I would like to find out whether people here would be interested in such sensors. More important, what would you like? I know, it is a very broad question, but I would like to have this discussion.
For our first version, I am envisioning a true minimalist design using one of our imagers and an Atmel AVR for processing. We could probably make the whole thing weigh less then a gram. If fully functional, the sensor will operate off a single LiPo battery- the vision sensor will have it's own integrated linear regulator. The AVR itself can be programmed with different firmware implement a different sensor using the same piece of hardware. I envision this being useful for air vehicles, ground vehicles, and non-robotic uses as well (sensor networks? home automation?)
The interface will likely be via I2C or SPI- I am partial to I2C because it allows multiple sensors to be hooked up on the same four-wire bus (ground, power, clock, data) but SPI is faster.
I'll start with this. I have more thoughts which I'll add on based on interest- I'm hoping all of you wear me out here!
At Centeye, we've been developing optical flow / vision sensors and integrating them on RC aircraft test beds for years now. Check out our website for details if you wish. In 2001 we obtained altitude hold on a fixed-wing aircraft (a foam Wingo, remember those?) using an early "Ladybug" sensor, and later in 2003 basic obstacle avoidance using a scratch-built aircraft. Some of this work was described in a Wired Online article way back in 2004. Ever since way back then many people have expressed an interest in these sensors.
Now, partially due to our participation in the Harvard RoboBee project and partially from our own funds, we are in a position to develop a portfolio of academic/experimentalist/consumer class sensors. The current batch of silicon vision sensors (yes- we design our own silicon!) is currently in fabrication, and should be finished sometime in February.
I would like to find out whether people here would be interested in such sensors. More important, what would you like? I know, it is a very broad question, but I would like to have this discussion.
For our first version, I am envisioning a true minimalist design using one of our imagers and an Atmel AVR for processing. We could probably make the whole thing weigh less then a gram. If fully functional, the sensor will operate off a single LiPo battery- the vision sensor will have it's own integrated linear regulator. The AVR itself can be programmed with different firmware implement a different sensor using the same piece of hardware. I envision this being useful for air vehicles, ground vehicles, and non-robotic uses as well (sensor networks? home automation?)
The interface will likely be via I2C or SPI- I am partial to I2C because it allows multiple sensors to be hooked up on the same four-wire bus (ground, power, clock, data) but SPI is faster.
I'll start with this. I have more thoughts which I'll add on based on interest- I'm hoping all of you wear me out here!
Replies to This Discussion
-
Permalink Reply by Matt on -
Your first version sounds perfect and wow that is lightweight. The main factor which would determine its success would be the software interface I think. It needs to be easy to implement. Could a sensor of this kind output a stream of forward & back translation, side to side translation and altitude values (translational movements in the xyz planes) for simple connection to a flight controller board or do you need a 360 degree ring of cameras to do this?
I can envisage something like this in an outdoor quadcopter eliminating the need for GPS position hold and aiding the barometric pressure hold. It could fight against wind gusts just like a hoverfly.
Wishlist: An optical flow system designed for a quadcopter/multicopter weighing about 10g with larger lens based camera which outputs translation in the xyz planes, updating at ~50Hz. I wonder if you could use high quality mobile phone cameras for this?
-
Permalink Reply by Geoffrey L. Barrows on -
One of the challenges with using (passive) vision systems is that what you get at the imager is fundamentally an angular measurement once you've projected pixels back out to the visual field. Optical flow measurements become "degrees per second" or "radians per second" rather than "meters per second" and thus accumulated optical flow becomes a similar angular measurement. At the most basic level, OF = (v / d) * sin(theta) where OF is optical flow, v is relative velocity, d is distance, and theta is the angle between the velocity vector and the direction in which you are looking. If you are flying parallel to the ground and looking down, then theta=90 deg and the sin(theta) term drops out. To get linear x/y/z rate measurements, we'd either need v or d using some distance measuring method, such as stereo, ultrasound (as in the Parrot AR.Drone), or something else. One possibility is to use feature tracking and SFM (structure from motion) or SLAM-type techniques, but that is "probably" a bit much for this current iteration using an 8-bit AVR! :) Let me think about this.
A 360 degree ring can tell you about direction of heading and linear translation along those axes, but normalized to the distance to the surrounding objects. A 100m/sec airplane flying 100m above the ground will see the same OF as a 10cm/sec fruit fly traveling 10cm above the ground.
-
Permalink Reply by Geoffrey L. Barrows on
-
Thank You. Regarding holding a position, right now we can do this fairly well inside. Outside in the wind is currently a different story but we haven't worked on that yet and I agree it would be useful.
One thing I am curious about in general here is whether people want a turnkey system that they plug into the vehicle and it works (such as the FMA copilot systems) or more basic sensors that they'd want to hack and play with. I imagine there is desire for both.
-
Permalink Reply by Randy on -
I think your idea of the minimalist design sounds like a good idea. It gives people the flexibility to integrate the information in with whatever other sensor data they have to come up with a complete picture.
I also vote for the I2C interface. If you keep a cache of say the last 32 values that might make it more convenient for CPUs to come and get the values at a time that's convenient for them.
and keep it cheap! especially if we need to buy 9 of them!
-
Permalink Reply by Geoffrey L. Barrows on
-
How cheap? :) I do understand where you are coming from.
In addition to storing those 32 values, with just a few more cpu cycles it is often possible to constantly compress that into the single value that you really need.
I'm going to be very open here: Whenever I do a spreadsheet to estimate how much it will cost to do this, the biggest cost item is also the most variable and unknown- tech support! We really need to make these things super "easy to (ab)use"!
-
Permalink Reply by Randy on
-
Almost all sensors seem to cost $50 so you can go up from there. Go above $200 though and I think most hobbyists will shy away from it.
-
Permalink Reply by Geoffrey L. Barrows on
-
That's consistent with what I hear a lot from others.
It would be nice if in a few years you can go on Digikey and get one for $5.
-
Permalink Reply by Lorentz on -
Hello Geoffrey
Great work! As far as the wishlist is concerned, I'd like to use it on a fixed wing aircraft for
1) altitude hold
2) obstacle avoidance
3) autonomous landing
I would not care about it being a turnkey system. I prefer to tinker with things, but it's my personal view. A 50$ price per sensor could be OK if a single sensor allows to do at least one of the three functions above. I've some question about the inner working:
a) what kind of signal does it output ? x / y component of OF ? is it an analogic or digital signal ?
b) does it assume the optical flow is uniform across the whole image as do the sensors inside optical mices ?
c) what happens if there are different flows in the seen image ? (different directions / different speeds)
d) what happens if the sensor is looking at a fixed (sufficiently textured) image and it turns around the axis perpendicular to the image plane ?
e) (similar to question c)) what happens if the sensor is "walking" down a corridor, i.e. it sees flows with different directions on both sides of the image ?
My congratulations again and sorry for boring you with these question.
Thank you
-
Permalink Reply by Geoffrey L. Barrows on
-
It sounds like you have experience with optic mice sensors. The output is digital. Also the output depends on the specific firmware in the microcontroller- In the past we've used a program that assumes uniform optic flow, and we've also divided up the field of view into multiple regions each with it's own measurement. We've also in the past incorporated different types of algorithms running in parallel. We can pretty much do anything subject to the limitations of the microcontroller. Regarding d)- if the visual field is divided into multiple regions then regions towards the periphery would sense more translation than rotation. We also have algorithms that directly sense rotation, but have not yet flown them.
PS. These are good questions
-
Permalink Reply by Lorentz on
-
Thank you Geoffrey. It seems a great sensor to me. Looking forward to know the price target.
-
Permalink Reply by Doug Weibel on -
I think these sensors could be very useful for accomplishing lots fo UAV tasks. I would be interested in them for a variety of airframe platforms. For inexpensive near term sensors my priorities would be -
Fixed wing functions: Heading hold for auto-takeoff, pattern recognition (runway) and spot tracking for auto-land
Quad/helicopter functions: OF for stationkeeping
I would prefer a platform that allows tweaking.
Top Content
Groups
-
ArduCopter User Group
478 members
-
ArduPlane User Group
299 members
-
ArduCopter (traditional …
77 members
-
ArduPilot (legacy) User …
77 members
-
Moderators
50 members
-
ROS User Group
31 members
-
Team Prometheus
11 members
© 2012 Created by Chris Anderson.
