A new study led by electrical and computer engineering professor Viktor Gruev details how the new camera works with tumor-targeted drugs to see cancer in animal and human patients. The study is published in the journal Science Translational Medicine.
“Engineers spend incredible amounts of time and money developing the image sensors in cellphones,” said Gruev, who also is affiliated with Carle Illinois College of Medicine. “When we are out on the town, these devices can capture pictures that are perfect for social media, but when doctors are examining patients, they don’t care how nice the shot looks – they care how well the picture captures reality. The driving force in the camera market is simply incompatible with the technology required for medical diagnostics.”
Looking for a better way forward, the researchers turned to the compound eye of the mantis shrimp.
“The mantis shrimp has these incredible eyes,” said Steven Blair, an electrical and computer engineering graduate student and the lead author of the study. “Humans perceive three colors – red, green and blue – because of a single layer of light-sensitive cone cells that line our retina, but the mantis shrimp perceives upward of 12 colors thanks to the stacks of light-sensitive cells at the tip of its eye. The mantis shrimp’s little eyes can thus see things that humans can’t imagine – and do so in a fraction of the space.”
While the human eye perceives three colors – red, green and blue – the mantis shrimp perceives upward of 12 colors thanks to the stacks of light-sensitive cells at the tip of its eye.
Graphic courtesy Steven Drake/Beckman Institute for Advanced Science and Technology
In the hopes of replicating this visual system in a single imaging device, the team integrated advanced semiconductor devices and specialized optical filters. The technology can capture the three colors of visible light that a doctor would normally see as well as three colors of invisible near-infrared light that the doctor would miss. It can then be paired with multiple tumor-targeted probes that accumulate in cancerous tissue and emit near-infrared light, permitting the doctor to see exactly where the tumors are located in a patient. These probes are just now entering medical markets.
Click here to see a video explaining this research.
“The combination of this bioinspired camera and emerging tumor-targeted drugs will ensure that surgeons leave no cancer cells behind in the patient’s body,” said Goran Kondov, a professor and chief surgeon from North Macedonia who demonstrated this technology in the operating room. “This additional set of eyes will help prevent recurrence of the disease, providing patients a quicker and easier path to recovery. And the device can potentially be manufactured at low cost since it is so simple, making it accessible to hospitals around the world. “
The next step for the team is the integration of the camera with endoscopic systems to satisfy the demands of minimally invasive surgeries in resource-limited hospitals.
The Air Force Office of Scientific Research, the U.S. Department of Defense, the National Science Foundation and the Office of Naval Research supported this research.
Gruev also is affiliated with the Beckman Institute for Advanced Science and Technology, theCancer Center and bioengineering at Illinois.
Cancer lurking in tissue could be more easily found when looking through a butterfly’s eye.
]]>Beneath the water’s surface lays a hidden world: one that cannot be perceived by the human eye. When viewed through a special camera, however, rich polarization patterns are unveiled. These patterns can be used as an alternative approach to geolocation- the process of determining the geographic position of an object.
By: Amber Rose
University of Illinois Urbana-Champaign researchers have developed a novel method for underwater geolocalization using deep neural networks that have been trained on 10 million polarization-sensitive images collected from locations around the world. This new study, led by electrical and computer engineering professor Viktor Gruev, along with computer science professor David Forsyth, enables underwater geolocalization using only optical data while providing a tool for tethered-free underwater navigation.
These findings were recently published in the journal eLight.
“We are showing for the first time, you can geolocate yourself, or a camera, in a number of different conditions, whether in open ocean waters, clear waters or low visibility waters, at day, at night, or at depth,” says Gruev. “Once you have a sense of where you are, then you can start exploring and use that information to have a better understanding of the underwater world or even how animals navigate.”
Gruev explains that one of the main challenges with underwater navigation and geolocalization is that GPS signals cannot penetrate water- they bounce off the surface. “We are blind in terms of GPS signals underwater. We need to use different means and different technology for geolocating underwater.”
The current standard for geolocalization is using acoustic information, mainly obtained by sonar technology. This works by deploying many little sonar beacons which send signals that are triangulated to locate an object underwater. The issue, however, is that sonar only works in a small, defined area, while also being limited by its accuracy.
Another method currently utilized is using submersibles that are tethered to a larger vessel above the surface that has a GPS signal. Although the submersible can maneuver a little bit, it is ultimately limited by the movement of the vessel.
“It is an incredibly challenging problem to have a free-moving, underwater vehicle. The way we solve this problem is by developing specialized cameras and machine learning algorithms. By combining those, we can actually figure out the sun’s location and this is where polarization imaging comes into play.”
–Victor Gruev
Light waves from the sun move in all directions-it is unpolarized. When those waves pass through a filter, like the water’s surface, they are forced to move in only one direction-the light has been polarized. Polarization patterns are the result of light’s transmission from the air to the water and scattering by water molecules and other particles. The patterns underwater change throughout the day, and they depend on the location of both the observer and the sun. By analyzing these patterns alongside accurate date and time information, it is possible to then determine location.
The team collected ~10 million images with an underwater camera and an omnidirectional lens capable of recording the polarization patterns from four sites: a freshwater lake in Champaign, IL (visibility around 0.3 m), coastal sea waters in Florida Key, FL (visibility around 0.5-3 m), sea water in the bay of Tampa, FL (visibility around 0.5 m), and a freshwater lake in Ohrid, North Macedonia (visibility exceeding 10 m). Images were taken in a variety of conditions (clear vs murky waters), depths, and times of the day- even at night when underwater light intensity is significantly weaker.
“We think of life as dull if we can’t see anything, if we cannot see the hands in front of us. But if we can see polarization properties of light, we can geolocate, even in muddy waters. And actually, life is pretty rich in terms of polarization,” Gruev says.
These images were used to train a neural network-a method of artificial intelligence to learn and improve accuracy over time. “The way we did this was to collect 10 million images of the sun from underwater,” Forsyth explains. “Each image was tagged with where it was taken, and the elevation of the sun. Those images were then passed into a learning system and the system was adjusted until it gave a precise location.” Using these machine learning techniques has helped improve location accuracy to 40-50 km, with the possibility of improving accuracy even further.
This technology presents new opportunities for people and robots to navigate underwater. Oceans account for over 70% of Earth’s surface area, yet very little is known about it. Data we do know about these bodies of water comes from monitoring via satellites 20-30 miles above the surface. In situautonomous sampling robots could provide more precise monitoring of water properties such as water temperature, salinity, oxygen levels and other related parameters.
The recent OceanGate Titan submersible search and rescue efforts have highlighted the need for accurate geolocation abilities. In order to locate the submersible at any possible depth, efforts were split into two distinct regions, near the ocean surface and near the seafloor, due to the limitations of current technology. Deep water efforts are significantly more challenging than near-surface operations, which have more technological options, and rely mainly on sonar. Not only is sonar unreliable over a large area, but it also often creates echoes that conceal an object’s precise location. Gruev says, “This polarization imaging technology will enable smaller autonomous robots to roam around the first 200-300 meters where light penetrates into the water and where our technology works very well and can help during search and rescue missions.”
“It is difficult to understand just how big the oceans are, how much water there is, how far away from anything you can be, and how hard it is to find anything out there. The biggest technological problem, up until the early 19thcentury, was simply knowing where you were at sea. And it remains really, really difficult.”
–David Forsyth
Viktor Gruev is also an affiliate of Beckman Institute for Advanced Science and Technology, the Department of Bioengineering and the Carle Illinois College of Medicine at UIUC.
Other contributors to this work include Xiaoyang Bai (Department of Electrical and Computer Engineering at UIUC), Zuodong Liang (Department of Electrical and Computer Engineering at UIUC), Zhongmin Zhu (Department of Electrical and Computer Engineering at UIUC) and Alexander Schwing (Department of Electrical and Computer Engineering and Department of Computer Science at UIUC).
This research was funded by the Office of Naval Research and U.S. Air Force Office of Scientific Research.
“Polarization-Based Underwater Geolocalization with Deep Learning”
There are many creatures on our planet with more advanced senses than humans. Turtles can sense Earth’s magnetic field. Mantis shrimp can detect polarized light. Elephants can hear much lower frequencies than humans can. Butterflies can perceive a broader range of colors, including ultraviolet (UV) light.
Inspired by the enhanced visual system of the Papilio xuthus butterfly, a team of researchers have developed an imaging sensor capable of “seeing” into the UV range inaccessible to human eyes. The design of the sensor uses stacked photodiodes and perovskite nanocrystals (PNCs) capable of imaging different wavelengths in the UV range. Using the spectral signatures of biomedical markers, such as amino acids, this new imaging technology is even capable of differentiating between cancer cells and normal cells with 99% confidence.
This new research, led by University of Illinois Urbana-Champaign electrical and computer engineering professor Viktor Gruev and bioengineering professor Shuming Nie, was recently published in the journal Science Advances. Both Gruev and Nie are affiliates of the Cancer Center at Illinois.
Small Variations
“We’ve taken inspiration from the visual system of butterflies, who are able to perceive multiple regions in the UV spectrum, and designed a camera that replicates that functionality,” Gruev says. “We did this by using novel perovskite nanocrystals, combined with silicon imaging technology, and this new camera technology can detect multiple UV regions.”
UV light is electromagnetic radiation with wavelengths shorter than that of visible light (but longer than x-rays). We are most familiar with UV radiation from the sun and the dangers it poses to human health. UV light is categorized into three different regions—UVA, UVB and UVC— based on different wavelength ranges. Because humans cannot see UV light, it is challenging to capture UV information, especially discerning the small differences between each region.
Butterflies, however, can see these small variations in the UV spectrum, like humans can see shades of blue and green. Gruev notes, “It is intriguing to me how they are able to see those small variations. UV light is incredibly difficult to capture, it just gets absorbed by everything, and butterflies have managed to do it extremely well.”
The Imitation Game
Humans have trichromatic vision with three photoreceptors, where every color perceived can be made from a combination of red, green and blue. Butterflies, however, have compound eyes, with six (or more) photoreceptor classes with distinct spectral sensitivities. In particular, the Papilio xuthus, a yellow, Asian swallowtail butterfly, has not only blue, green and red, but also violet, ultraviolet and broadband receptors. Further, butterflies have fluorescent pigments that allow them to convert UV light into visible light which can then be easily sensed by their photoreceptors. This allows them to perceive a broader range of colors and details in their environment.
Beyond the increased number of photoreceptors, butterflies also exhibit a unique tiered structure in their photoreceptors. To replicate the UV sensing mechanism of the Papilio xuthus butterfly, the UIUC team has emulated the process by combining a thin layer of PNCs with a tiered array of silicon photodiodes.
PNCs are a class of semiconductor nanocrystals that display unique properties similar to that of quantum dots—changing the size and composition of the particle changes the absorption and emission properties of the material. In the last few years, PNCs have emerged as an interesting material for different sensing applications, such as solar cells and LEDs. PNCs are extremely good at detecting UV (and even lower) wavelengths that traditional silicon detectors are not. In the new imaging sensor, the PNC layer is able to absorb UV photons and re-emit light in the visible (green) spectrum which is then detected by the tiered silicon photodiodes. Processing of these signals allows for mapping and identification of UV signatures.
Healthcare and Beyond
There are various biomedical markers present in cancerous tissues at higher concentrations than in healthy tissues—amino acids (building blocks of proteins), proteins, and enzymes. When excited with UV light, these markers light up and fluoresce in the UV and part of the visible spectrum, in a process called autofluorescence. “Imaging in the UV region has been limited and I would say that has been the biggest roadblock for making scientific progress,” explains Nie. “Now we have come up with this technology where we can image UV light with high sensitivity and can also distinguish small wavelength differences.”
Because cancer and healthy cells have different concentrations of markers and therefore different spectral signatures, the two classes of cells can be differentiated based on their fluorescence in the UV spectrum. The team evaluated their imaging device on its ability to discriminate cancer-related markers and found that is capable of differentiating between cancer and healthy cells with 99% confidence.
Gruev, Nie and their collaborative research team envision being able to use this sensor during surgery. One of the biggest challenges is knowing how much tissue to remove to ensure clear margins and such a sensor can help facilitate the decision-making process when a surgeon is removing a cancerous tumor.
“This new imaging technology is enabling us to differentiate cancerous versus healthy cells and is opening up new and exciting applications beyond just health,” Nie says. There are many other species besides butterflies capable of seeing in the UV, and having a way to detect that light will provide interesting opportunities for biologists to learn more about these species, such as their hunting and mating habits. Bringing the sensor underwater can help bring a greater understanding of that environment as well. While a lot of UV is absorbed by water, there is still enough that makes it through to have an impact and there are many animals underwater that also see and use UV light.
***
Viktor Gruev is also an affiliate of Beckman Institute for Advanced Science and Technology, the department of bioengineering and the Carle Illinois College of Medicine at UIUC.
Shuming Nie is also an affiliate of Beckman Institute for Advanced Science and Technology, and the department of electrical and computer engineering.
Other contributors to this work include Cheng Chen (department of electrical and computer engineering at UIUC), Ziwen Wang (department of bioengineering at UIUC), Jiajing Wu (College of Engineering and Applied Sciences at Nanjing University, China and School of Chemistry and Chemical Engineering at Yangzhou University, China), Zhengtao Deng (College of Engineering and Applied Sciences at Nanjing University, China), Tao Zhang (College of Engineering and Applied Sciences at Nanjing University, China), Zhongmin Zhu (department of electrical and computer engineering at UIUC), Yifei Jin (department of electrical and computer engineering at UIUC), Benjamin Lew (department of electrical and computer engineering at UIUC), Indrajit Srivastava (department of electrical and computer engineering at UIUC) and Zuodong Liang (department of electrical and computer engineering at UIUC).
This research was funded by the U.S. Air Force Office of Scientific Research, Office of Naval Research, the National Science Foundation, the National Institute of Health, and the University of Illinois Institutional Funds.
Video provided by the Cancer Center at Illinois Communications team.
ave designed an imaging sensor inspired by the visual system of mantis shrimp. These flamboyant creatures are capable of capturing 16 spectral channels and 6 polarization channels. We have mimicked the polarization sensing capabilities by combining metallic nano structures with an array of imaging elements. This bio-inspires polarization sensor can capture intensity and color information in high resolution (4 Mpixels) which has enabled many applications ranging from: early cancer detection in the gastric track to underwater imaging applications.
Checkout our papers related on this topic.
]]>