Transforming Graphics for All
Pradeep Sen's research on computer graphics, visualization, and gaming
Soon the spectacular special effects of blockbuster movies won’t be just the province of George Lucas or the wizards at Pixar. You'll be able to create them, too. And the exquisite lighting, stunning sharpness, and perfect composition of an Ansel Adams photograph will be at your fingertips.
These concepts are part of Pradeep Sen’s vision for the future of filmmaking, photography, and more. Sen, assistant professor of electrical and computer engineering, is working with a team of students to advance computational photography, real-time rendering, and the visualization of complex data to develop more efficient, effective ways to capture and experience our vision of the world.
Professional Photos for Everyone
Sen and his students are harnessing the power of online digital photography databases to help ordinary people take better pictures. Imagine a photograph of a child against a drab background and a separate picture of a beautiful landscape. What if you could blend the two into one eye-catching new photo? Or suppose you want to take a photograph of a famous landmark but crowds ruin the scene. A tool that could erase the crowd and fill in the holes accurately would be handy.
Current technologies help make those changes, but they’re not easy to use and the results aren’t seamless. That's where Sen's research comes in. His team is developing algorithms that leverage online photo databases to correct lighting and find information to fill the empty spaces in photos where objects have been erased. Professionals use similar tools; Sen plans to make them available to everyone. "What could you do if you had a magic camera? We want to empower everyone with automated tools to help them produce professional results," he says. The final software would have applications beyond photography, including enhancing medical imaging, complex data analysis, and architectural rendering.
Sen works in computational photography, an emerging field within computer graphics where digital images, new camera technologies, and massive computation are integrated to produce new kinds of images. He notes that the computational photography approach is fundamentally different from traditional image processing tools that modify photographs using pixel information. "Instead of looking only at the information contained within the pixels of a single image and modifying them, we look at the system as a whole," he explains. "“We take into account the camera, the lighting, online image databases, and more so we can leverage all of that data to generate the best picture possible."
Shedding New Light on Relighting
In other research, Sen and his team studied the physics of light to address the challenge of image-based relighting. Changing the lighting of a scene, or relighting, is needed when images and footage are captured under different lighting conditions than the final desired product. Relighting involves acquiring the reflectance function, the function that measures the way the light strikes an object and how it is transformed by the scene into outgoing light. The entire process is a time consuming, data intensive effort.
Sen addressed the relighting challenge with his research on a new kind of imaging technique called dual photography. The process harnesses Helmoltz Reciprocity, the physical law that states light pathways are the same regardless of the direction the light is flowing, to accelerate acquiring the reflectance function. Sen demonstrated that the flow of light emitted from a projector is symmetric in both directions and could be reversed, effectively transforming the projector into a virtual camera. "Dual photography allows us to use a projector to both emit and capture light," says Sen. "It's a much more efficient way to capture the data than a system based on multiple projectors and one camera."
In his latest research, Sen is accelerating the acquisition of the reflectance function even further. He is applying ideas from compressed sensing, which uses new algorithms to compress the information in a signal so that it can be captured more efficiently. Sen's approach requires only a small set of simple black-and-white patterns to do the same task that initially required complex illumination patterns and sophisticated processing. The simplicity of this approach makes it more practical for industrial applications, including use in a real movie studio. "We've taken ideas from physics and applied mathematics to create a novel framework that accelerates light transport acquisition. It could be a new way to solve many graphics problems," explains Sen.
Visualizing New Opportunities
Sen is also helping the next generation of engineers develop diverse graphics and visualization skills. Since joining UNM in 2006, Sen has expanded the graphics curriculum and helped start the Interdisciplinary Film and Digital Media Program, a joint undergraduate program offered by the School of Engineering and the College of Fine Arts. Now in its second year, the IFDM offers core classes and electives that give students a breadth of technical and creative skills, which they can apply to innovative digital technologies in the real world.
"Engineering students that come out of this program won’t just be good at graphics or game development," says Sen. "I want them to be good engineers who can think in bigger terms so they can work at Intel and Microsoft just as easily as for game development companies or special-effects houses."
Soheil Darabi, a doctoral student in computer graphics and computer engineering, is just one of the students taking advantage of the new opportunities at UNM. He works closely with Sen on compressing sensing research. "With this application based research, I’m learning the importance of being very precise and being conscious of even the smallest details," he says. "In research, and this industry, you come across new kinds of problems all the time. I’m learning the problem solving tools that I’ll need for those challenges."
Sen, along with Computer Science Assistant Professor Joe Kniss, has also started the UNM Advanced Graphics Lab within the School of Engineering. The AGL is a laboratory dedicated to the technical side of computer graphics, digital media, and visualization. It offers courses and research projects for students interested in working with computer graphics, especially the different facets of game development. "This unique set of opportunities will give our students real-world experience as well as some exposure which they can use in their resume when looking for a job after graduation," says Sen.
One AGL class is an introductory computer graphics class requiring small groups of students to develop a video game for the Xbox gaming console. The team designs the game story, programs it, and works with artists to develop the scenery and music or develops them on their own. Unlike many final projects that are never seen after the end of the semester, Sen plans to distribute the games through Xbox LIVE so gamers around the world can play them.
This past spring, eight teams produced Xbox games and a handful were selected to go through the licensing process. Working with STC.UNM, the corporation that patents and markets intellectual property at the university, Sen and the students are navigating the legal issues so the games can be launched to Xbox subscribers. Several of Sen’s students received awards for their copyrights during the STC.UNM Creative Awards event in April.
Sen says the game development project is good exposure for students and the university. "When we put these games on the Xbox console, we’re connecting with a whole new audience that might not learn about engineering and what we're doing at UNM. If kids play the games and see that UNM students created them, they might be inspired to consider engineering and game development as a career."
Working on Xbox games is just the first step. Sen is planning to expand the curriculum to other technologies, including the iPhone. As he continues his cutting edge research and the educational opportunities expand for students interested in computer graphics and visualization, UNM engineers will soon bring better, bolder visions of our world to big and small screens near you.