Please note that JavaScript and style sheet are used in this website,
Due to unadaptability of the style sheet with the browser used in your computer, pages may not look as original.
Even in such a case, however, the contents can be used safely.
Volume 80 (Sep 24, 2008)
Restoring out-of-focus images with super-resolution technology without additional image data
The proliferation of 1seg terrestrial digital broadcasting, digital cameras and more has not only made video and still images an integral part of our lives but has also led to growing needs for improved resolution. NEC Electronics, in collaboration with NEC Laboratories, has developed a super-resolution technology capable of sharpening out-of-focus images or smoothing rough edges in low-resolution video and still images when they are enlarged for display. Here we will take a closer look at the secret behind this new technology.
The advent of a technology that delivers clearer images
User needs such as the desire to view 1seg terrestrial digital broadcasts (a mobile terrestrial digital audio/video and data broadcasting service used in Japan and Brazil), digital still camera images and old video footage with higher-quality resolution are rising amidst the advancement of digitalization of every device imaginable. Just as these needs were beginning to rise, NEC Electronics announced that it had successfully developed a technology capable of meeting them. As a representative of ordinary users, I set out to determine to what extent this technology could actually improve resolution, and went straight to the developers themselves to get the scoop on this new technology.
Creating clearer images easily and inexpensively
This technology is super-resolution technology jointly developed by NEC Electronics and NEC Laboratories. In addition to sharpening out-of-focus images and smoothing rough edges by analyzing and processing information contained in one frame of data, this technology refines the contours of people and objects during the image magnification process, thereby compensating for image quality and enhancing color rendition for crisp images. Simply put, it is a technology used to sharpen out-of-focus images. "Our primary focus was image quality," says Yoshihiro Matsuura, manager in charge of marketing. Comparing the images below of a photo as it appears prior to and following use of this technology makes this very evident. The out-of-focus marbles from the first image appear to be more transparent in the second image, and the letters in the background are notably more distinct and clearer in appearance. I also had the opportunity to take a look at a few color and monochrome images, and the contours and details of the subjects in those photographs were very crisp as well.
Adoption of this technology enhances image data in 1seg terrestrial digital broadcasts by five times, from QVGA resolution (320 x 240 pixels) to WVGA (800 x 480 pixels), for display of clearer images on mobile phones and car navigation systems. Use of this technology also makes it possible to achieve crisp images in 1920 x 1080-pixel HGTV broadcasts by boosting image data in the 640 x 480-pixel VGA format ordinarily used for TV broadcasts and DVD storage to six times the resolution. But rather than actually increasing resolution itself, super-resolution technology only serves to improve the quality of resolution, thereby rendering it impossible to reproduce collapsed images.
According to Matsuura, "Clearer images can be achieved easily and inexpensively." That is the concept behind this technology. There already exists an algorithm for improving image quality by calculating data between successive multiple frames, but since the processing load is heavy and a large amount of memory resources is required, it is difficult to use this algorithm to create hardware capable of real-time processing. The greatest feature of this newly developed technology is that it enables super-resolution processing to be performed in real time with just one frame of image data. And since a large volume of external memory is unnecessary, use of this technology can also lead to lower costs. Moreover, modification of system configurations is unnecessary and connection to existing systems is simple. The biggest advantage of this technology is therefore its ability to reduce cost and simplify connection to existing embedded systems.
The great challenge of determining how to take this multiple-frame technology and transform it into a single-frame technology
Development of this single-frame super-resolution technology began in 2007. Amidst the proliferation of 1seg devices such as mobile phones in Japan, one of the issues that soon came to light was the roughness of images that occurred due to the previously mentioned resolution gap between image sources and display screens. Meanwhile, in overseas markets where many users connect their mobile phones to TVs or LCD monitors to view text messages or games, the roughness of images enlarged for display on such screens had become an issue. The search for a way in which to deal with these issues was what led to the development of this new technology.
Ordinarily, bilinear or bicubic technologies that interpolate pixels—for which shortages arise during the enlarging process—are used to display enlarged images. Sharpness processing of enlarged images is typically carried out via sharpness processing; however, with ordinary sharpness processing, noise is emphasized and quasi-contours are more likely to occur. Three years ago, NEC Electronics began collaborating with NEC Laboratories to develop a multiple-frame super-resolution technology capable of improving resolution by analyzing information from successive multiple images and correcting/restoring images between multiple frames. However, due to the sheer volume of analysis data, external components such as large-capacity memory are a necessity. This not only means increases in mounting area size, cost and power dissipation, but also increased difficulty in terms of real-time processing, thereby presenting numerous challenges in terms of embedding the technology in mobile phones and other consumer devices (Figure 1). This is what led this team of developers to begin considering whether or not it would be possible to create a processing method with which clearer images could be achieved with less analysis data by processing image data from one frame rather than multiple frames.
According to Ryoji Yanase, in charge of core and evaluation board development, "Although we officially started development back in September 2007, our endeavor turned out to be a continuing process of trial and error." Using the algorithm newly developed at NEC Laboratories as a base, the developers paid particular attention to image quality as they earnestly carried out evaluations and trials over and over again in an effort to determine how to improve the sharpness of out-of-focus images and achieve high-quality resolution. Yet at the beginning, they found themselves unable to achieve the high-quality resolution they were hoping to attain.
Takeshi Takanashi, in charge of design, recounts, "The most difficult challenge we faced was paring down bit width so as to keep the amount of hardware in check while at the same time making sure we weren't compromising image quality." Having made improvement after improvement, the team finally finished development two days before the final deadline. Takanashi comments, "We didn't intend to cause everyone to worry, but we were really focused on fine-tuning the details. This meant that we ended up pushing it down to the wire, but we were extremely confident in the image quality we were finally able to obtain." Yanase says of the end result, "Our customers were very receptive, and we were fortunate to receive very high acclaim." One of the outstanding features of this technology is that it can be embedded in either the output side or the receiving side of the image. Yanase goes on to add, "To my knowledge, there are no other companies carrying out development using a concept like this one."
"We've reached our goal, but are determined to make further improvements in the future."
Takanashi states, "I believe we've accomplished our original goal for this technology, and I have great confidence in what we have achieved." Yanase speaks eagerly about plans for the future saying, "I feel this attests to the solidarity of our team. Yet there is still much room for improvement. We have only just begun."
The newly developed single-frame super-resolution technology was featured at the 11th ESEC (Embedded Systems Expo) held in May 2008 at Tokyo Big Sight in Japan, where it garnered much attention. NEC Electronics plans to begin by providing cell-based ICs incorporated with this technology. Someday in the near future, it will become possible to view clear, beautiful images on all sorts of devices. And it may just be NEC Electronics' single-frame super-resolution technology that makes that dream a reality.
196KB)