RGB "Bayer" Color and MicroLenses

Bayer color filter array is a popular format for digital acquisition of color images [1]. The pattern of the color filters is shown below. Half of the total number of pixels are green (G), while a quarter of the total number is assigned to both red (R) and blue (B).

In order to obtain this color information, the  color image sensor is covered with either a red, a green, or a blue filter, in a repeating pattern. This pattern, or sequence, of filters can vary, but the widely adopted “Bayer”  pattern, which was invented at Kodak, is a repeating 2x2 arrangement.

Photographs & Text adapted from Photobit

When the image sensor is read out, line by line, the pixel sequence comes out GRGRGR, etc., and then the alternate line sequence is BGBGBG, etc. This output is called sequential RGB (or sRGB).

Since each pixel has been made sensitive only to one color (one spectral band), the overall sensitivity of a color image sensor is lower than a monochrome (panchromatic) sensor, and in fact is typically 3x less sensitive. As a result, monochrome sensors are better for low-light applications, such as security cameras. (It is also why human eyes switch to black and white mode in the dark).

MicroLenses

Microlenses and a metal opaque layer above the silicon funnel light to the photo-sensitive portion of each pixel. On their way, the photons of light pass through a color filter array (CFA) where the process begins of obtaining color from the inherently “monochrome” chip. (Actually, “panchromatic” is an apter term, since sensors respond across the spectrum; the word monochrome comes from television use and refers to black and white).

White Balance, Bayer Interpolation and Color matrix Processing

White Balance and Color Correction are processing operations performed to ensure proper color fidelity in a captured digital camera image. In digital cameras an array of light detectors with color filters over them is used to detect and capture the image. This sensor does not detect light exactly as the human eye does, and so some processing or correction of the detected image is necessary to ensure that the final image realistically represents the colors of the original scene. 

Each pixel only represents a portion of the color spectrum and must be interpolated to obtain an RGB value per pixel.  The Bayer color filter array (CFA) pattern, shown above, is a popular format for digital acquisition of color images [1]. Half of the total number of pixels are green (G), while a quarter of the total number is assigned to both red (R) and blue (B).

This note describes conversions from Bayer format data to RGB and between RGB and YUV (YCrCb) color spaces. We also discuss two color processing operations (white balance and color correction) in the RGB domain, and derive the corresponding operations in the YUV domain. Using derived operations in the YUV domain, one can perform white balance and color correction directly in the YUV domain, without switching back to the RGB domain.

1.)  White Balance & Bayer Interpolation

The first step in processing the raw pixel data is to perform a white balance operation.  A white object will have equal values of reflectivity for each primary color: ie:

R = G = B

An image of a white object can be captured and its histogram analyzed.  The color channel that has the highest level is set as the target mean and the remaining two channels are increased with a gain multiplier to match.  For example, if Green channel has the highest mean, gain ‘a’ is applied to Red and gain ‘b’ is applied to Blue.

G’ = R’a = bB’

The White Balance will vary, based on the color lighting source (Sunlight, Fluorescent, Tungsten) applied to the object and the amount of each color component within it.  A full color natural scene can also be processed in the same fashion.  This “Gray World” method assumes that the world is gray and the distribution of primaries color will be equal. 

The “White Patch” method attempts to locate the objects that are truly white, within the scene; by assuming the whites pixels are also the brightest (I = R+G+B).   Then, only the top percentage intensity pixels are included in the calculation of means, while excluding any pixels that may have any channel that is saturated.

Bayer Interpolation

To convert an image from the bayer format to an RGB per pixel format, we need to interpolate the two missing color values in each pixel. Several standard interpolation methods (nearest neighbor, linear, cubic, cubic spline, etc.) were evaluated on this problem in [2]. The authors have measured interpolation accuracy as well as the speed of the method and concluded that the best performance is achieved by a correlation-adjusted version of the linear interpolation. The suggested method is presented here.

Interpolating red and blue components

 Figure 1: Four possible cases for interpolating R and B components

As suggested in [2], R and B values are interpolated linearly from the nearest neighbors of the same color. There are four are possible cases, as shown in Figure 1. When interpolating the missing values of R and B on a green pixel, as in Figure 1 (a) and (b), we take the average values of the two nearest neighbors of the same color. For example, in Figure 1 (a), the value for the blue component on a shaded G pixel will be the average of the blue pixels above and below the G pixel, while the value for the red component will be the average of the two red pixels to the left and right of the G pixel.

Figure 1 (c) shows the case when the value of the blue component is to be interpolated for an R pixel. In such case, we take the average of the four nearest blue pixels cornering the R pixel. Similarly, to determine the value of the red component on a B pixel in Figure 2 (d) we take the average of the four nearest red pixels cornering the B pixel.

Interpolating the green component

By [2], green component is adaptively interpolated from a pair of nearest neighbors. To illustrate the procedure, consider two possible cases in Figure 2.

Figure 2: Two possible cases for interpolating G component

 In Figure 2 (a), the value of the green component is to be interpolated on an R pixel. The value used for the G component here is

In other words, we take into account the correlation in the red component to adapt the interpolation method. If the difference between R₁ and R₃ is smaller than the difference between R₂ and R₄, indicating that the correlation is stronger in the vertical direction, we use the average of the vertical neighbors G₁ and G₃ to interpolate the required value. If the horizontal correlation is larger, we use horizontal neighbors. If neither direction dominates the correlation, we use all four neighbors.

Similarly, for Figure 2 (b) we will have

 To conclude this section, note that if the speed of execution is the issue, one can safely use simple linear interpolation of the green component from the four nearest neighbors, without any adaptation

According to [2], this method of interpolation executes twice as fast as the adaptive method, and achieves only slightly worse performance on real images.  For even fast updates only two of the four green values are averaged.  However, this method displays false color on edges or zipper artifacts.

Color Saturation Matrix

The operation for saturation can be applied at the same time as the color correction matrix.  Unlike the color correction matrix, the saturation matrix does not rotate the vectors in the color wheel:

K is the saturation factor
K=1 means no change
K > 1 increases saturation
0<K<1 decreases saturation,  K=0 produces B&W ,  K<0 inverts color

A sample table of matrix values are calculated and shown below:

 
The saturated image can be further processed with an additional color correction matrix to compensate for cross-talk induced by the micro-lens and color filter process, lighting and temperature effects.  The combination matrix ([color correction matrix] *  [saturation matrix]) results in a closer to true world color representation, but an increase in noise.  Typically, the blue pixel has the lowest pixel response and the highest Crosstalk from the Green and Red light.  The resulting noise after matrix operation is a high degree of blue noise.

A monochrome image can now be easily obtained from a color image by setting K=0

2. Conversion between RGB and YUV

 We give two commonly used forms of equations for conversion between RGB and YUV formats. The first one is recommended by CCIR [3]

(2.1)

The second form is used by Intel in their image processing library [4], and may be more suitable for implementation:

(2.2)

In either case, resulting values of Y, U and V should be clipped to fit the appropriate range for the YUV format (e.g. [0,255] for a 24-bit YUV format). The inverse conversion may be accomplished by:

  (2.3)

3. White balance operation in RGB and YUV domains

The white balance operation is defined as a gain correction for red, green and blue components by gain factors A_R, A_G and A_B, respectively, i.e.

(3.1)

The new (white-balanced) values for red, green and blue are R_wb, G_wb and B_wb. To derive the equivalent form of this operation in the YUV domain, we proceed as follows. First, write equation (2.1) as

(3.2)

where  is the vector in the RGB space,  is the corresponding vector in the YUV space, , and C is the appropriate matrix of conversion coefficients. Similarly, (3.1) can be written as

(3.3)

where  is the vector in the RGB space modified by white balance operation (2.4), and . We want to determine what is the corresponding vector in the YUV domain, without having to revert back to the RGB domain. Vector  is found by substituting  for x in (3.2)

.

Let , so that . Then . Substitute this expression for x back into (3.2) to obtain:

(3.4)

This equation provides the connection between y and  without involving x or  (i.e. without going back to the RGB domain). Manipulating (3.4) and using the fact that for nonsingular matrices  [5], we get that white balance operation in the YUV domain is

(3.5)

Expressing components of  from (3.5) we get

Terms with leading coefficient less than 10^-3 have been dropped.  

References

[1]     B. E. Bayer, Color imaging array, US Patent No. 3971065.

[2]     T. Sakamoto, C. Nakanishi and T. Hase, “Software pixel interpolation for digital still cameras suitable for a 32-bit MCU,” IEEE Trans. Consumer Electronics, vol. 44, no. 4, November 1998.

{3}  http://www.northpoleengineering.com/rgb2yuv.htm