iPhone 3D Programming : Image-Processing Example: Bloom

Whenever I watch a classic Star Trek episode from the 1960s, I always get a good laugh when a beautiful female (human or otherwise) speaks into the camera; the image invariably becomes soft and glowy, as though viewers need help in understanding just how feminine she really is. Light blooming (often called bloom for short) is a way of letting bright portions of the scene bleed into surrounding areas, serving to exaggerate the brightness of those areas. See Figure 1 for an example of light blooming.

Figure 1. Original image, crude bloom, and Gaussian bloom

For any postprocessing effect, the usual strategy is to render the scene into an FBO then draw a full-screen quad to the screen with the FBO attached to a texture. When drawing the full-screen quad, a special fragment shader is employed to achieve the effect.

I just described a single-pass process, but for many image-processing techniques (including bloom), a single pass is inefficient. To see why this is true, consider the halo around the white circle in the upper left of Figure 1; if the halo extends two or three pixels away from the boundary of the original circle, the pixel shader would need to sample the source texture over an area of 5×5 pixels, requiring a total of 25 texture lookups (see Figure 2). This would cause a huge performance hit.

Figure 2. 5×5 filtering area

There are several tricks we can use to avoid a huge number of texture lookups. One trick is downsampling the original FBO into smaller textures. In fact, a simple (but crude) bloom effect can be achieved by filtering out the low-brightness regions, successively downsampling into smaller FBOs, and then accumulating the results. Example 1 illustrates this process using pseudocode.

Example 1. Algorithm for “crude bloom”

// 3D Rendering: Set the render target to 320x480 FBO A. Render 3D scene. // High-Pass Filter: Set the render target to 320x480 FBO B. Bind FBO A as a texture. Draw full-screen quad using a fragment shader that removes low-brightness regions. // Downsample to one-half size: Set the render target to 160x240 FBO C. Bind FBO B as a texture. Draw full-screen quad. // Downsample to one-quarter size: Set the render target to 80x120 FBO D. Bind FBO C as a texture. Draw full-screen quad. // Accumulate the results: Set the render target to the screen. Bind FBO A as a texture. Draw full-screen quad. Enable additive blending. Bind FBO B as a texture. Draw full-screen quad. Bind FBO C as a texture. Draw full-screen quad. Bind FBO D as a texture. Draw full-screen quad.

This procedure is almost possible without the use of shaders; the main difficulty lies in the high-pass filter step. There are a couple ways around this; if you have a priori knowledge of the bright objects in your scene, simply render those objects directly into the FBO. Otherwise, you may be able to use texture combiners to subtract the low-brightness regions and then multiply the result back to its original intensity.

The main issue with the procedure outlined in Example 1 is that it’s using nothing more than OpenGL’s native facilities for bilinear filtering. OpenGL’s bilinear filter is also known as a box filter, aptly named since it produces rather boxy results, as shown in Figure 3.

Figure 3. Zoom on the original image, crude bloom, and Gaussian bloom

A much higher-quality filter is the Gaussian filter, which gets its name from a function often used in statistics. It’s also known as the bell curve; see Figure 4.

Figure 4. Gaussian function

Much like the box filter, the Gaussian filter samples the texture over the square region surrounding the point of interest. The difference lies in how the texel colors are averaged; the Gaussian filter uses a weighted average where the weights correspond to points along the bell curve.

The Gaussian filter has a property called separability, which means it can be split into two passes: a horizontal pass then a vertical one. So, for a 5×5 region of texels, we don’t need 25 lookups; instead, we can make five lookups in a horizontal pass then another five lookups in a vertical pass. The complete process is illustrated in Figure 5. The labels below each image tell you which framebuffer objects are being rendered to. Note that the B₀–B₃ set of FBOs are “ping-ponged” (yes, this term is used in graphics literature) to save memory, meaning that they’re rendered to more than once.

Figure 5. Gaussian bloom with 10 FBOs

Yet another trick to reduce texture lookups is to sample somewhere other than at the texel centers. This exploits OpenGL’s bilinear filtering capabilities. See Figure 6 for an example of how five texture lookups can be reduced to three.

Figure 6. Five samples versus three samples

A bit of math proves that the five-lookup and three-lookup cases are equivalent if you use the correct off-center texture coordinates for the three-lookup case. First, give the row of texel colors names A through E, where C is the center of the filter. The weighted average from the five-lookup case can then be expressed as shown in Weighted average over five texels.

Example. Weighted average over five texels

(A + 4*B + 6*C + 4*D + E) / 16 = A/16 + B*4/16 + C*6/16 + D*4/16 + E/16

For the three-lookup case, give the names F and G to the colors resulting from the off-center lookups. Weighted average over three texels shows the weighted average.

Example. Weighted average over three texels

(5*F + 6*C + 5*G) / 16 = F*5/16 + C*6/16 + G*5/16

The texture coordinate for F is chosen such that A contributes one-fifth of its color and B contributes four-fifths. The G coordinate follows the same scheme. This can be expressed like this:

F = (A + 4*B) / 5 = A/5 + B*4/5

G = (E + 4*D) / 5 = E/5 + D*4/5

Substituting F and G in Weighted average over three texels yields the following:

(A/5 + B*4/5)*5/16 + C*6/16 + (E/5 + D*4/5)*5/16

This is equivalent to Weighted average over five texels, which shows that three carefully chosen texture lookups can provide a good sample distribution over a 5-pixel area.

1. Better Performance with a Hybrid Approach

Full-blown Gaussian bloom may bog down your frame rate, even when using the sampling tricks that we discussed. In practice, I find that performing the blurring passes only on the smaller images provides big gains in performance with relatively little loss in quality.

2. Sample Code for Gaussian Bloom

Enough theory, let’s code this puppy. Example 2 shows the fragment shader used for high-pass filtering.

Example 2. High-pass filter fragment shader

varying mediump vec2 TextureCoord; uniform sampler2D Sampler; uniform mediump float Threshold; const mediump vec3 Perception = vec3(0.299, 0.587, 0.114); void main(void) { mediump vec3 color = texture2D(Sampler, TextureCoord).xyz; mediump float luminance = dot(Perception, color); gl_FragColor = (luminance > Threshold) ? vec4(color, 1) : vec4(0); }

Of interest in Example 2 is how we evaluate the perceived brightness of a given color. The human eye responds differently to different color components, so it’s not correct to simply take the “length” of the color vector.

Next let’s take a look at the fragment shader that’s used for Gaussian blur. Remember, it has only three lookups! See Example 3.

Example 3. Blur fragment shader

varying mediump vec2 TextureCoord; uniform sampler2D Sampler; uniform mediump float Coefficients[3]; uniform mediump vec2 Offset; void main(void) { mediump vec3 A = Coefficients[0] * texture2D(Sampler, TextureCoord - Offset).xyz; mediump vec3 B = Coefficients[1] * texture2D(Sampler, TextureCoord).xyz; mediump vec3 C = Coefficients[2] * texture2D(Sampler, TextureCoord + Offset).xyz; mediump vec3 color = A + B + C; gl_FragColor = vec4(color, 1); }

By having the application code supply Offset in the form of a vec2 uniform, we can use the same shader for both the horizontal and vertical passes. Speaking of application code, check out Example 4. The Optimize boolean turns on hybrid Gaussian/crude rendering; set it to false for a higher-quality blur at a reduced frame rate.

Example 4. Rendering engine (bloom sample)

const int OffscreenCount = 5; const bool Optimize = true; struct Framebuffers { GLuint Onscreen; GLuint Scene; GLuint OffscreenLeft[OffscreenCount]; GLuint OffscreenRight[OffscreenCount]; }; struct Renderbuffers { GLuint Onscreen; GLuint OffscreenLeft[OffscreenCount]; GLuint OffscreenRight[OffscreenCount]; GLuint SceneColor; GLuint SceneDepth; }; struct Textures { GLuint TombWindow; GLuint Sun; GLuint Scene; GLuint OffscreenLeft[OffscreenCount]; GLuint OffscreenRight[OffscreenCount]; }; ... GLuint RenderingEngine::CreateFboTexture(int w, int h) const { GLuint texture; glGenTextures(1, &texture); glBindTexture(GL_TEXTURE_2D, texture); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_RGBA, GL_UNSIGNED_BYTE, 0); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0); return texture; } void RenderingEngine::Initialize() { // Load the textures: ... // Create some geometry: m_kleinBottle = CreateDrawable(KleinBottle(0.2), VertexFlagsNormals); m_quad = CreateDrawable(Quad(2, 2), VertexFlagsTexCoords); // Extract width and height from the color buffer: glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &m_size.x); glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &m_size.y); // Create the onscreen FBO: glGenFramebuffers(1, &m_framebuffers.Onscreen); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.Onscreen); glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, m_renderbuffers.Onscreen); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.Onscreen); // Create the depth buffer for the full-size offscreen FBO: glGenRenderbuffers(1, &m_renderbuffers.SceneDepth); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.SceneDepth); glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, m_size.x, m_size.y); glGenRenderbuffers(1, &m_renderbuffers.SceneColor); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.SceneColor); glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, m_size.x, m_size.y); glGenFramebuffers(1, &m_framebuffers.Scene); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.Scene); glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, m_renderbuffers.SceneColor); glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, m_renderbuffers.SceneDepth); m_textures.Scene = CreateFboTexture(m_size.x, m_size.y); // Create FBOs for the half, quarter, and eighth sizes: int w = m_size.x, h = m_size.y; for (int i = 0; i < OffscreenCount; ++i, w >>= 1, h >>= 1) { glGenRenderbuffers(1, &m_renderbuffers.OffscreenLeft[i]); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.OffscreenLeft[i]); glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, w, h); glGenFramebuffers(1, &m_framebuffers.OffscreenLeft[i]); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.OffscreenLeft[i]); glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, m_renderbuffers.OffscreenLeft[i]); m_textures.OffscreenLeft[i] = CreateFboTexture(w, h); glGenRenderbuffers(1, &m_renderbuffers.OffscreenRight[i]); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.OffscreenRight[i]); glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, w, h); glGenFramebuffers(1, &m_framebuffers.OffscreenRight[i]); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.OffscreenRight[i]); glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, m_renderbuffers.OffscreenRight[i]); m_textures.OffscreenRight[i] = CreateFboTexture(w, h); } ... } void RenderingEngine::Render(float theta) const { glViewport(0, 0, m_size.x, m_size.y); glEnable(GL_DEPTH_TEST); // Set the render target to the full-size offscreen buffer: glBindTexture(GL_TEXTURE_2D, m_textures.TombWindow); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.Scene); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.SceneColor); // Blit the background texture: glUseProgram(m_blitting.Program); glUniform1f(m_blitting.Uniforms.Threshold, 0); glDepthFunc(GL_ALWAYS); RenderDrawable(m_quad, m_blitting); // Draw the sun: ... // Set the light position: glUseProgram(m_lighting.Program); vec4 lightPosition(0.25, 0.25, 1, 0); glUniform3fv(m_lighting.Uniforms.LightPosition, 1, lightPosition.Pointer()); // Set the model-view transform: const float distance = 10; const vec3 target(0, -0.15, 0); const vec3 up(0, 1, 0); const vec3 eye = vec3(0, 0, distance); const mat4 view = mat4::LookAt(eye, target, up); const mat4 model = mat4::RotateY(theta * 180.0f / 3.14f); const mat4 modelview = model * view; glUniformMatrix4fv(m_lighting.Uniforms.Modelview, 1, 0, modelview.Pointer()); // Set the normal matrix: mat3 normalMatrix = modelview.ToMat3(); glUniformMatrix3fv(m_lighting.Uniforms.NormalMatrix, 1, 0, normalMatrix.Pointer()); // Render the Klein bottle: glDepthFunc(GL_LESS); glEnableVertexAttribArray(m_lighting.Attributes.Normal); RenderDrawable(m_kleinBottle, m_lighting); // Set up the high-pass filter: glUseProgram(m_highPass.Program); glUniform1f(m_highPass.Uniforms.Threshold, 0.85); glDisable(GL_DEPTH_TEST); // Downsample the rendered scene: int w = m_size.x, h = m_size.y; for (int i = 0; i < OffscreenCount; ++i, w >>= 1, h >>= 1) { glViewport(0, 0, w, h); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.OffscreenLeft[i]); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.OffscreenLeft[i]); glBindTexture(GL_TEXTURE_2D, i ? m_textures.OffscreenLeft[i - 1] : m_textures.Scene); RenderDrawable(m_quad, m_blitting); glUseProgram(m_blitting.Program); } // Set up for Gaussian blur: float kernel[3] = { 5.0f / 16.0f, 6 / 16.0f, 5 / 16.0f }; glUseProgram(m_gaussian.Program); glUniform1fv(m_gaussian.Uniforms.Coefficients, 3, kernel); // Perform the horizontal blurring pass: w = m_size.x; h = m_size.y; for (int i = 0; i < OffscreenCount; ++i, w >>= 1, h >>= 1) { if (Optimize && i < 2) continue; float offset = 1.2f / (float) w; glUniform2f(m_gaussian.Uniforms.Offset, offset, 0); glViewport(0, 0, w, h); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.OffscreenRight[i]); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.OffscreenRight[i]); glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenLeft[i]); RenderDrawable(m_quad, m_gaussian); } // Perform the vertical blurring pass: w = m_size.x; h = m_size.y; for (int i = 0; i < OffscreenCount; ++i, w >>= 1, h >>= 1) { if (Optimize && i < 2) continue; float offset = 1.2f / (float) h; glUniform2f(m_gaussian.Uniforms.Offset, 0, offset); glViewport(0, 0, w, h); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.OffscreenLeft[i]); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.OffscreenLeft[i]); glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenRight[i]); RenderDrawable(m_quad, m_gaussian); } // Blit the full-color buffer onto the screen: glUseProgram(m_blitting.Program); glViewport(0, 0, m_size.x, m_size.y); glDisable(GL_BLEND); glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffers.Onscreen); glBindRenderbuffer(GL_RENDERBUFFER, m_renderbuffers.Onscreen); glBindTexture(GL_TEXTURE_2D, m_textures.Scene); RenderDrawable(m_quad, m_blitting); // Accumulate the bloom textures onto the screen: glBlendFunc(GL_ONE, GL_ONE); glEnable(GL_BLEND); for (int i = 1; i < OffscreenCount; ++i) { glBindTexture(GL_TEXTURE_2D, m_textures.OffscreenLeft[i]); RenderDrawable(m_quad, m_blitting); } glDisable(GL_BLEND); }

In Example 4, some utility methods and structures are omitted for brevity, since they’re similar to what’s found in previous samples.

Keep in mind that bloom is only one type of image-processing technique; there are many more techniques that you can achieve with shaders. For example, by skipping the high-pass filter, you can soften the entire image; this could be used as a poor man’s anti-aliasing technique.

Also note that image-processing techniques are often applicable outside the world of 3D graphics—you could even use OpenGL to perform a bloom pass on an image captured with the iPhone camera!