posted April 9th, 2008
As you may know, Naked light no longer primarily uses Core Image, but instead uses the Naked Anatomy Engine. Anatomy is similar to Core Image that they both abstract OpenGL and the OpenGL Shading Language for 2D graphics, but Anatomy is designed directly for use with Naked light, which offers us lots of speed and stability bonuses. Naked light pushed Core Image to its limits, and well, broke it with all kinds of edge cases that Core Image wasn’t designed for—so while I was holding off on Anatomy at first, it soon become imperative.
That said, Core Image still has lots of great filters I want to take advantage of, so it’s necessary to have a bridge between Anatomy and Core Image. I do this through an OpenGL extension called the Framebuffer Object architecture, which let’s us use FBOs (framebuffer objects).
In order to draw something onto your screen, all graphics architectures must draw something into a special bitmap called a buffer (or, in OpenGL parlance, a framebuffer). FBOs are a special buffer that can then be used as a texture and redrawn into other buffers.
I can also use FBOs (and other textures) in Core Image by creating a CIImage object with the imageWithTexture:size:flipped:colorSpace: message.
So the process goes like this:
- We render the nodes to the left of a Core Image node (or group of Core Image nodes) into an FBO.
- We process the FBO using the filters in the Core Image node.
- We render that into a new FBO, setup as a CIContext.
- We then process that FBO in Anatomy through the remaining Anatomy filters.
Which works great, except for a few disastrous bugs. We had a twofer last week, this bug was a five-fer.
Things looked good when zoomed out. When you zoomed in, and the lower left corner of the canvas was out of view, portions of images would get shifted around, portions wouldn’t get drawn, mayhem occurred.
First, Step 1 would reuse the same FBO for different sizes. This is obviously wrong, and now Naked light throws away old FBOs when they start to smell and have the wrong coordinates.
Second, Step 4 wasn’t placing the final FBO in the right place in Anatomy coordinates. A few betas ago I changed Anatomy’s coordinate system to be resolution independent. I didn’t make that fix here, apparently. Instead of placing something at 1 inch, for example, the code would try to replace the image at 72 pixels. But it actually placed things at 72 inches, which is, pretty clearly, not good.
Third, Step 3 wasn’t rendering the Core Image result in the right coordinates in our FBO-based CIContext. Since part of the FBO would never get drawn into, you would see garbage leftover scraps of images in VRAM. On the left is the rendered FBO going into Core Image, on the right is the FBO I rendered the Core Image result into.
This is actually a poor image: I just restarted, and my VRAM is relatively unfragmented. Ordinarily, where you just see faint gray diagonal lines is a hot, hot mess.
It turns out I was using some pretty suspicious-looking math, including a fairly unconventional trick of setting the left and bottom bounds of the FBO to the negative of the left and bottom bounds of the rectangle the view is cropped to. Apple insists, “The Core Image coordinate space has the origin in the bottom left corner of the screen. You should configure the OpenGL context in the same way.” I quickly fixed that and got things working.
Apparently.
It turns out that negative trick was required to keep Core Image filters with point inputs happy. It kept the coordinate space consistent from FBO to FBO. Without it, zooming in and out caused all of these filters’ points to shift as you did so. So to keep Step 2 working, I had to undo this.
Fourth, I finally fixed the bug in Step 3 while keeping everything else working. It was actually really easy: I had assumed that the origin of my FBO would always be right in the lower left corner, as Apple suggested. This code must have predated the bug fix I had just previously broke again. It’s fixed, and now the output FBO matches the input FBO with zero shifting.
Finally, a slight aesthetic fix was needed. For speed reasons, I only send into Core Image what I expect out of it. This results in visual anomalies, but is fine for previewing—as Naked light doesn’t yet render final-quality images (that comes in Beta 4 or 5, I haven’t decided which yet. In Core Image terms, I wrongly assume that the Region of Interest is the Domain of Definition. You can see this in the error image: the image is bordered in black. I now clamp the rendering so that the edge pixels repeat indefinitely, which helps, but doesn’t alleviate, this not-really-an-issue.
May 27th, 2008 at 5:27 AM
Phew! Few things frustrate me as much as graphics bugs
Good thing I’m not you 
Good job, keep it up! Really looking forward to this app 