Exploring the 'color' blend mode

In my ongoing adventures in creating image manipulation tools for Matlab, I have spent quite a bit of time getting desirable behavior from blending and adjustment tools.  Although most of these problems share a common set of core difficulties, perhaps one of the more intuitive ways to explore the issue is through the common "color" layer blending mode.

In image editing tools such as GIMP and Photoshop, image layers can be combined or blended using different pixel blending modes.  While modes such as the contrast enhancing modes don't suggest their operating methods so directly, certainly the component and arithmetic modes do.  Addition and division modes suggest the underlying math.  Hue and saturation suggest the channels to be transferred between foreground and background.  The detail-oriented observer might begin to raise questions at this point.  Hue? Saturation?  In what model are we working?

What about the 'color' mode?  Mere observation suggests that it's a combination of hue and saturation information being transferred, but again, the details are critically important. Various sources online suggest that this mode performs a HS transfer in HSV... or HSB... or HSL... or HSY?  One begins to wonder if anybody knows what they're talking about.

As my experience lies with GIMP, I based my expectations on its behavior.  Certainly, I can't test my blending algo against software I don't have; however, it might be worth keeping in mind that other suites may use different approaches.  The passing impression of online resources suggests that GIMP uses HSV or HSL, Photoshop uses CIELAB, and according to simplefilter.de, PaintShopPro uses some bastard HSL/HSY swap that has people so baffled that they ask questions of forums only to get more nonsensical guidance.   Let's sort this out as best we can by simple experiment.

I assume at this point a few things about what is desired.  Forgive my loose language hereon, but in this pixel blending mode, what should be transferred is the chroma information independent of the luminosity-related information.  In other words, the hue and colorfulness should change, but not the perceived brightness.  White should remain white, black should remain black.  Acronyms make it sound like a simple task, don't they?

Let's start with HSV, HSL, and HSI.  These are polar models used to express the content of the sRGB color space.  The convenience of these familiar models is simply the manner in which their polar format allows intuitive selection and adjustment of colors.

HSV is commonly visualized as a hexagonal cone or prism defined by piecewise math.  HSI is expressed as a cylinder or cone defined by simple trigonometry.  HSL is often described as a hexagonal bicone.  What's with all the different geometry?  All we want to do is separate HS from L/V/I, right?  Feel free to look at the math, but I think it's more valuable to the novice (me) to see how the isosurfaces associated with these models are projected into the sRGB space.

Isosurfaces of Value and Intensity as projected into the RGB cube

Value, the V in HSV, is defined as the the maximum of the RGB channels.  This means that the isosurfaces of V are cube shells in RGB.  Intensity (I in HSI) is the simple average of the channels.  This translates to planar isosurfaces.  At this point, it might start to look like HSI would provide more meaningful separation of color information from brightness information.  Does it make sense that [1 1 1] and [1 0 0] have the same brightness metric as they do in HSV?  Let's try to do an HS swap in these models and see what happens.

Foreground image

Background Image

Color blend in HSV

Color blend in HSI

Well, those look terrible.  White background content is not retained, and the image brightness is severely altered.  What happens with HSL?

Color blend in HSL

Well that looks better.  In fact, that looks exactly like GIMP behaves.  Checking the source code for GIMP 2.8.14, this is exactly how GIMP behaves. Let's look for a moment at why the white content is preserved.  While I previously compared the projected brightness isosurfaces to mislead the reader as I had misled myself, the answer lies in the projection of the saturation isosurfaces.  Simply put, it is the conditional nature of the calculation of S in HSL which creates its bicone geometry and allows comparable S resolution near both the black and white corners of the RGB cube.

Saturation isosurfaces as projected into the RGB cube from HSV, HSI, HSL

So we know how GIMP does it.  What are better ways?  While HSL helps prevent the loss of white image regions, the overall brightness still suffers.  Let's start with the vague suggestion that Photoshop uses CIELAB.  At this point, I can only make educated guesses at the internals, but intuition would suggest that we might try using CIELCHab, the polar expression of CIELAB.  In naivety, I perform a CH swap to obtain the following.

Color blend in CIELCHab

Now things are actually worse.  Both black and white regions are lost.  To understand why, I put together a tool for visualizing the projection of sRGB into other color spaces such as CIELAB.

The trajectory of an out-of-gamut point in unconstrained LCHab and HuSLab

Here, we see the volume containing all RGB triplets whose values lie within standard data ranges (e.g. 0-255 for uint8).  Unlike the case with HSV, HSL, and HSI, the sRGB space is a subset of the CIELAB space.  A color point which lies outside the depicted volume would have pixel values that are either negative or larger than allowed by the datatype.  Upon conversion back to RGB, these values would be truncated to the standard range. In this manner, color points outside the cube will be mapped to points on its surface. Since the truncation occurs after conversion to RGB, the trajectory of out-of-gamut color points will be parallel to the axes of the cube.  It then stands to reason that the orientation of the neutral axis means that out-of-gamut points tend to be mapped away from the white and black corners of the cube and toward the primary-secondary edges.

If the goal remains to isolate the effects of color manipulation and image brightness, the geometry of this projection poses an issue.  To transfer the radial position of a primary or secondary corner to a pixel which is near the extent of the L axis would be to push the pixel outside the cube.  If we were to rotate the hue of a pixel which lies on a primary or secondary corner, it would again move outside the cube.  How can we manage the existence and graceful conversion of these points?   To be honest, I'm not sure I know the best methods.  All I can offer are my current approaches.

One approach would be to do data truncation prior to RGB conversion.  Of course, the maximum chroma for a given L and H is not simple to calculate, but this would keep the trajectories of out-of-gamut points in a plane of constant L.  This produces good results, but reveals an issue with working methods.

Color blend in CIELCHab

Ideally, we would try to do all our image operations in the uniform space (LAB/LUV) and only perform the conversion to RGB for final output or screen display.  At the very least, we could convert to RGB with no data truncation, allowing for negative and supramaximal pixel values to retain the information stored by OOG points.  For my efforts in creating standalone tools with RGB input and output, this best approach seems terribly impractical.  When constrained by compatibility with Matlab's image viewer and other image handling functions, we're forced to decide whether it's appropriate to lose chroma information by truncating in LCH.  Even though it causes no ill effects for a single operation, it means that successive operations will tend to compress the chroma content of the image to a small subset of the projected RGB space.

This is more of a problem for image and color adjustment than for a single channel swapping blend operation, but it's something to keep in mind.  I'll probably come back to the topic of normalized versions of LCH at a later date.

If implementing a constrained color model like this is beyond the realm of practical effort-costs, consider this simple dirty method to retain brightness in a color blend.  Simply copy the background luma, perform a HS channel swap in HSL, then re-establish the original luma using your favorite luma-chroma transformation.  This might seem like it would waste a lot of time with converting twice, but it's often the fastest method among those I've described.  The results are similar to the normalized methods, and existing conversion tools can typically be used.

Color blend in HSL with Y preservation

For an example of the methods, feel free to check out the image blending function I've posted on the Mathworks file exchange. The conversion tools for LCHab, LCHuv, HuSLab, HuSLuv, and HSY are included in the parent submission.