Modern Linux distros featuring either GNOME, Unity or KDE offer fairly easy configuration of color management, this system level configuration mostly pertains to the handling of an ICC display profile.

If you have set a display profile via your system configuration tool (The Color applet in System Settings for GNOME or Unity), there are a few things to keep in mind.

An ICC display profile consists of two main parts. First the so-called "vcgt", which corrects for whitepoint (this is most noticeable on laptops which shift from being very blueish to a bit more yellowish) and gamma. The "vcgt" is loaded into X11 and applied to your whole screen, so all applications automatically benefit. On a GNOME or Unity desktop this is done by GNOME Settings Daemon during login.

The second is the rest of the ICC profile, and this has to be processed in color management enabled application (typically via liblcms2). So we need a mechanism to pass the actual ICC profile to our applications without having to configure them all individually.

The oldest mechanism is the _ICC_PROFILE atom, which allows a single display profile to be defined for your system (this obviously fails for dual head configurations). You can check if a profile is setup like so:

$ xprop -display :0.0 -len 14 -root _ICC_PROFILE

On a GNOME or Unity system the _ICC_PROFILE is setup by GNOME Setting Daemon during login.

On some systems there may be a _ICC_PROFILE_1 and _ICC_PROFILE_2 atom to facilitate dual head configurations. While darktable should pick up on those, I'm not sure how well other applications support this mechanism.

The another relatively new mechanism is colord by Richard Hughes, which is an infrastructure daemon, which by it self does very little, as it's mostly just an information/configuration store. But modern GNOME, Unity or KDE desktops store their display profile setups in colord, so applications can query colord which profile to apply depending on which screen they are displayed on (in case of a dual head configuration). Currently few applications are colord aware however. darktable is one of those few.

If you haven't setup a display profile yourself that doesn't per-se mean there is no display profile active. Modern desktops actually query the display (via EDID) itself about its advertised color coverage, from which an automatic display profile is generated. An easy way to check if such an automatic profile was generated:

# ls -l ~/.local/share/icc/edid-*.icc

For more in-depth information the following articles are highly recommended:

• http://blog.pcode.nl/2013/04/14/display-profiles-generated-from-edid/
• http://blog.pcode.nl/2012/01/29/color-management-on-linux/

darktable's current (1.0/1.1/1.2) implementation

First off I want to make something very clear, darktable's darkroom mode is fully color managed and should always be used in situations where you are evaluating color. Since darktable is colord enabled (and if colord is properly setup) it should even render correct color in dual head configurations.

In darktable's lighttable mode there are however a few things you need to be aware of. When importing new RAW photos into darktable, it will display the RAWs embedded thumbnail (which has been fully processed by the camera software) and the display profile is currently NOT being applied to these embedded thumbnails. Keep in mind, that these thumbnails have been processed by the proprietary firmware of the camera, so even if we applied the display profile it would still not match the processing that is done by darktable by default. It is however possible to disable the use of these embedded thumbnails altogether in darktable's preferences, forcing all RAWs to be processed by darktable's own imaging pipeline. This will slow down thumbnail generation by a few orders of magnitude, which is why this isn't our default behavior.

Once you've entered darkroom mode for a RAW, the image has been processed in darktable's imaging pipeline including the application of the display profile, so this result is now kept as an accurate thumbnail for that particular RAW.

If you've read that well, you might have noticed that we currently keep the display profile pre-applied in our thumbnail cache. Which is good for performance, and works just fine in most use-cases, but can result in ugly behavior for example when you change/replace your display, since darktable will still show thumbnails applied with the display profile of your previous display. In such a case you can forcibly remove the thumbnail cache like so:

$ rm -Iv ~/.cache/darktable/mipmaps-*

example

let take this photo as an example.Here, we will get rid of the fluorescent orange wall. We will use the color zone module to desaturate reds, but doing so would also desaturate the lighthouse railings. We will use masks to avoid that.we now add a masks to limit the effect of the module to the wall only. Using the masks combobox from the module, we add a curve shape :And here is our final image :

masking interfaces

The masks combobox inside modules let you add new shapes, reuse existing shapes. But if you want to perform more complex tasks, you will have to use the masks manager which is in the left panel:Right-clicking on a shapes will open a popup menu with all the possible actions.

editing shapes :

Advanced tips

Cloning :

Warning :

• there is no brush painting. masks are currently limited to curves
• very complex, self-intersecting curves might not render correctly

The general aim of the module is transferring the look and feel of one image onto another. You can use it if you intend to harmonize the image's colors of a photo series - or you may want to completely change the atmoshpere of an image, maybe even into an unnatural, surrealistic look. As its purpose is mainly an artistic one, the new module was put into the effect group.

Mathematically "color mapping" uses an enhanced model of the groundbreaking work of Eric Reinhard et.al. [0]. The basic idea is to make a statistical match of lightness and colors between images by mapping their mean and variance values in an Lab-like color space. Our enhanced version analyses the color space of the images in form of color clusters to allow for a more fine-grained match of colors. In addition we use histogram equalization for lightness matching instead of the simpler mean/variance method.

To use this module two steps are required:

• First you open the source image in darkroom mode and acquire its color characteristics by pressing the "acquire as source" button. A set of color clusters is generated and displayed in the source clusters area. Each cluster is represented by a set of nine color swatches with the mean value in the center surrounded by swatches indicating the color variance within that cluster. The clusters are sorted in ascending order by their weight, which reflects the relative number of pixels that contribute to the clusters.
• Next you open your target image in darkroom mode. darktable has remembered the previously collected source clusters; if they are not yet displayed, press the reset button in the module's expander bar. You now press the "acquire as target" button to generate a corresponding set of color clusters for your target image. It is displayed in the target clusters area.

When both source and target clusters are collected an automatic color mapping is applied to the target image. In its default settings the overall effect can be quite exaggerated. A set of sliders gives you control of the effect's strength. You can also use blending operator "normal" to tame the effect. As the color mapping module comes early in the pixelpipe, you have a lot of freedom to further finetune the colors with modules like tone curve or color correction.

The number of color clusters to apply - from 1 to 5 - is a central parameter. In the most simple case only one color cluster is applied to all colors of the target image. This will provide a color cast as if the scene was illuminated with a light in the corresponding color. Sometimes this is what you aim for but frequently the situation is more complex. Often the source and target image contain different dominant colors and you want them to be mapped in a more smooth way.

Let's look at an example. We will take the following image as our target image. It shows a scene of Valencia's Ciudad de las Artes y de las Ciencias.

First we go for a grotesque case with strong color shifts to make the effects obvious. I want to transfer the greenish color atmosphere of this image (an abandoned cold war shelter's washroom) to our target image.

If I only take one cluster I get the following output.

All colors are shifted towards the dominant green tint of the source image. The picture changes a bit if I take two clusters.

Within the source image a first cluster now represents the greens, the second cluster the more neutral tones. In our target image one cluster represents the more blueish colors and the second the more reddish ones. The cluster mapping will apply the green tones of the source image onto the blue tones of the target image leaving the other colors neutral. This gives the overall impression of a sky with poisoned green clouds. And here finally the image if we take five clusters:

Now the color variety of the target image is much better preserved. Still the greenish tint of the source image has found its way into the target. The overall result is more credible but also less pronounced as in the previous two examples. Generally a higher number of clusters leads to a less pronounced effect.

The mapping between source and target clusters is done automatically. Parameter "color dominance" gives you a level of control. If this parameter is set to zero, clusters are mapped according to color proximity. Each target cluster is checked against each source cluster and the color adjustment is done with the closest match. Depending on the source and target image this can lead to different target clusters being mapped to the same source cluster. If "color dominance" is set to its maximum 100%, the mapping is based on cluster weight. Instead of color proximity now the number of pixels contributing to a cluster plays the major role. The dominant color of the target image will be mapped to the dominant color of the source image. This is a bit similar to using a lower number of color clusters but allows a flexible adjustment without the need to recollect clusters in both source and target image.

Typically the colorizing effect at 0% is rather subtle whereas it is quite bold at 100%. In-between values lead to a transition between the extremes. This transition is incremental - the mapping flips at certain threshold values which depend on the source/target images.

Let's take a new source image: one of the Gehry buildings in Düsseldorf's Medienhafen shot shortly after sunset.

Obviously the image has two dominant colors: blue and orange. So let's take two color clusters. Here is the result if we set parameter "color dominance" to 0%.

The same source and target clusters, now applied with a color dominance of 100% give a much stronger effect.

In the last two cases I set parameter "histogram equalization" to 80%. This parameter controls how strong the tonal contrast of the source image is transferred onto the target image.

You need to do some experiments with the number of clusters and the other parameters of this module by yourself to find the best suited settings for your aim. The current status of this module can be found in darktable's master branch and will be part of darktable's next feature release.

[0] Erik Reinhard, Michael Ashikhmin, Bruce Gooch, Peter Shirley, "Color Transfer between Images", IEEE Computer Graphics and Applications, September/October 2001

Modules that can be instantiated multiple times have a new icon in their header, next to the "reset" button. Clicking that icon will open a pop-up menu that allows you to create a new instance of the module, change the order in which the different instances are applied, or delete an instance of the module. Each instance can have its own parameters, can be activated or deactivates separately and can use presets. Note that the last instance of a module can't be deleted, it can only be deactivated.

Keeping with darktable's philosophy, you can only rearrange modules of the same kind. The order of modules of different kind is optimized by the darktable developers in order to minimize quality loss. The different modules are, as usual, applied from bottom to top.

first example

To illustrate this new feature, let's take a blake&white image :

Now we will add 2 new instances of graduated density. We now have three instances, including the first one.
We set up each instance with a different colored gradient.

Here is what can be achieve :

real life example

The previous example is nice, but apart for illustrating diffraction, it is not that useful.
Multi-instance really shows its power when used with selective adjustments and the next example will show you how to do that.

Darktable does not currently implement masks, but we have a very powerful alternative, conditional blending, that allows to select region based on their color and luminance rather than their position.
So here is another example using conditional blending :

Let's start from this simple image :
This image is nice but we want to saturate the sand a little more and turn the rest of the image into black and white.

First, let's work on the sand in the foreground. We will use the colorzone plugin to selectively increase the saturation of the yellow color. The result is shown below
Now that the sand is done, let's desaturate the background.

First we create a second instance of the colorzone plugin using the new icon next to the "reset" button.
Then we restring the area where that instance will be applied using blendif.

Our background is much darker than our foreground, so it is very easy to use blendif to select the background.

Now we simply desaturate the image by pulling down the saturation line. This gives us the following final image

presets, styles...

Each instance of a module can be considered like a completely independent module. In particular, each instance can have presets applied to it independently from the other presets.

A nice trick is to use a temporary preset to easily copy/paste the settings of a given instance to a new instance. A fine tuned blendif selection can thus be copied from one instance to another (and usually inverted right away) to easily apply different presets to different areas.

Styles, on the other hands are applied to images and not to modules. This means that styles contains the information on the number of instances used and how each instance was set. If you create a style from our first image you could easily apply our "diffraction effect" to any image you want

Note : great thanks to Jeremy Rosen for is help in the writing of this blog post

today we have a request to you, the users of darktable.

As you might know there is an annual meeting of developers and users of free and open source graphics applications (like gimp, inkscape, blender, scribus, and also darktable), the Libre Graphics Meeting. This year it's held in Madrid, Spain, and it's a great opportunity to discuss things with other developers face-to-face. Last year this has brought us native colord support in darktable.

Since traveling around the world can be quite expensive at times we need YOUR help. If you think that you can spare some coin please consider to donate to this Pledgie campaign. The current status is that every Cent will go to tatica's travel fund.

In case you don't know who tatica is, she is doing the Spanish translation of darktable and promised to use her time at LGM to translate the usermanual, too. So we are aware that only Spanish speaking people will directly benefit from her going to LGM, but everyone else is welcome to show appreciation for her work, too. And maybe it will encourage others to follow her example and also translate the usermanual to your language next. :)

Thank you for reading
the darktable team

PS: If you happen to be in Madrid mid April some of us would love to meet you.

Lightroom is a great software but it will probably never run on GNU/Linux, Adobe does not seem to have any plan for this platform (and not only for Lightroom, but this is not the subject). As a long time user of Lightroom wanting to move to GNU/Linux (which is my main working platform) I have started working on a Lightroom import support for darktable. Today I have a Windows dual-boot just to run Lightroom, a waste of disk space and a waste of time as I need to reboot when I want to process my pictures.

The import support is fully automatic and done in two phases. First when importing pictures the tags are imported, then when entering the darkroom the development process for the picture being edited are imported. This import is based on the Lightroom .xmp sidecar. The chance is that both XMP can live together as Lightroom ones are named <BASENAME>.xmp whereas darktable ones are named <BASENAME>.<ext>.xmp.

What can be imported? Many of the Lightroom developments, but it is not possible to have a 100% accurate conversion process. That's why at the moment there is no batch support. Importation is done on the darkroom, one picture at a time as it requires manual adjustments.

We can classify the importation support in three categories:

1. 100% accurate:
   • crop
   • rotation
   • flip
   • tags
2. Mostly accurate:
   • exposure / blacks
   • grain
   • tone curve (only lightness supported)
   • color zones
   • local contrast
3. Needs tweaking:
   • vignette - The forms are not imported (rounded box effect).
   • spots removal - There is two kind of processing on Lightroom (clone, heal).

The work on this import module will continue, trying to be more accurate where it can be. At least I feel that this makes the migration less painful by giving the photographers an good help to quickly recover the development work done with Lightroom.

But let's look at an example, first the image as imported on Lightroom without any modifications:

Then the picture as developed on Lightroom (crop, tone curve, blacks and local contrast):

Then the picture as imported in darktable:

Note that to be closer to the Lightroom rendering I found that decreasing the saturation to 0.90 in the color correction module and adding 0.10 in exposure is a good starting point. This is not done by the exporting module as it is really a matter of taste.

[update 20/12/2012: `how to profile your camera' includes instructions with the new gen-profile script]
[update 15/12/2012:
no more recompile needed, updated the instructions in the benchmark section and how to run make.sh. ]

to summarize the current situation in dt: we have a lot of cool tools wrapped around great algorithms with almost all the knobs you need to get perfect results. while you can actually get really great results it's this sheer number of knobs that makes finding a good parameter set quite a time consuming task. even creating per-iso presets is not straight forward, as most of the current modules depend on a lot more stuff early on in the pipe (whitebalance, exposure, basecurve, etc).

in the following, i'd like to look into automating this process and (after some calibration work, which could be optional, but less optimal) then something closer to a single-click solution. the plan is:

• design a module where the denoising performance only depends on camera and iso setting.
• profile (camera, iso) pairs and store presets for each one of those (can be interpolated potentially).
• auto-apply them in switched-off state, so you can enable the right thing by just clicking the module's `off button' once.

background

a lot of research is centered around denoising images corrupted by additive white gaussian noise. this means that your input signal x is observed as

unfortunately, digital cameras don't follow that pattern, so the noise is mostly dominated by some sort of signal-independent sensor noise and signal-dependent photon noise (or `shot noise'). this non uniform noise follows a poissonian-gaussian distribution [2]

a poissonian is a discrete distribution, emerging from quantized photon behaviour. alpha converts this to a number range we use in image processing ([0, 1] with extrapolated hdr headroom in the case of dt). to be able to apply general denoising procedures, we need to stabilize this noise via a transform [0].

denoising is then done in three steps:

1. transform to map variance to unity uniformly all over the image,
2. denoise with a generic method against additive white gaussian noise,
3. backtransform

academia seems to fight additive white gaussian noise with current king-of-the-hill bm3d [6] and extensions. dt uses bilateral filters or non-local means [5]. the latter combined with a relatively simple extension [4] has been shown to perform quite well (fast, next to no implementation overhead for us).

there are very elegant wavelet based methods, too [3], so i also tried to drop in our wavelet code using bayes-shrink with an overcomplete wavelet transform, i.e. do a soft threshold shrinkage with threshold:

the results had certain wavelety artifacts, so the following is using our nlmeans code to remove additive gaussian noise.

example

to illustrate the above noise model, here's a plot of noise standard deviation sigma vs. brightness from a canon eos 5dm2 at iso 3200:

note how noise increases at higher brightness (caused by the poissonian distribution) and has a global offset (caused by the additive gaussian). only the green channel is saturated, the other two channels absorb too much of the light to go all the way to the right.

noise is estimated via mad (median of absolute deviations) on the finest hh coefficients of a wavelet transform, assuming zero mean. this is why the calibration shot needs to be out of focus, we want to minimize the spill of edge information into the hh coefficients. the median estimator is robust to a few outliers, but we still want to violate the assumption that the finest wavelet coefficients only contain noise as little as possible.

to fit an analytical model to these curves, we approximate

and fit this noise model to it [2]:

note that our fit in the above graph (smooth curve, green) goes below zero, which can be caused by the black level (which is subtracted in these input numbers).

these fits can be used to construct conversion curves [1], which look as follows:

applying those to an image, and then computing the noise levels vs. brightness again, we get this nice flat graph that mostly shows sigma=1 (which indicates white additive gaussian noise, yay!):

this is easy to process with your favourite denoising filter (wiener filter and dct, bm3d, wavelet-based methods, nlmeans etc).

all that remains to be done is to map the values back to their original range. note that the algebraic inverse of the mapping is the wrong thing to do; we want an unbiased inverse [1].

results

this table shows some numerical results in terms of peak signal to noise ratio before and after denoising.
the reference image is the mean of two iso 100 shots.

(*) handtuned using old nlmeans/equalizer and handtuned using the new module.

interestingly, i get better results out of iso 1600 than from 800. this can have a lot of causes and it would have been a good idea to measure the camera's native iso levels instead of powers of two.

here are a couple of 1:1 screen grabs to illustrate the numbers. from top to bottom: iso 3200, 6400, 12800 and 25600 with one-click denoise results:

iso 25600 seems to have trouble with the black level, judging by the color cast (lifting the black level manually helps a little, fixing whitebalance instead of using auto made things worse). apart from that i'm still not going to recommend it for everyday use, but maybe downsized resolutions will be useful for some purposes at times.

how to profile your camera

the parameters (a, b) have to be found for each (camera, iso) pair. these are then stored in a preset for a new denoising module. there is a generic (poissonian only) preset, but you can't dial in these values manually, you have to create a preset for your camera and iso (if it doesn't already exist).

to generate a profile, dt ships a couple of scripts and tools in tools/noise/. it will create presets as well as a couple of detailed data plots used to verify the usefulness of the result, such as the curves seen above.

the steps in detail:

taking noise profile images

take one shot per iso. it has to be out-of-focus and have a widespread histogram, i.e. be over- and underexposed at the same time (see example image below). if you want to be thorough, take 2-3 to verify the numbers.
this image looks very green, because the `raw linear' style has been applied to it (find it in darktable/tools/noise). this makes sure we get linear camera raw rgb with no basecurves, no whitebalance, etc in the way. it is essential that you apply this style.

you can either take those shots manually or have

~/darktable/tools/noise/gen-profile

take them for you via gphoto2. you need gphoto2, exiv2 and gnuplot installed to run the script (well, plus various coreutils).

your images should look something like this:

generate plots and fits

open a terminal, go to the directory you just exported your images to. from there, type

~/darktable/tools/noise/gen-profile -d .

this will create a ton of informative pdf files, which should look like the above examples. if the fit doesn't match the curves, or the curves look wildly different than the general shape in the plots above, something went wrong (most likely your image wasn't overexposed enough, so check the histogram). this script will output a file `presets.txt' with copy/pastable presets for your camera, to be shipped with dt. one line will look something like:

{n_("canon eos 5dm2 iso 3200"), "Canon", "EOS 5D Mark 2", 3200, {1.0f, 1.0f, {4.494e-05, 4.494e-05, 4.494e-05}, {-1.063e-06, -1.063e-06, -1.063e-06}}},

the exif data (model, maker, and iso) is used to do automatic matching of the correct preset, so you only need to enable the module.

additionally, this script will copy your darktable database, inject the newly created presets for you to test, and instruct you how to run dt with that library for a test.

run a benchmark (optional)

this requires additional shots. since we want to verify what we did was reasonable, we want (and need) independent input.

to test the presets you just created, you have to run darktable as instructed at the end of the script, something like:

darktable --library /tmp/library.db

take one image for each iso setting you're interested in (same iso sequence as above). for iso 100, take 2-3 of them. it is essential that these images only differ in iso and exposure time (take them with a sturdy tripod and aperture priority mode).

create a reference image (use the `create hdr' button in lighttable mode with a couple of iso 100 shots).

you don't need to apply any styles this time, leave the images as natural as possible (but all the same).

export your images as float pfm to the same directory (filename iso$(EXIF_ISO) is a good idea), also put the reference as `reference.pfm' in there.

apply the newly created presets to your images, export them in that directory too (float pfm, use iso$(EXIF_ISO)_denoised as a filename for example).

then run

~/darktable/tools/noise/benchmark.sh

to verify that the psnr values are reasonable. the denoised version should always have a higher psnr than the input (even for iso 100), the values should be in the range of the above table.

ship it!

send your new presets to one of our mailing lists, or upload them to some pastebin or similar and let us know in irc. we might also be interested in the pdf files with the plots, to verify your fits worked out and everything looks fine.

using the new `denoise (profiled)' module

the goal of this work was to give you a simple, single-click denoiser that always just works (tm). there are still a few options exposed for your tweaking pleasure, though these affect how the nlmeans core works. the conversion curves are not exposed to the user, but there is a generic preset for non-profiled cameras which will try to roughly do something reasonable.

• patch size: 0 will degenerate to a bilateral filter. increasing the number will increase sharpness in your output, but you might have to increase `strength' to counter balance the effect.
• strength: make denoising stronger (> 1) or weaker (< 1). mostly a matter of taste. the default setting was chosen to maximize psnr in my tests (with a patch size of 1) .

future work

[update 23/12/2012: this interpolation is now happening automatically in git master when you enable or reset the module]
 

the bottom line

if you're not into signal processing and don't like to profile your gear, just hope someone already measured your camera, enable the new denoising module and enjoy parameter free optimal denoising. it's magic!

references

• [0] paul prucnal, bahaa saleh: transformation of image-signal-dependent noise into image-signal-independent noise, optics letters, vol. 6, no. 7, 1981.
• [1] markku makitalo, alessandro foi: optimal inversion of the generalized anscombe transformation for poisson-gaussian noise, ieee trans. image process.
• [2] alessandro foi et al.: practical poissonian-gaussian noise modeling and fitting for single-image raw-data, ieee trans. image process. vol. 17, no. 10, 2008.
• [3] frolian luisier, thierry blu: a new sure approach to image denoising: interscale orthonormal wavelet thresholding, ieee trans. image process. vol.16, no. 3, 2007.
• [4] charles kervrann and jerome boulanger: optimal spatial adaptation for patch-based image denoising. ieee trans. image process. vol. 15, no. 10, 2006.
• [5] antoni buades, bartomeu coll, jean-michel morel: a non-local algorithm for image denoising. proc. cvpr 2005.
• [6] k. dabov, a. foi, v. katkovnik, and k. egiazarian: image denoising with block-matching and 3d filtering. proc. spie electronic imaging, 2006.

appendix

in this short time, you guys contributed presets for:

• canon eos-1ds mark ii
• canon eos 20d
• canon eos 30d
• canon eos 350d
• canon eos 400d
• canon eos 40d
• canon eos 450d
• canon eos 50d
• canon eos 550d
• canon eos 5d mark ii
• canon eos 5d mark iii
• canon eos 600d
• canon eos 60d
• canon eos 7d
• canon powershot g10
• canon powershot s90
• konica minolta dynax 5d
• nikon d300
• nikon d3100
• nikon d5000
• nikon d5100
• nikon d600
• nikon d7000
• nikon d80
• nikon d800
• nikon d90
• olympus e-30
• olympus e-400
• olympus e-420
• olympus e-m5
• olympus e-pl1
• panasonic dmc-fz18
• panasonic dmc-g3
• panasonic dmc-g5
• panasonic dmc-gf1
• panasonic dmc-gx1
• pentax k100d
• pentax k10d
• pentax k200d
• pentax k-5
• pentax k-7
• pentax k-x
• sony dslr-a200
• sony dslr-a230
• sony dslr-a550
• sony dslr-a700
• sony nex-3
• sony nex-6
• sony nex-7
• sony nex-c3
• sony slt-a55v
• sony slt-a65v

which seem to work great, thanks for that!

this also enables us to do nice cross-camera noise comparisons, like so:

lower would be better for both axes, but note that lower gaussian noise level might just mean higher black level. so i wouldn't base my camera purchase on these plots just yet.

note how the four versions of the eos 400d gather around almost the same spot. three of those have been taken at almost the same time, and one of them comes from a different day with different lighting. this shows that the method can produce consistent results (and that you might gain some accuracy by averaging a few runs). it would now be interesting to see how far away from that cluster a different device of the same type is, to verify using someone else's presets is a good idea.

someone noted that, at times, noise still stays in your image, visible as large color blotches (not fine grained anymore). these can be alleviated by the equalizer, using something similar to this style.

RAW format support
darktable uses two libraries to read RAW files. Most common RAW formats are read via the RawSpeed library (by Klaus Post), more esoteric RAW formats not supported by RawSpeed are read via LibRaw (which in turn is based on Dave Coffin's dcraw). While most vendors have only a single RAW format there are camera to camera variations.
So if darktable isn't recognizing your RAW files at all (or they show up as corrupted images), it is always a good idea to check if darktable's development version (git master) does read them (before running darktable git master, please do backup your database and images).

If it still doesn't work, it's a good idea to check if the latest dcraw (the version that comes with the distros tends to lag behind) can handle the files:

Install the build dependancies liblcms-dev(el) libjpeg-dev(el) and libjasper-dev(el) (or whatever your distro calls them)

# wget http://www.cybercom.net/~dcoffin/dcraw/dcraw.c
# gcc -o dcraw -O4 dcraw.c -lm -ljasper -ljpeg -llcms
# ./dcraw -v -w -o 1 -f -T IMG_0000.RAW

If it doesn't work you may want to supply Dave Coffin with sample RAW files.
Also, please mail the RAW samples to our devel mailing list so we can take a look.

Sensor support
darktable currently only supports regular Bayer style sensors (which probably accounts for 99% of available cameras). This means darktable will not be able to handle Sigma FOVEON based cameras. A lot of Fuji cameras have weird variations of the standard Bayer scheme which make them incompatible with darktable, too. We currently have no plans to support these cameras either.

Basecurve
By default RAW files are more or less linear, meaning they look fairly dark/murky by default. So one needs to apply a curve to make it look decent. For darktable we try to get a ballpark match with the camera's JPEG output with regard to tonality, to do this we use the so-called basecurve, which we manually derived from RAW+JPG samples. Typically the basecurves seem to be vendor specific (but not so much camera specific).

That said, dynamic image processing technologies like Nikon's Active D-light, Canon's Auto Lighting Optimizer, Samsung's Smart Range, Sony's Dynamic Range Optimizer, etc do throw a monkey wrench into RAW converters ability to approximate camera output in any way. So when doing comparison make sure you've turned the above off.

Color Matrices
The plain red, green and blue values we get straight from the sensor are never a good representation of color. This is fixed by characterizing the sensor's behaviour in a so called XYZ matrix (often referred to as a color matrix). There are two kinds in darktable (and you need at least one of those to get decent color rendition in darktable):

The standard color matrix is a color matrix we get from dcraw (which in turn gets most of them via Adobe's DNG Converter).

Take a sample RAW

Convert it to DNG via Adobe DNG Converter (make sure you have the latest version, assuming Adobe updated the converter for your particular camera model)

exiv2 -pt MY.DNG 2>/dev/null | grep CalibrationIlluminant2
# Exif.Image.CalibrationIlluminant2 Short 1  21

Assuming the above is 21:

# exiv2 -pt MY.DNG 2>/dev/null | grep ColorMatrix2
Exif.Image.ColorMatrix2 SRational 9  7798/10000 -2562/10000 -740/10000 -3879/10000 11584/10000 2613/10000 -1055/10000 2248/10000 5434/10000

The enhanced color matrix which we calculate from user submitted color charts.

• http://blog.pcode.nl/2010/06/28/darktable-camera-color-profiling/

Neither of the above color matrices have the goal of emulating the camera's JPEG color rendition. So even though the basecurve will in most cases give you similar tonality, the colors are unlikely to match.

White balance presets
darktable gets a library of white balance presets from the UFRaw project. This means you can select camera white balance presets in darktable's user interface. Having white balance presets for your camera is a convenience, it's not critical. But if they are missing, you can contribute them:

Upgrade the camera firmware (use a full, known reliable battery), be aware that it can brick your camera if done improperly. (having the latest firmware is preferable, but not critical, if you are uncomfortable with the risks involved in upgrading your camera you should not attempt it)

Shoot a single RAW file for each of the camera's white balance presets (make sure you don't have any white balance finetuning enabled) (image content doesn't matter) (these are usually 5-10 shots). Make sure you have a recent version of ExifTool installed (if not, you need to edit wb_extract.pl and point it to your updated copy).

Download http://ufraw.cvs.sourceforge.net/viewvc/ufraw/ufraw/wb_extract.pl

perl wb_extract.pl  2>/dev/null

(If you are adept at Perl, you could help us/UFRaw by revising the above script)

Some cameras optionally offer the ability to tune their white balance presets. You can include these as well (but the numbers of shots increase rapidly, as there are typically 5-10 base presets, offering as much as 19 steps of finetuning per preset). For the time being you need to shoot either all or none of the finetuning steps as darktable hasn't implemented finetuning interpolation yet. If you do, use only the blue-amber axis, leave the green-purple axis centered (0).

When contributing your white balance presets please do include your camera's firmware version (for future reference).

EXIF support & Lens Recognition
darktable uses the Exiv2 library to read EXIF data from image files. As with all RAW formats, while there is common ground, there can be camera to camera differences.

You can check what Exiv2 can (or can't) read from your RAW file using the exiv2 command line utility:

# exiv2 -pt IMG_0000.RAW

A common problem is lenses not being recognized. Most RAW files store the lens identifiers as a number, which Exiv2 translates into a lens name via a lookup table. Vendors don't advertise these numbers, so this lookup table is put together from community submissions. Another problem that arises is that the main camera vendors don't account for third party lenses. So a Canon and a Sigma lens can end up reported as the same number in which case it's hard to tell for Exiv2 which is which. I think a heuristic is used where focal lengths are compared to see which matches best (assuming both variants are known).

If you have any issues with EXIF you can report these to the Exiv2 project, don't forget to include sample files. Also when reporting issues with lens detection, always include the full lens name in the bug report, so "Canon 18-55" isn't useful (as there are 4 (or whatever) variants of that lens). Even if there aren't variants, still include the full/proper lens name.

While darktable doesn't use ExifTool for anything, it's still a good idea to check if ExifTool handles your files well, too, and if not report this to the ExifTool project.

Many distros ship older versions of Exiv2/Exiftool. Be very careful when updating your system copies of these libraries as you may inadvertently break other applications and/or create problems with future operating system upgrades.

Lens correction
Assuming Exiv2 translated the LensID in EXIF in to a proper lens name. The Lensfun project may or may not have correction information for your lens. For the lenses that do have correction information this is typically limited to distortion correction. If there isn't any correction information for your lens yet, you can calibrate it yourself (don't forget to submit the results to the Lensfun project, assuming they work well, please do verify this before submitting):

These articles might be helpful:

• http://libregraphicsworld.org/blog/entry/creating-lens-distorsion-models-with-hugin-lens-calibrator
• http://lensfun.berlios.de/lens-calibration/lens-calibration.html
• http://wilson.homeunix.com/lens_calibration_tutorial/
• http://www.victordima.ca/darktable-lensfun-hugin-and-a-pentax-25mm-f1-4-cctv-lens/

Final notes

If the above doesn't help you in assessing your problem and you need to contact us, please do keep the following in mind:

Where appropriate include RAW samples so developers can check your problem (please use English language file sharing services)

For some issues it's useful to upload screenshots, however please make screenshots of darktable using an English GUI:

# LANG=C darktable

This blog post will go through a simple workflow when working with high dynamic ranged images using darktable and the modules for processing, you need use Darktable 1.1RC for this guide. The example image used in the screenshots can be downloaded at following link: AtriumMorning

How to make an HDR image

I'm not going into details of the process of making an HDR image, there are many guides out there describing manual methods or automatic ones which some cameras have, but basically, take a bracket shot of your scene and import them into darktable and do no processing at all, export to 16bit tiff and import the tiff files into luminance hdr where you use its align and merge HDR functionality, when HDR is merged and cooked just save the image as EXR image format which you load into darktable for further processing.

Processing the HDR image using darktable

Step 1, load your HDR image into darktable, and enter darkroom, the image will probably look very strange and the first thing is disable all auto applied modules by selecting the "active group" and turn off all modules.

Step 2,  the first thing to do now is to settle a base for further processing correcting the image exposure, the approach differs between processing a day or night shoot. If you have a night shot with a few light sources you should now correct the exposure of the light source to bring back as much detail into light source without lowering its intensity so it end up looking dull. If you process a daylight shoot for example a room with windows where you have a indoor scene and sunlit outdoor scene, the approach will be to adjust exposure so the darkest part of the image eg. shadows contains details and have a nice dark tone. This is just a starting point for both night and day shoots, you will likely get back to this step for fine tuning the end result so don't spend too much time on this step for now, just do a rough estimation of what you think you want according to the 2 approaches described above.

Step 3, we should now apply a tone map operator, which purpose is to compress a high dynamic ranged image into a lower range suitable for display on screen etc. There are 2 variants of tone map operators, global and local, local tone map operators tries to preserve contrast, some is better than others but in general will make the image look unrealistic and are commonly used to do "artistic"  HDR images. This blog post is not about doing cartoons and therefore i will not go though a process of doing such thing. At our hands we have 'global tonemap' module in darktable which has 3 standard implementations of tonemappers, reinhard, filmic response curve and drago. The choice of operator is just a choice of what get closest to what you want. All three of them have their own pros and cons but is usable in both day and night shoots, I use all three of them depending of what result I'm after. Choose a global tone map operator and tune available controls to your like, if you processing a day shot you want to bring details in highlights and if you are on a night shot the shadows are what you aim to bring back. You will probably depending on tone map operator choice and its controls go back to step 2 to fine tune the exposure. The detail slider within the global tonemap module will try to recover some contrast details which is lost in the tonemap operation.

Step 4, the image can sometimes look somewhat dull and local contrast enhancement comes to the rescue, darktable have 2 approaches of local contrast, the equalizer can enhance contrast locally and the module named local contrast which at the time writing only is available in the source repo but will be available in the next release of darktable.

Additionally, you can use any module to enhance the image in the way you want. Play around and you will find how to get closer to the end result you are searching for, but remember not to touch exposure slider when you archived the result from above steps, that will ruin the process and you will have to start over from begin.

Geotagging module

For quite some time people have asked us for a way to geotag their images from within darktable. While that is a nifty feature for sure and really helpful when you take pictures outside of a studio we always had to say something along the lines of “sorry, we don't have that yet”. Some day however Henrik decided to give it a try and started work in his geo branch. Things started to come together nicely and everything looked really promising, but unfortunately he was a little short in free time so the progress stalled and the code started to bitrot. Since it would be a pity to throw away all the great work Henrik did I kind of adopted the branch and set sails to add the missing bits and pieces to make geotagging a new feature of darktable.

Granted, a few details are still missing and I wouldn't be surprised if a few quirks and bugs can be found, too, but nevertheless I decided that the code was ready for prime time and merged it back into master. So everyone using either the experimental builds from Pascal's PPA, compiles darktable from git or has some other way to run the latest and greatest development version of darktable can try all of this stuff.

A full fledged GPS receiver and a small logger

While the easiest way to get geotagged images is using a camera with a built-in GPS receiver (or an external receiver attached to the camera) most people don't have that. So we somehow have to assign geo locations to the images. Basically there are two ways to do that: for one you can do it manually for every image or you can use a GPS receiver to record a GPX track while you take your pictures (you can either use a cheap GPS logger or a full fledged GPS receiver, maybe even your phone). If you just want to try this you can take the manual route but if you plan to geotag thousand of pictures taken during your vacation I would advice to spend a little money on some hardware.

Some background information

If you are lucky enough to have a camera which already stores coordinates in the EXIF data of the images you can skip this paragraph and also the next one. If you want to tag the images manually you can skip the next two paragraphs, too. Or just keep reading, it might be interesting after all. ;)

So, let me explain the intended workflow to assign geo locations to your images making use of the GPX file stored by your GPS tracker. First of all let me explain what these GPX files are. The GPS receiver calculates its current position based on the information it receives from some satellites and stores them in a GPX file, together with the current date and time (and some other data which we are not interested in). Basically it is a mapping between time and position, telling us where the device has been at a given time. As you probably know, the EXIF data of the images also contains a time stamp. So all we have to do is take the time stamp of the image, look-up the position of the GPS tracker at that point in time and assume that camera and GPS device were in close proximity. Easy. Well, at least in theory. In reality there are a few problems which you need to keep in mind.

First, most cameras don't have the exact time (GPS devices however have it, they are basically showing the time from atomic clocks sent by the GPS system). At least my camera is almost always off a little. Maybe you are better in taking care of setting the embedded clock. Anyway, if the time of camera and GPS tracker differs the location will also be wrong. So we might need to fix the time stored with the image.

The second problem is more systematical. The time stored in the EXIF data doesn't contain a time zone. At least on all cameras I have seen so far. So what exactly does a time stamp of “12:00:00” mean? Well, it's noon, but where? Most people set it to their local time, but what is that? The image doesn't tell us. All we know is that the GPX file uses UTM time (at least with my Garmin) or contains the time zone used if it doesn't. In order to correctly find the position of the image we need to know if it's noon in London (UTM), New York, Berlin, Sydney or any other place of the world.

Adding coordinates to the images

Calculating the time offset

As we have just seen we need to take care of the time in two places: the offset in the images and the time zone needed to make sense of the EXIF data. That is also the order in which you have to tackle them: first the offset, then the time zone. To fix the drift of the camera assigned time stamp you can either enter it manually into the offset input field in the geotagging module in lighttable mode or let darktable help you. All you need is a picture taken of a reliable time source. This can be any precise clock or even better the time displayed on your GPS device – provided it has a display to show you the time. When you have such an image selected you can click on the button next to the offset entry (currently it has a looking glass as its icon, that might change though) and darktable will present you an entry box at the bottom of the window. Just enter the time that is shown on the clock or GPS device that you took a picture of and hit enter (or click OK). As a result you will get the difference between the time you entered and the one associated with the image in its EXIF data entered into the offset field on the right. Now all you have to do is selecting all the images you want to geotag (and that are probably suffering from the same time offset) and click the apply button (currently represented by a check mark). This will alter the time in darktable's internal database for these pictures, so you will also see the change in the image information module on the left. However, in the current state darktable will NOT attach this changed time to your exported images – they will still use the original value the camera wrote, but that is supposed to change.

Loading a GPX file

Now that you have a bunch of images with a corrected time you can apply a GPX track. Click the corresponding button in the geotagging module and navigate to the GPX file. Before confirming that dialog you should make sure that the time zone selector is showing the right one for your camera. Once that is confirmed you can click on Open. Should you ever make a mistake with the time zone selection you can just come back and reapply the GPX file with a different time zone.

Watching the images on the map

Map mode

At this point I suppose that your pictures have a location assigned – either by a nifty camera which does it on its own or by you loading a GPX file – or that you want to add that information manually. In any case you should switch to the map mode by selecting it in the header of the window (top right hand, maybe you have to expand the top panel and/or hit ctrl-h to see it). You should now see a map in the center and some new modules on the right. Make sure to display the film strip on the bottom of the window (expand the bottom panel and/or hit ctrl-f). If your images are already tagged with a location you can now double click them in the film strip and the map will be zoomed to the image. You can also pan the map by clicking and dragging the mouse and zoom it using the mouse wheel (yes, we will add keyboard shortcuts). If you want to geotag an image manually or are not happy with the location that it currently has you can put them on the map by dragging them from the filmstrip and dropping them on the map. Currently it's not possible to drag them around on the map, you always have to take them from the filmstrip! In order to help you finding the place on earth where you want to put the image to there is a location module on the right hand with which you can search for city names, points of interest, …

TL;DR

First adjust the time offset of the images, then import the GPX file and make sure to use the correct time zone. Drag&drop images from the film strip to the map afterwards when you think that you have to.