A new year is coming on us quickly, so how about a nice new website to go with it?

houz and I have been working hard over the past few months to migrate the old website from Wordpress to a new static site, using Python/Pelican. This should make things more secure and safer for both you and us (see the problems that rawsamples.ch had for the perils of using a db-driven backend for a website). Not to mention it makes collaboration and contributing a bit easier now, as the entire site gets its own GitHub repository (I’ll be eagerly awaiting your pull requests).

improving contrast with the local laplacian filter

sometimes difficult lighting situations arise which, when taking photographs, result in unappealing pictures. for instance very uniform lighting on a cloudy day may give dull results, while very contrasty illumination (such as back lit) may require to compress the contrast to embrace both highlights and shadows in the limited dynamic range of the output device.

refer to the following two shots as examples:

Rawsamples.ch is a website with the goal to:

…provide RAW-Files of nearly all available Digitalcameras mainly to software-developers. [sic]

It was created by Jakob Rohrbach and had been running since March 2007, having amassed over 360 raw files in that time from various manufacturers and cameras. Unfortunately, back in 2016 the site was hit with an SQL-injection that ended up corrupting the database for the Joomla install that hosted the site. To compound the pain, there were no database backups … :(

Have you ever over-exposed your images? Have you ever noticed that your images look flat and dull after you apply negative exposure compensation? Even though the over/underexposed warning says there is no overexposure? Have you ever wondered what is going on? Read on.

the problem

First, why would you want to know which pixels are overexposed, clipped?

Consider this image:

… Why is the sky so white? Why is the image so flat and dull?

modern sensor capture an astonishing dynamic range, namely some sony sensors or canon with magic lantern’s dual iso feature.

this is in a range where the image has to be processed carefully to display it in pleasing ways on a monitor, let alone the limited dynamic range of print media.

example images

use graduated density filter to brighten foreground

using the graudated density iop works well in this case since the horizon here is more or less straight, so we can easily mask it out with a simple gradient in the graduated density module. now what if the objects can’t be masked out so easily?

[update 2016/07/31: there was a section about intermediate export to csv and manually changing that file. this is no longer needed, exporting the style directly from darktable-chart is fine now.]

motivation

for raw photography there exist great presets for nice colour rendition:

• in-camera colour processing such as canon picture styles
• fuji film-emulation-like presets (provia velvia astia classic-chrome)
• pat david’s film emulation luts

unfortunately these are eat-it-or-die canned styles or icc lut profiles. you have to apply them and be happy or tweak them with other tools. but can we extract meaning from these presets? can we have understandable and tweakable styles like these?

Most modules in darktable are working on changing pixels color, lightness, etc. Few modules are moving pixels and when they do they are doing it in a very constraint way like to rotate, fix the lens’ distortions or remove spots.

The liquify module offer more ways to move pixels around by applying some free style distortions to parts of the image. There is three tools to help doing that:

• point
• line
• curve

Due to the heated debate lately, a short foreword:

We do not want to harass, insult or criticize anyone due to his or her choice of operating system. Still, from time to time we encounter comments from people accusing us of ignorance or even disrespect towards Windows users. If any of our statements can be interpreted such, we want to apologize for that – and once more give the full explanation of our lacking Windows support.

Today we are happy to announce a big new feature that we will not only ship with the big 2.0 release later this year but also with our next point release, 1.6.4, which is due in about a week: even more darkroom modules!

One of the big strengths of darktable has always been its varied selection of modules to tweak your image. However, that has also been one of the main points of criticism: too much, too many and too complicated to grasp. To make it easier for the user to deal with the flood of tools darktable has had the “more modules” list since many years. It changed its appearance a few times, we added module categories, allowed to select favorite modules, and all of that has proven to be useful. Thus there have always been people that approached us with great new ideas for new modules, especially since we moved to GitHub a while ago with its powerful Pull Request system, yet we couldn’t accept many of them. Some were not that great codewise, some didn’t really fit our product vision – and then there were some that looked nice and certainly benefited some users, but we felt it wasn’t generic enough to justify polluting our module list even more. Of course this was a bad situation, after all these people invested quite some time into providing us with a new feature that we turned down. No one likes to waste their time.

If you overexpose a photo with your digital camera you are in trouble. That’s what most photography related textbooks tell you – and it’s true. So you better pay close attention to your camera’s metering while shooting. However, what to do when the “bad thing” happened and you got this one non-repeatable shot, which is so absolutely brilliant, but unfortunately has some ugly signs of overexposure?

In this blog article I’d like to summarize how darktable can help you to repair overexposed images as much as possible. I’ll cover modules which have been part of darktable for a long time but also touch the brand new module “color reconstruction”.

darktable (and some other projects, like for example ufraw) don’t do any real lens detection or correction by itself. We depend on two libraries which in most cases are provided by the Linux distribution you’re using.

Lens Detection

Many image files contain metadata about how the image was created. In case of digital camera images, a standard called Exif is used, this standard allows a camera to record many details about how an image was taken. However Exif is not a singular well defined specification, there is a common part that is well defined, and there are the so-called MakerNotes. The MakerNotes are parts of Exif that each vendors gets to do with whatever they like. They are typically completely undocumented, and have to be reverse-engineered to be able to handle them in any way. For most vendors this reverse engineering has been done to some degree and at least parts of the MakerNotes can be deciphered most of the time.

Pat David has a great blog on photoediting in GIMP. I recently read his post on luminosity masks and was fairly impressed. Can darktable do something similar? Yes – they’re a special case of parametric masks.

I thought I’d post a quick tutorial on luminosity masks using parametric masks. First, I strongly suggest you read Pat David’s post and thoroughly understand what’s going on.

A quick and simplistic explanation follows: Normally, if we make a selection and, say, adjust the brightness dramatically in that selection, we get a sharp (and ugly) transition near the edge of the selection:

Figure 1: the gradient test image – download and play with it in darktable

People using image editors or similar (raster) graphics editors are probably familiar with histograms. You also have them in almost all digital cameras. In darktable you can find it very prominently in the top right corner of darkroom mode, but also as a backdrop of modules like levels, tonecurve and similar.

From a mathematical point of view they are a diagram displaying the amount of pixels in the image that have a specific colour, lightness, value or similar measure. The horizontal axis represents the brightness while the vertical axis corresponds to the amount of pixels that have that brightness.

wouldn’t it be great if you could judge sharpness of your images in lighttable mode? this mode is limited to small and medium sized thumbnails of your images, so we can deliver the required speed to browse a lot of them.

to tell whether or not you got the focus right during the shoot, we would like to look at the full resolution. the most you get out of lighttable mode will look like this:

The general picture on the modern Linux desktop

Modern Linux distros featuring either GNOME, Unity or KDE offer fairly easy configuration of color management, this system level configuration mostly pertains to the handling of an ICC display profile.

If you have set a display profile via your system configuration tool (The Color applet in System Settings for GNOME or Unity), there are a few things to keep in mind.

An ICC display profile consists of two main parts. First the so-called “vcgt”, which corrects for whitepoint (this is most noticeable on laptops which shift from being very blueish to a bit more yellowish) and gamma. The “vcgt” is loaded into X11 and applied to your whole screen, so all applications automatically benefit. On a GNOME or Unity desktop this is done by GNOME Settings Daemon during login.

In darktable, selective editing was a long awaited feature. Our development version now allow limiting module effects to a region of the image.

Remember the old times, the red light of the darkroom, the smell of the developing bath …

Remember when you were using your hands or a small piece of cardboard to achieve some masking …

Now you can do that in darktable.

example

let take this photo as an example.

One of the upcoming new feature in darktable is the ability to use the same development module several times. By applying the same module multiple times and combining them with blendif it is possible to do some effects that could not be achieved previously without using external tools like the gimp.

Modules that can be instantiated multiple times have a new icon in their header, next to the “reset” button. Clicking that icon will open a pop-up menu that allows you to create a new instance of the module, change the order in which the different instances are applied, or delete an instance of the module. Each instance can have its own parameters, can be activated or deactivates separately and can use presets. Note that the last instance of a module can’t be deleted, it can only be deactivated.

… and how to get rid of it.

[update 02/05/2018 The information how to create camera noise profiles is outdated please read this tutorial instead!]

[update 20/12/2012: ‘how to profile your camera’ includes instructions with the new gen-profile script]

[update 15/12/2012: no more recompile needed, updated the instructions in the benchmark section and how to run make.sh.]

to summarize the current situation in dt: we have a lot of cool tools wrapped around great algorithms with almost all the knobs you need to get perfect results. while you can actually get really great results it’s this sheer number of knobs that makes finding a good parameter set quite a time consuming task. even creating per-iso presets is not straight forward, as most of the current modules depend on a lot more stuff early on in the pipe (whitebalance, exposure, basecurve, etc).

Update: Some of the information on this page is oudated. See the github wiki page on camera support for up-to-date instructions.

Say you’re running darktable, you’ve just bought a brand spanking new camera and it’s not supported yet by darktable. Here is a list of things that need to be done (typically we’d recommend to check this before actually buying anything, often you’ll be able to find sample raw files online):

Introduction

This blog post will go through a simple workflow when working with high dynamic ranged images using darktable and the modules for processing, you need use darktable 1.1RC for this guide. The example image used in the screenshots can be downloaded at following link: AtriumMorning

How to make an HDR image

I’m not going into details of the process of making an HDR image, there are many guides out there describing manual methods or automatic ones which some cameras have, but basically, take a bracket shot of your scene and import them into darktable and do no processing at all, export to 16bit tiff and import the tiff files into luminance hdr where you use its align and merge HDR functionality, when HDR is merged and cooked just save the image as EXR image format which you load into darktable for further processing.

in an ideal world, an image is piecewise smooth. it has soft gradients, some detail and edges. in particular there’s no noise and the edges are sharp. given these assumptions, you can do a lot of cool things to your pictures, using techniques like frequency space editing, wavelets, or local histograms.

darktable’s equalizer module demonstrates some of this, using the wavelet approach. you can use it to sharpen and denoise, enhance or attenuate certain frequencies in your image, while keeping the edges intact.

false color highlights seem to be an issue frequently, so here’s some quick faq about it. alexandre, please excuse all the outward references ;)

why are my highlights magenta?

that’s how the sensor works. it collects a couple of photons, at some point it fills up and rejects to deliver any more useful information past this point. unfortunately that doesn’t happen at the same time for all color channels.

how does the sensor work?

usually digital cameras come with a color filter array (CFA) of absorptive filters in front of an array of CCD or CMOS sensor which collect electrons proportional to the incoming photons according to the photoelectric effect.

Recent builds from git will bring you a new executable, “darktable-cli”. With this tool you can export images using the processing power of darktable on the command line.

The simplest way to call the utility is

darktable-cli <input> <output>

This will take the input image, look for the XMP file associated with it, process it at maximal resolution and write the output to output, trying to guess the output format from the output filename. You can also explicitly give a XMP file by running

We heard quite some voices from users requesting better possibilities to contribute to the project. Here they are.

In our dev meeting (or if you had a closer look: even before) we decided to ditch the old bug tracker in favor for a new one: Redmine, hosted by PolarFox, just on the same server as our website lives. This comes with some long-wanted features – and more liberty in configuration.

darktable is all about providing you the best tools in order to get the most out of your photographies. This blog entry will explain how an existing feature can help you get more detailed exports and it will try to give you a glimpse of what is cooking in an unscheduled but upcoming version of darktable for even better detail preservation.

Make sure to enable High Quality Resampling for exporting your photographies

In darktable, the pixel pipeline is responsible for processing your photography from demosaic up to the point it is passed over to the output subsystem (saving in tiff, jpeg…).

For quite some time darktable supports tethering your camera. What was missing all the time was live view. This is about to change though. If you use master (and have a camera supporting it) you can now either use the eye button in the “camera settings” module or hit ‘v’ on your keyboard to start live view from within tethering mode. Since the preview is scaled to fit your screen it might be a good idea to hide some of the side panels. If you are using a Canon EOS (I only tested this with my 40D) you can also use your middle mouse button to zoom into the preview. Another click brings you back to regular live view.

I’m the self-appointed maintainer of darktable on Solaris, which is a fairly easy gig to keep on top of.

Here’s why that is so: darktable’s codebase is very portable. It’s not riddled with operating system-specific assumptions; it uses standard C (with some C++), and apart from the OpenCL support every prerequisite library is buildable on Solaris with gcc or g++. I’d prefer to use Oracle Solaris Studio because that’s my work compiler, but there’s no great incentive for me to beat up on all the prerequisites to make them behave.

or “How to drive away the evil skull”

At all times main memory was one of the most limited resources in computing. Although from 20 years to now the memory setup of a typical desktop PC has increased by a factor of several thousands (from less than a megabyte to a few gigabytes), we still need to consider how to efficiently handle that resource.

The reason of course lies in the increasing demands of modern applications. Today it might be darktable which is the single most challenging software to hit the boundaries of your system.

this is one of the oldest modules in darktable. it appeared to me that it probably lacks an example to discover how useful it can be … so here goes the example.

this started off to be a wrapper around the gegl:whitebalance operation, which works in Lab color space and is able to give dark and bright colors a different color tint, interpolating between the two for mid tones.

so suppose you have the following image:

or “If one slider is not enough”

Diligent readers of our small blog series are already aware of the blending feature that darktable offers as part of many modules. Instead of just handing over their result to the subsequent module in pixelpipe, “blending modules” take a moment to reconsider. Based on the blend setting they will take their original output together with their input and do a re-processing. As an example refer to here, where we took blend mode “overlay” with module “lowpass” to do shadow recovery.

Many readers will have already heard about GPU processing and the fact that darktable can make use of OpenCL to improve performance. As we still lack a detailed documentation of that topic, please find here a few explanations and howtos.

The Background

Processing high resolution images belongs to the more demanding tasks in modern computing. Both, in terms of memory requirements and in terms of CPU power, getting the best out of a typical 15, 20 or 25 Megapixel image can quickly bring your computer to its limits.

One of the remaining shortcomings of digital cameras is their rather low dynamic range in comparison to analog – especially black-and-white – film. Scenes with strong differences between highlights and shadows are very difficult to capture. Even if they are exposed properly with no blown-out highlights they will too often only give acceptable results after extensive post-processing.

Fortunately, darktable is progressing with a high pace. Some days ago I wrote an article on how to recover shadows with a technique using lowpass filter plus blend mode (" Using lowpass filter to recover shadows "). In between a new, even better module called “shadows and highlights” was integrated into darktable, that obsoletes this technique.

or “How to bring the jungle back”

Since its early beginnings darktable has a tone curve module that is able to alter the gray level distribution of an image. Recently we did an enhancement: tone curve is now able to control the full Lab color space with separate curves for the L, a and b channel. People who are used to curve tools in RGB, at first might get puzzled over the results of these three curves; they show marked differences to the typical RGB curve. Especially a and b channels need to be dealt with in the right way; not doing so will give you strong off-colors. To spare you frustration here are some explanations and examples.

or: how to rescue your shot after the fact.

also: how to use color zones for black and white.

sometimes i’m surprised by what kind of data is hidden in my raw images, and i want to pass this on to those of our users who happily take pictures in jpg. actually it’s just a short story about a typical communication problem between me and my camera and the way darktable moderates that, after the fact.

you might have noticed our equalizer tool, and been confused by it and the many controls. that’s probably partly because you didn’t see a similar thing before, we had to develop it first.

very short history

behind the ui is a powerful frequency domain processing technique, based on wavelets. the most commonly used wavelets are based on the lifting scheme [swe97], work in a data-independent way, and are decimated (i.e. the coarse coefficients are much more sparse than the fine ones). while this allows for very fast implementations, data-independent wavelets can lead to blurring or ringing around edges (depending on what you do to the coefficients during image enhancement). raanan fattal had quite an inspiring paper at siggraph 2009 [fat09] introducing edge weights into the lifting scheme to overcome this. while the method is fast, produces okay results (the legacy equalizer version I was based on this), it has some problems caused by the decimation. in particular, decimated wavelets are not shift-invariant, which means your results will change if you slightly crop the image for example. actually the same author also had a solution for that earlier already [far07], in a different context, and not quite as fast.

different kind of saturation

there are many different ways of tuning saturation, darktable does offer a few alternative ways to alter saturation and the reason for this post is to clarify what they do and how they work. the image on left is the original untouched image used for the different examples below, use it as a reference for comparing the results of the different kind of saturation described below, the resulting effects is exaggerated to make it easier to spot the differences.

since i probably tend to make this more technical than any reader would like to, here’s the take home message:

• much faster import of folders
• much faster thumbnail creation for first-time images
• much improved scalability wrt concurrency
• much improved scalability wrt total number of images in your database (should be good up to around 500 million images)
• much improved robustness (no more deleting ~/.cache/darktable/mipmaps all the time, yay)

some context:

darktable’s light table mode shows you your image collection in arbitrary order and filtered by arbitrary queries to the underlying database. that means that there is quite some uncertainty about which thumbnails you are going to need in the next second. so we rely on a caching mechanism that stores a few thumbnails, regenerates them as needed, and evicts not so frequently used thumbnails from the cache.