basecurve-0

about basecurves

the purpose of the basecurve is to make the otherwise scene-referred linear (linear raw rgb) color look good on your output devices. this is done independently of any color managed transforms which are also done in the pipeline, so we can establish a certain look independent of the devices. this will affect how highlights and shadows are balanced against each other, the overall contrast of the image, as well as color saturation. it basically boils down to:

"Friends don’t let friends view scene-linear imagery without an
“S-shaped” view transform." [0]

which is by the way a good reference to read up on, with comparison images between linear, naive gamma mapped, and images with s-curves applied to them.

in darktable we try to match approximately how vendors process their images by guessing the basecurves visually/manually. if your camera is not supported yet or you don't like the result or just want to play with it, there is also a way to measure the basecurve from a raw and jpg pair.

 

measuring the basecurve

you'll need a raw file and a corresponding jpg from your camera. from this we can try to guess the basecurve applied in the jpg. credits for the initial version of this tool go to torsten bronger.

if you think about it it's surprising that we can get useful information from this, since there is a lot of processing done on chip on your raw data to create the jpg. for example there is white balancing, a color matrix, and then the basecurve (plus a lot more which we're conveniently going to ignore). actually the tool we use to solve for all these things at the same time is ignorance, so for best results you should make sure your input image makes the impact of those factors minimal:

  • take a blurry shot, to rule out effects of sharpening, noise, and denoising.
  • try to get as much dynamic range as you can, blacks and blown highlights (you probably need to go one or two stops over what you think you need to make the raw overexpose, too).
  • Shoot using the camera's default image style (often called "standard" not "neutral") if you have special image filters, hipster modes, dynamic light enhancement things in your camera, switch them off (such as `Nikon Active D-Light', `Canon Auto Lighting Optimizer (ALO)', `Sony Dynamic Range Optimizer (DRO)' or Samsung Smart Range, etc).
  • try to make the target appear as black and white as possible (after camera white balancing).

a suitable shot might look like this user contributed shot here:

example

you will need a git checkout of darktable and build and run the tool as follows. note that you don't need to compile darktable itself from git, also you don't need most build dependencies. in the directory of your local git checkout, navigate to the tools/basecurve subdirectory and execute a few commands:

cd tools/basecurve
make
dcraw -6 -W -g 1 1 -w input.raw

# now create the jpg by either /one/ of the two:
# use thumbnail
dcraw -e input.raw && mv inputraw_thumb.ppm inputjpg.ppm
# use jpg you took in raw+jpg mode
convert input.jpg inputjpg.ppm

# create basecurve
./basecurve input.ppm inputjpg.ppm
gnuplot plot
evince basecurve.pdf

that should look something like

basecurve-0

basecurve-1

where the last plot compares the green channel to the fitted spline which will be exported for use in darktable. to improve the fit you can give a third parameter to the basecurve tool to set the number of nodes. 20 is the maximum.

 

using your measured curve in darktable

to use your newly created curve, all you need to do is to copy/paste the output of the script above into a shell. beware, you're altering your database directly! please back it up first!

it might look something like:

echo "INSERT INTO presets VALUES('measured basecurve','','basecurve',2,X'00000000c2b8d03a2c113f3bfbf7fe3a2c11bf3cc304b33c7c36a13d21750b3e2c113f3e1577ba3ec496ba3e62e7213f7c36213ff4135c3f0000803f0000803f0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000020000000000000000000000',1,X'00000000180000000000C842000000000000000000000000000000000000000000000000000000000000000000000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F00000000000000000000803F0000803F',7,0,'','%','%','%',0.0,51200.0,0.0,10000000.0,0.0,100000000.0,0.0,1000.0,0,0,0,0,2);" | sqlite3 ~/.config/darktable/library.db

if you restart darktable after doing that, you should have a new preset by the name you gave above (`measured basecurve') in the basecurve module. enjoy!

 

references

[0] Cinematic Color
From Your Monitor to the Big Screen
A VES Technology Committee White Paper
Oct 17, 2012

21 thoughts on “about basecurves

  1. Hello. Are you sure in script? How is it possible? I think “gnuplot plot” has to have the options or file. Part “# create basecurve” makes me feel depressed and doesn’t work..

  2. Unfortunately, no. May be my English doesn’t allow me to understand.. I couldn’t find directory “tools/basecurve” and file “plot”

  3. Exactly! Because for me “git” is the bad man like villain.)) I’ll try again. Thank you.

  4. Thanks a lot for this tutorial!
    The above code inserts a locked preset into darktables database. And the apply-presets are somehow weird. How can i insert an editable preset?

    • ok, i somehow solved the problem by editing the database directly with sqlitebrowser.

  5. Thanks for this tip, and I’ll give it a try. Are you interested on samples for any cameras to get these curves? EOS 70D and 30D over here

    • well that is a good question.. i’m going to say yes, i’m interested in your data. but we’re not yet sure what to do with it. not sure it is a good idea to maintain per-camera curves, sometimes these are the same for many cameras of a certain vendor. also there are questions how to present that in the gui if the list is 100s of camera models long.

      i guess we’ll see once we have a couple of them (probably best to send it to the developers mailinglist).

  6. Does the base curve allow the full brightness range of the RAW file to be mapped into the limited brightness range of a JPG?

    I would love to be able to run a second base curve so camera independent profiles could be created to mimic the soft clipping of film stock or to allow some HDR like processing.

    • our curves don’t clip, they are extrapolated for values > 1 (using some exponential approximation of the behaviour in the last 10 percent of the brightness levels below 1 or similar).

      so yes, curves in dt (shaper curves in color profiles, basecurve, and tonecurve) map the full hdr range [0,\infty) to [0, \infty). you're only just looking at the [0,raw range] x [0,1] range in the gui.

  7. I bought an Olympus OM-D E-M1 and can edit RAW files but am not sure if Darktable has full RAW support for the E-M1. Also I notice there are no denoise profiles for the E-M1 and wonder how I can contribute those?

    • These are hard to understand instructions. How do you take photos that are both under and over exposed at the same time? I can get one but neve both.

    • Taking a picture that has both over and under-exposure isn’t that hard if you have a flash and a doorway:

      Have a half open door.
      Have the flash illuminate a wall behind that door.
      Place yourself on the other side.
      Aim so half the image is covered by the door, the other half sees the wall behind the door.
      Set exposure so the door will be under-exposed.
      Make the flash bright enough to over-expose the wall, half a step over should be enough.
      Have manual foxus so neither the door or the wall behind it are in focus.
      Shoot.

      You should get it after a few tries.

  8. I’ve tried w/ my coolpix P330 and at first glance using a custom base curve seems better.

    Thanks!

  9. The problem with DT is /when/ the basecurve module is applied. The mentioned “S-shaped” view transform in a cinematic color pipeline takes place at the very end, to make viewing images on a low dynamic range display meaningful.

    I.e. all color corrections, shadow/highlight balancing, etc., take place in a linear space, /before/ this view transform is applied.
    But DT applies the basecurve at the /beginning/ of the pipe, pushing everything into a non-linear color space on which all subsequent modules operate. That’s the opposite of what the quoted “Cinematic Color” paper suggests.

    I pointed this out several times on the list. To really make DT a linear image processing tool, you need to disable the basecurve completely and apply it, at the end of the pipe, by other means (because there is no way to reorder the modules in DT, for the user, at this time).

    • probably the cited paper isn’t a very good fit. i just liked the quote, because it makes a strong point about s-curves without really explaining why tf you want them. i can understand if people don’t like the overly contrasty look that you get on the big screen at times.

      that said, dt works in `perceptually linear’ Lab (with some hdr/out of gamut extension), so all the linear rgb considerations only hold up to a point.

      i even still have a branch that applies the basecurve as part of the output color profile module, must be years old by now.

      major concerns about that are:
      - can’t move the basecurve because of backward compatibility
      - don’t want to move it after colorout since that is output display dependent
      - needs to work in rgb, so the output color profile module could do Lab->rgb (standardised)->curve->output profile rgb (device dependent)

      we can experiment with that again after 1.4, maybe just introducing a basecurve2 module with a different position in the pipe (would be device dependent output rgb, but good enough for a test).

      the profiled curves would still be valid as we are ignoring the color matrix.

  10. > Shoot using the camera’s default image style (often called “standard” not “neutral”)

    I’m curious about the above. Presumably you should shoot with the curve you want to replicate. Or am I missing something?

    For most cameras the default settings are a bit cheesy with over saturated colours loss of detail in highlights etc. Having the neutral base curve would be great to build further tweaks on.

    • good point about the neutral base curve.. lately i’ve become fond of using the canon base curve i measured for my 5d mark 2, but remove all the vertices from it but the three leftmost ones. this effectively turns it into a linear curve, but pulls down the blacks a bit (much like the generic `dark contrast’ preset, but looks a little better imo).

      often this looks a lot more neutral and realistic, less eye-popping and oversaturated than the in-camera base curve. the small bump in the blacks i justify with sensor noise which cannot decrease dark values below 0, so it effectively biases dark areas to be a little too bright, resulting in washed out blacks. i just want to pull those down to counteract that effect.