Questions about raw formats and conversions

Support
21

The command is dynamic based on the image. Here’s the code:

const TiffOutput = ["-T"]
const WriteToStdout = ["-Z", "-"]
// -o [0-6]  Output colorspace (raw,sRGB,...):
const Colorspace = ["-o", "1"]
// disable auto-flip/auto-rotate, because libraw is sometimes wrong:
const IgnoreOrientation = ["-t", "0", "-j"]

// this is already time()d by sharpReadable
export async function dcraw_emu(src: PosixFile): Promise<Readable> {
  const start = Date.now()

  const srcDim = await dimensions(src)
  if (srcDim == null) {
    return logger.throw(
      "Cannot decode RAW " +
        src +
        ": no EXIF dimensions." +
        DoNotSendErrorFlag +
        NonRetriableErrorFlag
    )
  }

  const fitDim = ImageSize.largestFit().outputSize(srcDim)

  const maybeHalfSize: string[] = []

  if (fitDim != null && 4 * dmegapixels(fitDim) < dmegapixels(srcDim)) {
    logger.debug("Large original source: using -h")
    maybeHalfSize.push("-h")
  }

  const cmd = await dcrawEmuNativePath()
  const args = [
    ...TiffOutput,
    ...WriteToStdout,
    ...Colorspace,
    ...maybeHalfSize,
    ...IgnoreOrientation,
    ...Settings.dcrawEmuArgs.values,
    src.nativePath
  ]
  const timeout = 5 * minuteMs
  const opts = {
    encoding: "buffer",
    timeout,
    maxBuffer: 250 * MB
  }
  logger.debug("dcraw_emu()", { cmd, args, opts })
  const childProc = await execFile(cmd, args, timeout, opts)

...

If you run photostructure info --debug $rawfile or photostructure sync-file --debug $rawfile you’ll see the dcraw_emu() log line emitted to stdout (along with screenfuls of other verbosity).

22

Looks like an option is getting fed to dcraw_emu that isn’t expected (-Z). Additionally, there’s a solitary “-” as the third index that looks like it would also be a problem.

Below are the relevant parts from the output of the command you suggested

2021-04-05T23:53:40.367Z info-4124 debug ChildProcess execFile() { command: ‘/usr/bin/dcraw_emu’,
args:
[ ‘-T’,
‘-Z’,
‘-’,
‘-o’,
‘1’,
‘-t’,
‘0’,
‘-j’,
‘-q’,
‘0’,
‘-w’,
‘/Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG’ ],

2021-04-05T23:53:40.374Z info-4124 warn SharpReadable strategy failed for /Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG: Error: readableToFile() failed: Problem from /Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG: Unknown option “-Z”.⁶⁷

23

Well dang, -Z - should redirect the output to stdout (this is so I can avoid disk I/O, which is horrible on Windows).

$ tools/linux-x64/libraw/dcraw_emu 
dcraw_emu: almost complete dcraw emulator
Usage:  tools/linux-x64/libraw/dcraw_emu [OPTION]... [FILE]...
...
-Z <suf>  Output filename generation rules
          .suf => append .suf to input name, keeping existing suffix too
           suf => replace input filename last extension
          - => output to stdout

Are you running v1.0.0-alpha.2? What OS?

(Ah: I believe you’re running alpha.1, which relied on the system’s older version of dcraw_emu. PhotoStructure needs the latest and greatest, so in alpha.2, it’s actually included in the build, so upgrading to alpha.2 should address this).

(I just tested the binary on Ubuntu 12.04 x64 and it rendered a .TIFF from a .DNG: if you want to email me an example .DNG, I can see what’s going on).

24

Docker container on ubuntu20

/ps/app $ photostructure -V
1.0.0-alpha.2

Emailing you a file. Also, another possible culprit is that the image was originally an .AWR from a sony RX100VI and was converted to a DNG during lightroom import.

25

I suspect the previews were generated from a prior version (alpha.1 or v0.9.1): when I just imported it into my scratch library, and the colors look correct.

If you choose “Resync this asset” from the ⋮ menu, PhotoStructure should rebuild the previews and the pink cast should go away. If it doesn’t, tell me!

Oops: I just saw you’re using Docker, not node. I’ll verify that Docker isn’t picking up the incorrect dcraw_emu.

26

DARN. I just saw that the Dockerfile was still installing the older version from the Alpine apk repo.

I’ll update the Dockerfile to build libraw natively for Alpine. This will be fixed in the next build (probably 1.0.0-beta.1).

Thanks again for the heads-up, @Thomas_Hollier !

27

I have hundreds of photos (lossless compressed) from at least 3 Fuji cameras that show up as a black image. They were imported under version 0.9 but still not viewable with the 1.0.0 release. I did try re-sync’ing the asset but nothing changed. Should Fuji lossless compressed RAW files be displaying here? I’ve downloaded the original (from within Photostructure) and they open fine in Capture One, Affinity etc. Thanks.

28

Apologies!

If you send me an email with an example raw image that isn’t rendering correctly I can take a look at what’s going on.

30

(I deleted the link to the shared image as soon as I downloaded it, just in the interests of privacy).

So, at least on PhotoStructure for Node on Ubuntu 20.04.2 LTS on x64, it seems to import ok:

image
image1690×804 112 KB

@joshanno what OS are you using, and what version and edition of PhotoStructure?

31

I’m on Mac OS Big Sur 11.4

Version 1.0.0.

I imported these with a beta version. I did try to re-import one of the images later from a different directory but I didn’t see a change. The majority of Fuji RAW display fine so I suspect it’s just the lossless compressed.

Perhaps I should delete the currently imported versions and try again?

32

Following up:

I installed a new version of 1.0.0 on a different mac entirely, re-imported from the source files, and now these images display properly. So, there is a work-around the fact that the current library doesn’t want to show them (despite being the same version). I’ve begun to re-create the library from scratch on the original Mac and it’s working as expected so far.

33

Thanks for following up—glad it’s working now!