Questions about raw formats and conversions

This is super helpful, thanks!

I would add a vote/suggestion that just about any file appear as a blank file icon - maybe this is a bad idea, but sometimes you have a random PDF or something in a directory and if they don’t appear at all, they kind of get lost.

Not sure if there is a better solution to that kind of thing.

Welcome to PhotoStructure, @bdillahu !

One of my very first PhotoStructure beta users said “Oh, this would be perfect for organizing my recipes on PDF!”… (sometimes feedback knocks you back a bit!) Conceptually, as long as PhotoStructure can render a file in the browser, it could be imported into the library, but I think there are many general file-sharing apps already available, and it would certainly pull the product in different directions. I’ll think about this more.

Thanks… for the welcome.

I was into testing with you back in 2018/2019, but life pulled me away a while.

Been looping back and catching up with things a little. You’ve come a long ways :slight_smile:

I agree that I’m not sure it’s a “best fit”… but there is a lot of overlap - my struggle is how to handle files that are associated with my pictures.

Just as an example off the top of my head - if I have a vacation, I will have a collection of pictures, but along with that I may have an itinerary I saved (which kind of makes part of the “trip history”), some PDF brochures of places we visited, whatever. How best to juggle those together. Right now I usually have my “archive” of trip stuff and my photos over in “photo land”.

Where do these collide? Not sure.

But please concentrate on v1 and v2 and v3 and just let these thoughts niggle in the back of your mind for when they are appropriate :slight_smile:

Hello! Another new user her.

Stumbled across PhotoStructure a couple of days ago, and have to say it looks very promising!

Just a bit disappointed to learn that it doesn’t support .cr3 - at least yet - but I hope that will change soon.

Also, would like to add my support for .pdf’s - another, more professional usecase is that model and property release forms may need to follow the photos into storage (the photos may be commercially unusable without these legal forms).

1 Like

Good news, everyone!

I’m going to ship an initial alpha release with the older version of LibRaw that I’ve done testing with, but during the alpha, I’ll update to 0.20.2 which includes the .cr3 support (and GoPro, and Fujifilm, and a bunch of other goodness!), so .cr3 support is coming soon!

1 Like

Thanks, that’s great news!

An added bonus to the .cr3 support:

The Fujifilm X-Pro2 with compressed raws is my preferred camera / file format when just having fun with a camera, so this was going to be my next question. I’m very happy to see that fujifilm compressed raws is included in that release! :slight_smile:

2 Likes

@mbrakes can you share an example Fujifilm X-Pro2 image with me so I can verify the mimetype is correct and that LibRaw is parsing it properly?

The new v1.0.0-alpha.2 (hopefully going to be released today or tomorrow) contains the latest released version of LibRaw, including .cr3 support. More details here: https://forum.photostructure.com/t/v1-0-0-alpha-1-is-out-alpha-testers-assemble/369

Hi Matthew!

Below you’ll find a WeTransfer download link to an X-Pro2 .RAF file (expires within one week)

It is almost certainly a raw file of the losslessly compressed type, as I’ve had this as the default setting on my camera since I bought it :slight_smile:

X-Pro2 RAW file (.RAF)

1 Like

@mbrakes looks like the new LibRaw works! :tada:

Indeed - hooray! :partying_face:

I am running alpha 2 and it looks like libraw is still generating pinkish images from my sony RX100 VI.
Can you let me know what the command line format is so I can run some tests in the docker container. I’m assuming you are using the dcraw_emu command?

If you were using an earlier version, I didn’t make PhotoStructure alpha.2 automatically rebuild previews (as it seemed that only a few people saw this issue). If you click “resync this asset” while viewing any pinked images, it’ll force-rebuild the previews.

I’ll get you that command as soon as I’m at my desk. If you DM or email me an example raw image, that’d be great (nothing with anything private, please!)

The command is dynamic based on the image. Here’s the code:

const TiffOutput = ["-T"]
const WriteToStdout = ["-Z", "-"]
// -o [0-6]  Output colorspace (raw,sRGB,...):
const Colorspace = ["-o", "1"]
// disable auto-flip/auto-rotate, because libraw is sometimes wrong:
const IgnoreOrientation = ["-t", "0", "-j"]

// this is already time()d by sharpReadable
export async function dcraw_emu(src: PosixFile): Promise<Readable> {
  const start = Date.now()

  const srcDim = await dimensions(src)
  if (srcDim == null) {
    return logger.throw(
      "Cannot decode RAW " +
        src +
        ": no EXIF dimensions." +
        DoNotSendErrorFlag +
        NonRetriableErrorFlag
    )
  }

  const fitDim = ImageSize.largestFit().outputSize(srcDim)

  const maybeHalfSize: string[] = []

  if (fitDim != null && 4 * dmegapixels(fitDim) < dmegapixels(srcDim)) {
    logger.debug("Large original source: using -h")
    maybeHalfSize.push("-h")
  }

  const cmd = await dcrawEmuNativePath()
  const args = [
    ...TiffOutput,
    ...WriteToStdout,
    ...Colorspace,
    ...maybeHalfSize,
    ...IgnoreOrientation,
    ...Settings.dcrawEmuArgs.values,
    src.nativePath
  ]
  const timeout = 5 * minuteMs
  const opts = {
    encoding: "buffer",
    timeout,
    maxBuffer: 250 * MB
  }
  logger.debug("dcraw_emu()", { cmd, args, opts })
  const childProc = await execFile(cmd, args, timeout, opts)

...

If you run photostructure info --debug $rawfile or photostructure sync-file --debug $rawfile you’ll see the dcraw_emu() log line emitted to stdout (along with screenfuls of other verbosity).

Looks like an option is getting fed to dcraw_emu that isn’t expected (-Z). Additionally, there’s a solitary “-” as the third index that looks like it would also be a problem.

Below are the relevant parts from the output of the command you suggested

2021-04-05T23:53:40.367Z info-4124 debug ChildProcess execFile() { command: ‘/usr/bin/dcraw_emu’,
args:
[ ‘-T’,
‘-Z’,
‘-’,
‘-o’,
‘1’,
‘-t’,
‘0’,
‘-j’,
‘-q’,
‘0’,
‘-w’,
‘/Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG’ ],

2021-04-05T23:53:40.374Z info-4124 warn SharpReadable strategy failed for /Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG: Error: readableToFile() failed: Problem from /Lightroom/Volume02/2019_catalog/2019/2019-08-15/DSC08313.DNG: Unknown option “-Z”.⁶⁷

Well dang, -Z - should redirect the output to stdout (this is so I can avoid disk I/O, which is horrible on Windows).

$ tools/linux-x64/libraw/dcraw_emu 
dcraw_emu: almost complete dcraw emulator
Usage:  tools/linux-x64/libraw/dcraw_emu [OPTION]... [FILE]...
...
-Z <suf>  Output filename generation rules
          .suf => append .suf to input name, keeping existing suffix too
           suf => replace input filename last extension
          - => output to stdout

Are you running v1.0.0-alpha.2? What OS?

(Ah: I believe you’re running alpha.1, which relied on the system’s older version of dcraw_emu. PhotoStructure needs the latest and greatest, so in alpha.2, it’s actually included in the build, so upgrading to alpha.2 should address this).

(I just tested the binary on Ubuntu 12.04 x64 and it rendered a .TIFF from a .DNG: if you want to email me an example .DNG, I can see what’s going on).

Docker container on ubuntu20

/ps/app $ photostructure -V
1.0.0-alpha.2

Emailing you a file. Also, another possible culprit is that the image was originally an .AWR from a sony RX100VI and was converted to a DNG during lightroom import.

I suspect the previews were generated from a prior version (alpha.1 or v0.9.1): when I just imported it into my scratch library, and the colors look correct.

If you choose “Resync this asset” from the ⋮ menu, PhotoStructure should rebuild the previews and the pink cast should go away. If it doesn’t, tell me!

Oops: I just saw you’re using Docker, not node. I’ll verify that Docker isn’t picking up the incorrect dcraw_emu.

DARN. I just saw that the Dockerfile was still installing the older version from the Alpine apk repo.

I’ll update the Dockerfile to build libraw natively for Alpine. This will be fixed in the next build (probably 1.0.0-beta.1).

Thanks again for the heads-up, @Thomas_Hollier !

I have hundreds of photos (lossless compressed) from at least 3 Fuji cameras that show up as a black image. They were imported under version 0.9 but still not viewable with the 1.0.0 release. I did try re-sync’ing the asset but nothing changed. Should Fuji lossless compressed RAW files be displaying here? I’ve downloaded the original (from within Photostructure) and they open fine in Capture One, Affinity etc. Thanks.