Questions about raw formats and conversions

[Wanderweger]

Hi, another newbie post (sorry).

Can you help me with three questions about how raw images are handled.

I have a directory containing Canon CR2 images and have auto management on. Am I correct that Photostructure just copies the CR2 images into the new library. Assuming that is the case, then is there a jpg image also created somewhere for the photostructure viewer (and if so where), or does it render on the fly?

Is there a list of formats that photostructure supports.

If there is an image in my original library that is not supported does auto file management just ignore it. I’m hoping to collect everything in a single place and am a little concerned that unrecognized files will be ‘lost’ when creating the new library, but not sure whether there’s a solution to this.

Thanks so much.

[mrm]

@Wanderweger

another newbie post (sorry)

No apologies, that’s what this forum is for!

@Wanderweger

Am I correct that Photostructure just copies the CR2 images into the new library.

Yes.

@Wanderweger

is there a jpg image also created somewhere for the photostructure viewer (and if so where), or does it render on the fly?

The thumbnails and preview images are JPEGs that live in your PhotoStructure library. See https://photostructure.com/faq/library/#where-are-my-librarys-database-settings-and-preview-files

Rendering is all done at import time to make browsing fast.

@Wanderweger

Is there a list of formats that photostructure supports?

V0.9.1 uses dcraw, which supports many raw formats.

v1.0 (which I hope to have alpha releases ready in a week or two) switched to libraw, which supports many more recent raw formats. The only raw format that I’m aware of that it doesn’t support yet is cr3, which they say will be ready soon.

@Wanderweger

If there is an image in my original library that is not supported does auto file management just ignore it.

Currently, yes, but thinking about it more, it seems like it would be better to import it if possible: most raw images have fairly large embedded JPEG images that I can use for full-screen. I’ll think about this some more: thanks for bringing this to my attention!

[mrm]

@Wanderweger

Is there a list of formats that photostructure supports?

Version 0.9.3 and 1.0.0 will import files with the following mimetypes:

  "image/gif",
  "image/jpeg",
  "image/pjpeg",
  "image/png",
  "image/tiff",
  "image/webp"

as well as the following raw types:

const librawSupportedMimeTypes = new Set([
  // TODO: add hasselblad, leaf, leica, phase one
  // See https://www.dechifro.org/dcraw/#cameras
  "image/x-adobe-dng",
  "image/x-canon-cr2",
  // "image/x-canon-cr3", // TODO: these aren't supported by libraw yet
  "image/x-canon-crw",
  "image/x-epson-erf",
  "image/x-fuji-raf", // < both are found in the wild
  "image/x-fujifilm-raf", // < both are found in the wild
  "image/x-kodak-dcr",
  "image/x-kodak-k25",
  "image/x-kodak-kdc",
  "image/x-minolta-mrw",
  "image/x-nikon-nef",
  "image/x-nikon-nrw",
  "image/x-olympus-orf",
  "image/x-panasonic-raw",
  "image/x-panasonic-rw2",
  "image/x-pentax-pef",
  "image/x-samsung-srw",
  "image/x-sigma-x3f",
  "image/x-sony-arw",
  "image/x-sony-sr2",
  "image/x-sony-srf"
])

If HEIF support is installed, image/heif and image/heic mimetypes are also imported.

[Wanderweger]

Thanks. Thinking along these lines, there are other image-related formats that I’d also like to have imported (just copied, not converted) so they are all saved in once place. For example, I have an Insta360 camera that generates INSV files for 360 panoramic info in addition to the DNG files. Would it make sense to include these to the import list (or even better, an option to add files of a custom type to the config file)?

[mrm]

@Wanderweger

an option to add files of a custom type to the config file)?

Hmm. Let me think about that.

There are a bunch of filters that are applied to determine if a file is relevant for importing. One of those is dimensions. If ExifTool can extract dimensions from the .insv, then event cleared that hurdle.

The code that copies files into the library needs to be able to extract a “capture at” time in order to know what subdirectory it should copy it into. Do these files have always that metadata? (You can use the info tool to see what PhotoStructure thinks).

Can you send me a DM or email with an example file set (the insv and dng)?

Ideally the files would have matching metadata to merge them into the same asset, and then a different variant (something PhotoStructure knows how to render, like the dng) can be picked as the “primary” or “shown” asset file.

Is it ok to copy a file into your library even if it can’t be rendered by PhotoStructure? (This is what you’re asking, right?)

[Wanderweger]

Thanks. I’ve sent you an email with some example files.

Just copying would be great - I don’t expect Photostructure to render the file. However, a bonus would be to display the file name and/or capture time in the ‘image square’ so at least the user would know it exists?

[mrm]

Thanks for sharing those examples!

I’ve just rebuilt PhotoStructure’s internal list of supported file extensions using the latest version of ExifTool, and the .insp extension was recently added

With this change, the next version of PhotoStructure will properly import these files.

(They’ll be rendered as distorted panoramas, though. )

[Wanderweger]

This is super helpful, thanks!

[bdillahu]

I would add a vote/suggestion that just about any file appear as a blank file icon - maybe this is a bad idea, but sometimes you have a random PDF or something in a directory and if they don’t appear at all, they kind of get lost.

Not sure if there is a better solution to that kind of thing.

[mrm]

Welcome to PhotoStructure, @bdillahu !

One of my very first PhotoStructure beta users said “Oh, this would be perfect for organizing my recipes on PDF!”… (sometimes feedback knocks you back a bit!) Conceptually, as long as PhotoStructure can render a file in the browser, it could be imported into the library, but I think there are many general file-sharing apps already available, and it would certainly pull the product in different directions. I’ll think about this more.

[bdillahu]

Thanks… for the welcome.

I was into testing with you back in 2018/2019, but life pulled me away a while.

Been looping back and catching up with things a little. You’ve come a long ways

I agree that I’m not sure it’s a “best fit”… but there is a lot of overlap - my struggle is how to handle files that are associated with my pictures.

Just as an example off the top of my head - if I have a vacation, I will have a collection of pictures, but along with that I may have an itinerary I saved (which kind of makes part of the “trip history”), some PDF brochures of places we visited, whatever. How best to juggle those together. Right now I usually have my “archive” of trip stuff and my photos over in “photo land”.

Where do these collide? Not sure.

But please concentrate on v1 and v2 and v3 and just let these thoughts niggle in the back of your mind for when they are appropriate

[mbrakes]

Hello! Another new user her.

Stumbled across PhotoStructure a couple of days ago, and have to say it looks very promising!

Just a bit disappointed to learn that it doesn’t support .cr3 - at least yet - but I hope that will change soon.

Also, would like to add my support for .pdf’s - another, more professional usecase is that model and property release forms may need to follow the photos into storage (the photos may be commercially unusable without these legal forms).

[mrm]

@mbrakes

Just a bit disappointed to learn that it doesn’t support .cr3 - at least yet - but I hope that will change soon.

Good news, everyone!

I’m going to ship an initial alpha release with the older version of LibRaw that I’ve done testing with, but during the alpha, I’ll update to 0.20.2 which includes the .cr3 support (and GoPro, and Fujifilm, and a bunch of other goodness!), so .cr3 support is coming soon!

[mbrakes]

Thanks, that’s great news!

An added bonus to the .cr3 support:

The Fujifilm X-Pro2 with compressed raws is my preferred camera / file format when just having fun with a camera, so this was going to be my next question. I’m very happy to see that fujifilm compressed raws is included in that release!

[mrm]

@mbrakes can you share an example Fujifilm X-Pro2 image with me so I can verify the mimetype is correct and that LibRaw is parsing it properly?

The new v1.0.0-alpha.2 (hopefully going to be released today or tomorrow) contains the latest released version of LibRaw, including .cr3 support. More details here: https://forum.photostructure.com/t/v1-0-0-alpha-1-is-out-alpha-testers-assemble/369

[mbrakes]

Hi Matthew!

Below you’ll find a WeTransfer download link to an X-Pro2 .RAF file (expires within one week)

It is almost certainly a raw file of the losslessly compressed type, as I’ve had this as the default setting on my camera since I bought it

X-Pro2 RAW file (.RAF)

[mrm]

@mbrakes looks like the new LibRaw works!

[mbrakes]

Indeed - hooray!

[Thomas_Hollier]

I am running alpha 2 and it looks like libraw is still generating pinkish images from my sony RX100 VI.
Can you let me know what the command line format is so I can run some tests in the docker container. I’m assuming you are using the dcraw_emu command?

[mrm]

If you were using an earlier version, I didn’t make PhotoStructure alpha.2 automatically rebuild previews (as it seemed that only a few people saw this issue). If you click “resync this asset” while viewing any pinked images, it’ll force-rebuild the previews.

I’ll get you that command as soon as I’m at my desk. If you DM or email me an example raw image, that’d be great (nothing with anything private, please!)