Image Simulation

Ask the Community
1

After using various image simulation tools (which shall remain unnamed), I have come to the conclusion that they are in general a) a really useful tool for quickly conveying if a system is ‘good’ or ‘bad’ and b) extraordinarily confusing and opaque.

Half the time it is not totally clear what the image simulation is actually doing, and thus the faith I have in it is pretty low, which is a shame for how useful the tool can be. I’m curious to hear other peoples thoughts. Do you all use image simulations to demonstrate if a system is ‘good’ or not? If so do you recommend one tool over another, or do you rely on other metrics to assure yourself that the system truly meets requirements, and the image simulation is just aesthetic?

2

I think the image simulation tools in a lot of commerical software suffer from choices which trade some performance gain for reduced accuracy, often to an unnecessary degree. For example, one of the more popular programs defaults to 64x64 array sizes. It is unclear if this is before or after padding; if before, that is really the edge of acceptable. For example, secondary/fifth order spherical aberration cannot be Nyquist sampled on that grid, and a circular aperture will non-negligibly depart from a truthful representation of a circle. If the latter, it is wholly unacceptable.

I find that the knobs and dials made available are not very well explained for a subject matter expert to tune them, and at the same time there are too many for someone who is more of a raybender to handle.

Then there is the issue of a truthful representation of the system. Many assumptions or modeling choices are needed to produce faithful image simulations beyond the ancillary details of grid sizes, etc. These cover things like how to represent polychromatic light (number of wavelengths, weighting, placing all the PSFs on equal grids, …), or the fill factor of the pixel and other elements of the system transfer function that are related to the detector. As an example of a modern camera, the Fujifilm GFX-50 has a linear fill factor on its pixels of ~65%. A default choice model with 100% fill factor will produce images which have less aliasing than the real thing. This is not a knob even presented to the user of the program. Or what to do about the fact that imaging systems, generally, are only locally shift invariant.

This is all to say, image simulation is a lot more complicated than ray tracing. I think making the ‘easy for a non subject matter expert to use, but highly accurate’ image simulation program is very difficult.

I think it is fairly inappropriate to use image simulation to verify requirements, but it is very effective at showing what the system “means” in nontechnical terms.

3

Which really hits home when you adjust the settings to get back that accuracy and deal with the time it takes to run the simulation! Granted, its no easy task to run a fully featured image simulation, so I understand why, but overall I don’t think this trade off should be as opaque as it currently is.

Thank you. This point is so hard to communicate because of all the details that have a large impact on the output! I have struggled to feel comfortable with any large simulation tool that is not extremely transparent about its methodologies and explains its “knobs” in a thorough manner.

I would also add that a good image simulation can capture the qualitative aspects of the system that aren’t well represented in the other more technical metrics. This of course comes with the caveat that producing the simulation is very difficult for all the reasons listed :slight_smile:

4

I think you will struggle to get the commercial lens design programs to one nine of accuracy. Two nines is probably impossible on the basis of knowledge errors alone, but ditto on the basis of approximations in their modeling.

I don’t think fully featured image simulation is that difficult, or has to be slow; this sample I made for elsewhere does the simulation and reverses the blur with a Wiener filter, plus graphic output, in 19 lines of python. A non-airy disk PSF, whether it be loaded from a measurement or from a model, is << 10 lines of code to change to. The same for polychromatic. As you scale this API from toy monochromatic problems to real ones with warts, the information density makes it something you have to read slowly, and choices about grid parameters, etc, require careful thought. The forward pass of that model takes 895 ms on a laptop. It has between four and five nines of accuracy, which does not erode by using a less simplistic PSF or set of parameters.

I would also add that a good image simulation can capture the qualitative aspects of the system that aren’t well represented in the other more technical metrics. This of course comes with the caveat that producing the simulation is very difficult for all the reasons listed

You can use image simulation to form a model of data for measurements, say an image of a slanted edge to measure MTF from, or a PSF on a detector to measure FWHM or 1/e^2, etc. Unless you are in a heavily oversampled (say, Q>6) regime, I think the accuracy is not good enough. And large Q runs slowly.

5

I think this is a great insight. If you can model an image simulation of something well expect, MTF for example, and you get similar results for the image simulation of the MTF and the calculated MTF, that provides a nice sanity check regarding how the image analysis was set up.

The real caveat here is that to setup the image simulation, much less the deeper check to verify the simulation, require deep knowledge about not only the optical designa dn system, but about the underlying programming as well and how the calculation is being performed. In your prysm example (great demo btw), it is fairly transparent what is being done. Someone could easily look through the underlying code to understand exactly what is going into an image simulation and therefore understand how to set it up, but in other software scenarios that may not be easily accomplished (or may be outright impossible to find).

All this suggests that it is essential to a) understand the underlying algorithms and b) to be really intentional and careful when setting up an image simulation. As nice as it would be, it in fact is not a tool that can be trivial setup for any system and the results trusted.

6

Verifying image simulation with measurements (like MTF) is not really trivial, either. Perhaps the most interesting test cases are aliased systems, which can violate assumptions made in modeling. Modeling aliased systems is not really a special case, but it is very difficult. In my experience, the best algorithm can depend on the data, so defaults must be chosen based on a criteria like “usually among the best” or “rarely among the worst.” Placing qualifiers on the quality of the result usually does nothing to instill a great sense of accuracy in the user. This is not to say it can’t be done. If you read the docstring for the conv method, it gives a terse outline of how the algorithm handles gridding to maintain accuracy always. Beyond this, the only thing you really have to do if you want the last word in accuracy is do things outside the natural order, deferring interaction with aliased data until the last moment possible. This prevents the aliases from corrupting the lower frequencies during intermediate steps.

On the measurement side - measurement algorithms are complicated, too. It would be foolish to think one can implement a suitably robust piece of code quickly since the improvements are usually made in response to its failure on some data, and exposure to that data comes with time. This is also not to say it is impossible to develop your own high quality measurement algorithms.

I’ve cross compared prysm to MTF mapper simulating a few different targets, which all have different aliasing properties. Frans and I have shown that the image simulation tool in MTF-mapper itself1, and prysm2 hit those without special user intervention to the 1e-6 level of accuracy, which is a few digits better than, say, a Trioptics or Optikos MTF bench will even report their measurements to.

prysm’s API is designed to express ideas more or less in English, with modeling/simulation parameters available as arguments. The default argument cases are designed with a number of clever tricks to maintain accuracy. This means some of its internal algorithms are more complicated than the naive approach, and may have an unexpected time complexity, for better or worse.

1 MTF-Mapper uses a discrete point non-gridded method which is more akin to rayleigh-sommerfeld, with some differences

2 prysm is a classical numerical methods approach

Testing your code in MTF measurement scenarios is probably more punishing than usual scenes, since the targets are all going to be aliased where a natural scene may have little to no appreciable aliasing. The benefit is that you know truth in the analytic cases. I suppose a natural scene has a concept of truth as well, since you can compare the Fourier spectrum, but simply acquiring an accurate Fourier spectrum of an image is harder than fft(im) which is something probably not all that well appreciated by most engineers and scientists.

7

I broadly agree, and I certainly agree that verifying image simulation with measurements is not trivial. I suppose the heart of the matter is I would argue an image simulation can never show what the true image will look like as well as an actual image from the real system.

Prysm does an excellent job of making it easy, and understandable in human language, to setup an image simulation and it takes into account a lot of parameters when performing the simulation, from my limited experience it is now my software of choice for image simulations.

However, my point more so is how do I know any of these simulations are actually ‘good’ or ‘correct’? This is kind of the allegory of the cave situation. At best we are creating what we hope are accurate models in software to describe the infinitely complex (if we grind down to the extreme minutia) of a real world system. Therefore, we can never fully capture what physically will happen, with no effort on our part, in the real world by using a simulation. Instead we hope we cover our bases well enough and that the simulation does a pretty good job of showing how the image will form. But, how do I ever know if the image simulation is right without then getting my actual system and taking a real picture and comparing the two?

I don’t have a good answer, and you have done a great job expanding on this idea and have given me a lot to think about so thank you!

8

However, my point more so is how do I know any of these simulations are actually ‘good’ or ‘correct’? This is kind of the allegory of the cave situation. At best we are creating what we hope are accurate models in software to describe the infinitely complex (if we grind down to the extreme minutia) of a real world system. Therefore, we can never fully capture what physically will happen, with no effort on our part, in the real world by using a simulation. Instead we hope we cover our bases well enough and that the simulation does a pretty good job of showing how the image will form. But, how do I ever know if the image simulation is right without then getting my actual system and taking a real picture and comparing the two?

I think a lot of that comes down to experience for choosing simulation parameters, e.g. number of samples across a pupil. You can always test for asymptotes though. Most error sources decrease with larger arrays (aliasing, the PSF not ever reaching zero, …). You can watch for convergence by running your simulation at different grid parameters. This is a simialr idea to hyperparameter tuning in the machine learning community.

You can test your simulation for analytic cases. For example, a synthetic image of a slanted edge as taken by a lens with a circular aperture and rectangular pixels of known fill factor has an MTF which is the product of the diffraction limited MTF (a large arccos expression) and sincx, sincy. You can verify that with a measurement on the synthetic image, using a measurement routine you trust. I personally trust MTF Mapper, to at least 4 or 5 nines.

Phase retrieval in the modern Fienup group approach – via nonlinear optimization – also can tell you about the quality of your simulation based on model-data consistency. You can give it parameters for things like number of wavelengths (say) and have it tell you when it is good enough. Or ask it what “Q” is for your data. Alden Jurling wrote some papers on that idea.