autoemx.core.particle_statistics module

Particle size statistics: summary statistics of the equivalent diameters of particles, and their export.

The statistics use the frames of a particle size statistics measurement (SampleLedger.size_measurements) and, if PowderMeasurementConfig.include_eds_auto_particles (default), the frames searched by EDS on the same sample, before or after it, each frame under a segmentation (SampleLedger.frames, SampleLedger.segmentations). Each result is stored in the sample ledger (SampleLedger.size_results): results of other segmentations, frames or measurements are kept. For samples measured before frames searched by EDS were recorded, the particles measured by EDS with automatic particle selection (selection ‘auto’) count instead.

Independent of the microscope: used both at the end of a particle statistics measurement (EM_Particle_Finder.get_particle_stats) and to recalculate statistics from saved particle sizes (autoemx.runners.process_particle_statistics).

Files saved in the sample folder (<suffix> is optional):

• <sample_ID>_Par_sizes<suffix>.csv: ID, frame, area and equivalent diameter of each particle;

• <sample_ID>_Par_size_stats<suffix>.csv: summary statistics of the equivalent diameters;

• <sample_ID>_Par_size_distribution_hist<suffix>.<format>: size distribution plot (histogram and cumulative distributions of the equivalent diameters).

autoemx.core.particle_statistics.SIZE_STATS_MEASURE = 'Equivalent particle diameter in μm'

Description of the measure, first column of the statistics file.

autoemx.core.particle_statistics.CI_QUANTITIES = {'D10': 'D10', 'D43': 'D43', 'D50': 'median', 'D90': 'D90', 'Dv10': 'Dv10', 'Dv50': 'Dv50', 'Dv90': 'Dv90'}

key in compute_particle_size_stats()).

Type:

Statistics given a confidence interval (name

autoemx.core.particle_statistics.N_BOOTSTRAP_RESAMPLES = 200

Number of bootstrap resamples of the confidence intervals.

autoemx.core.particle_statistics.MIN_BOOTSTRAP_UNITS = 5

Minimum number of resampled units (frames, or particles without frame records) in each size band for confidence intervals: with fewer, the bootstrap underestimates the variability.

autoemx.core.particle_statistics.compute_particle_size_stats(eq_diameters_um: Sequence[float], weights: Sequence[float] | None = None) → Dict[str, int | float]

Summary statistics of particle equivalent diameters, optionally weighted.

Parameters:

• eq_diameters_um (sequence of float) – Equivalent diameters of the particles, in µm.

• weights (sequence of float, optional) – Non-negative (number) weight of each particle (see band_particle_weights()); particles of weight 0 are left out. Default: all particles count equally.

Returns:

Number-weighted: n_par_analysed (particles counted), mean, stdev (population standard deviation), median, max, min and the percentiles D10, D25, D75, D90. Volume-weighted (each particle also weighted by the volume of a sphere of its diameter, as measured by laser diffraction): D43 (mean, D[4,3]) and the percentiles Dv10, Dv50, Dv90. In µm.

Return type:

dict

Raises:

ValueError – If no diameter (or no particle of positive weight) is given.

autoemx.core.particle_statistics.edge_correction_factor(bbox_width_px: int, bbox_height_px: int, frame_width_px: int, frame_height_px: int, edge_margin_px: int) → float

Miles-Lantuéjoul correction of a particle measured in a frame where particles touching the edge are left out.

A particle is kept only if its bounding box stays more than edge_margin_px pixels from every edge of the frame, which is less likely the larger it is. Its weight is multiplied by the inverse of the fraction of frame positions where it is kept, W·H / ((W - w - 2m - 1)(H - h - 2m - 1)), so that large particles are not under-counted.

Raises:

ValueError – If the particle cannot fit in the frame without touching its edge.

class autoemx.core.particle_statistics.SizeObservations(eq_diameters_um: ndarray, areas_um2: ndarray, corrections: ndarray, scan: ndarray, frame_ids: List[str | None], particle_index: List[int | None], particle_keys: List[Tuple[str, int] | None], scans: List[ParticleSizeBand], n_bands: int)

Bases: NamedTuple

Particle sizes from the scans of a sample (size bands and frames searched by EDS), with what is needed to weight them (see size_observations()).

eq_diameters_um: ndarray

Equivalent diameter (µm), area (µm²) and frame-edge correction of each observation.

areas_um2: ndarray

Alias for field number 1

corrections: ndarray

Alias for field number 2

scan: ndarray

index in scans (size bands first, then EDS scans), or -1 for a particle measured by EDS that only shares the weight of the band of its diameter (no EDS scan recorded).

Type:

Scan of each observation

frame_ids: List[str | None]

Frame of each observation, index of its particle in the particles given (only for those), and its (frame_id, index) in its frame (only for particles of frames).

particle_index: List[int | None]

Alias for field number 5

particle_keys: List[Tuple[str, int] | None]

Alias for field number 6

scans: List[ParticleSizeBand]

Size bands of the measurement, then EDS scans.

n_bands: int

Alias for field number 8

autoemx.core.particle_statistics.size_observations(particles: Sequence[ParticleInfo], bands: Sequence[ParticleSizeBand] = (), eds_scans: Sequence[ParticleSizeBand] = (), excluded_frame_ids: Iterable[str] = (), excluded_particles: Iterable[Tuple[str, int]] = ()) → SizeObservations

Observations of particle sizes: the particles counted in the frames of the size bands and in the frames searched by EDS (eds_scans), and the particles given (measured by EDS, sharing the weight of the band of their diameter), except in the frames left out (excluded_frame_ids and frames overlapping a larger one, see overlapping_frame_ids()) and the particles left out (excluded_particles, as (frame_id, index)).

autoemx.core.particle_statistics.frames_overlap(center_um, half_size_um, centers_um, half_sizes_um) → ndarray

Whether a frame (centre and half width and height on the sample, in µm) overlaps each of other frames (arrays of centres and half sizes), beyond a touching edge.

autoemx.core.particle_statistics.size_ranges_overlap(d_ranges_um, d_range_um) → ndarray

Whether diameter ranges (array of (d_min, d_max)) overlap a diameter range, beyond a common bound.

class autoemx.core.particle_statistics.EdsSearchFrame(frame_id: str, center_um: Tuple[float, float], half_size_um: Tuple[float, float], d_range_um: Tuple[float, float], frame_width_um: float)

Bases: NamedTuple

Frame searched by EDS, on the sample: centre and half width and height (µm), and size range of its scan.

frame_id: str

Alias for field number 0

center_um: Tuple[float, float]

Alias for field number 1

half_size_um: Tuple[float, float]

Alias for field number 2

d_range_um: Tuple[float, float]

Alias for field number 3

frame_width_um: float

Alias for field number 4

autoemx.core.particle_statistics.eds_search_frames(eds_scans: Sequence[ParticleSizeBand]) → Tuple[List[EdsSearchFrame], ndarray]

Frames searched by EDS with a known position, except those left out of the statistics as overlapping another (overlapping_frame_ids()), and the equivalent diameters (µm) of the particles counted in them.

autoemx.core.particle_statistics.overlapping_frame_ids(scans: Sequence[ParticleSizeBand]) → List[str]

Frames left out of the statistics because they overlap, on the sample, a frame of a scan with an overlapping size range (which counts the same particles): of overlapping frames, the largest is kept (the first recorded, if as large). Frames of different size bands, whose size ranges do not overlap, are never compared. Frames without position are kept.

autoemx.core.particle_statistics.weighted_size_observations(particles: Sequence[ParticleInfo], bands: Sequence[ParticleSizeBand] = (), excluded_frame_ids: Iterable[str] = (), eds_scans: Sequence[ParticleSizeBand] = (), excluded_particles: Iterable[Tuple[str, int]] = ()) → Tuple[SizeObservations, ndarray, List[ParticleSizeBand], List[ParticleSizeBand]]

Observations of particle sizes (see size_observations()) and their weights, so that the statistics describe the sample: each counts its edge correction (edge_correction_factor()) over the area of the scans able to observe its size, so that each size counts in proportion to its number of particles per unit area of the sample, whatever the number of particles collected in each size band.

Returns:

• observations (SizeObservations)

• weights (numpy.ndarray) – Weight of each observation (all 1 without bands or EDS scans).

• bands, eds_scans (list of ParticleSizeBand) – The bands and EDS scans with their number_density_per_mm2 (particles they counted per mm²) and, for bands, n_eds_particles (particles measured by EDS sharing their weight).

autoemx.core.particle_statistics.bootstrap_confidence_intervals(particles: Sequence[ParticleInfo], bands: Sequence[ParticleSizeBand] = (), excluded_frame_ids: Iterable[str] = (), confidence: float = 0.95, n_resamples: int = 200, rng: Generator | None = None, eds_scans: Sequence[ParticleSizeBand] = (), excluded_particles: Iterable[Tuple[str, int]] = ()) → Dict[str, Tuple[float, float]]

Confidence intervals of the size statistics (CI_QUANTITIES), by bootstrap.

The frames of each scan (size band, or frames searched by EDS) are resampled with replacement, with their particles and area, so that both the size distribution and the number density of each scan vary as between measurements of the same sample; the statistics are then weighted as in weighted_size_observations(). Particles of a scan without frame records (and without scans, all particles) are resampled individually. Particles measured by EDS sharing the weight of a band are kept.

Parameters:

• particles (sequence of ParticleInfo) – Particles included in the statistics.

• bands (sequence of ParticleSizeBand, optional) – Size bands of the measurement.

• excluded_frame_ids (iterable of str, optional) – Frames left out of the statistics.

• confidence (float, optional) – Confidence level (default 0.95).

• n_resamples (int, optional) – Number of bootstrap resamples.

• rng (numpy.random.Generator, optional) – Random generator (default: a new one).

• eds_scans (sequence of ParticleSizeBand, optional) – Frames searched by EDS included in the statistics.

• excluded_particles (iterable of (frame_id, index), optional) – Particles of frames left out of the statistics.

Returns:

Name (D10, D50, D90, D43, Dv10, Dv50, Dv90) to (low, high) bounds in µm; empty if a scan has fewer than MIN_BOOTSTRAP_UNITS frames (or particles, without frame records).

Return type:

dict

autoemx.core.particle_statistics.ci_relative_half_widths(size_stats: Dict[str, int | float], confidence_intervals: Dict[str, Tuple[float, float]], quantities: Iterable[str]) → Dict[str, float]

Half-width of the confidence interval of each of quantities (e.g. ‘D50’), relative to its value.

autoemx.core.particle_statistics.select_size_stats_particles(particles: Sequence[ParticleInfo], include_eds_auto_particles: bool = True) → List[ParticleInfo]

Particles of the ledger counted in the size statistics besides those of the frames: the particles measured by EDS with automatic particle selection (selection ‘auto’, with an area), if include_eds_auto_particles, for samples without frames searched by EDS (measured before these were recorded), in ledger order.

autoemx.core.particle_statistics.build_size_result(powder_meas_cfg: PowderMeasurementConfig, bands: Sequence[ParticleSizeBand] = (), eds_scans: Sequence[ParticleSizeBand] = (), particles: Sequence[ParticleInfo] = (), n_par_target: int | None = None, measurement_id: int | None = None, segmentation_id: int | None = None, eds_segmentation_id: int | None = None, excluded_frame_ids: Sequence[str] = (), excluded_particles: Sequence[Tuple[str, int]] = (), missing_frame_ids: Sequence[str] = (), precision_reached: bool | None = None) → ParticleSizeStatsResult

Particle size statistics result of frames under a segmentation (see size_stats_inputs()), weighted as in weighted_size_observations(), to store in the sample ledger (SampleLedger.append_size_result, which gives it its id).

Parameters:

• powder_meas_cfg (PowderMeasurementConfig) – Settings of the measurement (of the EDS acquisition, without measurement).

• bands (sequence of ParticleSizeBand, optional) – Size bands of the measurement, and frames searched by EDS, with their frames under the segmentation.

• eds_scans (sequence of ParticleSizeBand, optional) – Size bands of the measurement, and frames searched by EDS, with their frames under the segmentation.

• particles (sequence of ParticleInfo, optional) – Particles measured by EDS counted (samples without frames searched by EDS, see select_size_stats_particles()).

• n_par_target (optional) – Recorded in the result.

• measurement_id (optional) – Recorded in the result.

• segmentation_id (optional) – Recorded in the result.

• eds_segmentation_id (optional) – Recorded in the result.

• precision_reached (optional) – Recorded in the result.

• excluded_frame_ids (sequence, optional) – Frames and particles (as (frame_id, index)) left out of the statistics, recorded in the result; the area of the frames left out is not counted.

• excluded_particles (sequence, optional) – Frames and particles (as (frame_id, index)) left out of the statistics, recorded in the result; the area of the frames left out is not counted.

• missing_frame_ids (sequence of str, optional) – Frames left out because they have no result of the segmentation, recorded in the result.

Raises:

ValueError – If no observation has a positive weight.

autoemx.core.particle_statistics.working_frame(frame: FrameRecord, segmentation_id: int) → ParticleStatsFrame | None

A frame of the ledger with its particles under a segmentation (None if not segmented with it).

class autoemx.core.particle_statistics.SizeStatsInputs(bands: List[ParticleSizeBand], eds_scans: List[ParticleSizeBand], particles: List[ParticleInfo], missing_frame_ids: List[str])

Bases: NamedTuple

What the size statistics of a sample are computed from (see size_stats_inputs()).

bands: List[ParticleSizeBand]

Alias for field number 0

eds_scans: List[ParticleSizeBand]

Alias for field number 1

particles: List[ParticleInfo]

Alias for field number 2

missing_frame_ids: List[str]

Alias for field number 3

autoemx.core.particle_statistics.size_stats_inputs(ledger: SampleLedger, measurement_id: int | None = None, segmentation_id: int | None = None, eds_segmentation_id: int | None = None, include_eds_auto_particles: bool = True) → SizeStatsInputs

Size bands of measurement measurement_id with their frames segmented with segmentation_id, and the frames searched by EDS segmented with eds_segmentation_id (one scan per size range and frame width), from the ledger frames; frames without result of the segmentation are left out (missing_frame_ids). Without frames searched by EDS (samples measured before these were recorded), the particles measured by EDS with automatic selection count, if include_eds_auto_particles.

autoemx.core.particle_statistics.eds_segmentation_for(ledger: SampleLedger, segmentation_id: int | None = None) → int | None

Segmentation of the frames searched by EDS counted with the frames of segmentation segmentation_id: the same if they have its results, else the last one they have (None without frames searched by EDS).

autoemx.core.particle_statistics.default_result_inputs(ledger: SampleLedger, include_eds_auto_particles: bool | None = None) → dict

What the size statistics of a sample are computed from by default (keyword arguments of recalculate_particle_stats()): those of the active result; else the last completed measurement under the segmentation it was acquired with, and the frames searched by EDS (under the same segmentation if they have its results, else under the last one they have) if include_eds_auto_particles (default: the setting of the measurement, or of the EDS acquisition).

autoemx.core.particle_statistics.recalculate_particle_stats(ledger: SampleLedger, particles_to_filter: Tuple[str, int]] | None=None, frame_ids_to_filter: Iterable[str] | None = None, include_eds_auto_particles: bool | None = None, measurement_id: Any = <object object>, segmentation_id: Any = <object object>, eds_segmentation_id: Any = <object object>) → ParticleSizeStatsResult | None

Particle size statistics of a sample recalculated from its ledger frames, leaving out some particles or frames (not stored: see SampleLedger.append_size_result).

By default (see default_result_inputs()), from what the active result was computed from (the last completed measurement and the frames searched by EDS, without result): the measurement, its segmentation and the segmentation of the frames searched by EDS can be given instead (eds_segmentation_id=None: frames searched by EDS not counted). A sample measured only by EDS gets statistics from its EDS frames (or, measured before these were recorded, from its particles measured by EDS with automatic selection).

Parameters:

• ledger (SampleLedger) – Ledger of the sample (not modified).

• particles_to_filter (iterable of (frame_id, index), optional) – Particles to leave out. Default (None): those left out of the active result, in the frames segmented by the same segmentation (particles are numbered per segmentation).

• frame_ids_to_filter (iterable of str, optional) – IDs of the frames whose particles are left out (exact matches). Default (None): those left out of the active result.

• include_eds_auto_particles (bool, optional) – Whether to count what EDS measured. Default: the setting of the active result (of the measurement, or of the EDS acquisition).

• measurement_id (int or None, optional) – What to compute the statistics from (default: see above).

• segmentation_id (int or None, optional) – What to compute the statistics from (default: see above).

• eds_segmentation_id (int or None, optional) – What to compute the statistics from (default: see above).

Returns:

The statistics, recording the particles and frames left out (those found), or None if nothing is left.

Return type:

ParticleSizeStatsResult or None

autoemx.core.particle_statistics.size_distribution(ledger: SampleLedger, result: ParticleSizeStatsResult | None = None) → Tuple[ndarray, ndarray]

Equivalent diameters (µm) and weights of the particle sizes behind a statistics result of a sample: what save_particle_size_histogram() plots.

Parameters:

• ledger (SampleLedger) – Ledger of the sample.

• result (ParticleSizeStatsResult, optional) – Statistics of the sample (default: the active one, else those computed by default).

Returns:

eq_diameters_um, weights – Diameters with a positive weight, and their weights.

Return type:

numpy.ndarray

autoemx.core.particle_statistics.particle_sizes_dataframe(particles: Sequence[ParticleInfo]) → DataFrame

Table of particle sizes: ID, frame, area (µm²) and equivalent diameter (µm) of each particle.

The equivalent diameter is computed from the area when a particle does not record it.

Raises:

ValueError – If a particle has no area.

autoemx.core.particle_statistics.save_particle_size_histogram(eq_diameters_um: Sequence[float], output_dir: str, sample_id: str, output_file_suffix: str = '', weights: Sequence[float] | None = None, size_stats: Dict[str, int | float] | None = None, image_format: str = 'png', bins: int = 20) → str

Save the particle size distribution plot as <sample_id>_Par_size_distribution_hist<suffix>.<image_format>.

Histogram of the number fraction of particles (weighted, if weights are given) by equivalent diameter, with logarithmic bins when the diameters span more than a factor 10, and cumulative number and volume distributions; the D10, D50 and D90 of size_stats (see compute_particle_size_stats()) are marked.

Returns:

Path of the saved image.

Return type:

str

autoemx.core.particle_statistics.save_particle_statistics(particles: Sequence[ParticleInfo], output_dir: str, sample_id: str, output_file_suffix: str = '', verbose: bool = False, bands: Sequence[ParticleSizeBand] | None = None, excluded_frame_ids: Iterable[str] = (), image_format: str = 'png', confidence_intervals: Dict[str, Tuple[float, float]] | None = None, eds_scans: Sequence[ParticleSizeBand] = (), excluded_particles: Iterable[Tuple[str, int]] = ()) → DataFrame

Compute the particle size statistics and save the particle sizes, the statistics and the histogram.

With size bands, the statistics are weighted by band (see band_particle_weights()): the particle sizes file then gives the weight of each particle (relative, 1 on average over the particles counted), and the statistics file the number of bands.

Parameters:

• particles (sequence of ParticleInfo) – Particles measured by EDS included in the statistics (each with area_um), for samples without frames searched by EDS.

• output_dir (str) – Folder where the files are saved (the sample folder).

• sample_id (str) – Sample ID, prefix of the file names.

• output_file_suffix (str, optional) – String added to the file names.

• verbose (bool, optional) – If True, logs the statistics.

• bands (sequence of ParticleSizeBand, optional) – Size bands of the measurement, to weight the statistics.

• excluded_frame_ids (iterable of str, optional) – Frames left out of the statistics, whose area is not counted.

• image_format (str, optional) – File format of the size distribution plot (default: ‘png’).

• confidence_intervals (dict, optional) – 95 % confidence intervals of the statistics (default: computed, see bootstrap_confidence_intervals()), saved as <name>_CI95_low and <name>_CI95_high.

• eds_scans (sequence of ParticleSizeBand, optional) – Frames searched by EDS included in the statistics.

• excluded_particles (iterable of (frame_id, index), optional) – Particles of frames left out of the statistics.

• EDS (Each particle of a frame is listed with its frame and its index in the frame; particles measured by)

• ID. ((particles) with their)

Returns:

One row: the measure described, followed by the statistics of compute_particle_size_stats() (and n_bands with size bands).

Return type:

pandas.DataFrame

Raises:

ValueError – If no particle is given, a particle has no area, or no particle has a positive weight.

autoemx.core.particle_statistics.size_result_label(ledger, result) → str

Readable name of a statistics result: id, measurement, segmentations, particles, frames and particles left out, e.g. ‘Result 2 · measurement 0 · SAM2 · 80 px · 412 particles · active’.