dials.algorithms.refinement¶
Table of Contents
Contains classes used to manage the reflections used during refinement, principally ReflectionManager.

class
dials.algorithms.refinement.reflection_manager.
BlockCalculator
(experiments, reflections)[source]¶ Bases:
object
Utility class to calculate and set columns in the provided reflection table, which will be used during scanvarying refinement. The columns are a ‘block’ number and an associated ‘block_centre’, giving the image number in the centre of the block

class
dials.algorithms.refinement.reflection_manager.
ReflectionManager
(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, trim_scan_edges=0.0, outlier_detector=None, weighting_strategy_override=None, verbosity=0)[source]¶ Bases:
object
A class to maintain information about observed and predicted reflections for refinement.
This new version keeps the reflections as a reflection table. Initialisation is not complete until the ReflectionManager is paired with a target function. Then, prediction can be done, followed by outlier rejection and any random sampling to form the working subset.

filter_obs
(sel)[source]¶ Perform a flex array selection on the managed observations, so that external classes can filter according to criteria not available here

finalise
(analysis=None)[source]¶ Complete initialisation by performing outlier rejection and any requested subsetting. If a list of results from a CentroidAnalysis object is provided, these may be used to determine outlier rejection block widths

get_accepted_refs_size
()[source]¶ Return the number of observations that pass inclusion criteria and can potentially be used for refinement

get_centroid_analyser
(debug=False)[source]¶ Create a CentroidAnalysis object for the current reflections

get_free_reflections
()[source]¶ Return all reflections that were accepted for refinement but not chosen in the working set


class
dials.algorithms.refinement.reflection_manager.
StillsReflectionManager
(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, trim_scan_edges=0.0, outlier_detector=None, weighting_strategy_override=None, verbosity=0)[source]¶ Bases:
dials.algorithms.refinement.reflection_manager.ReflectionManager
Overloads for a Reflection Manager that does not exclude reflections too close to the spindle, and reports only information about X, Y, DelPsi residuals

dials.algorithms.refinement.reflection_manager.
calculate_entering_flags
(reflections, experiments)[source]¶ calculate entering flags for all reflections, and set them as a column of the reflection table.
Contains classes used to provide weighting schemes as strategies for ReflectionManagers.

class
dials.algorithms.refinement.weighting_strategies.
ConstantWeightingStrategy
(wx, wy, wz, stills=False)[source]¶ Bases:
object

class
dials.algorithms.refinement.weighting_strategies.
ExternalDelPsiWeightingStrategy
[source]¶ Bases:
dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for stills refinement. This version uses statistical weights for X and Y and assume that the Delta Psi part is already provided in the reflection table

class
dials.algorithms.refinement.weighting_strategies.
StatisticalWeightingStrategy
[source]¶ Bases:
object
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement

class
dials.algorithms.refinement.weighting_strategies.
StillsWeightingStrategy
(delpsi_constant=1000000)[source]¶ Bases:
dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement. This version uses statistical weights for X and Y and a fixed constant for the delta Psi part, defaulting to 1000000
Refiner is the refinement module public interface. RefinerFactory is what should usually be used to construct a Refiner.

class
dials.algorithms.refinement.refiner.
Refiner
(reflections, experiments, pred_param, param_reporter, refman, target, refinery, verbosity=0, copy_experiments=True)[source]¶ Bases:
object
Public interface for performing DIALS refinement.
 Public methods:
 run rmsds get_experiments get_matches get_param_reporter parameter_correlation_plot selection_used_for_refinement predict_for_reflection_table predict_for_indexed
Notes
 The return value of run is a recorded history of the refinement
 The experiments accessor provides a copy of the experiments used by refinement
 The model accessors provide copies of those models that might be modified by refinement (beam, crystal and detector) TO BE DEPRECATED
 get_matches exposes the function of the same name from the privately stored reflection manager
 The return value of selection_used_for_refinement is a flex.bool

get_parameter_correlation_matrix
(step, col_select=None)[source]¶ Return the correlation matrix between columns of the Jacobian at the specified refinement step. The parameter col_select can be used to select subsets of the full number of columns. The column labels are also returned as a list of strings

predict_for_indexed
()[source]¶ perform prediction for all the indexed reflections passed into refinement and additionally set the used_in_refinement flag. Do not compose the derivatives of states of the model as this is expensive and they are not needed outside of a refinement run

predict_for_reflection_table
(reflections, skip_derivatives=False)[source]¶ perform prediction for all reflections in the supplied table

print_exp_rmsd_table
()[source]¶ print useful output about refinement steps in the form of a simple table

print_panel_rmsd_table
()[source]¶ print useful output about refinement steps in the form of a simple table

print_step_table
()[source]¶ print useful output about refinement steps in the form of a simple table

class
dials.algorithms.refinement.refiner.
RefinerFactory
[source]¶ Bases:
object
Factory class to create refiners

classmethod
config_parameterisation
(params, experiments, refman, do_stills)[source]¶ Given a set of parameters, create a parameterisation from a set of experimental models.
 Params:
 params The input parameters experiments An ExperimentList object
Returns: A tuple of the prediction equation parameterisation and the parameter reporter.

static
config_refinery
(target, pred_param, constraints_manager, verbosity)[source]¶ Given a set of parameters, a target class, a prediction parameterisation class and a constraints_manager (which could be None), build a refinery
 Params:
 params The input parameters
Returns: The refinery instance

static
config_refman
(reflections, experiments, do_stills, verbosity)[source]¶ Given a set of parameters and models, build a reflection manager
 Params:
 params The input parameters
Returns: The reflection manager instance

static
config_restraints
(det_params, beam_params, xl_ori_params, xl_uc_params, gon_params)[source]¶ Given a set of user parameters plus model parameterisations, create a restraints plus a parameterisation of these restraints
 Params:
 params The input parameters det_params A list of detector parameterisations beam_params A list of beam parameterisations xl_ori_params A list of crystal orientation parameterisations xl_uc_params A list of crystal unit cell parameterisations gon_params A list of goniometer parameterisations
Returns: A restraints parameterisation

classmethod
Contains classes for refinement engines. Refinery is the shared interface, LevenbergMarquardtIterations, GaussNewtonIterations, SimpleLBFGS and LBFGScurvs are the current concrete implementations

class
dials.algorithms.refinement.engine.
AdaptLbfgs
(*args, **kwargs)[source]¶ Bases:
dials.algorithms.refinement.engine.Refinery
Adapt Refinery for LBFGS minimiser

class
dials.algorithms.refinement.engine.
AdaptLstbx
(target, prediction_parameterisation, constraints_manager=None, log=None, verbosity=0, tracking=None, max_iterations=None)[source]¶ Bases:
dials.algorithms.refinement.engine.Refinery
,scitbx_lstbx_normal_equations_ext.non_linear_ls
,scitbx.lstbx.normal_eqns.non_linear_ls_mixin
Adapt Refinery for lstbx

class
dials.algorithms.refinement.engine.
DisableMPmixin
[source]¶ Bases:
object
A mixin class that disables setting of nproc for multiprocessing

class
dials.algorithms.refinement.engine.
GaussNewtonIterations
(target, prediction_parameterisation, constraints_manager=None, log=None, verbosity=0, tracking=None, max_iterations=20, **kwds)[source]¶ Bases:
dials.algorithms.refinement.engine.AdaptLstbx
,scitbx.lstbx.normal_eqns_solving.iterations
Refinery implementation, using lstbx Gauss Newton iterations

convergence_as_shift_over_esd
= 1e05¶

damping_value
= 0.0007¶

gradient_threshold
= 1e10¶

max_shift_over_esd
= 15¶

run
()[source]¶ To be implemented by derived class. It is expected that each step of refinement be preceeded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

step_threshold
= None¶


class
dials.algorithms.refinement.engine.
Journal
[source]¶ Bases:
dict
Container in which to store information about refinement history.
This is simply a dict but provides some extra methods for access that maintain values as columns in a table. Refinery classes will use these methods while entering data to ensure the table remains consistent. Methods inherited from dict are not hidden for ease of use of this object when returned to the user.

add_row
()[source]¶ Add an element to the end of each of the columns. Fail if any columns are the wrong length

del_last_row
()[source]¶ Delete the last element from the each of the columns. Fail if any columns are the wrong length

reason_for_termination
= None¶


class
dials.algorithms.refinement.engine.
LBFGScurvs
(*args, **kwargs)[source]¶ Bases:
dials.algorithms.refinement.engine.AdaptLbfgs
Refinery implementation using cctbx LBFGS with curvatures

class
dials.algorithms.refinement.engine.
LevenbergMarquardtIterations
(target, prediction_parameterisation, constraints_manager=None, log=None, verbosity=0, tracking=None, max_iterations=20, **kwds)[source]¶ Bases:
dials.algorithms.refinement.engine.GaussNewtonIterations
Refinery implementation, employing lstbx Levenberg Marquadt iterations

add_constant_to_diagonal
(mu)[source]¶ Add the constant value mu to the diagonal of the normal matrix

mu
¶

report_progress
(objective)[source]¶ Callback within the refinement main loop that can be overridden to report the value of the objective function (and possibly) other details for longrunning methods

run
()[source]¶ To be implemented by derived class. It is expected that each step of refinement be preceeded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

tau
= 0.001¶


class
dials.algorithms.refinement.engine.
Refinery
(target, prediction_parameterisation, constraints_manager=None, log=None, verbosity=0, tracking=None, max_iterations=None)[source]¶ Bases:
object
Interface for Refinery objects. This should be subclassed and the run method implemented.

get_correlation_matrix_for_step
(step)[source]¶ For each type of residual (e.g. X, Y, Phi), decompress and return the full 2D correlation matrix between columns of the Jacobian that was stored in the journal at the given step number. If not available, return None

jacobian_condition_number
()[source]¶ Calculate the condition number of the Jacobian, for tracking in the refinement journal, if requested. The condition number of a matrix A is defined as cond(A) = A inv(A). For a rectangular matrix the inverse operation refers to the MoorePenrose pseudoinverse. Various matrix norms can be used, resulting in numerically different condition numbers, however the 2norm is commonly used. In that case, the definition is equivalent to the ratio of the largest to smallest singular values of the matrix: cond(A) = sig_(A) / sig_min(A). That is the calculation that is performed here.
The condition number is a measure of how accurate the solution x to the equation Ax = b will be. Essentially it measures how errors are amplified through the linear equation. The condition number is large in the case that the columns of A are nearly linearlydependent (and infinite for a singular matrix). We use it here then to detect situations where the correlation between effects of different parameter shifts becomes large and therefore refinement is problematic.
Note, the Jacobian used here does not include any additional rows due to restraints terms that might be applied, or any parameter reduction due to constraints. Therefore this condition number relates to the pure linearised (GaussNewton) step, which might not actually be what the refinement engine uses. It can be indicative of issues in the fundamental set up of the least squares problem, even if these issues are avoided in practice (e.g. by use of an algorithm like LevenbergMarquardt, inclusion of restraints or parameter reduction).

run
()[source]¶ To be implemented by derived class. It is expected that each step of refinement be preceeded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

set_nproc
(nproc)[source]¶ Set number of processors for multiprocessing. Override in derived classes if a policy dictates that this must not be usercontrolled

split_jacobian_into_blocks
()[source]¶ Split the Jacobian into blocks each corresponding to a separate residual


class
dials.algorithms.refinement.engine.
SimpleLBFGS
(*args, **kwargs)[source]¶ Bases:
dials.algorithms.refinement.engine.AdaptLbfgs
Refinery implementation, using cctbx LBFGS with basic settings
Contains classes used to construct a target function for refinement, principally Target and ReflectionManager.

class
dials.algorithms.refinement.target.
LeastSquaresPositionalResidualWithRmsdCutoff
(experiments, reflection_predictor, ref_man, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶ Bases:
dials.algorithms.refinement.target.Target
An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and phi, terminating on achieved rmsd (or on intrisic convergence of the chosen minimiser)

class
dials.algorithms.refinement.target.
LeastSquaresPositionalResidualWithRmsdCutoffSparse
(experiments, reflection_predictor, ref_man, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶ Bases:
dials.algorithms.refinement.target.SparseGradientsMixin
,dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoff
A version of the LeastSquaresPositionalResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments

class
dials.algorithms.refinement.target.
SparseGradientsMixin
[source]¶ Mixin class to build a sparse Jacobian from gradients of the prediction formula stored as sparse vectors, and allow concatenation of gradient vectors that employed sparse storage.

class
dials.algorithms.refinement.target.
Target
(experiments, reflection_predictor, ref_manager, prediction_parameterisation, restraints_parameterisation=None, gradient_calculation_blocksize=None)[source]¶ Bases:
object
Abstract interface for a target function class
A Target object will be used by a Refinery. It will refer to a Reflection Manager to get a list of observations. It will perform reflection prediction on those observations and update the reflection manager with those predictions. It will then query the reflection manager to get a list of accepted reflections with observed and calculated positions. These are the reflections for use in refinement. It obtains the gradients of reflection positions from a relevant prediction parameterisation object. With all of this information in place, it calculates the value of the target function, the gradients of the target function and auxiliary information (e.g. RMSDs).
Concrete instances of this class implement the actual target function calculation. The base class should not determine what type of target function is used (e.g. least squares target), or limit whether the object is used for a detector space & phi residual, or a reciprocal space residual. This should all be set by a derived class.

calculate_gradients
(reflections=None, callback=None)[source]¶ delegate to the prediction_parameterisation object to calculate gradients for all the matched reflections, or just for those specified

compute_functional_gradients_and_curvatures
(block=None)[source]¶ calculate the value of the target function and its gradients. Set approximate curvatures as a sideeffect

compute_residuals_and_gradients
(block=None)[source]¶ return the vector of residuals plus their gradients and weights for nonlinear least squares methods

compute_restraints_functional_gradients_and_curvatures
()[source]¶ use the restraints_parameterisation object, if present, to calculate the least squares restraints objective plus gradients and approximate curvatures

compute_restraints_residuals_and_gradients
()[source]¶ delegate to the restraints_parameterisation object, if present, to calculate the vector of restraints residuals plus their gradients and weights for nonlinear least squares methods

get_num_matches_for_experiment
(iexp=0)[source]¶ return the number of reflections currently used in the calculation

get_num_matches_for_panel
(ipanel=0)[source]¶ return the number of reflections currently used in the calculation

predict
()[source]¶ perform reflection prediction for the working reflections and update the reflection manager

predict_for_free_reflections
()[source]¶ perform prediction for the reflections not used for refinement

predict_for_reflection_table
(reflections, skip_derivatives=False)[source]¶ perform prediction for all reflections in the supplied table

rmsd_names
= ['RMSD_X', 'RMSD_Y', 'RMSD_Phi']¶

rmsd_units
= ['mm', 'mm', 'rad']¶

rmsds_for_reflection_table
(reflections)[source]¶ calculate unweighted RMSDs for the specified reflections. Caution: this assumes that the table reflections has the keys expected by _rmsds_core

set_prediction_parameterisation
(prediction_parameterisation)[source]¶ For circumstances where the PredictionParameterisation object was not available at initialisation, set it with this method

set_restraints_parameterisation
(restraints_parameterisation)[source]¶ For circumstances where the RestraintsParameterisation object was not available at initialisation, set it with this method

split_matches_into_blocks
(nproc=1)[source]¶ Return a list of the matches, split into blocks according to the gradient_calculation_blocksize parameter and the number of processes (if relevant). The number of blocks will be set such that the total number of reflections being processed by concurrent processes does not exceed gradient_calculation_blocksize


class
dials.algorithms.refinement.target_stills.
LeastSquaresStillsResidualWithRmsdCutoff
(experiments, reflection_predictor, ref_man, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶ Bases:
dials.algorithms.refinement.target.Target
An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and minimum rotation to the Ewald sphere, DeltaPsi. Terminates refinement on achieved rmsd (or on intrisic convergence of the chosen minimiser)

predict_for_reflection_table
(reflections, skip_derivatives=False)[source]¶ perform prediction for all reflections in the supplied table

rmsd_names
= ['RMSD_X', 'RMSD_Y', 'RMSD_DeltaPsi']¶

rmsd_units
= ['mm', 'mm', 'rad']¶


class
dials.algorithms.refinement.target_stills.
LeastSquaresStillsResidualWithRmsdCutoffSparse
(experiments, reflection_predictor, ref_man, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶ Bases:
dials.algorithms.refinement.target.SparseGradientsMixin
,dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoff
A version of the LeastSquaresStillsResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments