experiment

experiment

Define classes and functions for the Experiment class (running sims and comparing them to data)

Classes

Name	Description
Experiment	Class for running a single sim and comparing it to data.
Fit	A class for calculating the fit between the model and the data. Note the

Experiment

experiment.Experiment(
    sim=None,
    pars=None,
    dataloader=None,
    data_kwargs=None,
    flags=None,
    label=None,
    **kwargs,
)

Class for running a single sim and comparing it to data.

Parameters

Name	Type	Description	Default
sim	Sim	the sim to run; if None, will be created when run_model is called	None
pars	dict	dictionary of parameters passed to Sim	None
flags	dict	which analyses to run; see fp.experiment.default_flags for options	None
label	str	label of experiment	None
kwargs	dict	passed into pars	{}

Methods

Name	Description
compare	Create and print a comparison between model and data
compute_fit	Compute how good the fit is
get_crude_death_rate	Calculate the crude death rate from the model results.
plot	Plot the model against the data
post_process_results	Compare the model and the data
run	Run the model and post-process the results
run_model	Create the sim and run the model
summarize	Convert results to a one-number-per-key summary format. Returns summary,
to_json	Export results as JSON.

compare

experiment.Experiment.compare()

Create and print a comparison between model and data

compute_fit

experiment.Experiment.compute_fit(*args, **kwargs)

Compute how good the fit is

get_crude_death_rate

experiment.Experiment.get_crude_death_rate(fp_df=None, sim_df=None)

Calculate the crude death rate from the model results.

plot

experiment.Experiment.plot(
    do_show=None,
    do_save=None,
    filename='fp_experiment.png',
    axis_args=None,
    do_maximize=True,
)

Plot the model against the data

post_process_results

experiment.Experiment.post_process_results(compute_fit=True, **kwargs)

Compare the model and the data

run

experiment.Experiment.run(
    pars=None,
    keep_people=False,
    compute_fit=True,
    **kwargs,
)

Run the model and post-process the results

run_model

experiment.Experiment.run_model(pars=None, **kwargs)

Create the sim and run the model

summarize

experiment.Experiment.summarize(as_df=False)

Convert results to a one-number-per-key summary format. Returns summary, also saves to self.summary.

Parameters

Name	Type	Description	Default
as_df	bool	if True, return a dataframe instead of a dict.	False

to_json

experiment.Experiment.to_json(
    filename=None,
    tostring=False,
    indent=2,
    verbose=False,
    **kwargs,
)

Export results as JSON.

Parameters

Name	Type	Description	Default
filename	str	if None, return string; else, write to file	None
tostring	bool	if not writing to file, whether to write to string (alternative is sanitized dictionary)	False
indent	int	if writing to file, how many indents to use per nested level	2
verbose	bool	detail to print	False
kwargs	dict	passed to savejson()	{}

Returns

Name	Type	Description
		A unicode string containing a JSON representation of the results,
		or writes the JSON file to disk

Examples::

json = exp.to_json()
exp.to_json('results.json')

Fit

experiment.Fit(
    data,
    sim,
    weights=None,
    keys=None,
    custom=None,
    compute=True,
    verbose=False,
    **kwargs,
)

A class for calculating the fit between the model and the data. Note the following terminology is used here:

- fit: nonspecific term for how well the model matches the data
- difference: the absolute numerical differences between the model and the data (one time series per result)
- goodness-of-fit: the result of passing the difference through a statistical function, such as mean squared error
- loss: the goodness-of-fit for each result multiplied by user-specified weights (one time series per result)
- mismatches: the sum of all the losses (a single scalar value per time series)
- mismatch: the sum of the mismatches -- this is the value to be minimized during calibration

Parameters

Name	Type	Description	Default
sim	Sim	the sim object	required
weights	dict	the relative weight to place on each result (by default: 10 for deaths, 5 for diagnoses, 1 for everything else)	None
keys	list	the keys to use in the calculation	None
custom	dict	a custom dictionary of additional data to fit; format is e.g. {‘my_output’:{‘data’:[1,2,3], ‘sim’:[1,2,4], ‘weights’:2.0}}	None
compute	bool	whether to compute the mismatch immediately	True
verbose	bool	detail to print	False
kwargs	dict	passed to cv.compute_gof() – see this function for more detail on goodness-of-fit calculation options	{}

Example::

sim = cv.Sim()
sim.run()
fit = sim.compute_fit()
fit.plot()

Methods

Name	Description
compute	Perform all required computations
compute_diffs	Find the differences between the sim and the data
compute_gofs	Compute the goodness-of-fit
compute_losses	Compute the weighted goodness-of-fit
compute_mismatch	Compute the final mismatch
plot	Plot the fit of the model to the data. For each result, plot the data
reconcile_inputs	Find matching keys and indices between the model and the data

compute

experiment.Fit.compute()

Perform all required computations

compute_diffs

experiment.Fit.compute_diffs(absolute=False)

Find the differences between the sim and the data

compute_gofs

experiment.Fit.compute_gofs(**kwargs)

Compute the goodness-of-fit

compute_losses

experiment.Fit.compute_losses()

Compute the weighted goodness-of-fit

compute_mismatch

experiment.Fit.compute_mismatch(use_median=False)

Compute the final mismatch

plot

experiment.Fit.plot(
    keys=None,
    width=0.8,
    font_size=18,
    fig_args=None,
    axis_args=None,
    plot_args=None,
    do_show=True,
)

Plot the fit of the model to the data. For each result, plot the data and the model; the difference; and the loss (weighted difference). Also plots the loss as a function of time.

Parameters

Name	Type	Description	Default
keys	list	which keys to plot (default, all)	None
width	float	bar width	0.8
font_size	float	size of font	18
fig_args	dict	passed to pl.figure()	None
axis_args	dict	passed to pl.subplots_adjust()	None
plot_args	dict	passed to pl.plot()	None
do_show	bool	whether to show the plot	True

reconcile_inputs

experiment.Fit.reconcile_inputs(verbose=False)

Find matching keys and indices between the model and the data

Functions

Name	Description
compute_gof	Calculate the goodness of fit. By default use normalized absolute error, but
diff_summaries	Compute the difference of the summaries of two FPsim calibration objects, and print any

compute_gof

experiment.compute_gof(
    actual,
    predicted,
    normalize=True,
    use_frac=False,
    use_squared=False,
    as_scalar='none',
    eps=1e-09,
    skestimator=None,
    **kwargs,
)

Calculate the goodness of fit. By default use normalized absolute error, but highly customizable. For example, mean squared error is equivalent to setting normalize=False, use_squared=True, as_scalar=‘mean’.

Parameters

Name	Type	Description	Default
actual	arr	array of actual (data) points	required
predicted	arr	corresponding array of predicted (model) points	required
normalize	bool	whether to divide the values by the largest value in either series	True
use_frac	bool	convert to fractional mismatches rather than absolute	False
use_squared	bool	square the mismatches	False
as_scalar	str	return as a scalar instead of a time series: choices are sum, mean, median	'none'
eps	float	to avoid divide-by-zero	1e-09
skestimator	str	if provided, use this scikit-learn estimator instead	None
kwargs	dict	passed to the scikit-learn estimator	{}

Returns

Name	Type	Description
gofs	arr	array of goodness-of-fit values, or a single value if as_scalar is True

Examples::

x1 = np.cumsum(np.random.random(100))
x2 = np.cumsum(np.random.random(100))

e1 = compute_gof(x1, x2) # Default, normalized absolute error
e2 = compute_gof(x1, x2, normalize=False, use_frac=False) # Fractional error
e3 = compute_gof(x1, x2, normalize=False, use_squared=True, as_scalar='mean') # Mean squared error
e4 = compute_gof(x1, x2, skestimator='mean_squared_error') # Scikit-learn's MSE method
e5 = compute_gof(x1, x2, as_scalar='median') # Normalized median absolute error -- highly robust

diff_summaries

experiment.diff_summaries(
    sim1,
    sim2,
    skip_key_diffs=False,
    output=False,
    die=False,
)

Compute the difference of the summaries of two FPsim calibration objects, and print any values which differ.

Parameters

Name	Type	Description	Default
sim1	sim / dict	the calib.summary dictionary, representing a single sim	required
sim2	sim / dict	ditto	required
skip_key_diffs	bool	whether to skip keys that don’t match between sims	False
output	bool	whether to return the output as a string (otherwise print)	False
die	bool	whether to raise an exception if the sims don’t match	False
require_run	bool	require that the simulations have been run	required

Example::

c1 = fp.Calibration()
c2 = fp.Calibration()
c1.run()
c2.run()
fp.diff_summaries(c1.summarize(), c2.summarize())