Model State Serialization#
The IDAES framework has some utility functions for serializing the state of a
Pyomo model. These functions can save and load attributes of Pyomo components,
but cannot reconstruct the Pyomo objects (it is not a replacement for pickle).
It does have some advantages over pickle though. Not all Pyomo models are
picklable. Serialization and deserialization of the model state to/from json is
more secure in that it only deals with data and not executable code. It should
be safe to use the from_json()
function with data from untrusted sources,
while, unpickling an object from an untrusted source is not secure. Storing a
model state using these functions is also probably more robust against Python
and Python package version changes, and possibly more suitable for long-term storage
of results.
Below are a few example use cases for this module.
Some models are very complex and may take minutes to initialize. Once a model is initialized it’s state can be saved. For future runs, the initialized state can be reloaded instead of rerunning the initialization procedure.
Results can be stored for later evaluation without needing to rerun the model. These results can be archived in a data management system if needed later.
These functions may be useful in writing initialization procedures. For example, a model may be constructed and ready to run but first it may need to be initialized. Which components are active and which variables are fixed can be stored. The initialization can change which variables are fixed and which components are active. The original state can be read back after initialization, but where only values of variables that were originally fixed are read back in. This is an easy way to ensure that whatever the initialization procedure may do, the result is exactly the same problem (with only better initial values for unfixed variables).
These functions can be used to send and receive model data to/from JavaScript user interface components.
Examples#
This section provides a few very simple examples of how to use these functions.
Example Models#
This section provides some boilerplate and functions to create a couple simple test models. The second model is a little more complicated and includes suffixes.
from pyomo.environ import *
from idaes.core.util import to_json, from_json, StoreSpec
def setup_model01():
model = ConcreteModel()
model.b = Block([1,2,3])
a = model.b[1].a = Var(bounds=(-1e5, 1e5), initialize=2)
b = model.b[1].b = Var(bounds=(-100, 100), initialize=20)
model.b[1].c = Constraint(expr=b==10*a)
a.fix(2)
return model
def setup_model02():
model = ConcreteModel()
a = model.a = Param(default=1, mutable=True)
b = model.b = Param(default=2, mutable=True)
c = model.c = Param(initialize=4)
x = model.x = Var([1,2], initialize={1:1.5, 2:2.5}, bounds=(-10,10))
model.f = Objective(expr=(x[1] - a)**2 + (x[2] - b)**2)
model.g = Constraint(expr=x[1] + x[2] - c >= 0)
model.dual = Suffix(direction=Suffix.IMPORT)
model.ipopt_zL_out = Suffix(direction=Suffix.IMPORT)
model.ipopt_zU_out = Suffix(direction=Suffix.IMPORT)
return model
Serialization#
These examples can be appended to the boilerplate code above.
The first example creates a model, saves the state, changes a value, then reads back the initial state.
model = setup_model01()
to_json(model, fname="ex.json.gz", gz=True, human_read=True)
model.b[1].a = 3000.4
from_json(model, fname="ex.json.gz", gz=True)
print(value(model.b[1].a))
2
This next example show how to save only suffixes.
model = setup_model02()
# Suffixes here are read back from solver, so to have suffix data,
# need to solve first
solver = SolverFactory("ipopt")
solver.solve(model)
store_spec = StoreSpec.suffix()
to_json(model, fname="ex.json", wts=store_spec)
# Do something and now I want my suffixes back
from_json(model, fname="ex.json", wts=store_spec)
to_json#
Despite the name of the to_json
function it is capable of creating Python
dictionaries, json files, gzipped json files, and json strings. The function
documentation is below. A StoreSpec
object provides the function with details on what to store and how to handle
special cases of Pyomo component attributes.
- idaes.core.util.model_serializer.to_json(o, fname=None, human_read=False, wts=None, metadata=None, gz=None, return_dict=False, return_json_string=False)[source]#
Save the state of a model to a Python dictionary, and optionally dump it to a json file. To load a model state, a model with the same structure must exist. The model itself cannot be recreated from this.
- Parameters:
o – The Pyomo component object to save. Usually a Pyomo model, but could also be a sub-component of a model (usually a sub-block).
fname – json file name to save model state, if None only create python dict
gz – If fname is given and gv is True gzip the json file. The default is True if the file name ends with ‘.gz’ otherwise False.
human_read – if True, add indents and spacing to make the json file more readable, if false cut out whitespace and make as compact as possible
metadata – A dictionary of additional metadata to add.
wts – is What To Save, this is a StoreSpec object that specifies what object types and attributes to save. If None, the default is used which saves the state of the complete model state.
metadata – additional metadata to save beyond the standard format_version, date, and time.
return_dict – default is False if true returns a dictionary representation
return_json_string – default is False returns a json string
- Returns:
If return_dict is True returns a dictionary serialization of the Pyomo component. If return_dict is False and return_json_string is True returns a json string dump of the dict. If fname is given the dictionary is also written to a json file. If gz is True and fname is given, writes a gzipped json file.
from_json#
The from_json
function puts data from Python dictionaries, json files,
gzipped json files, and json strings back into a Pyomo model. The function
documentation is below. A StoreSpec
object provides the function with details on what to read and how to handle
special cases of Pyomo component attributes.
- idaes.core.util.model_serializer.from_json(o, sd=None, fname=None, s=None, wts=None, gz=None, root_name=None)[source]#
Load the state of a Pyomo component state from a dictionary, json file, or json string. Must only specify one of sd, fname, or s as a non-None value. This works by going through the model and loading the state of each sub-component of o. If the saved state contains extra information, it is ignored. If the save state doesn’t contain an entry for a model component that is to be loaded an error will be raised, unless ignore_missing = True.
- Parameters:
o – Pyomo component to for which to load state
sd – State dictionary to load, if None, check fname and s
fname – JSON file to load, only used if sd is None
s – JSON string to load only used if both sd and fname are None
wts – StoreSpec object specifying what to load
gz – If True assume the file specified by fname is gzipped. The default is True if fname ends with ‘.gz’ otherwise False.
- Returns:
Dictionary with some performance information. The keys are “etime_load_file”, how long in seconds it took to load the json file “etime_read_dict”, how long in seconds it took to read models state “etime_read_suffixes”, how long in seconds it took to read suffixes
StoreSpec#
StoreSpec
is a class for objects that tell the to_json()
and from_json()
functions how to read and write Pyomo component attributes. The default
initialization provides an object that would load and save attributes usually
needed to save a model state. There are several other class methods that
provide canned objects for specific uses. Through initialization arguments, the
behavior is highly customizable. Attributes can be read or written using callback
functions to handle attributes that can not be directly read or written (e.g.
a variable lower bound is set by calling setlb()). See the class documentation below.
- class idaes.core.util.model_serializer.StoreSpec(classes=None, data_classes=None, ignore_missing=True, suffix=None, suffix_filter=None)[source]#
A StoreSpec object tells the serializer functions what to read or write. The default settings will produce a StoreSpec configured to load/save the typical attributes required to load/save a model state.
- Parameters:
classes – Dictionary of classes to read/write. The class is the dictionary key and the value is a list or tuple with two elements 1) a list or tuple of attributes to read/write and 2) a filter function. The attributes can be an empty list and the filter function can be None. Filter functions if present only apply to reading components and do not affect writing. Filter functions takes two arguments. The first is the object being read to and the second is the state dictionary being read from. The state dictionary contains keys for the attributes that were written. Based on the state of the component and the stored state of the component, the filter function returns a list of attributes to read. For example, a filter function can be used to only read the values of variables that were fixed when they were written. The order of the class keys is important and should go from specific classes to general ones. For example, if a UnitModel is a Block, but you want to store extra attributes for a UnitModel, UnitModel should come before Block in the key set.
data_classes – This takes the same form as the classes argument, the classes should be component data types.
ignore_missing – If True will ignore a component or attribute that exists in the model, but not in the stored state. If false an exception will be raised for things in the model that should be loaded but aren’t in the stored state. Extra items in the stored state will not raise an exception regardless of this argument.
suffix – If True store suffixes and component ids. If false, don’t store suffixes. This is a legacy option. The preferred way to store a Suffix of not is just to include Suffix in classes or not.
suffix_filter – None to store all suffixes or a list of suffixes to store.
- get_class_attr_list(o)[source]#
Look up what attributes to save/load for an Component object. :param o: Object to look up attribute list for.
- Returns:
A list of attributes and a filter function for object type
- get_data_class_attr_list(o)[source]#
Look up what attributes to save/load for an ComponentData object. :param o: Object to look up attribute list for.
- Returns:
A list of attributes and a filter function for object type
- set_read_callback(attr, cb=None)[source]#
Set a callback to set an attribute, when reading from json or dict.
- set_write_callback(attr, cb=None)[source]#
Set a callback to get an attribute, when writing to json or dict.
- classmethod value(only_not_fixed=False)[source]#
Returns a StoreSpec object to store variable values only.
Structure#
Python dictionaries, json strings, or json files are generated, in any case the structure of the data is the same. The current data structure version is 3.
The example json below shows the top-level structure. The
"top_level_component"
would be the name of the Pyomo component that is being
serialized. The top level component is the only place were the component name does
not matter when reading the serialized data.
{
"__metadata__": {
"format_version": 3,
"date": "2018-12-21",
"time": "11:34:39.714323",
"other": {
},
"__performance__": {
"n_components": 219,
"etime_make_dict": 0.003}
},
"top_level_component":{
"...": "..."
},
}
The data structure of a Pyomo component is shown below. Here "attribute_1"
and "attribute_2"
are just examples the actual attributes saved depend on
the “wts” argument to to_json()
. Scalar and indexed components have the
same structure. Scalar components have one entry in "data"
with an index of
"None"
. Only components derived from Pyomo’s _BlockData
have a "__pyomo_components__"
field, and components appearing there are keyed
by their name. The data structure duplicates the hierarchical structure of the
Pyomo model.
Suffixes store extra attributes for Pyomo components that are not stored on the
components themselves. Suffixes are a Pyomo structure that comes from the AMPL
solver interface. If a component is a suffix, keys in the data section are the
serial integer component IDs generated by to_json()
, and the value is the
value of the suffix for the corresponding component.
{
"__type__": "<class 'some.class'>",
"__id__": 0,
"data":{
"index_1":{
"__type__":"<usually a component class but for params could be float, int, ...>",
"__id__": 1,
"__pyomo_components__":{
"child_component_1": {
"...": "..."
}
},
"attribute_1": "... could be any number of attributes like 'value': 1.0,",
"attribute_2": "..."
}
},
"attribute_1": "... could be any number of attributes like 'active': true,",
"attribute_2": "..."
}
As a more concrete example, here is the json generated for example model 2 in Examples. This code can be appended to the example boilerplate above. To generate the example json shown.
model = setup_model02()
solver = SolverFactory("ipopt")
solver.solve(model)
to_json(model, fname="ex.json")
The resulting json is shown below. The top-level component
in this case is given as “unknown,” because the model was not given a name. The
top level object name is not needed when reading back data, since the top level
object is specified in the call to from_json()
. Types are not used when
reading back data, they may have some future application, but at this point they
just provide a little extra information.
{
"__metadata__":{
"format_version":3,
"date":"2019-01-02",
"time":"10:22:25.833501",
"other":{
},
"__performance__":{
"n_components":18,
"etime_make_dict":0.000955581665039062
}
},
"unknown":{
"__type__":"<class 'pyomo.core.base.PyomoModel.ConcreteModel'>",
"__id__":0,
"active":true,
"data":{
"None":{
"__type__":"<class 'pyomo.core.base.PyomoModel.ConcreteModel'>",
"__id__":1,
"active":true,
"__pyomo_components__":{
"a":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":2,
"_mutable":true,
"data":{
"None":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":3,
"value":1
}
}
},
"b":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":4,
"_mutable":true,
"data":{
"None":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":5,
"value":2
}
}
},
"c":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":6,
"_mutable":false,
"data":{
"None":{
"__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
"__id__":7,
"value":4
}
}
},
"x":{
"__type__":"<class 'pyomo.core.base.var.IndexedVar'>",
"__id__":8,
"data":{
"1":{
"__type__":"<class 'pyomo.core.base.var._GeneralVarData'>",
"__id__":9,
"fixed":false,
"stale":false,
"value":1.5,
"lb":-10,
"ub":10
},
"2":{
"__type__":"<class 'pyomo.core.base.var._GeneralVarData'>",
"__id__":10,
"fixed":false,
"stale":false,
"value":2.5,
"lb":-10,
"ub":10
}
}
},
"f":{
"__type__":"<class 'pyomo.core.base.objective.SimpleObjective'>",
"__id__":11,
"active":true,
"data":{
"None":{"__type__":"<class 'pyomo.core.base.objective.SimpleObjective'>",
"__id__":12,
"active":true
}
}
},
"g":{
"__type__":"<class 'pyomo.core.base.constraint.SimpleConstraint'>",
"__id__":13,
"active":true,
"data":{
"None":{
"__type__":"<class 'pyomo.core.base.constraint.SimpleConstraint'>",
"__id__":14,
"active":true
}
}
},
"dual":{
"__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
"__id__":15,
"active":true,
"data":{
"14":0.9999999626149493
}
},
"ipopt_zL_out":{
"__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
"__id__":16,
"active":true,
"data":{
"9":2.1791814146763388e-10,
"10":2.004834508495852e-10
}
},
"ipopt_zU_out":{
"__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
"__id__":17,
"active":true,
"data":{
"9":-2.947875485096964e-10,
"10":-3.3408951850535573e-10
}
}
}
}
}
}
}