Model State Serialization

The IDAES framework has some utility functions for serializing the state of a Pyomo model. These functions can save and load attributes of Pyomo components, but cannot reconstruct the Pyomo objects (it is not a replacement for pickle). It does have some advantages over pickle though. Not all Pyomo models are picklable. Serialization and deserialization of the model state to/from json is more secure in that it only deals with data and not executable code. It should be safe to use the from_json() function with data from untrusted sources, while, unpickling an object from an untrusted source is not secure. Storing a model state using these functions is also probably more robust against Python and Python package version changes, and possibly more suitable for long-term storage of results.

Below are a few example use cases for this module.

  • Some models are very complex and may take minutes to initialize. Once a model is initialized it’s state can be saved. For future runs, the initialized state can be reloaded instead of rerunning the initialization procedure.

  • Results can be stored for later evaluation without needing to rerun the model. These results can be archived in a data management system if needed later.

  • These functions may be useful in writing initialization procedures. For example, a model may be constructed and ready to run but first it may need to be initialized. Which components are active and which variables are fixed can be stored. The initialization can change which variables are fixed and which components are active. The original state can be read back after initialization, but where only values of variables that were originally fixed are read back in. This is an easy way to ensure that whatever the initialization procedure may do, the result is exactly the same problem (with only better initial values for unfixed variables).

  • These functions can be used to send and receive model data to/from JavaScript user interface components.

Examples

This section provides a few very simple examples of how to use these functions.

Example Models

This section provides some boilerplate and functions to create a couple simple test models. The second model is a little more complicated and includes suffixes.

from pyomo.environ import *
from idaes.core.util import to_json, from_json, StoreSpec

def setup_model01():
    model = ConcreteModel()
    model.b = Block([1,2,3])
    a = model.b[1].a = Var(bounds=(-1e5, 1e5), initialize=2)
    b = model.b[1].b = Var(bounds=(-100, 100), initialize=20)
    model.b[1].c = Constraint(expr=b==10*a)
    a.fix(2)
    return model

def setup_model02():
    model = ConcreteModel()
    a = model.a = Param(default=1, mutable=True)
    b = model.b = Param(default=2, mutable=True)
    c = model.c = Param(initialize=4)
    x = model.x = Var([1,2], initialize={1:1.5, 2:2.5}, bounds=(-10,10))
    model.f = Objective(expr=(x[1] - a)**2 + (x[2] - b)**2)
    model.g = Constraint(expr=x[1] + x[2] - c >= 0)
    model.dual = Suffix(direction=Suffix.IMPORT)
    model.ipopt_zL_out = Suffix(direction=Suffix.IMPORT)
    model.ipopt_zU_out = Suffix(direction=Suffix.IMPORT)
    return model

Serialization

These examples can be appended to the boilerplate code above.

The first example creates a model, saves the state, changes a value, then reads back the initial state.

model = setup_model01()
to_json(model, fname="ex.json.gz", gz=True, human_read=True)
model.b[1].a = 3000.4
from_json(model, fname="ex.json.gz", gz=True)
print(value(model.b[1].a))
2

This next example show how to save only suffixes.

model = setup_model02()
# Suffixes here are read back from solver, so to have suffix data,
# need to solve first
solver = SolverFactory("ipopt")
solver.solve(model)
store_spec = StoreSpec.suffix()
to_json(model, fname="ex.json", wts=store_spec)
# Do something and now I want my suffixes back
from_json(model, fname="ex.json", wts=store_spec)

to_json

Despite the name of the to_json function it is capable of creating Python dictionaries, json files, gzipped json files, and json strings. The function documentation is below. A StoreSpec object provides the function with details on what to store and how to handle special cases of Pyomo component attributes.

idaes.core.util.model_serializer.to_json(o, fname=None, human_read=False, wts=None, metadata=None, gz=None, return_dict=False, return_json_string=False)[source]

Save the state of a model to a Python dictionary, and optionally dump it to a json file. To load a model state, a model with the same structure must exist. The model itself cannot be recreated from this.

Parameters
  • o – The Pyomo component object to save. Usually a Pyomo model, but could also be a subcomponent of a model (usually a sub-block).

  • fname – json file name to save model state, if None only create python dict

  • gz – If fname is given and gv is True gzip the json file. The default is True if the file name ends with ‘.gz’ otherwise False.

  • human_read – if True, add indents and spacing to make the json file more readable, if false cut out whitespace and make as compact as possilbe

  • metadata – A dictionary of additional metadata to add.

  • wts – is What To Save, this is a StoreSpec object that specifies what object types and attributes to save. If None, the default is used which saves the state of the compelte model state.

  • metadata – additional metadata to save beyond the standard format_version, date, and time.

  • return_dict – default is False if true returns a dictionary representation

  • return_json_string – default is False returns a json string

Returns

If return_dict is True returns a dictionary serialization of the Pyomo component. If return_dict is False and return_json_string is True returns a json string dump of the dict. If fname is given the dictionary is also written to a json file. If gz is True and fname is given, writes a gzipped json file.

from_json

The from_json function puts data from Python dictionaries, json files, gzipped json files, and json strings back into a Pyomo model. The function documentation is below. A StoreSpec object provides the function with details on what to read and how to handle special cases of Pyomo component attributes.

idaes.core.util.model_serializer.from_json(o, sd=None, fname=None, s=None, wts=None, gz=None, root_name=None)[source]

Load the state of a Pyomo component state from a dictionary, json file, or json string. Must only specify one of sd, fname, or s as a non-None value. This works by going through the model and loading the state of each sub-component of o. If the saved state contains extra information, it is ignored. If the save state doesn’t contain an entry for a model component that is to be loaded an error will be raised, unless ignore_missing = True.

Parameters
  • o – Pyomo component to for which to load state

  • sd – State dictionary to load, if None, check fname and s

  • fname – JSON file to load, only used if sd is None

  • s – JSON string to load only used if both sd and fname are None

  • wts – StoreSpec object specifying what to load

  • gz – If True assume the file specified by fname is gzipped. The default is True if fname ends with ‘.gz’ otherwise False.

Returns

Dictionary with some performance information. The keys are “etime_load_file”, how long in seconds it took to load the json file “etime_read_dict”, how long in seconds it took to read models state “etime_read_suffixes”, how long in seconds it took to read suffixes

StoreSpec

StoreSpec is a class for objects that tell the to_json() and from_json() functions how to read and write Pyomo component attributes. The default initialization provides an object that would load and save attributes usually needed to save a model state. There are several other class methods that provide canned objects for specific uses. Through initialization arguments, the behavior is highly customizable. Attributes can be read or written using callback functions to handle attributes that can not be directly read or written (e.g. a variable lower bound is set by calling setlb()). See the class documentation below.

class idaes.core.util.model_serializer.StoreSpec(classes={<class 'pyomo.core.base.param.Param'>: (('_mutable', ), None), <class 'pyomo.core.base.var.Var'>: ((), None), <class 'pyomo.core.base.boolean_var.BooleanVar'>: ((), None), <class 'pyomo.core.base.expression.Expression'>: ((), None), <class 'pyomo.core.base.block.Block'>: (('active', ), None), <class 'pyomo.core.base.constraint.Constraint'>: (('active', ), None), <class 'pyomo.core.base.suffix.Suffix'>: ((), None)}, data_classes={<class 'pyomo.core.base.var._GeneralVarData'>: (('fixed', 'stale', 'value', 'lb', 'ub'), None), <class 'pyomo.core.base.boolean_var._GeneralBooleanVarData'>: (('fixed', 'stale', 'value'), None), <class 'pyomo.core.base.param._ParamData'>: (('value', ), None), <class 'int'>: (('value', ), None), <class 'float'>: (('value', ), None), <class 'str'>: (('value', ), None), <class 'pyomo.core.base.expression._GeneralExpressionData'>: ((), None), <class 'pyomo.core.base.block._BlockData'>: (('active', ), None), <class 'pyomo.core.base.constraint._GeneralConstraintData'>: (('active', ), None)}, ignore_missing=True, suffix=None, suffix_filter=None)[source]

A StoreSpec object tells the serializer functions what to read or write. The default settings will produce a StoreSpec configured to load/save the typical attributes required to load/save a model state.

Parameters
  • classes – Dictionary of classes to read/write. The class is the dictionary key and the value is a list or tuple with two elements 1) a list or tuple of attributes to read/write and 2) a filter function. The attributes can be an empty list and the filter function can be None. Filter functions if present only apply to reading components and do not affect writing. Filter functions takes two arguments. The first is the object being read to and the second is the state dictionary being read from. The state dictionary contains keys for the attributes that were written. Based on the state of the component and the stored state of the component, the filter function returns a list of attributes to read. For example, a filter function can be used to only read the values of variables that were fixed when they were written. The order of the class keys is important and should go from specific classes to general ones. For example, if a UnitModel is a Block, but you want to store extra attributes for a UnitModel, UnitModel should come before Block in the key set.

  • data_classes – This takes the same form as the classes argument, the classes should be component data types.

  • ignore_missing – If True will ignore a component or attribute that exists in the model, but not in the stored state. If false an exception will be raised for things in the model that should be loaded but aren’t in the stored state. Extra items in the stored state will not raise an exception regardless of this argument.

  • suffix – If True store suffixes and component ids. If false, don’t store suffixes. This is a legacy option. The preferred way to store a Suffix of not is just to include Suffix in classes or not.

  • suffix_filter – None to store all suffixes or a list of suffixes to store.

classmethod bound()[source]

Returns a StoreSpec object to store variable bounds only.

get_class_attr_list(o)[source]

Look up what attributes to save/load for an Component object. :param o: Object to look up attribute list for.

Returns

A list of attributes and a filter function for object type

get_data_class_attr_list(o)[source]

Look up what attributes to save/load for an ComponentData object. :param o: Object to look up attribute list for.

Returns

A list of attributes and a filter function for object type

classmethod isfixed()[source]

Returns a StoreSpec object to store if variables are fixed.

set_read_callback(attr, cb=None)[source]

Set a callback to set an attribute, when reading from json or dict.

set_write_callback(attr, cb=None)[source]

Set a callback to get an attribute, when writing to json or dict.

classmethod value(only_not_fixed=False)[source]

Returns a StoreSpec object to store variable values only.

classmethod value_isfixed(only_fixed)[source]

Return a StoreSpec object to store variable values and if fixed.

Parameters

only_fixed – Only load fixed variable values

classmethod value_isfixed_isactive(only_fixed)[source]

Return a StoreSpec object to store variable values, if variables are fixed and if components are active.

Parameters

only_fixed – Only load fixed variable values

Structure

Python dictionaries, json strings, or json files are generated, in any case the structure of the data is the same. The current data structure version is 3.

The example json below shows the top-level structure. The "top_level_component" would be the name of the Pyomo component that is being serialized. The top level component is the only place were the component name does not matter when reading the serialized data.

{
    "__metadata__": {
        "format_version": 3,
        "date": "2018-12-21",
        "time": "11:34:39.714323",
        "other": {
        },
        "__performance__": {
            "n_components": 219,
            "etime_make_dict": 0.003}
    },
    "top_level_component":{
      "...": "..."
    },
}

The data structure of a Pyomo component is shown below. Here "attribute_1" and "attribute_2" are just examples the actual attributes saved depend on the “wts” argument to to_json(). Scalar and indexed components have the same structure. Scalar components have one entry in "data" with an index of "None". Only components derived from Pyomo’s _BlockData have a "__pyomo_components__" field, and components appearing there are keyed by thier name. The data structure duplicates the hierarchical structure of the Pyomo model.

Suffixes store extra attributes for Pyomo components that are not stored on the components themselves. Suffixes are a Pyomo structure that comes from the AMPL solver interface. If a component is a suffix, keys in the data section are the serial integer component IDs generated by to_json(), and the value is the value of the suffix for the corresponding component.

{
    "__type__": "<class 'some.class'>",
    "__id__": 0,
    "data":{
      "index_1":{
          "__type__":"<usually a component class but for params could be float, int, ...>",
          "__id__": 1,
          "__pyomo_components__":{
            "child_component_1": {
              "...": "..."
            }
          },
          "attribute_1": "... could be any number of attributes like 'value': 1.0,",
          "attribute_2": "..."
      }
    },
    "attribute_1": "... could be any number of attributes like 'active': true,",
    "attribute_2": "..."
}

As a more concrete example, here is the json generated for example model 2 in Examples. This code can be appended to the example boilerplate above. To generate the example json shown.

model = setup_model02()
solver = SolverFactory("ipopt")
solver.solve(model)
to_json(model, fname="ex.json")

The resulting json is shown below. The top-level component in this case is given as “unknown,” because the model was not given a name. The top level object name is not needed when reading back data, since the top level object is specified in the call to from_json(). Types are not used when reading back data, they may have some future application, but at this point they just provide a little extra information.

{
  "__metadata__":{
    "format_version":3,
    "date":"2019-01-02",
    "time":"10:22:25.833501",
    "other":{
    },
    "__performance__":{
      "n_components":18,
      "etime_make_dict":0.000955581665039062
    }
  },
  "unknown":{
    "__type__":"<class 'pyomo.core.base.PyomoModel.ConcreteModel'>",
    "__id__":0,
    "active":true,
    "data":{
      "None":{
        "__type__":"<class 'pyomo.core.base.PyomoModel.ConcreteModel'>",
        "__id__":1,
        "active":true,
        "__pyomo_components__":{
          "a":{
            "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
            "__id__":2,
            "_mutable":true,
            "data":{
              "None":{
                "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
                "__id__":3,
                "value":1
              }
            }
          },
          "b":{
            "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
            "__id__":4,
            "_mutable":true,
            "data":{
              "None":{
                "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
                "__id__":5,
                "value":2
              }
            }
          },
          "c":{
            "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
            "__id__":6,
            "_mutable":false,
            "data":{
              "None":{
                "__type__":"<class 'pyomo.core.base.param.SimpleParam'>",
                "__id__":7,
                "value":4
              }
            }
          },
          "x":{
            "__type__":"<class 'pyomo.core.base.var.IndexedVar'>",
            "__id__":8,
            "data":{
              "1":{
                "__type__":"<class 'pyomo.core.base.var._GeneralVarData'>",
                "__id__":9,
                "fixed":false,
                "stale":false,
                "value":1.5,
                "lb":-10,
                "ub":10
              },
              "2":{
                "__type__":"<class 'pyomo.core.base.var._GeneralVarData'>",
                "__id__":10,
                "fixed":false,
                "stale":false,
                "value":2.5,
                "lb":-10,
                "ub":10
              }
            }
          },
          "f":{
            "__type__":"<class 'pyomo.core.base.objective.SimpleObjective'>",
            "__id__":11,
            "active":true,
            "data":{
              "None":{"__type__":"<class 'pyomo.core.base.objective.SimpleObjective'>",
              "__id__":12,
              "active":true
              }
            }
          },
          "g":{
            "__type__":"<class 'pyomo.core.base.constraint.SimpleConstraint'>",
            "__id__":13,
            "active":true,
            "data":{
              "None":{
                "__type__":"<class 'pyomo.core.base.constraint.SimpleConstraint'>",
                "__id__":14,
                "active":true
              }
            }
          },
          "dual":{
            "__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
            "__id__":15,
            "active":true,
            "data":{
              "14":0.9999999626149493
            }
          },
          "ipopt_zL_out":{
            "__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
            "__id__":16,
            "active":true,
            "data":{
              "9":2.1791814146763388e-10,
              "10":2.004834508495852e-10
            }
          },
          "ipopt_zU_out":{
            "__type__":"<class 'pyomo.core.base.suffix.Suffix'>",
            "__id__":17,
            "active":true,
            "data":{
              "9":-2.947875485096964e-10,
              "10":-3.3408951850535573e-10
            }
          }
        }
      }
    }
  }
}