# GPflow 2 Upgrade Guide¶

This is a basic guide for people who have GPflow 1 code that needs to be upgraded to GPflow 2. Also see the Intro to GPflow with TensorFlow 2 notebook.

## Kernel Input Dims¶

The `input_dim`

parameter has been removed from the `Kernel`

class’s initialiser. Therefore all calls to create a kernel must be changed to remove the `input_dim`

parameter.

For example:

```
-gpflow.kernels.SquaredExponential(1, variance=1.0, lengthscales=0.5)
+gpflow.kernels.SquaredExponential(variance=1.0, lengthscales=0.5)
```

**Note**: old code may still run without obvious errors against GPflow, since many kernels take an optional numerical value as their first parameter. You may not get the result you expect though!

## Parameter and tf.Variable¶

The `Parameter`

class in GPflow 1 was a separate class from `tf.Variable`

. The `params_as_tensors`

decorator or the `params_as_tensors_for`

context manager were required to turn them into a tensor that could be consumed by TensorFlow operations.

In GPflow 2, `Parameter`

inherits from `gpflow.Module`

(a `tf.Module`

subclass) that wraps a `tf.Variable`

, and can directly be used in place of a tensor, so no such conversion is necessary.

References to `params_as_tensors`

and `params_as_tensors_for`

can simply be removed.

## Parameter Assignment¶

In GPflow 2 the semantics of assigning values to parameters has changed. It is now necessary to use the Parameter.assign method rather than assigning values directly to parameters. For example:

```
# Initializations:
-likelihood.scale = 0.1
+likelihood.scale.assign(0.1)
```

In the above example, the old (GPflow 1) code would have assigned the value of `likelihood.scale`

to 0.1 (assuming that likelihood is a `Parameterized`

object and scale is a `Parameter`

), rather than replacing the `scale`

attribute with a Python float (which would be the “normal” Python behaviour). This maintains the properties of the parameter. For example, it remains trainable etc.

In GPflow 2, it is necessary to use the `Parameter.assign`

method explicitly to maintain the same behaviour, otherwise the parameter attribute will be replaced by an (untrainable) constant value.

To change other properties of the parameter (for example, to change transforms etc) you may need to replace the entire parameter object. See this notebook for further details.

## Parameter trainable status¶

A parameter’s `trainable`

attribute cannot be set. Instead, use `gpflow.set_trainable()`

. E.g.:

```
-likelihood.trainable = False
+gpflow.set_trainable(likelihood, False)
```

## SciPy Optimizer¶

Usage of GPflow’s Scipy optimizer has changed. It has been renamed from `gpflow.train.ScipyOptimizer`

to `gpflow.optimizers.Scipy`

and its `minimize`

method has changed in the following ways:

Instead of a GPflow model, the method now takes a zero-argument function that returns the loss to be minimised (most GPflow models provide a

`model.training_loss`

method for this use-case; gpflow.models.SVGP does not encapsulate data and provides a`model.training_loss_closure(data)`

closure generating method instead), as well as the variables to be optimised (typically`model.trainable_variables`

).The options (

`disp`

,`maxiter`

) must now be passed in a dictionary.

For example:

```
-optimizer = gpflow.train.ScipyOptimizer()
-optimizer.minimize(model, disp=True, maxiter=100)
+optimizer = gpflow.optimizers.Scipy()
+optimizer.minimize(
+ model.training_loss,
+ variables=model.trainable_variables,
+ options=dict(disp=True, maxiter=100),
+)
```

Any additional keyword arguments that are passed to the `minimize`

method are passed directly through to the SciPy optimizer’s minimize method.

## Model Initialisers¶

In many cases the initialiser for the model will have changed. Typical changes include:

Instead of separate parameters for

`X`

and`Y`

, some models now require a single`data`

parameter containing a tuple of the X and Y data.The

`kern`

parameter has been renamed to`kernel`

.

For example, for the `GPR`

model:

```
-model = GPR(X, Y, kern=kernel)
+model = GPR(data=(X, Y), kernel=kernel)
```

Models that do not take a `likelihood`

argument because they hard-code a Gaussian likelihood (GPR, SGPR) now take a `noise_variance`

argument that sets the initial value of the likelihood variance.

## SVGP Initialiser¶

The SVGP model’s initialiser no longer accepts X and Y data. Instead this data must be passed to the various computation methods of the model (`elbo`

, `training_loss`

etc).

In the Introduction to GPflow 2 notebook there is an example of how to use SVGP with optimisation using mini-batches of data.

In addition, SVGP’s `Z`

parameter has been removed. To pass-in inducing points use the `inducing_variable`

parameter. Also `SVGP`

’s `feature`

attribute has been renamed to `inducing_variable`

.

## Autoflow¶

The `@autoflow`

decorator has been removed. Since eager execution is the default in TensorFlow 2 this is no longer necessary.

You may wish to consider wrapping functions that were previously wrapped in the `@autoflow`

decorator in the `tf.function`

decorator instead, to improve performance (but this is not necessary from a functionality point of view).

## Use of tf.function¶

Wrapping compute-heavy operations such as calculating a model objective or even the optimizer steps (such as `tf.optimizers.Adam().minimize()`

) with `tf.function`

is crucial for efficient computation.

**Note**: you should ensure that functions wrapped in `tf.function`

are only passed **tensors** (not numpy arrays or other data structures, with the exception of a small number of bool or enum-style flags), or the decorator will re-compile the graph each time the function is passed new objects as its arguments. See the TensorFlow documentation on re-tracing for further details.

You can convert a numpy array to a tensor by using `tf.constant`

. For example: `compiled_function(tf.constant(numpy_array))`

.

## Model Compilation¶

Models no longer need to be compiled before use. Remove all calls to the `compile`

method.

## Sessions and Graphs¶

GPflow only supports eager execution, which is the default in TensorFlow 2. It does not support graph mode, which was the default execution mode in TensorFlow 1. Therefore all references to Sessions and Graphs should be removed. You should also remove references to the `gpflow.reset_default_graph_and_session`

function.

**Warning**: code that creates graphs (for example `tf.Graph().as_default()`

) will disable eager execution, which will not work well with GPflow 2. If you get errors like “’Tensor’ object has no attribute ‘numpy’” then you may not have removed all references to graphs and sessions in your code.

## Defer Build¶

The `defer_build`

context manager has been removed. References to it can simply be removed.

## Return Types from Auto-flowed Methods¶

GPflow methods that used the `@autoflow`

decorator, like for example `predict_f`

and `predict_y`

, will previously have returned NumPy Arrays. These now return TensorFlow tensors. In many cases these can be used like NumPy arrays (they can be passed directly to many of NumPy’s functions and even be plotted by matplotlib), but to actually turn them into numpy arrays you will need to call `.numpy()`

on them.

For example:

```
def _predict(self, features: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
mean, variance = self._model.predict_f(features)
- return mean, variance
+ return mean.numpy(), variance.numpy()
```

## Parameter Values¶

In GPflow 1, `Parameter.value`

was a property that returned the numpy (`np.ndarray`

) representation of the value of the Parameter.

In GPflow 2, `Parameter`

behaves similar to TensorFlow’s `tf.Variable`

: `Parameter.value()`

is a method that returns a constant tf.Tensor with the current (constrained) value of the Parameter. To obtain the *numpy* representation, use the `Parameter.numpy()`

method:

For example:

```
-std_dev = np.sqrt(model.likelihood.variance.value)
+std_dev = np.sqrt(model.likelihood.variance.numpy())
```

## Model Class¶

The `Model`

class has been removed. A suitable replacement, for those models that do not wish to inherit from `GPModel`

, may be `BayesianModel`

.

## Periodic Base Kernel¶

The base kernel for the `Periodic`

kernel must now be specified explicitly. Previously the default was `SquaredExponential`

, so to maintain the same behaviour as before this must be passed-in to the `Periodic`

kernel’s initialiser (note that `active_dims`

is specified in the base kernel).

For example:

```
-Periodic(1, active_dims=[2])
+Periodic(SquaredExponential(active_dims=[2]))
```

## Predict Full Covariance¶

The `predict_f_full_cov`

method has been removed from `GPModel`

. Instead, pass `full_cov=True`

to the `predict_f`

method.

For example:

```
-f_mean, f_cov = model.predict_f_full_cov(X)
+f_mean, f_cov = model.predict_f(X, full_cov=True)
```

## Predictive (log)density¶

The `predict_density`

method of GPModels and Likelihoods has been renamed to `predict_log_density`

. (It always returned the predictive *log*-density, so no change in behaviour.)

## Settings / Configuration¶

In GPflow 2, the `gpflow.settings`

module and the `gpflowrc`

file have been removed. Instead, there is `gpflow.config`

.

`gpflow.settings.float_type`

has changed to `gpflow.default_float()`

and `gpflow.settings.int_type`

has changed to `gpflow.default_int()`

. `gpflow.settings.jitter`

/`gpflow.settings.numerics.jitter_level`

has changed to `gpflow.default_jitter()`

.

These default settings can be changed using environment variables (`GPFLOW_FLOAT`

, `GPFLOW_INT`

, `GPFLOW_JITTER`

, etc.) or function calls (`gpflow.config.set_default_float()`

etc.). There is also a `gpflow.config.as_context()`

context manager for temporarily changing settings for only part of the code.

See the `gpflow.config`

API documentation for more details.

## Data Types¶

In some cases TensorFlow will try to figure out an appropriate data type for certain variables. If Python floats have been used, TensorFlow may default these variables to `float32`

, which can cause incompatibilities with GPflow, which defaults to using `float64`

.

To resolve this you can use `tf.constant`

instead of a Python float, and explicitly specify the data type, e.g.

```
tf.constant(0.1, dtype=gpflow.default_float())
```

## Transforms¶

These have been removed in favour of the tools in `tensorflow_probability.bijectors`

. See for example this Stackoverflow post.

GPflow 2 still provides the `gpflow.utilities.triangular`

alias for `tfp.bijectors.FillTriangular`

.

To constrain parameters to be positive, there is `gpflow.utilities.positive`

which is configurable to be either softplus or exp, with an optional shift to ensure a lower bound that is larger than zero. Note that the default lower bound used to be `1e-6`

; by default, the lower bound if not specified explicitly is now `0.0`

. Revert the previous behaviour using `gpflow.config.set_default_positive_minimum(1e-6)`

.

## Stationary kernel subclasses¶

Most stationary kernels are actually *isotropic*-stationary kernels, and should now subclass from `gpflow.kernels.IsotropicStationary`

instead of `gpflow.kernels.Stationary`

. (The `Cosine`

kernel is an example of a non-isotropic stationary kernel that depends on the direction, not just the norm, of \(\mathbf{x} - \mathbf{x}'\).)

## Likelihoods¶

We cleaned up the likelihood API. Likelihoods now explicitly define the expected number of outputs (`observation_dim`

) and latent functions (`latent_dim`

), and shape-checking is in place by default.

Most of the likelihoods simply broadcasted over outputs; these have now been grouped to subclass from `gpflow.likelihoods.ScalarLikelihood`

, and implementations have been moved to leading-underscore functions. `ScalarLikelihood`

subclasses need to implement at least `_scalar_log_prob`

(previously `logp`

), `_conditional_mean`

, and `_conditional_variance`

.

The likelihood `log_prob`

, `predict_log_density`

, and `variational_expectations`

methods now return a single value per data row; for `ScalarLikelihood`

subclasses this means these methods effectively sum over the observation dimension (multiple outputs for the same input).

## Priors¶

Priors used to be defined on the *unconstrained* variable. The default has changed to the prior to be defined on the *constrained* parameter value; this can be changed by passing the `prior_on`

argument to `gpflow.Parameter()`

. See the MCMC notebook for more details.

## Name Scoping¶

The `name_scope`

decorator does not exist in GPflow 2 anymore. Use TensorFlow’s ``name_scope`

<https://www.tensorflow.org/api_docs/python/tf/name_scope?version=stable>`__ context manager instead.

## Model Persistence¶

Model persistence with `gpflow.saver`

has been removed in GPflow 2, in favour of TensorFlow 2’s checkpointing and model persistence using the SavedModel format.

There is currently a bug in saving GPflow models with TensorFlow’s model persistence (SavedModels). See https://github.com/GPflow/GPflow/issues/1127 for more details; a workaround is to replace all trainable parameters with constants using `gpflow.utilities.freeze(model)`

.

Checkpointing works fine.