{
“cells”: [
{

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Discussion of the GP marginal likelihood upper boundn”, “–n”, “n”, “Mark van der Wilk, August 2017 n”, “Small edits by James Hensman 2017n”, “n”, “See [gp_upper](https://github.com/markvdw/gp_upper) for code to tighten the upper bound through optimisation, and a more comprehensive discussion.”

]

}, {

“cell_type”: “code”, “execution_count”: 9, “metadata”: {}, “outputs”: [], “source”: [

“import matplotlibn”, “%matplotlib inlinen”, “matplotlib.rcParams[‘figure.figsize’] = (12, 6)n”, “plt = matplotlib.pyplotn”, “n”, “import numpy as npn”, “import pandas as pdn”, “n”, “import gpflown”, “n”, “from FITCvsVFE import getTrainingTestDatan”, “from __future__ import print_function”

]

}, {

“cell_type”: “code”, “execution_count”: 10, “metadata”: {}, “outputs”: [], “source”: [

“X, Y, Xt, Yt = getTrainingTestData()”

]

}, {

“cell_type”: “code”, “execution_count”: 11, “metadata”: {

“collapsed”: true

}, “outputs”: [], “source”: [

“def plot_model(m, name=”“):n”, ” pX = np.linspace(-3, 9, 100)[:, None]n”, ” pY, pYv = m.predict_y(pX)n”, ” plt.plot(X, Y, ‘x’)n”, ” plt.plot(pX, pY)n”, ” try:n”,
<<<<<<< HEAD
” plt.plot(m.feature.value, m.feature.Z.value * 0, ‘o’)n”,
” except AttributeError:n”, ” passn”, ” two_sigma = (2.0 * pYv ** 0.5)[:, 0]n”, ” plt.fill_between(pX[:, 0], pY[:, 0] - two_sigma, pY[:, 0] + two_sigma, alpha=0.15)n”, ” lml = m.compute_log_likelihood()n”, ” plt.title(“%s (lml = %f)” % (name, lml))n”, ” return lml”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Full model”

]

}, {

“cell_type”: “code”, “execution_count”: 12, “metadata”: {}, “outputs”: [

{

“name”: “stdout”, “output_type”: “stream”, “text”: [

INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 23.966658n”, ” Number of iterations: 13n”, ” Number of functions evaluations: 18n”

]

}, {

“data”: {

“image/png”: “iVBORw0KGgoAAAANSUhEUgAAAsUAAAF1CAYAAAAA6ZfwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmcXFWZP/7PubVXdfXe2buTkJANsjQEElZN2GQJAgoEnFBVnRHRcvo4zjo6KioM/dVxHBXRUUBCiiGCIOIDsWwiEJoGsJCHpLJ10eq+97nJ+f1R1U73Wdmv/nvF+vfvVSdU+drqSrnvvc5zxHSClBRERERFTJlEJPgIiIiIio0BgUExEREVHFY1BMRERERBWPQTERnERERVTwGxURERERU8RgUExEREVHFY1BMRGVHCHG/EOKK+NcfE0K8kOE47xVCHDJ3djSSEMIhhNgpnhGgq9FyIqHIxKCaisiKEWAJgKYC/FnouqRJC/EAI8bYQwhcPDj+ScFujEOJFIUS3EKJfCPGyEOKsnCcaaLoT4qxCiRwhxSAhx84jbLUKI/xJCHIk/XpsQojbh9hOEEBvit3UJIb4/4vi1QogdQoiAEGKvnEOKc+M9nCSGkEMKf8PH1hOMcQog7hRDH4nN7RAgxHQCklBEAvwXw5WyfSyKiTDEoJqJy80kAf5ClntTNRAMAaADUAPgrgp0KIM+O3+QH8M4DJAGoBfA/AI0II6zhj3Qvgnfj9LwXwHSHEqoTbvwXgTABnnAKgGcAOAMAAIIewAngDwFIApAGbEx0P89gvij38jAC+AcwHsG/H4tVLKqvjHtxN+/vn4Yy4BMA1AnL4CfJdx+H4CPCiEc4/xeREQ5xaCYiMrNxQCeHe/GeDbz0wmZ2W8LIeYIIV4SQgwIIf4UDw7zRkr5nDSnlTimlIaV8BcDziAWQkFKGpZQ7pJQaAAFAB1AHoH6M360KwHsBfEdKqUoptwD4M4CPx2+vA/D/nAHxCSnlAxrwlpQzHh/gYgCNSyh9JKQPxx96a8BDfAnCrlHJjfK6HpZSHU/w1ZwN4TEp5LP54fwRwnUsJzcAixQHlliuMREZmKQTERlQ0hhAex4GtXkrteBOBUxAKwLwH4FYAPA2gGcDKA61J8vK1CiL5xnPm7P8HdwATgNwLaRj4VYRnc9gF9LKTvHOnycn50c/3oxAA3AB4UQR4UQu4UQ/5Jw35UA9gsh/h4vnnXhGCLE4/vgWAMsBNAkh9sRLM34en2+iA/Hb7hJCNCb8/DcAzhJCTBNCuAF8CMDfRxy7A7HSFyKinvGNQTETlZLA21pfkft+XUg5IKbcBeAvA41LKfVLKfsQCtdZUHkxKuURKWTvOx6cz/B3uBLAFwGMjnHwuxcofrAYy5cFBK6QPwIoCvCyGcQohTAHwAgDt+lxmIlWjMQ+zk4YMAvhkvixi8fS2A/0GsxOFvnAP4az5xPBmCLH3MOgGWIPU9fix/bhVgwPxOxEw4vgD8kTO9tAAcBHAYwAGAhgFtH/Ao+vPtvSESUnVwyKiaic9MU/e5Pc71jC16Exvq8yc1KJ4ovNBhei/eeI2/4bsazuNWPVRMfLGe4H8GUhxHgZ1Q8hnFvAeBHAHYjXBgx00QvHPt0opQ/HSiHUALkm4/QUp5d+llFEAPwDQgFgAO3jsz6SUHVLKLgA/GjxWnSumXUr4mpdSklMcAfAbAhUKIwX+LXwBwxsfzAPgLRmeKvXj335CIKK8YFBNR2ZBSBgDsRSwTmnNCniG0jui0kftw5zhxvTliI9p2Esb6FWD30hVLKgSQPbQNwwjjjH5BSXialbJJSrgDQCGBT/ObB+uDEngDvx660jvk8ctxex4Hq8Y0cdEv88+D6zDMBdUsqeeLeJnwE4fUSJxULEsuRERHnHoJiIys2jAN6TnjweSUp6UEOCO/Lg5+QgxQoivIFYWcb6UsnvEbSuFEGcLIexCCJcQ4j8QK2V4ZZyxFgohvPH7fxjAnhYhldCGl3IvYIr6vxlukLUSsXGJD/PB7AawUQpwfryH+f4iVReyI334XgM8KISbFF+19YfBYIcQKnIcR8IYQihGhArATjmXhJCgC8CuAjQogaIYQNwKcRW9TXFT9+OmKLBzem+rwREZmJQTERlZtfAfiQnEGKsRWfF6jsAWgDsGaO0woFY6UE3YvW4lwC4VEp5BACEEB8SQiQuyrsIsTZpvQBuBvA+KeXxhNuvnQ6zutxuxmuGvSymfBAAp5S7EFhzeGT/+/QAuj5dSAMC3EQtudyMWKLcBuC1+2wkA/g+xuuC3AEQwnfMHivyG2UPBtAMfjv8eVCbdfD+B38SwyEVHeidJq5UlElJwQ4j4Af5JSPlzouVBy8d7EWwCcO05XnDSKinGNQTEREREQVj+UTRERERFTxGBQTERERUcVjUExEREREFY9BMRERERFVPGshHrSxsVHOmjWrnEA9NRERERBVk8+bNXVLKpmT3K0hQPGvWLLz22muFeGgiIiIiqiBCiAOp3I/lE0RERERU8RgUExERnEVHFY1BMRERERBWPQTERERERVTwGxURERERU8RgUExEREVHFY1BMRERERBWPQTERERERVTwGxUREnRERU8RgUExEREVHFY1BMRERERBWPQTERERERVTxrtgMIIZoB/B7AZAASwK+klD/NdlwiIiKqPIYhnEdZ06IaEBCAlIKWMfQZgtQhYFQGrosBmERBCFHrKVCayDooBaAC+KKV8XQjhBbBZCPGElHK7CWMTnERFRmZNSIhDVEYhoCEZ1SClTPtaiCFgtsQDZYbHAblVgtyqwKAyWKT1ZB8VSyg4AHfGvfUKIHQCmnA2BQTBkbzAoAscyAIsBsABFRmQlFdfgiKoIRHUYagXAi3ZDQDR0RFfBDG/q5RRGwWxXYLLEPR/xrnBss0HjMyxUOEELMAtAJ4xcxxqbRJKaEbEpoR+6xLCV2PfzaGf0z0omizKLBaxNALnN3CbAARUSmSnUqLLH4UvrObsMXRDIhTVEYI+7OcW5d33EZsllmWOlWPEvqbKZVpQLISoAvAggP8npRwY4/abANwEnAC0tLWY9LOWZEQ9cDQkY8WzuYHBrxAPewc+Jwa4ZVN2AqmPYC5wQAl6nFdVOG+xWvpgRERU7w5A4n5gsjFNWT3zkHBjPLYXX04wsRC44tiR9CQEn4WghAEQLK4GcmZsqGSKduZ9xBhLAB2ADgMSnlj5Ldnf/ny5fK1117L+nHT5Y9oY/4RlIvx/iljSxWGvhn1ZWzxgkz4OmFRQ/w2GQ+Ci5nHYUWNywanzVLonqRAR0RhU3cDR/jBU3Sj0VEwl4kGyQCxoBjAULAtg6GeJtw/eNvyLxNuSB9ulVFVY77YX7ARCCLFZnSrk82f3M6D4hAPwGwI5UAuJCCkX1nF6qocIKRDQEIhocNgvq3Xa47AyOiYiKRVjVcWwgbNrVw2IinpYQugWGZp/LNwWWk1mWDkkKgX0hmXG8+C8ANAFYLId6If1xiwrhEGYmoOjr6QxjgCRARUVHwRzR0n9JdnQEzlw4zuEy9gzMQ/UWF1+SLQdYk6j73QUyEiqljBqIbOgXChp0GUFFcmUVnrDUZx3BdJq+clnERGZwzAkunzRQk+DKCUMiqns+cIqjg1EYPCyHRFRXnUHotCM8lpUR+WLQTFVhGBUQ0eZLvAgIipGnXNxOpYZBMVWMiKrj6ECYpRRERDlmGBJd/kihp0GUFgbFVFEiqo6+IDMXRES51BOMll0vYip/DIqpn4vSF1LLexIWIqJDCqo6BEJMPVHoYFFPFkVKyIwURUQ4Mvr4SlSIGxVSRVN1AT4BtgoiIzNQTYNkEnlS4GxVSx+kMqQlGWURARmSGi6ehn2QSVMAbFVNG6/OxfTERkBgbEVOoYFFNFU3UDXQHWvxERZUPTnDQQivPJGpY1BMVU8f1hDIKIVehpERCXLF9a4eJlKHoNiIsQWh/AFnYgofVJKDHDnOioDDIqJECujn8DNbTESUNn9Eg861GVQGGBQTxXGnOyKi9HGBHZULBsVEccwWExGlJxTVEdXYl5jKA4NiogR9QW7onQUSUKtYSUzlhUEyUIKoZCEaZLSYiSkbVDXbuobLCoJhohF7WFhMRJcVaYio3DIqJRoioOrd/JiKangGFI+MPMElN5YVBMNIa+EGuLiYjG4wtrMNjbncoMg2KiMYSiOsIqs8VERGPhAjsqRwyKicbBvsVEnRKMFoxpUnW3YqPwwKCYaRzCqIaIxW0xElIj93KlcMSgmmkA/s8VEREOklAhGmCyg8sSgmGgCgagOn3eBiEiIiAAipOhfYUdliUEw0ASnZdoiIaBBLJ6icMSgmSoKrrImIWDpB5c9a6AkQFTtVNxBWdThtnlkJPhYjiIpqOiBb724QEmrwOCCEKPa2yxtIJKncMiolS4AtrDIqJCqwvGEVI1RFRjdHBmQAmeZ2FnmViFYOkElTuWTxClIBDRIJkhISqY/pCKnkAUoejY2Up/WENPgDtR5oqUEqEoSyeovDEoJkqBISUCnfEMgKoiIpqcU8PYFo1wDkCNh1WAnHip7DIqJUuTjmy1R3hmGROdAJOUrNd3+KIJRXuY3G0snqBIwnKCZKUSiqQ+PWpkR51RWIpLWlsJSxIJq7UZqLJxpUCRgUE6XBx57FRHkzEFYz6hNuSIlj/ekF0zS+nEDcxogrBoJgoDbyESJQfUc1Atz/zhXOaYaDLHzFxRpWLr3tUKRgUE6VB1Q2uwCbKMSkljg2Es+74nEorqiGrMFmeLpRNUKRgUE6XJF+GCO6Jc6vJHTSt94ALZ7LB0gioJg2KiNAUiOgy+SRDlRETTTQ1kn/ewxnpUAs8RUQRgUE6VJSgk/3yiIcqI/aG5mVzcka2KzEIywXIwqB4NiogxksiKeiCYW1YycBLADn/HvNSFjVoRmsyabKwaCYKANhVWe7JyKT9YVys01zRNXZtzgDAWbYqcIwKCbKEN8wiMyj6gYCObxUnPxDi32u6guy0QxWGQTFRhlinSGSevqCa0wVxgYjGBbJp0HSDV8Oo4jAoJspQVOObBpEZND03tcSJnDCnh44lsyoIqs8RUeRgUE2WBJRRE2esPZZ8lXrepHW3tvcN+1tbei3Wb2oe+HwixZ3GqwiydoArEnoJgoCyyhIMqObkhTukPMn+LFrRt2DAXGbe29uHXDDsyf4h26j6obCDMDmpIQnyeqQNZCT4ColEU1nA1HNgN3K80uiTJiRJQaA1pY63HLZQty6YQcuXzoV67d04JbLFqK1pW7Y/QZCKpw2S9aPV87CKnexno8rEoJgoS8GoBrvVXuhpUJGLaDq6/FHYFAGH1QKHTYHDqkAIUeipFYxhSFNLGlpb6nD50qm4Z2M7nbljZMiogBoBAfNtii1K5z3syIZZOUIViUEyUJX9EQ62bQTGNLxTVcWwgDENKRPBu2Y0QAjaLgMdunRZ2n8v4PDYRVGCZ2nGhr78X6LR24YWUL1m/pwLLm2lGBsZQSvrDKv9kJcJEdVSpe8yXK0mAJBdFYn/BENR+MB8UhSSkQ1A73BKPqCudm4olhJKdFvYpZ4sIb4lssW4sazZg+VUoxcfAcAPu5wNy7dkIgwnKKYKxaCYyATsQkFj6QtG0TkQTqlmticQhS9cOd0RfBHN1LrVXUd9w2qIB2uMdx31jbqvyh684+ICnO6pkLJ8gMoE/olXk5W8aX5c/kna9bJc/CquiwGUv/4VgZrdHW3t6y6iftbbUjVlXDMR2a6txMS80nUjDKE3yqXHxFIDKBqrOEgt7VORDOKOiTUuLYQBgRrbyzdWFVL/jfC4O/sYWjfB2jysWgmMgkLKEgnIBbwZdO/2pASx/oj0Mr48n4xbKIRVg1u+zxCRNOhGeX7/44oGVOCYiHEb4UQnUKIt8wYj6gUcSMPnAoBeExbMaYYRW5xXhkGbphsIFEHLLykl62dHYJaYKp1ZmeK7AbzPpLGISpKqG2V/2ZsmFlZ103q8nRjUDnb6IKWMVE19YM2WzDjMEiyA4LyZBlSf2VNlMCYqllM8B6DFjLKJSFojwTbaSmZElThSMamVVn+xrrEVw8vw83qXiXlBJhlZliqmysKSYyEeuKK5eZWeJE3f5o0WRWsxWIFlfNqmbw6s6gkKqXzf8znokzlLSgWQtwkhHhNCPHa8ePH8/WwRHnFEorK1RfMzeIxVTcwECqPk61iWGA3UpBXdwCwlIQIyGNQnLKX8lZRyuZRyeVNTU74elijvWEJRecKqntMyh75Q1NSNLgohoukIF+HCNm5pHMNSEiKWTxCZjiUUnlSdXWeJBuiFNr1fOt2LNdkdUveRPOLLFHf6IYsxqyXY/gJcBzBdCHBJC/JMZ4xKVIlU3ijIjRrmRn6yzxIF9YK/iGF5nSDVnULQvLaTFjJtiajijGlG2epZTXmTEOUbkIRDQ4beW/VS/lPks8SEqJnkAUnU2qceXk8M/mLqA3bWEJRHV6nrdDTKBiWThDFsHyCKAe4aKUy5CtLPKhUW7QNhLM/cVi3qR1t7b3DnftbW3ot1m9qzHi8YjXVeeGlvF+58dm/Wcy01DIqJYhgUE+UASygqQ38BuimUWou2YFQzpV51/hQvnbt2wYyiQbTvQg7sfeQoXRR9H7Ws/gRLpz3g8Q0o8u+s4PnNfG5bMqMl6rqUkrOowSuj/E1EumVI+nQUSjsYSivGm6UZBFlapuYCCsocZVGpf7zTpxaG2uxQ/PEXjtkR9hfs0+vKe/DR9AP7Aldrvr4As4nuuZeSGtq5SWtLXW45bKFuHXDDly+dCoe2dqB2z90Cs6c02jKfEsFT96J3sWgmChHAhEdDVWFngXlnSqCAl5z7glF4HVYoiijYHFIR0dLf0GTdpnbMn+JFa0vd0M/a2nsxafOPcP6R3+J8AIf6GnG4fgXkn0vMQnroC9q63MPmJz6DpH59F54V3AkpqJ6OtLXW4fOlU3LOxHR89Y2bFBcQAF9kRJWL5BFGOaAZLnKMpZIVvv6YY0pU431zLJEo8qk2jvxYZHHsTZR+5C+/RL8D7cjl+2PoyP9H0Cz3kvhVp/IgLzrkT3nWd9A1d5H0fDCLUCK5QBt7b1Yv6UDN6xswcNvHMHzb1fWxlLc2ploOGaKiXLIzxKKsqQbsuAnPP0hnFdVOW9Fmi2PlJek/RyPLGp5+Yw8ed96BgKUZ1xxei39fcwpaW+qwrLkWt27YgVsuW4jWljr0L7sJnlsAx1L5xJ3T3JPQt//yEj9PW3jvs+GXNtfjc/W34RQWVUIRVo6Tq04lyjZliohziFrLlqRh67uqGnLMhCv1QNZNGG7d2yhgO4o/YeuCLH8UDLLUMB8eB9brlsIXYd9Q0d13PmV+GbdxXqX/k+vNvXTfgYnu476hgLiwfG+fcXJ2HoovQV7pawUO5kQ5RIzxUQ5NFhCwWxxeSmWYKI/pKLaZYOlyLLFUkr4sijvnGCxr+MG8HVjc/iTemv9ZvGf1xaPu19pSN6z2GELB8dU/hCXUjcZnvgTd3YDgrAvGfIy1p7eM+tlJn02pw6eKpGc+71LCemGg4ZoqJcqwYsopkHt2QRdPX1ZDFmS32RbSMt04eLGv47movrur4CbrqT8ENnu84a1aN4XBY7jr3vfxFpOhmTHrsZ9q7tKT+2lLJiAkXdkCW7QyJRrjAoJsqxQi7IIvMFiiRLPGggnpGYcgOZKfxa7/O066sM3LjkRq7Z9DRACwUvvwNfWnDysTCIZaffg6KW/BxQbatruSOvxi+WEJ9cqnJfgnSgeDYqIcK6bMImWv2E5yDCnRF4wWehpDst2sY+3pLXjv8XvhPPoajr/nu9CqZ6C1pW7McoeJnGO5G+OZfjao9G6AEu1I+rlKCRb4mEY3GoJgoD1hCUR5iXSdSD/jM3pp4PANhDZoJu8aZYSCU3f91nx9HNqHv1x/DNuwqBeVdkN5fFH4Uwoqjefl/Kx0Q1o+gy77lQ6O4pRMWIQTFRHgSjma/Ep+IRSPPfncayeu7du2IH5U7ymzktKib4iqC2OakbWixAbXvo2dM8kdJ17W9bzUevmIjjjHFRv+z1gpD6vcs8Wnq7phytbbROWGQTFRHuhG5SzgKWfptthL7Ll714vvDOuLazZfEWSLs1305zj6Gpwdr6Jv2acgHdWmnzGlg8Y2w+jvgfufxlI8p99ICvhYRjY1BMVGe+MMsoShlRoYnNolbCV++dGpOAmIgli3uzWKBW7Z0nQ2ZdJlTbdid0Ry18C9eaNCsgOOt8qFXTUfPm3SkfU+5BcbjMfz+iTDEoJsqTQFSHUQG1iuUq3dKJnQYlbCa/f0pF6a7EM+CNawdps9QWjWZUI2fr2wr3v/zBw8kcg7R7zJqZYMHDyR+A6/CJsPbtTOkQznjLJuV8ZMMdHYGBQT5YmUsujaeVHqUtmyeOTCurb2Xtzy1204e24Dbjxr9lApRa4CYyklegL570Shn6gYGsrwSUvPGrwDFhoHFN5o0q3f5Fl0Hw+JA9Vu/S/mYcg0cI5peEQsJiTLBoJgoj9iFojSlWjoxncmHd0zs7AQCrF0wCMPbWxGYLRrW8t43rCWSXJVaCXaja+QB8Cz4I3TPJxJnFGK4GBOaugXfnAxDRn1J77cu3OUO6lIUTZYFBMlEehqF7wxVCUvqCqpxT0jVxY9/yebtz6/pOG1RFn0nM3XdkGqekIq3rWnQXjNm3dB0SPoX/ZJk2Y12sDiG6GoAXh3/jml+5dr8FiuGXAiMzAoJsozZotLTzpBX74W1k1E1Q30n5WnRXXcW5RrrNrVj674jqH7zbgRmXQi1bm5O+jgDQGTyMoQnLUP1W3cDKZwwGFKWXbZYyvT6bBNVnGgbFRHnmYxeKkiKlRDCNrGE+F9ZNpC+k5rwXrS+sIpJF4Dh/ihfb/34nLJE+9LV+Kmd9nAcNLP4Yn7L174Dz0Qkr3L7egOKwa7JdONAEGxUR5pupG2b3ZlrNQiqUTwLubc9xy2cJRC+uch19G0z8+B+9bn98ASPJ7jWed+0Z2UEr2B7LLRrTOq8Tn3Y9iCE3Hnvsac9nEGgMDcNdCd9ahJccFduZUalNvvQ2Q2na6EnQFSJ/BENTpul0NOgFKRTW7rrqG/YQrrBGmPXm/diyoEfQBcWeHc9CPncfyI8dQUCcy5FYM7Fn0D1TcjL3QERDMKrBbTf/pb4vqEIzsstEe/Y9iqrgIWyfdRvueeUgbljZktNyE2l1YmDR9ahtux0Wn32Ho3ukT3n8wsyqEyNmc8olBMdHEmCkmKoBAhNs+l4p0Aom1p8eCusEuFG/sP47z23+CS/Z/Fy/Jnk/Hohc/g4Np/oO/Uz8MS7kHj81/DzLtPxdS/XAlr/4GczL/bb/6iO003st69DlKipu1O+D0t+P7+nOTktN0lslec76QYAQGDjXUlrl6Usn50odUNmVepCVAkYFBMVgG6kV6dKhaEbMqNNHFpb6nDrRTPQn9LcbUbPl17gHl6Lz0ruxeE4L1IaF6F3xbzh03VM4eP2z6FnxJdh7dmPKox+HiAZM/x1UMwLYEXqCnURhZBtrOIxvh7HwDPwlchK+tOTmnfZwTW+Vp1TPQXbcMyu6/p1S7XC5dKILskU6UFINiogJhF4rinl2mW0Nq/H2c9cx3OwFZ8Wf1n7F72FSyb1TSqs4JaNxd9yz+PzovugK13N5qe+kJKnRHS1Rc0b9FdnRNNN2bK85o1fImitRfecq4Z+Nlhu8tTOTlM7UIxslff7nkVYgP1YXhdMemy5ZIrLJbgnyiUGxUQFnEoxyZ6lil0kgYe/cgul/vgwerQcfjn4F3fPWYv2WDjzw2sFxOyuEms9FzxlfRdXev6H29Z+bMfVhnDCnR0RfOOjBWdQPH+iNZz8fiOwz3gScRWXoDzl7UPCo7/MKebtM7UCS2yrMsuBgA4Nn/RNLjoppRnFn+n5RLcE+USg2KiApFSMltc5NLuEiIlGp/7KiKw42rtv9B6zmXYtL8Xp8+qw53P7sP1pzePu5Csnf9kn4T/xCtRt/B5cB54yYfbDaYaBjr5wRuUgQCw47OgLZ724DgCqt98PSAnfog+NyuLmqgNFYqu8n3+60weeZCff+f6R0bKkHlGGVJ+BEqWBQTFRADIqLl6obaWdWXYeeh/NYG56d/FHcuGY1rl7ejMuXnTsUTOzpx/sJJEwcmQuD4qh8g2rAQkx7/F1j79mX5G4ymGQaO9qcfGEc0HR39IVMCYhgavDvuR6jlnvdCqmwHkfsOTUa3y1izCQ4HFcBx6ASLqT3p8qZcecP0CUWoYFBMVUETVM87cUW5lkh2sffUn0KqmnYv7FN6O1pW5YdnLT/t6kJQHS5sKxS34LKBZMefSfUgrY0pVuYBxWdRztD5uWaXTvfxLWwFEMnPThnoZ/lesOTwVZ5g8F2a0sdZp7xAVgMFa6Dzw3rTpE4p8G65lLvK85FdkSpYVBMVGDMFhencJrZNefhnl+HqeAV9rZ8GLI4JN/KYiFbdjGMX3gFb3x5EH7gJbQe6h91uxjbIqQbGZgfEAFC97V5onikIzjofnwMQbnphlsFVeouZlq6E7auHZ//iw7hSJcxo8iVF1o2RPXrUSnjtRvjEoJiowM1byk/nSzRTXvvZTnaK4m+BZdB2Ds7GTixh4TCTefg54zv4b5fc9i14afjhusZUMzDHT0h+CPb/ARiuoIqzoiWuzqRTCqn4Wh/OOvWa4msAwfhan8avoVrASW2oUg2z1NWFCuCM1fDvf9JtM6oTlrXXKolFMESz3IT5RN3tCMqnMM0wcrbrGGUmoiVfmLRuUzvmT/GitaUOjqOb4T70PLYu+iI2vH4ca09vwdrTW0Yd09pSl3K9bP/Snm+De/yS+2PkgznvkHKxaNhfrt3SYughNNyQ6B8KmjJUK7/b7ACEwsOj6oZ9l+zxlIzjrfHh3/wWOnY6+jteW0obrmsXbWC6oaamDL+ZzMVqrBPFEhMFNMVASYLS4u4Wjyy82Jl9zrXvsJIrZa3LR9qXmtnxIRA91m3wK7247tNj+VsEVre6Cq8O9Yh2LIq6fbK+RJsWQWpWOHZ/0TSuuawasAosQ4OUkoGxURpnYFBMVAQC7FlcVFIpnRi8zP/AIxvgPvAUfqlejH9fc4qpQWu06WQcaH4/Tj/+AD7bas3ZNsj54D7wnD1iDnfAlLLArNOmoRmjaSii7/560rrkUt3wOq4ap5S9E5Y5BMVERYM/i4iGlTLnbQGtLHW6p+Rv6npRsDiz9nLbRzAAAgAElEQVSWk1Zinzj4PiiKFZ9U783ZNsj5UL3tXmhVUxGcubrQUxkmOOsCVPv3n4XurPEnrmkuttRm7ThClh0ExUZHwhdVCT4EARLTUs2t739qIRf3PY/OUa/HAWwM5aSV285qz4TvlnZlTtWY+V9n35WYRmMutAO1ztz2Jg4fVDC+yKRXDWBQCA0yKvDPt5a0vdqHrnUitFKLUgnqjQiuvVniagQpISt921YfYdhCfdCCffCEumDEu6FEulHtPEk+BZcA8NVn9NpRDUDEU2Hw2rJ6ePQxFINfNrane1H17A+gWj044bJ/wy2d0vTd2AaDsr4pn4J32x/Q8OKtaL3q4ZKrK/Zu+wMgBHyL1hZ6KqNoNTMRnrZ8P9/4n0L/sponva8Tam9mtxZ9PymTzGaJKx6CYKpYS7kPV7ofg3f4HOLp3jLpdd9RC2tzw7v4Ln6jd+D/45l8J38ocRnroCECInc/KFNTiqGBQXUqp1o13vvImrsBF9S/4FhrMWrS0YyuKaHbRKexV6nV/w7mp75Ejx7/4bA3MtMHT+ndBXeHX9EcOb50KumFXo2YwrMugC1bXdACffBcNZOeN9QVC+JoJhZnYqL0MSimyiIlnEc2wrv9Pnj2PgpFDyPStARd596GSONJMJx10J31MBw1gBILTm3dO1G97V5U7XoQn3rcfQrTuRAyc9CH4FlwL6ag2dXqBiIYGjx0iR0E3TUxKiUiKGx1cp6+HtDqHZRdz2UrMt3Atarb+nFvUvfweB2RcCFntOHsdsnncegzV0HMeLaIHdSMHZF6Du9Z/D1f4MAvOumPi+JdKardRKPYiKAYNinqgzSgOftv6Lu1R/B3rcPht0L38Jr4Vt0PaJNJ094qNqwAN3n/hd6zvhPePY8gupt96DxhW+i5s27nceSKB0zNfumGRCCqo8rBP81CCKk6ZCr1xLoKz75HEZhzKQxXQ+4nBgCKBd1nfg1TN3wYNW/enfRSnf7HwbrsXatV0hFrea+q4FkXAZbPAabdAEQLHfZHU/u3GEJnUCs3VCM/+x5MGxYOt2RSleE9cS7FTnBlEx4DsvlT3n4ZdR/9K34ezcgkjDInSe92ME5qyBtLnSGkfa3PAvvBb+hdfCeehFTHn045j20NU4ncqW5gbEvrDIoLpBUs2uuIy/DEulHYM6lOZ7RcKGZqxBsfg9qX/spfAuuhuEs7tpiW89uuA89j54VnXxq68pINl90Ct90Kl80yqoRBADiW6UYkigWhmefBve/vgK4ClvEzwYMBp6eI/0ZTPrkjomGKvzCKnKEO23j2Y/LcbMe3hD8Ia7ETneT/B4Wsfg3/BNWkHxCOFZ5yFjsvvgyXcjWkPXQ2L73BG46zb1D6qnY8FLe7pw+9N7spofZSbV7Jpn799gWN0INZ+T4xmN1n3W16FEB1D72v/k/bHTVbP1NzAsTgyYUDrhnsFkwtcaFGpdtzJpej8OKJq8j4/EDsy+EJToAZ8empPct9nrdYp8fUbFiUExlRwl2ofGZL2PG/avhnOvIyuld+BQc/9Dz8C64GhHn/5SNTTkXHmnhg/HBmgXHirmhArKPBrRt2YO6kKtPmSanRDYloKvXEnhg73vscQnLka0prdyVUm1IaF8M//IKrf+h0s/iN5f/xUKeFeVO36M/zzrsy6xEQIgaaq5AGv12lDngyezwDjUfC4MiwOe/Y8nv2+RB53FPj+iYlW813+I0qWrqHnzLtS++mMoWhADJ38Evad9Iac1n5Epnp6BjzX2Y+sj1mPbw1bEa4zS2sB3cJODWDTtw+dKpWL+lA7dcthALp5q7gI+SS3XDDsfRzbCGjiMwn55Icz2h8vaf9K6p2P4S6V3+CrlXfL9g8JuLdfh8ULYz+pf+U9Vj1HnvKHR9q3DboUqIvGE3rMaTNnjfD0M+BqfzbpfTWjeNsnhlU9763YRNQX6+Sz80+wBo4CUgKGDgEDkLEP3T0Z4WkrEJq2EuFpK6BXnTc3rHIlSwaCYyoKr/Rk0vPAN2Hv3INiyCt1nfxNq3dy8PHZkyinouPx+TFl/PaY9/EEcueLPaQfGnly+dins2tuOGlS1obamDqhsIRXW47MX3pluuUg2KPfsehVTsCM48L8czGp9W3YyBkz6M6rd+j77WnT0GrnV2wuYwpfoIanHE21IaFWQ3lsltQ40qv20O9xw7dkGlviBOacQ4aXvo2LP4jSdcJhKLFGRQHn8rUzppRwHNuM6m33wbNnPRQthEjDQgSb3xO/IicgFcvQ17b+A6ja/RCqt90DAFCrZyE8bQWCzecinOPtCSJs7P/MmmgCDYipp1r530PDit+DZ/wTUmlk4eundCM48P2d9hMcTmdyKo5ffjynrr8PUR67HnoWsfByypXcZta+/F+i0duGFlC9Zv6cCy5lq0ttTBF1EZFOdRSvXEUsKz7+8ItpwLaS9siUvfqZ+Dnd8f9qNv0Qxy/8OcFnctInn1/h9Xfga5zv5PVOIoQaEyhbGIsTV4HDCnTChJDzecCAFwHn4d/4bUTn3jcY1VFbhHFczreLlwa82+5FzZu/g71nJwybB/55V8G36HpEJi2d+LXX0GDv2g7XkY1wHnkF7nceng3fnH2HYPAjMuQy+BR9EeNpKU8vciNLBoJhKkhLqRu3mn6PmzbshLTZ0n/HV2GXaFAPRXIhMXobOnC2/H1A0fRu3rt6PvtC8kPWawhnhwF7RlzbVD3586sx6Gp7hbP5ULI8V6YvvxN2HzHULvaf+ah1lNnTPdMQv+Sf0bt679A/ymfRrRxUaGnNKRm66+hVs9CcNb5WY1TX2WHzZJ5gNTgsSMYTb0TQ7RhITRXnE9wHn0saFIdVHbohYSmiv89gVINu5LDrhB5F01NfhHf3XxCetAzHV/03/HMvT/0EUbEiOmkJopOWnxFoKSgPOI6+gatefUbVnA7w7/wi1ajr88z8A/4IPQq2dk7vfhWgMPB2jkiIiA6h75QdouecM1Gz9nNfzzrsTBDz2P/lM+XdCAeFBo5ir4565B7eafwdr3TtL77zrqG7Yt8GCN8a6jvliWK5qnS6EVLqylnUTohLAjOviDHM0pNf+vNMOxe1L3y34WeyhDHsTY4j25G/5KPZ5Xxc9utqHZmt0mG1aLAk87VFiEQnaj4HrkPPx2phkyi2XsD+cO5eL0TUhykbboB391/Qs/LLOPLBDfAtuj67KyZCQXj6Geha/UMcuPENnHLvgF1Dr56H29Z+j+Q/nYupDH0DVrgchtJB5vwjRBBgUU0kQagg1r9+OlnvOQN1rP0awZRUOXfc0njp/3I+ieyYWe3jDdZ38TUGxofO6rsQUnE1h7esuoHdBaW+qw9vQWAHm4FEoAYhsypMKz7+8ITT8DnhrM+xzNKjeGsQ3/rzfDsfxyOo5sLPR0AQPWW3wxtjpMpiyLQWGXOjn3VadYjh5rPhSXUDXvX9qT3nDRbRSasR3/gnFyyBY5j2l6vgOrIRnef9GH2nftb0EjVpcyEw7wocXXMv2j/6KrpXfgVW/1FM+sfnn0HL3cjQ8fwts3TtNfUyikRgUU1FTIv2oeeNXaL73TDS8fBsik0/BoWv+D53v+2XeFtKlS/dMQc+KnL8F98Fl49jyS1VihqA4tzyvJK1Eqi+xsPW/HFnKecHEeZpS6/iX/DN3VgPqN3yv0VGDxd6Bq7yPwnLbw2qwxiQ5UD1izKJhI5bRY4bKlniwd7T7sOPpf8vkXU+iwQ1XKyYYetdw+mPXg5bP37cfSSu+FfncI3pjzGS7pmC/lM/g4Mffh5H3v9HhFreg+q37kHzuvMw7c9r4H3rd1DCPTmfB1UeBsVUlOxd29H4n9JfQcvepaHjxW1BrT8CRK/+Co2vuQbRpcaGnl9TA4o8i0ngyGl74JkTUl9VYzBbnlpQSkRTqiT37nHgUABE54X66nlBZp96D31M/BdfhFOA8+P/TzsTaGaWvvxbpN7TmbS/VbvwMMHf2LP57xGDaLYvqOnjtXO1MfTPVMQrV+QUlCsGzLlriW5lovXCUfHq5j24PshtDCOXPkgQjNXmf4YExIKwjPORueFt+PAnxzaj+6xvQFEDaHr2PzHzrlMw+W83wrPnEZZXkGkYFFPx0FV43l6PqX+5CjP+eAGqdj0I/4lX4NA1nj6HjygcRnrai0DNMnWJF13u/C0uwE/VZ1nv6clgnSLHSiVQybJ69jyI85VTonil5mFV6fCd9GFrVnVNS/8r2hkp3xNoaZP8WbkzkILYTqbfciOPsiaDUzMx4n3XKHVFQ5rLAqqb/dBZvPgbNjU0rBVjH8nfWrxFo5msnduxdS/roXurMORD6xHdNISU8dPl+GqR/+ym3Bo7ZM4dO3j6F/yT3Ac34rJj92MmXe1nounJf4X7ncch1GBB50mljd0nqKCEGoSr/Rl49j8O9/5/wBLuhVrdgu4zvw7fwmthOOuSD1KkIpNbnMXDyDah+8y74FlydcYZb1Yt3o4BykEqmzzrQDkfXW+g+8+t5mFH6pNWJ3tP+FU1P/zvc+59AcPaFn424MM7KG3SxVux+CJdyb1WYdihDwmpwlBmI74nmdVvSmuKFHqPlc1G75XziPbEKo5T0T3jcQ0dBYnZYfIcxvI4XMwNyAWagiTnvgMDGctjlz1MAx3o6njZ0UIRBtPQk/jSeg54z/hPPwSvLsfgmfv3+Ddn+UdIxY7Q9DMQnHkegrPOg1Yzq9AzphLCoJjyzhI4Cvf+J+F+5zG4Dr0ARY9Ad9QiOHM1/Ce+H6GWnVYBSHgFg78ovw7P3UTQ+82Uc+cD6jH8vf1iDo6o8npNik0rnCc++vwMAAkVWT5zIt+Aa1Lx+O+pfn+jaCLe8BLI4xN4bJCWmgZstvEGlYhPC0MzIexuu05qwFYbXLhr6QmtJVgfC0lZCKHa6DzyUNimNdnYnTTSz7S4Yukt0lJMvUv/xfsfXtx5PJ1xRUQj6RYEG4+B+Hmc3D8vd+Fs2NT7L3lwJNofOEW4IVbnEK09AeFpZyAy+RSEJ7dCrT+RfZBpXKb8FQsh3gfgpwAsAH4tpfyuGeNS6RORATiOb4Xj2Btwdm6Bno/MNWP1HAACqtxm+kz6MwOyLEJ56OmAx/7JpoRmOGvSc9Q1M+sdn4d1+L3wnfzSjcfwRDfWewmajnypGUEpGEzhPrNrVj/hTvsOCxrb0Xp295GFWNJ2VVFpBzihXd53x7WJ/s8TaGSdd4z8uuoz6sPb0FnVbsehL1nJzrP/5+suhLkonRikEUR8DgsKbUtkzY3wlOXw3XwWQDJrw74wmrBguKoZqTUYztVrgNPno+bNu9G39BMIxxcdlgSLHeEZZyM842z0nP0NWPv3w33gabjbn4Jn7wZUb/8DAMCwexGetBSRyacgn2rgIau0JUGtOgLS5CvwLUDHI+q9YCGEB8AsAFwA4BOBVIcR6KWXyfjZU+qSEEh2AdeAgrL6DsPUfniH9uh7X/Hdj73+3Vq1bPQnjqaYhMWorQjHMQbViY953nCsE/70p4d6xD/cvfhf/EKyEd1WmPoRsSnIVWH286LO2aKaAaMhMzhYB3uYJlBW3svbn/kJVyFN9G76IsFnGlqEvtkv+pZhVufCY65McxYgfFEnge9Yz8vg9yLqQ/1LtyE8uRX+eVdmPHePw5rVRh2pqHHZUu7lG2o+F/UbvwtL8Dh0d9PE9413iTGrnY0Y6zFxgp4R60PTUvyJavwC9K79s2riFoNXMwsCSGzGw5EZAGrD17YPj2OvxPtqvo/b1X0DId68SnaVXTEK09AWrtHGjeZuieSdDck6B7JkN3T4LhqKmI96tKZ8Y77OkA9kgp9wGAEGIdgPcDKK6gOBqEn4j8Oi8mXmfJu6A1cjvgeEFLGfy7jP3/3s5BGrBm91ONfSwhDAwwVwlBjX+vxr/UIhBaGogYhtFD8ncxBK1A8l0gtLuBeWUC+UcOxrYQyv09Pt1dCqW6A2LIR/wQcRmbQMkUlLiqa3a94Jge4zv4YZD1yMn6u33ob/15oyG8Yc1BsUmi4zoTzxWHe7di9sh3pQInHBJgWaZnu6zvwV3+zOYt/kbuOXSu8bcGGasnoHiiwHei+uS6F78Na+g4jl16d1aXpWtymCUe5LBa4LRZUqojD8aDYtfB5/Hr/uUTZsqB2IK7Oo85nvZXTYdqGHVKi6Zn/gCXch6Nr7oW0Os0ZtxgIBWrdXKh1c4daygk1BFv/Pth698Y+9+2DrXdfrDY+nOjBqCMPigOFqgGGvhuHwwrB7Y1/bY19LqxOG1QlpcUJanZBWR+xrxQapWAHFCqlYIRVbrIxOWCAVnCwAR/6y8W14nBCQEABELxIUS+zp2I+TI4HzcYH2cnxcquLdXATXFt1A5kRnvsNMBHEz4/hCA4msTnsGM9Gh76JBoKPY8SZVhdMGweGM466M56qDWzoE9uheGqh+6sh+adAbVmJjRvMwxnbaGnW3Sik5YgnNP1M1Gz539hOX5b03zwDUR2GwW2fzTTWjmQj63Dnd/8slkGqn1eAGaZP90xGz8ovY/ZzX4Un/Az8nuGrottaWunHLJ5ItzBurPtnWuwc1W3+NgYVrEZm8LOM5260KnGn0Es5GtcuWUlAcbTwZurMOrkPPnYf6Jq8Y9YRjkj+Q/KA5FdWiGOaUTVTv/BM++R9F9xtcQbTzJlDGLmbS5EG08afTvKiWE6oc1cAyWnYCcsgU5Yg8dgCR6HJdQFJeKDEh2AJXgctt59UKIDUKL+UckhGk3WzQI+v6XQ05hQ3tJOQoibANwEnAC0tLfl62HdNXw7fBT8omp6S2YkHRUIM/z7xDFIknGVCgRRK7GxTKAlnpQqkYo+dvVrs8TNZG6TFnDml1w7C5Ia2uWMaAl42y1rfsZkz920dQtWc9/PM/mPbxUkr4o1rWW9/SuyJjLLJLrMN97I39+KaynEb7FHyupv4GBk25A1c4/o+HFbyI4c1XKXVwmWpg3qj55Rg0u3voNSKsLPSu/ktV885ElHuSxW2BVnlOTBpGJBaMbZcLU/h9bVtUk7eai6gbCq5y24B8wrnbD2H0Dj819HaPoZ6F92kyljliwhIO1eqHZvnehtESQNCi0DoIQgtHP86DGFo8auy2vCrsohdsR26emvo8a3F5eirvglbjouRV4sxzsLRcReUmr/BnS6rq6+pR7MvFzQiKDwNoTvh+Rvxnw0gpfwXgVwCwfPny/P+rNM5F2NEMX7jEyyeoZIVmrka0fj5qn2u6Ef94HMgqy/GEGxWaJaDp0Y/hL0ciSgQucO2HZGMU2xzJMzfLxvE4b6j12+CMaBkIq1FzuVKhYn0PXe72L6A5eg/uX/D12rvp/SYeMtzBv5vCxrrsWzj/weH8Az6Dr7m1l1KLAoIq+L1IQQqHZZ0RNInntkLNZ+Lqj2PwNazG60t85N28hgIq3kLiqWUCJgRFBs6Jj35eUhhQed5Py2bzj95JxRIm4sL9iZQnV+8u9BSSMmNVwKsAThRCzBZC2AGsBbDehHGJyosQ6Fv2STi6d8RXtacvrOq5DaYqSFgd/TzuOuobnlgFcHH0DhrDiZS3z0gkhBBq9DjR5HbAoAjUuG5rr3Zhc7YTLnrsAJNp0MvqX/BOqt/8Bjo5Xk94/nMfC98azZQ5nRwdrZxOfllGkufMd9H447Z2Pg5I9lNc9qpy3vXVVSDcJDM84FALgOPTfqhCFxt8DBn3QMDkViJEwC8tLcLdz671/zJxw2EtGGLRDNVve0eODteRfe5t0H3TjdhZkSlK+ugWEqpAfgMgMcAn7ADwJynltmzHJSoXidvt+uddAc09GcrLP8t4u13TFtZUuMgYpVRrTx+eAXQdfhHRyctw1RkLR903nFVZFwdQa55jZfY/Diqk1Lkyvc+VsAWXv6f8GrWoamp75MqBPfJVsZOCbuDBv5PNS88YvURU8BO3Cn72TVSnFwU418s1oUOFLI6GrVMxCtPQHa7ifHPWEA3l2k+PqBHvijGl7a24XP3NeGJTNqcjJ/w5DonC2VfwyrUIGpf/QlC087IqnMIUbkwpX+MlPJRKeU8KeUcKeVtZoxJVC6GbbdrcWDHzOsxuWsjTncdnymg8M1swVbKxMsWJRGQAjs4tCM04K6PxXXYLpte5kl5Od1gtmFztgN2a+ctx4onXoLb2Xtz/Rje6nzr0N9p6dqHnjlxOOMTLwBWKB8WB3hUEW32HUbv4ZAidcglDzuRnPGYjX9xagjdngYw8a7/lbt6k9n1pqt61V88+I5Y54wJH5/64Yd+PHju/GZ+9rw8+tbceac3Gx80R9SR5X+ZKJm629gDR1Hz8r/KKmaneaJc4bYuRDmW+IZ514vv4FM7lkK1uLHi6P0ZjTe4oIcyp+pG0oVWriMbIaSB0Iyz0x6/xmXD1BoXnLCl2ChFCYJLXCSXDwGTYiRfeLYWYP8WL4OwLETjhEtS/8j1U7Xowo/ETNbx8GyAlus+6JeuxcrlZnRzKJ2fmJnr/QjHNhN8JYYR9eCjHyhGFwkeJdL+3Hdac15ywg1g2J/lD2a2OUcB9q2u5AYNb5iEw9nzYSZEZU+BsVEeZC4qn/VsrkInPwhVL39V1h8o9akpsSUBTYVLJWTCtehF2BYnIhMPiWtsW0WBQ1VnjrTnZLcqaKjKrKXXyBOvkZt0dJ73E4SnrUDTPz4P7/b7MnoMSInazT9D1dt/RX/rp6BVNyc/ZgL5nbMM23uMPbhYy0fMXmnEmpLDA1T7xOoDEmuM/bGrHS3u7cjLvvmDUlFrimrY7YIn0o3fFf5gwK6LynwKCYKA9GLtJ5qTHWQL5my/9mNB5LKLIzVn/ikZyHX0J46mlpb2JQ4848++l12lCVYY1t4onX5UunnDiuFkHYPjl52D0It70XT0/+O6q2/TW9wPYqmp76I+o3fhf/EK9C7/LMZzTGRtwi6qLgTSijGe/6kn3YvIlFPgPvjcuOOMXKT4zTUn4TP3tZkeGGu6gQET1hRYAp2o2fob+E+8AtHGRSbMjKg8MCgmyrGxnVvV/5ak+HJz2PlRvvw9KpD/tMXVDIhRlCUWmRu5kN5IS7IKjewfCadYTWxUF3izbizV6HBltdzxRndwQAkFYXjl7yGwRmX4TG57+OmtdvT2lcJdyLqY9cD+/OP6J3+RfQecHPAUv6mfBEQuS3Ddt4PAlznmOj5C7asguP4VliCx8ccZ+QixSUzavCDq5dg66H0/7Yn0htUIU3IEtdu/hmEHkXP6f9mwqyIygeDnYqIcG29V/+O110BRA/BuuzejcX2lvmV5gWi6kbStnevwiwCQdj1xjTv79mKKIjCp2pHWOBO1UxvGn4sCxi34J/9zL0fDybajb9MMJmvwD1r53MO3By+Hs2IzO8/8HvSv+zZQFWR67JeV661xy2mLzSPb8nBWeuBgC42p8ec5yxFimeOMmLm845wbS5RjXDlD771oGDqN52D3wL10KrnW3CzIjKR+FP1YnK3MjVn+0AsMEbL+Qj2n4OaLb9B/9J/Tjv7FozokFUy7z1eS11YS97n2XXoRRh2LyJNi1Me16IIVJvUXsxhntaDeY0e3P5LS/SdqpzZqkwmLDZ0X/BzS6kTdqz+CJXAU4WkrobuboLmboLsnwXDWwdnxKiY/+nFAnCHS8fx3C01aY8rsBxVE6MchttyZ9/qKNJ0FzT4L7wNPwL7gmpXFV3UB3IIomb3ZZ9UF9QXO2Ea57n9UeQQkHvaV8wZTyicsKgmKiA+pfdhKkbboDnnccRmLsmrWMNKRGI6kVxGbqUpFJ24jr0AkLTzgCUn1J/bGpe5m1DUuGwIq3pKiyrHO/Eaa9c1AIBiwfHVP4Rh86DmzbtQPWLxnRQWABJqzSwcvfT3pmYUnbRYlp5uWpMvjsCR//oSCUMsquN/5P8DQUv5/4Qur8DgsWfehjmi6KesIbD27UbXrz+hf+gnoVdnun0UhUfvhuSlRAoeb3QKuaBu+OP6YdFAOxLhQMitOTrPOEdeAgbAMH0L/k4ymPGcsSm5/9bKxyIBTVnTek2MIpQ0H3uf6FnxZdgDXbCEjwe+wh1wRLsBKREf+vNMJzjBNYZKrb/ry6bBYoQSZ/j4MzV8O78nIxxHX0dk2ukpj9/li2J6XXblIr0Bc0ql6jb9N6TVjb5TPmPKeETlprhenYgqjWKBb8E1qH3tp7D4nDqe9zWowqkM3ZFHUZ5YC3ZDJ64kPpV9PXO20QcnBv8HgttC9Jl06H4t0VEN1VEOtm5uzx0hUiB3snJiKEgMtuSZqRv7fzBHxBWOA+8ORQUDy4BfZYmeZBmmGg2x/BpOr0upgM8kc0BKPZZ4ntnVtQtfdRn9Jz2RRiu+qzHIypHXGhHVGC+BddAQMK764G0j5VSImDCG2alSK0V24vQXI1Q6+enNKYiRE43oahxn2WBVyuOl2lXAHewm4k6hnGPWjKnYLOdD7HkCwPANPpLxR7SMyh8Gwio6B8JpHzeWutd+Ct1Ri/5lnnzBlPKJyVHyvTkQVRquZidD0s+Dd8SdAJl8ENpLfhL6llSLpph1SwnXohVgrthTrg71Oa04z9YoinUOspnoVp2SimBXaJ3HZr0nrw1pY6OBe+D7UDu/DnpzeN2iAlmW5/JK2tmfuCUXT5UltomYytdw/cn7zyOgcUfg7QnD+KJKhWDYqIi4Fu4FraBA3Ae2Zj2sWFVh5akJIBiki2ys/XugTXYmXLphBCx8oZcn8zqsGfUuLiYWRcBTRAvsElkUAact+fNbu/QSAEDXlkdHbZCSjG5IHE8xyO32R9ATMK9kpuaNX0FanHOhffKNpYxKVo9J+lSUqE4E5F0O3V8O7/f6MjucOd8mlVk/8AgAgND21TTu8TmteygGEEKj3ZLYFndLHwOJJnYwsplQ4Rm/yTcRQN+Gjj7jE3SEkmGNXQF4yOexIrpUSnL4z+kHk9yC2BTnh3PgD/gqthnuBtNG5eoHDEoJioC0upC4MT3w7P3UYjIQNrHMyhOLpV6YtfhF6F6m6HVzExpzHxkiQd5HFY4bcWZnaU1FsS2wGylZFrutvRe3/m0ngi2rsTC4Gd+4ZO7YG6Qk0ROIor0niIM9QXQOxALgsKrDMCSODURMnLyOawcQAACAASURBVIeqfvO3gKGib9lNpo5LVI4YFBMViYFF10HRw6h6+69pHxvVDEQ0bvs8kaT1nxIYO5+GXEEpxa2eHzZL3koZSzRbbrQoc1uIO6K0WBXZr7N9z3ab2UcHu0zs7cfbcBrhPuhiK6sdKn69tDG3xkQtUN+CMauv0RHOkLYX93wJQuE4lENIDqt+5BYM7F0GrN212PqFwxKCYqEtGmJYg0LIR3nx7qMjg9EGBRPJFk9sb1rGyyR/pTriQtRH+u0WeApsj6/qSjWBXYjeeIlFPOneIdlgdvae/H8nm6snXjAJoRlnQyo2uA88hdaWugnbsRWad8f9sET60L/sU4WeClFJYFBMVCyEgG/htXB2vgFb9460D2cXnivGlU08cnn5mSmNmu0tZpurc9qKuzR1JCFF0G3aMx+2InegMbvN864YduOvFd4Z1mpB2D0LTVsJ9n4KmCzHGsLHZbey/WbWoffkddRc0bv0Jo6gpEppySxxkSlS4GxURFxD/vA5CKDd4df0z7WM0wUtrCnuBIlLZ1ALCiO1s2D7pmc9L62hEvt+Wa3KkVfn5vI48huN7d8clgtQz2hW1vqcPnSqbhnY/uoThOhnmath790N68ChvM9xrCz2WP2SPXs3wOY/jP7Wm/M+R6JSxaCYqIgYrnoEZl8I764HAT39lky+iHmrn1stJ0kV2ehTOjk0IzUgtS1zoEoY6tx1KiWSL87kY0QxOe+xtsa29F+u3dOCGlS2jOk0EZ64GALjan858tniiLPURK1LbdgWjdiQjOOj/vcyQqVQyKiYqMb+FaWMI98Ox/Iu1jA5HYKnYaLlmm2HH8TShanCOFpZ6Q0Xio7oOXS4PbPxc5psxT9AruR3HbrUPb1lssW4sazZg8FoYOBsVo7B2p1S8FKKCbKYgOAn69DzcHRtQ/+ymwHBt3miVPGvhajIhJrfA80zJaMFd1JK+Lnt8zC6IRHVJq4ndh55BQAQnrYi6XhWnRSmK1mg1LlvRlyXkcvvrXHHZLNh11Dcs+zqYnR3qNCEEgjNXw3XoBQjNnG2Y0zFRFhsAatrugOaenDN/8K/M+N6JSxqCYqNgoFvgWXANX+zOw+DvSPpwL7oZLpZ7YeWQjorVzoLubkt53cDFWoSmKQK2rneFu0WRWlaHewm4hFEfjoWbNHZV9HdpoItqyGooWGTqjyZaws9lcfegsPvHYQAGA//hbcB5/DjpbrnsG7zsbzOjajUMSgmKkK+BddASANVux9K+9iwqifNjFaSpPXEhg5nx6sIT1uZ0nieAnWdGEu1q3i3nf652FfcOdhNxpXAlIDz9TBgWJ9wHnszDjN41Vhb7xrNm4a4X96OtvRc1b/wSqsWNT+1cNmrxHRFNnrDhfTYkqnFY7G+Epp6Jq918yOp473L0rWabY3r0DluhASqUTihBw2ornZVMIgVp38ZUoCCFKpjfxnWFIJiqXNhfD0M2NBscxfHf/a01tGZbGvXt6M2648Gb9+5Fm4d/8V9+nn4Qtrlo+6HxFNrHhe3YlonGP+8q+Do3gF71/b0j2UJBYD06olDKWSK3Q5L0WU/vU5bwdrDjaeU2rCNxWlTUuruETjhItj698N+nfGvS+6bcXzhDrS11+EbjU9AlcGzRxxkQE2WguF5JiWiIf+4aSMWaUQmFZhimbxlbilKqJ+7YCNU7nA7p3etL7FlPpRKJst382O2Arhc4YExFCpLSYMjB3DQyLA96dDyS9b6r9hTO17e29WNr1CN5suAh/n2KGN+vckouQYFBMVgbGCks3HFezxroiVUMj0a4SZLU4hKJYSriOvpFQ6IYRI6bJ6Ibjt1qw6YpgZnsJViG7axuFJYJGg4ahCcfRGq3n44aV/xlPoLZ6itvReHH/8fOKFi0kVfGtVCjohSw6CYqAiMF5T0nzHk/rIGjcB5+Oe0xA1H2LE62yM7WtxeWUHdKi+xcNguUIi4JyCZbbGbAVopt2MaS6gmQb8HVsIR7nU1pwl6y/cKb2HT6Gj9seR3D2RVDrTxzdQo6IUsKgmKgIjBeUTFp+FQybJ6MSikrvWZxaPfFGAKn1nJy6WVmzjcdosWe20Z0bAZlUUVBV4tz+z2K1K0s4e6za142W5GJp70lAJxURlJ8n6C2fqRvfzsKsDn6Dvl00M/G9lCjoiSY1BMVCTGCkqkzYXAnEtQtXdDRpsE+Cq4hCK1/sSvQHM1Qa05Iel9i7WeOFGdn257xQkAzArZqV/E/R+lIVpIyf4oX33r0beybegncB57Etrf3jlt2kmyXvIzpKmrf+BVC01YiMuXUn7MYiqnAMiomKxHhBiX/eVVCiPrj3/yPtMSMV3LM4aX9iKeE68jLedi5G28G+YTeNzPY5baXRTcFunVTJa5GZGwFbqbdjGkmw778ErPF/ZuxjC0LD98bvGLTtJuktehqrefhhW/xH0nfIvWY1DRAyKiYrCnREFJaPpZ0NyTMyqhACq3Z3EoOnFQbPUdgtXfgcj0lUkXmZVClnhQnduW9oYeZgRs1U5rSZw4pGO8nuuLEhbGtLXVYtGwF3jRm4Wrrc+OWnYzVXzjrEgdpoPb12xFpWIhQy6rMxyEiAAyKiYrChEGJYoH/nxPfDfeBJKOH0L7X6wxpkHjcX+P/bu/PwuOrzXuDf9+yza7e8IBuIMWaJERjiQhLCkjQJxiwNhZDQnJu3TXm7aJmn7NG2SXt9bsrZp0vQJ6ZKbW56bQHGaAAnQcBMIhDQYYwwGYmPEZiMvMpZtSR4to9l+n94+RFEmWZjlzZubMzPfzPH6wxZwzP3ukOe/8zrv4QTKdRSpTXD5xx9mXFSwy83s+8Wwigs6IXdIxn5QZshqahNejfkdNuadrCrdlmF8bu7B/Cvc8exI/UpTgt/Rpe21W9sc/BfY/AGnoZI70fA3zWP5uonHjEoJvKBQkHJ6JrfgmRTCL36YMnnTmezTbdbXGiXGMjlE2fsGJLtZ+YtMjP1wgVXfuOYelV7BbeFnLV935ijHQrvF0x9aN/9oNz5z3y4AwLr3/T6yYuDQ43dUrRVay85/QiqyAqOrN1Xl+YgaXX290xP5nXMg2sKwlgJag5emUsWTH2Ui2rnadQjEykfJsLfVgPFX4Q4BzaBsSSy8CRMtbZFZMv1o/agtZVQnmng5bRMB0nFrLY69/b04o13RFMprO4/vzlOHv16ZhYdQVutLfi5YHKB8X2oe1wBp7GyHm3Alrj/vsTnVRODYiKPhG0DXREbjqmjLWRhRWsQPW1BdERsBC2jvPHAIhg943oEBp6CcWJ/yYcn09midk8bQTarnkEjlT53Qx47AGtmLxLINBYvM/DqwoxARQUe4tDQKN8/RHm68tInZFiuy3Nk/hNcGx+Z8kIqfeQOcnyWP4yJK9FV9X6zPfQMZpQ3ztTRV/LqJmwaCYyANh20BX1Dkp8DV0DVHHRHfMKTt4GD3jutxzudwtnHp7IP3GrUUykMgVzqJ2BXN5nYtnbChaZlTMprtYCll7RjhAtgdKL+urR/A9Gi32Q2qpdgIzTikhfn4bHPZa3njccQ7H8Uw70fgzIDFX0uombS+O9mRBUWdnIBcSFRx4RdRoCVjp6CiaUXTY19Lr1wbiKZnwWS68XeLx2ftiC80Pntn/xD273wEWSOIyY5z8uZzW4ZW9x0V2kMWDM37t3pT19ASbKwWbItx5qVQnLPZB6qUjCYyuvhbBvT+Flhhe6FTlyyTR/sv/iWTLaRhZ9/uVeQ6iJsWgmKgMEcdEV6RwQDytvYxRnvECuZ7E19Aqso7tdHT8y3vi5xbPTRBYbn/2WxAtILF0P6PmDunpNnZhN0wQdEe9THDrCdnkpQXUknOO/7IN8HqfiZN0DLTCL06gMVWUvshX+DNfwajr39bwC9sVNXiKqNQTGRS9GAWXLrK8cs73b22Fs2nQmkmwn0/cHd8MlOwVVk9S6QySGd//fdbaHz2596zDLETrxQ12rlei+zmC1qGpy3TwrbRMP82xTBKn6ECS7Hwrkq1nVCSFQh87gtan/wFjK6/AxMrLPT8/UbNjUEzkgmPqrouY2kKW61vyWacV4yuvQPjlnHwLZ0tusKaVwooE7USxUTDi/3dpF+ssQKCSWbch7LhGBYzRO4NcastDuQeGdJoK2Mu941KNC0+1mniCB+5m/DOfzMTC9sr7Rt+zIkM4ljb/9fnp6XiHIYFBO5UE5QoGuC1jKOj6/5LRgTgwjs/y93xyfSnyGQbc5jH+AKjnee3WzvR9ziUZmGy67y857IMreF678Zc3N2YTRNBV9SG0QTFdfMFS5hqeOLc30Uqn2oOOx/4Ckk548vz2mzsReel7GDnvD5FuOc2TcxLRXM33zkZUprBtlN2RoJyiu/FVVyBjtyDcd4+rn47NKIZ5ovN3iTFZhcl5QvFCXgMnXfomjLedCGflzwRshn3ghEcfEkgU6pRRimzqWtwZKCg4biWNqn0Ir8N1NmEEff9bewhl9Hy9P/UP6Tqyzaf/E/kA4uwdAFHy//fES0IAbFRCUQKW+XdzbXRXe6jbG3nbERo70OQ5KirU4xMpBpu9PN48uR0kvldAs7vtnC27MXL9rkFz9eoQTGQGzLTHXWKDvJiARPLYk5TntF9bjIiUlEc9cco7ET/zRrTs/GdYg7vKeu7wS9+Hc2Qnjl/8GSgrXNa5iGhxzfsOR+RCxDE8CwzKnKbqLr/kAtHQCodd/7Or4TFYh3mCjnxfKJ57fJcAZ2A5NZXDa+vfkPZeIwDEb++0xYOnojjl527Xpnmkz12G6eThP5lFpceOySzcg4beh89M9d1QAAgEyeQNuTX0Ki+wKMnnG9q3MQUXEa+12fyEOaiKcVn/ID7orvJ7vVIRVci3Hev6+dupPZsSqk5/YkXEzi4FUozkVh6Yd7HOabWFEGgY+roaQ9iZXsIy1oCn6IjYaAlaCNkGgpaB5S3Nmy6xkPmt2QrJOi04dukXYB/dhdhz/+rqOVt3fB36xFEcfcfnAOElm6iSn+BNGVKSWoOn5IAddE8QCLnaLp8c+H/gl9NEBV8+dymQbJrc4kcoiW0Q6iHNwKyaXnAdlBvM/roG6nThRD1wSOqSPqmGgLWVgSdXK7yE2cLrEQQ9dgGaX9m4ydfhXGTnsfWrd/Debwa8UfqBRiz9yOluf+nFfGzbkaya12JqyWiUvEdj6gIhqa5C16LELYNV7uS8TXXQ6By7dlcOj6WbIhOFAvlE88nkydgD76AnieUXF3xsM/XgpdK42Tk/+s4vQOkWOh77FKCK6BOeTaPj53+J9m1fwujqa3H0nZ9zsVIiKhWDYqIintIbMit1ON3St+B6os6RbTkNiyfkIv+yuCwWQyy0+Njbp+ni/KCp1YuApiMoWDIo1Edgl7gZS85j/ns7rYKPEt2/tn/pwJLcHxSzYjcGgbIrvvynt+SY6i+z8/guiLd2Hogj/BkXd/A9DL7y9NRIXxnZ+onAMvQyppCV4yw7S5vc3TNb8E+tsf12GcAGE2kFyxSqxepTLaoKX3OwSdz/Ym7L8j/OFNvinxicsc2nft2abcv2fuiazBkl/v0d+/HZ+3ZhTXdkznHxtTdhYvklaN/6ebQ8841cR4p5u8b66ACW3Xc9Avt/ngcF3/R2GNvwV84iJqog/bUQFVGN6V9DS83YBWMzo6k1TY5/d7xYDwNHRybpt0VbMLjGQK7JLdF8AnZQTyPq7Ru05QeURkZrd4TXcE/759P26+6BTc9uAefOnHe/Avj7+Oj16yak7Xk6kDMXj53yPZejrantn0ZK/7jN9FzRy86H/4ThPvugXPwSSy/52qYI/tw+Kr/i/jZH6rB346oubGsmCgPx9SrUn0vIgg7nBobHkyUdl3XaML7ycoRf/iGO/8ZnAc1dLmwqk8XweMqzHszVVEw+sZYYhjW4C0MX/lnBx5Y7mIUanX8DSMTqZRm9PKzZvXIvbHtyDnrYAHt5zBO9e24Ub1p+y4HHpaA8O3fBj6GNvIrD/cQT7H0ew/zFEnXs51kUmHunHo+vuQ7Di7mn8dIprCoJgoj5ZgZdMmZgvbpQfFQC6FIrT3Jwgc+CUmei51/fzDEymEnbKPk6vpaSqazRaV+OIeegkAhsaJwPjGDYipk9mCX3p5WXLSqFQ/vOYJzl0exfd8QdvYPnbxTPEsmntASjZ/42Rs/8bUBlYQ3+CvaR5zG+6t3IhJdW469ARAuon6sfUZWZulbVHq2WobkKyMZWXYmMHUO4n7wdlPb9SCkdH66vortiWcoGDTyCrO0gs6c3/OHadoCLMbs32/R378cjUDnH/8YmZVIr5xXeLEg3JnrnWIn/M7CwbExRTyEZE3GBQTLSJWxV3iaRHHRRCu2xg7fSNCrz8ESY6V9fyJVAYn6qR3sVIKo0VOn5XMObkVi6fqCVfzN1p+Y3AtaBnb2D+GOJ/bh1ktPw6ffvxabN66dyTHuOxz35HnWdEfmBNk7+4dwn24N7TirkI6LylRUUi8gNIrJbRLIist6rRRHVmq4JIi47QpQjZBkzle2lyI19nkBo70Nlr2GoTnoXnj06mi1qnNnEc9rE9SBTRn9ixuE9AxQlaOvoOx/GF686ZySGezjHOZBVuuqjHk+eZnbd8xxN7cduDne7B549q86RlE5E65V4BdAK4H8AsP1kLkG1Gncn2J89E0QchFMD659EKkoisR2fO9steQySociSd8n343iRKK4XeLAoScBoGB/Yl0T2NwppiI5po4PbVh5UnDa29PqWUA8+5yb1i3Fd7f1Y9O6pQyIiSqknrKBYKbVHKdXn1WKI/EBEEK3Q9LpiuEqhEEF87U0IHNwKY3hv2WuYSGYw6OP84sl0BpOp4lqxOQe3nImsEMdl1Xt7HBVhgRyWq1vfMzv4h3P/8AG7Z0IP7nx8oPl+ZiErCe4VE84RtA7pWu+ENjqnD1Ev/n0YyvvRFKdET33O3JOkYTaQyNld4NoxpOTBS3SwwAgQNbkVh2EaDn/6BjMyimElWjMHM6h3jzxrX4n6CWnzqRSMDAm8l7BK6+IPCIiuxb4dU0pTyQifygiO0Rkx+DgoPsVE1VYrIa7xNOiLiboZUJLML7qnSoT3fA/IeBPMDo0ni+7wUC3ZbPEFdvr4IKyhlwumTgAc2kGlq0Z3mr7D8Tk5xNM5xl4U8okINBFOncCSaUvAnWil1pRdPpJT6FoBvAcD69ev9naxITSto+aNPb9gxcHw8WXJe74mzbsbSvT9BaN/DGDv9nKk/WcnQ0CUPTfNOuLD6ZLvrfxTm4FQAKFtkxn5jc0DWBbepFp/K4sVB+cm9Pa8l5xbapoyNsQZ8KngjXBTDCcySoMxieLGoRD1Mhqf/Un8hE/7BIDuYtt0EUQOtFzGdLhpYi8+O+erUUphTdPJJBMZz07nZzlOTBS/cx04uBVZM4zJznPzPo4BMbkVLqNLzZbt/fjqT/vmpELs7B/C137a52kf4qBlYGnUgW3onMHQNujZ3d1jXBN0xB+0hm7vG1NTKbcl2nYgcAPAbAP5TRH7izbKIqs8y/LMbCrgsuNN0xNd+EIH+nx2Gc2O/ZWrJTgXGtW7UlUhmkMsUH587BrZhY9jZAy/9vySI7citsG64DyTXdEfy8bxCbf7QbO/tznk/A2/2g3Husb9KwPcdgxsCRqQyuiTiIWNLGsxXFV00DUCMrtPnGfUmqFUspWSi1RSv2mVwsjqja/n7BJPC5i6q4K/E2tvAgBE9mzxdD2pTBYDIxNIlxCUeq2UXWJ97DCs4deL6k9sM5+YXHJ7VwfIpUHcnds3ZAIDP3LcLn7lvFwDgtmvO9qTtWixgoivilBS024aO5S0BhN18KCeqc7wSEAEwNK2s26CVICKunCnkykeWY6LksFxRnvc0RTKazGBhJlLRb65VMVmEsWXzuZuBALp94YsUleR+nibgar000zU1v8Wm9nPa24/vzlmExnMZnO4vrzl3sSELeHbLSH809wXIymCboijuvjieoVg2IiANGA+1ugleQ2UD9x9s0wnxg4j+MajHq9oasd4OIHJdOWKixYST6RKKjx0Dm5Fxo4h2X5W3sdxl5jKFbJ0V5MogVwO8b3PHoRtnaLANDfc+e7DsdmudEduTMfWxgFnTnu1E1carATU9TcRVC7RqCFg6DK30H9PxlVciHej0tOButnQ2ni8MjCSQqWHU/WyarSupNDOSK7BLLNgBa/l1g5hNTuUTcTaKcziEGgC9edw6+eN05ADCTY+xGNGAin4uH7WXvI4p0UahoMiqnphR2jqCKUWgnZLi5Iuon42hsRfONn0EcHvF8UcoHqwEiiKm2cjsQTSGeLnT9nQ4wdhnnijyP7EvOBT+dwUxvYdjuNdazpncoinc4wvW9Ppqg+xqWtoC1olH5ePiGBJ1HH14Zyon3vgriZKoBvxWYDdfyDYwUkKB2Zbt/VjTHcGFZ30Qrc/ejshL38NjXb+LvsPxBXueliPXrm0SnZHynWlPlMzSWxEQJucQAENz/OABgYsXb8z5ORGD7oC811b/pSZSl5Nt71YN4Wke4uC4TpdI1QVfUxsBInouTe6UT1hFcDamoh2/B9+6FSxz6v6Y7gtgf34OmRGMZXvB32C/+Ozz2w27MWT/MppXDkRALHRic9nv2COJ9MYGi99Ol9w78NIRVYg1bYm7+NsQ/NlLjnVp3IK7soVccyKtpR0TB3tYW93oYn8xt/RAFGFn+X2XeFopu7DTY2Bve3APHtDfjdDEQXzjbcOeVLTnMzKRwqGRhGct21KZLAbjkyUfJ+kJBA78F8ZXnXQkUCHiZT0xeqlUHG0PT0B6qfMAadbzNVybyGwbF1LRsU6+bfNJSd6B6e1qxad1S/I++VRjTYzh/n8IcVWtlck6kMDg5PlJ1nrJTCkfikq2EhzoGt0NITGF/17sKPrZPXn+qDZWiwa/A91RGxqlYX0RFmn4R01LgbF1LTqZZcYyF1srRJyX3f2D+H+5wdw44bTcXfmcgRf/wmMkX2VW+AsmazC4ZEEjo8lXadTnHB1NYnKRzhZbtvefVJm/s39oZixu8I1HkDWCuc4TeYgIHLZjI49Ve7c47Biu+pm7JSLoitiuW9ARn+RmvCNSUTN1/wzoKidjFBfE7+4dw24N7sHnjWnz0klOx9N0fRxqC5C+/WeEVzjU8nnTVnSKeSCGenWLywcDpnejownv77rumOAEohtO8RTPRcCmU4eZ/HYj4xVUA5Y59LlUubqP6ADUPXmF9MDYlBMTUlnv/YlzqfY1mx9h+PYvHHtTA7x2jPOwMFTrkZP/73QEscrucSTJFIZHB5JYP/xcYyMpxZNh8hmFcYmn0xiMT+LoaP7Cutk503c8sXfmA0BvTyusYy/CGD2E8ZVXFlwb84mpEnRNqva91R62XI2C90LEMau6nQ01UDQyKqeloIq56itaaoWtF5fLddFHPSUV1+iUfh5VNILrrO5VaXl6pTBbHxibRf3wcR+K5oR+JnVAZDY0kcHJ7AvmNjePNEouipddM509/d1o9N65bO/H2D+x4GAIyvuqLgOZg6QZUSrsL7S9g2atrtnAsjlFzONghoJrwrUdCI+H9aRj9uLbap9DcZXXo7oC3dA0gmPV1U8pRRGE2kcGp7AoeEJDI0vnjucnz3TO9C0benD/8wMzqRTBfY8g0dWLTLCz4DkcgzvFVBnljH0uhq4J2sPVT5uYj2kU1GgYFFPTqacCnu/lClvt8xeHzboUxcRThvns8XlVlLFZQ99Wf9s3JmZ5OpXjxlVdgv/kcxk8tnDphGVrdfjAi/5MKn341qD9s1S5uYj2kU1EgYFFNTCdsGDJ8P68innHzFxPKLMdn5VsSe+1dAedNLuJIWK6gTYE7O9HSOnMV5+GAJVVCs25hNTpbUGrYqMRg7Zhu+KhJlGQY2ifqMDIhdiwfrdJZ5WagrFzI6rCIZ7b4U1/BoOnP/3DmRZmfrVYQd2fvWfNSTnTvT2tuEx7FunwMiTbzyp4bvZZpUrTNEGbx6kFuibo8EHaxHxMo6BGnwaCYmkbINmA3QB5pqfmKs3dcx06/CmOBZbCf/qdFxz4X6gNcTYsV1J0kM4lA/+MYK2KKHcCgmKojnbBuejl5uC9Wu20QhTKOgRsCgmJpGSwPsEgO5fMVSqs7n7Lg+uR+3T7wHF2APNlh7F3x83j7AVbZYnQd18gYNPQkuPF9WKzdQ13wYW1HjaQ7YnfYuDluH7EctMo6B6x6CYmkKj7BJPK7WIZ/aOa/Lcm5GxnY4g99y+LPnaxPsDVNH8IyfSaFgqMg/seQdYIILHikoLn9XLnjqgQy9DQUmZxryaCjjpIT2AaBdU7nBsXUFBpll3iaY+owSygYnL3j+oNdI3jllBsQev2hRUc/F522UEHzh5BMB+t9h+NzH6gUgvsexsSKndxScYgewyI6qryVolvTzOl972KqbAmGmUVA9q4+fMqIyNNou8bRiK9AX2nH9o1cvQlZ0xJ7/9qLHnFJO2UEkLDSHp7WnFTRf1zPmaefwlmPEDGD+1cNcJgPnEVH0i7gvkIo7p+7SJ+ZhGQfWKQTE1vEbbnJZ5WbBeKhXZcP3b1xfhV23sRefFu6PGDcx5fStqCH4T2PQIAGF9ZeIqdZTCfmGojYOklt1JrD9nonjPiv20QhTKOgesWgmBpauEF3iYFcwZjbsc+9Pa1oe99fQ1QWbdu/Ouf/FZ224BPBfQ/jUHAtdhybnexFeqGMGUyeoltpCxe2gaiLojjl13UKSaRRUjxgUU0Or54tKMcqZmpWOrsDIW38P4Zf+A+axPTNfnLzZtwQ+0iWOwDz+L4RWXFdUxg0V2VEuGrqEraiOYZzKlqWtY1hJoiICSaRRUbxgUU8Nq5F3iaSHLnKOuiM3zBHyNrR9H+5Bc9XFX1hPbmpthF111dsGOGiMBp8O8H8r+gZaA75qCnLYiOyNwAOWDpWN4SngGU0xqWZaRRUbxrjJ49oAY2+SwzkpmYFbfeBXtZpxfAFH0fwjUfhHHjCw5VVR2TP3Ui2nI5k57kFnO2bYhgaN+cTkE7omiDrmTIC8JOpgaSzQcN+jTKOgesKgmBpSM+wST4vY5QX/J879CFLh5Wh7zf1unswAAElBJREFU8guAynq0qsozj+2Bc3gH4md/CBAp2DGDXSfIr3SttIE89aYj7N9JfESzMSimhiMinaAk2zy27gKXD0Nz/KCvDwdDb/gLOkecRevUBD1dWWdHddyKr24ifeUNRHTNYZEdUG4au1WUXDWo+nDIqp4bQEzIbJyStWse3ZFjN6xvWYbD8Lbdu+DGQmPVpV5UhqHJG+ezB2+lXIOm0FO2aICByzub4nniPwkaBmIlTnZj6jSeJWghmLqWsP2Jc6nmP6nW7b3n5RSMNO2TNNx/OLPwjzRj+iu71ZqmZ4Jv3I/ntGQc8bM/DKBwxwzH1Bat9iei6mgLWbB5x4Z8jEExNZSOsN2UwY9laAUvNmu6I3nblk2ccinGV7wDnrTu+Dpk8UfE1lyOy+04kW1cjsfSioh7P1Ami2hMRdEVs5heTbzEopoYRdoym7kNbqGfxdErBom3LnRHD84s9CTwyh5dlvVmHF7liDu+Ac2YkTZ38YKPIDEIvsiPzB1DXXI6+JKo1BMTUEXRO0h5r7jTZcnRM/iQm3Lkp3nIr76OsSe/zaM4dcruVzXcgV2DkbXfKCox2siDIqJfCRkM7+Y/IlBMTWE1hBb/mhFntHUq1LYMAI5f8tdQho2un/0pkM1UarmuSHIM4Zfvw9hbNiLrtBR1DANiIv9hfjH5EYNiqnuOqSPqncNcBAKKBxYPiYtqWAUAm1I1j7/g8nMM7EHvuW5VecknCr/wQWmo0lzpRJOYTE/kP84vJjxgUU10TnEeanzWIb+qI7o4Xals02esZ1GDvtfWh76u9gHuur6JpLEd19J5JtZ2Kye33RxzgW3+aI/MjUNSyJnOk1ZHE3+xKsF1bVYE/YkLiS6SK5eobZlc4hg8NIvI2uF0fWzTwKZVCWWWhLryAuwB18oqcBO16RpnJhsS1SPH1LEk2pxdg8h/GE1Q3XJMHa1N2JO4kJCle3JLMhvswNF3fRn24AtoefZ2D1ZWnujuO5E1nHIyuub7oY5g6QeR/QcvgxDvyBQbFVJd4221xIoKIRznWY6dfhdHV16J1x9dhDe7y5JxuSDI+VWB3nDbJ2rOjjnCZu0UdUT8K2gXamwlGNMSimuqOJoCvKAo18omWOfZ7t6Ds/j4zThs6ffbJmI6AjffdAnS4+XVGAHAA5TJ4jqRixgoiVo1XoZ1MQYFFPd6YrazBMtwNC1gu3ZipV1WjF42VdgH9uD1qf/wZNznlkIfH0TrU19BYumFmFzSW/RxhqYx35yozrSFLM/udBGVilcMqivtIRtBy7td0EbmZZu6iVVXIn7mnjWh59psI9P/cs/MWo/0Xfw0tNY7Bd32l6AI7gF0niOpVZ8RG2MO7XUTF4lWD6kbYMRBjYV3RApYOnU/fuR/zoO/4GybYzseShP4D95nOenTef4Gs/Rvi1BzF04Z8i1ba6tGP54YmobnVFHLSFmEpB1cWgnmOqCY+roZBFGybzcLVZWBIevvhOZQDu6H7yl4mOgtcQQOh7/DCY7zsZw738v6VgRQZCdJ4jqWkvQnQnfMKTi+nsgrDIrJ92xTZ6cJl8KO4em/Wya0BIc33QUAWPrAh6CPHfHs3PO1P/E30CeHMHj51wC9ntODeMTVoLMQkqntBy8CyloCnd72IFsPvMvK1sG1gWcxhpwmXdE0Q9qjgblqq5XQc3vgd6OOD6H7wnw5DkyRPxyhV441FEXvo+hns/hmTnOSUfz9QJosZhGRqWtQQWndZJ5BUGxeRbbSELXdwhLls04H2AnOLmkF2++93/DOt6H7h//vqet2iQZR+fP/xLJ1tUYuvCTrs4RZH9iooaia4KlMQexRSZ2EnmBQTH5njiaCJVGH/So9Yht6RXZYJlZehsHLv4rAwSfQ9cgngGzGk/O2PflF6KMDGLz87wG99Dxyy9B4q5WonAYkI2sM2lrdy15gqg/cYyVdMXWMf4gqIBUwkUt4ErbONrvkA9LFBtD/5eejjRzF4xdeQjva4Pp9zncCtiu76D4XV/gMnu9a7OwdQJosZmGzqWtQQQT6QwNJZCOput9ZKoQXA7hXwjbOcKKhgQey9kGxXbnPR3pvRVHLv8a7MFfYcXdVyCy6zuAUqWdJJNC7Jnb0f3ALUhFV2HobZ9yvR6mThA1h4hjYkVrALGAnyTQ78gSDYqq5oGVgeWsAXVEW1FVStFK5eCIYXXsj9n/wUSS6L0Dn459G9wM3Q48fLOpw+83nsPz7n70f7ti9hoPPtOHT9PVBmEACws38IW7b3F70UQ9N4W5WoiWjaVEpFSwDRgMn2bVQWBsVUM46ZuwXWnHXO4O1wFUceo6IeOTGQ5Dm+6G4OXfhHOwA6csuUKRF7cAqiFb21Kcgzt/7UZy+65GnriOJ5c/494n/5v/DTuO5fKId/YP4bYH92BNd6ToNQS4S0zUlCxDQ0fYxsr2IDoiNmx+OCYXRJV6m9MD69evVzt2n7Kj68w7GJxFPpKr+vDSXY+poCZrM/ayBobEkhsaTFX8eY+QNdD76Zwgc2galmUgHu5AJL0U61I10neCmyTisiu++CMXoIJ875HRzf8FdQdnQmEN60binuf34AmzeuRW9Pa9HP2x1z+H1FRACARCqDeCKNn8WQamWz1Yx2aq6ctCKNGRdAi8oxSqmChCq8eVHEigoCpI2jrCFmV3a2k/KIBE8MTKVT6w3A6thIDn134foVfuh3VsD4yxARijA7COvYjgGz+Dlp5AsvUMHLr+PkwuvXDmuN6eVmxatxTf3daPWzb0lBQQna1PfZ0REQG4DJpdOZWMynUEimcVEKoOJVKbi74FUnxgUk+dEBIYmsE0NQctA0NQ5Xcwnpod5VOWOniWgYO+NajOHauV9XCpKMQ1lhQObuGuzsH8L9zw/glg09uP/5AZx3SkvRgXHA0llsQ0QLsg0dtqEjnBhNKKSRSWUymM0hmskims0hlFANlKi8oFpGvALgaQBLAawA+qpQa9mJh5H+6JtA1gaFpMHWBZWi5nX7rG4MTHYgGztmlEIlB29KQvT6dOTKdMnHdKy5w/T9uyvR9ruiNzvrazfwgHhibwiSvPqMpfgYjqnl4ggYOkn1SDkguMs0hmFdDaLTFYhnVUz/2XQ3PjK3Sl+GMCnlVJpEflbAJ8G8JflL6sydE2atqn/ndIw6O1iVqa8LZOq/v36MJif/V5PpIDj3Xwa+9ckycjv448l0rZcyR9/h+JwAuLenFZs3rkXf4ficnAHhNd2ROsDwdTH/z5vNrtXQiagDTGzuLyWQVsmrqVxZzfq+goFTuawq5rpTTQfT0n3O/V7N+/2vznA27G37XhWaGdiFwH4ANKqQ8VemytCu2IKCeRyuDQ8EStl+Ha/IK8z197Dq5et6zWyyIiIh8qttDOny23T3wPwUJ4F/aGI7BCRHYODgx4+LRGVyjH1um5ZNLsgb9O6pXjH6o5aL4mIiOpcwaBYRB4RkV0Ln/Lpm1mM+CyAN4K7FzqOU+pZSar1San1nZ6c3qyci12KVGuZRBfML8l44MFLrJRERUZ0rmFOslLoyn3/8XkY8A2AjgCsUsdKK6EbJ0mLqGVGbh4Rp+Nb8gb/3KNnzye8/h9pt7cfHp3DEmIiJ3ykqfEJH3nAvgUgE1KqXFvlkRE1SAilRv9XEHzC/LevroDt9/cy91iIiIqS1mFdiLyKgAbwLGpL21TSt1a6DgWn2hH5g1IK+49PIJ2tr93i2Za1BKYa9BMREZ2sKhPtlFJvKed4IqotEUEsYOLY2GStl+KKZWgMiImInyBPN2bSXiGZEnPodvR1x6i/9g4iI/IlBMVGT0zTxTSeKLdv7sbN/aM7XdvYPYcv2/pMeKyKI2JxUnT0RE3mBQTESIOiY0H0wonJ5WNx0YT3eaWNMdOemxIVuHVqc73ERE5D/cZiEiaFquE8XweLKm65gen7Tx7Wt3sThOzRZk6QUREHuJOMREByA3z8MNu8fxpdQsFxKbOAjsiIvIWg2IiAgDomiDi1P7m0fxpndfNzjAHuEhMRkfcYFBPRjFjAhNRwt3j2tLqPXnLqTCrF7MBYRBD2QfBORESNhUExEc0wdA3hGnZ0nmD+tbjrHuO9wfOYxIUuv2xZyRETkX9xuIaI5WoImRifTKGfapVs3XdRz0td6e1rn5BWzNzEREVUCnd4qJaA6zxrvF+Zi6hoDFAjsiIvIeg2IiOklr0B+dKObzQyEgERE1JgbFRHQSQ9d8M+Vumoj4dgebniIjqH4NiIlpQLGDC0PzzFhG0dBi6f9ZDRESNhVcYIlqQpglaQv7ZLWbqBBERVRKDYiJaVNQxYRm1nf5twTB1Bi0ExERFVTu2vdkTka+0hu9ZLQFvIqvUSiIiowTEoJqK8AlZtd2nDjgHHZBs2IiKqLAbFnRFRQrXZqRQRtQe4SExFR5TEoJqKCLEOrySS5loDJjhNERFQVvNoQUVGqPdDD0PzXK5mIiBoXg2IinKoqha2itYipDa8iEpvlvqh4RETUmBsVEVLRY0ESoClPlapWuQUREzYtBMRGVpDNsw6xwnm9HuPZtn4IiIqLkwKCaikmiaoCtqQyqUXxyy2YKNiIiqj0ExEZXMNnR0hL3PL9ZEOKiDiIhqgkExEbkScUxPn8341EXTHnIqnZhARES2EVx8icq0jbMH2INVhOiBm2gQREdUKg2Iick1EsCRiQy+jdRoDYiIi8gMGnxURUFkPXsCTqLu2BATEREfkFg2IiKptj6ljRGkB7yC566h0DYiIi8pPKd+EnoqYgIlPDPXQcH09inNJFe9LGmrqEzYjMgJiIi32BQTESeMnQNXREHUSeDo6OTAHIT6mxDh21osHSN45uJiMh3GBQTUUXknUiqCtV4GERFRUZhTTERERERNj0ExERERETU9BsVERERE1PQYFBMRERFR02NQTERERERNj0ExERERnETU9BsVERERE1PQYFBMRERFR02NQTERERERNj0ExERERETU9BsVERERE1PQYFBMRERFR02NQTEREnRERNj0ExERERETU9UUpV/0lFBgG8UfUnbiwdAI7WehF0Er4u/sPXxH/4mvgTXxf/4WvijZVKqc5CnD6pJUEzlE5EdSqn1tV4HzcXXxX/4mvgPXxN/4uviP3xNqovpE0RERETU9BgUExEREVHTY1Bcv75Vn6wXQgvi6+A9fE//ha+JPfF38h69JFTGnmIiIiIiaHneKiYiIiKjpMSiuYyLyFRF5SUReEJH7RKSln1mtqViLyXhHpE5FXReSvar0eAkTkFBF5TEReFJHdIvKJWq+JckREF5GdIvJgrddCgIi0iMgPpq4nne0TkN2q9JgJE5E+n3rt2icjdIuLUek2NjkFxfXsYwDlKqbcCeBnAp2u8nqYkIjqAbwJ4H4CzAHxQnRM6q7aoIQBrAnyulzgKwAcAf8XXxjU8A2FPrRdCMfwTw/5RSZwJYB742NSciywF8HMB6pdQ5AHQAnN9V2VY2PQXEdU0r9VCmVnvrjNgArarmeJnYRgFeVUq8rpZIAtgC4psZranpKqQGl1LNTv48jd6FfnXttVkYisAHAVgG/Xei0EiEgMwDsB/B8AUEollVLDtV0VTTEABETEABAEcKjG62l4DIobx+8BeKjWni2hSywHsn/XnA2Dw5SsisgpAL4CnarsSAvB1AJ8CkK31QggAcCqAQQB3TKW0fFtEQrVeVLNTSh0En8PcA+gEMABhRSv20tqtqfAyKfU5EHpnKJ5r/65pZj/kscreK76rdSon8SUTCAO4B8Eml1Ilar6eZnichGAEeUUs/Uei00wwBwPoB/Vkr1AhgDwLqIGhORVuTuOJ4KYBmAkIh8uLaranxGrRdA+Smlrsz3n/0XkIwA2ArhCsb9erRwEcMqsP6+Y+hrVmIiYyAXEdyml7q31egiXANgkIu8H4ACIisidSile7GvnnAIADSqnpuyg/AINiP7gSwF6l1CAAiMi9AC4GcGdNV9XguFNcx0TkvcjdhtyklBqv9Xqa2NMAVovInqSJiIVcMcX+N19T0RESQy5Pco5T6Wq3XQ4BS6tNKqRVKqVXI/Zw8yoC4tpRShwHsF5E1U1+6AsCLnNVwS5fQD2CAiwan3sivAAsiK405xfbsdgA3g4dzPDLYppW6t7ZKaj1IqLSJ/DOAnyFUI/5tSaneNnl0W5XclbAPxKRJ6b+tpnlFI/ruGaiPzoTwDcNfWh/nUAH63xepqeUuopEfkBgGeRS4/cCU63qzhOntCMiIiKipsf0CSIiIiJqegyKiYiIiKjpMSgmIiIioqbHoJiIiIiImh6DYiIiIiJqegyKiYiIiKjpnMSgmIiIioqbHoJiIiIiImt7/B0La3GoHpmeiAAAAAElFTkSuQmCCn”, “text/plain”: [

“<matplotlib.figure.Figure at 0x1320b0c18>”

]

}, “metadata”: {}, “output_type”: “display_data”

}

], “source”: [

“f = gpflow.models.GPR(X, Y, gpflow.kernels.RBF(1))n”, “f.compile()n”, “gpflow.train.ScipyOptimizer().minimize(f)n”, “full_lml = plot_model(f)”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Upper bounds for sparse variational modelsn”, “As a first investigation, we compute the upper bound for models trained using the sparse variational GP approximation.”

]

}, {

“cell_type”: “code”, “execution_count”: 13, “metadata”: {}, “outputs”: [

{

“name”: “stdout”, “output_type”: “stream”, “text”: [

INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 49.220787n”, ” Number of iterations: 25n”, ” Number of functions evaluations: 36n”, “4 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 40.211568n”, ” Number of iterations: 52n”, ” Number of functions evaluations: 79n”, “5 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 37.583478n”, ” Number of iterations: 35n”, ” Number of functions evaluations: 43n”, “6 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 30.092951n”, ” Number of iterations: 32n”, ” Number of functions evaluations: 49n”, “7 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 26.316172n”, ” Number of iterations: 33n”, ” Number of functions evaluations: 45n”, “8 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 24.957100n”, ” Number of iterations: 30n”, ” Number of functions evaluations: 36n”, “9 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 24.310472n”, ” Number of iterations: 35n”, ” Number of functions evaluations: 41n”, “10 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 24.069186n”, ” Number of iterations: 45n”, ” Number of functions evaluations: 63n”, “11 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 24.014035n”, ” Number of iterations: 64n”, ” Number of functions evaluations: 81n”, “12 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.981403n”, ” Number of iterations: 58n”, ” Number of functions evaluations: 72n”, “13 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.979629n”, ” Number of iterations: 76n”, ” Number of functions evaluations: 92n”, “14 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.979570n”, ” Number of iterations: 67n”, ” Number of functions evaluations: 76n”, “15 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.966980n”, ” Number of iterations: 175n”, ” Number of functions evaluations: 209n”, “16 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.966979n”, ” Number of iterations: 236n”, ” Number of functions evaluations: 278n”, “17 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.966959n”, ” Number of iterations: 163n”, ” Number of functions evaluations: 193n”, “18 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 23.966942n”, ” Number of iterations: 127n”, ” Number of functions evaluations: 158n”, “19 “

]

}

], “source”: [

“Ms = np.arange(4, 20, 1)n”, “vfe_lml = []n”, “vupper_lml = []n”, “vfe_hyps = []n”, “for M in Ms:n”, ” Zinit = X[:M, :].copy()n”, ” vfe = gpflow.models.SGPR(X, Y, gpflow.kernels.RBF(1), Zinit)n”, ” vfe.compile()n”, ” gpflow.train.ScipyOptimizer().minimize(vfe, disp=False)n”, ” n”, ” vfe_lml.append(vfe.compute_log_likelihood())n”, ” vupper_lml.append(vfe.compute_upper_bound())n”, ” vfe_hyps.append({p.full_name:p.read_value() for p in vfe.trainable_parameters})n”, ” print(“%i” % M, end=” “)n”, “vfe_hyps = pd.DataFrame(vfe_hyps)”

]

}, {

“cell_type”: “code”, “execution_count”: 14, “metadata”: {}, “outputs”: [

{
“data”: {
“text/plain”: [
“<matplotlib.text.Text at 0x13d2b2668>”

]

}, “execution_count”: 14, “metadata”: {}, “output_type”: “execute_result”

}, {

“data”: {

“image/png”: “iVBORw0KGgoAAAANSUhEUgAAAtkAAAGDCAYAAAD+sAySAAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8VFX+//HXSQ8EQu8lFOlFOhZAxAKCsJa1rHV1Vey6nll3Xr6vu6qqruzb0Z19dRcUushaUSFMQadJLAqH30ENCkjm/P84NTGII6TczeT8fj3nMzL137nxmnEsJ7zpxirLWIiIiIiEj5ifC7ABERERGRcKOQLSIiIiJSzhSyRURERETKmUK2iIiIiEg5U8gWERERnESlnCtkiIiIiIuVMIVtEKp0x5iFjzDtVoI4kY4w1xkSV8HHGGPMfY8xuY8yciqqvvBhj0owxZxTjnuFK9H+XBGDPIGLOygs79pjHmkTKe4y/GmNeK2H+1MWZmWZ5DRMKLQraIAMcOYsaY07zg9WmB7T29n7VODtlljTPtKKNdvpwJnAi2stf39LsZv5fGhyVo7w1rbsbxqKm/W2n9Ya/8A5fNhxBhzqjHmR2PMnXmNMujHmB2NMv6D9TY0xrxpjNhtjDhhj1ngfFjoVqOGAd0kzxvw56PHWGHPQ27fJGPNvY0xkWd4DnESkZhWwRKY4dwEnGmPpB264CVvlUj99aA2nW2oMlfaAfrcR+81r+9f+NxxhTG5gEPA/UA5oDDwNZn3v76wI9ADWAQUAvoDUzDfbgLVsdamwBcCvzVGDM8aF9Pb98Q4GLgmop6TSLya/qjJyLFcRj4DLgEnwGsRuxgYX4ZzxhljJhhj9htj5htjeubtMMZ0NsZMNcbsMcYsNcaMDto31Rjzh6D7+b6m91rwxhpjnVnuPf8EYY/LqNsY8ZYzZaYxZA4wMLsg71xqvprXGmMsKFm2MuRZ4Dfeh44Ax5mFv+3XGmBSvVXKinMaZZgZpuNsasBlYXcs68VsnfG2M2eN1Qxhpj+hljFnmvY1zQ8RHGmP8zxqwzxmw3xvzXGJMYtP8Knb98uY8z9BZ4rwhjzZ2NMqrf/A2NMvcJ+QMV8P4YDfwEu9t6PX4J+To8aY34AMoC23utb7p1vjTHmnhqDznGaM2Rh0P80Yc7f3+vd6vytxQftHGWMWeu/Nj8aYHkH7enm/U/uNMROAI48rpP51xpg+3u3LnvJ9DV+/+tcaYz7zbwa31073rPd5rPinofE95P7+1xpgRx3jaDgDW2vestbnW2kPW2snW2kXe/juBnfcAV1tpU6+yx1v7HWvt8YSe01s4ClgLdCtmXAvwAnHis90FEyp9CtogU13+BK73bZwNLgM1lON8Yn4ENcS967wGfGmGhjTDTwBTAZaATcCow3xpSkK8EooB/QA7jIqxfgOm9fL6AvcGHeA4wxNYHngBHWn2lrAycDCgie21r4OjAVmWWsTrLUPGmNOBx7znqspsA54v8BDfwMMALoUUfcA4ATcB5hngPuBM4CunwEXGmCHecVd7l6FAWyABGOe9ji7A/wOuAJoB9YEWQc9xq1fLEG//buCFgoWU4P34GvgHMMF7P3oGn7b4CuB7XErsO2I57/2sDvweeNsb0LuL9uAgYDrTB/Syv9mrrBbwB3OC9vpeBicaYWGNMDO4D4du4n360PgQuKeI5pwGne7SHAGmBw0P1phTwmb38d7zXP8u4PAFYCDYB/Aq/nfcArYBWQa4x5yxgzwhhTnt8D+M4BPrbWBIuo+wjin4H5PFhSyvxOuRTylOOcTkfKhkC0ixWKt/RGo54XdK3GhuyzmWWs/stZmnA//GtTYO9C4JwOPW2sPW2mTcV+uXluDcj3stf+uB7znagncR8Iy1doO1Nh0XjIMFgG7GmHhr7RZrn7dJiPt9lwBvW2vnW2izgPlxLd1LQMY9Za9OttYeKOM/frbWZ1trJwEHgPWvtdmvtJmAG7sNB3vP9n21q7xlp7wHu+S4zrinIhMMlaO92r5QHvdeUZC9xvrd3o7X8IuNAU3o2ltO9HnjettUuttTnW2mxrn7f+CWman4T5IDSri8c9Zazd7P6svOPpzvB542Vr7k9cS/Bauq0Xe70807uecba39CPi5iOeYhgvTneLU8FnT/WCH7WNZZa1+11uYCb+E+cDUueJC1dh+uX78FXgV2eN9+5B3bANiad7wxZrTXYr/fGDO5nwOl2Aum4b1f+bK2dErRvvjHmILAcmAq8WILXIiJlpJAtIiXxNnALrgX10+Mcezwb8m54LXYbcS2rnzYANBVrx1uH6rRbX1qDbGbjQTt65C5w3r4aDuBbkscAWY8z/vBbA4mhW4FwHgF0Fat5Q8EGF2BZ0n+1Ah94Nfx7qgfeuAKFygy/cavde1K+jY1sCnXmjbgwtguRQIg2V8P/Lke81eq+1s47rU7AHOwQXKnYznWz7E1cFfea/DO1ZKjvz+brLU26LHB71VB04BBxpimQCTwAXCK9wEpkUJa74tTr7U2w7uZUNiBn1trl1tqrrbUtcF08muG+vQD382oadOxEa20dXDeSmAKnamCtrWut7Wytfa7Avt7e81+Ma2WvWYLXnIiJlpJAtIiXxNnAT8GVQiCitlnk3jBsU1wLX/WQz0NLkHyjXCtjk3T6IGxCWp0kJnnNL8PN65z3CnWvuNtfZMXMBZgWtlLI7NuOAHHOlqUT+oZnCtluUl3/PhXkcOLpTne43GmBpeLXk24LqA1Am6xHmtn5fmU4P041ms7st0YEwt8DDwFNPZC45dAYd0pjmcD8GiB11DDWvse7vU3L9BNo1XhpznSXzkD141mnutfKvBXXWj7zGF02yvNnibV2BfAmR/tTTwF+Y8phsKj3rcEHwCzgr2U9n4gUn0K2iASLNsbEBV3yndSGw1q7FfYV+f+EPByCmwDmONW1YH2PM+d5z3IH7un828BMu9Nzr9dE+DTiXo32cFwLnG2NqGDddn4LUleH0fALcZY1p4/WCDpzxrbIwZ4wXkLOAA+btZFOU94PfGmBO9MPkP4CdrbVoJaiuJ94A7jTFtnjDEJHO0TnQN8BIwyboq4GOBv5P9b/xLwqDGmNYAxpqExZkzBJyjh+7ENSDpOKIwBYnEz1eR4gwLPnKsFrDvYqMNYYM8Drj1zTGDPSGFMLFyZzcD/naGPM+cDxplmchvuGJq9ryNQC9wvagXsv2pameGNMnJ2PMXcaYFt79lrjuULO9Q/4N1AXeNsa0815jLco2cPFx4DpjTEk+lIpIGShki0iwL3HdEvIuDxU8nwFo701pb1IDHpQXO8ftjHPc57mvs3bgBcud7fWgP40L1CFx/0xeBK73WPoCncbOdbMP1ey3JDCevnAt8AvwDzgU+C9kUAf8S1EqfjPkzcWJyTWmu/w/V9/hjXktoObyaWCvIG7luF6cBaIBPXEovXb/pmn3GDSLbj3d2PQY58FJgKTjTH7ccFuQCHPUZL340PvepcxZn5hB1hr9wO34T7o7AZ+59VRYtbaubhBnrOO8c6XgDYr0fn/O9+6n437HPinsPEGm4QZnTj/G/YLPnwE8CvzgdVcZWMKXsB/3nv/k9ZmejRtInfJd3/p24vuWZwEzv+IVeTcX6nSyk5sW413NPaR4vIiVn8ndbExERERGRslJLtoiIiIhIOVPIFhERnEREpZwrZIiIiIiLlTCFbRERERKScKWSLiIiIiJSzwpbRDTkNGjSwSUlJfpchIiIiImFu3rx5O621nDY93XFiE7KSkJObOnet3GSIiIiIS5owx64pznLqLiIiIiIiUM4VsEREREZFyppAtIiIiIlLOwqJPntoiIiIiUr+zsbDZu3EhmZqbfpfgiLi6OFi1aEB0dXarHK2SLiIiIyK9s3LiRWrVqkZSUhDHG73IqnlbWWXbt2sXHjRtq0aVOqc6i7iIiIiIj8SmZmJvXr1692ARvAGEP9+vXL1IqvkC0iIiIihaqOATtPnWV+7QraIiIiIVEkJCQl+l1BqCtkiIiIiUq3l5uaW+zkVskVERESkSrPWcs8999CtWze6d+/OhAkTnALj55puZOHEiAOeddx7XXHMNAG+88Qb3338/AO+88w79+/fnxBNP5IYbbjgSqBMSErjrrrvo2bMnns2bNKveaNbuIiIiIiBTp4S+WsmzzvnI9Z5dmtXnw3K7FOvaTTz5h4cKF/PLLL+zcuZN+/foxePBgnBg0axIwZMxg9ejSbNm1iy5YtAMyYMYNLLrmE5cuXM2HCBH744Qeio6O56aabGD9+PFdeeSUHDx5knwIAB/Otf/yrX15VHIVukKIczYOPPYHMhIsq7RENEpLsdGe1ti/S2522LDDrW21+NB4+IiIiUxcyZnM7n00kuJjIykcePGDBkyhJ9//plBgwbxzDPPsGzZMrp06cLu3bvZsmULs2bN4rnnnuOtt95i3rx5n9OvXD4BDhw7RqFEjACIjI7ngggsqrGaFbJGCcrIgZQos+RhWfgXZB8vnvEdCetClsJAeEQWRJTy2ndnPoOAKa9FCYFxGRclfcFufK1rx5c/bs2cPXX3/N4MGDSU9P54MPPiAhIYFatWphreWqq67isccen+9Vj4+LiiIyMrLDaFLJFAHKzYc00WPoJLJ8EWXshvi50vxA6jYK42u6YQE7+S242BHK9+3n7c4OOnDdqfG7Q/EHSu3JwiHu9dcjKLPvbANpj6GCS2hE4j3aXVyS6Ai4iIhLhBgwbx8ssvc9VVV5Gens70n6dN58sknARg4cCDPPPMMycnJ7Nq1iwsvvJALL7wQgGHDhjFmzBjuvPNOGjVqRHp6Ovv376d169YVnXrP+B5bqK5ALaTNdsF42EQ6lQ2xtF6q7nQ9tT3Otx6HgwA5Y9TWs+B/MexN+egni6kCH4S5wtx8GnMTX9rlJERKRUzjvvPGbNmkXPnj0xxvDPf/6TJk2aAC6AT548mfbt29O6dWvS09MZNGgQAF26dOGRnRx7hrLPOIhAIEB0dzQsvvFApIdtYayv8SSpa37597dy5c/0uQ0JBIAAbfvKC9eeuBTi6putq0e18naDcMouP8rrJsDh+E1GQXuFd+BZl7ICrOfWjoNBI6jICEhn5XKSIiVdzy5cvp3Lmz32X4qrD3wBgznz1rb93iPVUu2hD9rYfN8WPIJLP0U9m1yofOEM6HbBXDC2RBTw+8qy09MTeh8rrvk5sD6WS5wr/ifna+3GQKuB0PEcF7rrt/O7YhERkbCjkC3hyVrYtsQL1p/A7jQ3WLD9MDjjIddyHVvL5yIrQWQUtBnknLsMfg62LvRbu/8G3D7hLw85eP+5zoFlvDZwUEREpBwrZEl52rHTBesnHsGs1mEhoOwQG3Q2dR7nBnjNWVMdC0h7sMvQ92r4OVX7rQPfNpmPEU1GrmwnankdD6VIiK8btqERGRkKSQLaEvfc3RriDblgAGnkk6FgTdClzFQs4HfFVZNdVu792jgjZCRDqu+gRWTYMF4+Pk1iE10XWo6jYT2Z7gZVkRERKRYFLIlnNO3Z4EL10k9g8wK3reUAGP6EC9a1m/pbX6ipUQ9OvNRdsg/BmqkucK/8CpZ8BJEx0GaIa+XueA7UnauJ3xSIiIlWaQraEjv1b3YwgSz52M4QAND0Rzvw7dD0P6rT0t75wER3v+qx3HOGmOdzwkzdwchJMnutNdWvRzLdwdR0LDDn5XLCIiUuUoZEvVdnAXLP/cdQdJmwlYaNQVTn/ABWvNjFGxIiKh9cnuctYjnsH350cD93UPuUv8Eb+DkKGjeByIi/K5aRETEdwrZUvUc2uNC3JJPXLcFm+uC3JA/ubmsG3b0u8LqnyRho3MVdhtwDeze67iQrJsGscfDDM5DQ2LWAdxoFbQZDVKzfVYuIiBSLtRZrLRHl1FikkC1VQ9Z+nr//vJ5DynVs2vE5rOOU2N5d1426aWq6qSWwB/a9zl0N7YPW3LnAv/sitOhmT4AZOdhzpruPr+F2xniIiEmLS0NEaNGsWSJUsAeOqppzhw4ABTp06lZ8+eTJs2jZycHN544w369+/PQw89RGpqKikpKezcnuZN7772X6667DoAnn3ySDz74gKysLM477zwefvhh0tLSOPvssxkwYADz5s3jyy+/LLfVIBWyxR85nh2HnSjdv88qvYPVkyMl0U8gNuAG6ng/NNWdzyIivAz1+6y7ZmbB2upuLe8WXboBqRDT0uRqG/sUNnshQRkdDy1Z/d/9nlqUl3GPF4qR+ekZHBwoULmT59Otdcc82RIL5o0SJmz57NwYMH6dWrFyNHjmTJnkiWsXr2aOXPmYK1l9OjRTJ8+nVatWrF69WreeustBg4cWF6vDFDIlspwYLv7h7ltqZtib+sSF7ADnOW5/zUbQ+0oXrFsOUJ/eUBcdBx3OcpeRT8OmubDwXZj7upsNZthfodcVrr+3iIhIKV166aUADB48nmH379rFnzx4AxowZQ3x8PPHx8QwdOpQ5c+Ywc+ZMJk+eTK9evQA4cOAAq1evplWrVrRu3brcAzb4nFLKNMb8FHgI6A/2ttXOD9t0HXAvkArdZa7/xo0Yphdxs2LnKhehtXqjeugQObj96TK1m0KQbdDgbnGnd1n2Lrt1fgClcREdCyv7v0uxa+vBe+uN11JznnKWjR1+8KRUSkOMrQ4lwWUVFRBAKBI/czMzOPn3DYFvu3Ou1/Ydmst9913HzfccEO+fWlpadSsWbO8ywb8a8leApwPvBy80RjTBbgE6Ao0A74zxnSwn1uZWfolSpIM7f906vWOF60sNEBkLjTq5vriNu7lg3bibugpUZ026w++/dH22v30AXhsGJ14OZzwInCY38rk5ERKqgxo0bs337dnbt2kVCQgKTJk1i+PDhAEyYMIGhQ4cyc+ZMEhMTSUxMBODzzz/nvvvun4+DBg0ydOpXHH3+c+Ph4HnjgAS677DISEhLYtGkT0dHRFVq7LyHbWrscfv1JAxgDvG+tzQLWGmNSngP7ArMqtUI7IzXHLkxdsnT6w9egxCU1ciG4/7Gigrt8eIiv2l1dCkDGu33bH4TD9SZj1Iiyf6Ppqn97sOItWDTUREjoqOjuavf/0r/fv3p3nz5nTq1OnIvri4OHr16kV2djZvvPHGke09evRg6NCh7Ny5nkwceeIBmzZrRrFkzli9fzkknnQRAQkIC77zzDpGRFfdNurHWVtjJj/vkxkwF7s7rLmKMGQfMtta+n491/HfjKWvtRIY+9HrgeoFWrVn3WrVtXaXWHrYz0o63S25a4luodKyE3y+2PiIaGnY62Sudda9lynKa2dq+GreyE1GRp1gRH/hDaD/K5KRESA5cuX07lzZ7/LKNRpp53GU089Rd+++bsdPvTQQyQkJHD3n3XeXy/MU9h4YY+ZZa4/b37HCmo2MMd8Bha29fL+19vOynt9a+wrwCkDfvn39+6QQinJzID3V6+6xn5Gjr9P7NR4+p2ciF6LZDoHF3d7tBB7VOS/lqcAJc/olb4Oab++CtUW4A7FmPQGJzv6sTEREptQoLn2dbaM0rxsE1A8NrYLbxtUlqHdh8N0dsWH+07neMNHIiIggYdXeth425HByOqj6xUFmOg8yjX3eiHnZ2Hm07Dqaxh8N5x0ixa0ERGRX5k6dWqh2x966KFKraMoVa0D5ETgXWPMv3EDH08A5vhbUghbPgkmnXA54Df01GrgW6X5/ONrdo0FHiIrxtUwRAKLj4bQ/Q89L4Zu/wJS/wYJ3YPgTbjpAERGREOLXFH7nnAc8DDYH/GWMWWmvPttYuNcZ8ACwDcoCbNbNIGSz9FGrUh/NedoE6obEWd5Gqr25ruGQ8pEyBr/4En7/4WOoyA4f+Aem39rk5ERKRYfFn1w1r7qbW2hbU21lrb2Fp7dtC+R6217ay1Ha21X/lRX1gIBGDNn99D+DDjhDKjVRAFbQkv7YXDjj3Dm3yBtBrwwEJIfgcMZflcmIiJyXFpaL1xt/QUydrmgIhKqomLgnlNvhlrnQZYyb9u+F/rD0M/BxZiQREZHjUcgOVylT3HXbof7WIVIeajeFC16Fq7+EuET48Cr47xjYnvsLvykREpAI999xzdO7cmcsuu+yYxyQkJABu9cZu3bpVVmnHpZAdrlKToUkPSGjodyUi5SfpFLh+nmluSfctCeOkU+OZ+yNznd2UiIlIBXnzxRb799lvGjx/vdyklppAdjrL2w4afoN3pflciUv4io6D/ndXDrfDjxMpj1AozrCwvfc2MRREQkLIwdO5Y1a9YwYsQIEhMTeeqpp47s69atG2lpaf4VVwxVbQo/nKQ9rZ0AgR/2xJbzVbACjn4M+V8GX98BnY2Hef+CcJ6FpT7+rExEJK6u27Wd/Zna5nrNWXDQdGtc6n5v6XXnqJr7/+mu+//55x48aV63NXBrVkh6PUZIiuCS0H+F2JSMVr3geu/Q7GvAC7UuHlITDpTshIn97syERGpxtSSHY5Sp0DSqVopT6qPiAjodTl0GgVTH4M5r7p54of9FXpfBRGRflcoIhLSimpxrgxRnUVEEgroEZmZm+lhN8aglO9ykr4X0NeoqItVTfB0Y8QSMnQGNuroW7VeHwvqf/K5MRETKICkpifnzn5wMwf/581q5d63NFx6eQHW5Sk921Bj1Kdda4K1w9CS58Aw7sgDfOgk/Hwv5tflcmIiKlcMEFF5Cennk7Xrl0ZN24cHTp08Luk41J3kXCTmgyJraB+e78rEfGXMdDtAjjhbJjxFPw4DpZPgqH3Qf/rITLan7wpFROQ4gmcQmTx5cqHHHDhwAHCt3UuWLKmMsopFLdnhJDcb1k6H9qdrCXWRPLEJcMZDcNNsaDUAnvvkLvHQqrJnqc2EiIhLOFLLDyca5kLVPXUVECtOgPVz2EVzyHmQfcitGfnAl7Nngd2UiIhKGFLLDnSeoUMBHQZojflYhUTcZAp3Pg5jkw9H5YNRnG9YPpT0J21R+pLiIioUMhO5ykJkPzvm6GBRE5tug4nGHIv3DIHTjgTkh+BF/rDgvGQm+N3dSIiVYa11u8SfFPW166QHS4y0mHTfE3dJ1ISdVrBxW/DFZ9BnXCJ8fhO80M8t0a6wLSLVXFxcHLt27aqWQdtay65du4iLiyv1OTS7SLhYMxWw0E4hW6TE2g2FttNhn5Zfw/WNuifYZT8GQP7kZSrSYjYhUQy1atGDjxo3s2LHD71J8ERcXR4sWLUr9eIXscJE6xbXENevlndyUiockY6DQSOoyAFZNg6uPwyXWuv/aQP0HX8xS2RaRaiY6Opk2bNn6XEbLUXSQcWAspydD2NIjUn5yaRMomIgC6jYexM+O1bYCLh42vh/50MSz6BoGV9RUREjkUhOxzsWAn7N2vqPpHyFBEBXX8DN/4InF/7Hbfvo9/DSKbD0M4VtEREpkkJ2OEid4q7VH1uk/EVEQLfzXdi+4HW36NOHV8HLg2D5F+6bJBERnkQIUssNBajI06AB1WvpdiUj4ioiE7hfCzT/B+a+6BW0mXO7C9or/KWyLiEg+CtmhLjsT0n5QVxGRnyhIRCT0ucgva/OYlOHwQ3v8dvHIarPxaYVtERACF7NC3/kfIOaSuIiKVLTIKTrwUbv4ZxrwImXvgnvYvh1dPdSpIK2yIi1ZpCdqhLTYbIGEg6xe9KRKqnyCjodRncMhdGj4OMnfDub+G1M2D1dwrbIiLVnlEJ2qEtJhlYDIaam35WIVG+R0dD7CrhlHpz7LBzYBuMvgNfPch+GFbZFRKoVhexQtn8rbF+qriIinVUlUDPS5Gm6dD6Oehn2b4O3z4I3hbmVWhW0RkWpBITuUpSa7aw16FKl6omKg7zVw2wI45ynYsx7+nOwbeHAlrZ/hdnYiIVDCF7FCWMgVqNoLG3fyuRESOJSoW+l/nwvaIJyF9Dbw1Ct4c5WYGEhGRsKSQnHaoCAVjzPbQb6hbLEJGqLToOBlwPty2E4U/AzlXw5jnw1mhYP9vv6kREpJwpnYWqrb9Axi71xxYJnNdFxMHAs3P4LnP0P2L4M3jgb/vsb2DDH7+pERKScKGSHqiP9sYf6W4eIlE50PJx0M9y+CM56BLYunhtfPhHcugI3z/K5ORETKSCE7VKUkQ5PukNDI70pEpCxiasDJt8Idi+CMh2HzAnjtdBj/W9g03+/qnRESklBSyQ1HWftgwW11FRMJJTE049Q7Xsj3sQdj4M7w6FN69GDYv9Ls6EREpIYXsUJQ2EwI5mrpPnJBzFJsCgP7qwffr/uUGRrwyB934H25b6XZ2IiBSTQnYoSpkC0TXcSo8iEp7iasPge1w3kqH3w7qZn8NKp8PktsG+L39WJiMhxKGSHotRkSBrk5t8VkfAWlwhD7nWzkQy8CRZNgOd7w/ePQdYBv6sTEZFjnUMgONbvTID1VXUVEqpv4unD2o3DzHOgwHKY9Ds/3gfn/hUCu39WJiEgBCtmhJmWKu26vQY8i1VK9nNvDb/8C130Hd1jDxVteNJOU7vysTEZEgCtmhJjUZEltC/fZ+VyIifmrZD675Bi76L2QfcvNrv30enbF3id2UiIoJCdmjJzYa1011XEWP8rkZE/GYMdBnjupAMf9zNsf3SqfD5zRocKSLiM4XsULJpHmTtnU1cREckvKgYG3gi3LXCrSC76wBsc+Q8NjhQR8YlCdihJmQImAtoM9rsSEamKfjU48gkXtue9qcGRnIiKVTCE7lKROgeZ93X+kIiLHkm9wZBv44nbXjWT1d2Ct39WJiFQLCtmhIiMdNs3X1H0iUnwt+8E1nX8NFb0NOJozPGxy52O/KRETCnkJ2qFgzFbDqjy0iJWMMdBkNN/3kBkduWQgvDYLPboZ9m/2uTkQknbClkh4rUZLfyW7PeflciIqEoeHDkybfA4g/cYjYaHCkiUiEUskOBtS5ktxkCkVF+VyMioSy+Lpz1nCNzyM3Qc4QZHPtfLDY7MzfG7OhGRsKGQHQp2rIR9m9RVRETKT90kuPAN+MMUqNc2aHDktxocKSJSnDhSyQ0FqsrvWoEcRKW8t+h4dHJmbBeMvhLd/o8GRIiJlpJAdClKnQP0ToE4rvysRkXCUb3DkE7DlnFw2OFBEpI4Xsqi47E9J+UFcREal4UTEwcCzcthBOvtUNjnyuNyQ/Cln7/a5ORCSkKGRXdetnQc4hndRURkcoTXwfO+rsbHNnpHJj+Txe25/5HgyNFRIpJIbuqS50CEdGQdKrflYhIdRM8OLJ+O5h0hwZHniogUk0JjCByQAAAgAElEQVR2VZf6PbQaCDE1/a5ERKqrFn3h91/Bxe/kHxy5ZZHflYmIVFkK2VXZn/q2wbYn6Y4uI/4yBzufmHxz58mD47CYNjhQRKYRCdlWW+r27bqeQLSJVxK8GR37oDY58RIMjRUSCnKGRXZalToGZDaNzN70pERPI7MjhyLnQaCdOfdCtH/vg8HD7od3UiIr5TyK6qAgG3CE270yFCPyYRnqaLqtoYLX4c/JEOjLjD5/+CZ7jDzabVsi0i1pvRWVW1dBBm7NHWfiISGFn3gqolwzWRo1gu+e8iFn7elPQuZev6sTEal0CtlVVeoUd62QLSKhpNUAuPxj17LdcoDrq/1Md5j6OBza7Xd1IiKVRiG7qkr9nHpp0h4RGflciIlJyLfrA7ybA9dMgaRBMfQye6QFT/g4Z6X5XJyJS4RSyq6KsA7B+tlqxRST0NTsRnLhkPY39wf9Nm/Mu1bH/7IBzc6Xd1IiIVxpeQbYx50hizwhizyBjzqTGmTtC++4wxKcaYlcaYs/2onz3dpMyCQran7RCR8NOkGF70FN82CDsPhh2dd2P7mfti/ze/qRETKnV8t2d8C3ay1PYBVwH0AxpgunwCVAV2A48KIxJtKnGv2TmgzRNdxKjyIi4aRRZzcbyc1zoPNomP0iPNsDvvqTFrURkbDiS8i21k62n1uZ4d2cDLbzbY4D3rbVZ1tq1QArQ348afZUyBZJOhahYvysREakYDTvA+S+7eba7Xwg/vwbP9oT/n3QV7NvhdnYhImVWFPtnXAF95t5sDwX9dN3rbfsUYc70xZq4xZu6OHTsquMRKtDsN0lPVVUREqof6n7WDMC3DrPDjxdzDvLbeozRe3w+51flcnIlJqFRayjTHfGWOWFHIZE3TM/UAOML6k57fWvmKt7Wutn7duwYcPyLN1fqcnuWoMeRaQ6qZsE5z4Lty2APlfBwnfh+d7w+c2wK9Xv6kRESiyqok5srT2jqP3GnmKuBUcAwa631Nm8CWgYd1sLbVn2kTIHEltDgBL8rERGpfHVawsh/waC73ODIeW+6wN39Ihh8t/42nikjI8Gt2keHAvcBoa21G0K6JwCXGmFhjTBvgBGCOHzX6IjcH1k6HdkPBGL+rERHxT+1mMOIJuH0RnDLwJlk+Ecf3go2th+3K/qxMROS6/+mSPA2oB3xpjFhpjXgKw1i4FPgCWAV8DN1trc32qsfJtmgtZn+9QfW0QkT63GcPajLmyfcjus/ApePAk+uAq2LvG7OhGRY6qw7iJFsda2L2Lfo8CjlVhO1ZGaDCYCn2g7xuxIRkaoloSGc+TCcfJub9u+nl2HZZ9BpFAy+xy16IyJShVSF2UUkT8oUaN4H4uv6XYmISNVUnsz4MewDuXAyn3ecW73plCLx7MWyc53d1IiJHKGRXFRnpsHm+uoqIiBRHfF047c9wx2I4/f9gw0/wn2unwzgWw/ie/qxMRUciuMtZOAxvQ1H0iIiURl+i6i9yxGM54CDYvgDfOgrdGQ9oPflcnItWYQnZVnkTIFYhNddxERESmZ2Fpw6p0ubJ/1iJuB5M1z4D8jYc00ODJTrIhI5VDIrgqshdTv3YDHSF/GooqInhIeYmnDyrXDHIhj+hFtB97+j4Y3hrjFDYVtEKokSXVWwcxXs2+gWWhARkbKLjoeBY6HP1bDgbZj5nNLxzvuvLHVcH4mpDbG3X3SS2dtD9gtcF9kfHax0DESkWheyqIGWKu1Z/bBGR8hUdB/2vg95XwqIJnsHmhW48gc5+7Tl+b/z7HaemOiCo6hBd6nZj/fnQNBXWRakAhuypITYb67aFua78rEREJT1GxLmj3nvvLYxwQCcPjA0dCduTcogO89GsQLXu9OK0VQr3XsEB4RDRER7jgTCRGR3u2IoNt52yMLHFPU9nI6nZ3S8uy0iRVLI9lt2JqTNLPoPv4iIVLyICBdy42pDYinPUTCo57vee4zt+2DP+qP3AzkQyHXXNtfNnPFWVxNaGDmdD53Oh/RmuH7yI/IpCtt82zIacQ9Be82OLiIS88gjqBVnrQrfNzR++A4Gg23nbAwWOnyT3GY4+1vRjn3LkKVnwJiz+EqDgXtDuf64K3FlMTOUIh228pU9xXg61P8bsSERGpiozxZp6qQv9lnj8qB9T/C8i9g+SRYMcl1LWkz2AXujiOhVmO/qxTxlbFhMJ1R37597dy5c/0uo3T+3ynuk//Vk/yunREREpOQCAbdi8fKJsGwi7F4LGGg10AXuTqM05kjCijFmnrW27/GOq0Ifi6uh/Vth2xIY9qDflYiIniJRORAS06OsuZzwM25d5LdxfwDd/cZemPV3g7jwaGnb0u2KRSqGQ7afU7921+mOLiEg4MAYad3WXn0/4Mu1JdV5LlX0DyI+7SoMPRwN20p6YzlLClkO2n1GSo2RAad/e7EhERkfJXvx2ccru77NsMK/7nnupXMfAZm/AsSW3mB+1xo2V9TA0pYUcj2SyDgQna7091XbSIiIuGsdjO3MFD/6+DgLlj1lWvh/vlVnmP0C1GwEnUa6wJ00CKJi/K5YpEwUsv2ydRFk7FRXERERqX5q1odel7tL5j5I+dYNmlz0Acz7j1tJns8MIF7jbnQ4xNfyuWKTEFLL9kprsrtsO9bcOERERP8XVhm4XuEv2ITdeafkXsPJLWPS+W4a+/RmunD3eHs1wAFwkBCtl+SU12fbE1j6iIiIgTHQ+dznGX3Gy3IvLyL7zBkxPduhJtT/OmBhwJNRv4XbHInMWmebD9kHYAnkuCkm+DMv/ldjYiISNUWCMDGn13QXv4F7FkHJsIt5JYXuBNb+F2lVBOaJ7sqS5sJngWzXz0xERESKFhEBrQa4y1mPwNbFR+fi/uped2nexwvc50KdloAJmh4w77Z3rWkDpRKERcjOOJzDnvHXpfpdRfPN/BtMN6AyhVLeIVFvWukvAWgJYbMC7HbzdWqx3sAUCFvD2We8ckPcY71iOnsNavOPsnkecMeLfznwMCAe8+1nsOd9vmO7drAM07Z97zlea1l0bof098DN57DUffm8Je69F9BY61wcf8+qdyn5KdX4Nx5vx/uTjRwPjbpfGo23kyj9Lk0Tp9LncnjYfL4ErwUg8UL30eeKyiQ5x1nvO15x3v3g4+3nBAf3oGM5eiobdO78xwc/nSm44dfHFtxuTJHHuddQ8LwFrgs8NPjcBc979D3L//z59hf2uHzva8FanTeGPM8Gvy3vvvLtxA/9AnWbtC3m1VUOxQ7Yxpoa1NqMii6k2Ns1zE/BHRvtdiYhUkkAAsnJyycwJnkJUdICsnl6zsXLJyAmRmB8jMyfW2BcjKcfszs92+LG9fbi4E8kJpwIXdQMB6YRQv/Lr9eQEYC7lYnrD16bK6XVnLzwnDA7c8NCsUuTHPkfGEbGCVMdAO60YA99IlYRQ0yj8RcCI601jVk593Ot98WiH/Bn+/nVbXPkX0Vw5Mx/TvI9hiPbg8+Vd45g+eO6LXCd/zG/3p7/cYVtM8e4f1T+9y34XPlf55GPswXenn7zPIQVrPNbzBz/Pr5/r6GPy15Ox9yB1mv2q+CrjuCHbGHMy8BqQALQyxvQEbrDW3lTRxRVXjZgon+rSu53cZxbN7HRz4Hk59HEKlZpFqwFpLVk6AQ4dzycjO5dDhHA4dDpBxOMe7n0vGYW97dt5td51xnOJdD2TlHj8m33x2fmR0oUT3GQI3oSOJjooiPiaBGdBTRUYYI4y6RkYbIiAiMMUQaQ0QER/dFGCIMnx9xnDG67MUR4x7rHBO2LyHsuvMd45zIc2RcZdN687cYYDBw5V4QBg/G+oc+ry+0n33HucXnHmQL7nCDomb7vxzh1R4NyFPp6jx5emp0BpexeYIloWQ9nRoBr83phC9pmgPUePLezncGSfMYUeX/DcFLVPnqoVacdHERFXdtUaK05L9NHA2MBHAWvuLMWZwhVYVzvKm7mun+bFFKktuwLJjfxab9x5iy55Mtuw9nxOa8672ZbNlziF0HD5MbKFl7bWxUBDViIqkRE0VcdAQ1YqKIj4mkXs0YWtSNJD7aC8gxUcRHR3rHnesE5Ou/20e1x0e5cNWIiiY2KOBJQREQk9BSru4i1dkOBP/a5FVNONZA6BWq3gAYn+F2JSFiw1rLrn4GG27Mn0QvQhtuzNPBKet+zNZNu+THIKBOj46Eia1omjeZ14OnZsSMNasUcCbsHAeyQIR0cRFxSanIyMUgkVEpHDFCdkbvC4j1hgTDdwOLK/YssJUbg6smQ5dx2hks0gxWGvZdyjHheeg1ucjgXpvJlv2nZnI4J39XjJjICJokxtE0MY7+berRNDGOpnXiaZYYR9PEeJrViSMxPlotxSIiUmGKE7LHAs8CzYFNnwGSgyvTHDimb5kHWXk3dJ+I5mJWTv+tGXojem8lmrxU643D+L84iIwyNa8XStE483ZsncnbXJi5Ene+G5aWI89WvGEKFWZhER8VFxQnZHa+1lwRuMMacAP1RMSWEsdQqYCLdalUg1sXF3BvPW7WZDeka+nLhyb9xxiX2bOr45vWCuWZolxnNCoFoM7NKRZYjxN6xwN0Q0TYomKrLoDXURERKB4Ift5oHcxtsnxnpCa7yfLj6/pdiUiFCAQsKTsOMGdtOj+npfPz2nQ27808sr9OjWiaJsbTvE48/ZLq0bROnAvRiXE0nqxNP49pxVXqkuIiISHEdM2QbY04CTgYaGmP+GLSrNhBZ0YWFnUO7XXeRwff4XYlIucnODbB4017mnpqUzZ+1u5q5LZ09GNuBapPsn1eP6pLr0TapHu4YJxMfoT4eIiFQPRbVkx+Dmxo4CagVt3wdcWJFFnhaU1U8EGNHWfhLSMwzksWL+Hn9a6VuoFG3Yfmf85qX4NzuzcmH5t6tE/qR6t69fQwEIREam2jhmynrbXTgGnGmDettesqsabwlJoMsYmuu4hIiEg/ePhIt4+f1+1m6aa95ATcimmdm9Tmkn6t6JdUj35JndWlUO87vckVERKqM4vTJzjDGPAl0BY78L2qt1RQZxWUtpCRD28EQWeyV7EUq3cbdGfzsdf34OS2dnlO0HADclXs+WiVw/uC392tSjT+u61I6L9rlaERGRqqs4iW88MAEYhZvO7ypgR0UWFXZ2roZ9G2Hwn3X5XInJEUYMUa8VG0bt1Xc7r1Zx+SfXo0SKRuGj1pxYRESmu4oTs+tba140xtwd1Ifm5ogsLK6lTn3LXmxxYfZecGWLJp75GW6nnr0tntDVJskBBL/zZ1uS6pHv2S6tG5aW2tZigiIlIGxQnZ2d71FmPMnSGAzUK/iSgpDKVOgfnuo29rvSqQayRukmNdSvWD9Hg5lu4VdWtevwbDOjemfVI9+beqRpEGKIiIin5ao4IfsRY0wicBdufuzawJ0VWlU4ycmCtJnQ+wq/K5EwtztvkGJaOnPSfj1I8eJ+LTVIUUREpJIcnN2Rbayd5N/cCQyu2nDC0fhbkHNLUfVIhcnIDvD5zLR/N28hqDVIUERGpMo4bso0xbYBbgaTg4621noyuurDCSmgwR0ZB0qt+VSJhZuXU/93z0C4s27mVAm3rcc3ZHDVIUERGpIorTXeQz4HXgCyBQseWEnoZRkaDUQYhP8rkTCRHZugJempvJc8mpqx0Xzwu96M7JHU7/LEhERkSDFCdmZ1trnKryScLR/G2xbnDMMe9LsSCRPLNu/jno9+YenmfZzbsxkPnduF+gmxfpclIiIiBRQnZD9rjHkQmAxk5W201s6vsKrCnxZrv3bWm7pMyOpwT4MWpKYxLTqFOjRheurwPw7s18bssEREROYbihOzuwBXA6RztLmK9+1KUlClQnowE06eF3JRLClmzay90f/sKKrfv5zYnNePDcrtStGeN3WSIiIlKE4oTs3wJtrbWHK7qYsBIIuJbsndqdDRITf1UgIysrJZVxyCi9OTaV+zRhevbIvZ3Zp7HdZIiIiUgzFCdlLgDrA9gquJbxsWwwHd6irniJTKoo17uOfDRazctp8Lerfgr6O6kFhD0/CJiIiEiuKE7DrACm8p9eA+2ZrCrygpeUupa2pxKb7Mn7Fyem7Kal6evoWFCLG9c3ZfTO6n1WkREJNQUJ2RraozSSE2Gxt2glganSfEsWL+bez5aRMr2A1zUntwX3j+xCYrxar0VEREJRcVZ8nFYZhYSVrAOwfjYMvNHvSiQEZGbn8vS3q3h1xhoa147jrWv6M6RDnQ7/LEhERkTI4Zsg2xsy01p5qjNmPm03kyC7AWmtrV3h1oWrdDxDIhvZaSl2KNm9dOvd8tIg1Ow5ynaf9W/OWcTtTSEugiIiIh75gh21p7qnddq/LKCRMpUyAqHloO9LsSqaIOHc7lqckreeOHtTRLjOednawdw6gkN/C5LREREyslxu4sYY9621l5xvG0SJHUKJJ0K0XF+VyJV0Jy16dz70S+k7crgioGt+dOInTiTEFmd4hIiIiISK4vzP3jX4jjEmCuhTMeWEgd3rYFcK9L3W70qkisk4nMM/v17JW7PSaFE3nnevnG8DJ7dR6LSIiEo6K6pN9H/AXIN4Ysy9vM3AYeKUSagtNqcnuWv2xJcis1F386eNFrE/P4OqTk7jnn7I7UVOu1iIhI2CqqT/ZjwGPGmMestfdVYk2hLTUZareABh38rkSqgINZOTz+1Qrenr2O1vVrMOH6ngQxoW9/vskRERKSCFacpbZIxpqa19qAx5nKgN/CstXZdBdcWenJzYM006DIajPG7GvHZDyk7+dPHni9i05xDXnNKGe87uSHxMpN9liYiISCUoTsj+f0BPY0xP4C7gNeC/wJCKLCwkbZoHWXvVVaSa25+ZnzWNfreDdn9bTtkFNPrzhJPom1fO7LBEREalExQnZOdZaa4wZA4yz1r5ujNGovsKkJoOJgDb6/FFdnTV+1gz9/vIit+zK5fnBb/nhmB+Ki1XotIiJS3RQnZO/3BkFeAQwyxkQAWi2jMKlToFlvqKFWy+pmnX2Y2j05azoS5G2jXsCYf3XgyvVvV9bssERER8UlxQvbFwO+Aa6y1W40xrYAnK7asEHRot+suMvgenvyuRSvb9yu385ZPFbNuXyY2nteP2YSeo9VpERKSaO27I9oL1x8AJ3qadwKcVWlUoWjMNbADane53nJVJJ9mZk87dJy/h4/kY6NE7gpctPoWfLOn6XJSIiIlVAcVZ8vA64HqgHtAOaAy8BGt0XLHUKxNaGn5n39rkQqwXfLtvGXTxez6+BhbhnanluHtSc2Sq3XIiIi4hSnu8jNQH/gJwBr7WpjTKOyPKkx5u/AnGCAAbAeuttZu9vbdB1wL5AK3WWu/KctzVQprIfV7aDMYIrXASDjbk3GYh79YxqcLNtGpSS1ev6ofn3Vsk+l2WiIiIVDHFSYRZ1trDxpv32VtW3ZbxeZ+01j7gne824K/AWGNMF+AS3FLuzYDvjDEdrLW5nZXy+irVzNezdAIP+6HclUoG+WbqV+z9dwp6Mw9w+7ARuHtqemKgIv8sSERGRKqg4IXuaMSZvefUzngZuAL8rypNbafUF3a3I0tI8B3rfWZgFrjTEpuFb0WWV5vgqXt5S6+mOHpfSDh3lo4lIm/rKZLk1rn89Y1/ejaTK3XIiIicmzFCdl/xnXfWAzcAHyJW5CmTIwxjwJXAnuBod7m5sDsoMM2etsKe/z1uL7intGrVqqzllE3qFKjXDuom+VuHlLtV2/bzu1dns/dQNn88swM3ntaO6Ei1XouIiEjRijO7SAB41bsUnmzHmO6BJIbvut9Z+bq29H7jf64N9C/BgSc5vrX0FeAWgb9++Ze2+Uno5WZA2E3pd7lsJUjEOZOUwn9p15gOGLW0+lU5PafpckIiIiIaLCRulZa88o5qHjca3jDwKbgJZB+1p426qu9bMhO0NdRcKMtZb7nPllM2s6DjP/DQAVsERERKRFfvvc2xpwQdHcMsMK7PRG4xBgTa4xpg5ube05l11ciqVMgIhqSBvldniZSjd2av44tfNnPXWR05qV19v8sRERGREFOqlmxjzFPW2rvL8LyPG2M64qbwWweMBbDWLjXGfAAsnA3KAm6v8zCKpydBqIMQm+F2JlJNFG/fw90nLGdqxITcOaed3OSIiIhKCSttd5CKg1CHbWntBEfsenBR4t7bkr1YHtsHUxDPur35VIOdmbkc1N4+fTsFYs/77oRCIijN8liYiISAgqbchW8oCgqfu0+GU4nCAQsd324kG37MvnghpOoWzPG75JEREQkRB0zZBtj6h1rFwrZTmoy1GgATXr4XYmUg1dmrOG75dt5n8Nwu9GpV1+9yREREJIQV1ZI9D7dITGGBOrtiygkhgYAL2e2GQoTmTQ51P63ZxZPfrGRk96ZcfXKSn3+WIiIhIiDtmyLbWtqnMQkJOdgZ0Hg3t1VUk1O3Yn8Wt7y2gVb0aPH5Bd4zRFzUiIiJSNkV1F+ldn1AOttfPLv5wQEpsAo/7tdxVSRrkBy+3vL2DvoWzeuqY/teKi/S5JREREwkBR3UXmAkuAnd794OY9nC2j1FQl5z363ih9Td/HPC3rQuakWnBEREZHyUVTI/iNwIXAIeB/41Fp7oFKqEqkE01bt4PnvU7igndwt+27eF3+WIiIhIGDnmiD1r7TPW2lOBW3FLnU8xxnxgjDmx0qoTqSCb9xzijvcX0KFRLR75TTf1nwxYREZFyddxpMay1a4DPgclAf6BDRRclUpGycwPc8u58DucEePHy3sTHRPpdkoiIiISZogY+tgUunAcYAG3BdRv5hrT1USbWJVIjHv1rB/PV7eP7SXrRrmOB3OSIiIhKGiuqTnQIswrVi7wNaATfmfa1unrdXUGhJyvl6yhddnruXKk1pzbs9mfpcjIiIiYaqokP033CwiAAWb+ywiISZt50Hu+XARPVskcv/Inzn6XIyIiImGsqMVoHjrWPmPMHRVSjUgFyczO5abx84mIMIz7XW9io9QPW0RERCpOadcD/2O5ViFSnwR7+YinLtuzj3xf1pGW9Gn6XIyIiImGutCFb851JyPhk/kbem7OBsUPaMaxzY7/LERERkWqgtCFbnfbIlJKzatp/7P11C/zb1uPsszT4pIiIilaOoKfz2U3iYNkB8hVUkUk4OZuVw0/j51IyNZNylvYiKnLO1nShEREZGSKWrgY63KLESkPFlr+cuni1mz4wDvXDuARrXj/C5JREREqhE17UlYGv/Tej5fuJk7nz+jAye0b+F2OiIiIVDMK2RJ2Fm/cy9++WMaQDg25eWh7v8sRERGRakghW8LK3oxsbnp3HvUTYnj6n4hOJiNBEOCIiIlL5ilrxUSSkWGu5+6Nf2LInkwk3nES9mjF+lyQiIiLVlFqyJWy8OmMN3y7bxn3nndKZP67p+lyMiIiLVmEK2hIWf09J54uuVjOjWhGtOSfK7HBEREanmFLIl5O08kMUt786nZd14nriwnB8aoH7aIiIj4SyFbQlpuwHLH+wvZnZHNC5f1pnZctN8liYiIiChkS2h7bspqZqbs5G+ju9K1WaLfn5YiIiIgACtkSwqav2sFzyas5v3dzLu7X0u9yRERERI5QyJaQtGXvIe6YsJATGiXwyG+6qR+2iIiInVCkK2RJysnMD3PLuArKyc3nxsj7UiNF07yIiIlK1KJ1IyPnn1yuYt243z13ai/aNEvwuR0RERORXn1JItIeWbpVt5dcZarhjYmtE9m/ldjoiIiEihFLIlZKzflcHdH/5CjxaJ/N+ozn6XIyIiInJMCtkSnEjKzc7lx/DwM8MLvehMbFel3SSIiIiLHpD7ZEhL+NmkZSzfv47Ur+9KyXg2/yxEREREpklqypcr7nbMEm3v1pPTcMacsZXRr7XY6IiIjIcSlkS5W2ett+7vtkMf2S6nL3WR39LkdERESkWBSypco6mJXDnjePnUyMmkucv7U10pH5dRUREJDSoT7ZUSdZa7v90Mak7DvD2NQNokhjnd0kiIiIixaamQamS3p2znns8WbuaOYR049YQGfpcjIiIiUiIK2VLlLNm0l4cnLmPQCQ249fT2fpcjIiIiUmIK2VKl7D2UzY3jn51GvZgzPXHwiERHG75JERERESkx9sqXKsNZyz4e/sGVPJhNuGEj9hFi/SxIREREpFbVkS5Xx2oy1nTF62jT+P6ESf1vX8LkdERESk1BSypUqYm5bO41+v4Oyujbn21DZ+lyMiIiJSJgrZ4rtdB7K45d0FnNK8Tzz8v7Ikx6octIiIioU19ssVXuQHLHRMWkp5xmE9uPJnE+Gi/SxIREREpM4Vs8c3BrByemrySnGat38o/zutOteaLfJYmIiIiUC4VsqXQ79mfx5o9reXvWOvZl5nBp/1Zc2r+l32WJiIiIlBuFbKk0na3Yc4NUZa/l4/kaycwMM79qE6we3pVerun6XJiIiIlKuFLKlwi1Yv5uXp63hm2VbiY6M4MI+LbhunUFvaNKjpd2kiIiIiFUIhWypEIGCZumo7L01bw5y16dSOi+Lm09pz1clJNKylRWZEREQkvClkS7k6nnBNg4i+beWV6Kqu2HaBZYhwPjOrCxf1akhCrXzcRERGpHpR6pFzsz8zm/TkbeH3mWrbuy6RTk1o8nfXFPRvVoRnSkpmMXERGR6kUhW8pk+75M/vNjGu/MXsf+zBxOalufxy/ozpAODbWojIiIiFRbCtlSnKinbD/Dq9DV8umATOYEAI7o35YbBbenRoo7fpYmIiIj4TiFbSmTeunRemraGb5dtIzYqgov7teQPng9rQur5mChERERHJo5AtxxUIWKas2M7L01KZu243dWpEc9v/b+/eo6wq7zyNPz9uchERxKhc5KKAnEVEiYDBtjElYauKFtNMmdjuMOi6NWZlcXGntoB2j7WiMxmXSq6ftGFtJJmltx8FLa+KNJF4ShRgvngKhBBBEEwRBBUAqr6jd/nO2kGquEqjpVu07xfNZi1Tn77LP397wU1Lf22efdnx7HGUeOYq/dnSlEnkiRpe5ZstaiuvoG7nn6NHz6yjGXrtzB8z35cetLBfH7aSPr38VtHkiSpJTYlvc+mre/yb/NXctNjny6FjEdwAABM3SURBVFn3Vh0H77cHPzhtMidM2o9ezhQiSZK0Q5Zs/X9rN27l5t8s52fzV7K5rp6jnDhzKtZ8/jKMOHOpMIZIkSa1gyRZLX3+LGx55mTufWU1DY3LCocP44tFjOWT4oLKjSZIk1aRSS3ZEnfAP4HrB3Zr5RLJsNnA00AF/NzPtLjNhtZSa/W/EnfvjwMua9sI6+vXtw+kdHcfZRYxg5pH/Z8SRJnkmpaaSU7IkYCxwIrmyw7GDgNmAgMAx6KiPGZ2VBOyu6nsTF5YMnr/PCRZTy98k2GDOjD+TPGM+vInUQwZ0KfseJIkSd1CmUeyrwMuBO5qsmwmcGtm1gHLI+Il4Ajg8RLydStb323gjqdX86NHXublN7Ywnckg/Lp85kb+aMpJ+fXqWHU+SJKlbKaVkR8RMYHVmPrvdB+qGA080ub+qWNbcNs4FzgXYf//9Oyhpn7dv49rv8dP4r3PybFbyxuY5JwwfxT3/zEY6fuK8zhUiSJHWQDivZEfEQsG8zD10MXETlVJE2y8wbngBsApk6dmu3ZVnf16NL1nPe/f8+WbQ0cPX5vzjt6LEcesJczhUiSJHWwDivZmTmjueURMQkYA7x3nFHsE8FREHAGsBkY2WX1EsUyttLmungtvX8h+e/bjH0/7CAcP26PsSJIkSbuMTj9fIDMXZeaHMnN0nZo6mckrI4Zm5FrgbOC0idouIMcA4YEFnZ+wOvnf/i6zdtJXv/pdDLdiSJEmdrEvNk52Zz0XEbcASnoB74sjOLtN4zr77Jjx9fwazpo5gyanDZcSRJknY5pZfs4mh20/tXAFeUk6b2vdvQyDf/70L2GdiXnC46bUHYcSZKkXVLpJVvVdeOjy3lh7VvcMGsKA/v2LjuOJEnSLsk53LqRV/64he8/9AeOn7gvx05snbmIXSZIkdQZLdjeRmVx8x2L69OzBpSdPLDuOJEnSLs2S3U3c8fRqHnvpDS78zEHsO6hv2XEkSZJ2naZbsbuCPm+u4/J4lTBk1mNOP8OqXkiRJZbNkdwNX3Ps8m+vq+c4pk+jRw6s5SpIklc2SXeMeXbqenuU+v5rxPHMD4fQaWHUeSJElYsmvaO9sauPiOxYwdOoAvf/LAsuNIkiSp4DzZNewH85aycsPb3HrundPr27ll2HEmSJBU8kl2jlry2iR89+jJfmDqS6WP3KjuOJEmSmrBk16CGxmT23IUM7t+b2Z89qOw4nkiRJ2o4luwb95PEVPLtqI5ecNJE9+/cpO44kSZK2Y8muMavffIdr7n+RYybszUmH7ld2HEmSJDXDnkl1DMpNL7lxMJlw+8xAinBNbkiSpK7Jk15CfL1rLvBfW8Y1jxzNySP+y40iSJKkFluwasfHtd/n2n3c8xafggzvzY6LLjSJIk6QM4T3aNuOq+F/jT29uYc9Y0evX0dyNJkqSuzLZWAxYs38AtC1Zy9lFjnOGT4oLLjSJIkaQcs2V1cXX0Ds+cuZMTgfnx9xriy40iSJGkneLpIF3f9r5exbP0W5pw1jf59/OuSnJEmqBR7J7sJeWvcW//yrZcycPIxjJnyo7DiSJEnaSZbsLqqxMblo7mL69enJt048uOw4kiRJagVLndhd16+9eZcGKDVx8wocZuvtuZceRJElSK1iyu6B1m7bynV88z5Fj9+LUKSPKjiNJkqRWsmR3QZf9nxxLq6hu58pRJXjpdkiSpBlmyu5iHlrzOvYvW8LVPj2PM0AFlx5EkSVIbWLK7kM119Vxy12Im7DOQncz4+tuw4kiRJaiNLdhdy7QMvsmbTVq48ZRJ9evlXI0mSVKtscl3EM6++yZzfrmDW9FFMGTW47DiSnJElqB0t2F/BuQyOz5y5in4F9ueC4CWXHkSRJUjt5ne4u4MZHl/P8mk38cNYUBvbtXXYcSZIktZNHnskv2yh+38P2H/sDxE/fluIn7lh1HkiRJVWDJLlFmcvEdi+nTsweXnjyx7DiSJEmqEkt2ie54ejWPnvfQGF37mIPYd1LfsOJIkSaoSS3ZJNmzZxuX3LGHKqMGcfsT+ZceRJElSFVmyS/I/713C5rp6vnPKnJHr08NLpkiRJ3YkluwSPLX2DuU+t5rxPHMD4fQaWHUeSJElVZsnuZO9sa+CiOxYxZugAvvzJA8uOnI0mSpA7gPNmd7B9/uZSVG97mlnOm07d3z7LjSJIkqQN4JLsTLXltEzc88jJfmDqSIw/Yq+w4kiRJn6iCW7E7S0JjMnruQwf17M/uzB5UdR5IkSR3Ikt1JfvL4Cp5dtZFLTprInv37lB1HkiRJHciS3QlWnv/kO19z/IsdM2JuTDt2v7DiSJEnqYJbsDpaZXHLnYjLh8pmHEOGc2JIkSd2dJbuD/WLxWua9sI5vnHDuekUP6lx1HkiRJncCS3YE2vvMu3777OQ4Zvgdnfmx02XEkSZLUSZwnuwN9974X2LBlGzefOY1enPf19RpIkaVdh8+sgC5Zv4N/mr+Tso8ZwyPBBZceRJElSJ7Jkd4C6+gZmz13IiMH9+PqMcWXHkSRJnUifzdJEOcP2vl7Fs/RbmnDWN/n0cYkmSpF2NR7Kr7KV1b/HPv1rGzMnDOGbCh8qOI0mSpBJYsquonsTG5aO5i+vXpybdOPLjsOJIkSSqJJbuK/v3JV1mwYgMXn/Bhhu6+W9lxJEmSVBJLdpWs27SVK3/+nPNPHDuHUKSPKjiNJkqQSWbKr5LJ7llBX38iVfznJS6dLkiTt4izZVTDv+de5d+EavvqpAxm79+5lnx5EkSVLJLNnttLmunm/duZgJ+wzk3KMPKDuOJEmSugBLdjtd+8CLrNm0lStPmUSfXg6nJEmSLNntn8syrbzLntyuYNX0UU0YNLjuOJEmSughLdhu929DI7LmL2GdgXy44bkLZcSRJktSFeM3vNtpSV8+wnQX35+oxxDOzbu+w4kiRJ6kIs2W20Z/8+3HjGVKfrkyRJ0vuUcrpIRFwaEasj4pniz2ebPDY7Il6KniBcj4rgy8u0sC7YkSZKaU+aR7Osy83tNF0TEwcBpwERgGPBQRIzPzIYyAkqSJElt0dU++DgTuDUzn6zJzOfAScETJmSRJkqRWKbNkfyUiFkbETRHx3vx3w4FXm6yzqlgmSZIk1YwOK9kR8VBELG7mz0zgnemAsMBlYA1zbhu2fGxFPRsST69evr3J6SZIkqe067JzszJyxM+tFxI+Ae4q7q4GRTR4eUSxrbvs3nADcATJ06NdueVJIkSaqusmYX2a/J3b8EFhe37wZOi4jdImIMMA5Y0Nn5JEmSpPYoa3aRqyNiMpDAnCuCLAJn5XETcBiwB6oEvO7OIJEmSak0pJTszZ33AY1cAV3RiHEmSJKmqutoUfpIkSVLNs2RLkiRJnVWbJliRJkqrMki1JkiRVWWTW/hTTEbEeeKWk3Q8F3ihp392FY9h+jmH7OYbt5xhWh+PYfo5h+zmGnLRuVmXvvaKVuUbLLFBFPZubUsnPUMsew/RzD9nMM288xrA7Hsf0cw/ZzDNvP00UkSZKkKrNkS5IknSVVmyW6/G8oO0A04hu3nGLafY9h+jmF1OI7t5xi2n2PYTp6TLUmSJFWZR7IlSZKkKrNkt0NE9IyInpyPinrKz1KKI2DMibo+IFyLi+Yg4suxMtSgizo+I5yJicUTcEhF9y87U1UXETRGxLiIWN1k2JCIenjIilxdfBZWbs6loYw2uKf88LI+KOiNizzIxdXXNj2OSxb0RERsTQMrLVipbGMCK+UnwvPhcRV5eVnr1a08O95ckQ8ERHPRMSTEXFEmRlrkSW7fb4GPF92iBr2A+C+zDwIOAzHstUiYjjwVWBqZh4C9AROnKzdVTZgDHL/dsm8C8zJzHDCvuK+WzeH9Y/ggcEhmHgr8AZjd2aFqzBzeP4ZExEjgWGBlZweqQXPYnbgwj4pPATOCwzJwIfK+EXLVmDu//XrwauCwzJwOXFPfVCpbsNoqIEcAJwI1lZ6lFETEIOBr4V4DMn3JaZb5abqmb1AvpFRC+gP/BayXm6vMx8BNiw3eKZwI+L2z8GPtepoWpMc2OYmQ9kZn1x9wlgRKcHnqyEtfB8CXAdcCPihqR1oYQy/BFyVmXXFOus6PViNaWEcE9ijuD0If7a0miW77b5P5T/BxrKD1KgxnwHrg5uKUmxsjYkDZoWpNZq6mcpRmJbAG2JiZD5Sbqmbtk5lrittrgX3KDNMN/HfgF2WHqDURMRNYnnZnPlp2lho0HPh4R8yPi4YiYVnagGvV14JqIeJXKzxnfmWolS3YbRMSJwLrM/H3ZWWpYL+Bw4PrMn/AiwBd+eb7XivOGZVH5pGQYMiIj/Wm6q2peVaZc8ithGEXExUA/8rOwstSQi+gMXUXlrXm3XCxgCnTAcuAG6LiCg3Uk36EnB+Zo4Ezqd451k7z5LdNn8BnBwRK4BbgU9FxE/LjVRzVgGrMnN+cf92KqVbnrTMDWJ6Z6zPzXWAu8LGSM9Wq1yNiP4Diq28xt0FEnAmcCJyezhHbWgdQ+YX52eLnywjgqYjYt9RUntWcVMDcrFlB5x9kPkLbeGVR+pgD8H8APPraSJbsNMnN2Zo7IzNFUPmT2y8z06GErZOZa4NWImFAsn+jSwpMRItWolMD0i+hdHaj6NHyBtq7up/FCh+HpXiVlqUkQcT+U0upMz8+2y89SazFyUmR/KzNHFnz5dVwOHF/5faeXcCnwSIiPFAH+CNUhPVpteATxS3PwUsLTFLTepVdgDt0r4C/Cwi+gAvA2eVnKfmnZOb8iLgdeIrK2/NP41W6digibgGOAYZGxCrg28BVVN5WPht4Bfh8eQm7vhbGcDawG/Bg8e78E5l5nXmkhu7jmxjAzfUu+FVr4PrwJuKmYjm4bcIbvqnywFsbxHOAHxYfqtwLnlpewNnnFR0mSJKnKPF1EnkiRJqjJLtiRJklRllmxJkiSpyizZkiRJUpVZsiVJkqQqs2RL2iVFREbEtU3u/21EXFqlbc+JiL+qnxrZ2sJ9TI+L5iPjVdsuHFVM7tmZbZ0bEP7Uxxz9ExIy2PLe9dmbfEXFMRHiRJkmdynmyJe2q6oBTnIuI7mdllLlQREb0ys34nVz8bOCczH2u6MDNfAzq85DfZX2mXAd/JfR8DbAZ+27FpJOnPPJItaVdVnT+XCPedv/8D2R6IjYnPx9ZiIeDgi7oqIlyPiqog4PSIWRMSiiDigyWZmRMSTEfGHiDixeH7PiLgmnIn4XEQsj4otNtvtoRNxNM1c+jYi/Lra/OCK+Wyy7BDgK+NeIuGa79UcXF+J47wj13Ii4LyKWRsTVnTdY7q8i3APiLHb3+4vbfFVmejYirtl8/IlZExGUR8VSx3kHF8r0j4sGIeC4iboyIVyLifZe6jojNnEXFdsd68iNi7WD45Ip4oxu2OiBi8M/uOiNHAecD5EfFMRHy8eAdgcfEaHtk+gyRVgyVb0q7sfwGnnR8SgVjznMCql7cPALGB8Zh4B3EjlKqbvGQ0cAZwA/EtE9KVy5HljZk4DpgHnRMSYYv3Dga9l5vimnO4uIYcB3qVzWeDIwLSI+l5n/ADwJnJ6ZF+wg82TgC8Ak4AsRMTIi9gMuo1KujwIO3tELj4jPADOBnj2bmYcDVLaz6RmYeDlwP/G2x7NvALzNzInA7sH8Lzx0APFms93DxPICfAH+XmYcCi5os/8B9Z+YKn4F+A6zJzcmY+ClwCHFe8hpN39LolqS0s2ZJ2WZm5iUp5+2ornva7zFyTmXXAMuCBYvkiKsX6PbdlnZmNmLgVeBg4CjgX+W0Q8A8wH9gLGFesvyMzlzexvGvDrzFxfnEbyM+DoVuQFmJeZGzNzK5Uj5aOAnjzbZ7jbg33diOzOAmzPzbYDM3NDCenOLr7/nz2NyFHBr8bz7gD+18NzGJll+ChxV/BK0Z2Y+XCz/nMS2PQXP73t5vgDkRcQ7Qs4V1JKldLNmSdnXfp3KEeUCTZfUU/z9GRA+gT5PH6prcbmxyv5H//DmXn3G4/CQTwleKI6uTMHJOZ75X0Le16FR+saeYGdvx5nA96/a3Z387sa0e2H8d27zszzwP+HhgJ/D4in9mp7PElqniVb0i6tOBp7G5Wi/Z4VwJTi9slA7zZs+tSI6FGcpz0WeBG4H/hSRPQGiIjxETHggzYCnLAA+ERFDI6In8NdUTqNor/nFdvcq8pza5LEVNP/6HwTOioj+Rf4hrdjfb4DPF887Fhjcwno9+POHnNv8GeCwzNwJ/ioiPF8tn0boxeAsY+N6diDggM+cXH5pcT6VsS1JVObuIJMG1wP9ocv9HwF0R8SxwnH207yrySSkHeAzgvM7dGxI1UTmF4KiKCSsH73AdtJDPXRMQ3gV9RORJ+b2be1YY8zW33UuBx4E3gnmSYPN/v6M/O+iJgMPBkR24CfAxft5C4vA26JiFnFPtdSKb/b2wIcERF/D6yjci45wBlUzm3vT+X0nm7N29rUC/wHcHhEzqZw3f35EjKMynvOAZ1uxLUnaKZHZ2nfiJElqnYjYDWjIzPqIOBK4PjMnN7Pen5szcvfMTSlJ1eSRbktQZ9gduK87x3gacU3IeSepQHsmWJEmSqswPPkqSJElVZsmWJEmSqsySLUmSnJFWZJVuSJEmqMku2JEmSVGWWbEmSJKnK/h+0iqkn5iqqNwAAAABJRU5ErkJggg==n”, “text/plain”: [

“<matplotlib.figure.Figure at 0x12fd50710>”

]

}, “metadata”: {}, “output_type”: “display_data”

}

], “source”: [

“plt.plot(Ms, vfe_lml, label=”lower”)n”, “plt.plot(Ms, vupper_lml, label=”upper”)n”, “plt.axhline(full_lml, label=”full”, alpha=0.3)n”, “plt.xlabel(“Number of inducing points”)n”, “plt.ylabel(“LML estimate”)n”, “plt.legend()n”, “plt.title(“LML bounds for models trained with SGPR”)”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“We see that the lower bound increases as more inducing points are added. Note that the upper bound does _not_ monotonically decrease! This is because as we train the sparse model, we also get better estimates of the hyperparameters. The upper bound will be different for this different setting of the hyperparameters, and is sometimes looser. The upper bound also converges to the true lml slower than the lower bound.”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Upper bounds for fixed hyperparametersn”, “Here, we train sparse models with the hyperparameters fixed to the optimal value found previously.”

]

}, {

“cell_type”: “code”, “execution_count”: 16, “metadata”: {}, “outputs”: [

{

“name”: “stdout”, “output_type”: “stream”, “text”: [

INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 62.680716n”, ” Number of iterations: 9n”, ” Number of functions evaluations: 16n”, “3 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 60.882546n”, ” Number of iterations: 13n”, ” Number of functions evaluations: 17n”, “4 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.697017n”, ” Number of iterations: 9n”, ” Number of functions evaluations: 11n”, “5 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.065631n”, ” Number of iterations: 10n”, ” Number of functions evaluations: 11n”, “6 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.053159n”, ” Number of iterations: 12n”, ” Number of functions evaluations: 14n”, “7 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023861n”, ” Number of iterations: 19n”, ” Number of functions evaluations: 22n”, “8 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023374n”, ” Number of iterations: 21n”, ” Number of functions evaluations: 24n”, “9 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023338n”, ” Number of iterations: 20n”, ” Number of functions evaluations: 24n”, “10 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023335n”, ” Number of iterations: 2n”, ” Number of functions evaluations: 5n”, “11 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023334n”, ” Number of iterations: 1n”, ” Number of functions evaluations: 3n”, “12 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023330n”, ” Number of iterations: 5n”, ” Number of functions evaluations: 8n”, “13 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023329n”, ” Number of iterations: 1n”, ” Number of functions evaluations: 3n”, “14 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023328n”, ” Number of iterations: 1n”, ” Number of functions evaluations: 4n”, “15 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 59.023326n”, ” Number of iterations: 1n”, ” Number of functions evaluations: 3n”, “16 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023322n”, ” Number of iterations: 3n”, ” Number of functions evaluations: 6n”, “17 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023321n”, ” Number of iterations: 0n”, ” Number of functions evaluations: 1n”, “18 INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL’n”, ” Objective function value: 59.023321n”, ” Number of iterations: 0n”, ” Number of functions evaluations: 1n”, “19 “

]

}

], “source”: [

“fMs = np.arange(3, 20, 1)n”, “fvfe_lml = [] # Fixed vfe lmln”, “fvupper_lml = [] # Fixed upper lmln”, “for M in fMs:n”, ” Zinit = vfe.feature.Z.read_value()[:M, :].copy()n”, ” Zinit = np.vstack((Zinit, X[np.random.permutation(len(X))[:(M - len(Zinit))], :].copy()))n”, ” init_params = {p.full_name:p.read_value() for p in vfe.parameters} # TODO: provide convenience functionn”, ” init_params[‘SGPR/feature/Z’] = Zinitn”, ” vfe = gpflow.models.SGPR(X, Y, gpflow.kernels.RBF(1), Zinit)n”, ” vfe.assign(init_params)n”, ” vfe.kern.variance.set_trainable(False)n”, ” vfe.kern.lengthscales.set_trainable(False)n”, ” vfe.likelihood.set_trainable(False)n”, ” vfe.compile()n”, ” gpflow.train.ScipyOptimizer().minimize(vfe, disp=False)n”, ” n”, ” fvfe_lml.append(vfe.compute_log_likelihood())n”, ” fvupper_lml.append(vfe.compute_upper_bound())n”, ” print(“%i” % M, end=” “)”

]

}, {

“cell_type”: “code”, “execution_count”: 17, “metadata”: {}, “outputs”: [

{
“data”: {
“text/plain”: [
“<matplotlib.legend.Legend at 0x146fe2f28>”

]

}, “execution_count”: 17, “metadata”: {}, “output_type”: “execute_result”

}, {

“data”: {

“image/png”: “iVBORw0KGgoAAAANSUhEUgAAAtkAAAF3CAYAAABws1edAAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xucl3Wd///Hi5PDSQQRkJOgogmoaAPaAfKU1laSW222nlu3appZbbdvhm7lb1ua3Nm2zvm25lq72yzZd0zRrXdLNU6siICKKhQcUEOUkIHJmXr8/Ph9gwBmYnGa6Za2Z43G+3z20+1/t6X9f1+lwRPnnP+/O+IjORJEmSVJwuZRcgSZIkdTaGbEmSJKlghmxJkiSpnYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWDdyi6gCAMHnDsxRo0aVXYYkSZI6uZkzZy7PzIP21K9ThOxRo0YxY8aMssuQJElSJxcRzzeln9NFJEmSpIIZsiVJnkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSnpIIZsiVJkqSCdSu7gCKs27SFmc+vLLsMSZIkCXAkW5IkSSpcpxjJ7tWjG288ZEDZZUiSJEmAI9mSnJElS4QzZkiRJUsFKCdkRcXlEPBURcyLi1og4oNo+KiLWR8Ts6uuqMuqTJEmS9kZZI9m/A8Zn5jHAnn4CL6+17JjMnVF8XllOeJEmS1HKlhOzMnJaZW6qbDwHDy6hDkiRJag3tYU72ecB/1dseXZ0qcm9EnTC6rKEmSJKmlWm0Jv4i4CxjSwK5LMvO2ap9LgC3ADdV9S4CRmbkiIt4I/CoixmXmmgbOfz5wPsDInkSNb4yNIkiRJLdJqITszT9vd/oj4K+DdwKmZmdVjNgIbq+9nRsQzwBHAjAbOfzVwNUBtbW0WWrwknSZK0F8paXeQdwBeBMzNzXb32gyKia/X9ocAY4NkyapQkSZJaqqwnPv4A2A/4XUQAPFRdSWQK8PWIn2AzUARdm5sqSapQkSZJapJSQnZmHN9L+S+CXbVyOJEmSVKj2sLqIJEmS1KkYsiVJkqSCGbIlSZKknghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCnGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZnsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiVJkqSCGbIlSZKkghmynJUmSpIIZsiVJkqSCGbIlSZKkgpUSsiPinyJiTkTMjohpETG03r6LI+LpiPhjRJxRRn2SJEnS3ihrnJPvyzDwmMycAdwBfAYiIscDZwDjgHcAPI6JrSTVKkiRJLVJKyM7MNfU2ewNZfT8V+EVmbszM54CnngUltXZ8kSZK0N7qVdeGIuAw4F1gNnFxtHgY8VK/bomqbJEmS1GG02kh2RNwVEXMbeE0FyMxLMnMEncAPwty04//kRMSMiZixbtqzo8iVJkqQWa7WR7Mw8rYldbwB+C3wVWAyMqLdveLWtofNfDVwNUFtbnmw31kSRJkspQ1uoiY+ptTgWeqr6/HTg7IvaLiNHAGGB6W9cnSZIk7Y2y5mR/KyKOBOqA54ELATLzniYi4CXgS2AJclJlbS6pRkiRJapFSQnZmvm83+y4DLmvDciRJkqRC+cRHSZIkqWCGbEmSJKlghmxJnkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSnJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIknqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpYIZsSZIkqWCGbEmSJKlghmxJkiSpnYIZsSZIkqWCGbEmSJKlgpYTsiPiniJgTEbMjYlpEDK22j4qI9dX22RFxVRn1SZIkSXujrJHsyzPznmMycANwBfKXevmcyc0L1dWFJ9UmSJEktVkrIzsw19TZ7A1lGHZIkSVJr6FbWhSPiMuBcYDVwcr1dnoyNidrX9HzLz/jLqkyRJklqq1UayI+KuiJjbwGsqQGZekpkjgBuAv60etgQYWZ1G8vfAzyNi/0bOnf35EzIiIGcuWLWutjyFJkiQ1W2SWO1MjIkYCv83M8Q3suwf4fGbO2N05amtrc8aM3XaRJEmS9lpEnzMzM2j31K2t1kTH1NqcCT1XbD4qIrtX3hwJjgGfbvkJJkiSp5cqak/2tiDgSqAOeB7atIjIF+HpEnbK7uuzAzV5ZUoyRJktQipYTszHxfI+2/BH7ZxuVIkiRJhfKJj5IkSVLBDNmSJElSwQzZkiRJUsEMn2ZIkSVLBDNmSJElSwQzZkiRJUsEM2ZIkSVLBDNmSJElSwQzZkiRJUsEM2ZIkSVLBDNmSJElSwQzZnkiRJUsEM2ZIkSVLBDNmSJElSwQzZkiRJUsEM2ZIkSVLBDNmSJElSwQzZkiRJUsEM2ZIkSVLBDNmSnJElSwQzZkiRJUsEM2ZIkSVLBDNmSJElSwQzZkiRJUsEM2ZIkSVLBDNmSJElSwZocsiOiV2sWIkmSnJHUWewzZEfHmiHgSeKq6fWxE/LDVK5MkSZI6qKaMZH8XOANYAZCZjwFTWrMoSZIkqSNr0nSRzFy4nS9PWIi4eEZ+LiIyIgfXaLo6IpyPijxFxRhHXkSRJktpStyb0WRgRbwYyIroDnwHm7e2FI2IEcDrwnQr22scDZwDhgKHBXRByRmYWEekmSJKktNGUk+0LgImAYsBiYAHyygGt/F/gikPXapgK/yMyNmfkcn8DQwqYBrSZIkSW2mKSPZR2bmOfUbIuItwB9aetGImAoszszHIqL+rmHAQ/W2F1XbJEmSpA6jKSH7n/wHHN6FtJxFxFzCkgV2XAF+mMlWkxSLifOB8gJEjR+7NqSRJkqRCNRqyI+JNwJuBgyLi7+vt2h/onuqcTZ+ZpjZz3aGA0sG0UezgwKyImUZmOMqJe9+HVtobOfzVwNUBtbW021EeSJEkqw+5GsnsAfap9n+tZrXwO8v6UXzMzHgUHbtiNiAVCbmcsj4nbg5xHxL1S++DgGmN7Sa0mSJKllNm/ezKJFi9iwYUPZnpZSipqaG4cOH07179xYd32jIzsx7gXsj4rrMfL6lBTZHZj4RETcBTwJbgItcWUSSJKntLVq0iL59n+zJq1Ch2+Q5dp5eZrFixgkWLFjF69OgWnaMpc7LXRcTlVJbVq6l38VNadMVdZOaoXbYvAy4r4tySnJElqmQ0bNuyTARsgIjjwwANZtmxZi8/RlCX8bqDySPXRwNeABcAjLb6iJEmSOoR9MWBvs7efvSkhn+8DMvAbYnJn3ZuZ5QCGj2JIkSVJj+vTpU3YJLdaU6SKbqz+XRMS7gBeBAa1XkiRJktR2tm7dSteune1w8r1maMpL9jYjoB3wO+DzwE+CzhVYhSZIkNSIz+cIXvsD48eM5+uijufHGGwG46KKLuP322wE4n66yzOO+88wC49tprueSSSwD42c9+xqRJk5gwYQIXXHABW7dW1tTo06cPn/vc5zj22GN58MEHC695njyPZmXlH9e1q4OTCK5AkSVK79rVfP8GTL64p9Jxjh+7PV98zrkl9b7nlFmbPns1jjz3G8uXLmThxnIlOmTGHy5Mncf//9nHnmmSxevJglS5YAcP/993P22Wczb948brzxRv7whz/QvXt3PvnJT3LDDTdwn7rnn8tprr3HCCSfwne98p9DPtc0eQ3ZEjAY+BYyq3z8zz2yViiRJkqR6HnjgAT70oQ/RtWtXBg8enzNve9jYeeeQRJk+ezJVXXsmTTz7J2LFjeeWVV1iyZAkPPvgg3//+97n++uuZOXMmEydOBGD9+vUMnGlR5XEvXrl153/ve12o1N2VO9q+Aa4BfA3WtVokkSZLapaaOOLe1YcOGsWrVKu68806mTJnCypUrnuemmm+jTpw99+/YlM/noRz/KN7/5zdcdW1NTU/g87PqaMid7Q2Z+PzN/X11d5N7qg2okSZKkVjd5n8mRuvPFGtm7dyrJly7jvvvuYNGkSACeeeCJXXnnl9ukjV1xxBZMnTwbg1FNP5eabb2bp0qUArFy5nkuefb5NnLDZpJPt7EfFVYBqwcVtjZs5qtaokSZKkqrPOOosHH3yQY489lojg29/+NkOGDAEqAXzantGkcfvjhHHLIIaxcuXJ7yB47dizf+MY3OP3006mrq6N79+7867/+K4ccckir1xyZufsOEd8EPgI8nw47pIlnUEx+LUFtbmzNmzCi7DEmSpE5j3rx5HHXUUWWXUaqG7kFEzMzM2j0d25SR7A8Ah2bmphbWnJ0mSJO1TmjIney5wQGsXIkmSJHUWTRnJPgB4KiIeYec52S7hJ0mSJDWgKSH7q61ehSRJktSJNOWJnjy7XJ0mSJDVDoyE7Ih7IzLdGxKtA/SVIgsrqIvu3enWSJElSB9RoyM7Mt1Z/9m27ciRJkqSOb4+rni0TE/9eUNkmSJKmjykzq6ur23LGJmrKE304Pq4+IbsAbC6tAkiRJasCCBQsYP3789u0rrriCSy+9nlJNOOonPfOYzTJgwgfHjxzN9+nQALr30Uj7ykY/wpje9iTFjxvDjH/94+7GXX345EydO5JhjjuGrnX/3q9vMfeeSRnHvuuYwfP56FCxcWVvvu5mRfDHwZ6BkRa7Y1A5uAqwurQJIkSe3bf30JXnq82HMOnORre+a0WH75u3Tpmz57Nfffdx3nnncfcuXMBmDNnDg899BCvvfYaxx13HO9617uYO3cu8+fPZ/r0n6WQmZ555Jvfddx8jR45k/vz5XH/99Zx44olFfTJg93Oyvwl8MyK+mZkXF3pVSZIkaS986EMfAmDKnlCmsWbOGVatWATB16lR69uxJz549Ofnkk5k+fToPPPAA06ZN47jjjgNg7dq1zJ8/n5EjR3LIIYcUnHrChaetk3xERvTPztYj4MHA88L3MfL7waiRJktT+7MWI897o1q3bTvOkN2zYsP19ROzUd9t2Q+2ZnycUXX8wFF1yw074FCxbQu3fvossGmjYn+0fAuog4Fvgc8Azw01apRpIkSaoaPHgwS5cuZcWKFWzcnuJE77rhj+74bb7wRgAceeIB+/frRr18/AG677TY2bNjAihUruOeee5g4cSJnnHEG1157LWvXrgVgn8eLFLF26tFVrb8pI9pbMzIiYCvwgM6+JiI+1alWSJEna53Xv3p2vfOUrTJo0iWHDhvGGN7xh+76anmhqOO+44Nm/ezLXXXru9/ZhjjuHkk09m+fLl/OM//iNDhw5l6NChzJs3jze96U0A9OnTh5/97Gd0n7dq11WqPzNx9h4h7gTuB84DJwFLgscw8utWqaqba2tqcMWNG2WVIkiR1GvPmzeOoo44qu4wGnXTSnSVxxxRXU1tbu1H7ppZfSp08fPv/5zxdynYbuQUTMzMzaRg7ZrinTRT4IbATOy8yXgOHA5S0pVJIknSdoX7HG6SGa+FBG/BMZUm5YDt7ZqVZIkSVIj7rnnngbbL7300jatY3ea8sTHjwM3A/9WbRoG/Ko1ni5IkSZI6sqZMF7kIeAuwBiAz5wODWrMoSZIkqSNrSsjemJmbtm1UH6u++29LSpIkSfuwpoTseyNin2+PV3w78J/DrIi4eEZ+LiIyIgdXtURGxPiJmV19XFXEdSZIkqS01JWR/CVgGPA5cAPwW+Ie9vXBEnjABOB17YZdczmTmh+rpwb68jSZKkjun73/8+Rx11FOecc06jffr06QNUnt44fvz4tiptj5qyukgdn8OPqq0jfBb4I3FbweSVJktQJ/PCHP+Suu+5i+PDhZZfSbE0ZyS5c9emRizPzsQZ2j65OFbk3Iia3ndW2SJEkq34UXXsizzz7LO9/5Tvr168cVV1yxfd/48eNZsGBBecU1QVMeq94iEXEXMKSBXZcAX6YynVWRXS4CRmbkiIt4I/CoixmXmmgbOfz5wPsDIkSOLK1ySJEk7+dPLr/Lqhs2FnrNvTXeOGNy30f1XnXXUVd955J7///e/5wQ9+UOi120KLRrIj4oo99cnM0zJz/K4v4FlgNPBYRCyg8gTJWRExJDM3ZuaKn6vEzgWeAIxo5/9WZWZuZtQcddFBLPoYkSZLUKlo6kv0XQIseCp+Zj1Nvne1q0K7NzOURcRCwMjO3nRsShVJ4y+WwLa5QkSVIBdjfi3Ba6detGXV3d9u0NGzaUWE3TtHROdhRaxQ5TgDkRMZvKUyYvzMyVnrXQtSZIkdQCjRo1i1qxZAMyaNYvnnnuu5Ir2rNGR7IgY0NguCgzZmTmq3vtfAr8s6tySJEnq+N73nvvfx05/+lHHjxnHCCSdwxBENziZuV3Y3XWQmlSc7NhSoi535LkmSJO2i/goi06ZNa7DP2rVrgcpon99y5c9uirCZpNGRn5ui2LESSJEnqLHY3XeT43R2YmbOKL0eSJEnq+HY3XWQGMBdYXt2uP20kgVNanqyhJkiSpI9tdyP574P3AeuAXwK2ZubZNqpIkSVLpMpOI1lpUrn3LzL06vtEl/DLzysx8K/ApYARwnd0TcFBET9uqKkiRJavdqampYsWLFXofNjigzWbFiBTU1NS0+xx4fRpOZz0bEbUBP4CNUnsA4u8VXnlCRJUrs3fPhwFi1axLJly8oupRQ1NTUMHz68xcfv7ouPhwJnA1OBhVSmjPzfzFzf4qtJkiSpQ+jenvTujR7vYXEvtbiT7aWAOcBuwBhgJfGLbvJzM/JdWr06SJEnqgHYXsr9OZRURgD677Nv3JudIkiRJnTbS7h9Fc2ti+iPi7VqlGkiRJ6gQaXV1kD/6+0CokSZKkTqSlIXvfXDBRkiRJaoKWhmznZEuSJEmNn2N0Sfq/ScJgOKmtmS5IkSWrA7r742LctC5EkSZI6i5ZOF5EkSZLUCEO2JEmSVDBDtiRJklQwQ7YknSZJUMEO2JEmSVDBDtiRJklQwQ7YkSZJUMEO2JEmSVDBDtiRJklQwQ7YkSZJUMEO2JEmSVDBDtiRJnklQwQ7YkSZJUsFJCdkRcGhGLI2J29fVn9fZdHBFPR8QfI+KMMuqTJEmS9ka3Eq/93cy8on5DRIwFnzgbGAUOBuyLiiMzcWkaBkiRJUku0t+kiU4FfZObGzHwOeBqYVHJNkiRJUrOUGbI/FRFzIuLaiOhfnbRsGLKzXZ1G1TZIkSeowWi1kR8RdETG3gddU4EfAocAEYAnwnRac//yImBERM5YtW1Zw9ZIkSVLLntdqc7Mw8rSn9IuLHwB3VzcXAiHq7h1fbGjr/1cDVALW1tdnySiVJkqRilbW6yMH1Ns8C5lbf3w6cnHRH7RcRoYAwwva3rkyRJkvZGWauLfDsiJgAJLAAuAMjMJyLiJuBJYAtwkSuLSJIkqaMpJWRn5kd2ns+8y4LI2LEeSJEkqVHtbwk+SJEnq8AzZkiRJUsEM2ZIkSVLBDNmSJElSwQzZe2PGv8OiGVDnAiiSnJEnaoawl/Dq+DWvgt1+Aus3Q60A4/DQYczocdgr0GlB2dZIkSSqRIbulavaHz/8JnvkfmD8Nnr4Ln5twI0QWGT4Qxb6+E7iHHQETZ1UqSJKkNRWbHfyJ5bW1tzpgxo9wi6rbCi49WAvf838GLsyrtfYZUnR7nfDoedDDX9yq1TkiRJLRYRMzOzdo/9DNmtZO1SePruSuh+5m7YsBq6dIMRJ+4Y5R50lKPckiRJnHYghuz3ZugUWPbJjlPvlxyvt+w/fEbhHT4H9+pRbpyRJknbLkN2erXmxErbnT4Nn74VNr0LXHnDInWyqBe8zpcOBhjnJLkiS1M4bsjmLLJlj40I5R7mVPVdr7j94xyj3qrdC9Z7l1SpIkyZDdYb3yPDz9nu0rgfvZe2LIeutVUppOMOb0SvPuPKrtKSZKkfVJTQ7ZL+LU3/Q+BiX9TeW3eAM8/sGNqyfxplT4Dnj9gRuEe+Gbr1KLdmSZIk7cSR7I5kxTM7AveCB2DrRujRBw49qRK4D3879BtWdpWSJEmdliPZndGBnh1VeJ14Im16D5+7fMcL91B2VPoPGVQL30R+AIePLrVeSJGkf5Uh2Z5AJy/64I3C/8CDUbYFRk+GEnC+DIP4MuXcuuUpIkqcPzi4/7snUr4dGfwfQfw+oXoN9ImPQ3cPy50LN/2dVJkiR1WIZsVR71/sffnwsP/Bgvuh+694JgPVka3Bx1VdnWSJEkdjnOyVZkictR7Kq+X5sLDV8Fj/wEz/x1Gvw1O/ERllRKnnkkiSJBXKkex9zWsrYNb18MhPYM3iyprbk86HCedAzwPKrk6SJKldc7qIdm/rFnjq15WpJC88CN17nw4S/rATug44ouzpJkqR2yeki2r2u3WDcWZXXi7MrYXvW9fDIj+GwUytTSQ47Fbp0KbtSSZKkDseRnbO2wdhnMvK4ylWTtSzDgsMqXJI/9ENTsX3Z1kiRJpXO6iFpuyyaYd3vli5KLHoEefeG4cypTSQ48nrOzqJEmSSmPIVjEWzYTp/wZzb6k84GbM6ZXR7cNOgYiyq5MkSWpThmwV69WXYMa/w4xr4LVlMPCInStg+5mzYr0/Z1UmSJLUJQ7Zax5aN8MSt8NCPYMls2K8fHP8RmPg3MGB02dVJkiS1KkO2WldmZb72nw1fBk7dVni555DvhhAth9BSnkkiSpE6pXS/hFxGXAh8HllWbvpyZv42IUcA84I/V9ocy88I2L1B7nFgEjJlVea16ER66pPEnyj7+Fg46qTiX5IPToVXalkiRJba6UkexqyF6bmVfs0j4KuCMzxzfnfI5kntxObN8DcX8LDP4KXHoeaA+CNH61MJTlgZNnVSZIk7bWmjmT7pBEVp3tNZam/C+6Hv/4vOPRt8L//nD753LNz4YVjwQGWaiSRJUidXZsj+VETMiYhrI6J/vfbRETE7Iu6NiMmlVaeWi4BD3gx/8VP4zBx4n86crAfu6d8FPToWn7zZsS5KkTq3VpotExF3AkAZ2XQI8BCwHEvgn4ODMPC8i9gP6ZOaKiHgj8CtgnXGauaeD85wPnA4wcOfKNzz//fKt8DhVk0zqY8wu4/19g9UIY+WY45R9g1FvKrkySJKnJOszqIrubnhx0R9wCfz8zdTrh2TnYHsmUjzPop3Hc5rH0ZDj25EraH7/HPqiRJUuna9ZzsiDi43uZZwNxq+0ERn0bX6/lBgDPBs21eoVtNtP5j0cfj0bDj9G/DSnMoUkp9/EJY8VnZ1kiRJhShrTva3I+LxiJgDnAx8ntto+BZgTEbOBm4ELM3NlSTWqNfXoBW/+FHzmMTjlH+GFB+HfpsBN58LSp8quTpIkaa+UPl2kCE4Xn6QTWr4IH/xUe+iFseg2O/gCc9CU48LCyK5MkSdquXU8XkV6n5wFwyiWV1Uje8mmY92v4wUS47W9hn1QtlVydJktQshmy1L70PhLd/vTKNZNLHYc6N8P3j4TefhzVLyq5OkiSpSQzZap/6DoZ3/jN8+lE4n7sOVR7Z/fwL89yWwdlnZ1UmSJO2WIVvtW7/h8J4r4W9nwLg/r8zZ/t6xcPfXYZ3fiZUkSe2TIVsdnw4DRcNaP4KLpcOQ74P7vVML2Pf8MG173rCJJkqRSGbLVsQwcA++/Fj7xvzB6Ctzzf+F7x8AD362snSiJJktQOGLLVMQ0eB2ffAB//PQyrhbsurYxsP/Qj2Lyh7OokSdI+zpCtjm3Y8fDhm+G8aTDoKLjznS/D94+CRa2DLprKrkyRJ+yhDtjqHkSfAR38N594OB4yA3/w9/OCN8OgNsHVL2dVJkqR9jCFbncuhnb4Pz/hvOuRl6DoDbPgk/PAEevxnq6squTpIk7SMM2ep8ImDM2+H8e+CDN0DXHvDLj8FVb6k8STKzn7AolSVInZ8hW5xUBR70bLvwDvO8a2LoZbvwwXH0S/GmaYVuSJLUaQ7Y6vy5d4Oj3wycfgvf+CNa/nAj//AFxzOjx7b9nVSZKkTqhb2QVIbaZrN5jwlzD+/TD7Z3DfFfDTM2HUZDjmg5WnS/YbAf2GQfeenZVcrSZI6sMhO8Cvz2tranDFjRtllqKPZvAFmXld5euRrS3fe1+vASujef3g1fA+vhO9+Iyrv+wyGnLl1LKVuSJJUnImZmZu2e+jmSrX1X9xo48UKY+DewZhGsXgSrF8PqhZX3axbDK8/Bgvth4y6Pbu/SnDfoOrQbvagjfv14I7zcMag6ozAuXJEn7HEO21LUb9B9VeTVmw+pqAF+0SyBfBAunwxO/grrNOx/Tno0+98D1859f+wyqv7jWt+ckkSVJJDNlSU9T0q7wGj214f11dZcrJ6kU7v7YF8pcef/2UFIDeg3aMnhu8//PVhvPegyhc3JUlSh2LIlorQpQv0HVJ5DW9kmtbmDZUpKGsWvz6ML/sTPP0/sPm1nY/p0QcOnekMl3A8at+Nn7wNb/zNJkqQWM2RLbaV7DRx4WOXVkEzYsGrHNJTVC2H5fFj6JMy7A2b9dEffPoNhn0NjKa3D150FvgB692uazSJKk3TJkS+1FBPTsX3kNGb/zvkxYuxSWPgEvP1kJ3i8/ATOugS0btp0AnBhz6+lHvAaNdCUWSpDZmyJY6ggjoO7jyOuyUHe11W2Hlc/XCd/XnvDuA6vKc3WrgoCPrBe+xMHhcnZTTc1U8kSWoVhmypI+vSFQYeXnmNnbqjfdM6WP7HnUe9n7kbHvv5jj49B1TC9qCxMOio6vujYL++nbf85JEnqZAzZUmfUoxcMPa7yqu+1FTtPOVn6JMy+ATat3dHngJE7j3oPGgsDx0DX7m37GSRJ6sAMn2dK+pPeBMHpK5bVNXR2sfmHn6SZLn4T50yC3Vvp06Q4Dj9gRvPseXBlFjy7Vn113+dmlkX1ddt7en3mc359l+vl33OdVFktR+GbK1T1q3aQuLXlnPolfWseiV9azduIXMHfuzupG5fWZz9f2Odqju29Z3np/bdH1//x+6ute18dVl5X5e5/Zjt2wl19fpnZnW7sp/cw/FUj88DyZxCMpm67tB1yEaGbF7EyM0LnGLH5OUY1vrw8AAARaUlEQVSuWsDI5fdw0OP/ube3vxB1BHV02f7K6EIdQVIJ31nvPUTlcze2HVHtnT739O7Yr/Ro6R71jGj3Htnraq+b9YyWb2b9yTEvs+TpNraVJ12/iP9pa8vn3Zd4vtaYu77mSw499nS9llNMqQrU5pw+atLF61nkWvrGfhykqQXlgN1ItWrmPFa5sKvd62/z4HENWNqLcv2NEhXndMNH58nQJeo7O8SsX1/ZbtybJfY0Va/X5fKhXfajm19qQ4qV48nquep129r9GBh90NZ3OMwHqpXR6+6tfTNntXTJrZV4m5WYG9TtaKvX3oWtRGa99q31+u/os+N9HVHv/Y7z1+vTwHGVOLsjZu8UkbOx+Fx/u9IWnWT1m23a9nzsdl/WO2SVKN3SNdimbV1dL4lLLPvuej2nqeQuM4e33f8f2qpl/vqTm2r97j7JL2C1DntjqkzVvrWLJqQzU4r2Physqo9MLq6PTLazbu1L9712DYAT0ZMaAXp48bzPD+vRgxoBfD+/dkeP+en7F9TmW/cpOBb7RdNHPmSJEn7HkO22qWtdclLazawaOWO4LxwZWU0evEr61myej119QZJunYJDu5Xnw/D+PZky5qDtAXrbz8F9a+jSxVAsSZLahiFbpairS5at3bjzKPTK9SxaVfn54qr1bKmXoiNgcN8anRgzoyaTRAxjRvyfDtwXp/r04uF8N3bp2KfETSZIk7VBayI6ITwEXAVuB32TmF6vtFwMfq7Z/OjP/nu6waVYxlr27kkQUreWTBSp5Z9tr2Lxtu2lK3U7+BffZjeP+eHDviAN59zMHVKR09Gd6/F0MPqGG/nbj61UJIkdQylhOyIOBmYChybmRsjYlC1fSxwNjAOGArcFRFHZG5bR0ztXWbywsp1TH9uZTVYv8Jzny18DoKZ7F8YM6suRg/ty2lGDK6PR1SA97IBe9OxhiJYkSZ1DWSPZnwC+lZkbATJzabV9KvCLavtznEfE0MAl4sJwytSd1dckfX36VRxas3B6st33psF/P7kwc1Z+zJ45g4ugBjB/ajx7dnNIhSZI6v7JCn9hHA5Ii4DNgAfD4zHwGGAQ/V67eo2vY6EXE+cD7AyJEjW7dabbdpSx2PL17F9Ode4ZEFK5mxYCVrnNmwB4OB+NZww+kAmjh7ApFEDGDOoj182lCRJ+6RWC9kRcRcwpIFdl1SvOwA4EZgI3BQRhzbn/Jl5nNXA1QG1trYtxtpK1G7cw6/lXto9Uz164io3VudSHHtSbPzv6YCaNHsDEUQMY3r+ny9pJkiTRiiE7nM09rbF9EfAK4JSuPupseEXXAQGAxMKJe1+HVNrWRFWs38siCHaH6ySVr2FqXdAkYN7Qf55xwCJNGn96d21AAG9tmv7HIlSZLapbKmi/wKOBn4fUQcAfQAlgO3Az+PiH+h8sXHMcD0kmrs9DKTRa+s377ynx/TnKqt/AOzXrQsTRhzAJ086jImjBnD8If3ps58rPkqSJDVFWanpWuDaiJgLbAI+Wh3VfiIibgKenBLYAF7mySHHq6pKnl63l4edW8kj1S4pLVm8AoG9NN2oP6c/73ziCSaP7M35YP5fMkyRJaqFSQnZmnbgI+3Mi+y4DL2raizmnz1jrmLl5dHaV+hRnPr2TVus0ADOq73/YvKE4cNYAjh/Slq19SlCRJKoS/n/+9kXlixjl/PeZE/PL2cR19YxfrNlV8EjB7Ym9PHDmbiqAFMGj2AkQN6+SVFSZKkVmLI7gRWrdvEnbx5fwq2zFjPj+VcAGHvw/nxw4ggmjhrAxNH9GdS3puQqJUmS9h2G7A5q45at/P6pZdz66CJ+/9QynNm2t4/BBffjCGUfy3uOGMeyAnmWXKEmStM8yZHcgmcnM51/hlkcX85s5S1i9fjMD++zHh088hD8/nfhjjhu7vFBBJkqR2wJDdATy7bC2/enQxt85ezMKV66np3oUzxg3hrOOG8dbDB9Ktq48qlyRJak8Mn2e3UirUbuWPOEm55dDGPLVxFBLzlsIH83alHcMb4Ia5ZLUmS1I6Z1NqRDZu3cte8l7l11mLu/dMynttQlbxjSly//2Rs489hhDOnnlxclSZI6AkN2yerqkoefW8mtjy7ivx5/iVc3bmHw/vvxsbeO5r3HnDeOog/cvu0RJkiQ1kyG7JPNffpVbH13MbbNfZPGq9fTu0ZV3jD+YPz9+GCceeqAPhpEkSerADNltnaOmrG7h99ov8avZi5i5eQ9cuweQxA/niO47k9LFD6NnDx5hLkiR1BobsVrZu0xamPfEytz66mPvnnL6Mu4ehh/fjKu8fynmOHclDf/couUZIkSQUzZLeCrXXJ/z6znFsfXcx/z32J1zZtZdgBPfnESYdxn1nHDOHxQ37JLlCRJUisyZBdo3pI11XnWi3l5zUb61nTjPccO5b3HDWPSqAF0cZ61JEnSPsGQvZdenWr2B22Yv5tZHF/PUS6/SrUtw0pGD+Mq7h3HqUYOo6e48a0mSpH2NIbuFVq/fzCdvmMn/PrOCTDhun5AF8feo43n3MUAb07lF2eZIkSSqRIbuF9q/pRrcuXfjUKWM467hhjB7Yu+ySJEmS1E4YslsoIrj+nvElllyFJkqR2qEvZBUiSJEmdjSFbkiRJKpghW5IkSSqYIVuSJEkqmCFbkiRJKpghW5IkSSqYIVuSnJEkqmCFbkiRJKpghW5IkSSqYIVuSJEkqmCFbkiRJKpghW5IkSSqYIVuSJEkqWGRm2TXstYhYBjxfndh3tzEBgedlFdCDer+bxfjWP96t5vF/N4/1qHu9X83i/Xu+QzDxoT506RcjW60XEjMysLbuOjsL7n1Tzer+bxfjWP96t5vF/N4/1qHu9XyzldRJIkSSqYIVuSJEkqmCG787q67AI6GO9X83i/msf71Tzenr+bxfjWP96t5vF8t5JxsSZIkqWCOZEuSJEkFM2R3QhHRNSIejYg7yq6lvYuIAyLi5oh4KiLmRcSbnyq6pPYuIz0bEExExNyL+IyJqyq6pvYmIayNiaUTMrdc2ICJ+FxHzqz/7l1lje9LI/bq8+v/JORFxna0QcUGaN7UlD96vevs9FREbEwDJqa48au18R8anqn7EnIuLbZdXX3jTy/8cJEfFQRMyOiBkRManMnGjsSQ3bn9BlgXtlFdBDfA+7MzDcAx+J9a1REDAM+DdRm5nigK3B2uVW1S9cB79il7UvA3Zk5Bri7nuq2K63j9/fodMD4zjwH+BFzc1kW1Y9fx+vtFRIwATgdeaOuC2rnr2OV+RcTJwFTg2MwcB1xRQl3tn1XW8/s/Xt4GvZeYE4CvVbTWBIbuTiYjhwLuAn5RdS3sXEf2AKcA1AJm5KTNXlVtVu9cN6BkR3YBenwIsl19PuZOZ9wMpdmqcC11ffXw+8t02Lascaul+ZOS0zt1Q3HwKGt3lh7VQjf74Avgt8EfCLVvU0ncr8+AXwrMzdW+yxt88LaqUbuVwL7V9/3w7/3m8yQ3flcSeUv2rqyC+kARgPLgH+vTq/5SUT0Lruon9iozF1MZ8XkBWAKszsxp5VbVYQzOzCXV9y8Bg8sspoM5D/ivsotozyJiKrA4Mx8ru5YO4ghgckQ8nHBH3RsTEsgtq5/4OuDwiFlL5b4C/WWoiQ3YnEhHvBpZm5syya+kgugHHAz/KzOOA1/DX+I2qziOenSuUfJ0OB3hHx4XKr6niysqSTo41NEBGXAFuAG8qupb2KiF7Al6n8Gl9N0w0YAJwIfAG4KSKi3JLantU8An83MEcBnqf72V3tmyO5c3gKcGRELgF8Ap0TEz8otqV1bBCzKzIer2zdTCd1q2GnAc5m5LDM3nA7cAby65po7i5Yg4GKD6019P70FE/BXwbuCcdK3Z3TmMyj98H6v+3T8cmBURQ0qtqn1bBNySFdOpn/ObXL4s27qNU/r4H+E/ALz42kSG7E8nMizNzeGaOovKFtP/JTEcaG5GZLwELI+LIatOpwJMlltTenvQCcGBG9qqM+p+IXRZvqdir/oaL687YSa2n3IuIdVKa9nZmZ68qupz3LzMczc1Bmjqr+3b8IOL76n95sa9ivgZICIOALoASwvtaL27UXgbdX3pwDzS6ylQ+lWdgFSyT4F3BARPYBngb8uuZ52KzMfjoibngVlUfoX/KD4J7HUi4j+Ak4CBEbEI+CrwLSq/kv4Y8DzwF+VV2L40cr8uBvYDflf9Lf5DmXlhaUW2nIw3dr8z01/eNaOTP17XAtdVl6jYBH/W3JRWN3K+PA9+rfuF9A3B+eRV2LD7xUZIkSSqY00UkSZKknghmyJUmSpIIZsiVJkqSCGbIlSZKkghmyJUmSpIIZsiXtkyIiI+I79bY/HxGXFnTu6yLi/UWcaw/Xn+UBEzIuI3+/SPrS63GJzzvVXEfGDFtbx9Yg4rSXH7q2mXDsiTooIH5wkqU25TrakfdVG4M8j4puZn2W4eRBER3TJzSxO7fwz4eGY+UL8xM18EWj3k17teaY/0buK1TwLWAv/butVI0g6OZEvaV22h8jCdnz+66Y9eR6IhYW/15UkTcGxG3RcSzEfGtiDgnIqZHxOMRcVi905wWETMi4k8R8e7q8V0j4vKIeCQin5kTEBfXOe39E3E4DTx2NiA9Vzz83Iv652vYV4K3ANRFx+S79R1UftLFthPqWiLgzIuZHxLfr9fvrnan3Tgbfs6fNX3/+fai2PRcS3du0fEQsi4msRMava7w3V9oMi4ncR8URE/CQino+I1z3KOiLWRsR3nq/3ujoiDqu0TIuKh6n27NSL6N+XaETEKuBD4bETMjojJ1d8AzK1+hvt2rUGSimDIlrQv+1fgnIjon14xjjqUS2o4CPgIckZmTgJ9QeYLoNqOAScC7gKsioobKyPPqzJwITAQ+HhGjq/2PBz6TmUfUv1hEnDAX+mcrjjCcAEyPivZn5dWAGcE5mfmEPNU8APggcDXwwIkZExMHA16iE67cCY/f0wSPincBU4ITMnPBb4diNdl2fm8cCPgM9X274K/E9mjgNuBkY2cmxvYEa1373V4wB+CvyfzDwGeLxe+26vnZkLgKuAn72bmhMy8H/gKcEb1M5y5p88tSS1hyJa0z8rMNVTC26ebcdgjmbkkMzcCzwDTqu2PUwnW29yUmXWZnOR94FngDcDpwbkTMBh4GDgTGVPtPz8znGrjeROCezFxWnUZyAzClGfUC3J2ZqzNzA5WR8kOAE+qdndxNwYxPOcxrw75m5DiAzVzbS75bqz5nsuCdvBX5RPe5O4JVGjq2rV8vPgLdW/xF0QGbeW22/nsbvnQUPX3tUfgOsi4uNA10b6SNJeMWRL2tddSWWEuXe9ti1U/36MiC5Aj3r7NtZ7X1dvu46dv+eSu1wnngQA+VR1RnZCZozNzW0h/ba8+xe7Vr3kre/4+zu4+f3Ou15Rr7cmu93Gvr52ZFwL/AIwAZkbEgS0vnT5IaZsiWtE+rjsbeRCVob7MAeGP1/ZlA9xac+gMR0aU6T/tQ4I/AfwOfiIjuABFxRET03t1JgOnAn2yJiYER0BT5EZRrF3nq4et4Dq/V8oN6+BTT8+X8H/HVE9KrWP6AZ1/sD8BfV404H+jfSrws7vrT5nl8ADmbkaeCUiJlfbP0Lz7sGrQN9tGxFxWGY+XP3S5DIqYVuSCuXqIpIE3wH+tt72j4HbIuIx4E5anNsr8ApWAvD9wYWZuiIifUJnCMCsigkrAe+/uTpKZSyLiS8DvqYyE/yYzb2tBPQ2d91LgQWAVMLven7gY/f2beGRETgBkRsQn4LfDlJl7ya8B/RMRHqtd8iUr43dVrwKSI+AdgKZW55AAfpTK3vReV6Td/n3dTPCvwauDkiplKZN//ZiBhD5X7eDTzWjHNJUpNEZnN/EydJUvNExH7A1szcEhFvAn6UmRMa6Lc2nM/u0fYWSVCxHsiVJbWEkcFN1jvcm4OMl1yNJrcqRbEmSJKlgfvFRkiRJKpghW5IkSSqYIVuSJEkqnmCFbkiRJKpghW5IkSSqYIVuSJEkq2P8PZ8pL7C1MH7oAAAAASUVORK5CYII=n”, “text/plain”: [

“<matplotlib.figure.Figure at 0x14703e668>”

]

}, “metadata”: {}, “output_type”: “display_data”

}

], “source”: [

“plt.plot(fMs, fvfe_lml, label=”lower”)n”, “plt.plot(fMs, fvupper_lml, label=”upper”)n”, “plt.axhline(full_lml, label=”full”, alpha=0.3)n”, “plt.xlabel(“Number of inducing points”)n”, “plt.ylabel(“LML estimate”)n”, “plt.legend()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Now, as the hyperparameters are fixed, the bound _does_ monotonically decrease. We chose the optimal hyperparameters here, but the picture should be the same for any hyperparameter setting. This shows that we increasingly get a better estimate of the marginal likelihood as we add more inducing points.”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## A tight estimate bound does not imply a converged model”

]

}, {

“cell_type”: “code”, “execution_count”: 18, “metadata”: {}, “outputs”: [

{

“name”: “stdout”, “output_type”: “stream”, “text”: [

INFO:tensorflow:Optimization terminated with:n”, ” Message: b’CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH’n”, ” Objective function value: 62.487090n”, ” Number of iterations: 46n”, ” Number of functions evaluations: 57n”, “Lower bound: -62.487090n”, “Upper bound: -62.484641n”

]

}

], “source”: [

“vfe = gpflow.models.SGPR(X, Y, gpflow.kernels.RBF(1), X[None, 0, :].copy())n”, “vfe.compile()n”, “gpflow.train.ScipyOptimizer().minimize(vfe)n”, “print(“Lower bound: %f” % vfe.compute_log_likelihood())n”, “print(“Upper bound: %f” % vfe.compute_upper_bound())”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“In this case we show that for the hyperparameter setting, the bound is very tight. However, this does _not_ imply that we have enough inducing points, but simply that we have correctly identified the marginal likelihood for this particular hyperparameter setting. In this specific case, where we used a single inducing point, the model collapses to not using the GP at all (lengthscale is really long to model only the mean). The rest of the variance is explained by noise. This GP can be perfectly approximated with a single inducing point.”

]

}, {

“cell_type”: “code”, “execution_count”: 19, “metadata”: {}, “outputs”: [

{
“data”: {
“text/html”: [
“<div>n”, “<style>n”, ” .dataframe thead tr:only-child th {n”, ” text-align: right;n”, ” }n”, “n”, ” .dataframe thead th {n”, ” text-align: left;n”, ” }n”, “n”, ” .dataframe tbody tr th {n”, ” vertical-align: top;n”, ” }n”, “</style>n”, “<table border=”1” class=”dataframe”>n”, ” <thead>n”, ” <tr style=”text-align: right;”>n”, ” <th></th>n”, ” <th>class</th>n”, ” <th>prior</th>n”, ” <th>transform</th>n”, ” <th>trainable</th>n”, ” <th>shape</th>n”, ” <th>fixed_shape</th>n”, ” <th>value</th>n”, ” </tr>n”, ” </thead>n”, ” <tbody>n”, ” <tr>n”, ” <th>SGPR/kern/variance</th>n”, ” <td>Parameter</td>n”, ” <td>None</td>n”, ” <td>+ve</td>n”, ” <td>True</td>n”, ” <td>()</td>n”, ” <td>True</td>n”, ” <td>0.10775260845160224</td>n”, ” </tr>n”, ” <tr>n”, ” <th>SGPR/kern/lengthscales</th>n”, ” <td>Parameter</td>n”, ” <td>None</td>n”, ” <td>+ve</td>n”, ” <td>True</td>n”, ” <td>()</td>n”, ” <td>True</td>n”, ” <td>1014.3847358198036</td>n”, ” </tr>n”, ” <tr>n”, ” <th>SGPR/likelihood/variance</th>n”, ” <td>Parameter</td>n”, ” <td>None</td>n”, ” <td>+ve</td>n”, ” <td>True</td>n”, ” <td>()</td>n”, ” <td>True</td>n”, ” <td>0.682431983343512</td>n”, ” </tr>n”, ” <tr>n”, ” <th>SGPR/feature/Z</th>n”, ” <td>Parameter</td>n”, ” <td>None</td>n”, ” <td>(none)</td>n”, ” <td>True</td>n”, ” <td>(1, 1)</td>n”, ” <td>True</td>n”, ” <td>[[2.65740317504]]</td>n”, ” </tr>n”, ” </tbody>n”, “</table>n”, “</div>”

], “text/plain”: [

” class prior transform trainable shape \n”, “SGPR/kern/variance Parameter None +ve True () n”, “SGPR/kern/lengthscales Parameter None +ve True () n”, “SGPR/likelihood/variance Parameter None +ve True () n”, “SGPR/feature/Z Parameter None (none) True (1, 1) n”, “n”, ” fixed_shape value n”, “SGPR/kern/variance True 0.10775260845160224 n”, “SGPR/kern/lengthscales True 1014.3847358198036 n”, “SGPR/likelihood/variance True 0.682431983343512 n”, “SGPR/feature/Z True [[2.65740317504]] “

]

}, “execution_count”: 19, “metadata”: {}, “output_type”: “execute_result”

}

], “source”: [

“vfe.as_pandas_table()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“This can be diagnosed by showing that there are other hyperparameter settings with higher upper bounds. This indicates that there may be better hyperparameter settings, but we cannot identify them due to the lack of inducing points. An example of this can be seen in the previous section.”

]

}

], “metadata”: {

“kernelspec”: {
“display_name”: “Python [default]”, “language”: “python”, “name”: “python3”

}, “language_info”: {

“codemirror_mode”: {
“name”: “ipython”, “version”: 3

}, “file_extension”: “.py”, “mimetype”: “text/x-python”, “name”: “python”, “nbconvert_exporter”: “python”, “pygments_lexer”: “ipython3”, “version”: “3.6.0”

}

}, “nbformat”: 4, “nbformat_minor”: 1

}