02. MultipleParameterTutorial

In this tutorial, you will learn how to:

• Optimize the Objective Function with Multiple HyperParameters
• Define different types of Search Space

# Import HyperOpt Library
from hyperopt import tpe, hp, fmin
import numpy as np

Declares a objective function to optimize. Unlike last time, we will optimize the function with two Hyperparameters, $x$ and $y$.

$$ z = sin\sqrt{x^2 + y^2} $$

def objective(params):
    x, y = params['x'], params['y']
    return np.sin(np.sqrt(x**2 + y**2))

Just like last time, let's try visualizing it. But unlike last time, there are two Hyperparameters, so we need to visualize them in 3D space.

import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

x = np.linspace(-6, 6, 30)
y = np.linspace(-6, 6, 30)
x, y = np.meshgrid(x, y)

z = objective({'x': x, 'y': y})

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(x, y, z, cmap=cm.coolwarm)

ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')

plt.show()

Likewise, let's define the search space. However, this time, you need to define two search spaces($x, y$), so you put each of them in the dict().

space = {
    'x': hp.uniform('x', -6, 6),
    'y': hp.uniform('y', -6, 6)
}

Perfect! Now you can do exactly what you did at BasicTutorial!

best = fmin(
    fn=objective, # Objective Function to optimize
    space=space, # Hyperparameter's Search Space
    algo=tpe.suggest, # Optimization algorithm (representative TPE)
    max_evals=1000 # Number of optimization attempts
)
print(best)

100%|██████████| 1000/1000 [00:07<00:00, 127.90trial/s, best loss: -0.9999976342002768]
{'x': 4.278018218372159, 'y': 1.97095757186186}

Define different types of Search Space

• hp.randint(label, upper) searches the integer in the [0, upper) interval.
• hp.choice(label, list) searches for elements in the list.

def f(params):
    x1, x2 = params['x1'], params['x2']
    if x1 == 'james':
        return -1 * x2
    if x1 == 'max':
        return 2 * x2
    if x1 == 'wansoo':
        return -3 * x2

search_space = {
    'x1': hp.choice('x1', ['james', 'max', 'wansoo']),
    'x2': hp.randint('x2', -5, 5)
}

best = fmin(
    fn=f,
    space=search_space,
    algo=tpe.suggest,
    max_evals=100
)

print(best)

100%|██████████| 100/100 [00:00<00:00, 396.61trial/s, best loss: -12.0]
{'x1': 2, 'x2': 4}