How should the parallelism parameter relate to max_evals in Hyperopt?

In the context of Hyperopt, parallelism refers to the ability to evaluate multiple configurations or trials of hyperparameters simultaneously. The max_evals parameter specifies the maximum number of evaluations or trials that Hyperopt will perform.

When the parallelism parameter is set to be smaller by an order of magnitude than max_evals, it allows for a more manageable and efficient exploration of the hyperparameter space. This configuration ensures that while a significant number of trials can be executed, the system does not become overwhelmed by trying to evaluate too many at once, which could lead to resource contention and inefficiencies.

Moreover, having a lower parallelism relative to max_evals allows for better utilization of computational resources, making it feasible to track progress, manage jobs effectively, and gather results from ongoing trials without creating excessive load. It helps maintain the balance between exploration of the hyperparameter space and the practical limitations of the hardware or cluster being utilized.

Setting the parallelism parameter significantly higher than max_evals or too close can lead to diminished returns in performance and possible instability in the execution, while being set to half would also not efficiently leverage the potential evaluations available within max_evals. Therefore, having the parallelism smaller by an order of magnitude aligns well with best practices for using