additional termination conditions

[lacava]

As mentioned in #124, we want the user to be able to specify additional termination conditions for the recommender, besides a set number of analyses. Other ways the recommender could terminate:

• based on a stall condition: recommendations stop when a recommended algorithm has not improved the results for stall_count iterations.

  • implementation: either the AI class or Recommender class would have to keep track of recommendations for each dataset and their scores.
  • issues:
    • running in parallel would be an issue, since we need to know the outcome to decide whether or not to continue.
    • need to store the origin of a result (whether or not it came from a user experiment or the AI)

• based on a computational budget: recommendations stop when X minutes have passed.

  • implementation: the AI class keeps track of the wall-clock time, and could continue to add experiments to the queue according to a delta_time variable since original request was received.
  • issues: again hard to implement with parallel machines. we currently don't have a good way of knowing how long an experiment will take, unless we want to start making educated guesses.

[lacava]

recommendations stop when X minutes have passed

• object that stores start time, dataset, request
• terminate queue if termination conditions are met