<jupyter_start><jupyter_text>Using the Progress Callbacks with CPLEX OptimizerThis tutorial includes everything you need to set up decision optimization engines, build a mathematical programming model, then use the progress callbacks to follow the progress, capture the intermediate solutions and stop the solve on your own criteria.When you finish this tutorial, you'll have a foundational knowledge of _Prescriptive Analytics_.>This notebook is part of the **[Prescriptive Analytics for Python](https://rawgit.com/IBMDecisionOptimization/docplex-doc/master/docs/index.html)**>It requires an [installation of CPLEX Optimizers](http://ibmdecisionoptimization.github.io/docplex-doc/getting_started.html)Discover us [here](https://developer.ibm.com/docloud)Table of contents:-  [Describe the business problem](Describe-the-business-problem:--Games-Scheduling-in-the-National-Football-League)*  [How decision optimization (prescriptive analytics) can help](How--decision-optimization-can-help)*  [Use decision optimization](Use-decision-optimization)    *  [Step 1: Import the library](Step-1:-Import-the-library)    *  [Step 2: Set up the prescriptive model](Step-2:-Set-up-the-prescriptive-model)    *  [Step 3: Track the CPLEX progress](Step-3:-Track-the-CPLEX-progress)*  [Summary](Summary)****  How  decision optimization can help* Prescriptive analytics (decision optimization) technology recommends actions that are based on desired outcomes.  It takes into account specific scenarios, resources, and knowledge of past and current events. With this insight, your organization can make better decisions and have greater control of business outcomes.  * Prescriptive analytics is the next step on the path to insight-based actions. It creates value through synergy with predictive analytics, which analyzes data to predict future outcomes.  * Prescriptive analytics takes that insight to the next level by suggesting the optimal way to handle that future situation. Organizations that can act fast in dynamic conditions and make superior decisions in uncertain environments gain a strong competitive advantage.  With prescriptive analytics, you can: * Automate the complex decisions and trade-offs to better manage your limited resources.* Take advantage of a future opportunity or mitigate a future risk.* Proactively update recommendations based on changing events.* Meet operational goals, increase customer loyalty, prevent threats and fraud, and optimize business processes.  Use decision optimization  Step 1: Import the libraryRun the following code to import Decision Optimization CPLEX Modeling library.  The *DOcplex* library contains the two modeling packages, Mathematical Programming and Constraint Programming, referred to earlier.<jupyter_code>import sys
try:
    import docplex.mp
except:
    raise Exception('Please install docplex. See https://pypi.org/project/docplex/')<jupyter_output><empty_output><jupyter_text>Step 2: Set up the prescriptive model We will write a scalable model just in order to show how to use the progress callback API.This model is not important: it is a model that can take very long to solve and will find multiple intermediate solutions<jupyter_code>from docplex.mp.model import Model

def build_hearts(r, **kwargs):
    # initialize the model
    mdl = Model('love_hearts_%d' % r, **kwargs)

    # the dictionary of decision variables, one variable
    # for each circle with i in (1 .. r) as the row and
    # j in (1 .. i) as the position within the row    
    idx = [(i, j) for i in range(1, r + 1) for j in range(1, i + 1)]
    a = mdl.binary_var_dict(idx, name=lambda idx_tuple: "a_%d_%d" % (idx_tuple[0], idx_tuple[1]))

    # the constraints - enumerate all equilateral triangles
    # and prevent any such triangles being formed by keeping
    # the number of included circles at its vertexes below 3

    # for each row except the last
    for i in range(1, r):
        # for each position in this row
        for j in range(1, i + 1):
            # for each triangle of side length (k) with its upper vertex at
            # (i, j) and its sides parallel to those of the overall shape
            for k in range(1, r - i + 1):
                # the sets of 3 points at the same distances clockwise along the
                # sides of these triangles form k equilateral triangles
                for m in range(k):
                    u, v, w = (i + m, j), (i + k, j + m), (i + k - m, j + k - m)
                    mdl.add_constraint(a[u] + a[v] + a[w] <= 2)

    mdl.maximize(mdl.sum(a))
    return mdl<jupyter_output><empty_output><jupyter_text>It is perfect to demonstrate how to listen to CPLEX during its progress as it contains multiple intermediate solutions.  Step 3: Track the CPLEX progress  Basic track of the search progressWe will use a TextProgressListener to follow the CPLEX search. It will print on the standard output each time an intermediate solution is found.<jupyter_code>m5 = build_hearts(5)

from docplex.mp.progress import TextProgressListener
unfiltered_texter = TextProgressListener(filtering=False)
m5.add_progress_listener(unfiltered_texter)<jupyter_output><empty_output><jupyter_text>Solve with Decision OptimizationIf you're using a Community Edition of CPLEX runtimes, depending on the size of the problem, the solve stage may fail and will need a paying subscription or product installation.You will get the best solution found after ***n*** seconds, thanks to a time limit parameter.<jupyter_code>m5.solve(clean_before_solve=True)<jupyter_output><empty_output><jupyter_text>Let's remove the listener.<jupyter_code>m5.remove_progress_listener(unfiltered_texter)<jupyter_output><empty_output><jupyter_text>Track and store all intermediate solutions You just need to subclass the SolutionListener object and specialize the notify_* methods if needed.Here we will store all solutions all along the way.<jupyter_code>from docplex.mp.progress import SolutionListener
class MyProgressListener(SolutionListener):
    def __init__(self, model):
        SolutionListener.__init__(self, model)
        self.solutions = []
 
    def notify_solution(self, s):
        SolutionListener.notify_solution(self, s)
        self.solutions.append(self.current_solution)
 
    def get_solutions(self):
        return self.solutions
    
keeper = MyProgressListener(m5)
m5.add_progress_listener(keeper)
m5.solve(clean_before_solve=True)<jupyter_output><empty_output><jupyter_text>We have stored docplex.mp.solution.SolveSolution objects.We can iterate on them to query the objective values, the values of each variables...<jupyter_code>for s in keeper.get_solutions():
     print(s.objective_value)<jupyter_output><empty_output><jupyter_text>Implement our own aborterIt may be nice to be able to abort the CPLEX on your own criteria.For example, when the gap is converging very slowly, it may be a good idea to stop and use the last solution instead of waiting forever.You just need to subclass the ProgressListener and specialize the notify_* methods.<jupyter_code>from docplex.mp.progress import ProgressListener
class AutomaticAborter(ProgressListener):
    def __init__(self, max_no_solutions=10):
        self.last_incumbent_obj = -999999999
        self.nb_solutions = 0
        self.nb_non_improving = 0
        self.max_non_improving = max_no_solutions

    def notify_progress(self, progress_data):
        super(AutomaticAborter, self).notify_progress(progress_data)
        last_obj = self.last_incumbent_obj
        if progress_data.has_incumbent:
            if last_obj is None or progress_data.current_objective >= last_obj + 1e-5:
                self.nb_solutions += 1
                self.nb_non_improving = 0
                print('----> #solutions={}'.format(self.nb_solutions))
            else:

                # non improving move
                self.nb_non_improving += 1
                print('----> #non improving solutions={}'.format(self.nb_non_improving))
            self.last_incumbent_obj = progress_data.current_objective
        else:
            self.nb_non_improving += 1
        if self.nb_non_improving >= self.max_non_improving:
            if not self.has_aborted():
                print('!! aborting cplex, #solutions={0}, #non-improving: {1}'.format(self.nb_solutions,
                                                                                              self.nb_non_improving))
            self.abort()<jupyter_output><empty_output><jupyter_text>Let's build a bigger problem with more solutions.<jupyter_code>love14 = build_hearts(11)
auto_abort = AutomaticAborter(max_no_solutions=50)
love14.add_progress_listener(auto_abort)

love14.solve()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07414645361535808, 'fraction_numerical': 0.0073571674905161515, 'mean_word_length': 3.895891952729319, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 43, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '315287', 'license': 'Apache-2.0', 'n_tokens_mistral': 2447, 'n_tokens_neox': 2300, 'n_words': 871, 'path': 'jupyter/Progress+Callbacks+Local.jupyter-py36.ipynb', 'repo_name': 'vostertag/DO-Samples', 'size': '8699'}