Add MINLPTests

[odow]

The merge target for this is #575

Adding support for MINLPTests: https://github.com/jump-dev/MINLPTests.jl

My test locally was not promising.

Test Summary:          | Pass  Fail  Error  Total  Time
nlp-cvx-expr           |  183    18      8    209  6.7s
  nlp_cvx_expr_001_010 |    5                   5  0.0s
  nlp_cvx_expr_001_011 |    5                   5  0.0s
  nlp_cvx_expr_002_010 |    5                   5  0.0s
  nlp_cvx_expr_002_011 |    5                   5  0.0s
  nlp_cvx_expr_101_010 |    5                   5  0.0s
  nlp_cvx_expr_101_011 |    5                   5  0.0s
  nlp_cvx_expr_101_012 |    5                   5  0.0s
  nlp_cvx_expr_102_010 |    5                   5  0.0s
  nlp_cvx_expr_102_011 |    3                   3  0.0s
  nlp_cvx_expr_102_012 |    5                   5  0.0s
  nlp_cvx_expr_102_013 |    5                   5  0.0s
  nlp_cvx_expr_102_014 |    3     2             5  0.0s
  nlp_cvx_expr_103_010 |    5                   5  0.0s
  nlp_cvx_expr_103_011 |    5                   5  0.0s
  nlp_cvx_expr_103_012 |    3     2             5  0.0s
  nlp_cvx_expr_103_013 |    3     2             5  0.0s
  nlp_cvx_expr_103_014 |    5                   5  0.0s
  nlp_cvx_expr_104_010 |    5                   5  0.0s
  nlp_cvx_expr_105_010 |                 1      1  0.0s
  nlp_cvx_expr_105_011 |                 1      1  0.0s
  nlp_cvx_expr_105_012 |                 1      1  0.0s
  nlp_cvx_expr_105_013 |                 1      1  0.0s
  nlp_cvx_expr_106_010 |                 1      1  0.0s
  nlp_cvx_expr_106_011 |                 1      1  0.0s
  nlp_cvx_expr_107_010 |    3     2             5  0.0s
  nlp_cvx_expr_107_011 |    5                   5  0.0s
  nlp_cvx_expr_107_012 |    5                   5  0.0s
  nlp_cvx_expr_108_010 |    3     2             5  0.0s
  nlp_cvx_expr_108_011 |    5                   5  0.0s
  nlp_cvx_expr_108_012 |    5                   5  0.0s
  nlp_cvx_expr_108_013 |    3     2             5  0.0s
  nlp_cvx_expr_109_010 |    5                   5  0.0s
  nlp_cvx_expr_109_011 |    5                   5  0.0s
  nlp_cvx_expr_109_012 |    5                   5  0.0s
  nlp_cvx_expr_110_010 |    5                   5  0.0s
  nlp_cvx_expr_110_011 |    5                   5  0.0s
  nlp_cvx_expr_110_012 |    5                   5  0.0s
  nlp_cvx_expr_201_010 |    6                   6  0.0s
  nlp_cvx_expr_201_011 |    6                   6  0.0s
  nlp_cvx_expr_202_010 |    6                   6  0.0s
  nlp_cvx_expr_202_011 |    6                   6  0.0s
  nlp_cvx_expr_202_012 |    3     3             6  0.0s
  nlp_cvx_expr_202_013 |    3     3             6  0.0s
  nlp_cvx_expr_202_014 |    6                   6  0.0s
  nlp_cvx_expr_203_010 |                 1      1  0.0s
  nlp_cvx_expr_204_010 |                 1      1  0.0s
  nlp_cvx_expr_205_010 |    6                   6  0.0s
  nlp_cvx_expr_206_010 |                     None  6.3s
ERROR: Some tests did not pass: 183 passed, 18 failed, 8 errored, 0 broken.

It seems like the tolerance is quite bad?

nlp_cvx_expr_103_013: Test Failed at /Users/oscardowson/.julia/packages/MINLPTests/hkcTW/src/MINLPTests.jl:75
  Expression: isapprox(JuMP.value(variable), solution, atol = tol)
   Evaluated: isapprox(0.2501297534424012, 0.25; atol = 1.0e-6)

And there are some issues with the parser:

nlp_cvx_expr_105_010: Error During Test at /Users/oscardowson/.julia/packages/MINLPTests/hkcTW/src/MINLPTests.jl:149
  Got exception outside of a @test
  Gurobi Error 10003: Problem adding general constraint

nlp_cvx_expr_106_010: Error During Test at /Users/oscardowson/.julia/packages/MINLPTests/hkcTW/src/MINLPTests.jl:149
  Got exception outside of a @test
  Gurobi Error 10006: invalid parent index in general constraint NL (node 7)

[odow]

The JuMP models to reproduce are https://github.com/jump-dev/MINLPTests.jl/tree/master/src/nlp-cvx-expr

nlp_cvx_expr_105_010 is:

using JuMP, Gurobi
model = Model(Gurobi.Optimizer)
@variable(model, x, start = 0.1)
@variable(model, y)
@objective(model, Min, -x - y)
@constraint(model, exp(x - 2.0) - 0.5 <= y)
@constraint(model, log(x) + 0.5 >= y)
optimize!(model)

which simplifies to

julia> using JuMP, Gurobi

julia> model = Model(Gurobi.Optimizer);
Set parameter LicenseID to value 890777

julia> @variable(model, x);

julia> @constraint(model, exp(x - 1) <= 1)
exp(x - 1) - 1.0 ≤ 0

julia> optimize!(model)
ERROR: Gurobi Error 10003: Problem adding general constraint
Stacktrace:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.16%. Comparing base (729ecb9) to head (fd5da47).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #586   +/-   ##
=======================================
  Coverage   92.16%   92.16%           
=======================================
  Files           6        6           
  Lines        2630     2630           
=======================================
  Hits         2424     2424           
  Misses        206      206           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[simonbowly]

Updates on #575 sorted out the API errors in my local test. I'd expect to see some tolerance/convergence failures, it's still early days for us in NLP.

@odow the MINLPTests don't actually seem to be running in CI? Or did I miss something?

[odow]

CI is running: https://github.com/jump-dev/Gurobi.jl/actions/runs/11811230287/job/32904476402?pr=586

[simonbowly]

Ah, totally separate job, thanks

[simonbowly]

We should probably set a time limit. It's liable to spin forever trying to prove global optimality on a model with no bounds.

[odow]

So a tolerance issue is actually an artefact of how we think about nonlinear solutions. The objective is optimal within tolerance, but you've found a solution that satisfies the constraints, and yet is "far" away from the theoretical optimal primal point.

Here's an example:

julia> using JuMP, Gurobi

julia> begin
           model = Model(Gurobi.Optimizer)
           @variable(model, x)
           @variable(model, y)
           @objective(model, Min, x + y)
           @constraint(model, x^2 <= y)
           @constraint(model, -x^2 + 1 >= y)
           optimize!(model)
           solution_summary(model; verbose = true)
       end
Set parameter LicenseID to value 890777
Gurobi Optimizer version 12.0.0 build v12.0.0rc1 (mac64[arm] - Darwin 24.0.0 24A335)

CPU model: Apple M1
Thread count: 8 physical cores, 8 logical processors, using up to 8 threads

Optimize a model with 0 rows, 2 columns and 0 nonzeros
Model fingerprint: 0x0b797f74
Model has 2 quadratic constraints
Coefficient statistics:
  Matrix range     [0e+00, 0e+00]
  QMatrix range    [1e+00, 1e+00]
  QLMatrix range   [1e+00, 1e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [0e+00, 0e+00]
  RHS range        [0e+00, 0e+00]
  QRHS range       [1e+00, 1e+00]
Presolve time: 0.00s
Presolved: 5 rows, 7 columns, 10 nonzeros
Presolved model has 2 second-order cone constraints
Ordering time: 0.00s

Barrier statistics:
 AA' NZ     : 7.000e+00
 Factor NZ  : 1.500e+01
 Factor Ops : 5.500e+01 (less than 1 second per iteration)
 Threads    : 1

                  Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
   0   8.22993840e-01  1.96000000e-01  3.13e-01 1.60e+00  4.97e-01     0s
   1  -1.12124092e-01 -3.52459705e-01  4.45e-02 1.73e-01  7.36e-02     0s
   2  -2.46594364e-01 -2.66809783e-01  1.11e-03 4.74e-03  4.25e-03     0s
   3  -2.49953016e-01 -2.50338631e-01  1.22e-09 9.77e-05  9.02e-05     0s
   4  -2.49999253e-01 -2.50001508e-01  4.67e-12 3.85e-07  4.78e-07     0s
   5  -2.49999968e-01 -2.50000026e-01  5.34e-11 1.44e-08  1.34e-08     0s

Barrier solved model in 5 iterations and 0.00 seconds (0.00 work units)
Optimal objective -2.49999968e-01


User-callback calls 48, time in user-callback 0.00 sec
* Solver : Gurobi

* Status
  Result count       : 1
  Termination status : OPTIMAL
  Message from the solver:
  "Model was solved to optimality (subject to tolerances), and an optimal solution is available."

* Candidate solution (result #1)
  Primal status      : FEASIBLE_POINT
  Dual status        : NO_SOLUTION
  Objective value    : -2.50000e-01
  Objective bound    : -2.50000e-01
  Dual objective value : -2.50000e-01
  Primal solution :
    x : -5.00130e-01
    y : 2.50130e-01

* Work counters
  Solve time (sec)   : 3.58105e-04
  Simplex iterations : 0
  Barrier iterations : 5
  Node count         : 0

[odow]

@simonbowly here's one of the time limit examples. It's a univariate polynomial with one-sided domain.

julia> using JuMP, Gurobi

julia> begin
           model = Model(Gurobi.Optimizer)
           @variable(model, z >= 1)
           @objective(model, Min, z^3 - z^2)
           optimize!(model)
           solution_summary(model; verbose = true)
       end
Set parameter LicenseID to value 890777
Gurobi Optimizer version 12.0.0 build v12.0.0rc1 (mac64[arm] - Darwin 24.0.0 24A335)

CPU model: Apple M1
Thread count: 8 physical cores, 8 logical processors, using up to 8 threads

Optimize a model with 0 rows, 3 columns and 0 nonzeros
Model fingerprint: 0x893ba885
Model has 1 general nonlinear constraint (2 nonlinear terms)
Variable types: 3 continuous, 0 integer (0 binary)
Coefficient statistics:
  Matrix range     [0e+00, 0e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [1e+00, 1e+00]
  RHS range        [0e+00, 0e+00]
Found heuristic solution: objective 0.0000000
Presolve model has 1 nlconstr
Added 3 variables to disaggregate expressions.
Presolve time: 0.00s
Presolved: 10 rows, 7 columns, 23 nonzeros
Presolved model has 1 bilinear constraint(s)
Presolved model has 1 nonlinear constraint(s)
Warning: Model contains variables with very large bounds participating
         in nonlinear terms.
         Presolve was not able to compute smaller bounds for these variables.
         Consider bounding these variables or reformulating the model.


Solving non-convex MINLP

Variable types: 7 continuous, 0 integer (0 binary)

Root relaxation: unbounded, 1 iterations, 0.00 seconds (0.00 work units)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0  postponed    0         0.00000          -      -     -    0s
     0     0  postponed    0         0.00000          -      -     -    0s
     0     2  postponed    0         0.00000          -      -     -    0s
 771394  3487          -   83         0.00000 -5.137e+14      -   0.0    5s
 1571231  3764 -5.137e+14   69    1    0.00000 -5.137e+14      -   0.0   10s
 2362922  3723 -5.137e+14   75    1    0.00000 -5.137e+14      -   0.0   15s
 3162922  3557          -   80         0.00000 -5.137e+14      -   0.0   20s