Skip to content

Add examples for the ReSolve SUNLinSol interface - #13

Merged
shakedregev merged 6 commits into
resolve-developfrom
resolve/ida_examples
Nov 17, 2026
Merged

shakedregev merged 6 commits into
resolve-developfrom
resolve/ida_examples

Conversation

@JeffZ594

@JeffZ594 JeffZ594 commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

This pull request adds 4 examples that are located under examples/ida/resolve:

When building SUNDIALS the flags

-D SUNDIALS_ENABLE_RESOLVE=ON
-D ReSolve_DIR= <path/to/resolve/installation>
-D SUNDIALS_ENABLE_CXX_EXAMPLES=ON

need to be enabled for the ReSolve examples to be built.

To build SUNDIALS with CUDA, enable

-D SUNDIALS_RESOLVE_BACKENDS=CUDA
-D SUNDIALS_ENABLE_CUDA=ON

idaHeat2D_resolve.cpp: A heat equation example that runs on CPU, CUDA and HIP. This example works on CUDA and HIP by doing all work on the host and sending data to the device as necessary.

idaHeat2D_resolve_cuda.cu: A heat equation example that uses CUDA kernels to perform work on the device where possible.

Both heat examples have identical results to idaHeat2D_klu.c located under examples/ida/serial.

The results are as follows:

Time umax k nst nni nje nre h
0.00 9.75461e-01 0 0 0 2 2 5.15e-10
0.01 8.24056e-01 5 53 63 23 65 5.55e-04
0.02 6.88097e-01 5 69 81 24 83 9.99e-04
0.04 4.70961e-01 5 90 106 27 108 1.91e-03
0.08 2.16312e-01 5 113 130 27 132 1.72e-03
0.16 4.53210e-02 5 137 155 28 157 3.43e-03
0.32 1.98864e-03 5 173 193 29 195 6.18e-03
0.64 3.83238e-06 5 210 233 31 235 2.22e-02
1.28 0.00000e+00 1 227 255 34 257 1.78e-01
2.56 0.00000e+00 1 230 258 37 260 1.42e+00
5.12 0.00000e+00 1 231 259 38 261 2.85e+00
10.24 0.00000e+00 1 232 260 39 262 5.69e+00

Number of Error Test Failures: 2

Number of Nonlinear Solver convergence failures: 0

idaFoodWeb_resolve.cpp: A food web example that runs on CPU, CUDA and HIP. This example works on CUDA and HIP by doing all work on the host and sending data to the device as necessary.

idaFoodWeb_resolve_cuda.cu: A food web example that uses CUDA kernels to perform work on the device where possible.

Although both converge, these two examples vary in performance.

idaFoodWeb_resolve.cpp (CPU)

Time Bottom-left Top-right nst k h
0.00e+00 1.0000e+011.0000e+05 1.0000e+011.0000e+05 0 0 1.6310e-08
1.00e-03 1.0318e+011.0319e+05 1.0827e+011.0822e+05 32 4 1.0823e-04
1.00e-02 1.6189e+021.6189e+06 1.9735e+021.9735e+06 137 3 1.1679e-04
1.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 235 1 2.3789e-02
4.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 240 1 1.9031e-01
7.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 242 1 1.9031e-01
1.00e+00 2.4019e+022.4019e+06 2.7072e+022.7072e+06 243 1 3.8062e-01

idaFoodWeb_resolve_cuda.cu

Time Bottom-left Top-right nst k h
0.00e+00 1.0000e+011.0000e+05 1.0000e+011.0000e+05 0 0 1.6310e-08
1.00e-03 1.0318e+011.0319e+05 1.0827e+011.0822e+05 32 4 1.0823e-04
1.00e-02 1.6189e+021.6189e+06 1.9735e+021.9735e+06 139 3 1.0263e-04
1.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 244 1 3.0601e-02
4.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 255 1 1.5301e-02
7.00e-01 2.4019e+022.4019e+06 2.7072e+022.7072e+06 276 1 7.6503e-03
1.00e+00 2.4019e+022.4019e+06 2.7072e+022.7072e+06 295 1 1.5301e-02

Following are the solver statistics

Metric idaFoodWeb_resolve.cpp (CPU) idaFoodWeb_resolve_cuda.cu
Number of steps 243 295
Number of residual evaluations 457 572
Number of Jacobian evaluations 47 114
Number of nonlinear iterations 455 570
Number of error test failures 8 5
Number of nonlinear conv. failures 11 36
Number of step solver failures 4 28

In addition to these examples, the following modifications are made to the SUNMatrix_ReSolve class

  • Add getters for the Re::Solve memory space object and the ReSolve CSR matrix object

  • Changed the behaviour of SUNMatZero to only zero out the data array, keeping the sparsity pattern.

  • Please target the develop branch not main.

  • Review our Contributing Guide, and ensure that you sign your last commit (at minimum) as per the guide.

  • Provide a concise description of what your pull request does, and why it is needed/benefical.

  • Add a note about your change to the CHANGELOG.md and docs/shared/RecentChanges.rst files. Notice that the former is a markdown file and the latter is reStructuredText, so the formatting is slightly different.

  • After your PR is opened, ensure that all of the tests are passing (a SUNDIALS developer will have to allow the testing to run).

…e and the matrix object, and modify SUNMatZero to only zero out the data array.
@JeffZ594
JeffZ594 requested a review from pelesh August 12, 2026 02:05
@pelesh

pelesh commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Please add summary how solutions to the problem compare.

@shakedregev

Copy link
Copy Markdown
Collaborator

What is the command to run the examples?

@shakedregev shakedregev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still need to test this.

* 1 2 ns
* c = (c , c , ..., c ) , ns = 2 * np
*
* and the PDE's are as follows:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to follow. Maybe better to have it rendered in readthedocs or something.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a hold over from how the other built in examples explained the example. They have a better rendered explanation of the examples here https://sundials.readthedocs.io/en/latest/examples/ida/index-superbuild.html

@shakedregev

Copy link
Copy Markdown
Collaborator

Why is there one version for CPU/HIP/CUDA + another version of the code just for CUDA?

@shakedregev shakedregev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I am running the correct thing. Please specify exactly which examples to run. But:

 ./idaHeat2D_kry

idaHeat2D_kry: Heat equation, serial example problem for IDA 
         Discretized heat equation on 2D unit square. 
         Zero boundary conditions, polynomial initial conditions.
         Mesh dimensions: 10 x 10        Total system size: 100

Tolerance parameters:  rtol = 0   atol = 0.001
Constraints set to force all solution components >= 0. 
Linear solver: SPGMR, preconditioner using diagonal elements. 


Case 1: gsytpe = SUN_MODIFIED_GS

   Output Summary (umax = max-norm of solution) 

  time     umax       k  nst  nni  nje   nre   nreLS    h      npe nps
----------------------------------------------------------------------
  0.01   8.24106e-01  2   12   14    7    14     7   2.56e-03    8  21
  0.02   6.88134e-01  3   15   18   12    18    12   5.12e-03    8  30
  0.04   4.70711e-01  3   18   24   21    24    21   6.58e-03    9  45
  0.08   2.16509e-01  3   22   29   30    29    30   1.32e-02    9  59
  0.16   4.57687e-02  4   28   36   44    36    44   1.32e-02    9  80
  0.32   2.09938e-03  4   35   44   67    44    67   2.63e-02   10 111
  0.64   0.00000e+00  1   39   51   77    51    77   1.05e-01   12 128
  1.28   0.00000e+00  1   41   53   77    53    77   4.21e-01   14 130
  2.56   0.00000e+00  1   43   55   77    55    77   1.69e+00   16 132
  5.12   0.00000e+00  1   44   56   77    56    77   3.37e+00   17 133
 10.24   0.00000e+00  1   45   57   77    57    77   6.74e+00   18 134

Error test failures            = 1
Nonlinear convergence failures = 0
Linear convergence failures    = 0


Case 2: gstype = SUN_CLASSICAL_GS

   Output Summary (umax = max-norm of solution) 

  time     umax       k  nst  nni  nje   nre   nreLS    h      npe nps
----------------------------------------------------------------------
  0.01   8.24106e-01  2   12   14    7    14     7   2.56e-03    8  21
  0.02   6.88134e-01  3   15   18   12    18    12   5.12e-03    8  30
  0.04   4.70711e-01  3   18   24   21    24    21   6.58e-03    9  45
  0.08   2.16509e-01  3   22   29   30    29    30   1.32e-02    9  59
  0.16   4.57687e-02  4   28   36   44    36    44   1.32e-02    9  80
  0.32   2.09938e-03  4   35   44   67    44    67   2.63e-02   10 111
  0.64   0.00000e+00  1   39   51   77    51    77   1.05e-01   12 128
  1.28   0.00000e+00  1   41   53   77    53    77   4.21e-01   14 130
  2.56   0.00000e+00  1   43   55   77    55    77   1.69e+00   16 132
  5.12   0.00000e+00  1   44   56   77    56    77   3.37e+00   17 133
 10.24   0.00000e+00  1   45   57   77    57    77   6.74e+00   18 134

Error test failures            = 1
Nonlinear convergence failures = 0
Linear convergence failures    = 0

@JeffZ594

JeffZ594 commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

I updated the pull request to add instructions to make the ReSolve examples install. The example that you ran was for SUNDIALS's built in Krylov Solver. The specific examples to run are located under examples/ida/resolve. If the cmake flags in the pull request are enabled they should appear.

The version of the examples that is for CPU/HIP/CUDA just uses ReSolve for the linear solves and leaves everything else to be done on the host, but the CUDA only examples uses CUDA kernels to do work in the example like updating the Jacobian for the Heat Equation example.

@shakedregev

Copy link
Copy Markdown
Collaborator

Are we expecting some failures:

./idaFoodWeb_resolve 

idaFoodWeb_ReSolve: Predator-prey DAE example for IDA

Number of species ns: 2     Mesh dimensions: 20 x 20     System size: 800
Tolerance parameters:  rtol = 1e-05   atol = 1e-05
Linear solver: ReSolve KLU with CPU backend
CalcIC called to correct initial predator concentrations.

-----------------------------------------------------------
  t        bottom-left  top-right    | nst  k      h
-----------------------------------------------------------

0.00e+00   1.0000e+01   1.0000e+01   |   0  0   1.6310e-08
           1.0000e+05   1.0000e+05   |

1.00e-03   1.0318e+01   1.0827e+01   |  32  4   1.0823e-04
           1.0319e+05   1.0822e+05   |

1.00e-02   1.6189e+02   1.9735e+02   | 139  3   1.0263e-04
           1.6189e+06   1.9735e+06   |

1.00e-01   2.4019e+02   2.7072e+02   | 271  1   8.6202e-03
           2.4019e+06   2.7072e+06   |

4.00e-01   2.4019e+02   2.7072e+02   | 278  1   6.8961e-02
           2.4019e+06   2.7072e+06   |

7.00e-01   2.4019e+02   2.7072e+02   | 284  1   1.7240e-02
           2.4019e+06   2.7072e+06   |

1.00e+00   2.4019e+02   2.7072e+02   | 290  1   2.7585e-01
           2.4019e+06   2.7072e+06   |

-----------------------------------------------------------
Final run statistics: 

Number of steps                    = 290
Number of residual evaluations     = 502
Number of Jacobian evaluations     = 65
Number of nonlinear iterations     = 500
Number of error test failures      = 6
Number of nonlinear conv. failures = 18
Number of step solver failures     = 10
[regevs@login03.frontier resolve]$ ./idaHeat2D_resolve

idaHeat2D_ReSolve: Heat equation, serial example problem for IDA
          Discretized heat equation on 2D unit square.
          Zero boundary conditions, polynomial initial conditions.
          Mesh dimensions: 10 x 10        Total system size: 100

Tolerance parameters:  rtol = 0   atol = 1e-08
Constraints set to force all solution components >= 0. 
Linear solver: ReSolve SystemSolver using KLU 
       difference quotient Jacobian

Using CPU backend

IDACalcIC called with input boundary values = 0 

   Output Summary (umax = max-norm of solution) 

  time       umax     k  nst  nni  nje   nre     h       
 .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 
  0.00   9.75461e-01  0    0    0    2     2   5.15e-10 
  0.01   8.24056e-01  5   53   63   23    65   5.55e-04 
  0.02   6.88097e-01  5   69   81   24    83   9.99e-04 
  0.04   4.70961e-01  5   90  106   27   108   1.91e-03 
  0.08   2.16312e-01  5  113  130   27   132   1.72e-03 
  0.16   4.53210e-02  5  137  155   28   157   3.43e-03 
  0.32   1.98864e-03  5  173  193   29   195   6.18e-03 
  0.64   3.83238e-06  5  210  233   31   235   2.22e-02 
  1.28   0.00000e+00  1  227  255   34   257   1.78e-01 
  2.56   0.00000e+00  1  230  258   37   260   1.42e+00 
  5.12   0.00000e+00  1  231  259   38   261   2.85e+00 
 10.24   0.00000e+00  1  232  260   39   262   5.69e+00 

 netf = 2,   ncfn = 0 

@shakedregev

Copy link
Copy Markdown
Collaborator

@pelesh - are we ok with no HIP functionality?

@JeffZ594

JeffZ594 commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

This is the result from ./idaFoodWeb_bnd


Number of species ns: 2     Mesh dimensions: 20 x 20     System size: 800
Tolerance parameters:  rtol = 1e-05   atol = 1e-05
Linear solver: BAND,  Band parameters mu = 40, ml = 40
CalcIC called to correct initial predator concentrations.

-----------------------------------------------------------
  t        bottom-left  top-right    | nst  k      h
-----------------------------------------------------------

0.00e+00   1.0000e+01   1.0000e+01   |   0  0   1.6310e-08
           1.0000e+05   1.0000e+05   |

1.00e-03   1.0318e+01   1.0827e+01   |  32  4   1.0823e-04
           1.0319e+05   1.0822e+05   |

1.00e-02   1.6188e+02   1.9735e+02   | 127  4   1.4203e-04
           1.6189e+06   1.9734e+06   |

1.00e-01   2.4019e+02   2.7072e+02   | 235  1   3.9160e-02
           2.4019e+06   2.7072e+06   |

4.00e-01   2.4019e+02   2.7072e+02   | 238  1   3.1328e-01
           2.4019e+06   2.7072e+06   |

7.00e-01   2.4019e+02   2.7072e+02   | 239  1   6.2655e-01
           2.4019e+06   2.7072e+06   |

1.00e+00   2.4019e+02   2.7072e+02   | 239  1   6.2655e-01
           2.4019e+06   2.7072e+06   |

-----------------------------------------------------------
Final run statistics: 

Number of steps                    = 239
Number of residual evaluations     = 3339
Number of Jacobian evaluations     = 36
Number of nonlinear iterations     = 421
Number of error test failures      = 3
Number of nonlinear conv. failures = 10
Number of step solver failures     = 0 

I will look more into the step solver failures, but I think the error test failures and nonlinear convergence failures are normal, at least compared to the banded solver.

For the heat equation, the failures should be normal. The built in KLU solver has the exact same result as the ReSolve solver.

@shakedregev

Copy link
Copy Markdown
Collaborator

If these failures are expected, we can merge this.

@JeffZ594

JeffZ594 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

My understanding of the step solver failures is that:

At some step of the non linear solver there was a non convergence error.

When the solver gets this error it increments the step solver failure counter. If this error was non-recoverable, the entire solver will terminate.

However, if it is recoverable, the solver will reduce the step size at that step and attempt it again.

I think since the solver converged, it should be fine but I'm not sure if these failures are inherent to solving the food web problem with the KLU solver, or some error in my implementation since there's no benchmark to check the performance of the KLU solver with like with the heat example.

@JeffZ594

Copy link
Copy Markdown
Collaborator Author

@shakedregev @pelesh I believe the pull request is ready to merge now. The fortran check fails because the check fails to install something necessary to start the check, and the build and test checks also fail on the initial fork of SUNDIALS without my changes so I think these failures are outside of my control.

I didn't make the change to the explanation of the foodweb example to have it rendered because I don't think the rendered version of the comment in the .cpp file itself appears anywhere in the documentation, and there is a fully rendered explanation of the problem in a downloadable .pdf in the SUNDIALS documentation.

@shakedregev
shakedregev merged commit 3c2fc96 into resolve-develop Nov 17, 2026
6 of 10 checks passed
@shakedregev
shakedregev deleted the resolve/ida_examples branch November 17, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants