Tempus commits on 1/10 causing slow-downs in Albany dynamic problems

[ikalash]

Continuing the discussion of the following Albany issue https://github.com/gahansen/Albany/issues/243 which is really a Trilinos issue.

Some recent Tempus commits have slowed down substantially (5x) several Albany dynamic problems that utilize Tempus. Here are some sample timings for the Schwarz_Alternating_Dynamics_Cubes test in Albany along with Trilinos commit tags:

6ab3036 : 19.14 sec
1d0aae2 : 104.95 sec

This means that the commit 1d0aae2 (Tempus: move integrator output to observer) started this problem.

Here is some sample verbose output from the Albany CDash:

1/10 run: http://cdash.sandia.gov/CDash-2-3-0/testDetails.php?test=3387873&build=65282
1/11 run (slower one): http://cdash.sandia.gov/CDash-2-3-0/testDetails.php?test=3390813&build=65335

@trilinos/tempus

[ikalash]

I have dug into this a little bit and the problem has to do with there being many more observer calls. I have put print statements in Piro_ObserverToTempusIntegrationObserverAdapter_Def.hpp, observeEndIntegrator routine, to provide an example. Before the slow-downs, there are 836 calls to this routine; after, there are 175978 calls (for the serial alternating Schwarz dynamic cubes problem in Albany which we are debugging). Here is some sample output:

Before slowdown:

Time integration complete.

IKT observeEndIntegrator
IKT observeTimeStep
IKT observing solution time = 0.05
T final actual: 5.000e-02

F) Check the solution to the forward problem ...

After slowdown:

Time integration complete.

IKT observeEndIntegrator
IKT observeTimeStep
IKT observing solution time = 0.05
IKT observeEndIntegrator
IKT observeTimeStep
IKT observing solution time = 0.05
T final actual: 5.000e-02

F) Check the solution to the forward problem ...

For some reason, the observing of the solution gets called twice now after a time-integration is complete.

[sconde]

This was fixed by #2189