Allow I/O of multi-tile arrays and fields with undistributed dimensions

[billsacks]

This PR extends the IO routine undist_arraycreate_alldist to work with multi-tile arrays. This allows I/O of multi-tile arrays and fields with undistributed dimensions.

The implementation follows suggestions from @theurich and @oehmke .

I have tested this using billsacks/esmfprojects-multi_tile_io@eea516b. This writes a 6-tile field with 2 ungridded dimensions (dimensions 1 & 3 are gridded, 2 & 4 ungridded). I have manually inspected the output to verify that this field is being written correctly.

[billsacks]

I am marking this as a draft PR. Things still needed:

• I will add a unit test of this functionality
• I'd like @uturuncoglu to test this and make sure this works for him

[theurich]

@billsacks, this looks exactly as expected! While going through the code I made a tiny code simplification that I just pushed. Nothing critical, but since we are in there anyway, might as well.
One thing I was going back and forth was the question whether both single- and multi-tile cases should call into the same DistGrid::create(), which we could do. It would eliminate having to do the conditional branching. On the other hand, there are a few extra things to do for multi-tile, also down in the DistGrid::create() itself. So it might just be smarter to do the branching on this level as you have it. That is where I came out thinking about it now, so I think what you have is the right way to go.

[billsacks]

Thanks for the change @theurich . I had some local changes to update to latest develop (in an attempt to try to get things working with the issues in #112 ), so I ended up cherry-picking your changes onto this branch and then force pushing. (Let me know if this messes you up in any way, in which case I can undo this and redo it differently.)

[theurich]

@billsacks - good with me.

[billsacks]

One thing I was going back and forth was the question whether both single- and multi-tile cases should call into the same DistGrid::create(), which we could do. It would eliminate having to do the conditional branching. On the other hand, there are a few extra things to do for multi-tile, also down in the DistGrid::create() itself. So it might just be smarter to do the branching on this level as you have it. That is where I came out thinking about it now, so I think what you have is the right way to go.

Thanks for your thoughts on this point. I wondered about this when I was doing the implementation. I hadn't been totally sure if it would work to use the same call for the single-tile case, but even given that it would, I saw some advantage to keeping the standard single-tile case simple and easier to understand... so, basically, the same feelings you have.

[billsacks]

I have added unit tests of the new code, and it sounds like this is working for @uturuncoglu . I'm still working on the issue that @uturuncoglu identified – that we get warning messages like

20221213 104527.508 WARNING          PET144 ESMCI_PIO_Handler.C:1567 ESMCI::PIO_Handler::isOpen()   File, ufs.cpld.lnd.ini.2000-01-01-00000.tile1.nc, closed by PIO

However, I have found that this warning exists on develop, too. I'm looking into it to see if it's something caused by my earlier multi-tile I/O work, but I feel like the fix for that can (and maybe should) be separate from this PR.

So @theurich please let me know if this is okay to merge from your perspective.

[theurich]

@billsacks - please go ahead bringing this into develop.

[uturuncoglu]

Hi All, Maybe after merging this and having nightly testing, we could create a new beta snapshot. If we want to push this new I/O layer to NoahMP component. The new ESMF version needs to be installed all the platforms, which UFS is tested. It could take time. —ufuk

…

On Dec 15, 2022, at 3:53 PM, Bill Sacks ***@***.***> wrote: I have added unit tests of the new code, and it sounds like this is working for @uturuncoglu <https://github.com/uturuncoglu> . I'm still working on the issue that @uturuncoglu <https://github.com/uturuncoglu> identified – that we get warning messages like 20221213 104527.508 WARNING PET144 ESMCI_PIO_Handler.C:1567 ESMCI::PIO_Handler::isOpen() File, ufs.cpld.lnd.ini.2000-01-01-00000.tile1.nc, closed by PIO However, I have found that this warning exists on develop, too. I'm looking into it to see if it's something caused by my earlier multi-tile I/O work, but I feel like the fix for that can (and maybe should) be separate from this PR. So @theurich <https://github.com/theurich> please let me know if this is okay to merge from your perspective. — Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMBRZACQTKI2WR3UAK2XLWNOONNANCNFSM6AAAAAAS2IWLAY>. You are receiving this because you were mentioned.

[theurich]

I agree about the snapshot. My regular routine with respect to snapshots now is that I look on Saturday how the nightlies did (Fri->Sat). When looking good, I put a new tag onto develop. By Monday then we typically have a pretty complete test column for that new tag.
Assuming @billsacks merged this PR today, we'll have a snapshot tag with it by Monday following the outlined procedure.

[theurich]

@uturuncoglu - just to let you know that v8.5.0b10 includes the multi-tile PR merge. There were some hicups in the summarizer of the nightlies over the weekend, but I addressed that issue this morning. The summary table suggests that v8.5.0b10 is good to be used.

[uturuncoglu]

@theurich Thank. That is great. I'll open an issue in UFS side to have v8.5.0b10 at least on Orion and Cheyenne.

[uturuncoglu]

@theurich You can find the issue in the following link as for your reference,

ufs-community/ufs-weather-model#1542