Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor SmartRedis build system #341

Merged
merged 52 commits into from
Jun 2, 2023
Merged

Refactor SmartRedis build system #341

merged 52 commits into from
Jun 2, 2023

Conversation

billschereriii
Copy link
Contributor

@billschereriii billschereriii commented May 19, 2023 •
edited
Loading

Refactor the SmartRedis build system as planned in the design document:

  • add support for SR_BUILD and SR_LINK Make variables
  • add missing make dependencies
  • eliminate use of scripts for building the SmartRedis libraries
  • eliminate unnecessary deletion of SmartRedis builds, allowing repeat builds to be much faster
  • update examples, test cases, CI/CD to work with new build system
  • update Intel compiler in CI/CD to move away from deprecated compiler that will EOL in the next few months

@billschereriii billschereriii changed the title Moarmake Refactor SmartRedis build system May 19, 2023
@billschereriii billschereriii self-assigned this May 19, 2023
@billschereriii billschereriii added type: refactor Issues focused on refactoring existing code area: build Issues related to builds, makefiles, installs, etc area: test Issues related to the test suite area: python Issues related to the Python Client area: C++ Issues related to the C++ client area: C Issues related to the C client area: fortran issues related to the Fortran Client area: third-party Issues related to depencies and third-party package integrations area: examples Issues related to tutorials and examples area: CI/CD Issues related to continuous integration and deployment labels May 19, 2023
@billschereriii billschereriii marked this pull request as ready for review May 23, 2023 22:16
MattToast
MattToast reviewed May 30, 2023
View reviewed changes
Copy link
Member

@MattToast MattToast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python side of things looks great!! I was able to build a wheel using setup.py independent of the Makefile and everything is ending up where I expected it to in a venv/conda env! Simply withholding approval to give other reviewers a chance to look over

ashao
ashao requested changes May 30, 2023
View reviewed changes
Copy link
Collaborator

@ashao ashao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good Bill and I tried to break it on a couple of different machines. There are a couple of oddities/minor optimizations left:

  • hiredis and redis++ are always rebuilt with make lib[-with-fortran]. This seems to force the rebuild of a bunch of the C++ code in SmartRedis
  • libhiredis is still showing up in the install directory
  • I thought that at one point you were thinking of having a separate directory under install based on SR_BUILD and SR_LINK. Was this idea scrapped for simplicity?

@billschereriii
Copy link
Contributor Author

Looks really good Bill and I tried to break it on a couple of different machines. There are a couple of oddities/minor optimizations left:

  • hiredis and redis++ are always rebuilt with make lib[-with-fortran]. This seems to force the rebuild of a bunch of the C++ code in SmartRedis

I suspect this was a miss on reverting moving the libraries back to the install directory. Will fix

  • libhiredis is still showing up in the install directory
  • I thought that at one point you were thinking of having a separate directory under install based on SR_BUILD and SR_LINK. Was this idea scrapped for simplicity?

This had some nasty side effects and had to be rolled back. Once we get some stuff cleaned up inside the library, we'll get them out of the install directory

@billschereriii billschereriii requested a review from ashao May 31, 2023 22:49
ashao
ashao approved these changes May 31, 2023
View reviewed changes
Copy link
Collaborator

@ashao ashao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the comments. Can confirm that partial rebuilds work as expected. Please change that one line in the redis++ recipe and everything is good to go from my end.

mellis13
mellis13 requested changes Jun 1, 2023
View reviewed changes
Copy link
Collaborator

@mellis13 mellis13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR -- really nice simplification and improvement of the make system. Just a couple of requested changes, but nothing major.

@MattToast MattToast self-requested a review June 1, 2023 15:08
MattToast
MattToast approved these changes Jun 1, 2023
View reviewed changes
Copy link
Member

@MattToast MattToast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!!

mellis13
mellis13 approved these changes Jun 2, 2023
View reviewed changes
Copy link
Collaborator

@mellis13 mellis13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed one potential typo -- otherwise LGTM. Thanks for the hard work on this PR!

@billschereriii billschereriii merged commit 94f7188 into CrayLabs:develop Jun 2, 2023
@billschereriii billschereriii deleted the moarmake branch June 2, 2023 15:44
ashao pushed a commit to ashao/SmartRedis that referenced this pull request Jul 6, 2023
@billschereriii @ashao
Refactor SmartRedis build system (CrayLabs#341)
f2eb9ad 
Refactor the SmartRedis build system as planned in the design document:
 . add support for SR_BUILD and SR_LINK Make variables
 . add missing make dependencies
 . eliminate use of scripts for building the SmartRedis libraries
 . eliminate unnecessary deletion of SmartRedis builds, allowing repeat builds to be much faster
 . update examples, test cases, CI/CD to work with new build system
 . update Intel compiler in CI/CD to move away from deprecated compiler that will EOL in the next few months
 . make Python and Fortran optional parts of the build, not built by default

[ committed by @billschereriii ]
[ reviewed by @ashao @MattToast @mellis13 ]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashao ashao ashao approved these changes

@mellis13 mellis13 mellis13 approved these changes

@MattToast MattToast MattToast approved these changes

Assignees

@billschereriii billschereriii

Labels
area: build Issues related to builds, makefiles, installs, etc area: C++ Issues related to the C++ client area: C Issues related to the C client area: CI/CD Issues related to continuous integration and deployment area: examples Issues related to tutorials and examples area: fortran issues related to the Fortran Client area: python Issues related to the Python Client area: test Issues related to the test suite area: third-party Issues related to depencies and third-party package integrations type: refactor Issues focused on refactoring existing code
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@billschereriii @ashao @mellis13 @MattToast