Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI] Improve ray status for placement groups #18289

Merged
merged 6 commits into from
Sep 14, 2021

Conversation

rkooo567
Copy link
Contributor

@rkooo567 rkooo567 commented Sep 1, 2021

Why are these changes needed?

  • For no node type, we format resources in a prettier way. It also displays how many of resources are reserved by pgs.
  • For usage, it includes usage / total of placement group. Previously, it only prints the "usage" of placement group, which was confusing

After

Node status
------------------------------------------------------------
 1 node(s) with resources:
  CPU: 3.0
  object_store_memory: 157286400.0
  GPU: 66.0 (66.0 reserved in placement groups)
  pg_custom: 66.0 (66.0 reserved in placement groups)
  node:192.168.1.3: 1.0
  memory: 8279552000.0

Resources
------------------------------------------------------------
Usage:
 0.0/3.0 CPU
 66.0/66.0 GPU (0.0/66.0 used by placement groups)
 0.00/7.711 GiB memory
 0.00/0.146 GiB object_store_memory
 66.0/66.0 pg_custom (0.0/66.0 used by placement groups)

Before

Node status
------------------------------------------------------------
 1 node(s) with resources: {'GPU_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_3_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_1_326f40a13dccec23e14b78fc80262f71': 1.0, 'GPU_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'pg_custom_group_9da53bf2a1710a4433453f8e072ef810': 5.0, 'pg_custom_group_6e150e681d33f39699685637ca0fdc5b': 5.0, 'bundle_group_3_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'GPU_group_0_4acd757a53c53f736b72800076076b53': 1.0, 'bundle_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_3_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_2_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_3_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_2_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_1_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'bundle_group_080c9d088f3049e7bb4139a4c9fdba1a': 5000.0, 'bundle_group_6e150e681d33f39699685637ca0fdc5b': 5000.0, 'bundle_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_efb63fec901c967f94cb97a3928d52cf': 5.0, 'pg_custom_group_e17ecaca21cc323f91f83f01a5d3f376': 5.0, 'GPU_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_1_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_e6314fb8a23d64a2df3b6c92a2d03821': 5.0, 'GPU_group_2_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_3_4acd757a53c53f736b72800076076b53': 1000.0, 'GPU_group_4_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_868294ac2d7a48c9447a16e7e915bd54': 5.0, 'GPU_group_0_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4acd757a53c53f736b72800076076b53': 5.0, 'bundle_group_4_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'pg_custom_group_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'pg_custom_group_4_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_326f40a13dccec23e14b78fc80262f71': 5.0, 'GPU_group_0_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_1_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_1_4acd757a53c53f736b72800076076b53': 1.0, 'CPU': 3.0, 'bundle_group_4_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_1_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'GPU_group_4_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_3_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_0_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_868294ac2d7a48c9447a16e7e915bd54': 5.0, 'GPU_group_3_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_0_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_d6975df5cdeda9b201fb62beb6311494': 5.0, 'pg_custom_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_4_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_e17ecaca21cc323f91f83f01a5d3f376': 5000.0, 'pg_custom_group_2_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'GPU_group_76cfbe45659a1252fd4fe2f695ddd026': 5.0, 'pg_custom_group_1_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_1_326f40a13dccec23e14b78fc80262f71': 1000.0, 'GPU_group_0_efb63fec901c967f94cb97a3928d52cf': 1.0, 'GPU_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_d6975df5cdeda9b201fb62beb6311494': 5.0, 'GPU_group_1_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_0_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_2_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_080c9d088f3049e7bb4139a4c9fdba1a': 5.0, 'pg_custom_group_b47fe6e409dab4c127d34d05cf29017a': 5.0, 'object_store_memory': 157286400.0, 'bundle_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'GPU_group_3_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_1_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'pg_custom_group_2_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'bundle_group_d6975df5cdeda9b201fb62beb6311494': 5000.0, 'bundle_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'bundle_group_0_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'bundle_group_4_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_3_9da53bf2a1710a4433453f8e072ef810': 1.0, 'pg_custom_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'pg_custom_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5.0, 'GPU': 66.0, 'pg_custom_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'pg_custom_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_3_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_326f40a13dccec23e14b78fc80262f71': 5.0, 'GPU_group_3_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_4_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_0_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_2_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_3_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_4_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'pg_custom': 66.0, 'bundle_group_1_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_b47fe6e409dab4c127d34d05cf29017a': 5000.0, 'bundle_group_2_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_080c9d088f3049e7bb4139a4c9fdba1a': 5.0, 'bundle_group_efb63fec901c967f94cb97a3928d52cf': 5000.0, 'bundle_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'bundle_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'bundle_group_0_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_0_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'pg_custom_group_2_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_2_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_e6314fb8a23d64a2df3b6c92a2d03821': 5000.0, 'pg_custom_group_4_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_0_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_4_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_26addd2b76ca2377438e18791ea0b8f4': 1000.0, 'GPU_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_1_26addd2b76ca2377438e18791ea0b8f4': 1000.0, 'GPU_group_1_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_2_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_3_6e150e681d33f39699685637ca0fdc5b': 1.0, 'bundle_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'GPU_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_0_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_3_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'pg_custom_group_2_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_0_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_e6314fb8a23d64a2df3b6c92a2d03821': 5.0, 'GPU_group_1_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_2_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_1_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'pg_custom_group_4_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_0_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'bundle_group_2_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'pg_custom_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'bundle_group_3_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'pg_custom_group_4_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'pg_custom_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_4_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_9da53bf2a1710a4433453f8e072ef810': 5.0, 'pg_custom_group_1_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_1_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_4_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'bundle_group_4_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_3_326f40a13dccec23e14b78fc80262f71': 1000.0, 'GPU_group_4acd757a53c53f736b72800076076b53': 5.0, 'GPU_group_1_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_9da53bf2a1710a4433453f8e072ef810': 5000.0, 'pg_custom_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'GPU_group_efb63fec901c967f94cb97a3928d52cf': 5.0, 'pg_custom_group_3_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_0_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'bundle_group_1_4acd757a53c53f736b72800076076b53': 1000.0, 'bundle_group_0_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'bundle_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'bundle_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'pg_custom_group_1_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_3_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_76cfbe45659a1252fd4fe2f695ddd026': 5.0, 'GPU_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_2_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_2_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'GPU_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'GPU_group_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'bundle_group_2_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_4_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_2_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_2_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_0_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_1_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_4_4acd757a53c53f736b72800076076b53': 1000.0, 'bundle_group_2_4acd757a53c53f736b72800076076b53': 1000.0, 'pg_custom_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'GPU_group_3_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4acd757a53c53f736b72800076076b53': 5000.0, 'bundle_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'pg_custom_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_3_efb63fec901c967f94cb97a3928d52cf': 1.0, 'GPU_group_e17ecaca21cc323f91f83f01a5d3f376': 5.0, 'pg_custom_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_0_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_1_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'GPU_group_2_326f40a13dccec23e14b78fc80262f71': 1.0, 'node:192.168.1.3': 1.0, 'GPU_group_4_4acd757a53c53f736b72800076076b53': 1.0, 'bundle_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'GPU_group_4_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_1_4acd757a53c53f736b72800076076b53': 1.0, 'memory': 8279552000.0, 'bundle_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'pg_custom_group_2_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'GPU_group_2_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_0_4acd757a53c53f736b72800076076b53': 1000.0, 'pg_custom_group_3_d6975df5cdeda9b201fb62beb6311494': 1.0, 'GPU_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_3_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_1_9da53bf2a1710a4433453f8e072ef810': 1.0, 'pg_custom_group_4_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_0_326f40a13dccec23e14b78fc80262f71': 1.0, 'GPU_group_0_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_0_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_b47fe6e409dab4c127d34d05cf29017a': 5.0, 'GPU_group_1_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_3_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5.0, 'bundle_group_1_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'GPU_group_4_9da53bf2a1710a4433453f8e072ef810': 1.0, 'bundle_group_868294ac2d7a48c9447a16e7e915bd54': 5000.0, 'bundle_group_326f40a13dccec23e14b78fc80262f71': 5000.0, 'GPU_group_6e150e681d33f39699685637ca0fdc5b': 5.0, 'pg_custom_group_0_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_2_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_3_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'GPU_group_1_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5000.0, 'bundle_group_76cfbe45659a1252fd4fe2f695ddd026': 5000.0, 'GPU_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1.0}

Resources
------------------------------------------------------------
Usage:
 0.0/3.0 CPU
 66.0/66.0 GPU
 0.00/7.711 GiB memory
 0.00/0.146 GiB object_store_memory
 66.0/66.0 pg_custom

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@rkooo567
Copy link
Contributor Author

rkooo567 commented Sep 1, 2021

Btw, this is still confusing;

66.0/66.0 GPU (0.0/66.0 used by placement groups)

We uses 66 GPUs because they are all reserved by placement groups. I was thinking

0.0/0.0 GPU
0.0/66.0 GPU (reserved by placement groups)

this as an alternative format, but haven't implemented it because this can also be confusing...

@ericl
Copy link
Contributor

ericl commented Sep 2, 2021

test_cli failing

@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 2, 2021
@ericl
Copy link
Contributor

ericl commented Sep 2, 2021

66.0/66.0 GPU (0.0/66.0 used by placement groups)

This doesn't grammatically make sense. Does it mean 66 is reserved in placement groups, but 0 is used within the group?

Copy link
Contributor

@ericl ericl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment

Copy link
Contributor

@ericl ericl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

@rkooo567
Copy link
Contributor Author

rkooo567 commented Sep 2, 2021

@ericl

This doesn't grammatically make sense. Does it mean 66 is reserved in placement groups, but 0 is used within the group?

Yes. Right now the representation is

 66.0/66.0 GPU (X reserved in placement groups)

where X is "used" resources in placement groups.

and this will become

66.0/66.6 GPU (X/66.0 used in placement groups).

I agree this rep is a little weird. Another alternative is

  1. Separate out resources used by pgs
0.0/0.0 GPU # Original resources
X/66.0 GPU (reserved by placement groups) # PG reserved resources

or

  1. Use total reserved amount in the paranthesis and instead use the real usage
Y/66.0 GPU (66.0 reserved in placement groups) # Y == used (66) - pg_reserved (66) + pg_used (X)

One con of this option is that when your usages are mixed up with placement group usages, it is hard to read. Imagine

X CPU
Y Reserved
Z Used by pg

X + Z / 66.0 (Y reserved in placement groups) # but it is hard to know what are used by pg and what are not

@ericl
Copy link
Contributor

ericl commented Sep 3, 2021 via email

@rkooo567
Copy link
Contributor Author

rkooo567 commented Sep 3, 2021

@ericl @scv119 I'd like to get some final thoughts; I am thinking this;

66 CPU
56 pg reserved
4 used by CPU
40 used by pg reserved

# Current
60.0/66.0 (40 reserved in placement groups)

# New proposal
44.0/66.0 CPU (40 used, 56.0 reserved in placement groups )

Thought?

@ericl
Copy link
Contributor

ericl commented Sep 3, 2021 via email

@rkooo567 rkooo567 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 10, 2021
@rkooo567 rkooo567 requested a review from ericl September 10, 2021 01:34
@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 10, 2021
@rkooo567 rkooo567 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 14, 2021
@rkooo567 rkooo567 requested a review from ericl September 14, 2021 17:10
@ericl ericl merged commit 31e1638 into ray-project:master Sep 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants