[Sharepoint Online] Improve performance of Identity Sync #1204
Comments
artem-shelkovnikov
changed the title
8 tasks
|
☝️ if that PR's approach goes in, I'll need to also:
|
This was referenced Jul 10, 2023
seanstory
added a commit
to elastic/kibana
that referenced
this issue
Jul 10, 2023
## Summary Part of elastic/connectors#1204 This adds a configuration for the sharepoint online native connector to allow it to choose between approaches when fetching users for DLS. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jul 10, 2023
## Summary Part of elastic/connectors#1204 This adds a configuration for the sharepoint online native connector to allow it to choose between approaches when fetching users for DLS. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit fd2ecdf)
kibanamachine
referenced
this issue
in elastic/kibana
Jul 10, 2023
# Backport This will backport the following commits from `main` to `8.9`: - [Add RCF to pick user fetching strategy (#161546)](#161546) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sean Story","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-10T22:06:09Z","message":"Add RCF to pick user fetching strategy (#161546)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/connectors-python/issues/1204\r\n\r\nThis adds a configuration for the sharepoint online native connector to\r\nallow it to choose between approaches when fetching users for DLS.\r\n\r\n\r\n### Checklist\r\n\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fd2ecdfb27985044aa6e0c85662f23d5b000c255","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:EnterpriseSearch","v8.9.0","v8.10.0"],"number":161546,"url":"https://github.com/elastic/kibana/pull/161546","mergeCommit":{"message":"Add RCF to pick user fetching strategy (#161546)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/connectors-python/issues/1204\r\n\r\nThis adds a configuration for the sharepoint online native connector to\r\nallow it to choose between approaches when fetching users for DLS.\r\n\r\n\r\n### Checklist\r\n\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fd2ecdfb27985044aa6e0c85662f23d5b000c255"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/161546","number":161546,"mergeCommit":{"message":"Add RCF to pick user fetching strategy (#161546)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/connectors-python/issues/1204\r\n\r\nThis adds a configuration for the sharepoint online native connector to\r\nallow it to choose between approaches when fetching users for DLS.\r\n\r\n\r\n### Checklist\r\n\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"fd2ecdfb27985044aa6e0c85662f23d5b000c255"}}]}] BACKPORT--> Co-authored-by: Sean Story <[email protected]>
|
closing this, though https://github.com/elastic/enterprise-search-team/issues/5162 is a follow-up, because this change was insufficient. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As reported, syncing identities takes a long time:
Minor triaging showed the following:
We fetch members for each group for each user but instead we can fetch groups and its members with $expand query: https://graph.microsoft.com/v1.0/groups?$expand=members . It can reduce number of calls to Sharepoint dramatically.
Right now the number of calls is gonna be:
For each user (N users)
For each group (M groups per user)
Fetch members, so
N * M - for 300 users and 4 groups each it’s 1200 requests.
For 42000 users in the screenshot and assuming at least 10 groups, it's gonna be 420.000 requests each costing 5 resource units.
We need to optimize this logic to reduce sync time.
The text was updated successfully, but these errors were encountered: