Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EdgeDB query performance statistics #7814

Closed
wants to merge 21 commits into from
Closed

Add EdgeDB query performance statistics #7814

wants to merge 21 commits into from

Conversation

fantix
Copy link
Member

@fantix fantix commented Sep 27, 2024

Refs #7725, this is a quick'n'dirty first attempt the first implementation with a standalone extension that manipulates the PG query source text (and optionally the queryID to map to the cache key). The upper side is that the extension is kinda clean without patching PG codebase (easier to port), the down side is that we lose the raw SQL stats data for all.

  • Fix tests
  • Deal with SQL user permission issue
  • Fix missing hooks
  • Add EdgeQL view
  • Add backend capabilities
  • Handle multitenancy correctly
  • Play nice with persistent cache
  • Complete the reset function parameters
  • Count SQL queries in
  • Add columns that reflect compilation settings
  • Track EdgeQL scripts correctly
  • Fix CI builds
  • Add test

Sample results:

_localdev:main> select sys::QueryStats { * } filter .query not like '%QueryStats%' and .query not like '%sys::%' limit 2;
{
  sys::QueryStats {
    computed_fields: [],
    builtin: false,
    internal: false,
    name: '4ed18524-e5ee-6288-a0b0-9a171bcedd98',
    id: 4ed18524-e5ee-6288-a0b0-9a171bcedd98,
    query: 'SELECT 89',
    query_id: 5679466998613828232,
    query_type: SQL,
    compilation_config: Json("{\"allow_user_specified_id\": [\"false\"]}"),
    protocol_version: (major := 3, minor := 0),
    default_namespace: 'public',
    namespace_aliases: {},
    migration_name: 'initial',
    output_format: {},
    expect_one: {},
    implicit_limit: {},
    inline_typeids: {},
    inline_typenames: {},
    inline_objectids: {},
    plans: 0,
    total_plan_time: <duration>'0:00:00',
    min_plan_time: <duration>'0:00:00',
    max_plan_time: <duration>'0:00:00',
    mean_plan_time: <duration>'0:00:00',
    stddev_plan_time: <duration>'0:00:00',
    calls: 1,
    total_exec_time: <duration>'0:00:00.000001',
    min_exec_time: <duration>'0:00:00.000001',
    max_exec_time: <duration>'0:00:00.000001',
    mean_exec_time: <duration>'0:00:00.000001',
    stddev_exec_time: <duration>'0:00:00',
    rows: 1,
  },
  sys::QueryStats {
    computed_fields: [],
    builtin: false,
    internal: false,
    name: 'bcecb044-dd0b-af72-d014-345032436dfc',
    id: bcecb044-dd0b-af72-d014-345032436dfc,
    query: 'select
    (BB, (<__std__::int64>$0 + <__std__::int64>$1))',
    query_id: -4833294490262589582,
    query_type: EdgeQL,
    compilation_config: Json("{\"allow_bare_ddl\": \"AlwaysAllow\", \"query_cache_mode\": \"Default\", \"__internal_testmode\": false, \"store_migration_sdl\": \"NeverStore\", \"force_database_error\": \"false\", \"apply_access_policies\": true, \"allow_dml_in_functions\": false, \"allow_user_specified_id\": false, \"__internal_query_reflschema\": false, \"__internal_no_apply_query_rewrites\": false}"),
    protocol_version: (major := 2, minor := 0),
    default_namespace: 'bb',
    namespace_aliases: {},
    migration_name: 'm1ndk7eqxayvb5wdfkuv7kslyx2wnhr3e3afcmakqtgo4hjcxwspka',
    output_format: BINARY,
    expect_one: false,
    implicit_limit: 0,
    inline_typeids: false,
    inline_typenames: false,
    inline_objectids: true,
    plans: 6,
    total_plan_time: <duration>'0:00:00.001099',
    min_plan_time: <duration>'0:00:00.000086',
    max_plan_time: <duration>'0:00:00.000227',
    mean_plan_time: <duration>'0:00:00.000183',
    stddev_plan_time: <duration>'0:00:00.000048',
    calls: 10,
    total_exec_time: <duration>'0:00:00.000118',
    min_exec_time: <duration>'0:00:00.000007',
    max_exec_time: <duration>'0:00:00.000019',
    mean_exec_time: <duration>'0:00:00.000012',
    stddev_exec_time: <duration>'0:00:00.000004',
    rows: 0,
  },
}

@elprans
Copy link
Member

elprans commented Sep 27, 2024

the down side is that we lose the raw SQL stats data for all.

But we don't have to, do we? If we don't recognize a query as being a compiler product we can avoid modifying its state.

@fantix fantix force-pushed the query-stats branch 3 times, most recently from e49e222 to eee60ba Compare October 7, 2024 15:09
@fantix fantix force-pushed the query-stats branch 2 times, most recently from d59be85 to fb8ce27 Compare October 7, 2024 21:30
Also:

* Track normalized EdgeQL instead of original
* Track each command in scripts
* Skip info header if backend doesn't support stats
* Add back debug.edgeql_text_in_sql
* Exclude info header from the SQL hash
@fantix fantix marked this pull request as ready for review October 8, 2024 00:52
@fantix fantix requested a review from msullivan October 8, 2024 00:52
@fantix fantix requested a review from elprans October 8, 2024 00:52
static uint64
edbss_extract_stmt_info(char **query_str) {
if (strncmp(*query_str, EDB_STMT_MAGIC_PREFIX, strlen(EDB_STMT_MAGIC_PREFIX)) == 0) {
EdbStmtInfoSemState state;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What version of C will this be compiled with? Because you could just do this in C99:

Suggested change
EdbStmtInfoSemState state;
EdbStmtInfoSemState state = {0};

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, mostly assuming C99 in #7869 now

Comment on lines +1588 to +1589
setting_filter=lambda v: v.name in spec
and spec[v.name].affects_compilation,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make sure nothing secret goes in also?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only compilation setting that may include free text is force_database_error, which shouldn't have secrets. I can also add a double-safe to filter out extension configs in #7869.

Comment on lines +1570 to +1574
sql_info: Dict[str, Any] = {}
if (
not ctx.bootstrap_mode
and ctx.backend_runtime_params.has_stat_statements
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want whether we collect this to also depend on a config flag? This seems like a lot of extra stuff to store unconditionally.
(Curious about @elprans's opinion on this)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered in the meeting: we do need a config to turn the tracking on/off.

Follow-up question: do we want to make this config affects_compilation (thus controlling the compiled SQL with or without the magic JSON-comment prefix, but impacting query cache), or a runtime session config that the extension can read and behave (but we end up with prefixed SQLs all the time)?

@fantix fantix mentioned this pull request Oct 16, 2024
2 tasks
@fantix fantix closed this Nov 23, 2024
@fantix fantix deleted the query-stats branch November 23, 2024 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants