Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dados] br_bd_diretorios_brasil.etnia_indigena #1087

Merged
merged 3 commits into from
Jan 26, 2022

Conversation

gustavoairestiago
Copy link
Contributor

No description provided.

@github-actions
Copy link

Data Check Report

Table br_bd_diretorios_brasil/etnia_indigena

✔️ Check if table exists

SELECT "etnia_indigena" IN (
    SELECT `table_name`
    FROM `basedosdados-dev.br_bd_diretorios_brasil.INFORMATION_SCHEMA.TABLES`
) AS success

❌ Check if any column is completely null (BaseDosDados Exception)

WITH
    n_nulls AS (
        SELECT col_name, COUNT(1) nulls_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t,
        UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r'"(\w+)":null')) col_name
        GROUP BY col_name
    ),
    n_total AS (
        SELECT COUNT(*) total_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t
    )
SELECT col_name, nulls_count / total_count null_percent
FROM n_nulls, n_total

❌ Check if identifying column has unique values (BaseDosDados Exception)

SELECT 
  COUNT(
    DISTINCT CONCAT(
      IFNULL(SAFE_CAST(id_etnia_indigena AS STRING), " "), "&",
      "EOF"
    )
  ) / COUNT(*) unique_percentage 
FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t

Action Output

@github-actions
Copy link

Data Check Report

Table br_bd_diretorios_brasil/etnia_indigena

✔️ Check if table exists

SELECT "etnia_indigena" IN (
    SELECT `table_name`
    FROM `basedosdados-dev.br_bd_diretorios_brasil.INFORMATION_SCHEMA.TABLES`
) AS success

✔️ Check if any column is completely null

WITH
    n_nulls AS (
        SELECT col_name, COUNT(1) nulls_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t,
        UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r'"(\w+)":null')) col_name
        GROUP BY col_name
    ),
    n_total AS (
        SELECT COUNT(*) total_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t
    )
SELECT col_name, nulls_count / total_count null_percent
FROM n_nulls, n_total

✔️ Check if identifying column has unique values (100.00)

SELECT 
  COUNT(
    DISTINCT CONCAT(
      IFNULL(SAFE_CAST(id_etnia_indigena AS STRING), " "), "&",
      "EOF"
    )
  ) / COUNT(*) unique_percentage 
FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t

Action Output

@github-actions
Copy link

Data Check Report

Table br_bd_diretorios_brasil/etnia_indigena

✔️ Check if table exists

SELECT "etnia_indigena" IN (
    SELECT `table_name`
    FROM `basedosdados-dev.br_bd_diretorios_brasil.INFORMATION_SCHEMA.TABLES`
) AS success

✔️ Check if any column is completely null

WITH
    n_nulls AS (
        SELECT col_name, COUNT(1) nulls_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t,
        UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r'"(\w+)":null')) col_name
        GROUP BY col_name
    ),
    n_total AS (
        SELECT COUNT(*) total_count
        FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t
    )
SELECT col_name, nulls_count / total_count null_percent
FROM n_nulls, n_total

✔️ Check if identifying column has unique values (100.00)

SELECT 
  COUNT(
    DISTINCT CONCAT(
      IFNULL(SAFE_CAST(id_etnia_indigena AS STRING), " "), "&",
      "EOF"
    )
  ) / COUNT(*) unique_percentage 
FROM `basedosdados-dev.br_bd_diretorios_brasil.etnia_indigena` t

Action Output

- name: nome
bigquery_type: string
description: Nome da etnia ndígena
temporal_coverage:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indígena

@lucasnascm lucasnascm merged commit a391849 into master Jan 26, 2022
@lucasnascm lucasnascm deleted the br_bd_diretorios_brasil.etnia_indigena branch January 26, 2022 10:51
d116626 pushed a commit that referenced this pull request Feb 12, 2022
* [dados-bot] br_ms_vacinacao_covid19 (2022-01-23) (#1086)

Co-authored-by: terminal_name <github_email>

* [dados] br_bd_diretorios_brasil.etnia_indigena (#1087)

* Sobe diretorio etnia_indigena

* Update table_config.yaml

* Update table_config.yaml

* feat: conform Metadata's schema to new one

* fix: conform yaml generation to new schema

* fix: delete test_dataset folder

Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>
d116626 added a commit that referenced this pull request Feb 12, 2022
* [infra] fix dataset_config.yaml folder path (#1067)

* feat(infra) merge master

* [infra] conform Metadata to new metadata changes (#1093)

* [dados-bot] br_ms_vacinacao_covid19 (2022-01-23) (#1086)

Co-authored-by: terminal_name <github_email>

* [dados] br_bd_diretorios_brasil.etnia_indigena (#1087)

* Sobe diretorio etnia_indigena

* Update table_config.yaml

* Update table_config.yaml

* feat: conform Metadata's schema to new one

* fix: conform yaml generation to new schema

* fix: delete test_dataset folder

Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>

Co-authored-by: Ricardo Dahis <[email protected]>
Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>
lucascr91 added a commit that referenced this pull request Mar 14, 2022
* feat(infra): create version 1.6.2

* feat(infra): create version 1.6.2

* feat(infra): create version 1.6.2

* [infra] python-v1.6.2 (#1089)

* [infra] fix dataset_config.yaml folder path (#1067)

* feat(infra) merge master

* [infra] conform Metadata to new metadata changes (#1093)

* [dados-bot] br_ms_vacinacao_covid19 (2022-01-23) (#1086)

Co-authored-by: terminal_name <github_email>

* [dados] br_bd_diretorios_brasil.etnia_indigena (#1087)

* Sobe diretorio etnia_indigena

* Update table_config.yaml

* Update table_config.yaml

* feat: conform Metadata's schema to new one

* fix: conform yaml generation to new schema

* fix: delete test_dataset folder

Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>

Co-authored-by: Ricardo Dahis <[email protected]>
Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>

* feat(infra): 1.6.2a3 version

* feat(infra): 1.6.2a3 version

* fix(ingra): edit partitions and update_locally

* feat(infra): update_columns new fields and accepts local files

* [infra] option to make dataset public (#1020)

* feat(infra): option to make dataset public

* feat(infra): fix None data

* fix(infra): roll back

* fix(infra): fix retry in storage upload

* fix(infra): add option to dataset data location

* feat(infra): make staging dataset not public

* feat(infra): make staging dataset not public

* fix(infra): change bd version in actions

* fix(infra): add toml to install in ci

* fix(infra): remove a forget print

* fix(infra): fix location location

* fix(infra): fix dataset description

* feat(infra): bump-version

* feat(infra): temporal coverage as list in update_columns

* feat(infra): add new parameters to cli

* feat(infra): fix cli options

* [infra] change download functions to consume CKAN endpoints #1129  (#1130)

* [infra] add function to wrap bd_dataset_search endpoint

* Update download.py

* [infra] modify list_datasets function to consume CKAN endpoint

* [infra] fix list_dataset function to include limit and remove order_by

* [infra] change function list_dataset_tables to use CKAN endpoint

* [infra] apply PEP8 to list_dataset_tables and respective tests

* add get_dataset_description, get_table_description, get_table_columns

* [infra] fix dataset_config.yaml folder path (#1067)

* feat(infra) merge master

* fix files organization to match master

* remove download.py

* remove test_download

* Delete test_download.py

* remove test files

* remove test_download.py

* remove test_download.py

* remove test_download.py

* remove test_download.py

* add tests metadata

* remove test_download.py

* remove unused imports

* [infra] add _safe_fetch and get_table_size functions

Co-authored-by: lucascr91 <[email protected]>

* fix(infra): add a empty list to not a partition

* [infra] Adiciona suporte a Avro e Parquet (#1145)

* adiciona suporte a Avro e Parquet para upload

* Adds test for source formats

* [infra] update tests for avro, parquet, and csv upload

Co-authored-by: Gabriel Gazola Milan <[email protected]>
Co-authored-by: Isadora Bugarin  <[email protected] >
Co-authored-by: lucascr91 <[email protected]>

* [infra] Feedback messages in upload methods [issue #1059] (#1085)

* Creating dataclass config

* Success messages - create and update (table.py) using loguru

* feat: improve log level control

* refa: move logger config to Base.__init__

* Improving log level control

* Adjusting log level control function in base.py

* Fixing repeated 'DELETE' messages everytime Table is replaced.

* Importing 'dataclass' from 'dataclasses' to make config work.

* Fixing repeated 'UPDATE' messages inside other functions.

* Defining a new script message format.

* Definng standard log messages for 'dataset.py' functions

* Definng standard log messages for 'storage.py' functions

* Definng standard log messages for 'table.py' functions

* Definng standard log messages for 'metadata.py' functions

* Adds standard configuration to billing_project_id in download.py

* Configuring billing_project_id in download.py

* Configuring config_path in base.py

Co-authored-by: Guilherme Salustiano <[email protected]>
Co-authored-by: Isadora Bugarin <[email protected]>

* update toml

Co-authored-by: Ricardo Dahis <[email protected]>
Co-authored-by: Lucas Moreira <[email protected]>
Co-authored-by: Gustavo Aires Tiago <[email protected]>
Co-authored-by: lucascr91 <[email protected]>
Co-authored-by: Isadora Bugarin <[email protected]>
Co-authored-by: Gabriel Gazola Milan <[email protected]>
Co-authored-by: Isadora Bugarin  <[email protected] >
Co-authored-by: Guilherme Salustiano <[email protected]>
Co-authored-by: Isadora Bugarin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Data to add on BigQuery data-check-trigger
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants