Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[infra] Adiciona suporte a Avro e Parquet #1100

Closed
wants to merge 1 commit into from
Closed

[infra] Adiciona suporte a Avro e Parquet #1100

wants to merge 1 commit into from

Conversation

gabriel-milan
Copy link
Member

@gabriel-milan gabriel-milan commented Feb 7, 2022 •
edited by JoaoCarabetta
Loading

Motivação

  • Formatos de arquivo com tamanho reduzido e tempos de leitura menores implicam:
    • Menor espaço necessário para armazenamento
    • Menor tempo de processamento
    • Consequentemente, menores custos

Modificações

  • Corrige dependência inconsistente:
    • Para o extra "dev" era google-cloud-bigquery = "1.28.0"
    • Para o pacote em si, google-cloud-bigquery = "2.30.1"
    • Coloquei ambos como google-cloud-bigquery = "2.30.1"
  • Adiciona dependência pandavro para lidar com a interface Pandas <-> Avro
  • Adiciona suporte a Apache Avro e Apache Parquet para tabelas externas
    • O default continua sendo CSV para compatibilidade
  • Corrige a falta do argumento source_format na chamada to Table.init

TODO @isadorabugarin

  • Criar testes
  • Revisar implementação

@d116626 d116626 linked an issue Feb 7, 2022 that may be closed by this pull request
[infra] Suporte a parquet #1082
Closed
@d116626 d116626 changed the title [Feature] Adiciona suporte a Avro e Parquet [infra] Adiciona suporte a Avro e Parquet Feb 7, 2022
@JoaoCarabetta JoaoCarabetta requested review from isadorabugarin and removed request for JoaoCarabetta and d116626 February 7, 2022 23:08
@lucascr91 lucascr91 mentioned this pull request Mar 10, 2022
Merged
2 tasks
@lucascr91 lucascr91 closed this Mar 10, 2022
@lucascr91
Copy link
Collaborator

PR movido para #1145

@lucascr91 lucascr91 removed a link to an issue Apr 4, 2022
[infra] Suporte a parquet #1082
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@isadorabugarin isadorabugarin Awaiting requested review from isadorabugarin

Assignees

@gabriel-milan gabriel-milan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gabriel-milan @lucascr91