Questions tagged [parquet]

Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Filter by
Sorted by
Tagged with
1 vote
0 answers

Streaming Parquet file in chunks for write operation

I am taking beginner steps into DE and was tinkering with writing an ingestion script which does the following tasks: Reads data from a source (in this case a remote parquet file) Writes it to local ...
av abhishiek's user avatar
1 vote
3 answers

Write SQL Server table to Parquet file

This is a C# console application that creates Parquet file for SQL Server table. Given a 5GB SQL table, it takes nearly 65 minutes, so for our bigger tables (up to 500GB), it may end up taking days to ...
Srini's user avatar
  • 11
2 votes
1 answer

Stock data REST API extractor with GCP storage options

I'm currently working on an AI-driven trading system, the code below aims to extract ticker data from polygon REST API, this is a paid service so, in order to test the code you will need to subscribe /...
watch-this's user avatar
1 vote
0 answers

Optimized AWS Lamda purpose [closed]

I'm looking for optimized code with faster execution time. This writes CSV to parquet with the max size of 64 MB chunks. we are trying to concatenate the csv file contents being written as an parquet ...
user1510139's user avatar
3 votes
0 answers

Optimizing throughput when writing millions of rows with Parquet's row group

I'm building a basic data pipeline in .NET to extract data from tables in our RDBMS systems, that exports into columnar storage using Parquet. I'm using Parquet's row group to batch up the writes into ...
dcrdev's user avatar
  • 31