Questions tagged [parquet]
Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.
5
questions
1
vote
0
answers
1k
views
Streaming Parquet file in chunks for write operation
I am taking beginner steps into DE and was tinkering with writing an ingestion script which does the following tasks:
Reads data from a source (in this case a remote parquet file)
Writes it to local ...
- 111
1
vote
3
answers
1k
views
Write SQL Server table to Parquet file
This is a C# console application that creates Parquet file for SQL Server table.
Given a 5GB SQL table, it takes nearly 65 minutes, so for our bigger tables (up to 500GB), it may end up taking days to ...
- 11
2
votes
1
answer
154
views
Stock data REST API extractor with GCP storage options
I'm currently working on an AI-driven trading system, the code below aims to extract ticker data from polygon REST API, this is a paid service so, in order to test the code you will need to subscribe /...
1
vote
0
answers
51
views
Optimized AWS Lamda purpose [closed]
I'm looking for optimized code with faster execution time. This writes CSV to parquet with the max size of 64 MB chunks.
we are trying to concatenate the csv file contents being written as an parquet ...
- 11
3
votes
0
answers
837
views
Optimizing throughput when writing millions of rows with Parquet's row group
I'm building a basic data pipeline in .NET to extract data from tables in our RDBMS systems, that exports into columnar storage using Parquet. I'm using Parquet's row group to batch up the writes into ...
- 31