Questions tagged [etl]
ETL is an acronym for Extract-Transform-Load. This refers a database usage where data is extracted from a source, transformed into a different format, and loaded to a target database for analysis or reporting.
11
questions
1
vote
0
answers
28
views
Improving the readability, extensability, and optimization of a custom ETL Pipeline library (Pypeline.py)
I'm looking to improve the logic used in this ETL pipeline manager. The concept creates a structure:
Pipeline(Pipe(Task, [Task, ...]), [Pipe(...), ...])
...where a ...
- 53
1
vote
1
answer
71
views
How can I create a stronger ETL Pipeline library?
I've written a module for running ETL pipelines. The goal was to standardize API use in pipelines while making the pipeline easy to create and maintain. I'm not a Python guru by any means, and I'm ...
- 53
0
votes
0
answers
13
views
1
vote
0
answers
57
views
Extract, transform, and load Excel files
I am curious to know if there is a cleaner/better way to write this. Any suggestions would be greatly appreciated! I do ask some questions in comments within the code specific to certain functions/...
- 111
4
votes
1
answer
109
views
Implementing a data extraction pipeline with Bash
I wrote the following scripts to extract data from BigQuery and InfluxDB using the bigquery cli and the influx cli. The code works as expected. I included all files for completeness, but I'm primarily ...
- 319
1
vote
0
answers
36
views
Relational join of two datasets
Front Matter
I'm learning Scala and have not gotten used to functional programming and the language. I'm hoping a review of my naively implemented code can help me bridge my object-oriented ways to ...
- 111
1
vote
0
answers
1k
views
Handling slow processing the Spring batch way
We are using spring-batch to extract data from a SAP system, during processing we pass the extracted full texts to a tagging service, in the end the enriched documents are stored in Solr to be served ...
- 111
1
vote
0
answers
889
views
Snaplogic Script to group key-value pairs based on a certain ID field
Following is the script I'm using to group key-value pairs based on an ID field viz. interface_id:
...
- 499
2
votes
1
answer
909
views
Extract data from one column of SQL Server table
I have an SQL Server table from which I want to extract data from one column (DD) and return as multiple columns.
The data is in ...
- 311
3
votes
1
answer
553
views
ETL code to synchronize external system to internal database
I'm looking to see if my current evolution of ETL logic is getting in the ball park of "how it's done in the real world".
Specifically, do I use the metaphors correctly (...
- 1,212
5
votes
1
answer
369
views
MergeUtility for Entity Framework
Here is a utility that supports ETL/merging in Entity Framework.
If it's not appealing as a general purpose tool, why?
If it is appealing as a general purpose tool, how might the design be made ...
- 1,212