Questions tagged [etl]

ETL is an acronym for Extract-Transform-Load. This refers a database usage where data is extracted from a source, transformed into a different format, and loaded to a target database for analysis or reporting.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

ETL script that makes request and the log it into a table

...
user avatar
  • 1
1 vote
0 answers
40 views

Extract, transform, and load Excel files

I am curious to know if there is a cleaner/better way to write this. Any suggestions would be greatly appreciated! I do ask some questions in comments within the code specific to certain functions/...
user avatar
  • 11
2 votes
0 answers
49 views

Implementing a data extraction pipeline with Bash

I wrote the following scripts to extract data from BigQuery and InfluxDB using the bigquery cli and the influx cli. The code works as expected. I included all files for completeness, but I'm primarily ...
user avatar
1 vote
0 answers
31 views

Relational join of two datasets

Front Matter I'm learning Scala and have not gotten used to functional programming and the language. I'm hoping a review of my naively implemented code can help me bridge my object-oriented ways to ...
user avatar
  • 111
0 votes
0 answers
228 views

Dynamically configurable ZMQ filter with Flask API

I have a stream of ZMQ messages, I need to filter the right ones and save them in MongoDB. The trick is to have an option to set the filter conditions dynamically. I have come up with this solution ...
user avatar
  • 101
1 vote
0 answers
1k views

Handling slow processing the Spring batch way

We are using spring-batch to extract data from a SAP system, during processing we pass the extracted full texts to a tagging service, in the end the enriched documents are stored in Solr to be served ...
user avatar
  • 111
1 vote
0 answers
626 views

Snaplogic Script to group key-value pairs based on a certain ID field

Following is the script I'm using to group key-value pairs based on an ID field viz. interface_id: ...
user avatar
2 votes
1 answer
886 views

Extract data from one column of SQL Server table

I have an SQL Server table from which I want to extract data from one column (DD) and return as multiple columns. The data is in ...
user avatar
3 votes
1 answer
547 views

ETL code to synchronize external system to internal database

I'm looking to see if my current evolution of ETL logic is getting in the ball park of "how it's done in the real world". Specifically, do I use the metaphors correctly (...
user avatar
5 votes
1 answer
365 views

MergeUtility for Entity Framework

Here is a utility that supports ETL/merging in Entity Framework. If it's not appealing as a general purpose tool, why? If it is appealing as a general purpose tool, how might the design be made ...
user avatar