Questions tagged [etl]

ETL is an acronym for Extract-Transform-Load. This refers a database usage where data is extracted from a source, transformed into a different format, and loaded to a target database for analysis or reporting.

Filter by
Sorted by
Tagged with
0
votes
0answers
28 views

Save a dynamically created DAG in Airflow instead of registering it to the scheduler

I would like to save my dynamic DAGs and not have them autoscheduled by the Airflow scheduler and so I an not using the globals() utility. Is this the right way? I saw pickling_dag but doesn't seem ...
0
votes
0answers
175 views

Dynamically configurable ZMQ filter with Flask API

I have a stream of ZMQ messages, I need to filter the right ones and save them in MongoDB. The trick is to have an option to set the filter conditions dynamically. I have come up with this solution ...
1
vote
0answers
807 views

Handling slow processing the Spring batch way

We are using spring-batch to extract data from a SAP system, during processing we pass the extracted full texts to a tagging service, in the end the enriched documents are stored in Solr to be served ...
1
vote
0answers
460 views

Snaplogic Script to group key-value pairs based on a certain ID field

Following is the script I'm using to group key-value pairs based on an ID field viz. interface_id: ...
2
votes
1answer
774 views

Extract data from one column of SQL Server table

I have an SQL Server table from which I want to extract data from one column (DD) and return as multiple columns. The data is in ...
3
votes
1answer
545 views

ETL code to synchronize external system to internal database

I'm looking to see if my current evolution of ETL logic is getting in the ball park of "how it's done in the real world". Specifically, do I use the metaphors correctly (...
5
votes
1answer
361 views

MergeUtility for Entity Framework

Here is a utility that supports ETL/merging in Entity Framework. If it's not appealing as a general purpose tool, why? If it is appealing as a general purpose tool, how might the design be made ...