Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [google-bigquery]

BigQuery is Google's fully managed, scalable, serverless data warehouse. With BigQuery, you get a columnar and ANSI SQL database that can analyze petabytes of data in seconds.

Filter by
Sorted by
Tagged with
3 votes
1 answer
52 views

Trigger Cloud Function with Eventarc and then extract BigQuery table as csv files

Problem: Whenever new data are inserted, extract BigQuery tables as csv files and store them in Cloud Storage. My plan: Set up an Eventarc trigger based on Event method ...
hashaf's user avatar
  • 33
0 votes
1 answer
46 views

Efficiency of Apache Beam + BigQuery pipeline

This is my first Apache pipeline. It takes a JSON file and saves the correctly formatted rows in one table, and the misformed rows in another. My biggest worry is the efficiency. I have tested the ...
Yulia V's user avatar
  • 635
0 votes
1 answer
192 views

Execute several queries and save them to files in Bash

I wrote the following bash script to run queries saved in files and save them to csv files. The queries have to be run against a BigQuery database. I use the bq-cli ...
Moritz Loritz's user avatar
2 votes
2 answers
139 views

Calculates start time and end time of jobs in a dataproc cluster

I have the below function get_status_time which calculates the start time and the end time of the spark job which has already completed its run (status could be ...
Tushaar's user avatar
  • 55
2 votes
1 answer
57 views

building a shell command inside python function based on different conditions

I have written the below python function which is working fine but there are lot of code which looks redundant to me and hence i want to make it better by using best coding guidelines. Basically I am ...
Travelling Days's user avatar
3 votes
1 answer
49 views

How to fine tune the complex python function for creating merge command to run in bigquery

I created a python code to run the merge command in Google BigQuery, which is used to do UPDATE, INSERT, and DELETE in a single statement. I'd appreciate it if someone could assist me in fine-tuning ...
Tushaar's user avatar
  • 55
1 vote
1 answer
131 views

Creating SQL query for relative timeframes and multiple tables

What I want to achieve: I have several tables of the following schema: +------+-------+ | date | val | +------+-------+ | DATE | INT64 | +------+-------+ I want ...
Moritz Loritz's user avatar
3 votes
2 answers
157 views

Keeping only maximum date rows in a group

The following SQL code keeps only the MAX(date) rows with the same id and question values. I would like to know if there is a simpler/ shorter syntax returning the same result. ...
ZygD's user avatar
  • 133
2 votes
0 answers
44 views

SQL query that extracts a topic keyword from article titles

I have a query like this to select the only row with title contains specific programming topics. ...
phwt's user avatar
  • 359
3 votes
1 answer
1k views

Insert an SQL Server Table rows into a Bigquery table in one bloc

I want to insert all rows of an SQL server Table into a BigQuery Table having the same schema. The streaming insert row by row is very slow: to insert 1000 rows the execution of the code below took ...
user3569267's user avatar
2 votes
0 answers
379 views

Insert all SQL Server table rows in a Google BigQuery Table

I want to know the best way (best performance and reaction time) to insert all rows of an SQL table into a Google BigQuery table. Actually, I am looping over a query result and insert the rows one by ...
user3569267's user avatar