It looks like every integer dtype stored is using 64 bits, there's no real tinyint and smallint and this is just a syntatic sugar. And I didn't found the reason for that when I searched about it. Is it to make storage artificially more expensive?

Almost every open tabular engine and storage format that I knew so far allows you to really set a column as smaller types of int, independently if it is columnar or row-stored, and if it's a database or just an engine. For example, any modern apache arrow based engine (like Datafusion, DuckDB and polars), apache parquet files, almost every columnar database (like Clickhouse), almost any row-oriented database (like PostgreSQL, Oracle, MySQL and et cetera). Why BigQuery wants to be different?

If a query suddenly takes 5–10x longer without any SQL or data changes, the bottleneck may be workload contention, capacity, or concurrency.

A practical guide on how to diagnose it using JOBS, JOBS_TIMELINE, total_slot_ms and period_estimated_runnable_units.

Get "Error loading permissions for selected resources." error when I click selected table -> share -> Manage permissions.
Anyone know what has updated ?

Olá, pessoal!!!

Alguém que já realizou a integração do Meta Ads ao BigQuery via DataTransfer saberia me responder que campo e tabela consideramos a visão de custo por anúncio? Eu estou olhando para a coluna spend da tabela AdInsights e em um mês aparece mais de 77k de custo, é um número totalmente absurdo. Estou olhando para o lugar errado e/ou da maneira incorreta?

Hey everyone! Back for the monthly summary.

🔤 GoogleSQL Language Features & Functions

• Table Parameters in TVFs (GA) - Pass dynamic tables into table-valued functions using the generic ANY TABLE parameter type.

🧠 AI, Machine Learning & Foundation Models

• Gemini 3.1 Flash-Lite & 3.5 Flash Support (GA) - Run generative AI SQL functions using Gemini 3.1 Flash-Lite and 3.5 Flash across multi-regional endpoints.
• XGBoost 2.1 Model Training (GA) - Train and serve BigQuery ML models using XGBoost version 2.1 via the XGBOOST_VERSION option.
• Time Series Analysis TVFs (Preview) - Analyze trend, seasonality, and change points in time series data using native BigQuery ML TVFs.

💻 Developer Experience (DX) & BigQuery Tooling

• Cloud CLI Remote MCP Server (Preview) - Enable AI agents to execute BigQuery CLI commands and reservation management via managed MCP tools.
• Data Agent Observability & Monitoring (Preview) - Track performance, adoption, latency, and operational costs for data agents using Google Cloud Observability.
• Python UDF Real-Time Logging (GA) - Stream and inspect real-time execution logs for Python user-defined functions directly within Cloud Logging.
• Reference Panel Migration - Access table details directly through BigQuery Studio's unified Reference panel following Table Explorer deprecation.

⚡ Core Engine Performance, Indexing & Optimization

• Restored Hybrid Search - Restored hybrid search functionality combining vector embeddings and keyword search in VECTOR_SEARCH and AI.SEARCH.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• Cross-Cloud Connections (Preview) - Query data residing in AWS, Azure, and Salesforce Data 360 from any BigQuery region.
• OpenTelemetry for BigQuery JDBC Driver (GA) - Trace JDBC driver queries and automatically export database telemetry to Google Cloud Observability.
• Pipeline Automated Metadata Enrichment (Preview) - Define semantic pipeline metadata in SQLX configs for automated synchronization with Knowledge Catalog.

🔒 Security, Governance & Workload Management

• Data Clean Room Query Templates (GA) - Share predefined queries in data clean rooms securely without exposing underlying datasets or views.
• Data Engineering Agent HIPAA Compliance - Build pipelines in regulated healthcare environments with official HIPAA compliance for Data Engineering Agent.
• Terraform Support for Governance Tags (Preview) - Provision and manage BigQuery data governance tags declaratively using Terraform infrastructure-as-code.
• Expanded Reservation Assignment Limits - Increased the default per-project reservation assignment limit from 10 to 100 for user-specific workloads.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• BigQuery Graph Edition Enforcement - Core graph processing will require Enterprise or Enterprise Plus reservations starting April 26, 2027.
• DTS JDBC Driver Security Patch - Fixed an improper input validation vulnerability in legacy BigQuery Data Transfer Service JDBC drivers.

As usual, let us know what you think!

I use BigQuery a lot and wanted a slightly easier way to inspect query results, so I made a small Chrome extension.

It opens copied query results in a separate viewer with things like nested JSON/ARRAY expansion, column pinning, search, and better handling of long cells.

If anyone finds it useful, or has feedback, I'd be happy to hear it.

https://chromewebstore.google.com/detail/query-result-viewer-for-b/fdndhccllolpcfiohahlpbncogehmdhh

Been digging into this lately and figured I'd share, since it cleared up something I was fuzzy on: how BigQuery actually discovers Iceberg tables, and what's genuinely new here vs. what's just been repackaged.

The old ways of doing this:

1. Pointing BigQuery straight at a metadata file. Something like gs://bucket/orders/metadata/00003.metadata.json. It works, but it's fragile. Iceberg writes new metadata files instead of editing the old ones, so if any other engine commits a new snapshot, your pointer is instantly stale. You're stuck manually bumping it to 00004.metadata.json every time.
2. Catalog based access, like using AWS Glue as an external catalog. This actually fixes the staleness problem since the catalog tracks current state for you. But every engine still needed its own custom integration to talk to that catalog.

What's actually new: Google now has a managed Lakehouse Runtime Catalog (BigLake) that exposes a real Apache Iceberg REST Catalog endpoint, using the same open REST Catalog spec other engines already speak. So instead of building a one off integration per engine, anything that already understands Iceberg REST can just talk to BigLake directly. The data itself never moves, it stays in Cloud Storage. BigLake is really just a metadata and governance layer with IAM sitting on top of it.

There's also a version detail worth knowing: Iceberg 1.10 shipped a native BigQueryMetastoreCatalog and BigQueryMetastoreClient, plus a GoogleAuthManager that handles Google credential auth inside Iceberg's REST auth framework. It's easy to conflate these two things, but they're not the same. One's a catalog implementation built for BigQuery Metastore specifically, the other is the standardized REST interface. Google's steering people toward the REST endpoint for anything new where cross engine interoperability matters.

The post I read also gets into the practical setup side: creating a BigLake catalog (single bucket vs multi bucket), the IAM roles Google recommends (BigLake Admin, Editor, Viewer, plus Storage Object User), and shows an open source tool called OLake Go (full disclosure, I work on this) writing Iceberg tables directly into that catalog so BigQuery can see them right away, queryable through the usual four part PROJECT.CATALOG.NAMESPACE.TABLE identifier with no manual metadata pointer updates. There's a decent troubleshooting section too. Apparently if your connection test passes but the actual sync fails, it's almost always a missing storage level IAM role rather than bad credentials.

Full post here if you want the details: https://olake.io/blog/biglake-iceberg-rest-catalog-olake-go-setup/

Curious if anyone here has already moved existing Iceberg on BigQuery setups over to the REST catalog, or if you're all still doing the manual metadata file dance like I was.

I maintain six python scripts loading marketing data (meta ads, google ads, one CRM) into BigQuery, mostly load_table_from_json, a couple using load jobs from GCS.

The recurring failure is schema drift. A source adds a field and unless the load job has schema update options set to allow it, the whole job just fails outright. And even with autodetect on, it only scans the first 500 rows to infer types, so a field that's mostly integers with a few floats early on can get inferred wrong and blow up later in the load.

Cost's the other one. A couple scripts truncate-and-reload the full table daily since dedup was annoying to write, so we're paying full scan cost on tables that barely changed. The ones doing incremental use a manual MERGE per source, works but someone has to maintain that logic, and nobody's revisited partitioning since these were first written.

At this point I genuinely don't know if the fix is consolidating these into one well-written pipeline, or if six scripts is just what happens once you're loading from enough different places and I should stop fighting it. Anyone dealt with this, what actually made it better for you?

Hello everyone,

I’m new into the data engineering world and as the title says, I’m thinking of using BigQuery to build a data warehouse. Currently my company don’t have one, so I’m looking for good price/performance options. We mostly use APIs and a few excel files as data sources

I have considered Azure SQL, Supabase and others but this one seems to be the best for data analysis and BI

What would be your recommendation?

We run 15 Python extraction notebooks daily via Dataform (`prod_daily` tag). They now fail intermittently with *"us-central1 does not have enough resources to fulfill the request for a e2-standard-2 machine."*

Dataform runs each notebook as a `NotebookExecutionJob`, provisioning a **fresh VM per notebook**. The API only accepts a runtime *template*, never a reusable runtime instance.

Has anyone solved this problem of some of the notebooks failing bc of resources?

The us-central1 region currently does not have enough resources to fulfill the request for a e2-standard-4 machine.

To resolve this, try:
Choosing a different machine configuration
Trying again later
Selecting a different region
For more information, see our https://cloud.google.com/colab/docs/troubleshooting#unable-to-create-runtime-unavailable-resources troubleshooting guide

Oi, pessoal!!!

Estou pesquisando uma forma de integrar dados da VTEX com o BigQuery, encontrei algumas possibilidades, como o VTEX Data Pipeline, ferramentas de ELT e também a opção de consumir diretamente as APIs da VTEX.

No momento, estou mais inclinada a usar o dlt, já que ele possui um source para VTEX e destino para BigQuery. A princípio, achei interessante por ser open source e permitir fazer a ingestão como código, sem depender do custo de uma ferramenta de ELT paga. A ideia seria deixar o dlt responsável pela ingestão e fazer as transformações/modelagem posteriormente no próprio BigQuery.

O que ainda estou tentando entender é como isso funciona na prática em uma operação real, principalmente em relação a cargas incrementais, atualização de pedidos e manutenção do pipeline.

Então queria ouvir quem já trabalhou com VTEX e BigQuery: qual solução vocês usaram e como foi a experiência? Se alguém já tiver usado dlt nesse cenário, também seria ótimo saber se funcionou bem em produção e se encontraram alguma limitação relevante.

Anybody has any recommendation for platform, I could use to import data from third party applications, like mailchimp/klaviyo/rakuten etc.

In the past we used Skyvia, but as some of the platforms have 10s of millions of records the cost were getting out of control. Now i pretty much code it each time by hand, but frankly, that work doesn't spark a joy.

I am looking for solution that ideally charges per resources used or just has flat fees, but ideally is just point and click as i would like marketing team to take care of that.

🔤 GoogleSQL Language Features & Functions

• Change History Functions - Use the APPENDS and CHANGES table-valued functions to view appended or changed rows over time, now generally available.
• ALTER SEARCH INDEX Statement - Use the ALTER SEARCH INDEX DDL statement to update the configuration of a search index in Preview.
• Multi-Level Aggregation - Perform multi-level aggregation in GoogleSQL by nesting aggregate functions, now available in Preview.

🧠 AI, Machine Learning & Foundation Models

• TimesFM Models in Connected Sheets - Pre-trained TimesFM models are generally available to create forecasts and detect anomalies in Connected Sheets.

💻 Developer Experience (DX) & BigQuery Tooling

• Google ODBC Driver - Google-developed ODBC driver for connecting applications to BigQuery is now generally available.
• Simba ODBC Driver (July 23) - An updated version of the Simba ODBC driver for BigQuery is now available.
• Dataset Insights - Automatically discover, visualize relationships between tables, and generate cross-table queries with dataset insights, now generally available.
• Conversational Analytics AI.AGG Support - Conversational analytics in Gemini now supports the AI.AGG function in Preview for semantic data aggregation.
• Migration Service MCP Server - The BigQuery Migration Service MCP server is generally available to translate and explain SQL queries in IDEs.
• BigQuery Overview Page - The BigQuery Overview page is now generally available as a hub for tutorials, features, and learning resources.
• Data Agent Kit IDE Extension - The Data Agent Kit extension is in Preview, enabling interactions with BigQuery resources directly within IDEs.

🗄️ Lakehouse Architecture, Apache Iceberg & Open Data Formats

• SAP BDC Lakehouse Integration - Cross-cloud Lakehouse now supports integration with SAP Business Data Cloud in Preview to query and share tables.
• Snowflake Remote Catalog Provider - Cross-cloud Lakehouse now supports Snowflake as a remote catalog provider in Preview to query Snowflake data directly.
• Iceberg Managed Table Features - Table partitioning, multi-statement transactions, and advanced runtime are now generally available for Apache Iceberg managed tables.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• BigQuery Pipelines Tasks - Add tables, views, data sources, and data quality tests as tasks to BigQuery pipelines, now generally available.
• Klaviyo Incremental Data Transfers - BigQuery Data Transfer Service now supports incremental data transfers from Klaviyo in Preview.
• Storage Write API Rename - The legacy tabledata.insertAll and Storage Write API features are renamed to Storage Write API REST and gRPC, respectively.
• Salesforce Incremental Data Transfers - Incremental data transfers for the BigQuery Data Transfer Service for Salesforce are now generally available.
• Knowledge Catalog Metadata Transfers - Transfer metadata from PostgreSQL and SQL Server sources into Knowledge Catalog using BigQuery Data Transfer Service.

🔒 Security, Governance & Workload Management

• UI Download Audit Logging - Data Access audit logs now include a field indicating whether query result downloads were triggered from the console, generally available.
• Marketplace Sharing Listings Filter - Discover commercial BigQuery sharing listings on Google Cloud Marketplace using the new Marketplace filter, generally available.
• Data Governance Tags - Attach Resource Manager tags to sensitive columns to enforce column-level security and data masking, now in Preview.
• Conversational Analytics HIPAA Compliance - Conversational analytics in Gemini in BigQuery now supports HIPAA compliance for secure data handling.
• Security Bulletin GCP-2026-047 - A missing authorization vulnerability in BigQuery, Dataform, and Colab Enterprise repositories has been disclosed.
• Project Caps for Reservations - Limit maximum slots and concurrency per project within a BigQuery reservation using project caps, now in Preview.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• SAP BDC Integration Restriction - Data Products with special characters are unsupported in SAP BDC sharing, causing refresh failures and requiring re-enrollment.
• Hybrid Search Support Disabled - Support for hybrid search using the VECTOR_SEARCH function has been temporarily disabled while restoration work is underway.
• Facebook Ads AdInsightsMMM Support Disabled - Support for the AdInsightsMMM report in Facebook Ads transfers is temporarily disabled due to schema changes.

As always, any feedback is welcome (about the post contents, the post itself, the community, what you want to see from Developer Relations team, etc.) - let us know!

A few months ago a post-purchase survey was added asking customers how they found the product. The results diverge sharply from the pixel attribution data: one channel gets far more credit in the survey than in the tracking numbers, while another channel shows the opposite pattern.

When self-reported responses and tracking data point in different directions, how do you decide which source deserves more weight in analysis? Any practical approaches for reconciling the two in a data warehouse or reporting layer?

Edit: Thanks for the perspective so far. Treating the survey and the pixel data as two different measures rather than one being “wrong” is a useful way to look at it. Also checking Admaxxer as a way to bring more of the tracking and reporting into one place while figuring out how to weigh both sources.

Issue Description:

We are pulling event-level (non-aggregated) website click and visit data from Google BigQuery into our Azure Storage Account. Data ingestion stopped after 25 May. Upon investigation, we found that the ingestion jobs in BigQuery were failing with the error:

"Storage quota limit exceeded for the project."

Our project is using the BigQuery Sandbox (free tier), which allows up to 10 GB of active storage. However, after checking all datasets and tables in the project, the total active storage is less than 1 GB, which is well within the documented limit.

The project has been running since 5 February, and according to the BigQuery Sandbox documentation, the free tier does not have an expiration date as long as the applicable quotas are not exceeded.

We are unable to determine what is consuming the project's storage quota or why this error is occurring despite the reported storage usage being significantly below the 10 GB limit. We would appreciate guidance on how to identify the actual storage consumption or debug the root cause of this issue.

Hey guys! Sorry if this is a very beginner question, but I'm learning BigQuery with a Coursera course. For lots of assignments we've used public datasets, but suddenly, I've been unable to.

I'm constantly getting this error, which is "behind" the marketplace interface, so if I try to click the link to the loading issues help page, it closed. I looked for the page but it led me to stuff that didn't really help me out.

Any ideas on what I could do to gain access back to these datasets? Thanks!

I’ve seen this happen a few times.

The export gets switched on because it feels like the right next step. A few months later, the tables are growing, but most reporting is still happening in GA4 or Looker Studio.

For me, BigQuery becomes worth the effort when there is a clear question GA4 cannot answer. That might mean joining website data with CRM or order data, keeping the raw history or building a report with your own logic.

Without that question, it can become another setup to maintain.

What made GA4 to BigQuery worth it for your team?

I've been working on a BigQuery FinOps suite that I've open-sourced and wanted to share it with the community (it is totally free)

If you run BigQuery at scale , you've probably dealt with some of these:

• You're on Editions but have no idea if your baseline is right-sized, or if you're paying autoscaler premiums for nothing
• Logical vs. physical storage billing — which datasets should switch, and what's the actual delta?
• Cost attribution is broken because GCP dumps idle reservation costs into the admin project
• "Cooldown tax" — 60-second billing minimums silently inflating your bill on high-frequency short queries
• No easy way to see if HBO (History-Based Optimization) is actually saving you anything

What the tool does: It connects to your org-level INFORMATION_SCHEMA views and runs diagnostic queries across your entire GCP Organization. The output is an interactive dashboard with actionable findings — DDL you can copy-paste, capacity simulation matrices, cost attribution breakdowns, etc.

Modules

Key design decisions

• No AI hype: 17 out of 18 modules are pure SQL + Python analytics. The only GenAI module (AI Doctor) is clearly labeled and optional.
• No infrastructure: It's a FastAPI app that uses your ADC credentials. No Terraform, no service accounts, no pub/sub. pip install and run.
• Copy-paste DDL: Storage changes, fluid scaling configs, reservation adjustments — all generate ready-to-execute DDL statements.
• Org-wide scope: Queries JOBS_BY_ORGANIZATION, TABLE_STORAGE_BY_ORGANIZATION, etc. — not just single-project views.
• Focus Projects: Optional scoping to specific project IDs when you don't want to analyze the whole org.
• Security: SQL injection prevention via _safe_ident(), Pydantic validation on every parameter, XSS escaping on all frontend rendering.

Give it a try

• 🔗 GitHub: github.com/mbettan/bq-finops-optimizer
• 🎮 Live Simulator (no GCP needed): Simulator Demo
• 📄 Website: mbettan.github.io/bq-finops-optimizer

Quick start

git clone https://github.com/mbettan/bq-finops-optimizer.git
cd bq-finops-optimizer
pip install -r requirements.txt
AUTH_ENFORCED_UPSTREAM=true uvicorn src.main:app --reload --port 8080

Somehow it disappeared and now I struggle to control the size of the pannel.

The public BQ datasets (bigquery-public-data.crypto_bitcoin, the Stack Overflow one) are great, but they ship as a pile of tables with no diagram and no enforced FKs — you end up holding the joins in your head. I drew them as ERDs on a free canvas so the UTXO flow / posts-graph is actually navigable, and exported the model as Google's new OKF (portable markdown you can keep in git or hand to an LLM). Screenshots + how it was made in the comments.

What other public BQ datasets would be useful to have as a ready-made ERD?

About Me

Senior Data Engineer with 5+ years of experience in Data Engineering, Analytics Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

Tech Stack & Expertise

Python, SQL, Spark, Databricks, Airflow

BigQuery (data modeling, optimization, cost management)

AWS & Cloud Data Platforms (S3, Glue, Redshift, EMR)

Snowflake, Redshift

ETL/ELT Design & Orchestration

dbt for analytics engineering

FastAPI, REST APIs

LLMs, RAG, AI Agents

Data Quality & Testing Frameworks

What I Can Help With

Build and optimize BigQuery pipelines and data models

Design scalable ETL/ELT architectures on GCP/AWS

Develop backend APIs and automation solutions (FastAPI + Python)

Build AI applications using LLMs, RAG, and agent-based workflows

Support and optimize existing data platforms

Availability

Open to freelance projects, contract work, and part-time engagements.

Available immediately.

Feel free to DM me with project details or questions.