How to Set Up BigQuery Bulk Export for Search Console

How do you get your full Search Console data out of the interface?

You set up the bulk data export, which sends your Search Console performance data to BigQuery every day. It is configured once in Settings, it needs a Google Cloud project with two APIs enabled and two roles granted, and Google says the first export happens up to 48 hours after successful configuration.

This is the single highest-leverage setup for anyone running a site big enough that the Search Console interface starts feeling like a keyhole. If your blog has a few hundred pages you can live without it. Past a thousand pages it stops being optional, because the questions you want to ask cannot be asked in the interface at all.

I am going to walk through the actual configuration, and then through the four things that trip people up afterwards, because the setup is the easy part and the surprises come later.

Why would you need the bulk export at all?

Because the interface aggregates and truncates, and the export does not. Google describes the bulk export as containing all the performance data available to Search Console for your property, with the exception of anonymized queries. That phrase "all the performance data available" is doing a lot of work, and it is the reason this exists.

The practical difference shows up the moment you want to join Search Console data against anything else. Which of my pages gained impressions but lost clicks. Which queries moved between two specific dates across a whole directory. How many URLs appeared for a query at all. Those are trivial in SQL and impossible in the interface.

The other reason is retention. You control how long the data lives once it is in your own project, which matters when you are trying to understand a pattern over eighteen months rather than a quarter. That is the same reason a large site eventually outgrows screenshot-based reporting, something I got into when I wrote about what changes when your blog passes a thousand pages.

What do you need before you start?

A verified Search Console property that you own, a Google Cloud project, and the ability to grant IAM roles in that project. If you cannot grant roles, you will need someone who can, and it is worth sorting that out before you begin rather than halfway through.

You also need to decide where the dataset should live geographically, because you choose a dataset location during setup and that choice is not casually reversible. If your organisation has data residency requirements, settle them now with whoever owns that policy.

Finally, decide the project. Putting this in a scratch project you might delete is a mistake I would avoid, since the value of this data is entirely in its accumulation over time. Treat it as infrastructure from the first day. If you have not yet got the property itself set up properly, start with getting Search Console configured on the site before anything here.

How do you enable BigQuery and grant access?

Open the Google Cloud Console and switch to the project you chose. Go to APIs and Services, then Enabled APIs and Services, click to enable APIs and services, and enable both the BigQuery API and the BigQuery Storage API. Both are required, and enabling only the first is the most common way this fails silently.

Then go to IAM and Admin and grant access to the service account Google uses for this, which is search-console-data-export@system.gserviceaccount.com. That address is exact and it is the same for everyone, so if you are copying it from somewhere, copy it carefully rather than typing it.

Give that service account two roles. BigQuery Job User, which is bigquery.jobUser, and BigQuery Data Editor, which is bigquery.dataEditor. Both are needed because one lets the export run jobs and the other lets it write the results. Granting one and not the other produces a configuration that looks complete and never delivers data.

How do you turn the export on in Search Console?

Open Search Console for the property, go to Settings, and find Bulk data export. Enter your Cloud project ID, which is the project identifier rather than its display name, and choose a dataset name. Google defaults that name to searchconsole, and I would keep the default unless you have a strong reason not to.

Select your dataset location, then click Continue to initiate exports. At this point Search Console validates that it can actually write to the project, so an error here almost always means one of the two roles is missing or one of the two APIs is not enabled.

Then you wait. Google says the first export will happen up to 48 hours after your successful configuration and that it includes data for the day of the export. Nothing is wrong if your dataset is empty the next morning. Check again after two days before you start debugging.

What actually shows up in BigQuery?

Three tables. The first is searchdata_site_impression, which Google describes as containing performance data for your property aggregated by property. The second is searchdata_url_impression, which contains performance data for your property aggregated by URL. The third is ExportLog, which contains information about each successful export to one of the previous data tables.

The two data tables answer different questions and choosing wrongly is the most common early mistake. Site impression data is aggregated at the property level, so it is the right table for query-level trends. URL impression data is aggregated by URL, so it is the right table for anything page-specific, and it is much larger.

ExportLog is the one people ignore and should not. It tracks the agenda type, currently only SEARCHDATA, the namespace meaning which table, the data_date, an epoch_version that increments when data is revised, and the publish_time. That epoch_version field is how you know Google restated a day you already analysed.

What is missing from the export?

Anonymized queries, and the omission is deliberate. Google says rare queries are marked as anonymized and that the query field will be null when it's true, to protect the privacy of users making the query. So a null query is not a bug and not an export failure.

The URL impression table goes further. Google says that when under the threshold, some other fields like URL and country will be missing, again to protect user privacy. This means your row counts will not reconcile perfectly against the interface, and chasing that reconciliation is wasted effort.

There is also no history. Google is explicit that for historical data preceding setup, you must use the Search Console API or the reports. The export starts accumulating from configuration forward, which is the strongest argument for setting it up today rather than when you need it.

What will this cost you?

Google states that data is subject to Google Cloud storage and query costs, and that there is a free usage level. I am not going to quote you figures, because cloud pricing changes and the honest answer is to read Google Cloud's current pricing for your region before you commit.

What I can tell you is where the cost actually comes from, which is almost never storage and almost always careless querying. The URL impression table is partitioned by date, and a query that does not filter on the partition will scan everything you have ever accumulated. Write the date filter first, every time, before you write anything else.

Google also recommends putting an expiration on the partition rather than on the table, since expiring the table would delete all data. It says to set partition expiration to 14 days or longer. If you want long history, set it long and accept the storage, but set it deliberately rather than leaving it undefined.

What breaks this after it is running?

Schema changes. Google says plainly that schema changes will cause the export to fail, so do not add columns to these tables, do not rename anything, and do not let a well-meaning analyst "tidy up" the dataset. If you need derived columns, build them in a separate table or a view.

The second failure mode is silent, which makes it worse. Failed exports are not recorded in ExportLog, so an absence of rows is the only signal you get. The monitoring that works is checking that ExportLog has a fresh row for yesterday, not checking whether your dashboard looks normal.

The third is permissions drift, which happens when someone cleans up IAM and removes a service account they do not recognise. That address looks like something that should not be there, which is exactly why it gets removed. Document it wherever your team documents infrastructure, and connect the output to whatever you already use, whether that is a Looker Studio reporting view or a notebook.

What should you do next?

Set it up this week even if you have no immediate use for it, because the export only accumulates forward and the version of you that needs eighteen months of data will not be able to go back and get it. The configuration is a twenty-minute job and then it runs by itself.

Once it is running, write one query that filters on the date partition and counts rows per day from ExportLog. That single query is your monitoring, your sanity check, and the thing that tells you the pipeline is alive before anyone asks you a question that depends on it.

If you are running a large content site and you are not sure whether the interface is still enough for what you are trying to learn, reach out. The honest answer is sometimes that it is, and that is a cheaper conversation than a migration you did not need.