Google BigQuery 🔗

Supported drivers:

dbapi	default	driver	connection class
google-cloud-bigquery		`bigquery+google`	`google.cloud.bigquery.dbapi.Connection`

google-cloud-bigquery🔗

google-cloud-bigquery is the default dbapi driver for Google BigQuery in pydapper.

Installation🔗

pippoetry

pip install pydapper[google-cloud-bigquery]

poetry add pydapper -E google-cloud-bigquery

Google cloud storage alternate installation🔗

google-cloud-bigquery also has support for a more performant read api using pyarrow and remote procedure calls (RPC).
In order to get the performance benefits, no config is required, you simply install the google-cloud-bigquery-storage extra as well.
Read more about it in the google docs.

pippoetry

pip install pydapper[google-cloud-bigquery, google-cloud-bigquery-storage]

poetry add pydapper -E google-cloud-bigquery -E google-cloud-bigquery-storage

DSN format🔗

For bigquery, config is not actually passed through the dsn, so the dsn is extremely easy to define. The dsn simply tells pydapper what driver to use. Please see the connect and using examples below on how to pass config.

ExampleExample (Default Driver)

dsn = "bigquery+google:////"

dsn = "bigquery:////"

Example - `connect`🔗

By default the google client will look for the GOOGLE_APPLICATION_CREDENTIALS environment var

import pydapper

# export GOOGLE_APPLICATION_CREDENTIALS=/path/to/keystore.json

with pydapper.connect("bigquery+google:////") as commands:
    print(type(commands))
    # <class 'pydapper.bigquery.google_bigquery_client.GoogleBigqueryClientCommands'>

    print(type(commands.connection))
    # <class 'google.cloud.bigquery.dbapi.connection.Connection'>

    raw_cursor = commands.cursor()
    print(type(raw_cursor))
    # <class 'google.cloud.bigquery.dbapi.cursor.Cursor'>

Alternatively, you can construct a client yourself and pass it into connect...

import json
import pathlib

from google.cloud.bigquery import Client

import pydapper

credentials = (
    pathlib.Path("~", "src", "pydapper", "tests", "test_bigquery", "auth", "key.json").expanduser().read_text()
)

client = Client.from_service_account_info(json.loads(credentials))

with pydapper.connect("bigquery+google:////", client=client) as commands:
    print(type(commands))
    # <class 'pydapper.bigquery.google_bigquery_client.GoogleBigqueryClientCommands'>

    print(type(commands.connection))
    # <class 'google.cloud.bigquery.dbapi.connection.Connection'>

    raw_cursor = commands.cursor()
    print(type(raw_cursor))
    # <class 'google.cloud.bigquery.dbapi.cursor.Cursor'>

Example - `using`🔗

You might want to use a connection object that you constructed from some factory function or connection pool. In that case, you can pass that object directly into using...

import json
import pathlib

from google.cloud.bigquery import Client
from google.cloud.bigquery.dbapi import connect

import pydapper

credentials = (
    pathlib.Path("~", "src", "pydapper", "tests", "test_bigquery", "auth", "key.json").expanduser().read_text()
)

client = Client.from_service_account_info(json.loads(credentials))
dbapi_connection = connect(client=client)

commands = pydapper.using(dbapi_connection)
print(type(commands))
# <class 'pydapper.bigquery.google_bigquery_client.GoogleBigqueryClientCommands'>

print(type(commands.connection))
# <class 'google.cloud.bigquery.dbapi.connection.Connection'>

raw_cursor = commands.cursor()
print(type(raw_cursor))
# <class 'google.cloud.bigquery.dbapi.cursor.Cursor'>

Transactions🔗

BigQuery has no connection-level transactions, and the DBAPI reflects that:

Connection.commit() is a documented no-op, and there is no rollback() method at all.
The connection has no __enter__/__exit__ — with pydapper.connect("bigquery:////") works only because pydapper's context-manager proxying is conditional, and exiting the block performs no driver call (nothing is committed, rolled back, or closed).
GoogleBigqueryClientCommands does not declare AdapterCapability.TRANSACTIONS, so pydapper's commit(), rollback(), and transaction() raise UnsupportedFeatureError before the connection is touched.

Every statement runs as its own BigQuery job and is effectively committed on completion.

See Transactions and Context manager semantics for the cross-driver picture.

Google BigQuery🔗