Skip to main content

Changing the Snowflake Warehouse

When creating a connection, the user can specify a warehouse that should be used to refresh a job. However, in some use cases, you must override the warehouse for a subset of Nodes within a DAG.

Setting Warehouse in a Workspace​

When creating a Workspace for Snowflake, you enter the Role and Warehouse during the credential setup step. This warehouse becomes the default for all operations in that workspace.

Overriding Warehouse for Specific Nodes​

You can override the warehouse for specific Nodes within a DAG using a stage to set the current warehouse. This is useful in several scenarios:

  • Using larger compute resources to get a performance boost on a complex transformation.
  • Keep track of costs by running certain transformations on department-specific warehouses

To override the warehouse, define a stage anywhere within a UDN template:

{{ stage('Set Warehouse') }}
USE WAREHOUSE <warehouse_name>

Coalesce maintains the state of which warehouse is used before and after a Node is executed. When you execute the USE WAREHOUSE query, this persists for the entirety of the Node run but reverts to the previous warehouse when continuing to the next Node in the DAG.

Not supported in Development Workspaces at this time

Override warehouse is only supported in Environments.

Although USE WAREHOUSE can be executed in a development workspace, Coalesce does not revert the warehouse back to the default before continuing. In addition, within a development workspace, there is a possibility that the set warehouse affects other Nodes running in parallel.

Override warehouse will be coming to Development Workspaces in a future release.

Setting Warehouse for Tasks​

For task-based Nodes, you can specify the warehouse in the Scheduling Options:

  • Select Warehouse: Enter the name of the warehouse you want the task to run on without quotes (visible when Scheduling Mode is set to Warehouse Task).

You can also use the targetTaskWarehouse environment parameter to specify different warehouses across environments.