Transform Onboarding Guide
This guide walks you through everything you need to set up Coalesce Transform and deploy your first pipeline. Work through each section in order. For detailed instructions on any step, follow the linked guides.
Who This Guide Is For
Transform onboarding involves two main roles:
- Admins and setup owners: Configure accounts, connect data platforms, create Projects and Workspaces, and manage Environments.
- Developers: Build pipelines, add sources, create transformations, deploy, and run refreshes.
Prerequisites Checklist
Before starting, confirm you have:
- Cloud data warehouse access: Snowflake, Databricks, BigQuery, or Fabric. See Connection guides.
- Git repository: For version control (or plan to create one). See Coalesce Git Requirements.
- Basic SQL and data transformation concepts: Familiarity with SQL and data modeling.
- Admin access: To configure integrations and add team members. See Administrative Tasks.
- Google Chrome: The only supported browser. See System Requirements.
For network allowlisting, setup tasks, and more, see Setup Requirements.
Phase 1: Account and Organization Setup
Create Your Account
- Sign up through a trial or contact Coalesce for an enterprise account.
- Add team members and assign roles. See Add Users and Set Permissions.
- Configure SSO if needed. See Authentication.
Configure Network Access
- Allow inbound traffic from Coalesce IP addresses.
- Allow outbound traffic to Coalesce domains.
- For Snowflake: Configure network policies. For Databricks: Configure egress policies. See Network Requirements.
Connect Your Data Platform
- Choose your platform and follow the connection guide:
- Snowflake: Username/password, key pair, or OAuth
- Databricks: Unity Catalog required
- BigQuery: Service account authentication
- Fabric: See Fabric connection guide
- Test the connection before proceeding.
Phase 2: Project and Workspace Setup
Create Your Project
- Create a new Project from the Coalesce dashboard.
- Configure Git integration: choose provider (GitHub, GitLab, Bitbucket, or Azure DevOps).
- Add your repository URL.
- Create a personal access token for authentication.
- Use one Git repository per Project. See Create Your Project.
Set Up Version Control
- Each user has their own Git provider account.
- Each user creates a personal access token for Coalesce.
- Each user belongs to the organization's Git account.
- Decide your branch strategy (for example, feature branches, main for deployment).
- See Set Up Version Control for full details.
Create Your Workspace
- From your Project, click Create Workspace.
- Complete the Onboarding Wizard: name, description, connection.
- Configure Storage Locations and mappings.
- Connect the Workspace to your data platform using credentials from the Connect Your Data Platform section above.
- See Create a Workspace for step-by-step instructions.
Phase 3: Build Your First Pipeline
Add Sources
- From the Build screen, click + then Add Sources.
- Select the tables you want to add.
- Preview each source before adding.
- See Add a Data Source for details.
Build Transformations
- Add Nodes (Stage, Dimension, Fact, View, or Custom).
- Configure column transforms, joins, and filters.
- Check the Marketplace for pre-built Node types that fit your use case.
- See Transforms and Nodes for details.
Validate Your Pipeline
- Run the pipeline (or a subset) to populate tables.
- Preview data in each Node to verify transformations.
- Use the Problem Scanner to catch errors before deploying.
Phase 4: Environments and Deployment
Create Environments
- Go to Build Settings > Environments in your Workspace.
- Create Environments for DEV, QA, and Production (or your naming convention).
- For each Environment:
- Configure authentication (username/password, OAuth, or key pair).
- Set Storage Mappings (database and schema).
- Add Parameters if needed.
- Each Environment should map to a distinct database and schema. See Create Your Environments.
Deploy
- Ensure your Workspace is on the main branch with no uncommitted changes (for production).
- Go to Deploy and select your target Environment.
- Review the deployment plan.
- Deploy using the Coalesce App, CLI, or third-party tools.
- See Deployment Overview and Deploying to an Environment.
Set Up Refresh and Jobs
- Create Jobs by selecting the Nodes to include.
- Deploy before refreshing (Jobs run only on deployed Nodes).
- Schedule refreshes using the Coalesce Scheduler, CLI, Jobs API, or external tools.
- See Refresh Your Pipeline and Scheduling Jobs.
Phase 5: Team Rollout
- Add developers to the Project.
- Establish a Workspace strategy (one per branch, one per user, or one per feature).
- Designate a single developer to commit to the main branch.
- See Coalesce Best Practices for ongoing guidance.
Optional: Advanced Paths
AI Features
- Copilot: Use natural language to generate transformations. See Copilot and Migrating SQL to Coalesce with Copilot.
- AI-generated descriptions: Add descriptions to Nodes and columns for documentation and lineage.
See Coalesce AI for the full set of AI capabilities.
Programmatic Setup
- Project APIs: Automate project, Workspace, and Environment creation. See API documentation.
- CLI: Deploy and refresh from the command line. See CLI.
- Automation: Use APIs and CLI for CI/CD, Workspace provisioning, and Environment management.
Integrations
- Catalog: Sync lineage and documentation to Coalesce Catalog. See Catalog integration with Coalesce.
- Marketplace packages: Add pre-built Node types and patterns. See Marketplace.
- External orchestrators: Integrate with Airflow, GitHub Actions, GitLab, and others. See Third-Party DevOps Tools.
Get Help
Support Channels
- Shared Slack or Teams channel: Dedicated channel for your team and Coalesce Customer Success.
- Email: support@coalesce.io for quick assistance.
- In-app support: Click the question mark icon for the AI Assistant, or Get Help to open an email to support.
When contacting support, include your Environment ID, run ID, and error details. Use Copy All to Clipboard in the app to capture system information. See Contacting Support for full details.
Self-Service Resources
- Quick Starts: Snowflake Quick Start, Databricks Build Weather Analytics
- Foundational guide: Coalesce Foundational Hands-On Guide
- FAQ and troubleshooting: FAQ and Troubleshooting Deployments and Refreshes
What's Next?
- Coalesce Best Practices for ongoing workflow and deployment guidance
- Coalesce Catalog Onboarding Guide to add discovery and governance with Catalog
- Marketplace to explore pre-built Node types and patterns
- Troubleshooting Deployments and Refreshes if you run into issues