FireFly Analytics LogoFireFly Analytics
Architecture / IAM / Organizations

Organizations in FireFly

A comprehensive guide to how organizations are created, mapped, and managed in the FireFly platform, including Databricks resource mappings and automation status.

Overview

Organizations are the primary multi-tenancy unit in FireFly. Each organization is an isolated data silo with its own members, Service Principals, workspaces, and Unity Catalog configurations.

This document describes how organizations are created, what Databricks resources they map to, and what steps are automated vs manual.

Key Concepts

  • Multi-Tenant Isolation: Each org has isolated resources and members
  • SSO + SPN Authentication: Users auth via SSO, API calls via Service Principal
  • Unity Catalog Access: Per-org catalogs with group-based permissions
  • Role-Based Access Control: Owner, Admin, Member roles per organization

Automation Status

Not all organization setup steps are automated. Some require manual Databricks admin intervention:

  • Automated: Org creation, member management, SPN storage, workspace mapping
  • Manual: SPN creation in Databricks, group creation, catalog permissions

What Organizations Map To

Organizations map to resources in both the FireFly PostgreSQL database and Databricks. Understanding these mappings is crucial for proper organization setup.

FireFly Database Resources

organization

Core organization record with identity and configuration

  • id: Unique org identifier (org_timestamp_random)
  • name: Display name
  • slug: URL-friendly identifier
  • workspaceUrl: Primary Databricks workspace URL
  • ssoEnabled: Whether SSO is enabled (default: true)

member

User-organization relationships with role-based access

  • organizationId: Reference to organization
  • userId: Reference to user
  • role: "owner", "admin", or "member"

byodDatabricksSpns

Service Principal credentials for Databricks API access

  • organizationId: Reference to organization
  • clientId: SPN OAuth client ID
  • clientSecret: Encrypted SPN client secret

byodDatabricksWorkspaces

Workspace-to-SPN mappings for API routing

  • workspaceUrl: Databricks workspace URL
  • spnId: Reference to SPN credentials
  • deltaSharingGlobalMetastoreId: For Delta Sharing

organizationStorageSettings

Unity Catalog group and catalog configuration

  • primaryOrganizationGroup: Databricks group name
  • primaryOrganizationGroupId: Databricks group ID
  • organizationEditableCatalog: Writable catalog name

Databricks Resources

Organizations map to the following Databricks resources, which must be created manually:

Workspace

One or more Databricks workspaces per organization. The workspace URL is stored in the organization record.

Service Principal

OAuth credentials for API access. Created in Databricks and stored in FireFly for token exchange.

Unity Catalog Group

Access control for org members. Users must be added to this group to access org data.

Catalog

Data storage with granted permissions. The group receives READ/WRITE access to specific catalogs.

Database Schema

The following entity-relationship diagram shows all tables involved in organization management and their relationships.

Automation Status

Understanding which steps are automated and which require manual intervention is critical for organization setup and onboarding.

Automated (via APIs)

These steps are handled by FireFly

  • Organization creation in FireFly database (POST /api/admin/organizations)
  • Member addition to organizations (POST /api/admin/add-member)
  • SPN credential storage (POST /api/sso-spn/byod/databricks/spns)
  • Workspace mapping configuration (POST /api/sso-spn/byod/databricks/workspaces)
  • SCIM user ID lookup and mapping (POST /api/admin/databricks/accounts/map-scim-user)
  • Workspace validation (POST /api/sso-spn/byod/databricks/workspaces/validate)
  • Storage settings configuration

Manual Steps Required

These steps require Databricks admin

  • Create Service Principal in Databricks Account Console
  • Grant SPN access to workspace(s)
  • Create organization group in Databricks
  • Create catalog for organization in Unity Catalog
  • Grant Unity Catalog permissions to group (SELECT, MODIFY on catalogs)
  • Add users to Databricks group (no automated API)
  • Configure Delta Sharing providers and shares
  • Create uploads schema and user volumes

Complete Organization Setup Flow

The following diagram shows the complete flow for setting up a new organization. Red boxes indicate manual steps that require Databricks admin intervention.

Setup Phases Explained

1. Organization Creation

Admin creates organization via the Admin UI. This creates a record in PostgreSQL with name and slug.

2. Databricks Setup

A Databricks workspace must exist or be created for the organization to map to. Set the workspace URL for the organization and map the organization to the Databricks workspace.

3. Workspace Setup (MANUAL)

Admin must manually create Service Principal, grant workspace access, create Unity Catalog group, create catalog, and configure permissions in Databricks.

Note: This step is currently manual but can be automated via Terraform, or made self-service by building a UI wired to Databricks APIs that provides customers the ability to self-service the setup via an organization onboarding wizard.

4. FireFly Configuration

Admin adds SPN credentials to FireFly via BYOD settings, maps workspaces to SPNs, validates connections, and configures storage.

5. Member Management

Admin adds users as members in FireFly, creates user SPNs in Databricks (manual), maps users to SPNs in FireFly, and adds users to the Databricks group (manual) for data access.

Organization Creation Sequence

This sequence diagram shows the detailed flow for creating a new organization, including both automated and manual steps.

Creation API Details

POST /api/admin/organizations
{
  "name": "Acme Corporation",     // Required
  "slug": "acme-corp",            // Optional (auto-generated from name)
  "workspaceUrl": "https://acme.cloud.databricks.com",  // Optional
  "ssoEnabled": true              // Optional (default: true)
}

// Response
{
  "id": "org_1706123456_abc123",
  "name": "Acme Corporation",
  "slug": "acme-corp",
  "workspaceUrl": "https://acme.cloud.databricks.com",
  "ssoEnabled": true,
  "createdAt": "2024-01-24T12:00:00Z"
}

Member Onboarding Sequence

This sequence diagram shows the complete flow for adding a user to an organization and granting them access to Databricks resources.

Member Roles

Owner

  • Full administrative access
  • Can manage organization settings
  • Can add/remove other owners
  • Can delete the organization

Admin

  • Can manage members
  • Can configure settings
  • Cannot delete organization
  • Cannot manage owners

Member

  • Basic access to org features
  • Can view data and run queries
  • Cannot manage other users
  • Cannot modify settings

SSO-SPN Authentication Flow

This diagram shows how users authenticate via SSO and how their requests are authorized using the organization's Service Principal.

Key Authentication Points

  • Users authenticate via Okta (or other OIDC provider) - no Databricks account needed
  • Organization selection stores activeOrganizationId in session
  • All Databricks API calls use the organization's SPN credentials
  • SPN tokens are cached and refreshed automatically

API Reference

These are the key API endpoints for organization management.

Admin APIs

Require admin access (@databricks.com email)

GET /api/admin/organizations - List all organizations
POST /api/admin/organizations - Create organization
POST /api/admin/update-organization - Update organization
POST /api/admin/add-member - Add user to organization
POST /api/admin/remove-member - Remove user from organization
POST /api/admin/update-member-role - Change member role
POST /api/admin/databricks/accounts/map-scim-user - Map SCIM ID

BYOD Configuration APIs

GET /api/sso-spn/byod/databricks/spns - List SPNs
POST /api/sso-spn/byod/databricks/spns - Add SPN
GET /api/sso-spn/byod/databricks/workspaces - List workspaces
POST /api/sso-spn/byod/databricks/workspaces - Add workspace
POST /api/sso-spn/byod/databricks/workspaces/validate - Validate workspace

Frontend Locations

/admin/organizations - Manage organizations
/admin/organizations/[slug] - Manage org members
/admin/users - Manage users and SCIM mapping
/sso-spn/[orgId]/settings/bring-your-own-data - BYOD configuration

Key Takeaways

Understanding the organization model is essential for proper platform administration. Here are the key points to remember:

1

Organizations are Multi-Tenancy Units

Each org has isolated members, SPNs, and configurations

2

Creation is Partially Automated

FireFly DB records are created via API, but Databricks resources require manual setup

3

Service Principals are Per-Organization

Stored in byodDatabricksSpns and used for all Databricks API calls

4

Users Authenticate via SSO

But API calls use the organization's SPN credentials

5

Unity Catalog Access is Group-Based

Users must be added to the Databricks group (manual step)

Full Automation Roadmap

To fully automate organization setup, the following APIs would need to be integrated: Databricks Account API for SPN creation, SCIM API for group management, and Unity Catalog APIs for permissions.

Learn About Authentication