FireFly Analytics LogoFireFly Analytics
Solutions

SQL Editor

A native SQL query interface with warehouse integration, featuring multi-tab file editing, streaming results, and a catalog sidebar for data exploration.

Overview

Unlike the Notebook and Code Editors which embed Databricks Lakehouse Apps via iframe, the SQL Editor is implemented as a native React component. This provides tighter integration with your application's UI, state management, and user experience.

The SQL Editor communicates directly with Databricks SQL Warehouse APIs through Next.js API routes. All requests are authenticated using the SSO-SPN token acquisition flow, ensuring users never see Databricks login screens.

Why Native Instead of Iframe?

  • Custom UI/UX: Full control over the interface design and interactions
  • State Integration: Share state with other parts of your application
  • Feature Control: Add custom features like saved queries, query history, result export
  • Performance: Direct API calls without iframe overhead
  • Catalog Integration: Tight coupling with the Data Catalog for autocomplete

How It Works

The SQL Editor is a client-side React component that makes API calls to Next.js routes, which in turn communicate with Databricks SQL Warehouse APIs. The architecture follows a typical React application pattern with TanStack Query for data fetching and caching.

Architecture

Query Execution Flow

  1. User writes SQL in the Monaco-based editor with syntax highlighting and autocomplete
  2. Query is submitted via POST /api/databricks/sql/executeto the selected SQL Warehouse
  3. Server acquires SPN token using the user's mapped credentials
  4. API returns statement_id immediately (async execution)
  5. Client polls for results via GET /api/databricks/sql/status/{id}
  6. Results stream in and are displayed in the results table

User Experience

The SQL Editor features a three-panel layout optimized for data exploration and query development.

Features

  • Multi-tab file editing with persistence across sessions
  • Warehouse selection with real-time status indicator
  • Start/stop warehouse controls
  • Real-time query execution with streaming results
  • Query cancellation for long-running statements
  • Catalog sidebar for browsing tables and autocomplete
  • Result table with column metadata and row counts
  • Keyboard shortcuts (Cmd+Enter to run, Cmd+S to save)
  • Syntax highlighting for Spark SQL dialect
  • Error highlighting with line number references

Three-Panel Layout

Left Panel

Catalog Browser: Hierarchical tree view of catalogs, schemas, and tables. Click to expand, double-click to insert into query.

Top Right Panel

Query Editor: Monaco-based SQL editor with multi-tab support, syntax highlighting, and autocomplete from catalog metadata.

Bottom Right Panel

Results Table: Paginated results display with column headers, data types, and row counts. Supports large result sets.

Backend Architecture

The SQL Editor uses several API routes that interface with Databricks SQL Statement Execution API and SQL Warehouses.

API Routes

RouteMethodDescription
/api/databricks/sql/executePOSTSubmit SQL statement for execution
/api/databricks/sql/status/{id}GETPoll for query status and results
/api/databricks/sql/cancel/{id}POSTCancel a running query
/api/databricks/warehousesGETList available SQL warehouses
/api/databricks/warehouses/{id}/startPOSTStart a stopped warehouse
/api/databricks/warehouses/{id}/stopPOSTStop a running warehouse

Query Execution Example

// Execute SQL statement
const executeQuery = async (sql: string, warehouseId: string) => {
  // Submit query (returns immediately with statement_id)
  const response = await fetch("/api/databricks/sql/execute", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      warehouse_id: warehouseId,
      statement: sql,
      wait_timeout: "0s", // Return immediately, poll for results
    }),
  });

  const { statement_id } = await response.json();

  // Poll for results
  const pollResults = async (): Promise<QueryResult> => {
    const status = await fetch(`/api/databricks/sql/status/${statement_id}`);
    const data = await status.json();

    switch (data.status.state) {
      case "SUCCEEDED":
        return {
          columns: data.manifest.schema.columns,
          rows: data.result.data_array,
          rowCount: data.manifest.total_row_count,
        };
      case "FAILED":
        throw new Error(data.status.error.message);
      case "RUNNING":
      case "PENDING":
        await new Promise(r => setTimeout(r, 1000));
        return pollResults();
      default:
        throw new Error(`Unknown state: ${data.status.state}`);
    }
  };

  return pollResults();
};

Warehouse Management

The SQL Editor includes controls for managing SQL Warehouses, allowing users to start stopped warehouses and monitor their status.

// Warehouse selector with status and controls
function WarehouseSelector({ onSelect }) {
  const { data: warehouses } = useQuery({
    queryKey: ["warehouses"],
    queryFn: () => fetch("/api/databricks/warehouses").then(r => r.json()),
    refetchInterval: 10000, // Poll every 10 seconds for status updates
  });

  const startMutation = useMutation({
    mutationFn: (id: string) =>
      fetch(`/api/databricks/warehouses/${id}/start`, { method: "POST" }),
    onSuccess: () => queryClient.invalidateQueries(["warehouses"]),
  });

  return (
    <Select onValueChange={onSelect}>
      {warehouses?.map(wh => (
        <SelectItem key={wh.id} value={wh.id}>
          <span>{wh.name}</span>
          <Badge variant={wh.state === "RUNNING" ? "success" : "secondary"}>
            {wh.state}
          </Badge>
          {wh.state === "STOPPED" && (
            <Button size="sm" onClick={() => startMutation.mutate(wh.id)}>
              Start
            </Button>
          )}
        </SelectItem>
      ))}
    </Select>
  );
}

Enhancement Opportunities

The SQL Editor can be extended with additional features to improve productivity and collaboration.

Query History

Persist executed queries with timestamps, execution time, and row counts for easy re-running and analysis of past queries.

Saved Queries

Allow users to save, name, and organize frequently-used queries into folders for quick access and sharing.

Result Export

Export query results to CSV, Excel, or JSON formats for external analysis, reporting, and sharing with stakeholders.

Query Scheduling

Schedule queries to run at specific times or intervals with result delivery via email, Slack, or webhook.

Query Explain

Show query execution plans with cost estimates to help users optimize their queries before running.

Result Visualization

Add charting capabilities to visualize query results directly in the editor without external tools.