Fabric Migration Guide

This comprehensive guide walks you through migrating databases to Microsoft Fabric using QMigrator, including Azure App registration, connection setup, and end-to-end data migration.

Prerequisites

Before starting the migration process, ensure you have:

• Access to Azure Portal with appropriate permissions
• Microsoft Fabric workspace access
• QMigrator application credentials
• Source and target database connection details

Part 1: Azure App Registration (Service Principal)

Service Principal authentication is required for secure access to Microsoft Fabric resources.

Step 1: Create App Registration

1. Sign in to the Azure Portal
2. Search for App registration in the top search bar
3. Click New registration

Step 2: Configure Application Details

Fill in the required information:

• Name: Enter a descriptive name for your application (e.g., "QMigrator-Fabric-Connector")
• Supported account types: Select the accounts you want your app to support
• Redirect URI (Optional): Select Web under "Select a Platform" dropdown

Click Register to create the application.

Step 3: Capture Application Credentials

1. Select the newly created App Registration

Step 4: Generate Client Secret

1. Under Manage, select Certificates & secrets
2. Click New client secret
3. Provide a description and expiration period

Step 5: Grant Fabric Workspace Access

1. Open your Microsoft Fabric Workspace

2. Click Manage Access

   

3. Click Add people or groups

4. Search for and select your App name created in previous steps
5. Assign the Contributor role

6. Click Add to confirm

   

Part 2: QMigrator Configuration

Step 1: Login to QMigrator

1. Navigate to the QMigrator application URL

2. Click Login to access your account

   

Step 2: Configure Source Connection

Set up the connection to your source database.

1. Click on the Setup menu and navigate to Connections
2. Select migration source as Database
3. In the Connection Type field, choose Source
4. Click on Add Source to add new source

5. Provide the following connection details:

   

   • Connection Name: Reference name for the source
   • Hostname/IP: Source database host
   • Port: Database port number
   • Database Name: Source database name
   • Username: Database authentication username
   • Password: Database authentication password
   • .
   • .
   • .

6. Click Test Connection to verify connectivity

7. Click Save to store the connection

   

Step 3: Configure Target Connection

Set up the connection to your Microsoft Fabric target.

1. Click on the Setup menu and navigate to Connections
2. Select migration source as Database
3. In the Connection Type field, choose Target
4. Click Add Target button

5. Provide the following Fabric connection details:

   • Connection Name: Reference name (e.g., "Fabric_Mirror")
   • Workspace ID: Your Fabric workspace identifier
   • Tenant ID: Directory (tenant) ID from App registration
   • Client ID: Application (client) ID from App registration
   • Client Secret: Secret value from App registration
   • Target Type: Select Fabric service (Warehouse, Lakehouse, etc.)

   

6. Click Test Connection to verify

7. Click Save to store the connection

Part 3: End-to-End Migration

Step 1: Configure Migration Settings

Navigate to the E2E Migration screen and configure the following parameters:

Basic Configuration

Operation Settings

• Operation: Select migration operation type
  • Initial Data Load: Basic initial load without auto-deploying indexes
  • E2E Data Load: Complete migration including data transformation and CDC

Table Selection

• Tables: Multi-select tables to migrate (supports segment loading)

Advanced Configuration

Step 2: Execute Migration

1. After entering all details, click Execute
2. The system will initiate:
   • Schema extraction
   • Code conversion
   • Data migration

Step 3: Monitor Migration Status

Track your migration progress in the E2E Migration Status screen.

Extraction and Conversion Status

1. Select the operation from the dropdown menu
2. Monitor process status:
   • Initialize: Process started and initialized
   • Completed: Process finished successfully

Data Migration Status

1. Select Data Migration from the operation dropdown
2. Select both source and target connections
3. View generated configuration files ready for data migration
4. Review the config Excel file containing:
   • DAG (Directed Acyclic Graph) details
   • Table information
   • Chunk details
   • Category classification (large, small, LOB)

Part 4: Job Agent Automation

The Job Agent automates DAG execution for continuous migration operations.

Configure Job Agent

1. Navigate to the Job Agent screen
2. Select the Source Connection
3. Select the Target Connection
4. Click Start Agent

The Job Agent will automatically trigger all available DAGs from the audit config table, eliminating the need for manual execution.

Monitor Job Execution

View running DAGs and their status in real-time through the Job Agent interface.

Part 5: Scheduler and Task Management

Access Scheduler

1. Click on the Data Migration menu
2. Select the Scheduler screen
3. Login with your username and password
4. Click Browse and select DAG Runs to view active DAGs

View Task Details

1. Click on any running DAG to view its tasks
2. Each DAG consists of the following tasks:
   • Pre-validation: Initial checks before migration
   • DAG Execution: Main data migration process
   • Complete Validation: Post-migration verification

Understanding Task Architecture

• Each DAG task runs as an individual Pod in Kubernetes
• Concurrency settings determine the number of parallel pods
• Task statuses include:
  • Scheduled: Queued for execution
  • Queued: Waiting for resources
  • Running: Currently executing
  • Success: Completed successfully
  • Failed: Encountered errors

Task Management and Recovery

If a task shows Failed, Queued, or Stuck status:

1. Select the task in the scheduler UI
2. Click Clear to reset the task state
3. Select Downstream and Recursive options if needed
4. Verify the task selection
5. Click Clear to re-run the task

View Task Logs

1. Click on a specific task
2. Select Logs tab
3. Review runtime progress logs for troubleshooting

Part 6: Data Verification

After migration completes, verify data integrity:

1. Access your Microsoft Fabric workspace
2. Query the target tables
3. Compare row counts between source and target
4. Validate data types and constraints
5. Check for any data transformation issues

A successful migration should show matching record counts and data integrity across all migrated tables.

Best Practices

• Monitor Resource Usage: Adjust CPU limits based on migration performance
• Incremental Migration: Migrate tables in batches for large datasets
• Test Connections: Always test connections before starting migration
• Review Logs: Regularly check scheduler logs for early issue detection