Data Migration in Magento 2

Data Migration Tool

Overview

The Magento Data Migration Tool migrates data and settings from Magento 1 to Magento 2.

Installation

# Install the tool
composer require magento/data-migration-tool

# Or via marketplace
magento-module:install magento/module-data-migration-tool

Configuration

<!-- etc/config.xml -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_DataMigrationTool:etc/config.xsd">
    <steps title="Migration">
        <step title="Settings Migration">
            <migration>Migration\Step\Settings\Data</migration>
            <migration>Migration\Step\Settings\Log</migration>
        </step>
        <step title="Data Migration">
            <migration>Migration\Step\Data Integrity\Integrity</migration>
            <migration>Migration\Step\Data\Data</migration>
            <migration>Migration\Step\Data\Document\DocumentsToTransfer</migration>
            <migration>Migration\Step\Data\Delta\Delta</migration>
        </step>
        <step title="URL Rewrite">
            <migration>Migration\Step\UrlRewrite\Integrity</migration>
            <migration>Migration\Step\UrlRewrite\Data</migration>
        </step>
        <step title="Configurable">
            <migration>Migration\Step\Configurable\Integrity</migration>
            <migration>Migration\Step\Configurable\Data</migration>
        </step>
        <step title="EAV Migration">
            <migration>Migration\Step\Eav\Integrity</migration>
            <migration>Migration\Step\Eav\Data</migration>
        </step>
        <step title="Customer Attributes Migration">
            <migration>Migration\Step\CustomerAttributes\Integrity</migration>
            <migration>Migration\Step\CustomerAttributes\Data</migration>
        </step>
    </steps>
</config>

Running Migration

# Full migration (settings + data)
php bin/magento migrate:data --all etc/config.xml

# Settings only
php bin/magento migrate:settings etc/config.xml

# Data only
php bin/magento migrate:data etc/config.xml

# With verbose output
php bin/magento migrate:data --verbose etc/config.xml

# Resume from failure point
php bin/magento migrate:data --resume etc/config.xml

Migration Modes

# Dry run (no actual changes)
php bin/magento migrate:data --dry-run etc/config.xml

# Delta migration (only new/changed data)
php bin/magento migrate:data --delta etc/config.xml

# Reset migration (start over)
php bin/magento migrate:data --reset etc/config.xml

Custom Migration Scripts

When Custom Scripts Are Needed

Custom Data Mapping

namespace Vendor\Migration\Step\CustomOrderStatus;

class Data implements StepInterface
{
    public function __construct(
        private SourceInterface $source,
        private DestinationInterface $destination,
        private LoggerInterface $logger
    ) {}
    
    public function execute(): void
    {
        // Map custom order statuses
        $statusMap = [
            'custom_pending' => 'pending',
            'custom_processing' => 'processing',
            'custom_shipped' => 'complete',
        ];
        
        $orders = $this->source->getOrders();
        
        foreach ($orders as $order) {
            $newStatus = $statusMap[$order['status']] ?? $order['status'];
            
            $this->destination->updateOrderStatus(
                $order['entity_id'],
                $newStatus
            );
        }
    }
}

Custom Entity Migration

// Migrate custom module data
namespace Vendor\Migration\Step\CustomWarranty;

class Data implements StepInterface
{
    public function execute(): void
    {
        $sourceData = $this->source->select(
            "SELECT * FROM custom_warranty WHERE migrated = 0"
        );
        
        foreach ($sourceData as $row) {
            // Transform data
            $warranty = $this->transformWarranty($row);
            
            // Insert into new table
            $this->destination->insert('vendor_warranty', $warranty);
            
            // Mark as migrated
            $this->source->update(
                'custom_warranty',
                ['migrated' => 1],
                ['entity_id = ?' => $row['entity_id']]
            );
        }
    }
    
    private function transformWarranty(array $row): array
    {
        return [
            'order_id' => $row['sales_order_id'],
            'product_id' => $row['catalog_product_id'],
            'duration_months' => $row['warranty_period'],
            'price' => $row['warranty_price'],
            'status' => $row['is_active'] ? 'active' : 'inactive',
            'created_at' => $row['created_at'],
        ];
    }
}

Data Transformation

// Complex data transformation
public function transformCustomerData(array $magento1Customer): array
{
    $magento2Customer = [
        'email' => $magento1Customer['email'],
        'firstname' => $magento1Customer['firstname'],
        'lastname' => $magento1Customer['lastname'],
        'website_id' => $this->mapWebsite($magento1Customer['website_id']),
        'group_id' => $this->mapGroup($magento1Customer['group_id']),
        'default_billing' => $magento1Customer['default_billing'],
        'default_shipping' => $magento1Customer['default_shipping'],
        'created_at' => $magento1Customer['created_at'],
        'updated_at' => $magento1Customer['updated_at'] ?? date('Y-m-d H:i:s'),
    ];
    
    // Handle password hash migration
    if (!empty($magento1Customer['password_hash'])) {
        $magento2Customer['password_hash'] = $this->migratePassword(
            $magento1Customer['password_hash']
        );
    }
    
    // Map custom attributes
    $customAttributes = $this->getCustomAttributes($magento1Customer['entity_id']);
    foreach ($customAttributes as $attribute => $value) {
        $magento2Customer[$attribute] = $value;
    }
    
    return $magento2Customer;
}

Data Validation

Validation Techniques

Record Count Validation

public function validateRecordCounts(): array
{
    $tables = [
        'catalog_product_entity',
        'catalog_category_entity',
        'customer_entity',
        'sales_order',
        'sales_order_item',
    ];
    
    $results = [];
    
    foreach ($tables as $table) {
        $sourceCount = $this->source->count($table);
        $destCount = $this->destination->count($table);
        
        $results[$table] = [
            'source' => $sourceCount,
            'destination' => $destCount,
            'match' => $sourceCount === $destCount,
            'difference' => $destCount - $sourceCount
        ];
    }
    
    return $results;
}

Data Integrity Validation

public function validateProductData(): array
{
    $products = $this->destination->select(
        "SELECT entity_id, sku, name, price, status FROM catalog_product_entity"
    );
    
    $issues = [];
    
    foreach ($products as $product) {
        // Check required fields
        if (empty($product['sku'])) {
            $issues[] = ['product_id' => $product['entity_id'], 'issue' => 'missing_sku'];
        }
        
        if (empty($product['name'])) {
            $issues[] = ['product_id' => $product['entity_id'], 'issue' => 'missing_name'];
        }
        
        if ($product['price'] < 0) {
            $issues[] = ['product_id' => $product['entity_id'], 'issue' => 'negative_price'];
        }
        
        // Verify source exists
        $sourceProduct = $this->source->selectOne(
            "SELECT * FROM catalog_product_entity WHERE sku = ?",
            [$product['sku']]
        );
        
        if (!$sourceProduct) {
            $issues[] = ['product_id' => $product['entity_id'], 'issue' => 'no_source_match'];
        }
    }
    
    return $issues;
}

Business Logic Validation

public function validateOrderTotals(): array
{
    $orders = $this->destination->select(
        "SELECT entity_id, grand_total, subtotal, tax_amount, shipping_amount 
         FROM sales_order"
    );
    
    $issues = [];
    
    foreach ($orders as $order) {
        $calculatedTotal = $order['subtotal'] + $order['tax_amount'] + $order['shipping_amount'];
        
        // Allow for rounding differences
        if (abs($calculatedTotal - $order['grand_total']) > 0.01) {
            $issues[] = [
                'order_id' => $order['entity_id'],
                'issue' => 'total_mismatch',
                'expected' => $calculatedTotal,
                'actual' => $order['grand_total']
            ];
        }
    }
    
    return $issues;
}

Validation Report

public function generateValidationReport(): string
{
    $report = "Data Migration Validation Report\n";
    $report .= str_repeat('=', 50) . "\n\n";
    
    // Record counts
    $counts = $this->validateRecordCounts();
    $report .= "Record Counts:\n";
    foreach ($counts as $table => $data) {
        $status = $data['match'] ? 'PASS' : 'FAIL';
        $report .= sprintf("  %s: %d â†’ %d [%s]\n", 
            $table, $data['source'], $data['destination'], $status);
    }
    
    // Product validation
    $productIssues = $this->validateProductData();
    $report .= "\nProduct Issues: " . count($productIssues) . "\n";
    
    // Order validation
    $orderIssues = $this->validateOrderTotals();
    $report .= "Order Issues: " . count($orderIssues) . "\n";
    
    return $report;
}

Edge Cases and Troubleshooting

Common Edge Cases

Character Encoding Issues

// Fix UTF-8 encoding
public function fixEncoding(string $text): string
{
    if (!mb_check_encoding($text, 'UTF-8')) {
        return mb_convert_encoding($text, 'UTF-8', 'ISO-8859-1');
    }
    return $text;
}

NULL vs Empty Values

// Normalize NULL and empty values
public function normalizeValue($value)
{
    if ($value === null || $value === '') {
        return null;
    }
    return $value;
}

Duplicate SKU Handling

public function handleDuplicateSku(string $sku): string
{
    $counter = 1;
    $newSku = $sku;
    
    while ($this->skuExists($newSku)) {
        $newSku = $sku . '-' . $counter;
        $counter++;
    }
    
    return $newSku;
}

Broken Foreign Keys

public function fixBrokenReferences(): array
{
    $fixes = [];
    
    // Fix order items with invalid product references
    $items = $this->destination->select(
        "SELECT * FROM sales_order_item WHERE product_id NOT IN 
         (SELECT entity_id FROM catalog_product_entity)"
    );
    
    foreach ($items as $item) {
        // Find matching product by SKU
        $product = $this->destination->selectOne(
            "SELECT entity_id FROM catalog_product_entity WHERE sku = ?",
            [$item['product_sku']]
        );
        
        if ($product) {
            $this->destination->update(
                'sales_order_item',
                ['product_id' => $product['entity_id']],
                ['item_id = ?' => $item['item_id']]
            );
            $fixes[] = ['item_id' => $item['item_id'], 'fixed' => true];
        }
    }
    
    return $fixes;
}

Troubleshooting

# Check migration log
tail -f var/log/migration.log

# Resume failed migration
php bin/magento migrate:data --resume etc/config.xml

# Reset and retry
php bin/magento migrate:data --reset etc/config.xml

# Verbose debugging
php bin/magento migrate:data --verbose etc/config.xml 2>&1 | tee debug.log

# Check database connection
php bin/magento setup:db-check

# Verify source database
mysql -h source-host -u user -p -e "SELECT COUNT(*) FROM sales_order"

Practice Problems

0 / 2 solved

hard Custom Migration Script

Write a custom migration script to migrate custom warranty data from Magento 1 to Magento 2.

hard Data Validation Report

Create a comprehensive validation report comparing source and target databases after migration.

Flashcards

Question

What is delta migration?

Click to reveal answer

Answer

Migrating only new/changed data after initial full migration

Question

What does --dry-run do?

Click to reveal answer

Answer

Simulates migration without making actual changes

Question

When to use custom scripts?

Click to reveal answer

Answer

When standard tool can't handle custom data or transformations

Question

What does record count validation check?

Click to reveal answer

Answer

Same number of records in source and target tables

Question

How to resume failed migration?

Click to reveal answer

Answer

php bin/magento migrate:data --resume etc/config.xml

Revision Notes

Key Takeaways

1. Magento Data Migration Tool handles most standard data migration
2. Custom scripts needed for custom modules and transformations
3. Always run --dry-run first to validate migration plan
4. Delta migration (--delta) handles incremental data after initial full migration
5. Validate record counts, data integrity, and business logic
6. Handle edge cases: encoding, NULLs, duplicates, broken references

Interview Tips

• Walk through the Magento Data Migration Tool process
• When would you need custom migration scripts?
• How do you validate data integrity after migration?
• Describe a challenging data migration edge case you handled
• What's the difference between full and delta migration?

Cheat Sheet

Data Migration Cheat Sheet

Tool Commands:

migrate:settings - settings only
migrate:data - data migration
migrate:data --delta - incremental
migrate:data --dry-run - test
migrate:data --resume - continue

Custom Scripts:

Implement StepInterface
Transform data as needed
Handle edge cases
Log progress

Validation:

Record counts match
Required fields present
Business logic valid
Foreign keys intact

Edge Cases:

Character encoding
NULL vs empty
Duplicate SKUs
Broken references

Data Migration Tool

Overview

Installation

Configuration

Running Migration

Migration Modes

Custom Migration Scripts

When Custom Scripts Are Needed

Custom Data Mapping

Custom Entity Migration

Data Transformation

Data Validation

Validation Techniques

Record Count Validation

Data Integrity Validation

Business Logic Validation

Validation Report

Edge Cases and Troubleshooting

Common Edge Cases

Character Encoding Issues

NULL vs Empty Values

Duplicate SKU Handling

Broken Foreign Keys

Troubleshooting

Practice Problems

Quiz

Flashcards

Revision Notes

Key Takeaways

Interview Tips

Cheat Sheet

Data Migration Cheat Sheet