Regression Testing for Magento 2 Upgrades

Testing Pyramid for Upgrades

        /\         E2E Tests (10%)
       /  \        - Full checkout flow
      /    \       - Critical user journeys
     /------\
    /        \     Integration Tests (30%)
   /          \    - Module interactions
  /            \   - API endpoints
 /--------------\
/                \  Unit Tests (60%)
/                  \ - Core business logic
/--------------------\- Service classes

Test Priority Matrix

Priority | Area                    | Test Type    | Frequency
---------|-------------------------|-------------|------------------
P0       | Checkout/Payment        | E2E         | Every change
P0       | Catalog Display         | Integration | Every change
P0       | Order Processing        | Integration | Every change
P1       | Admin Order Management  | Integration | Daily
P1       | Customer Registration   | Integration | Daily
P1       | Search & Navigation     | Integration | Daily
P2       | CMS Pages               | Unit        | Weekly
P2       | Email Templates         | Unit        | Weekly
P3       | Admin Configuration     | Manual      | Pre-release

Upgrade Test Checklist

Pre-Upgrade:
- [ ] Full test suite passes on current version
- [ ] Baseline performance metrics captured
- [ ] Critical paths documented
- [ ] Test data prepared

During Upgrade:
- [ ] Run unit tests after composer update
- [ ] Run integration tests after setup:upgrade
- [ ] Verify admin panel loads
- [ ] Check frontend renders correctly

Post-Upgrade:
- [ ] Full regression suite passes
- [ ] Performance benchmarks met
- [ ] Security scan clean
- [ ] Manual smoke tests on critical paths

Identifying Critical Paths

Business Critical Flows

1. Product Discovery
   Homepage → Category → Product → Add to Cart

2. Checkout
   Cart → Shipping → Payment → Order Confirmation

3. Customer Account
   Login → Account Dashboard → Order History → Reorder

4. Admin Operations
   Login → Orders → Process Order → Invoice → Ship

5. Catalog Management
   Admin → Products → Edit → Save → Frontend Verify

Critical Path Test Scripts

// Checkout Critical Path Test
public function testCheckoutCriticalPath(): void
{
    // 1. Add product to cart
    $this->addProductToCart('simple-product', 2);
    $this->assertEquals(2, $this->getCartItemCount());
    
    // 2. Proceed to checkout
    $this->navigateToCheckout();
    $this->assertPageLoaded('checkout');
    
    // 3. Fill shipping information
    $this->fillShippingAddress([
        'firstname' => 'John',
        'lastname' => 'Doe',
        'street' => ['123 Test St'],
        'city' => 'Testville',
        'postcode' => '12345',
        'country_id' => 'US',
        'region_id' => 12,
        'telephone' => '555-0123'
    ]);
    
    // 4. Select shipping method
    $this->selectShippingMethod('flatrate');
    $this->assertShippingCost(10.00);
    
    // 5. Select payment method
    $this->selectPaymentMethod('checkmo');
    
    // 6. Place order
    $orderId = $this->placeOrder();
    $this->assertNotEmpty($orderId);
    
    // 7. Verify order in admin
    $this->adminLogin();
    $this->navigateToOrder($orderId);
    $this->assertOrderStatus($orderId, 'pending');
}

Monitoring Critical Paths

// Performance assertions during tests
public function testCheckoutPerformance(): void
{
    $startTime = microtime(true);
    
    $this->addProductToCart('simple-product');
    $cartTime = microtime(true) - $startTime;
    
    $this->assertLessThan(2.0, $cartTime, 'Add to cart too slow');
    
    $startTime = microtime(true);
    $this->navigateToCheckout();
    $checkoutTime = microtime(true) - $startTime;
    
    $this->assertLessThan(3.0, $checkoutTime, 'Checkout page too slow');
}

Automated Test Suite Structure

PHPUnit Configuration

<!-- phpunit.xml -->
<phpunit bootstrap="vendor/autoload.php"
         colors="true"
         stopOnFailure="false">
    <testsuites>
        <testsuite name="unit">
            <directory>tests/unit</directory>
        </testsuite>
        <testsuite name="integration">
            <directory>tests/integration</directory>
        </testsuite>
        <testsuite name="upgrade">
            <directory>tests/upgrade</directory>
        </testsuite>
    </testsuites>
    <coverage>
        <include>
            <directory suffix=".php">app/code/Vendor</directory>
        </include>
    </coverage>
</phpunit>

Upgrade-Specific Tests

// tests/upgrade/OrderProcessingTest.php
namespace Vendor\Module\Tests\Upgrade;

use Magento\TestFramework\Helper\Bootstrap;

class OrderProcessingTest extends \Magento\Framework\AppInterfaceTest\AbstractTestCase
{
    public function testOrderCreation(): void
    {
        // Ensure order creation works after upgrade
        $order = Bootstrap::getObjectManager()->create(
            \Magento\Sales\Api\Data\OrderInterface::class
        );
        // ... test order creation
    }
    
    public function testOrderStatusTransitions(): void
    {
        // Verify status transitions still work
        $statuses = ['pending', 'processing', 'complete'];
        foreach ($statuses as $status) {
            $this->assertTrue(
                $this->statusResolver->isValidTransition($status)
            );
        }
    }
}

CI/CD Integration

# .github/workflows/upgrade-test.yml
name: Upgrade Regression Tests
on:
  push:
    branches: [upgrade/*]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: root
          MYSQL_DATABASE: magento_test
        ports:
          - 3306:3306
      elasticsearch:
        image: elasticsearch:7.17
        ports:
          - 9200:9200
    steps:
      - uses: actions/checkout@v3
      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
          extensions: mbstring, intl, pdo_mysql
      - name: Install Dependencies
        run: composer install --prefer-dist
      - name: Setup Magento
        run: |
          php bin/magento setup:install \
            --db-host=localhost \
            --db-name=magento_test \
            --db-user=root \
            --db-password=root
      - name: Run Unit Tests
        run: vendor/bin/phpunit --testsuite unit
      - name: Run Integration Tests
        run: vendor/bin/phpunit --testsuite integration
      - name: Run Upgrade Tests
        run: vendor/bin/phpunit --testsuite upgrade

Test Data Management

// tests/_fixtures/product.php
return [
    'simple_product' => [
        'name' => 'Simple Test Product',
        'sku' => 'TEST-SIMPLE-001',
        'price' => 29.99,
        'qty' => 100,
        'is_in_stock' => 1,
    ],
    'configurable_product' => [
        'name' => 'Configurable Test Product',
        'sku' => 'TEST-CONFIG-001',
        'options' => [
            ['label' => 'Color', 'values' => ['Red', 'Blue']],
            ['label' => 'Size', 'values' => ['S', 'M', 'L']],
        ]
    ]
];

Building a Regression Suite

Test Coverage Strategy

Module Coverage Targets:
- Core (Catalog, Sales, Checkout): 80%+
- Custom Modules: 70%+
- Third-party: 50%+
- Total: 65%+

Critical Path Coverage:
- 100% of P0 flows
- 90% of P1 flows
- 70% of P2 flows

Maintenance Schedule

Daily:
- Run full regression suite
- Review failures
- Fix broken tests

Weekly:
- Review coverage metrics
- Update test data
- Remove flaky tests

Monthly:
- Refactor test code
- Update test infrastructure
- Review test effectiveness

Reporting

// Generate test report
$report = [
    'total' => $this->getTestCount(),
    'passed' => $this->getPassedCount(),
    'failed' => $this->getFailedCount(),
    'skipped' => $this->getSkippedCount(),
    'duration' => $this->getDuration(),
    'coverage' => $this->getCoveragePercent(),
    'failures' => $this->getFailureDetails()
];

// Output as JSON for CI/CD
echo json_encode($report, JSON_PRETTY_PRINT);

Flaky Test Management

// Mark known flaky tests
/**
 * @group flaky
 * @group upgrade
 */
public function testSearchReindex(): void
{
    $this->markTestIncomplete('Flaky: ElasticSearch timing');
    // Test implementation
}

// Retry mechanism
/**
 * @retry 3
 */
public function testCacheFlush(): void
{
    // Implementation with automatic retry
}

Regression Suite Reports

Report Components:
1. Executive Summary
   - Total tests: 1,247
   - Pass rate: 98.2%
   - Duration: 45 minutes
   - Coverage: 72%

2. Failed Tests
   - Test name
   - Failure reason
   - Screenshot (E2E)
   - Stack trace

3. Coverage Delta
   - New coverage: +2.3%
   - Decreased coverage: -0.5%
   - Uncovered: 12 critical paths

4. Performance Metrics
   - Average test duration
   - Slowest tests
   - Resource usage