Automated Testing in CI/CD

Test Pipeline Stages

Code Push/PR
  |
  +-- Stage 1: Static Analysis
  |     phpstan, phpcs
  |
  +-- Stage 2: Unit Tests
  |     Fast, isolated, no DB
  |
  +-- Stage 3: Integration Tests
  |     Database, services
  |
  +-- Stage 4: API Tests
  |     REST/GraphQL endpoints
  |
  +-- Stage 5: Functional Tests (MFTF)
  |     Browser-based
  |
  +-- Stage 6: Performance Tests
        Load testing

Pipeline as Code

# .github/workflows/test-pipeline.yml
name: Test Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  static-analysis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
      - run: composer install --prefer-dist
      - name: PHPStan
        run: vendor/bin/phpstan analyse --level=6 app/code/Vendor/
      - name: PHPCS
        run: vendor/bin/phpcs --standard=PSR12 app/code/Vendor/

  unit-tests:
    needs: static-analysis
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
          coverage: xdebug
      - run: composer install --prefer-dist
      - name: Unit Tests
        run: vendor/bin/phpunit -c dev/tests/unit/phpunit.xml.dist
      - name: Coverage Report
        uses: codecov/codecov-action@v3
        with:
          files: dev/tests/unit/coverage/clover.xml

  integration-tests:
    needs: unit-tests
    runs-on: ubuntu-latest
    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: root
          MYSQL_DATABASE: magento_test
        ports: ['3306:3306']
      redis:
        image: redis:6
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v3
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
      - run: composer install --prefer-dist
      - name: Setup Magento
        run: |
          bin/magento setup:install \
            --db-host=127.0.0.1 \
            --db-name=magento_test \
            --db-user=root \
            --db-password=root
      - name: Integration Tests
        run: vendor/bin/phpunit -c dev/tests/integration/phpunit.xml.dist

Key Takeaway

Test pipelines run stages sequentially with dependencies. Static analysis first, then unit tests, integration tests, and functional tests. Each stage gates the next.

JUnit XML Reports

<!-- phpunit.xml.dist -->
<phpunit>
    <logging>
        <log type="junit" target="test-results.xml"/>
        <log type="coverage-clover" target="coverage.xml"/>
        <log type="coverage-html" target="coverage-html/"/>
    </logging>
</phpunit>

GitHub Actions Test Reporting

- name: Run Tests
  run: vendor/bin/phpunit --log-junit test-results.xml
  
- name: Publish Test Results
  uses: mikepenz/action-junit-report@v3
  if: always()
  with:
    report_paths: test-results.xml
    summary: true

Coverage Reporting with Codecov

- name: Run Tests with Coverage
  run: |
    vendor/bin/phpunit \
      --coverage-clover coverage.xml \
      --coverage-html coverage-html/

- name: Upload Coverage
  uses: codecov/codecov-action@v3
  with:
    files: coverage.xml
    flags: unittests
    name: magento-unit-coverage
    fail_ci_if_error: false

Coverage Thresholds

# codecov.yml
coverage:
  status:
    project:
      default:
        target: 80%
        threshold: 1%
    patch:
      default:
        target: 70%

Test Dashboard

# Generate HTML report
- name: Generate Report
  run: |
    vendor/bin/phpunit --coverage-html coverage/ \
      --coverage-clover coverage.xml \
      --log-junit results.xml

- name: Deploy Report
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: coverage/

Key Takeaway

Use JUnit XML for test results, Clover XML for coverage, and Codecov for tracking. Set coverage thresholds to prevent regression.

PHPUnit Parallel Execution

# Using parallel test runner
vendor/bin/phpunit --testsuite "Unit" --testsuite "Integration" --parallel

# Using parallel-linter
vendor/bin/parallel-lint src/ --exclude vendor/

GitHub Actions Matrix Strategy

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - suite: Model
            path: Test/Unit/Model/
          - suite: Block
            path: Test/Unit/Block/
          - suite: Controller
            path: Test/Unit/Controller/
          - suite: Helper
            path: Test/Unit/Helper/
    steps:
      - uses: actions/checkout@v3
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
      - run: composer install --prefer-dist
      - name: Run ${{ matrix.suite }} Tests
        run: vendor/bin/phpunit --filter ${{ matrix.path }}

Splitting Tests by Time

# Split tests into balanced groups
groups:
  splitting:
    - group1: Test/Unit/Model/
    - group2: Test/Unit/Block/
    - group3: Test/Unit/Controller/
    - group4: Test/Unit/Helper/

Docker-Based Parallel Testing

# docker-compose.test.yml
version: '3.8'
services:
  test-1:
    build: .
    command: vendor/bin/phpunit --testsuite Suite1
  test-2:
    build: .
    command: vendor/bin/phpunit --testsuite Suite2
  test-3:
    build: .
    command: vendor/bin/phpunit --testsuite Suite3
  test-4:
    build: .
    command: vendor/bin/phpunit --testsuite Suite4

Result Aggregation

# Aggregate parallel results
- name: Merge Test Results
  run: |
    # Merge JUnit XML files
    vendor/bin/phpunit-merge-results \
      result-1.xml result-2.xml result-3.xml \
      --output merged-results.xml

Key Takeaway

Parallelization reduces test execution time. Use matrix strategy in GitHub Actions, Docker containers for isolation, and merge results for reporting.

MFTF in CI

mftf-tests:
  runs-on: ubuntu-latest
  services:
    mysql:
      image: mysql:8.0
      env:
        MYSQL_ROOT_PASSWORD: root
        MYSQL_DATABASE: magento_test
    chrome:
      image: selenium/standalone-chrome
      ports: ['4444:4444']
  steps:
    - uses: actions/checkout@v3
    - uses: shivammathur/setup-php@v2
      with:
        php-version: '8.1'
    - run: composer install --prefer-dist
    - name: Setup Magento
      run: bin/magento setup:install --db-name=magento_test --db-user=root --db-password=root
    - name: Run MFTF
      run: |
        vendor/bin/mftf run:suite adminLogin \
          --url=http://localhost \
          --selenium-host=selenium://chrome:4444
    - name: Generate Allure Report
      if: always()
      run: |
        vendor/bin/allure generate dev/tests/functional/Mftf/Allure/ \
          --output allure-results/

Test Environment Setup

#!/bin/bash
# scripts/setup-test-env.sh

echo "Setting up test environment..."

# Setup database
mysql -u root -proot -e "CREATE DATABASE IF NOT EXISTS magento_test;"

# Install Magento
bin/magento setup:install \
  --db-host=127.0.0.1 \
  --db-name=magento_test \
  --db-user=root \
  --db-password=root \
  --admin-user=admin \
  --admin-password=admin123 \
  --base-url=http://localhost/

# Deploy static content
bin/magento setup:static-content:deploy -f

# Enable developer mode
bin/magento deploy:mode:set developer

echo "Test environment ready!"

Smoke Tests After Deployment

smoke-tests:
  runs-on: ubuntu-latest
  needs: deploy
  steps:
    - name: Health Check
      run: |
        curl -f http://staging.example.com/rest/V1/store/storeConfigs || exit 1
        
    - name: Homepage Check
      run: |
        STATUS=$(curl -s -o /dev/null -w '%{http_code}' http://staging.example.com/)
        if [ $STATUS -ne 200 ]; then
          echo "Homepage returned $STATUS"
          exit 1
        fi
        
    - name: Checkout Flow
      run: |
        # Basic checkout smoke test
        curl -f http://staging.example.com/checkout/ || exit 1

Key Takeaway

Integrate MFTF into CI with Selenium services. Automate test environment setup. Include smoke tests after deployment to verify critical paths.