Running Tests in DevBox#

This page provides guidance on how to run Playwright tests in a DevBox environment, covering environment configuration, authentication, and test execution.

Setting Up Your DevBox Environment#

Before you can run Playwright tests, you need to configure your DevBox environment with the necessary parameters.

Configuring Environment Variables#

Open a PowerShell terminal in VS Code and set the required environment variables:

# Basic environment setup
$env:BROWSER = "chromium"
$env:HEADLESS = "false"
$env:RUN_TYPE = "nightly"
$env:RUN_ENV = "test"
$env:RUN_GEO = "us"

# Application-specific configuration
$env:ENV_TYPE = "makerShell"  # Replace with your application type
$env:ACCOUNT_ALIAS = "MakerShell_Nightly_Test_Us"  # Replace with your account alias

# Directory configuration - adjust paths to match your setup
$env:CONFIG_DIR = "D:\p\4\packages\playwright-integration-tests\src\config"
$env:OUTPUT_DIR = "D:\p\4\packages\playwright-integration-tests\"
$env:TEST_DIR = "D:\p\4\packages\playwright-integration-tests\src\apps"

# Authentication
$env:KEY_VAULT_PASSWORD = "your-key-vault-password"  # Contact security admin for this value

Note: For KEY_VAULT_PASSWORD, contact your team’s security administrator. This password is required for accessing credentials stored in Azure Key Vault.

Remember to reopen Visual Studio Code after making any changes to environment variables to ensure they take effect.

Configuring Environment Variables in launch.json#

For an improved development experience, you can configure environment variables in your VS Code launch.json file (in the .vscode directory):

{
  "type": "node",
  "name": "Login to Playwright New Framework",
  "request": "launch",
  "cwd": "${workspaceFolder}",
  "args": ["-c", "playwright.config.ts", "--alias=MakerShell_Nightly_Test_Us"],
  "program": "${workspaceRoot}/packages/playwright-integration-tests/src/login.ts",
  "outFiles": [
    "${workspaceRoot}/packages/playwright-integration-tests/lib/**/*.js"
  ],
  "sourceMaps": true,
  "env": {
    "BROWSER": "chromium",
    "HEADLESS": "false",
    "RUN_TYPE": "nightly",
    "RUN_ENV": "test",
    "RUN_GEO": "us",
    "ENV_TYPE": "makerShell",
    "CONFIG_DIR": "${workspaceRoot}/packages/playwright-integration-tests/src/config",
    "OUTPUT_DIR": "${workspaceRoot}/packages/playwright-integration-tests",
    "KEY_VAULT_PASSWORD": "" // Add your password here
  }
}

Authenticating to Test Environments#

After configuring your environment variables, you need to authenticate before running tests:

  1. Open the Debug tab in VS Code (Ctrl+Shift+D)

  2. Select “Login to Playwright New Framework” from the dropdown menu

  3. Click the play button to start authentication

This launches a browser window that will automatically authenticate using the configured credentials and create a storageState.json file with authentication information that tests will use.

Running Tests#

Once authentication is complete, you can run tests using several methods:

Method 1: Running from VS Code UI#

The easiest way to run a test is to use the VS Code UI:

  1. Open the test file in VS Code

  2. Right-click on the test file

  3. Select “Run Test” from the context menu

Method 2: Running from Terminal#

Run tests from the terminal where you’ve configured your environment variables:

# Run all tests
npx playwright test

# Or via npm script
npm run int-test

Running Specific Tests#

Execute a Single Test File#

npx playwright test modelApp.test.ts

Execute a Single Test Case#

Use the -g flag to run tests that match a specific pattern:

# Run tests that match a specific ID
npx playwright test -g "834453"

# Run tests that match a string in the title
npx playwright test -g "should create a new app"

Filtering Tests by Configuration#

By Run Type#

Use the RUN_TYPE environment variable to filter tests:

# Run only tests tagged as "runner" type
$env:RUN_TYPE = "runner"
npx playwright test

# Run tests that have multiple run types
$env:RUN_TYPE = "runner, performance"
npx playwright test

# Exclude specific run types (using ~ prefix)
$env:RUN_TYPE = "~nightly"
npx playwright test

# Include one type but exclude another
$env:RUN_TYPE = "nightly, ~performance"
npx playwright test

By Environment#

Filter tests by environment type:

# Run only production environment tests
$env:RUN_ENV = "prod"
npx playwright test

# Run only tests for a specific geography
$env:RUN_GEO = "us"
npx playwright test

# Run only tests for a specific application type
$env:ENV_TYPE = "makerShell"
npx playwright test

Handling Test Failures#

By default, failed tests don’t automatically retry. To enable retries:

# Retry failed tests once
$env:RETRIES = 1
npx playwright test

# Retry failed tests twice
$env:RETRIES = 2
npx playwright test

Debugging Tests#

Using VS Code Debugger#

To debug tests with breakpoints:

  1. Set breakpoints in your test files

  2. Open JavaScript Debug Terminal (from Terminal dropdown)

  3. Set your environment variables in this terminal

  4. Run your tests with Playwright commands

Using Playwright Inspector#

To use the Playwright Inspector for debugging:

# Run in debug mode
npm run test:debug

This opens the Playwright Inspector UI, allowing you to step through test execution, inspect the DOM, and visualize action targets.

Simulating a Lab Run#

To simulate how tests run in the lab environment:

  1. Run the login command from the root package folder:

    npm run login

  2. Run Playwright tests from a non-admin terminal:

    npm run int-test

Test Results and Artifacts#

After test execution completes, you can find:

  • A JUnit XML report in the outputDir specified in the configuration

  • Screenshots for failed tests (configured with screenshot: "only-on-failure")

  • Trace files for debugging (configured with trace: "retain-on-failure")

  • Videos of test failures (configured with video: "on-first-retry")

These artifacts help diagnose and fix test failures efficiently.