When it comes to integrating Jira, GitLab, and CodeBuild for a smooth CI/CD process, things can get messy. Add a testing framework like Playwright and Xray for test management, and you're in for a wild ride. Unfortunately, if you’ve read Xray’s documentation, you probably know it leaves a lot to be desired when it comes to guiding users on integrations involving CodeBuild and non-JUnit frameworks. So, I decided to tackle the problem and share the lessons learned along the way. If you're trying to make this work, this post might save you some headaches.
The Goal: Seamless Integration from Jira to CodeBuild
The pipeline we needed looked like this:
Jira: Trigger CI jobs when specific test executions are requested.
GitLab: Handle the CI pipeline that will trigger codeBuild
AWS CodeBuild: Execute the actual tests in a managed environment.
Xray: Manage the test results back in Jira.
On paper, it seems straightforward: Jira sends a webhook, GitLab runs the pipeline, CodeBuild executes the tests, and Xray collects the results. The reality? A tangled web of missing documentation and unexpected blockers.
Problem 1: Xray Documentation Assumes JUnit-Only Integration
Xray’s documentation focuses almost entirely on integrating JUnit-style test frameworks. While that’s fine for basic scenarios, it’s nearly useless if your tests are built in Playwright. There’s no clear explanation of how to:
Trigger a pipeline based on Jira test executions.
Send Playwright’s JSON or other custom formats to Xray.
Work around the lack of direct CodeBuild support in Xray.
After a lot of trial and error, I managed to bridge the gap.
Step 1: Webhooks from Jira to GitLab
The first step is ensuring that a test execution in Jira triggers the GitLab pipeline. You’ll need to ask your Jira admin for API keys and project settings permission. also you need to configure a webhook in Jira that targets your GitLab CI endpoint.
Project Settings > Automation > Create Rule
When:
Then:
Webhook Example:
{
"token": "<trigger-token>",
"ref": "main",
"variables": {
"JIRA_ISSUE_KEY": "{{issue.key}}",
"JIRA_SUMMARY": "{{issue.summary}}"
}
}This payload sends the issue key and summary to the GitLab pipeline, which can then dynamically decide which tests to run.
To trigger this Go to > Jira Testing Board > Test Plan > Edit > Actions > (title of your automation task)
Step 2: GitLab Pipeline for Playwright
In GitLab, the pipeline should trigger code build Here’s a simplified .gitlab-ci.yml configuration:
stages:
- test
trigger-codebuild:
stage: test
image: amazon/aws-cli:latest
script:
- echo $JIRA_ISSUE_KEY
- aws codebuild start-build --project-name ui-automation-playwright --environment-variables-override '[{"name":"JIRA_ISSUE_KEY","value":"'"$JIRA_ISSUE_KEY"'","type":"PLAINTEXT"}, {"name":"XRAY_CLIENT_ID","value":"'"$XRAY_CLIENT_ID"'","type":"PLAINTEXT"}, {"name":"XRAY_CLIENT_SECRET","value":"'"$XRAY_CLIENT_SECRET"'","type":"PLAINTEXT"}]'
rules:
- if: '$TRIGGER_SOURCE == "jira"'
when: always
tags:
- codebuild-ui-automation-playwright-$CI_PROJECT_ID-$CI_PIPELINE_IID-$CI_JOB_NAME (Ask your devops for (AWS) tags)
Add xray API keys into Gitlab Variables
This pipeline:
Triggers AWS Codebuild and there dependencies and Playwright browsers.
.
Step 3: Executing Tests in AWS CodeBuild
Using AWS CodeBuild adds scalability and flexibility but complicates the pipeline. GitLab must trigger CodeBuild, and CodeBuild needs to execute Playwright tests and return results.
CodeBuild Buildspec:
version: 0.2
phases:
install:
runtime-versions:
nodejs: 18
commands:
- echo "Updating system packages..."
- apt-get update && apt-get install -y libgbm1
- echo "Installing npm dependencies..."
- npm install
- echo "Installing Playwright browsers..."
- npx playwright install
build:
commands:
- echo Running tests...
- npx playwright test
post_build:
commands:
- echo "Uploading test results to Jira Xray for issue $JIRA_ISSUE_KEY with secret $XRAY_CLIENT_SECRET and id $XRAY_CLIENT_ID ..."
- echo "Generating Xray token..."
- >
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST --data "{\"client_id\": \"$XRAY_CLIENT_ID\", \"client_secret\": \"$XRAY_CLIENT_SECRET\"}" https://xray.cloud.getxray.app/api/v2/authenticate | tr -d '"')
- if [ -z "$TOKEN" ]; then echo "Failed to retrieve token"; exit 1; fi
- echo "Token generated successfully."
- ls
- |
curl -X POST \
-H "Content-Type: text/xml" \
-H "Authorization: Bearer $TOKEN" \
--data @results/test-results.xml \
"https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=AAA&testPlanKey=$JIRA_ISSUE_KEY"
artifacts:
files:
- 'results/*'
- '!node_modules/**/*'
discard-paths: yes
reports:
test-report:
files:
- 'test-results.xml'
base-directory: resultsThis configuration ensures Playwright runs on CodeBuild and outputs results compatible with Xray.
Step 3: Reading report on AWS and in Jira
in AWS CodeBuild based on this code
reports:
test-report:
files:
- 'test-results.xml'
base-directory: resultsyou will have reports into codebuild
But also into Jira Xray: all steps will be imported into Test Repository
Lessons Learned
Xray’s Documentation Falls Short:
Playwright integration is barely covered.
CodeBuild isn’t mentioned, requiring custom solutions.
Webhooks Are Your Friend:
Jira’s webhook feature is powerful but requires manual fine-tuning.
Keep It Lightweight for Merge Requests:
For merge requests, focus on smoke tests to reduce costs.
Custom APIs Help Where Official Ones Don’t:
In some cases, internal APIs (like Jira’s test repository API) can bridge gaps where official documentation fails.
Final Thoughts
Building a Jira > GitLab > CodeBuild pipeline with Playwright isn’t straightforward. Between incomplete documentation and tool limitations, you’ll need to get creative and use Chat GPT :D . That said, the flexibility this setup provides is worth the effort—once it works.
If you’re trying to build something similar, feel free to reach out. I’ve been through the chaos, so you don’t have to.
Comments