Automation test in SCI project

Automation test in SCI project

Offers a comprehensive guide to implementing automated testing. It covers the technology stack with a focus on JavaScript/TypeScript and the Playwright testing framework, including API generation and project structure. It explains how to use Playwright fixtures, how to customize them, and why. It provides practical advice on writing different types of test cases. These include general, E2E, and API tests. Reporting is covered with an overview and details of custom Playwright reporters, summarised reporting, and integrating with TestRail. The article also discusses the setup for nightly and manual testing on CircleCI, and concludes with plans for the future development of features in the test automation area.
Automation test in SCI project

1. Tech stack overview


1.1. Language stack: Javascript/Typescript

  • Using Typescript for strong type and reducing wrong attributes
  • Node package management tool: Yarn
  • Code lint: EsLint (simple-import-sort + prettier + ...)

1.2. Testing framework: Playwright

1.2.1. What Playwright?

  • Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox.
  • Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed.
  • Cross-language. Use the Playwright API in TypeScriptJavaScriptPython.NETJava.
  • Test Mobile Web. Native mobile emulation of Google Chrome for Android and Mobile Safari. The same rendering engine works on your Desktop and in the Cloud.

1.2.2. Why Playwright?

1.2.3. Playwright limitations

  • Does not support Microsoft Edge or earlier IE11.
  • Uses a desktop browser instead of a real device to emulate a mobile device

1.3. API Generator

1.3.1. Why API Generator?

  • Consume API is very clear and simple via classes and methods
  • Don't need any further confirmation between BE, FE and QA

Example:

// File: without_api_generator.ts
test
('Test case 1', async ({ request }) => { const newIssue = await request.post(`/repos/github-username/test-repo-1/issues`, { headers: { 'Accept': 'application/vnd.github.v3+json', // Add GitHub personal access token. 'Authorization': `token ${process.env.API_TOKEN}`, }, data: { title: '[Bug] report 1', body: 'Bug description', }, }) expect(newIssue.ok()).toBeTruthy() const issues = await request.get(`/repos/github-username/test-repo-1/issues`, { headers: { 'Accept': 'application/vnd.github.v3+json', // Add GitHub personal access token. 'Authorization': `token ${process.env.API_TOKEN}`, }, }) expect(issues.ok()).toBeTruthy() expect(await issues.json()).toContainEqual(expect.objectContaining({ title: '[Bug] report 1', body: 'Bug description', })) })
// File: with_api_generator.ts
test
('Test case 1', async ({ githubClient }) => { const newIssue = await githubClient.createIssue({ data: { title: '[Bug] report 1', body: 'Bug description', }, }) expect(newIssue.ok()).toBeTruthy() const issues = await githubClient.getIssues() expect(issues.ok()).toBeTruthy() expect(await issues.json()).toContainEqual(expect.objectContaining({ title: '[Bug] report 1', body: 'Bug description', })) })

1.3.2. How to generate API

  • BE will generate swagger.json based-on services' implementation
  • FE will generate API services from swagger.json via using swagger-typescript-api:
    • Overwrite http-client.eta to use APIContextRequest of Playwright instead of fetch
// File: http-client.eta
<% const { apiConfig, generateResponses, config } = it; %> import { APIRequestContext } from 'playwright-core'; export type QueryParamsType = Record<string | number, any>; export type ResponseFormat = keyof Omit<Body, "body" | "bodyUsed">; export interface FullRequestParams extends Omit<RequestInit, "body"> { /** set parameter to `true` for call `securityWorker` for this request */ secure?: boolean; /** request path */ path: string; /** content type of request body */ type?: string /** method of request */ method?: string /** query params */ query?: QueryParamsType; /** format of response (i.e. response.json() -> format: "json") */ format?: ResponseFormat; /** request body */ body?: unknown; /** base url */ baseUrl?: string; /** request cancellation token */ cancelToken?: CancelToken; } export type RequestParams = Omit<FullRequestParams, "body" | "query" | "path"> export type DataResponse<T, E> = { //[x: string]: any response: T error: E status: number statusText: string } export interface ApiConfig<SecurityDataType = unknown> { baseUrl?: string; baseApiParams?: Omit<RequestParams, "baseUrl" | "cancelToken" | "signal">; securityWorker?: (securityData: SecurityDataType | null) => Promise<RequestParams | void> | RequestParams | void; customFetch?: typeof fetch; } export interface HttpResponse<D extends unknown, E extends unknown = unknown> extends Response { data: D; error: E; } type CancelToken = Symbol | string | number; export enum ContentType { Json = "application/json", FormData = "multipart/form-data", UrlEncoded = "application/x-www-form-urlencoded", Text = "text/plain", } export class HttpClient<SecurityDataType = unknown> { public baseUrl: string = "<%~ apiConfig.baseUrl %>"; private apiContext: APIRequestContext; private securityData: SecurityDataType | null = null; private securityWorker?: ApiConfig<SecurityDataType>["securityWorker"]; private abortControllers = new Map<CancelToken, AbortController>(); private getXRequestId(method: string, url: string): string { const randomString = Math.random() return `${method.toLowerCase()}-${url}-${randomString}`.replace(/[^a-zA-Z0-9]/g, '-') } private customFetch = (url, fetchParams: Parameters<typeof fetch>[1]) => { const { 'Content-Type': contentType, ...headers } = fetchParams.headers as Record<string, string> const fetchOptions = { headers: { ...headers, ...(contentType !== ContentType.FormData ? { 'Content-Type': contentType, } : {}), }, method: fetchParams.method, } if (contentType === ContentType.FormData) { fetchOptions['multipart'] = fetchParams.body } else { fetchOptions['data'] = fetchParams.body } return this.apiContext.fetch(url, fetchOptions) } private baseApiParams: RequestParams = { credentials: 'same-origin', headers: {}, redirect: 'follow', referrerPolicy: 'no-referrer', } constructor(apiConfig: ApiConfig<SecurityDataType> = {}, apiContext: APIRequestContext) { Object.assign(this, apiConfig); this.apiContext = apiContext; } public setSecurityData = (data: SecurityDataType | null) => { this.securityData = data; } protected encodeQueryParam(key: string, value: any) { const encodedKey = encodeURIComponent(key); return `${encodedKey}=${encodeURIComponent(typeof value === "number" ? value : `${value}`)}`; } protected addQueryParam(query: QueryParamsType, key: string) { return this.encodeQueryParam(key, query[key]); } protected addArrayQueryParam(query: QueryParamsType, key: string) { const value = query[key]; return value.map((v: any) => this.encodeQueryParam(key, v)).join("&"); } protected toQueryString(rawQuery?: QueryParamsType): string { const query = rawQuery || {}; const keys = Object.keys(query).filter((key) => "undefined" !== typeof query[key]); return keys .map((key) => Array.isArray(query[key]) ? this.addArrayQueryParam(query, key) : this.addQueryParam(query, key), ) .join("&"); } protected addQueryParams(rawQuery?: QueryParamsType): string { const queryString = this.toQueryString(rawQuery); return queryString ? `?${queryString}` : ""; } private contentFormatters: Record<ContentType, (input: any) => any> = { [ContentType.Json]: (input:any) => input !== null && (typeof input === "object" || typeof input === "string") ? JSON.stringify(input) : input, [ContentType.Text]: (input:any) => input !== null && typeof input !== "string" ? JSON.stringify(input) : input, [ContentType.FormData]: (input: any) => Object.keys(input || {}).reduce((formData, key) => { const property = input[key]; formData[key] = property instanceof Blob ? property : typeof property === "object" && property !== null ? JSON.stringify(property) : `${property}` return formData; }, {}), [ContentType.UrlEncoded]: (input: any) => this.toQueryString(input), } protected mergeRequestParams(params1: RequestParams, params2?: RequestParams): RequestParams { return { ...this.baseApiParams, ...params1, ...(params2 || {}), headers: { ...(this.baseApiParams.headers || {}), ...(params1.headers || {}), ...((params2 && params2.headers) || {}), }, }; } protected createAbortSignal = (cancelToken: CancelToken): AbortSignal | undefined => { if (this.abortControllers.has(cancelToken)) { const abortController = this.abortControllers.get(cancelToken); if (abortController) { return abortController.signal; } return void 0; } const abortController = new AbortController(); this.abortControllers.set(cancelToken, abortController); return abortController.signal; } public abortRequest = (cancelToken: CancelToken) => { const abortController = this.abortControllers.get(cancelToken) if (abortController) { abortController.abort(); this.abortControllers.delete(cancelToken); } } public request = async <T = any, E = any>({ body, secure, path, type, query, format, baseUrl, cancelToken, ...params }: FullRequestParams): Promise<DataResponse<T, E>> => { const secureParams = ((typeof secure === 'boolean' ? secure : this.baseApiParams.secure) && this.securityWorker && await this.securityWorker(this.securityData)) || {}; const requestParams = this.mergeRequestParams(params, secureParams); const queryString = query && this.toQueryString(query); const payloadFormatter = this.contentFormatters[type || ContentType.Json]; const responseFormat = format || requestParams.format; const customFetchParams: Parameters<typeof fetch>[1] = { ...requestParams, headers: { ...(requestParams.headers || {}), ...(type ? { 'Content-Type': type } : {}), 'X-REQUEST_ID': this.getXRequestId(params.method, path), }, signal: cancelToken ? this.createAbortSignal(cancelToken) : requestParams.signal, } if (typeof body !== 'undefined' && body !== null) { customFetchParams['body'] = payloadFormatter(body) } return this.customFetch(`${baseUrl || this.baseUrl || ""}${path}${queryString ? `?${queryString}` : ""}`, customFetchParams).then(async (response) => { const returned: DataResponse<T, E> = { response: null, error: null, status: response.status(), statusText: response.statusText(), } if (responseFormat) { await response[responseFormat]() .then((data) => { if (response.ok()) { returned.response = data } else { returned.error = data } }) .catch((e) => { returned.error = e }) } else { const r = response as unknown as HttpResponse<T, E> returned.response = r.data returned.error = r.error } if (cancelToken) { this.abortControllers.delete(cancelToken) } return returned }); }; }
  • Use swagger-typescript-api to generate API services files
// File: gen-api.ts
import
{ render } from 'eta' import * as fs from 'fs' import * as path from 'path' import { generateApi, GenerateApiParams } from 'swagger-typescript-api' import { toCamelCase } from '/path/to/string/utilities' type RequestParameter = { name: string optional: boolean type: string description: string } type RequestPayload = { name: string optional: boolean type: string } type ParsedRequest = Request & { path: string parameters: RequestParameter[] payload?: RequestPayload } async function genOpenApi(swaggerPath: string, outputPath: string, typeAsAny = false) { const options: GenerateApiParams = { name: 'open-api.ts', output: outputPath, input: swaggerPath, templates: templatePath, httpClientType: 'fetch', generateUnionEnums: false, enumNamesAsValues: true, singleHttpClient: true, cleanOutput: true, modular: true, moduleNameFirstTag: true, generateResponses: true, generateClient: true, unwrapResponseData: true, extractEnums: true, extractResponseError: true, extractRequestBody: true, hooks: { onCreateRoute: (parsedRoute) => { const request = parsedRoute.request as ParsedRequest if (request.parameters.length > 0) { // Replace "{*abc}" to "${abc}" request.path = request.path.replace(/\{\*(\w+)\}/, '${$1}') } if (typeAsAny && request.payload) { request.payload.optional = true request.payload.type += ' | any' return { ...parsedRoute, request, } } }, }, } // Support type as any to check type validation rules of BE if (typeAsAny) { options.codeGenConstructs = (struct) => ({ ...struct, TypeField: ({ readonly, key, value }) => { return `${readonly ? 'readonly' : ''} ${key}?: ${value} | any` }, }) } // Generate open-api files // ------------------------------ try { const { files } = await generateApi(options) const fileList = files .filter((file) => !['data-contracts.ts'].includes(file.name)) .map((file) => { const fileNameWithoutExtension = file.name.replace('.ts', '') return { fileName: file.name, className: fileNameWithoutExtension, instanceName: toCamelCase(fileNameWithoutExtension), } }) .sort((a, b) => a.className.toLowerCase().localeCompare(b.className.toLowerCase())) const servicesConnectorPath = path.join(outputPath, 'services-connector.ts') fs.writeFileSync('/path/to/service-connector', renderTemplate('/path/to/service-connector-template', { fileList })) } catch (e) { console.error(e) } } // Generate open-api ;(async () => { await genOpenApi('/path/to/swagger.json', '/path/to/open-api/') await genOpenApi('/path/to/swagger.json', '/path/to/open-api-as-any/', true) })()

1.4. Folder structure

  • Folder structure follows monorepo architecture (Turborepo)

1.4.1. Project structure

  • .circleci contains CircleCI config files
  • apps contains testing apps
  • packages contains common packages which are used in testing app
    • common-tools contains code for common tools which will be used in testing apps. Example: gen-open-api , unify-test-case , etc...
    • eslint-config-custom contains base configuration for EsLint
    • playwright-common contains common components, pages, types, and utils which are used in test cases, fixtures, etc...
    • summary-report is a customer report to count total passed, failed, flaky, skipped, timed out, or interrupted. Is used for Slack notification
    • tsconfig contains base configuration for Typescript
  • .editorconfig defines coding styles for multiple developers working on the same project across various editors and IDEs
  • .eslintrc.js is EsLint root config
  • .gitignore contains GIT ignored file list
  • .npmrc stores the feed URLs and credentials that Npm uses
  • .prettierignore and .prettierrc contains configuration for prettier (Code Formatter)
  • package.json contains package information such as name, version, dependencies, scripts, etc...
  • README.md
  • turbo.json contains configuration for Turborepo
  • yarn.lock stores exactly which versions of each dependency were installed. More details at here

1.4.2. A testing app structure

  • __reports contains testing reports including Playwright, Allure, etc...
  • __results contains trace files which are used for Trace Viewer
  • data contains data for testing
  • libs contains helpers to support testing
    • api contains helpers for API testing
    • constants contains constants
    • fixtures contains test base for E2E and API testing
    • ui contains helpers for E2E testing
      • components contains components' fixtures such as globalHeadersidebarMenu, etc...
      • pages contains pages' fixtures such as loginPageswitchOfficePagecreateOfficePage, etc...
    • sql-scripts contains SQL scripts which are used for setup and teardown of testing
    • utils contains utilities
  • tests contains all test cases
  • tools contains utility tools which not belong to helpers. Example: gen-open-apiunify-test-case
  • .env.example contains an example for .env file
  • .eslintrc.js contains EsLint configuration
  • global.setup.ts contains global setup for all test cases
  • global.teardown.ts contains global teardown setup for all test cases
  • package.json contains package information such as name, version, dependencies, scripts, etc...
  • playwright.config.ts contains Playwright configuration
  • tsconfig.json contains Typescript configuration

2. Playwright Fixtures


2.1. What are Test fixtures?

  • Used to establish an environment for each test, giving the test everything it needs and nothing else.
  • Isolated between tests.
  • With fixtures, you can group tests based on their meaning, instead of their common setup.

2.2. Why fixtures?

  • Encapsulate setup and teardown in the same place so it is easier to write.
  • Reusable between test files - you can define them once and use them in all your tests. That's how Playwright's built-in page fixture works.
  • On-demand - you can define as many fixtures as you'd like, and Playwright Test will set up only the ones needed by your test and nothing else.
  • Composable - they can depend on each other to provide complex behaviors.
  • Flexible - tests can use any combinations of the fixtures to tailor the precise environment they need, without affecting other tests.
  • Simplify grouping - you no longer need to wrap tests in describes that set up environment, and are free to group your tests by their meaning instead.

2.2.1. Built-in fixtures

FixtureTypeDescription
pagePageIsolated page for this test run.
contextBrowserContextIsolated context for this test run. The page fixture belongs to this context as well. Learn how to configure context.
browserBrowserBrowsers are shared across tests to optimize resources. Learn how to configure browser.
browserNamestringThe name of the browser currently running the test. Either chromiumfirefox or webkit.
requestAPIRequestContextIsolated APIRequestContext instance for this test run.

2.3. Custom fixtures

To create your own fixture, use test.extend() to create a new test object that will include it.

// File: my-fixtures.ts
import
{ test as base } from '@playwright/test' import { TodoPage } from './todo-page' import { SettingsPage } from './settings-page' // Declare the types of your fixtures. type MyFixtures = { todoPage: TodoPage; settingsPage: SettingsPage; }; // Extend base test by providing "todoPage" and "settingsPage". // This new "test" can be used in multiple test files, and each of them will get the fixtures. export const test = base.extend<MyFixtures>({ todoPage: async ({ page }, use) => { // Set up the fixture. const todoPage = new TodoPage(page) await todoPage.goto() await todoPage.addToDo('item1') await todoPage.addToDo('item2') // Use the fixture value in the test. await use(todoPage) // Clean up the fixture. await todoPage.removeAll() }, settingsPage: async ({ page }, use) => { await use(new SettingsPage(page)) }, }) export { expect } from '@playwright/test'

Examples:

// File: without_fixtures.ts
const
{ test } = require('@playwright/test') const { TodoPage } = require('./todo-page') test.describe('todo tests', () => { let todoPage test.beforeEach(async ({ page }) => { todoPage = new TodoPage(page) await todoPage.goto() await todoPage.addToDo('item1') await todoPage.addToDo('item2') }) test.afterEach(async () => { await todoPage.removeAll() }) test('should add an item', async () => { await todoPage.addToDo('my item') // ... }) test('should remove an item', async () => { await todoPage.remove('item1') // ... }) })
// File: with_fixtures.ts
const
{ test } = require('@playwright/test') const { TodoPage } = require('./todo-page') test.describe('todo tests', () => { test.afterEach(async ({ todoPage }) => { await todoPage.removeAll() }) test('should add an item', async ({ todoPage }) => { await todoPage.addToDo('my item') // ... }) test('should remove an item', async ({ todoPage }) => { await todoPage.remove('item1') // ... }) })

3. How to write test case


3.1. Aliases

  • Purpose: to make test script look similar with other test frameworks like Jest, Mocha, Jasmine, etc...
  • Alias list:
    • describe is an alias of the test.describe
    • it is an alias of test
    • beforeAll is an alias of test.beforeAll
    • beforeEach is an alias of test.beforeEach
    • afterEach is an alias of test.afterEach
    • afterAll is an alias of test.afterAll
  • Example
// File: with-aliases.ts
describe
('Test suite', () => { beforeAll(async ({}) => { // beforeAll hook }) beforeEach(async ({}) => { // beforeEach hook }) it('Test case #1', async ({}) => { // Test case script }) it('Test case #2', async ({}) => { // Test case script }) afterEach(async ({}) => { // afterEach hook }) afterAll(async ({}) => { // afterAll hook }) })
// File: without-aliases.ts
test
.describe('Test suite', () => { test.beforeAll(async ({}) => { // beforeAll hook }) test.beforeEach(async ({}) => { // beforeEach hook }) test('Test case #1', async ({}) => { // Test case script }) test('Test case #2', async ({}) => { // Test case script }) test.afterEach(async ({}) => { // afterEach hook }) test.afterAll(async ({}) => { // afterAll hook }) })

3.2. Common test cases

3.2.1. Conventions

  • File name must be in lowercase. Can be a test suite or test case code. Example c12345.ts for test case and s56789.ts for test suite
  • Each test file must contain a TestRail link to the test case or test suite. Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
... it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => { ... })
  • Tip: Script yarn unify-testcase can help you:
    • Change your file name to be in lower-case
    • Prepend TestRail link to test case based-on your file name

3.2.2. Test script structure

Structure of test case contains following factors:

  • describe or test.describe to declare test suite
  • it or test to declare testcase
  • beforeAll or test.beforeAll to declare setup step for test suite
  • beforeEach or test.beforeEach to declare setup step for each test case
  • afterEach or test.afterEach to declare teardown step for each test case
  • afterAll or test.afterAll to declare teardown step for test suite

Example:

// File: with-test-suite.ts
describe
('Test suite', () => { beforeAll(async ({}) => { // beforeAll hook }) beforeEach(async ({}) => { // beforeEach hook }) it('Test case #1', async ({}) => { // Test case script }) it('Test case #2', async ({}) => { // Test case script }) afterEach(async ({}) => { // afterEach hook }) afterAll(async ({}) => { // afterAll hook }) })
// File: without-test-suite.ts
beforeAll
(async ({}) => { // beforeAll hook }) beforeEach(async ({}) => { // beforeEach hook }) it('Test case #1', async ({}) => { // Test case script }) it('Test case #2', async ({}) => { // Test case script }) afterEach(async ({}) => { // afterEach hook }) afterAll(async ({}) => { // afterAll hook })

3.3. E2E test cases

  • describeitbeforeAllbeforeEachafterEachafterAll and expect are imported from @/libs/fixtures/e2e
  • Available fixtures:
    • pagination
    • inlineNotification
    • toast
    • globalHeader
    • sidebarMenu
    • loginPage
    • switchOfficePage
    • newOfficePage
    • employeeListPage
    • employeeInvitePage
    • employeeDetailPage
    • emailSettingPage
    • emailTemplatePreviewPage
    • emailTemplateEditPage
    • sendPackageListPage
    • companySettingPage
    • senderInformationPage
  • Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
import { accountData } from '@/data/account' import { globalHeaderData } from '@/data/global-header' import { ACCOUNT_0_STATE } from '@/libs/constants/store' import { beforeEach, expect, it, use } from '@/libs/fixtures/e2e' use({ storageState: ACCOUNT_0_STATE }) beforeEach(async ({ sendPackageListPage }) => { await sendPackageListPage.goto() }) it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => { // User dropdown await globalHeader.openUserDropdown() await expect(globalHeader.userDropdown).toContainText(accountData.user[0].name) await expect(globalHeader.userSettingButton).toContainText(globalHeaderData.userDropdown.userSettingText) await expect(globalHeader.userSettingButton).toHaveAttribute( 'href', `${process.env.MFID_BASE_URL}${globalHeaderData.userDropdown.userSettingLink}` ) await expect(globalHeader.logoutButton).toBeVisible() await expect(globalHeader.logoutButton).toContainText(globalHeaderData.userDropdown.logoutButtonText) // Office dropdown await globalHeader.openOfficeDropdown() await expect(globalHeader.switchOfficeButton).toBeVisible() await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].tenantName) await expect(globalHeader.officeDropdown).toContainText(globalHeaderData.officeDropdown.officeBusinessNumberText) await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].businessNumber) await expect(globalHeader.switchOfficeButton).toContainText(globalHeaderData.officeDropdown.switchOfficeButtonText) })

3.4. API test cases

  • describeitbeforeAllbeforeEachafterEachafterAll and expect are imported from @/libs/fixtures/api
  • Available fixtures:
    • apiClient
    • navisClient
    • invoiceDeliveryClient
    • suzakuLoginPage
  • Example:
// File: c57134.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57134
import { accountData } from '@/data/receiver-account' import { describe, expect, it } from '@/libs/fixtures/api' /** * Endpoints: PUT /v1/user/terms */ describe('SCI-3365-Receiver can register MFID', () => { it('C57134-Verify agree the term process before user agrees the term', async ({ apiClient }) => { const loginResult = await apiClient.login(accountData.user[1].email, accountData.user[1].password) expect(loginResult.status).toBe(200) expect(loginResult.response).toHaveProperty('data') expect(loginResult.response.data.user).not.toBe(null) expect(loginResult.response.data.user.accept_terms).toBe(false) expect(loginResult.response.data.user.has_available_tenant).toBe(true) }) })

4. Reports


4.1. Report overview

  • Slack channel: #sci_automation_notification

  • Report type:

    • Daily: Every morning from Monday to Friday (GTM+7)
    • Monthly: TBD
  • Trigger type: nightly or manual

  • App name: sender or receiver

  • Testing type: api or e2e

  • Report format

    • Title: [{TRIGGER_TYPE}] [{APP_NAME}] {TESTING_TYPE} testing report for {RUN_DATE}

    • Content:

      • Total passed
      • Total failed
      • Total flaky
      • Total skipped (due to depending on test case is failed)
      • Total timed out (due to auto-waiting exceeding configured timeout)
      • Total interrupted
    • Example:

       

       

  • Link to report detail

    • Playwright report

    • Allure report

      allure_report

4.2. Custom reporter for Playwright

  • You can create a custom reporter by implementing a class with some of the reporter methods. Learn more about the Reporter API.
// File: my-awesome-reporter.ts
import
type { FullConfig, FullResult, Reporter, Suite, TestCase, TestResult, } from '@playwright/test/reporter' class MyReporter implements Reporter { onBegin(config: FullConfig, suite: Suite) { console.log(`Starting the run with ${suite.allTests().length} tests`) } onTestBegin(test: TestCase, result: TestResult) { console.log(`Starting test ${test.title}`) } onTestEnd(test: TestCase, result: TestResult) { console.log(`Finished test ${test.title}: ${result.status}`) } onEnd(result: FullResult) { console.log(`Finished the run: ${result.status}`) } } export default MyReporter
// File: playwright.config.ts
import
{ defineConfig } from '@playwright/test' export default defineConfig({ reporter: './my-awesome-reporter.ts', })

4.3. Summary Reporter

This report will calculate total passed, failed, flaky, skipped, timed-out and interrupted test cases as well as passed ratio and total duration. The result will stored in JSON format

// File: summary-reporter.ts
import
{ FullConfig, Reporter, Suite } from '@playwright/test/reporter' import * as fs from 'fs' import { getCurrentDate, convertMsToTime } from './path/to/helpers' export type AppName = 'sender' | 'receiver' export type TestType = 'e2e' | 'api' export type SummaryReporterOption = { outputFile: string appName: AppName testType: TestType } class SummaryReporter implements Reporter { private suiteList: Suite[] = [] private readonly outputFile: string private readonly appName: AppName private readonly testType: TestType constructor({ outputFile, appName, testType }: SummaryReporterOption) { this.outputFile = outputFile this.appName = appName this.testType = testType } onBegin(config: FullConfig, suite: Suite) { this.suiteList.push(suite) } onEnd() { let total = 0 let totalPassed = 0 let totalFailed = 0 let totalFlaky = 0 let totalSkipped = 0 let totalTimedOut = 0 let totalInterrupted = 0 let totalDuration = 0 this.suiteList.forEach((suite) => { suite.allTests().forEach((test) => { let isPassed = false let isFailed = false let isFlaky = false test.results.forEach((result) => { result.status === 'skipped' && totalSkipped++ result.status === 'timedOut' && totalTimedOut++ result.status === 'interrupted' && totalInterrupted++ result.duration if (result.retry === 0) { isPassed = result.status === 'passed' isFailed = result.status === 'failed' } else { if ((result.status === 'passed' && isFailed) || (result.status === 'passed' && isPassed)) { isFlaky = true isPassed = false isFailed = false } } totalDuration += result.duration }) isPassed && totalPassed++ isFailed && totalFailed++ isFlaky && totalFlaky++ total++ }) const passedRatio = (totalPassed / total) * 100 fs.writeFileSync( this.outputFile, JSON.stringify({ title: `[${this.testType.toUpperCase()}] [${this.appName.toUpperCase()}] Report for ${getCurrentDate()}`, total, totalPassed, totalFailed, totalFlaky, totalSkipped, totalTimedOut, totalInterrupted, passedRatio: isNaN(passedRatio) ? 'N/A' : `${passedRatio.toFixed(2)}%`, totalDuration: totalDuration === 0 ? 'N/A' : convertMsToTime(totalDuration), }) ) }) } } export default SummaryReporter

4.4. TestRail Reporter

This report will:

  • Detect TestRail test cases' id via test-case title
  • Create TestMilestone in the first day of a week (if does not exist)
  • Close TestMilestone in the last day of a week (if does not close)
  • Create TestRun whenever running an Automation Test
  • Close TestRun after running the Automation Test
  • Create TestResult for all detected test cases
// File: testrail-reporter.ts
import
TestRail from '@dlenroc/testrail' import { Reporter, TestCase, TestResult } from '@playwright/test/reporter' import dayjs from 'dayjs' import isoWeek from 'dayjs/plugin/isoWeek' import * as fs from 'fs' import { getCurrentDate, getCurrentWeek, getPrefixTitle, stripAnsi } from './path/to/helpers' dayjs.extend(isoWeek) export type AppName = 'sender' | 'receiver' export type TestType = 'e2e' | 'api' export type TestRailReporterOption = { outputFile: string host: string username: string password: string projectId: number suiteId: number appName: AppName testType: TestType } class TestRailReporter implements Reporter { private readonly testRailClient: TestRail private initTestMilestoneProcess: Promise<void> private testMilestone: TestRail.Milestone private testRun: TestRail.Run private readonly outputFile: string private readonly host: string private readonly projectId: number private readonly suiteId: number private readonly appName: AppName private readonly testType: TestType private testResultList: Array<TestRail.Result> = [] private readonly STATUS_MAPPING = { passed: 1, failed: 5, timedOut: 5, } constructor({ username, password, ...options }: TestRailReporterOption) { Object.assign(this, options) this.testRailClient = new TestRail({ host: options.host, username, password, }) } // Event handlers // -------------------------------- async onBegin() { this.initTestMilestoneProcess = this.initTestMilestone() } async onTestEnd(test: TestCase, result: TestResult) { await this.initTestResult(test, result) } async onEnd() { await this.initTestMilestoneProcess if (this.testResultList.length > 0) { await this.initTestRun() await this.updateTestResult() } this.genResult() await this.closeTestRun() await this.closeTestMilestone() } // Gen result // -------------------------------- genResult() { fs.writeFileSync( this.outputFile, JSON.stringify({ link: `${this.host}/index.php?/runs/view/${this?.testRun?.id || ''}`, }) ) } // TestRail businesses // -------------------------------- async initTestMilestone() { const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Milestone for ${getCurrentWeek()}` const milestoneList = await this.testRailClient.getMilestones(this.projectId, { is_completed: false, }) let testMilestone = milestoneList.filter((item) => item.name === name)[0] if (!testMilestone) { testMilestone = await this.testRailClient.addMilestone(this.projectId, { name, }) } if (!testMilestone.is_started) { testMilestone = await this.testRailClient.updateMilestone(testMilestone.id, { is_started: true, }) } this.testMilestone = testMilestone } async closeTestMilestone() { if (dayjs().day() === 5) { await this.testRailClient.updateMilestone(this.testMilestone.id, { is_completed: true, }) } } async initTestRun() { const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Run for ${getCurrentDate()}` const testRunList = await this.testRailClient.getRuns(this.projectId, { suite_id: this.suiteId, milestone_id: this.testMilestone.id, }) let testRun = testRunList.filter((item) => item.name === name)[0] if (!testRun) { testRun = await this.testRailClient.addRun(this.projectId, { suite_id: this.suiteId, milestone_id: this.testMilestone.id, name, }) } if (!testRun.include_all) { testRun = await this.testRailClient.updateRun(this.projectId, { include_all: true, }) } this.testRun = testRun } async closeTestRun() { await this.testRailClient.closeRun(this.testRun.id) } async initTestResult(test: TestCase, result: TestResult) { const testCaseIdList = test.title.match(/(C[0-9]+)/gi)?.map((item) => +item.replace(/c/i, '')) if (!Array.isArray(testCaseIdList) || testCaseIdList.length === 0) { return } if (result.status === 'skipped') { return } let runNumber = '' if (result.retry === 0) { runNumber = `First run` } else { runNumber = `Retry #${result.retry}` } let errorMessage = '' if (result.status === 'failed') { let errorDetails = '' errorDetails += stripAnsi(result.error.message) errorDetails += '\n' if (result.error.location) { errorDetails += ` at ${result.error.location.file}:${result.error.location.line}:${result.error.location.column}` } const errorArr = errorDetails.split('\n').map((item) => ` ${item}`) errorMessage += `\nError:\n\n${errorArr.join('\n')}` } this.testResultList = [ ...this.testResultList, ...testCaseIdList.map( (item) => ({ case_id: item, status_id: this.STATUS_MAPPING[result.status], comment: `${runNumber}\n\nDuration: ${result.duration}ms\n\n${errorMessage}`, } as unknown as TestRail.Result) ), ] } async updateTestResult() { await this.testRailClient.addResultsForCases(this.testRun.id, { results: this.testResultList, }) } } export default TestRailReporter

5. Nightly and manual test on CircleCI


Nightly and manual tests will run only on main branch

5.1. Nightly test

  • Will test api and e2e for all applications

  • Will be triggered at:

    • With UTC timezone: 8:00 PM on Sunday, Monday-Thursday
    • With GTM+7 timezone: 3:00 AM on Monday-Friday
  • Scheduled trigger on CircleCI:
    aaaaa

  • Purpose: ensures that you catch such problems within 24 hours when they occur

5.2. Manual test

  • Will be triggered manually when:

    • New code is added to main branch (can be merged PR or commit code directly to these branches)

    • Trigger via CircleCI

      • Step 1: Select main branch
      • Step 2: Click Trigger Pipeline button
        circleci111
  • Purpose: trigger api and e2e testing for a specified application on demand

6. Feature plans


  • Implement Monthly report about detected bugs
  • Auto create JIRA ticket when finding bug after running Automation Test
  • Generate test cases for testing type of parameters. Example: username: string param will be tested with following input:
    • null
    • ''
    • !@#$%^&*()
    • true
    • 123456789

More like this

A Summary of Live Coding a Simple CLI for gRPC Testing at Go Conference 2023
Jul 14, 2023

A Summary of Live Coding a Simple CLI for gRPC Testing at Go Conference 2023

Bear - Làm QA/QC Cũng Vui Như Chơi Game
Sep 16, 2022

Bear - Làm QA/QC Cũng Vui Như Chơi Game