Automation test in SCI project
1. Tech stack overview
1.1. Language stack: Javascript/Typescript
- Using Typescript for strong type and reducing wrong attributes
- Node package management tool: Yarn
- Code lint: EsLint (simple-import-sort + prettier + ...)
1.2. Testing framework: Playwright
1.2.1. What Playwright?
- Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox.
- Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed.
- Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java.
- Test Mobile Web. Native mobile emulation of Google Chrome for Android and Mobile Safari. The same rendering engine works on your Desktop and in the Cloud.
1.2.2. Why Playwright?
- Supports multiple-languages including Javascript/Typescript, Java, .NET, Python
- It supports multi-tab, multi-user, and iframe test cases.
- It is available as a VS Code extension with one-click-to-run tests and includes step-by-step debugging, selector exploration, and logging of new tests.
- It supports different types of testing, such as end-to-end, functional, and API testing.
- Supports automated accessibility testing with third-party plugins.
- It provides various debugging options like Playwright Inspector, Browser Developer Tools, VSCode Debugger, and Monitor Viewer Console Logs.
- It has built-in reporters like JSON, JUnit, and HTML Reporter. You can also create custom reports using Playwright
- Faster and more reliable than Cypress. There are a lot of online articles or topics that mention the unreliable of Cypress without a resolved solution:
- Cypress hanging on circleCI but pass locally
- Cypress Testing: stuck at wait-on http://localhost:3000
- Cypress Testing: stuck at wait-on http://localhost:3000 (StackOverflow)
- Cypress sometimes stalls/hangs with no output when running in Jenkins with Docker
- CYPRESS tests pass locally but are hanging on circleCI
- Cypress hung up trying to run a spec - won't resolve
- Cypress unexpectedly and randomly hangs when running tests in CI
1.2.3. Playwright limitations
- Does not support Microsoft Edge or earlier IE11.
- Uses a desktop browser instead of a real device to emulate a mobile device
1.3. API Generator
1.3.1. Why API Generator?
- Consume API is very clear and simple via classes and methods
- Don't need any further confirmation between BE, FE and QA
Example:
// File: without_api_generator.ts
test('Test case 1', async ({ request }) => {
const newIssue = await request.post(`/repos/github-username/test-repo-1/issues`, {
headers: {
'Accept': 'application/vnd.github.v3+json',
// Add GitHub personal access token.
'Authorization': `token ${process.env.API_TOKEN}`,
},
data: {
title: '[Bug] report 1',
body: 'Bug description',
},
})
expect(newIssue.ok()).toBeTruthy()
const issues = await request.get(`/repos/github-username/test-repo-1/issues`, {
headers: {
'Accept': 'application/vnd.github.v3+json',
// Add GitHub personal access token.
'Authorization': `token ${process.env.API_TOKEN}`,
},
})
expect(issues.ok()).toBeTruthy()
expect(await issues.json()).toContainEqual(expect.objectContaining({
title: '[Bug] report 1',
body: 'Bug description',
}))
})
// File: with_api_generator.ts
test('Test case 1', async ({ githubClient }) => {
const newIssue = await githubClient.createIssue({
data: {
title: '[Bug] report 1',
body: 'Bug description',
},
})
expect(newIssue.ok()).toBeTruthy()
const issues = await githubClient.getIssues()
expect(issues.ok()).toBeTruthy()
expect(await issues.json()).toContainEqual(expect.objectContaining({
title: '[Bug] report 1',
body: 'Bug description',
}))
})
1.3.2. How to generate API
- BE will generate
swagger.json
based-on services' implementation - FE will generate API services from
swagger.json
via using swagger-typescript-api:- Overwrite
http-client.eta
to useAPIContextRequest
of Playwright instead offetch
- Overwrite
// File: http-client.eta
<%
const { apiConfig, generateResponses, config } = it;
%>
import { APIRequestContext } from 'playwright-core';
export type QueryParamsType = Record<string | number, any>;
export type ResponseFormat = keyof Omit<Body, "body" | "bodyUsed">;
export interface FullRequestParams extends Omit<RequestInit, "body"> {
/** set parameter to `true` for call `securityWorker` for this request */
secure?: boolean;
/** request path */
path: string;
/** content type of request body */
type?: string
/** method of request */
method?: string
/** query params */
query?: QueryParamsType;
/** format of response (i.e. response.json() -> format: "json") */
format?: ResponseFormat;
/** request body */
body?: unknown;
/** base url */
baseUrl?: string;
/** request cancellation token */
cancelToken?: CancelToken;
}
export type RequestParams = Omit<FullRequestParams, "body" | "query" | "path">
export type DataResponse<T, E> = {
//[x: string]: any
response: T
error: E
status: number
statusText: string
}
export interface ApiConfig<SecurityDataType = unknown> {
baseUrl?: string;
baseApiParams?: Omit<RequestParams, "baseUrl" | "cancelToken" | "signal">;
securityWorker?: (securityData: SecurityDataType | null) => Promise<RequestParams | void> | RequestParams | void;
customFetch?: typeof fetch;
}
export interface HttpResponse<D extends unknown, E extends unknown = unknown> extends Response {
data: D;
error: E;
}
type CancelToken = Symbol | string | number;
export enum ContentType {
Json = "application/json",
FormData = "multipart/form-data",
UrlEncoded = "application/x-www-form-urlencoded",
Text = "text/plain",
}
export class HttpClient<SecurityDataType = unknown> {
public baseUrl: string = "<%~ apiConfig.baseUrl %>";
private apiContext: APIRequestContext;
private securityData: SecurityDataType | null = null;
private securityWorker?: ApiConfig<SecurityDataType>["securityWorker"];
private abortControllers = new Map<CancelToken, AbortController>();
private getXRequestId(method: string, url: string): string {
const randomString = Math.random()
return `${method.toLowerCase()}-${url}-${randomString}`.replace(/[^a-zA-Z0-9]/g, '-')
}
private customFetch = (url, fetchParams: Parameters<typeof fetch>[1]) => {
const { 'Content-Type': contentType, ...headers } = fetchParams.headers as Record<string, string>
const fetchOptions = {
headers: {
...headers,
...(contentType !== ContentType.FormData
? {
'Content-Type': contentType,
}
: {}),
},
method: fetchParams.method,
}
if (contentType === ContentType.FormData) {
fetchOptions['multipart'] = fetchParams.body
} else {
fetchOptions['data'] = fetchParams.body
}
return this.apiContext.fetch(url, fetchOptions)
}
private baseApiParams: RequestParams = {
credentials: 'same-origin',
headers: {},
redirect: 'follow',
referrerPolicy: 'no-referrer',
}
constructor(apiConfig: ApiConfig<SecurityDataType> = {}, apiContext: APIRequestContext) {
Object.assign(this, apiConfig);
this.apiContext = apiContext;
}
public setSecurityData = (data: SecurityDataType | null) => {
this.securityData = data;
}
protected encodeQueryParam(key: string, value: any) {
const encodedKey = encodeURIComponent(key);
return `${encodedKey}=${encodeURIComponent(typeof value === "number" ? value : `${value}`)}`;
}
protected addQueryParam(query: QueryParamsType, key: string) {
return this.encodeQueryParam(key, query[key]);
}
protected addArrayQueryParam(query: QueryParamsType, key: string) {
const value = query[key];
return value.map((v: any) => this.encodeQueryParam(key, v)).join("&");
}
protected toQueryString(rawQuery?: QueryParamsType): string {
const query = rawQuery || {};
const keys = Object.keys(query).filter((key) => "undefined" !== typeof query[key]);
return keys
.map((key) =>
Array.isArray(query[key])
? this.addArrayQueryParam(query, key)
: this.addQueryParam(query, key),
)
.join("&");
}
protected addQueryParams(rawQuery?: QueryParamsType): string {
const queryString = this.toQueryString(rawQuery);
return queryString ? `?${queryString}` : "";
}
private contentFormatters: Record<ContentType, (input: any) => any> = {
[ContentType.Json]: (input:any) => input !== null && (typeof input === "object" || typeof input === "string") ? JSON.stringify(input) : input,
[ContentType.Text]: (input:any) => input !== null && typeof input !== "string" ? JSON.stringify(input) : input,
[ContentType.FormData]: (input: any) =>
Object.keys(input || {}).reduce((formData, key) => {
const property = input[key];
formData[key] =
property instanceof Blob ?
property :
typeof property === "object" && property !== null ?
JSON.stringify(property) :
`${property}`
return formData;
}, {}),
[ContentType.UrlEncoded]: (input: any) => this.toQueryString(input),
}
protected mergeRequestParams(params1: RequestParams, params2?: RequestParams): RequestParams {
return {
...this.baseApiParams,
...params1,
...(params2 || {}),
headers: {
...(this.baseApiParams.headers || {}),
...(params1.headers || {}),
...((params2 && params2.headers) || {}),
},
};
}
protected createAbortSignal = (cancelToken: CancelToken): AbortSignal | undefined => {
if (this.abortControllers.has(cancelToken)) {
const abortController = this.abortControllers.get(cancelToken);
if (abortController) {
return abortController.signal;
}
return void 0;
}
const abortController = new AbortController();
this.abortControllers.set(cancelToken, abortController);
return abortController.signal;
}
public abortRequest = (cancelToken: CancelToken) => {
const abortController = this.abortControllers.get(cancelToken)
if (abortController) {
abortController.abort();
this.abortControllers.delete(cancelToken);
}
}
public request = async <T = any, E = any>({
body,
secure,
path,
type,
query,
format,
baseUrl,
cancelToken,
...params
}: FullRequestParams): Promise<DataResponse<T, E>> => {
const secureParams = ((typeof secure === 'boolean' ? secure : this.baseApiParams.secure) && this.securityWorker && await this.securityWorker(this.securityData)) || {};
const requestParams = this.mergeRequestParams(params, secureParams);
const queryString = query && this.toQueryString(query);
const payloadFormatter = this.contentFormatters[type || ContentType.Json];
const responseFormat = format || requestParams.format;
const customFetchParams: Parameters<typeof fetch>[1] = {
...requestParams,
headers: {
...(requestParams.headers || {}),
...(type ? { 'Content-Type': type } : {}),
'X-REQUEST_ID': this.getXRequestId(params.method, path),
},
signal: cancelToken ? this.createAbortSignal(cancelToken) : requestParams.signal,
}
if (typeof body !== 'undefined' && body !== null) {
customFetchParams['body'] = payloadFormatter(body)
}
return this.customFetch(`${baseUrl || this.baseUrl || ""}${path}${queryString ? `?${queryString}` : ""}`, customFetchParams).then(async (response) => {
const returned: DataResponse<T, E> = {
response: null,
error: null,
status: response.status(),
statusText: response.statusText(),
}
if (responseFormat) {
await response[responseFormat]()
.then((data) => {
if (response.ok()) {
returned.response = data
} else {
returned.error = data
}
})
.catch((e) => {
returned.error = e
})
} else {
const r = response as unknown as HttpResponse<T, E>
returned.response = r.data
returned.error = r.error
}
if (cancelToken) {
this.abortControllers.delete(cancelToken)
}
return returned
});
};
}
- Use
swagger-typescript-api
to generate API services files
// File: gen-api.ts
import { render } from 'eta'
import * as fs from 'fs'
import * as path from 'path'
import { generateApi, GenerateApiParams } from 'swagger-typescript-api'
import { toCamelCase } from '/path/to/string/utilities'
type RequestParameter = {
name: string
optional: boolean
type: string
description: string
}
type RequestPayload = {
name: string
optional: boolean
type: string
}
type ParsedRequest = Request & {
path: string
parameters: RequestParameter[]
payload?: RequestPayload
}
async function genOpenApi(swaggerPath: string, outputPath: string, typeAsAny = false) {
const options: GenerateApiParams = {
name: 'open-api.ts',
output: outputPath,
input: swaggerPath,
templates: templatePath,
httpClientType: 'fetch',
generateUnionEnums: false,
enumNamesAsValues: true,
singleHttpClient: true,
cleanOutput: true,
modular: true,
moduleNameFirstTag: true,
generateResponses: true,
generateClient: true,
unwrapResponseData: true,
extractEnums: true,
extractResponseError: true,
extractRequestBody: true,
hooks: {
onCreateRoute: (parsedRoute) => {
const request = parsedRoute.request as ParsedRequest
if (request.parameters.length > 0) {
// Replace "{*abc}" to "${abc}"
request.path = request.path.replace(/\{\*(\w+)\}/, '${$1}')
}
if (typeAsAny && request.payload) {
request.payload.optional = true
request.payload.type += ' | any'
return {
...parsedRoute,
request,
}
}
},
},
}
// Support type as any to check type validation rules of BE
if (typeAsAny) {
options.codeGenConstructs = (struct) => ({
...struct,
TypeField: ({ readonly, key, value }) => {
return `${readonly ? 'readonly' : ''} ${key}?: ${value} | any`
},
})
}
// Generate open-api files
// ------------------------------
try {
const { files } = await generateApi(options)
const fileList = files
.filter((file) => !['data-contracts.ts'].includes(file.name))
.map((file) => {
const fileNameWithoutExtension = file.name.replace('.ts', '')
return {
fileName: file.name,
className: fileNameWithoutExtension,
instanceName: toCamelCase(fileNameWithoutExtension),
}
})
.sort((a, b) => a.className.toLowerCase().localeCompare(b.className.toLowerCase()))
const servicesConnectorPath = path.join(outputPath, 'services-connector.ts')
fs.writeFileSync('/path/to/service-connector', renderTemplate('/path/to/service-connector-template', { fileList }))
} catch (e) {
console.error(e)
}
}
// Generate open-api
;(async () => {
await genOpenApi('/path/to/swagger.json', '/path/to/open-api/')
await genOpenApi('/path/to/swagger.json', '/path/to/open-api-as-any/', true)
})()
1.4. Folder structure
- Folder structure follows monorepo architecture (Turborepo)
1.4.1. Project structure
.circleci
contains CircleCI config filesapps
contains testing appspackages
contains common packages which are used in testing appcommon-tools
contains code for common tools which will be used in testing apps. Example:gen-open-api
,unify-test-case
, etc...eslint-config-custom
contains base configuration for EsLintplaywright-common
contains common components, pages, types, and utils which are used in test cases, fixtures, etc...summary-report
is a customer report to count total passed, failed, flaky, skipped, timed out, or interrupted. Is used for Slack notificationtsconfig
contains base configuration for Typescript
.editorconfig
defines coding styles for multiple developers working on the same project across various editors and IDEs.eslintrc.js
is EsLint root config.gitignore
contains GIT ignored file list.npmrc
stores the feed URLs and credentials that Npm uses.prettierignore
and.prettierrc
contains configuration for prettier (Code Formatter)package.json
contains package information such as name, version, dependencies, scripts, etc...README.md
turbo.json
contains configuration for Turborepoyarn.lock
stores exactly which versions of each dependency were installed. More details at here
1.4.2. A testing app structure
__reports
contains testing reports including Playwright, Allure, etc...__results
contains trace files which are used for Trace Viewerdata
contains data for testinglibs
contains helpers to support testingapi
contains helpers for API testingconstants
contains constantsfixtures
contains test base for E2E and API testingui
contains helpers for E2E testingcomponents
contains components' fixtures such asglobalHeader
,sidebarMenu
, etc...pages
contains pages' fixtures such asloginPage
,switchOfficePage
,createOfficePage
, etc...
sql-scripts
contains SQL scripts which are used for setup and teardown of testingutils
contains utilities
tests
contains all test casestools
contains utility tools which not belong to helpers. Example:gen-open-api
,unify-test-case
.env.example
contains an example for.env
file.eslintrc.js
contains EsLint configurationglobal.setup.ts
contains global setup for all test casesglobal.teardown.ts
contains global teardown setup for all test casespackage.json
contains package information such as name, version, dependencies, scripts, etc...playwright.config.ts
contains Playwright configurationtsconfig.json
contains Typescript configuration
2. Playwright Fixtures
2.1. What are Test fixtures?
- Used to establish an environment for each test, giving the test everything it needs and nothing else.
- Isolated between tests.
- With fixtures, you can group tests based on their meaning, instead of their common setup.
2.2. Why fixtures?
- Encapsulate setup and teardown in the same place so it is easier to write.
- Reusable between test files - you can define them once and use them in all your tests. That's how Playwright's built-in
page
fixture works. - On-demand - you can define as many fixtures as you'd like, and Playwright Test will set up only the ones needed by your test and nothing else.
- Composable - they can depend on each other to provide complex behaviors.
- Flexible - tests can use any combinations of the fixtures to tailor the precise environment they need, without affecting other tests.
- Simplify grouping - you no longer need to wrap tests in
describe
s that set up environment, and are free to group your tests by their meaning instead.
2.2.1. Built-in fixtures
Fixture | Type | Description |
---|---|---|
page | Page | Isolated page for this test run. |
context | BrowserContext | Isolated context for this test run. The page fixture belongs to this context as well. Learn how to configure context. |
browser | Browser | Browsers are shared across tests to optimize resources. Learn how to configure browser. |
browserName | string | The name of the browser currently running the test. Either chromium , firefox or webkit . |
request | APIRequestContext | Isolated APIRequestContext instance for this test run. |
2.3. Custom fixtures
To create your own fixture, use test.extend() to create a new test
object that will include it.
// File: my-fixtures.ts
import { test as base } from '@playwright/test'
import { TodoPage } from './todo-page'
import { SettingsPage } from './settings-page'
// Declare the types of your fixtures.
type MyFixtures = {
todoPage: TodoPage;
settingsPage: SettingsPage;
};
// Extend base test by providing "todoPage" and "settingsPage".
// This new "test" can be used in multiple test files, and each of them will get the fixtures.
export const test = base.extend<MyFixtures>({
todoPage: async ({ page }, use) => {
// Set up the fixture.
const todoPage = new TodoPage(page)
await todoPage.goto()
await todoPage.addToDo('item1')
await todoPage.addToDo('item2')
// Use the fixture value in the test.
await use(todoPage)
// Clean up the fixture.
await todoPage.removeAll()
},
settingsPage: async ({ page }, use) => {
await use(new SettingsPage(page))
},
})
export { expect } from '@playwright/test'
Examples:
// File: without_fixtures.ts
const { test } = require('@playwright/test')
const { TodoPage } = require('./todo-page')
test.describe('todo tests', () => {
let todoPage
test.beforeEach(async ({ page }) => {
todoPage = new TodoPage(page)
await todoPage.goto()
await todoPage.addToDo('item1')
await todoPage.addToDo('item2')
})
test.afterEach(async () => {
await todoPage.removeAll()
})
test('should add an item', async () => {
await todoPage.addToDo('my item')
// ...
})
test('should remove an item', async () => {
await todoPage.remove('item1')
// ...
})
})
// File: with_fixtures.ts
const { test } = require('@playwright/test')
const { TodoPage } = require('./todo-page')
test.describe('todo tests', () => {
test.afterEach(async ({ todoPage }) => {
await todoPage.removeAll()
})
test('should add an item', async ({ todoPage }) => {
await todoPage.addToDo('my item')
// ...
})
test('should remove an item', async ({ todoPage }) => {
await todoPage.remove('item1')
// ...
})
})
3. How to write test case
3.1. Aliases
- Purpose: to make test script look similar with other test frameworks like Jest, Mocha, Jasmine, etc...
- Alias list:
describe
is an alias of thetest.describe
it
is an alias oftest
beforeAll
is an alias oftest.beforeAll
beforeEach
is an alias oftest.beforeEach
afterEach
is an alias oftest.afterEach
afterAll
is an alias oftest.afterAll
- Example
// File: with-aliases.ts
describe('Test suite', () => {
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
})
// File: without-aliases.ts
test.describe('Test suite', () => {
test.beforeAll(async ({}) => {
// beforeAll hook
})
test.beforeEach(async ({}) => {
// beforeEach hook
})
test('Test case #1', async ({}) => {
// Test case script
})
test('Test case #2', async ({}) => {
// Test case script
})
test.afterEach(async ({}) => {
// afterEach hook
})
test.afterAll(async ({}) => {
// afterAll hook
})
})
3.2. Common test cases
3.2.1. Conventions
- File name must be in lowercase. Can be a test suite or test case code. Example
c12345.ts
for test case ands56789.ts
for test suite - Each test file must contain a TestRail link to the test case or test suite. Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
...
it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => {
...
})
- Tip: Script
yarn unify-testcase
can help you:- Change your file name to be in lower-case
- Prepend TestRail link to test case based-on your file name
3.2.2. Test script structure
Structure of test case contains following factors:
describe
ortest.describe
to declare test suiteit
ortest
to declare testcasebeforeAll
ortest.beforeAll
to declare setup step for test suitebeforeEach
ortest.beforeEach
to declare setup step for each test caseafterEach
ortest.afterEach
to declare teardown step for each test caseafterAll
ortest.afterAll
to declare teardown step for test suite
Example:
// File: with-test-suite.ts
describe('Test suite', () => {
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
})
// File: without-test-suite.ts
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
3.3. E2E test cases
describe
,it
,beforeAll
,beforeEach
,afterEach
,afterAll
andexpect
are imported from@/libs/fixtures/e2e
- Available fixtures:
pagination
inlineNotification
toast
globalHeader
sidebarMenu
loginPage
switchOfficePage
newOfficePage
employeeListPage
employeeInvitePage
employeeDetailPage
emailSettingPage
emailTemplatePreviewPage
emailTemplateEditPage
sendPackageListPage
companySettingPage
senderInformationPage
- Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
import { accountData } from '@/data/account'
import { globalHeaderData } from '@/data/global-header'
import { ACCOUNT_0_STATE } from '@/libs/constants/store'
import { beforeEach, expect, it, use } from '@/libs/fixtures/e2e'
use({ storageState: ACCOUNT_0_STATE })
beforeEach(async ({ sendPackageListPage }) => {
await sendPackageListPage.goto()
})
it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => {
// User dropdown
await globalHeader.openUserDropdown()
await expect(globalHeader.userDropdown).toContainText(accountData.user[0].name)
await expect(globalHeader.userSettingButton).toContainText(globalHeaderData.userDropdown.userSettingText)
await expect(globalHeader.userSettingButton).toHaveAttribute(
'href',
`${process.env.MFID_BASE_URL}${globalHeaderData.userDropdown.userSettingLink}`
)
await expect(globalHeader.logoutButton).toBeVisible()
await expect(globalHeader.logoutButton).toContainText(globalHeaderData.userDropdown.logoutButtonText)
// Office dropdown
await globalHeader.openOfficeDropdown()
await expect(globalHeader.switchOfficeButton).toBeVisible()
await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].tenantName)
await expect(globalHeader.officeDropdown).toContainText(globalHeaderData.officeDropdown.officeBusinessNumberText)
await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].businessNumber)
await expect(globalHeader.switchOfficeButton).toContainText(globalHeaderData.officeDropdown.switchOfficeButtonText)
})
3.4. API test cases
describe
,it
,beforeAll
,beforeEach
,afterEach
,afterAll
andexpect
are imported from@/libs/fixtures/api
- Available fixtures:
apiClient
navisClient
invoiceDeliveryClient
suzakuLoginPage
- Example:
// File: c57134.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57134
import { accountData } from '@/data/receiver-account'
import { describe, expect, it } from '@/libs/fixtures/api'
/**
* Endpoints: PUT /v1/user/terms
*/
describe('SCI-3365-Receiver can register MFID', () => {
it('C57134-Verify agree the term process before user agrees the term', async ({ apiClient }) => {
const loginResult = await apiClient.login(accountData.user[1].email, accountData.user[1].password)
expect(loginResult.status).toBe(200)
expect(loginResult.response).toHaveProperty('data')
expect(loginResult.response.data.user).not.toBe(null)
expect(loginResult.response.data.user.accept_terms).toBe(false)
expect(loginResult.response.data.user.has_available_tenant).toBe(true)
})
})
4. Reports
4.1. Report overview
Slack channel: #sci_automation_notification
Report type:
- Daily: Every morning from Monday to Friday (GTM+7)
- Monthly: TBD
Trigger type:
nightly
ormanual
App name:
sender
orreceiver
Testing type:
api
ore2e
Report format
Title:
[{TRIGGER_TYPE}] [{APP_NAME}] {TESTING_TYPE} testing report for {RUN_DATE}
Content:
- Total passed
- Total failed
- Total flaky
- Total skipped (due to depending on test case is failed)
- Total timed out (due to auto-waiting exceeding configured timeout)
- Total interrupted
Example:
Link to report detail
Playwright report
Allure report
4.2. Custom reporter for Playwright
- You can create a custom reporter by implementing a class with some of the reporter methods. Learn more about the Reporter API.
// File: my-awesome-reporter.ts
import type {
FullConfig, FullResult, Reporter, Suite, TestCase, TestResult,
} from '@playwright/test/reporter'
class MyReporter implements Reporter {
onBegin(config: FullConfig, suite: Suite) {
console.log(`Starting the run with ${suite.allTests().length} tests`)
}
onTestBegin(test: TestCase, result: TestResult) {
console.log(`Starting test ${test.title}`)
}
onTestEnd(test: TestCase, result: TestResult) {
console.log(`Finished test ${test.title}: ${result.status}`)
}
onEnd(result: FullResult) {
console.log(`Finished the run: ${result.status}`)
}
}
export default MyReporter
- Now use this reporter with testConfig.reporter.
// File: playwright.config.ts
import { defineConfig } from '@playwright/test'
export default defineConfig({
reporter: './my-awesome-reporter.ts',
})
4.3. Summary Reporter
This report will calculate total passed, failed, flaky, skipped, timed-out and interrupted test cases as well as passed ratio and total duration. The result will stored in JSON format
// File: summary-reporter.ts
import { FullConfig, Reporter, Suite } from '@playwright/test/reporter'
import * as fs from 'fs'
import { getCurrentDate, convertMsToTime } from './path/to/helpers'
export type AppName = 'sender' | 'receiver'
export type TestType = 'e2e' | 'api'
export type SummaryReporterOption = {
outputFile: string
appName: AppName
testType: TestType
}
class SummaryReporter implements Reporter {
private suiteList: Suite[] = []
private readonly outputFile: string
private readonly appName: AppName
private readonly testType: TestType
constructor({ outputFile, appName, testType }: SummaryReporterOption) {
this.outputFile = outputFile
this.appName = appName
this.testType = testType
}
onBegin(config: FullConfig, suite: Suite) {
this.suiteList.push(suite)
}
onEnd() {
let total = 0
let totalPassed = 0
let totalFailed = 0
let totalFlaky = 0
let totalSkipped = 0
let totalTimedOut = 0
let totalInterrupted = 0
let totalDuration = 0
this.suiteList.forEach((suite) => {
suite.allTests().forEach((test) => {
let isPassed = false
let isFailed = false
let isFlaky = false
test.results.forEach((result) => {
result.status === 'skipped' && totalSkipped++
result.status === 'timedOut' && totalTimedOut++
result.status === 'interrupted' && totalInterrupted++
result.duration
if (result.retry === 0) {
isPassed = result.status === 'passed'
isFailed = result.status === 'failed'
} else {
if ((result.status === 'passed' && isFailed) || (result.status === 'passed' && isPassed)) {
isFlaky = true
isPassed = false
isFailed = false
}
}
totalDuration += result.duration
})
isPassed && totalPassed++
isFailed && totalFailed++
isFlaky && totalFlaky++
total++
})
const passedRatio = (totalPassed / total) * 100
fs.writeFileSync(
this.outputFile,
JSON.stringify({
title: `[${this.testType.toUpperCase()}] [${this.appName.toUpperCase()}] Report for ${getCurrentDate()}`,
total,
totalPassed,
totalFailed,
totalFlaky,
totalSkipped,
totalTimedOut,
totalInterrupted,
passedRatio: isNaN(passedRatio) ? 'N/A' : `${passedRatio.toFixed(2)}%`,
totalDuration: totalDuration === 0 ? 'N/A' : convertMsToTime(totalDuration),
})
)
})
}
}
export default SummaryReporter
4.4. TestRail Reporter
This report will:
- Detect TestRail test cases' id via test-case title
- Create
TestMilestone
in the first day of a week (if does not exist) - Close
TestMilestone
in the last day of a week (if does not close) - Create
TestRun
whenever running an Automation Test - Close
TestRun
after running the Automation Test - Create
TestResult
for all detected test cases
// File: testrail-reporter.ts
import TestRail from '@dlenroc/testrail'
import { Reporter, TestCase, TestResult } from '@playwright/test/reporter'
import dayjs from 'dayjs'
import isoWeek from 'dayjs/plugin/isoWeek'
import * as fs from 'fs'
import { getCurrentDate, getCurrentWeek, getPrefixTitle, stripAnsi } from './path/to/helpers'
dayjs.extend(isoWeek)
export type AppName = 'sender' | 'receiver'
export type TestType = 'e2e' | 'api'
export type TestRailReporterOption = {
outputFile: string
host: string
username: string
password: string
projectId: number
suiteId: number
appName: AppName
testType: TestType
}
class TestRailReporter implements Reporter {
private readonly testRailClient: TestRail
private initTestMilestoneProcess: Promise<void>
private testMilestone: TestRail.Milestone
private testRun: TestRail.Run
private readonly outputFile: string
private readonly host: string
private readonly projectId: number
private readonly suiteId: number
private readonly appName: AppName
private readonly testType: TestType
private testResultList: Array<TestRail.Result> = []
private readonly STATUS_MAPPING = {
passed: 1,
failed: 5,
timedOut: 5,
}
constructor({ username, password, ...options }: TestRailReporterOption) {
Object.assign(this, options)
this.testRailClient = new TestRail({
host: options.host,
username,
password,
})
}
// Event handlers
// --------------------------------
async onBegin() {
this.initTestMilestoneProcess = this.initTestMilestone()
}
async onTestEnd(test: TestCase, result: TestResult) {
await this.initTestResult(test, result)
}
async onEnd() {
await this.initTestMilestoneProcess
if (this.testResultList.length > 0) {
await this.initTestRun()
await this.updateTestResult()
}
this.genResult()
await this.closeTestRun()
await this.closeTestMilestone()
}
// Gen result
// --------------------------------
genResult() {
fs.writeFileSync(
this.outputFile,
JSON.stringify({
link: `${this.host}/index.php?/runs/view/${this?.testRun?.id || ''}`,
})
)
}
// TestRail businesses
// --------------------------------
async initTestMilestone() {
const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Milestone for ${getCurrentWeek()}`
const milestoneList = await this.testRailClient.getMilestones(this.projectId, {
is_completed: false,
})
let testMilestone = milestoneList.filter((item) => item.name === name)[0]
if (!testMilestone) {
testMilestone = await this.testRailClient.addMilestone(this.projectId, {
name,
})
}
if (!testMilestone.is_started) {
testMilestone = await this.testRailClient.updateMilestone(testMilestone.id, {
is_started: true,
})
}
this.testMilestone = testMilestone
}
async closeTestMilestone() {
if (dayjs().day() === 5) {
await this.testRailClient.updateMilestone(this.testMilestone.id, {
is_completed: true,
})
}
}
async initTestRun() {
const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Run for ${getCurrentDate()}`
const testRunList = await this.testRailClient.getRuns(this.projectId, {
suite_id: this.suiteId,
milestone_id: this.testMilestone.id,
})
let testRun = testRunList.filter((item) => item.name === name)[0]
if (!testRun) {
testRun = await this.testRailClient.addRun(this.projectId, {
suite_id: this.suiteId,
milestone_id: this.testMilestone.id,
name,
})
}
if (!testRun.include_all) {
testRun = await this.testRailClient.updateRun(this.projectId, {
include_all: true,
})
}
this.testRun = testRun
}
async closeTestRun() {
await this.testRailClient.closeRun(this.testRun.id)
}
async initTestResult(test: TestCase, result: TestResult) {
const testCaseIdList = test.title.match(/(C[0-9]+)/gi)?.map((item) => +item.replace(/c/i, ''))
if (!Array.isArray(testCaseIdList) || testCaseIdList.length === 0) {
return
}
if (result.status === 'skipped') {
return
}
let runNumber = ''
if (result.retry === 0) {
runNumber = `First run`
} else {
runNumber = `Retry #${result.retry}`
}
let errorMessage = ''
if (result.status === 'failed') {
let errorDetails = ''
errorDetails += stripAnsi(result.error.message)
errorDetails += '\n'
if (result.error.location) {
errorDetails += ` at ${result.error.location.file}:${result.error.location.line}:${result.error.location.column}`
}
const errorArr = errorDetails.split('\n').map((item) => ` ${item}`)
errorMessage += `\nError:\n\n${errorArr.join('\n')}`
}
this.testResultList = [
...this.testResultList,
...testCaseIdList.map(
(item) =>
({
case_id: item,
status_id: this.STATUS_MAPPING[result.status],
comment: `${runNumber}\n\nDuration: ${result.duration}ms\n\n${errorMessage}`,
} as unknown as TestRail.Result)
),
]
}
async updateTestResult() {
await this.testRailClient.addResultsForCases(this.testRun.id, {
results: this.testResultList,
})
}
}
export default TestRailReporter
5. Nightly and manual test on CircleCI
Nightly and manual tests will run only on main
branch
5.1. Nightly test
Will test
api
ande2e
for all applicationsWill be triggered at:
- With
UTC
timezone: 8:00 PM on Sunday, Monday-Thursday - With
GTM+7
timezone: 3:00 AM on Monday-Friday
- With
Scheduled trigger on CircleCI:
Purpose: ensures that you catch such problems within 24 hours when they occur
5.2. Manual test
Will be triggered manually when:
New code is added to
main
branch (can be merged PR or commit code directly to these branches)Trigger via CircleCI
- Step 1: Select
main
branch - Step 2: Click
Trigger Pipeline
button
- Step 1: Select
Purpose: trigger
api
ande2e
testing for a specified application on demand
6. Feature plans
- Implement
Monthly
report about detected bugs - Auto create JIRA ticket when finding bug after running Automation Test
- Generate test cases for testing type of parameters. Example:
username: string
param will be tested with following input:null
''
!@#$%^&*()
true
123456789