![HTTP Adventure: [Part 2] The evolution of HTTP/2](https://static-careers.moneyforward.vn/Blog/HTTP%20series/%5BPart%202%5D%20The%20evolution%20of%20HTTP2.png)

Automation test in SCI project

1. Tech stack overview
1.1. Language stack: Javascript/Typescript
- Using Typescript for strong type and reducing wrong attributes
- Node package management tool: Yarn
- Code lint: EsLint (simple-import-sort + prettier + ...)
1.2. Testing framework: Playwright
1.2.1. What Playwright?
- Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox.
- Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed.
- Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java.
- Test Mobile Web. Native mobile emulation of Google Chrome for Android and Mobile Safari. The same rendering engine works on your Desktop and in the Cloud.
1.2.2. Why Playwright?
- Supports multiple-languages including Javascript/Typescript, Java, .NET, Python
- It supports multi-tab, multi-user, and iframe test cases.
- It is available as a VS Code extension with one-click-to-run tests and includes step-by-step debugging, selector exploration, and logging of new tests.
- It supports different types of testing, such as end-to-end, functional, and API testing.
- Supports automated accessibility testing with third-party plugins.
- It provides various debugging options like Playwright Inspector, Browser Developer Tools, VSCode Debugger, and Monitor Viewer Console Logs.
- It has built-in reporters like JSON, JUnit, and HTML Reporter. You can also create custom reports using Playwright
- Faster and more reliable than Cypress. There are a lot of online articles or topics that mention the unreliable of Cypress without a resolved solution:
- Cypress hanging on circleCI but pass locally
- Cypress Testing: stuck at wait-on http://localhost:3000
- Cypress Testing: stuck at wait-on http://localhost:3000 (StackOverflow)
- Cypress sometimes stalls/hangs with no output when running in Jenkins with Docker
- CYPRESS tests pass locally but are hanging on circleCI
- Cypress hung up trying to run a spec - won't resolve
- Cypress unexpectedly and randomly hangs when running tests in CI
1.2.3. Playwright limitations
- Does not support Microsoft Edge or earlier IE11.
- Uses a desktop browser instead of a real device to emulate a mobile device
1.3. API Generator
1.3.1. Why API Generator?
- Consume API is very clear and simple via classes and methods
- Don't need any further confirmation between BE, FE and QA
Example:
// File: without_api_generator.ts
test('Test case 1', async ({ request }) => {
const newIssue = await request.post(`/repos/github-username/test-repo-1/issues`, {
headers: {
'Accept': 'application/vnd.github.v3+json',
// Add GitHub personal access token.
'Authorization': `token ${process.env.API_TOKEN}`,
},
data: {
title: '[Bug] report 1',
body: 'Bug description',
},
})
expect(newIssue.ok()).toBeTruthy()
const issues = await request.get(`/repos/github-username/test-repo-1/issues`, {
headers: {
'Accept': 'application/vnd.github.v3+json',
// Add GitHub personal access token.
'Authorization': `token ${process.env.API_TOKEN}`,
},
})
expect(issues.ok()).toBeTruthy()
expect(await issues.json()).toContainEqual(expect.objectContaining({
title: '[Bug] report 1',
body: 'Bug description',
}))
})
// File: with_api_generator.ts
test('Test case 1', async ({ githubClient }) => {
const newIssue = await githubClient.createIssue({
data: {
title: '[Bug] report 1',
body: 'Bug description',
},
})
expect(newIssue.ok()).toBeTruthy()
const issues = await githubClient.getIssues()
expect(issues.ok()).toBeTruthy()
expect(await issues.json()).toContainEqual(expect.objectContaining({
title: '[Bug] report 1',
body: 'Bug description',
}))
})
1.3.2. How to generate API
- BE will generate
swagger.jsonbased-on services' implementation - FE will generate API services from
swagger.jsonvia using swagger-typescript-api:- Overwrite
http-client.etato useAPIContextRequestof Playwright instead offetch
- Overwrite
// File: http-client.eta
<%
const { apiConfig, generateResponses, config } = it;
%>
import { APIRequestContext } from 'playwright-core';
export type QueryParamsType = Record<string | number, any>;
export type ResponseFormat = keyof Omit<Body, "body" | "bodyUsed">;
export interface FullRequestParams extends Omit<RequestInit, "body"> {
/** set parameter to `true` for call `securityWorker` for this request */
secure?: boolean;
/** request path */
path: string;
/** content type of request body */
type?: string
/** method of request */
method?: string
/** query params */
query?: QueryParamsType;
/** format of response (i.e. response.json() -> format: "json") */
format?: ResponseFormat;
/** request body */
body?: unknown;
/** base url */
baseUrl?: string;
/** request cancellation token */
cancelToken?: CancelToken;
}
export type RequestParams = Omit<FullRequestParams, "body" | "query" | "path">
export type DataResponse<T, E> = {
//[x: string]: any
response: T
error: E
status: number
statusText: string
}
export interface ApiConfig<SecurityDataType = unknown> {
baseUrl?: string;
baseApiParams?: Omit<RequestParams, "baseUrl" | "cancelToken" | "signal">;
securityWorker?: (securityData: SecurityDataType | null) => Promise<RequestParams | void> | RequestParams | void;
customFetch?: typeof fetch;
}
export interface HttpResponse<D extends unknown, E extends unknown = unknown> extends Response {
data: D;
error: E;
}
type CancelToken = Symbol | string | number;
export enum ContentType {
Json = "application/json",
FormData = "multipart/form-data",
UrlEncoded = "application/x-www-form-urlencoded",
Text = "text/plain",
}
export class HttpClient<SecurityDataType = unknown> {
public baseUrl: string = "<%~ apiConfig.baseUrl %>";
private apiContext: APIRequestContext;
private securityData: SecurityDataType | null = null;
private securityWorker?: ApiConfig<SecurityDataType>["securityWorker"];
private abortControllers = new Map<CancelToken, AbortController>();
private getXRequestId(method: string, url: string): string {
const randomString = Math.random()
return `${method.toLowerCase()}-${url}-${randomString}`.replace(/[^a-zA-Z0-9]/g, '-')
}
private customFetch = (url, fetchParams: Parameters<typeof fetch>[1]) => {
const { 'Content-Type': contentType, ...headers } = fetchParams.headers as Record<string, string>
const fetchOptions = {
headers: {
...headers,
...(contentType !== ContentType.FormData
? {
'Content-Type': contentType,
}
: {}),
},
method: fetchParams.method,
}
if (contentType === ContentType.FormData) {
fetchOptions['multipart'] = fetchParams.body
} else {
fetchOptions['data'] = fetchParams.body
}
return this.apiContext.fetch(url, fetchOptions)
}
private baseApiParams: RequestParams = {
credentials: 'same-origin',
headers: {},
redirect: 'follow',
referrerPolicy: 'no-referrer',
}
constructor(apiConfig: ApiConfig<SecurityDataType> = {}, apiContext: APIRequestContext) {
Object.assign(this, apiConfig);
this.apiContext = apiContext;
}
public setSecurityData = (data: SecurityDataType | null) => {
this.securityData = data;
}
protected encodeQueryParam(key: string, value: any) {
const encodedKey = encodeURIComponent(key);
return `${encodedKey}=${encodeURIComponent(typeof value === "number" ? value : `${value}`)}`;
}
protected addQueryParam(query: QueryParamsType, key: string) {
return this.encodeQueryParam(key, query[key]);
}
protected addArrayQueryParam(query: QueryParamsType, key: string) {
const value = query[key];
return value.map((v: any) => this.encodeQueryParam(key, v)).join("&");
}
protected toQueryString(rawQuery?: QueryParamsType): string {
const query = rawQuery || {};
const keys = Object.keys(query).filter((key) => "undefined" !== typeof query[key]);
return keys
.map((key) =>
Array.isArray(query[key])
? this.addArrayQueryParam(query, key)
: this.addQueryParam(query, key),
)
.join("&");
}
protected addQueryParams(rawQuery?: QueryParamsType): string {
const queryString = this.toQueryString(rawQuery);
return queryString ? `?${queryString}` : "";
}
private contentFormatters: Record<ContentType, (input: any) => any> = {
[ContentType.Json]: (input:any) => input !== null && (typeof input === "object" || typeof input === "string") ? JSON.stringify(input) : input,
[ContentType.Text]: (input:any) => input !== null && typeof input !== "string" ? JSON.stringify(input) : input,
[ContentType.FormData]: (input: any) =>
Object.keys(input || {}).reduce((formData, key) => {
const property = input[key];
formData[key] =
property instanceof Blob ?
property :
typeof property === "object" && property !== null ?
JSON.stringify(property) :
`${property}`
return formData;
}, {}),
[ContentType.UrlEncoded]: (input: any) => this.toQueryString(input),
}
protected mergeRequestParams(params1: RequestParams, params2?: RequestParams): RequestParams {
return {
...this.baseApiParams,
...params1,
...(params2 || {}),
headers: {
...(this.baseApiParams.headers || {}),
...(params1.headers || {}),
...((params2 && params2.headers) || {}),
},
};
}
protected createAbortSignal = (cancelToken: CancelToken): AbortSignal | undefined => {
if (this.abortControllers.has(cancelToken)) {
const abortController = this.abortControllers.get(cancelToken);
if (abortController) {
return abortController.signal;
}
return void 0;
}
const abortController = new AbortController();
this.abortControllers.set(cancelToken, abortController);
return abortController.signal;
}
public abortRequest = (cancelToken: CancelToken) => {
const abortController = this.abortControllers.get(cancelToken)
if (abortController) {
abortController.abort();
this.abortControllers.delete(cancelToken);
}
}
public request = async <T = any, E = any>({
body,
secure,
path,
type,
query,
format,
baseUrl,
cancelToken,
...params
}: FullRequestParams): Promise<DataResponse<T, E>> => {
const secureParams = ((typeof secure === 'boolean' ? secure : this.baseApiParams.secure) && this.securityWorker && await this.securityWorker(this.securityData)) || {};
const requestParams = this.mergeRequestParams(params, secureParams);
const queryString = query && this.toQueryString(query);
const payloadFormatter = this.contentFormatters[type || ContentType.Json];
const responseFormat = format || requestParams.format;
const customFetchParams: Parameters<typeof fetch>[1] = {
...requestParams,
headers: {
...(requestParams.headers || {}),
...(type ? { 'Content-Type': type } : {}),
'X-REQUEST_ID': this.getXRequestId(params.method, path),
},
signal: cancelToken ? this.createAbortSignal(cancelToken) : requestParams.signal,
}
if (typeof body !== 'undefined' && body !== null) {
customFetchParams['body'] = payloadFormatter(body)
}
return this.customFetch(`${baseUrl || this.baseUrl || ""}${path}${queryString ? `?${queryString}` : ""}`, customFetchParams).then(async (response) => {
const returned: DataResponse<T, E> = {
response: null,
error: null,
status: response.status(),
statusText: response.statusText(),
}
if (responseFormat) {
await response[responseFormat]()
.then((data) => {
if (response.ok()) {
returned.response = data
} else {
returned.error = data
}
})
.catch((e) => {
returned.error = e
})
} else {
const r = response as unknown as HttpResponse<T, E>
returned.response = r.data
returned.error = r.error
}
if (cancelToken) {
this.abortControllers.delete(cancelToken)
}
return returned
});
};
}
- Use
swagger-typescript-apito generate API services files
// File: gen-api.ts
import { render } from 'eta'
import * as fs from 'fs'
import * as path from 'path'
import { generateApi, GenerateApiParams } from 'swagger-typescript-api'
import { toCamelCase } from '/path/to/string/utilities'
type RequestParameter = {
name: string
optional: boolean
type: string
description: string
}
type RequestPayload = {
name: string
optional: boolean
type: string
}
type ParsedRequest = Request & {
path: string
parameters: RequestParameter[]
payload?: RequestPayload
}
async function genOpenApi(swaggerPath: string, outputPath: string, typeAsAny = false) {
const options: GenerateApiParams = {
name: 'open-api.ts',
output: outputPath,
input: swaggerPath,
templates: templatePath,
httpClientType: 'fetch',
generateUnionEnums: false,
enumNamesAsValues: true,
singleHttpClient: true,
cleanOutput: true,
modular: true,
moduleNameFirstTag: true,
generateResponses: true,
generateClient: true,
unwrapResponseData: true,
extractEnums: true,
extractResponseError: true,
extractRequestBody: true,
hooks: {
onCreateRoute: (parsedRoute) => {
const request = parsedRoute.request as ParsedRequest
if (request.parameters.length > 0) {
// Replace "{*abc}" to "${abc}"
request.path = request.path.replace(/\{\*(\w+)\}/, '${$1}')
}
if (typeAsAny && request.payload) {
request.payload.optional = true
request.payload.type += ' | any'
return {
...parsedRoute,
request,
}
}
},
},
}
// Support type as any to check type validation rules of BE
if (typeAsAny) {
options.codeGenConstructs = (struct) => ({
...struct,
TypeField: ({ readonly, key, value }) => {
return `${readonly ? 'readonly' : ''} ${key}?: ${value} | any`
},
})
}
// Generate open-api files
// ------------------------------
try {
const { files } = await generateApi(options)
const fileList = files
.filter((file) => !['data-contracts.ts'].includes(file.name))
.map((file) => {
const fileNameWithoutExtension = file.name.replace('.ts', '')
return {
fileName: file.name,
className: fileNameWithoutExtension,
instanceName: toCamelCase(fileNameWithoutExtension),
}
})
.sort((a, b) => a.className.toLowerCase().localeCompare(b.className.toLowerCase()))
const servicesConnectorPath = path.join(outputPath, 'services-connector.ts')
fs.writeFileSync('/path/to/service-connector', renderTemplate('/path/to/service-connector-template', { fileList }))
} catch (e) {
console.error(e)
}
}
// Generate open-api
;(async () => {
await genOpenApi('/path/to/swagger.json', '/path/to/open-api/')
await genOpenApi('/path/to/swagger.json', '/path/to/open-api-as-any/', true)
})()
1.4. Folder structure
- Folder structure follows monorepo architecture (Turborepo)
1.4.1. Project structure
.circlecicontains CircleCI config filesappscontains testing appspackagescontains common packages which are used in testing appcommon-toolscontains code for common tools which will be used in testing apps. Example:gen-open-api,unify-test-case, etc...eslint-config-customcontains base configuration for EsLintplaywright-commoncontains common components, pages, types, and utils which are used in test cases, fixtures, etc...summary-reportis a customer report to count total passed, failed, flaky, skipped, timed out, or interrupted. Is used for Slack notificationtsconfigcontains base configuration for Typescript
.editorconfigdefines coding styles for multiple developers working on the same project across various editors and IDEs.eslintrc.jsis EsLint root config.gitignorecontains GIT ignored file list.npmrcstores the feed URLs and credentials that Npm uses.prettierignoreand.prettierrccontains configuration for prettier (Code Formatter)package.jsoncontains package information such as name, version, dependencies, scripts, etc...README.mdturbo.jsoncontains configuration for Turborepoyarn.lockstores exactly which versions of each dependency were installed. More details at here
1.4.2. A testing app structure
__reportscontains testing reports including Playwright, Allure, etc...__resultscontains trace files which are used for Trace Viewerdatacontains data for testinglibscontains helpers to support testingapicontains helpers for API testingconstantscontains constantsfixturescontains test base for E2E and API testinguicontains helpers for E2E testingcomponentscontains components' fixtures such asglobalHeader,sidebarMenu, etc...pagescontains pages' fixtures such asloginPage,switchOfficePage,createOfficePage, etc...
sql-scriptscontains SQL scripts which are used for setup and teardown of testingutilscontains utilities
testscontains all test casestoolscontains utility tools which not belong to helpers. Example:gen-open-api,unify-test-case.env.examplecontains an example for.envfile.eslintrc.jscontains EsLint configurationglobal.setup.tscontains global setup for all test casesglobal.teardown.tscontains global teardown setup for all test casespackage.jsoncontains package information such as name, version, dependencies, scripts, etc...playwright.config.tscontains Playwright configurationtsconfig.jsoncontains Typescript configuration
2. Playwright Fixtures
2.1. What are Test fixtures?
- Used to establish an environment for each test, giving the test everything it needs and nothing else.
- Isolated between tests.
- With fixtures, you can group tests based on their meaning, instead of their common setup.
2.2. Why fixtures?
- Encapsulate setup and teardown in the same place so it is easier to write.
- Reusable between test files - you can define them once and use them in all your tests. That's how Playwright's built-in
pagefixture works. - On-demand - you can define as many fixtures as you'd like, and Playwright Test will set up only the ones needed by your test and nothing else.
- Composable - they can depend on each other to provide complex behaviors.
- Flexible - tests can use any combinations of the fixtures to tailor the precise environment they need, without affecting other tests.
- Simplify grouping - you no longer need to wrap tests in
describes that set up environment, and are free to group your tests by their meaning instead.
2.2.1. Built-in fixtures
| Fixture | Type | Description |
|---|---|---|
| page | Page | Isolated page for this test run. |
| context | BrowserContext | Isolated context for this test run. The page fixture belongs to this context as well. Learn how to configure context. |
| browser | Browser | Browsers are shared across tests to optimize resources. Learn how to configure browser. |
| browserName | string | The name of the browser currently running the test. Either chromium, firefox or webkit. |
| request | APIRequestContext | Isolated APIRequestContext instance for this test run. |
2.3. Custom fixtures
To create your own fixture, use test.extend() to create a new test object that will include it.
// File: my-fixtures.ts
import { test as base } from '@playwright/test'
import { TodoPage } from './todo-page'
import { SettingsPage } from './settings-page'
// Declare the types of your fixtures.
type MyFixtures = {
todoPage: TodoPage;
settingsPage: SettingsPage;
};
// Extend base test by providing "todoPage" and "settingsPage".
// This new "test" can be used in multiple test files, and each of them will get the fixtures.
export const test = base.extend<MyFixtures>({
todoPage: async ({ page }, use) => {
// Set up the fixture.
const todoPage = new TodoPage(page)
await todoPage.goto()
await todoPage.addToDo('item1')
await todoPage.addToDo('item2')
// Use the fixture value in the test.
await use(todoPage)
// Clean up the fixture.
await todoPage.removeAll()
},
settingsPage: async ({ page }, use) => {
await use(new SettingsPage(page))
},
})
export { expect } from '@playwright/test'
Examples:
// File: without_fixtures.ts
const { test } = require('@playwright/test')
const { TodoPage } = require('./todo-page')
test.describe('todo tests', () => {
let todoPage
test.beforeEach(async ({ page }) => {
todoPage = new TodoPage(page)
await todoPage.goto()
await todoPage.addToDo('item1')
await todoPage.addToDo('item2')
})
test.afterEach(async () => {
await todoPage.removeAll()
})
test('should add an item', async () => {
await todoPage.addToDo('my item')
// ...
})
test('should remove an item', async () => {
await todoPage.remove('item1')
// ...
})
})
// File: with_fixtures.ts
const { test } = require('@playwright/test')
const { TodoPage } = require('./todo-page')
test.describe('todo tests', () => {
test.afterEach(async ({ todoPage }) => {
await todoPage.removeAll()
})
test('should add an item', async ({ todoPage }) => {
await todoPage.addToDo('my item')
// ...
})
test('should remove an item', async ({ todoPage }) => {
await todoPage.remove('item1')
// ...
})
})
3. How to write test case
3.1. Aliases
- Purpose: to make test script look similar with other test frameworks like Jest, Mocha, Jasmine, etc...
- Alias list:
describeis an alias of thetest.describeitis an alias oftestbeforeAllis an alias oftest.beforeAllbeforeEachis an alias oftest.beforeEachafterEachis an alias oftest.afterEachafterAllis an alias oftest.afterAll
- Example
// File: with-aliases.ts
describe('Test suite', () => {
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
})
// File: without-aliases.ts
test.describe('Test suite', () => {
test.beforeAll(async ({}) => {
// beforeAll hook
})
test.beforeEach(async ({}) => {
// beforeEach hook
})
test('Test case #1', async ({}) => {
// Test case script
})
test('Test case #2', async ({}) => {
// Test case script
})
test.afterEach(async ({}) => {
// afterEach hook
})
test.afterAll(async ({}) => {
// afterAll hook
})
})
3.2. Common test cases
3.2.1. Conventions
- File name must be in lowercase. Can be a test suite or test case code. Example
c12345.tsfor test case ands56789.tsfor test suite - Each test file must contain a TestRail link to the test case or test suite. Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
...
it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => {
...
})
- Tip: Script
yarn unify-testcasecan help you:- Change your file name to be in lower-case
- Prepend TestRail link to test case based-on your file name
3.2.2. Test script structure
Structure of test case contains following factors:
describeortest.describeto declare test suiteitortestto declare testcasebeforeAllortest.beforeAllto declare setup step for test suitebeforeEachortest.beforeEachto declare setup step for each test caseafterEachortest.afterEachto declare teardown step for each test caseafterAllortest.afterAllto declare teardown step for test suite
Example:
// File: with-test-suite.ts
describe('Test suite', () => {
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
})
// File: without-test-suite.ts
beforeAll(async ({}) => {
// beforeAll hook
})
beforeEach(async ({}) => {
// beforeEach hook
})
it('Test case #1', async ({}) => {
// Test case script
})
it('Test case #2', async ({}) => {
// Test case script
})
afterEach(async ({}) => {
// afterEach hook
})
afterAll(async ({}) => {
// afterAll hook
})
3.3. E2E test cases
describe,it,beforeAll,beforeEach,afterEach,afterAllandexpectare imported from@/libs/fixtures/e2e- Available fixtures:
paginationinlineNotificationtoastglobalHeadersidebarMenuloginPageswitchOfficePagenewOfficePageemployeeListPageemployeeInvitePageemployeeDetailPageemailSettingPageemailTemplatePreviewPageemailTemplateEditPagesendPackageListPagecompanySettingPagesenderInformationPage
- Example:
// File: c57293.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57293
import { accountData } from '@/data/account'
import { globalHeaderData } from '@/data/global-header'
import { ACCOUNT_0_STATE } from '@/libs/constants/store'
import { beforeEach, expect, it, use } from '@/libs/fixtures/e2e'
use({ storageState: ACCOUNT_0_STATE })
beforeEach(async ({ sendPackageListPage }) => {
await sendPackageListPage.goto()
})
it('C57293 Verify menu header after user logs on the SCI system successfully', async ({ globalHeader }) => {
// User dropdown
await globalHeader.openUserDropdown()
await expect(globalHeader.userDropdown).toContainText(accountData.user[0].name)
await expect(globalHeader.userSettingButton).toContainText(globalHeaderData.userDropdown.userSettingText)
await expect(globalHeader.userSettingButton).toHaveAttribute(
'href',
`${process.env.MFID_BASE_URL}${globalHeaderData.userDropdown.userSettingLink}`
)
await expect(globalHeader.logoutButton).toBeVisible()
await expect(globalHeader.logoutButton).toContainText(globalHeaderData.userDropdown.logoutButtonText)
// Office dropdown
await globalHeader.openOfficeDropdown()
await expect(globalHeader.switchOfficeButton).toBeVisible()
await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].tenantName)
await expect(globalHeader.officeDropdown).toContainText(globalHeaderData.officeDropdown.officeBusinessNumberText)
await expect(globalHeader.officeDropdown).toContainText(accountData.office[0].businessNumber)
await expect(globalHeader.switchOfficeButton).toContainText(globalHeaderData.officeDropdown.switchOfficeButtonText)
})
3.4. API test cases
describe,it,beforeAll,beforeEach,afterEach,afterAllandexpectare imported from@/libs/fixtures/api- Available fixtures:
apiClientnavisClientinvoiceDeliveryClientsuzakuLoginPage
- Example:
// File: c57134.ts
// Link: https://moneyforwardvietnam.testrail.io/index.php?/cases/view/57134
import { accountData } from '@/data/receiver-account'
import { describe, expect, it } from '@/libs/fixtures/api'
/**
* Endpoints: PUT /v1/user/terms
*/
describe('SCI-3365-Receiver can register MFID', () => {
it('C57134-Verify agree the term process before user agrees the term', async ({ apiClient }) => {
const loginResult = await apiClient.login(accountData.user[1].email, accountData.user[1].password)
expect(loginResult.status).toBe(200)
expect(loginResult.response).toHaveProperty('data')
expect(loginResult.response.data.user).not.toBe(null)
expect(loginResult.response.data.user.accept_terms).toBe(false)
expect(loginResult.response.data.user.has_available_tenant).toBe(true)
})
})
4. Reports
4.1. Report overview
Slack channel: #sci_automation_notification
Report type:
- Daily: Every morning from Monday to Friday (GTM+7)
- Monthly: TBD
Trigger type:
nightlyormanualApp name:
senderorreceiverTesting type:
apiore2eReport format
Title:
[{TRIGGER_TYPE}] [{APP_NAME}] {TESTING_TYPE} testing report for {RUN_DATE}Content:
- Total passed
- Total failed
- Total flaky
- Total skipped (due to depending on test case is failed)
- Total timed out (due to auto-waiting exceeding configured timeout)
- Total interrupted
Example:

Link to report detail
Playwright report

Allure report

4.2. Custom reporter for Playwright
- You can create a custom reporter by implementing a class with some of the reporter methods. Learn more about the Reporter API.
// File: my-awesome-reporter.ts
import type {
FullConfig, FullResult, Reporter, Suite, TestCase, TestResult,
} from '@playwright/test/reporter'
class MyReporter implements Reporter {
onBegin(config: FullConfig, suite: Suite) {
console.log(`Starting the run with ${suite.allTests().length} tests`)
}
onTestBegin(test: TestCase, result: TestResult) {
console.log(`Starting test ${test.title}`)
}
onTestEnd(test: TestCase, result: TestResult) {
console.log(`Finished test ${test.title}: ${result.status}`)
}
onEnd(result: FullResult) {
console.log(`Finished the run: ${result.status}`)
}
}
export default MyReporter
- Now use this reporter with testConfig.reporter.
// File: playwright.config.ts
import { defineConfig } from '@playwright/test'
export default defineConfig({
reporter: './my-awesome-reporter.ts',
})
4.3. Summary Reporter
This report will calculate total passed, failed, flaky, skipped, timed-out and interrupted test cases as well as passed ratio and total duration. The result will stored in JSON format
// File: summary-reporter.ts
import { FullConfig, Reporter, Suite } from '@playwright/test/reporter'
import * as fs from 'fs'
import { getCurrentDate, convertMsToTime } from './path/to/helpers'
export type AppName = 'sender' | 'receiver'
export type TestType = 'e2e' | 'api'
export type SummaryReporterOption = {
outputFile: string
appName: AppName
testType: TestType
}
class SummaryReporter implements Reporter {
private suiteList: Suite[] = []
private readonly outputFile: string
private readonly appName: AppName
private readonly testType: TestType
constructor({ outputFile, appName, testType }: SummaryReporterOption) {
this.outputFile = outputFile
this.appName = appName
this.testType = testType
}
onBegin(config: FullConfig, suite: Suite) {
this.suiteList.push(suite)
}
onEnd() {
let total = 0
let totalPassed = 0
let totalFailed = 0
let totalFlaky = 0
let totalSkipped = 0
let totalTimedOut = 0
let totalInterrupted = 0
let totalDuration = 0
this.suiteList.forEach((suite) => {
suite.allTests().forEach((test) => {
let isPassed = false
let isFailed = false
let isFlaky = false
test.results.forEach((result) => {
result.status === 'skipped' && totalSkipped++
result.status === 'timedOut' && totalTimedOut++
result.status === 'interrupted' && totalInterrupted++
result.duration
if (result.retry === 0) {
isPassed = result.status === 'passed'
isFailed = result.status === 'failed'
} else {
if ((result.status === 'passed' && isFailed) || (result.status === 'passed' && isPassed)) {
isFlaky = true
isPassed = false
isFailed = false
}
}
totalDuration += result.duration
})
isPassed && totalPassed++
isFailed && totalFailed++
isFlaky && totalFlaky++
total++
})
const passedRatio = (totalPassed / total) * 100
fs.writeFileSync(
this.outputFile,
JSON.stringify({
title: `[${this.testType.toUpperCase()}] [${this.appName.toUpperCase()}] Report for ${getCurrentDate()}`,
total,
totalPassed,
totalFailed,
totalFlaky,
totalSkipped,
totalTimedOut,
totalInterrupted,
passedRatio: isNaN(passedRatio) ? 'N/A' : `${passedRatio.toFixed(2)}%`,
totalDuration: totalDuration === 0 ? 'N/A' : convertMsToTime(totalDuration),
})
)
})
}
}
export default SummaryReporter
4.4. TestRail Reporter
This report will:
- Detect TestRail test cases' id via test-case title
- Create
TestMilestonein the first day of a week (if does not exist) - Close
TestMilestonein the last day of a week (if does not close) - Create
TestRunwhenever running an Automation Test - Close
TestRunafter running the Automation Test - Create
TestResultfor all detected test cases
// File: testrail-reporter.ts
import TestRail from '@dlenroc/testrail'
import { Reporter, TestCase, TestResult } from '@playwright/test/reporter'
import dayjs from 'dayjs'
import isoWeek from 'dayjs/plugin/isoWeek'
import * as fs from 'fs'
import { getCurrentDate, getCurrentWeek, getPrefixTitle, stripAnsi } from './path/to/helpers'
dayjs.extend(isoWeek)
export type AppName = 'sender' | 'receiver'
export type TestType = 'e2e' | 'api'
export type TestRailReporterOption = {
outputFile: string
host: string
username: string
password: string
projectId: number
suiteId: number
appName: AppName
testType: TestType
}
class TestRailReporter implements Reporter {
private readonly testRailClient: TestRail
private initTestMilestoneProcess: Promise<void>
private testMilestone: TestRail.Milestone
private testRun: TestRail.Run
private readonly outputFile: string
private readonly host: string
private readonly projectId: number
private readonly suiteId: number
private readonly appName: AppName
private readonly testType: TestType
private testResultList: Array<TestRail.Result> = []
private readonly STATUS_MAPPING = {
passed: 1,
failed: 5,
timedOut: 5,
}
constructor({ username, password, ...options }: TestRailReporterOption) {
Object.assign(this, options)
this.testRailClient = new TestRail({
host: options.host,
username,
password,
})
}
// Event handlers
// --------------------------------
async onBegin() {
this.initTestMilestoneProcess = this.initTestMilestone()
}
async onTestEnd(test: TestCase, result: TestResult) {
await this.initTestResult(test, result)
}
async onEnd() {
await this.initTestMilestoneProcess
if (this.testResultList.length > 0) {
await this.initTestRun()
await this.updateTestResult()
}
this.genResult()
await this.closeTestRun()
await this.closeTestMilestone()
}
// Gen result
// --------------------------------
genResult() {
fs.writeFileSync(
this.outputFile,
JSON.stringify({
link: `${this.host}/index.php?/runs/view/${this?.testRun?.id || ''}`,
})
)
}
// TestRail businesses
// --------------------------------
async initTestMilestone() {
const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Milestone for ${getCurrentWeek()}`
const milestoneList = await this.testRailClient.getMilestones(this.projectId, {
is_completed: false,
})
let testMilestone = milestoneList.filter((item) => item.name === name)[0]
if (!testMilestone) {
testMilestone = await this.testRailClient.addMilestone(this.projectId, {
name,
})
}
if (!testMilestone.is_started) {
testMilestone = await this.testRailClient.updateMilestone(testMilestone.id, {
is_started: true,
})
}
this.testMilestone = testMilestone
}
async closeTestMilestone() {
if (dayjs().day() === 5) {
await this.testRailClient.updateMilestone(this.testMilestone.id, {
is_completed: true,
})
}
}
async initTestRun() {
const name = `[SCI] ${getPrefixTitle(this.appName, this.testType)} Run for ${getCurrentDate()}`
const testRunList = await this.testRailClient.getRuns(this.projectId, {
suite_id: this.suiteId,
milestone_id: this.testMilestone.id,
})
let testRun = testRunList.filter((item) => item.name === name)[0]
if (!testRun) {
testRun = await this.testRailClient.addRun(this.projectId, {
suite_id: this.suiteId,
milestone_id: this.testMilestone.id,
name,
})
}
if (!testRun.include_all) {
testRun = await this.testRailClient.updateRun(this.projectId, {
include_all: true,
})
}
this.testRun = testRun
}
async closeTestRun() {
await this.testRailClient.closeRun(this.testRun.id)
}
async initTestResult(test: TestCase, result: TestResult) {
const testCaseIdList = test.title.match(/(C[0-9]+)/gi)?.map((item) => +item.replace(/c/i, ''))
if (!Array.isArray(testCaseIdList) || testCaseIdList.length === 0) {
return
}
if (result.status === 'skipped') {
return
}
let runNumber = ''
if (result.retry === 0) {
runNumber = `First run`
} else {
runNumber = `Retry #${result.retry}`
}
let errorMessage = ''
if (result.status === 'failed') {
let errorDetails = ''
errorDetails += stripAnsi(result.error.message)
errorDetails += '\n'
if (result.error.location) {
errorDetails += ` at ${result.error.location.file}:${result.error.location.line}:${result.error.location.column}`
}
const errorArr = errorDetails.split('\n').map((item) => ` ${item}`)
errorMessage += `\nError:\n\n${errorArr.join('\n')}`
}
this.testResultList = [
...this.testResultList,
...testCaseIdList.map(
(item) =>
({
case_id: item,
status_id: this.STATUS_MAPPING[result.status],
comment: `${runNumber}\n\nDuration: ${result.duration}ms\n\n${errorMessage}`,
} as unknown as TestRail.Result)
),
]
}
async updateTestResult() {
await this.testRailClient.addResultsForCases(this.testRun.id, {
results: this.testResultList,
})
}
}
export default TestRailReporter
5. Nightly and manual test on CircleCI
Nightly and manual tests will run only on main branch
5.1. Nightly test
Will test
apiande2efor all applicationsWill be triggered at:
- With
UTCtimezone: 8:00 PM on Sunday, Monday-Thursday - With
GTM+7timezone: 3:00 AM on Monday-Friday
- With
Scheduled trigger on CircleCI:

Purpose: ensures that you catch such problems within 24 hours when they occur
5.2. Manual test
Will be triggered manually when:
New code is added to
mainbranch (can be merged PR or commit code directly to these branches)Trigger via CircleCI
- Step 1: Select
mainbranch - Step 2: Click
Trigger Pipelinebutton
- Step 1: Select
Purpose: trigger
apiande2etesting for a specified application on demand
6. Feature plans
- Implement
Monthlyreport about detected bugs - Auto create JIRA ticket when finding bug after running Automation Test
- Generate test cases for testing type of parameters. Example:
username: stringparam will be tested with following input:null''!@#$%^&*()true123456789
![HTTP Adventure: [Part 2] The evolution of HTTP/2](https://static-careers.moneyforward.vn/Blog/HTTP%20series/%5BPart%202%5D%20The%20evolution%20of%20HTTP2.png)

Tips to ensure the quality of CSV Import feature in Web Applications

