Principal Data Engineer

RESPONSIBILITIES:

• Develop and operate cross-company data infrastructure.
• Manage data integration and data pipelines.
• Aggregate data from various internal products, services, and
• CRM tools into a data lake.
• Appropriately distribute the data collected in the data lake to analysis and ML platforms.
• Build and maintain data analysis infrastructure.
• Optimize DWH performance.
• Ensure data quality.

REQUIREMENTS: 

Must have:
.      - AI software development:

• Experienced in developing AI algorithms or understanding how it works from scratch
• Developed an Application Programming Interface as an internet service
• Operated and manipulated an API service by using appropriate software
• Controlled and tuned accuracy through operating data

        - Data engineering:

• Data Warehousing
• ETL Processes
• Cloud Platforms (AWS, Azure, GCP)
• Python
• SQL
• Big Data Technologies (Hadoop, Spark, Kafka)
• Data Modeling
• Data Quality
• Data Security
• Version Control (Git)

        - Tools: 

• Configuration management: Terraform
• CI/CD: GitHub Actions
• Monitoring and logging: Datadog, Cloud Monitoring, CloudWatch
• Project management: JIRA Cloud, Miro,...
• Documentation: Kibela, Google Workspace
• Spark: 2-3 years of experience
• Airflow: 1 year (As a user not administrator)

        - Cloud architecture:

• Experienced developing a system using some cloud services on AWS, GCP, or Azure(AWS is preferable).
• Understood cloud service components from architectural perspective(To know what can do, how it works and what should be paid attention)
• Designed and integrated a system with appropriate security.

Nice to have:

• Good Communication skills and an open-mind.
• Good at English (Four skills).