Senior Automation Engineer

We are seeking a Senior Automation Engineer with deep expertise in leading-edge test automation for complex, real-time, LLM-powered conversational platforms. You will architect, implement, and evolve sophisticated automated testing strategies focused on integration and functional validation of our web application and Slack/Microsoft Teams integrations.

Our product features highly dynamic components: multi-turn LLM-driven conversations, real-time event flows via WebSockets, cross-channel interactions, and rigorous quality assurance of AI-generated coaching outputs to mitigate hallucinations, ensure faithfulness, and maintain enterprise-grade compliance. You will establish robust, low-maintenance automation pipelines incorporating self-healing capabilities, natural language test generation, agentic execution, and advanced LLM evaluation to deliver exceptional reliability, performance, and scalability. Close collaboration with engineering, product, ML, and operations teams will be essential to enforce high-confidence quality gates and accelerate secure, rapid releases.

Key Responsibilities

• Provide senior-level leadership in test automation, owning the architecture and delivery of production-grade automated testing solutions that protect mission-critical AI coaching functionalities and integrations.
• Design and implement scalable automation frameworks for functional, integration, end-to-end, and regression testing across Next.js/React frontends, Node.js backends, and real-time conversational workflows
• Develop advanced automated API testing for RESTful services, microservices, and event-driven architectures, with rigorous validation of non-deterministic LLM responses through deterministic assertions and semantic checks.
• Lead automated testing of Slack and Microsoft Teams integrations, encompassing bot logic, interactive components, webhooks, notifications, multi-turn dialogues, and real-time event processing to guarantee consistent, channel-agnostic user experiences.
• Architect automated validation suites for WebSocket-based real-time communication, event-driven systems, low-latency data pipelines, and dynamic AI response coherence in conversational contexts.
• Integrate state-of-the-art LLM quality control automation, including hallucination detection, faithfulness scoring, bias monitoring, semantic similarity assessment, and production regression testing of model deployments in partnership with the ML team.
• Champion industry-leading practices such as AI-augmented self-healing tests (e.g., via tools like Mabl, Testim, or Applitools), natural language test authoring (e.g., testRigor, KaneAI), agentic test orchestration, visual AI validation, and continuous test optimization to eliminate flakiness in AI-interactive UIs and flows.
• Establish rigorous standards for test architecture, code quality, security in testing, and observability. Mentor team members, conduct thorough test code reviews, and cultivate a culture of engineering and quality excellence.
• Collaborate cross-functionally to convert product, AI, and compliance requirements into automated, high-assurance testing solutions that support rapid, safe iteration.
• Own continuous enhancement of test coverage, pipeline efficiency, and observability, proactively implementing improvements to sustain top-tier performance amid growing Fortune 500-scale adoption.

• Core Technical Expertise
• Frontend: In-depth proficiency testing modern React/Next.js applications using Playwright (preferred for cross-browser reliability and speed) or Cypress, with emphasis on dynamic, AI-interactive UIs.
• Backend & API: Expert-level automation of Node.js services, scalable APIs, microservices, and event-driven architectures utilizing Jest, Mocha, or Supertest.
• DevOps & Cloud: Advanced mastery of CI/CD orchestration (GitHub Actions, GitLab CI, Jenkins), containerization (Docker, Kubernetes), AWS infrastructure provisioning and testing environments, infrastructure as code (Terraform preferred), cloud security testing practices, and pipeline observability (e.g., integration with monitoring tools like Datadog or Prometheus for test metrics and AI workload tracing).

Highly Valued / Cutting-Edge Competencies

Modern Automation Frameworks: Hands-on experience with Playwright, Cypress, or AI-enhanced platforms featuring self-healing and agentic capabilities (e.g., Mabl, testRigor, Applitools, Testim, Katalon).

• LLM & AI Evaluation: Proven expertise with leading LLM evaluation frameworks such as DeepEval (for pytest-style testing and metrics like G-Eval, faithfulness, hallucination), RAGAS (for retrieval-augmented pipelines), Braintrust, LangSmith (LangChain ecosystem), or Arize Phoenix—enabling automated scoring of semantic relevance, bias, toxicity, and production quality gates.
• Python Proficiency: Strong command of Python for test orchestration, data-driven scenarios, ML model validation, and integration with libraries such as Pytest.
• Real-Time & Integration: Extensive automation experience with Slack/Microsoft Teams bots, webhooks, interactive elements, and WebSocket/event-driven real-time systems (including tools supporting WebSocket testing like Postman, Hoppscotch, or custom Playwright extensions).

Key Competencies

• Elite Individual Contributor: Demonstrated ability to independently own and deliver end-to-end, high-stakes automation initiatives that measurably elevate platform reliability and velocity.
• Advanced Problem-Solving: Exceptional capacity to devise innovative, efficient architectures for validating non-deterministic, multi-component conversational AI systems.
• Strategic Communication: Superior ability to articulate sophisticated testing strategies, architectural rationale, risk assessments, and outcomes to technical peers, product leaders, and executive stakeholders.
• Quality-Centric Leadership: Unwavering dedication to engineering secure, compliant, user-focused automation frameworks that prioritize AI trustworthiness, data protection, and flawless enterprise experiences