Software Engineer

We are hiring a staff software engineer (IC4) for the Telemetry Data Platform team, which carries product usage telemetry for the entire ServiceNow platform on a stack built with Kubernetes, Kafka, and ClickHouse. We are distributed across Israel, India, and the Americas. This is a full stack role with its center of gravity firmly on the infrastructure side — expect roughly two-thirds of your time on telemetry infrastructure, data pipelines, Kubernetes and DevOps work, and the rest on the services and product surfaces on top. You will be equally comfortable designing a distributed system and owning it in production. If you like following a system from the event that fires to the chart that renders — and you want the pager for it — you will feel at home. You will lead design and delivery of complex, multi-service features, own technical decisions within a domain, and are fully self-directed — escalating only genuine ambiguities and driving decisions to closure. Of the ten critical skills at this level, incident response management is the only one set at Expert; the rest are Experienced. What you’ll do Telemetry infrastructure and data pipelines - Design, build, and own backend services for distributed data streaming and processing that handle high-volume, high-cardinality event data with predictable latency and no silent data loss. - Build and maintain Kafka-based streaming pipelines and the pipeline components that feed ClickHouse. - Own data modeling and query performance in ClickHouse — partitioning, sort keys, materialized views, retention, and the cost curve that comes with all of it. - Partner with product and platform teams to shape requirements for telemetry ingestion and processing, then drive the solutions to production. Kubernetes, DevOps, and production ownership - Deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines that make releases repeatable and safe to roll back. - Own observability for what you build — meaningful metrics, useful logs, real tracing, and alerts that fire on customer impact rather than on noise. - Debug and resolve production incidents independently, participate in on-call, run root-cause analysis, and operate against defined service level objectives. Full stack engineering and technical leadership - Own the full development lifecycle for your work, and build the backend services and APIs that expose telemetry to internal consumers, product surfaces, and AI agents — including API contracts, data models, and schema migrations. - Contribute to the analytics front end — dashboards, funnels, and exploration tools — with an eye on performance against large result sets, and improve the web and mobile capture SDKs. - Lead the design of complex, multi-service features across team boundaries, drive decisions to closure, and write the design docs and postmortems that outlive the conversation. - Raise the bar through code review, test strategy, and automation coverage, and mentor engineers on the team. Qualifications Experience and education requirements - The job profile defines IC4 by scope, independence, and impact rather than years served. The minimums below are the screening bar for this requisition. - 8+ years of Backend software engineering experience - 4+ years of Designing and operating distributed systems - 3+ years Kubernetes in production, including Helm and CI/CD - 3+ years Kafka or equivalent streaming and data pipelines - 3+ years On-call and production ownership - Bachelor’s degree in computer science, software engineering, or a closely related technical field - A master’s degree may offset up to one year of the experience minimum. Equivalent practical experience is considered where technical depth is clearly demonstrable. Required qualifications - Strong backend development experience in Java, Python, Go or equivalent, used in production at scale. - Hands-on experience building distributed systems for data streaming, processing, and storage. - Production experience deploying and managing services on Kubernetes, including Helm and CI/CD pipelines. - Working knowledge of Kafka or an equivalent messaging and streaming system. - Solid SQL skills and hands-on experience with a columnar or analytical data store, including query optimization and physical data modeling. - Demonstrated depth in incident responseand customer escalations - Enough front-end capability to build and debug a data-heavy interface: JavaScript or TypeScript with React or Angular. Desired qualifications - Production experience with ClickHouse or another OLAP or columnar store — sharding, replication, materialized views, cost tuning. - Exposure to observability tooling: OpenTelemetry, metrics and logging stacks, alerting platforms. - Experience building product analytics or telemetry platforms, or with stream processing frameworks such as Flink or Spark. - Client-side instrumentation — web or mobile SDK development, event capture, session and consent handling. AI skills - Artificial intelligence strategy is a critical skill at Experienced proficiency for IC4 — it applies to how you build and to what you build. - Confident use of AI coding assistants such as GitHub Copilot or Cursor, paired with the critical eye to review AI-generated code as rigorously as any other — catching logic errors, security anti-patterns, and missed edge cases rather than treating model output as production-ready. - AI-assisted debugging and trace analysis, and a clear understanding of the data privacy rules governing what goes into a prompt. - Integrating LLM APIs into production features — and knowing when a model is the wrong tool for the job. Working knowledge of RAG pipelines, embedding stores, and agent frameworks; familiarity with the Model Context Protocol is a plus. - Exposing telemetry to AI agents through tool interfaces agents can use safely: predictable schemas, bounded results, clear error semantics, sane cost controls. - Designing safeguards for non-deterministic components — retry logic, circuit breakers, human-in-the-loop checkpoints — and hardening against latency spikes and hallucinated output reaching a customer. - Instrumenting AI features in production and judging their quality from telemetry and evaluation rather than impressions.