Senior Data Engineer

Senior Data Engineer Tech Stack: Python, Airflow, Scrapy, Azure AI Search, Azure, SQL. Come ambitious and hungry! About the Role Join our core engineering team to build the data backbone of our AI-powered accounting platform. You will take full ownership of our data ingestion and processing infrastructure—transforming raw, unstructured web data into high-quality search indexes for our LLMs. Responsibilities Architect resilient scraping infrastructure: Build and maintain high-volume, compliant web scrapers using Scrapy to ingest financial and regulatory data from diverse sources. Power the AI Context Window: Design pipelines to clean, chunk, and index data specifically for Azure AI Search, ensuring our RAG systems have the most relevant and up-to-date context. Orchestrate complex workflows: Design and optimize data pipelines (ETL/ELT) using Apache Airflow, ensuring data quality and timely delivery. Anti-Bot Evasion & Proxy Management: Implement sophisticated strategies to handle CAPTCHAs, IP rotations, and headless browsing to ensure 99.9% pipeline uptime. Requirements 5+ years of data engineering experience, with a heavy focus on Python. Deep knowledge of web scraping and building scraping pipelines at scale (handling anti-bot countermeasures, dynamic content, and headless browsers). Experience configuring and optimizing Azure AI Search indexes (vector search, semantic search, hybrid retrieval). Proficiency with Apache Airflow for DAG authoring and scheduling. Strong SQL skills and experience modeling data for analytics. Nice to Have Familiarity with running workloads on Kubernetes. Experience fine-tuning ranking algorithms or scoring profiles in search indexes. Knowledge of LLM integration patterns (RAG). Interested? Join us in our mission to supercharge professionals. Apply for this role