Data Automation Engineer

We are looking for Data Acquisition Research Engineers. They will develop and maintain large-scale web scraping and data acquisition systems. The role focuses on designing, developing and constantly maintaining resilient data collection and acquisition systems and scrapers for complex web platforms. This includes environments where adversarial and reverse engineering may be required. You will develop robust data pipelines and ensure reliable collection of large volumes of structured and unstructured data.

Responsibilities

• Develop and maintain data acquisition systems: web scrapers and automated data collection
• Develop data extraction methods that work with data repositories such as Websites, APIs, and GraphQL endpoints
• Research and solve data collection challenges in adversarial environments (rate limits, blocking, anti-bot systems, ...)
• Develop and integrate scalable and reliable data pipelines
• Monitor scrapers and ensure data quality and pipeline stability
• Work closely with other teams to transfer collected data into downstream systems

Requirements

• Strong Python development experience
• Experience with web scraping frameworks (Scrapy, Playwright, Selenium, Puppeteer, etc.)
• Familiarity with GraphQL APIs and REST APIs
• Solid understanding of networking fundamentals (HTTP/HTTPS, proxies, headers, cookies, rate limiting)
• Experience handling anti-bot protections and detection mechanisms
• Experience with CI/CD workflows for testing and deploying scraping infrastructure
• Ability to parse and process HTML, JSON, and dynamic web content