We’re seeking a Data Engineer to join our team and manage the backbone of our business: our massive market data platform. This role is central to improving the efficiency, quality, and utility of the data we provide to our customers.
Key Responsibilities
- Database Management: Oversee the performance, scaling, and maintenance of our large-scale market data database to ensure high availability and reliability.
- Performance Optimization: Proactively identify and implement improvements to data pipelines, storage, and retrieval processes to increase efficiency and reduce latency.
- AI/ML Integration: Lead the effort to integrate Artificial Intelligence (AI) and Machine Learning (ML) tools to optimize the dataset, including data repair, anomaly detection, and feature engineering.
- Feature Development: Collaborate with product and customer-facing teams to design and build new data features and interfaces that enhance the value and usability of our data for clients.
- Quality Assurance: Develop and implement rigorous data validation and quality checks to maintain the integrity and accuracy of the market data.
- Tooling & Automation: Select, implement, and maintain the necessary tools and technologies to automate data engineering tasks and ensure best practices.
Required Skills & Experience
- Proven experience as a Data Engineer, Database Administrator, or similar role, preferably in a financial services or market data environment.
- Expertise in managing and optimizing large, complex datasets (VLDBs).
- Strong proficiency in SQL and experience with various database systems (e.g., PostgreSQL, Snowflake, NoSQL).
- Experience with cloud data platforms (e.g., AWS, Azure, GCP).
- Familiarity with ETL methodologies and technologies.
- Familiarity with various data warehousing techniques and technology stacks.
- Familiarity with large TSDB deployments.
- Familiarity with PostgreSQL optimization techniques such as indexing, sharding, partitioning, etc.
- Working knowledge of Python or other relevant programming languages for data manipulation and scripting.
What We're Looking For
A proactive problem-solver with a strong background in data architecture who is excited by the challenge of managing a high-volume, real-time market data platform and driving innovation through AI/ML optimization. Most importantly, the engineer has a drive for learning and creating robust, sustainable solutions.
