Sr. Big Data Engineer

Chicago, Illinois
Hirewell

US Citizen or Green Card only

Must be capable to commute to downtown Chicago eventually

Sr. Staff Engineer

We are looking for a Senior Staff Big Data Engineer to join our Data Engineering team. This position will be the Senior technical resource driving architecture for the integration of large 3rd party partner integrations with companies like Facebook, Google and Twitter to name a few. The ideal candidate is naturally curious, dedicated, detail-oriented with a strong desire to work with awesome people in a highly collaborative environment. This position will require the ability to own and lead data initiatives on a cross-functional team.

What you’ll do:

• Migrate existing data pipelines from on-prem regional data centers to AWS and GCP.
• Architect a new modern event driven architecture with both batching and streaming
• Adjust existing pipelines to fit the AWS processing model such as integration with S3 and EMR, migrate to open source version of hadoop, adjustments to security model, etc…
• Working on Big Data technologies such as Hadoop, MapReduce, Kafka, and/or Spark in columnar databases
• Architect, design, code and maintain components for aggregating tens of billions of daily transactions
• Lead the entire software lifecycle including hands-on development, code reviews, testing, deployment, and documentation for streaming and batch ETL’s and RESTful API’s
• Partner and work closely with the QA Engineers to develop automated tests
• Participate in training and mentoring of junior team members

You should apply if you have most of this:

• 15+ years of experience designing and building data-intensive applications
• 10+ years architecting systems in a big data ecosystem using MapReduce, Spark, MPP Data Warehouses, and sql/nosql databases.
• 10+ years recent hands-on experience with object oriented languages (Java, Scala, Python)
• 5+ years Hands on experience building production level systems in a cloud environment (AWS or GCP)
• Excellent interpersonal and communication skills in English
• Proven experience leading the design and execution of event driven architectures for distributed systems
• Experience designing systems for performance, scalability, and reliability
• In-depth understanding of object oriented programming concepts
• Low level working knowledge of collections, multi-threading, JVM memory model, etc.
• Solid understanding of database fundamentals and SQL
• Understanding the full software development life cycle, agile development and continuous integration
• Ability to clearly communicate with team-members in a cross-matrix environment

What puts you over the top:

• Built systems in a containerized environment with familiarity in Docker, ECS, Kubernetes
• Exposure to Data Warehousing solutions like Snowflake and BigQuery
• Prior ad tech experience