Full Stack Engineer (Web Scraping)

Job description


Daltix is a fast-growing, successful start-up from Belgium with offices in Boom, Ghent and Lisbon. We bring real-time data-driven insights to the world of retail and fast moving consumer goods. How are we doing this? With a cutting-edge big data platform that gathers, processes, analyzes and enhances huge amounts of unstructured retail data. We then tie it all together with a suite of in-house business intelligence tools and clever data science techniques that are used daily by our customers to stay ahead of the competition.

Today, Daltix is a market leader in the Benelux region serving household names like Makro, Lidl, Dreamland, Jumbo, Greenyard and Unilever. Thanks to the success of our platform, we’re now scaling up across Europe and are looking for exceptional engineers to help us do it.


Sounds awesome! What do I need for this role?

As an experienced Full Stack Engineer you’ll join our tight knit Data Collection team to help build the next generation of Daltix’s distributed data collection & web scraping platform that lies at the heart of our business. To succeed, you’ll need to be a team player with strong knowledge and experience across the stack - with a particular focus on backend.

We are a data-driven company and ingest hundreds of gigabytes of raw data daily, across 60 sites. Your task will be to help the team triple that without losing data completeness or quality. To do this, you will need to be comfortable working with an eclectic mix of technologies and approaches to ensure we can reliably and repeatedly extract the data we need, when we need it regardless of how sophisticated the target is.


Tasks and areas you’ll be deeply involved with

  • Developing and scaling out the company’s next generation Javascript web scraping framework (modular+testable ES6, puppeteer/DTP, custom browser extensions etc.)
  • Helping us transition to a fully containerised, cloud native production environment
  • Building out our orchestration, metrics, alerting and tracing infrastructure
  • Improving our CI/CD pipelines
  • Developing and implementing novel solutions to defeat the most well defended targets (fingerprint avoidance, advanced proxy strategies, reverse engineering mobile APIs)
  • Improving our data transformation and QA systems
  • Boosting test coverage and overall code quality
  • Mentoring junior engineers to help them grow and unlock their potential

About our stack

  • Daltix’s infrastructure runs on top of Amazon Web Services and most of our production code is written in Python and Javascript.
  • We’re big on Hashicorp (Terraform, Packer, Vagrant, Consul... and soon Nomad)
  • We use CircleCI for integration/testing and a modern stack for logging, monitoring and alerting (Prometheus, Grafana, Alert Manager, Loki)
  • Where it makes sense, we heavily leverage serverless
  • In addition to Docker and Terraform, we also use Ansible for provisioning and configuration management
  • We follow Agile and use Jira to keep on top of our backlogs.

What's great about our engineering culture?

  • We work hard but it’s never “Crunch-Time” for us. No sleeping under desks or 16-hour days.
  • We use the best technology to solve problems, whether it’s something new or something established.
  • Team of smart, collaborative people working together to solve hard, interesting problems for real customers.


Requirements

What you need to be a perfect fit for this role

Required

  • At least 3-5 years of strong programming experience in Python and/or Javascript.
  • Exposure to Docker and some DevOps, TechOps or Network Operations experience with public cloud (GCP/AWS/Azure).
  • Knowledge of core TCP/IP networking principles and design.
  • Linux OS-level knowledge and proficient with command-line/scriptingand configuration management principles
  • Experience with relational and “nosql” databases/warehouses (e.g. Postgres, Redis, DynamoDB, Snowflake)
  • Experience with front-end dev (classic and modern SPAs)
  • Experience writing and consuming REST APIs (including knowledge of JWTs)
  • High proficiency in spoken and written English.
  • Able to adapt to changing priorities and learn quickly.

Big Plus

  • Experience with and understanding of large-scale web crawling.
  • You have experience working with highly concurrent, fault-tolerant distributed systems.
  • You have experience with big data technologies (such as Spark, Dask, Airflow, Cassandra, Kafka/Pulsar, Elasticsearch/Solr/Lucene)
  • You have experience with serverless, message queues and micro services architecture
  • Security minded with an understanding of cloud possibilities and limitations (especially in the areas of distributed systems, load balancing, networking and massive data storage).
  • You get energy from working in a highly complex and challenging startup environment with a high tech product.

If you still need more convincing… 

  • We are a young, entrepreneurial and fast-growing company; you will have the opportunity to shape our future and make a positive impact on our clients’ business
  • You will be offered a competitive wage in a talented, international team of experts
  • Flexible work arrangements with a lot of autonomy in what you do and where you do it - we trust you to know your schedule and work when you feel most productive
  • We’ve got great infrastructure for working remotely (easy to drop-in hangouts, a lot of communication over Slack and Github)
  • Great collaborative environment. Everyone wants to improve and get better at coding.
  • An open company culture where we play as hard as we work. You’ll get the chance to meet and work with industry professionals and help lift the company to the next level.
  • You will be able to participate in relevant training to stay at the top of your field.
  • Be part of an interesting and dynamic start-up, enjoying a front row seat as we scale up.
  • A cosy office in the heart of Lisbon
  • Health Insurance coverage
  • We never run out of coffee/snacks/fresh fruit/beer
  • Occasional company drinks and team events
  • Besides developing your technical skills you will also have the opportunity to grow into the following skill sets: Platform/architectural lead, project management/owner, team leading & coaching.


Join us on the path to becoming the Google of retail and change the way retailers and suppliers work.