This job is closed
Applications are no longer available for this announcement. Explore current related opportunities below.
Job description
Job Description
Create crawlers for each source URL using Python modules (Scrapy, Selenium, Requests, BeautifulSoup, Splash).
Develop and maintain Scrapy pipelines and middlewares to manage crawler output.
Build crawlers for all types of websites, overcoming technical roadblocks.
Manage crawlers to handle IP bans, geolocation bans, CAPTCHAs, and bot‑blocking services.
Write SQL queries for database operations using Python modules such as SQLAlchemy.
Deploy the Python scripts and crawlers to Linux‑based AWS servers.
Skills Must Haves
Strong hands‑on experience in Python programming.
Good experience with scraping libraries such as Requests, BeautifulSoup, Selenium and Scrapy.
Proven 3+ years working with web development frameworks such as Flask, Django, FastAPI, Tornado, and pandas, and building APIs and services using REST.
Experience with any RDBMS and strong SQL knowledge.
Clear understanding of object‑oriented concepts.
Proficient understanding of code versioning tools like Git.
Nice to Have
Familiarity with UI frameworks such as Angular or ReactJS.
Strong unit‑testing and debugging skills.
Knowledge of data engineering platforms such as Airflow.
Experience with Docker.
Knowledge of Agile and Scrum practices; experience with Jira and Confluence is a plus.
Experience with ETL and Celery.
Excellent interpersonal skills and ability to work with a diverse team.
Qualifications
Python
Web Scraping using Requests, BeautifulSoup, Selenium, and Scrapy
Frameworks such as Flask, Django, FastAPI, Tornado, pandas
Building APIs and services using REST
#J-18808-Ljbffr
Worksite address
san francisco, CA, 94199, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.