Project:

ListBank

Goal:

Scraping Japanese Site. Managing Scraped data and deliver to clients as requirement

My Role:

DB designer, Backend and Frontend Developer

Used Technologies:

Python, Beautifulsoup, Selenium, Django, MongoDB, ReactJS,Html, CSS, git

Project Details

• System to store and serve clean scraped data as requirements .
• Used beautifulsoup, selenium to scrape data.
• No-sql(MongoDB) database to store unstructured data.
• Developed function to store clean data from uploaded file(csv, excel) into mongoDB.
• Search and filter function with different parameter and download data as csv or excel format.
• Used djongo as Database driver to use Django ORM with MongoDB.
• Used React DOM to load all pages at a time and update page segments/data without full page load which feels the system much faster.