Case study
Crypto Web Scraper with Puppeteer
A web scraping application built with JavaScript and Node.js to automate cryptocurrency data collection from Yahoo Finance. The project uses Puppeteer for browser automation and data extraction, integrates an external exchange-rate API to convert USD values to BRL, and dynamically presents the processed results in an HTML table.
Technologies
The problem
Financial data available on web pages does not always provide a suitable API interface for specific experiments or automation scenarios. The challenge was to automatically collect cryptocurrency information directly from a web page, structure the extracted data, perform additional calculations, and present the information in a readable format.
The solution
A Node.js application was developed using Puppeteer to control a browser and navigate through Yahoo Finance pages. The application identifies relevant elements through DOM selectors, extracts cryptocurrency names, prices, and percentage changes, and organizes the collected information into structured data. After the extraction process, an external exchange-rate API is queried to retrieve the current USD-to-BRL conversion rate. The resulting data is processed and used to dynamically generate an HTML page containing cryptocurrency prices, changes, and converted BRL values.
Responsibilities
Developed the end-to-end scraping solution, including browser automation, DOM selector definition, data extraction, value normalization, exchange-rate API integration, and generation of the HTML interface used to present the collected information. The application responsibilities were also separated into classes and utility functions to keep scraping, data processing, and presentation logic organized and decoupled.
Technical decisions
Puppeteer was selected to control a real browser and access content directly from the page DOM. Scraping responsibilities were encapsulated into classes, separating selector configuration from the actual extraction process. Data extraction uses page.evaluate() to execute DOM queries directly within the browser context. Asynchronous programming was used to coordinate navigation, scraping operations, and external API requests. Extracted values are normalized before calculations, while an exchange-rate API provides the USD-to-BRL conversion used by the application. The native Node.js HTTP module was used to serve the dynamically generated HTML page without introducing an additional web framework.
Results
The final application was capable of automatically collecting cryptocurrency information, processing the extracted values, and enriching the data with runtime currency conversion. The project resulted in a functional integration of browser automation, web scraping, data processing, external API consumption, and dynamic HTML generation.