← Back to projects

GramHarvest - Instagram Data Collection Extension

Status: completed•September 12, 2025

A powerful Chrome extension built with React + TypeScript + TailwindCSS for comprehensive Instagram data collection. Extract detailed post information including metadata, engagement metrics, and media links from Instagram profiles with advanced scraping modes and export capabilities.

React 18TypeScriptViteTailwindCSS v4Chrome Extension Manifest v3Gulp
View on GitHub

Overview

GramHarvest is a sophisticated Chrome extension designed for comprehensive Instagram data collection. Built with modern web technologies including React 18, TypeScript, and TailwindCSS v4, this extension enables users to extract detailed post information from Instagram profiles. The tool features dual scraping modes, real-time progress tracking, and multiple export formats, making it a powerful solution for social media data analysis and research.

Key Features

  • Enhanced Data Collection: Extract comprehensive post data including URLs, author information, captions, thumbnails, engagement metrics, and timestamps
  • Dual Scraping Modes: Auto-scroll mode for automatic profile scraping and manual scroll mode for user-controlled collection
  • Multiple Export Formats: Download data as JSON, CSV, or Excel (.xlsx) with proper formatting
  • Real-time Progress Tracking: Live updates on scraping progress with background operation support
  • Smart End Detection: Advanced algorithms to detect when profile content has been fully loaded
  • Comprehensive History: View and manage all past scraping jobs with data preview capabilities
  • Modern UI: Instagram-inspired gradient design with backdrop blur effects and responsive layout

Technical Implementation

Architecture Overview

The extension follows Chrome Extension Manifest v3 architecture with a React-based popup interface and content scripts for Instagram page interaction. It uses Vite for fast development builds and Gulp for extension packaging, ensuring optimal performance and developer experience.

Frontend Architecture

  • React 18 with TypeScript: Component-based UI with type safety and modern React features
  • Vite Build System: Fast development server and optimized production builds
  • TailwindCSS v4: Utility-first CSS framework for responsive, modern styling
  • Extension Popup: 360px max-height interface with scrollable content and smooth animations

Extension Components

Content Scripts

  • Instagram page interaction and data extraction
  • Scroll event handling with debouncing
  • DOM parsing for post metadata collection
  • Background operation support

Service Worker

  • Background processing for scraping operations
  • Data storage and history management
  • Export format handling

Popup Interface

  • Real-time progress updates
  • Settings management (auto-scroll, export format)
  • History viewing and data preview
  • Export functionality

Data Processing Pipeline

  1. Content Detection: Identify Instagram profile pages and post elements
  2. Data Extraction: Parse post metadata including engagement metrics and media URLs
  3. Duplicate Prevention: Smart filtering to avoid collecting duplicate content
  4. Storage Management: Local browser storage with history tracking
  5. Export Processing: Format conversion for JSON, CSV, and Excel outputs

Challenges & Solutions

  1. Instagram DOM Changes: Instagram frequently updates their interface structure

    • Solution: Robust selector strategies with fallback mechanisms and regular updates

  2. Rate Limiting and Detection: Avoiding Instagram's anti-scraping measures

    • Solution: Intelligent scrolling patterns and request throttling with user-controlled pacing

  3. Data Consistency: Ensuring accurate extraction of engagement metrics and timestamps

    • Solution: Multiple parsing strategies for number formats (K, M, B) and date handling

  4. Extension Performance: Managing memory usage during large profile scraping

    • Solution: Background processing, chunked data handling, and efficient DOM queries

  5. Cross-browser Compatibility: Ensuring consistent behavior across Chrome versions

    • Solution: Manifest v3 compliance and modern web API usage

Performance Optimizations

  • Debounced scroll event handling to reduce processing overhead
  • Background operation to prevent UI blocking
  • Efficient DOM queries with modern selectors
  • Memory management for large datasets
  • Optimized export processing with streaming for large files

Security Features

  • Local-only data storage with no external transmission
  • No usage data collection or tracking
  • Open-source transparency for security auditing
  • User responsibility for ethical data collection practices
  • Clear privacy policy and license terms

What I Learned

  • Chrome Extension Development: Deep understanding of Manifest v3 architecture and extension lifecycle
  • Modern React Patterns: Advanced hooks usage and component optimization techniques
  • Web Scraping Ethics: Balancing technical capabilities with responsible data collection
  • Cross-platform Development: Building for browser extension environment constraints
  • UI/UX for Developer Tools: Creating intuitive interfaces for technical users

Future Plans

  • Additional social media platform support
  • Advanced data analysis and visualization features
  • API integration for automated workflows
  • Mobile companion app development
  • Enhanced export formats and cloud storage integration
  • Machine learning integration for content categorization

Comments