Insta Swiper

clgp

2026/02/14

Categories: Tags:

Building IG Swiper: An Instagram Photo Gallery Scraper

🎯 What is IG Swiper?

IG Swiper is a full-stack web application designed to crawl and display Instagram user photos in an elegant, swipeable gallery format. Think of it as your personal Instagram photo archive manager - it downloads photos from public Instagram profiles and presents them in a beautiful, user-friendly interface where you can browse, organize, and enjoy the content.

💡 The Purpose & Motivation

Instagram is a wonderful platform for sharing visual content, but it has limitations:

🏗️ Architecture & Technology Stack

The project is built as a modern full-stack application with clear separation of concerns:

Backend (Node.js + Express)

Frontend (React + Vite)

Deployment & Process Management

🛠️ The Development Journey

Phase 1: Core Scraping Engine

The first challenge was understanding Instagram’s API structure. Instagram doesn’t provide an official public API for downloading photos, so I had to reverse-engineer their web interface:

// The core scraping approach uses authenticated requests
async function instagramRequest(url, options = {}) {
  const cookieHeader = getCookieHeader();
  const csrfToken = getCsrfToken();

  const headers = {
    "User-Agent": "Mozilla/5.0 ...",
    Cookie: cookieHeader,
    "X-CSRFToken": csrfToken,
    "X-IG-App-ID": "936619743392459",
    // ... other headers
  };

  return fetch(url, { ...options, headers });
}

Key learnings:

Phase 2: Handling Different Post Types

Instagram has various content types that needed special handling:

  1. Single image posts - Straightforward image downloads
  2. Carousel posts - Multiple images in one post (handled via carousel_media array)
  3. Videos - Skipped during image crawling (media_type === 2)
  4. High-resolution images - Accessed via image_versions2.candidates[0].url
// Extract images from carousel posts
if (item.carousel_media) {
  for (const carouselItem of item.carousel_media) {
    if (carouselItem.media_type === 2) continue; // Skip videos

    const url = carouselItem.image_versions2?.candidates?.[0]?.url;
    if (url) {
      allImages.push({ id: carouselItem.id, display_url: url });
    }
  }
}

Phase 3: Database Design

I chose Better-SQLite3 for several reasons:

The database schema tracks:

Phase 4: Crawl State Management

One of the trickiest parts was implementing a robust crawl control system:

let crawlState = {
  isActive: false,
  shouldStop: false,
  currentUser: null,
  imagesFound: 0,
};

export function stopCrawl() {
  crawlState.shouldStop = true;
  const stoppedCount = stopAllCrawling(); // Update DB
  return { success: true, message: `${stoppedCount} user(s) stopped` };
}

Challenges solved:

Phase 5: Frontend Development

The frontend needed to be both functional and beautiful:

Phase 6: Deployment Strategy

Deploying locally with public access required:

  1. PM2 Configuration:
module.exports = {
  apps: [
    {
      name: "ig-swiper",
      script: "./backend/server.js",
      env: {
        NODE_ENV: "production",
        PORT: 3002,
      },
    },
  ],
};
  1. Cloudflare Tunnel: Secure HTTPS access without port forwarding
  2. Environment variables: Separate dev/prod configurations
  3. Build optimization: Production build with Vite

🚀 Features & Capabilities

Current Features

✅ Batch downloading - Crawl up to 500 posts per user
✅ Carousel support - Extract all images from multi-image posts
✅ Rate limiting - Intelligent delays to avoid Instagram blocks
✅ Status tracking - Real-time crawl progress updates
✅ Error handling - Graceful failure recovery
✅ Duplicate prevention - Skip already-downloaded images
✅ Cookie-based auth - Use your Instagram session for access
✅ Production deployment - PM2 + Cloudflare Tunnel ready

Future Enhancements

🔜 User authentication - Multi-user support with role-based access
🔜 Search & filtering - Find images by caption or date
🔜 Collections - Organize images into custom albums
🔜 Export options - Bulk download as ZIP files
🔜 Video support - Download and play videos
🔜 Privacy controls - Public/private galleries

📊 Technical Challenges & Solutions

Challenge 1: Instagram Rate Limiting

Problem: Instagram blocks aggressive scraping
Solution: Implemented 1.5s delays, max 500 posts limit, and exponential backoff on errors

Problem: Instagram sessions expire, breaking the scraper
Solution: Cookie validation checks and user-friendly error messages to refresh cookies

Challenge 3: Image Storage

Problem: Where to store potentially thousands of images
Solution: Local filesystem with UUID filenames, configurable via IMAGES_DIR env variable Problem after problem: Local machine still the main storage

Challenge 4: Concurrent Crawls

Problem: Multiple users crawling simultaneously could cause conflicts
Solution: Global crawl state with stop flags and database-level status tracking

Challenge 5: Production Deployment

Problem: Running locally but accessible publicly without exposing home network
Solution: Cloudflare Tunnel for secure HTTPS access + PM2 for process stability

🎓 Key Learnings

  1. Reverse engineering APIs - Instagram’s web API isn’t documented, but network inspection reveals patterns
  2. Rate limiting importance - Respectful scraping prevents IP bans
  3. State management complexity - Coordinating async operations requires careful planning
  4. Database choice matters - SQLite is perfect for single-user, local-first applications
  5. Deployment isn’t just hosting - Process management, monitoring, and tunneling are crucial

🔒 Ethical Considerations

IG Swiper is designed for personal use only:

🎯 Conclusion

Building IG Swiper was an educational journey through full-stack development, API reverse engineering, and production deployment. The project demonstrates:

📦 Project Resources

Built with ❤️ by a developer who loves beautiful photo galleries and offline-first applications.

>> Home

Comments