Guides

Practical, step-by-step walkthroughs for common scraping and automation tasks.

Scraping E-Commerce Product Data

Extract product names, prices, images, and availability from an online store and set up daily monitoring to track changes over time.

Create a new Extract flow — In the dashboard, click New Flow and select the Extract mode.
Enter the store URL — Paste the product listing page URL (e.g. https://store.example.com/products).
Describe what to extract — In the description field, write: "Extract all products with name, price, image URL, and stock status." Scraper's AI generates the extraction rules automatically.
Review the generated extraction rules — Verify the selectors and field mappings in the step editor. Adjust any selectors that target the wrong elements.
Run and verify output — Click Run Now and inspect the extracted JSON in the run results panel.
Set up a schedule for daily price monitoring — Go to the flow's Schedule tab and set a cron expression like 0 8 * * * to run every morning at 8 AM.

Example output:

[
  {
    "name": "Wireless Headphones Pro",
    "price": 79.99,
    "imageUrl": "https://store.example.com/images/headphones-pro.jpg",
    "inStock": true
  },
  {
    "name": "USB-C Charging Cable",
    "price": 12.49,
    "imageUrl": "https://store.example.com/images/usb-c-cable.jpg",
    "inStock": true
  },
  {
    "name": "Laptop Stand Adjustable",
    "price": 34.99,
    "imageUrl": "https://store.example.com/images/laptop-stand.jpg",
    "inStock": false
  }
]

Tips

Pagination — Use a Loop step with a "Next Page" click action to iterate through all result pages automatically.
Dynamic pricing — Some stores render prices via JavaScript. Scraper waits for the page to fully render before extracting, but you can add an explicit Wait step if prices load asynchronously.
Lazy-loaded images — Add a Scroll step before extraction to trigger lazy-loaded images. Set the scroll distance to cover the full product grid.

Monitoring Prices and Getting Alerts

Watch for price drops on specific products and get notified instantly through Slack, Discord, or email when a threshold is crossed.

Create a Monitor flow — Click New Flow and select the Monitor mode. This enables automatic change detection between runs.
Configure extraction for price fields — Add extraction rules targeting the price element. Use the number transform to strip currency symbols and parse the value.
Set a threshold alert — In the Alerts tab, create a threshold rule: "Notify when price drops below $X." Choose the severity level (info, warning, or critical).
Connect a notification channel — Go to Settings > Notifications and add your Slack webhook URL, Discord webhook, or email address.
Set the schedule — Configure the flow to run every 6 hours with the cron expression 0 */6 * * *.

Example alert configuration:

{
  "alerts": [
    {
      "type": "threshold",
      "field": "price",
      "operator": "lt",
      "value": 50,
      "severity": "critical",
      "message": "Price dropped below $50!"
    },
    {
      "type": "change_detected",
      "field": "inStock",
      "severity": "warning",
      "message": "Stock status changed"
    }
  ],
  "notifications": {
    "slack": "https://hooks.slack.com/services/T.../B.../xxx",
    "email": "alerts@yourcompany.com"
  },
  "schedule": "0 */6 * * *"
}

Tips

Avoiding false positives — Use the number transform to normalize prices before comparison. This prevents alerts caused by formatting changes (e.g. "$49.99" vs "49.99 USD").
Meaningful thresholds — Set thresholds based on actual price history rather than arbitrary values. Review the first few runs to establish a baseline before configuring alerts.
Multiple channels — Route critical alerts to Slack for immediate attention and info-level alerts to email for daily digests.

Automating Form Submissions

Automate filling and submitting web forms such as contact forms, applications, or search queries using the Interact flow mode.

Create an Interact flow — Select the Interact mode when creating a new flow. This mode supports browser actions like clicking, typing, and navigating.
Map form fields using selectors — Add Fill steps for each form field. Use CSS selectors like input[name="email"] or #contact-form textarea to target the right elements.
Use variables for dynamic data — Reference variables in your Fill values using double curly braces: {{name}}, {{email}}, {{message}}. Variables are passed in when triggering the run via API.
Add a confirmation extraction step — After submitting, add an Extract step to capture the confirmation message or reference number. This verifies the submission succeeded.
Handle multi-page forms with Condition steps — Use Condition steps to check which page you are on and branch your flow logic accordingly. Add Click and Wait steps between form pages.

Example flow definition:

{
  "name": "Contact Form Submission",
  "mode": "interact",
  "url": "https://example.com/contact",
  "steps": [
    { "type": "fill", "selector": "input[name='name']", "value": "{{name}}" },
    { "type": "fill", "selector": "input[name='email']", "value": "{{email}}" },
    { "type": "fill", "selector": "textarea[name='message']", "value": "{{message}}" },
    { "type": "click", "selector": "button[type='submit']" },
    { "type": "wait", "selector": ".confirmation", "timeout": 5000 },
    {
      "type": "extract",
      "selector": ".confirmation",
      "fields": ["referenceNumber", "statusMessage"]
    }
  ],
  "variables": {
    "name": "Jane Doe",
    "email": "jane@example.com",
    "message": "I'd like to learn more about your services."
  }
}

Tips

CAPTCHAs — Scraper cannot solve CAPTCHAs automatically. For forms protected by CAPTCHAs, consider using the API provider's direct submission endpoint instead, or use a CAPTCHA-solving integration.
File uploads — Use the Upload step type to attach files to file input fields. Provide a URL to the file or a base64-encoded string.
Multi-step forms — For wizard-style forms, chain Fill and Click steps for each page. Add Wait steps between pages to ensure the next form section has loaded before filling it.