Bulk File Downloader and Image Downloader: Web Pages, PDFs

What you get

Every result is one row in the run's dataset. Download it as JSON, CSV, Excel or HTML, or read it through the Apify API.

The main fields of each row:

• File (fileUrl)
• Name (fileName)
• Type (fileType)
• Bytes (fileBytes)
• Width (imageWidth)
• Height (imageHeight)
• Alt text (altText)
• Source link (url)
• error

What people use it for

• Product images: every photo of a list of product pages in the largest size offered, with alt text and size.
• Document collection: the PDF, Word, Excel and CSV files linked from report, tender or dataset pages.
• Site migration and backup: the images, video and files of the pages you move, with a checksum for each.
• Image datasets: pictures with their alt text, width and height, ready to label, train or index.
• Broken media checks: every image and file a page uses that answers with an error, listed with the reason.

Run it on Apify

1. Open bulk-file-downloader on Apify and sign in, or make an Apify account.
2. On the Input tab, fill in the fields listed below. The form opens with example values filled in.
3. Click Start. Rows appear on the Output tab while the run works.
4. Export the rows as JSON, CSV, Excel, XML or HTML, or read them through the Apify API.
5. To run it again on a schedule, save the input as a task and add a schedule in the Apify Console.

What you can set

• File links (fileUrls): Direct links to files of any type (images, video, audio, PDF, documents, archives). Each one is downloaded whatever its type. A link that turns out to be a web page comes back as a row that says so: put it under Pages instead.
• Pages (pageUrls): Web pages to download files from: the images it shows (the largest size it offers), its video and audio, and the files it links to, filtered by File types. Pages are read as HTML without a browser, so files a page loads with JavaScript only are not found.
• File types (fileTypes): Which files to take from Pages. Direct file links are always downloaded. Options: Images, Videos, Audio, PDF files, Documents (Word, Excel, PowerPoint, OpenDocument, text, CSV, EPUB), Archives (ZIP, RAR, 7z, TAR, GZ). Default: ["image"].
• Only files from the page's own site (sameSiteOnly): Take only files on the host of the page or its subdomains, leaving out files on other hosts (many sites keep their images on a separate CDN host, so this can drop them). Default: false.
• Maximum files per page (maxFilesPerPage): Caps the files taken from one page, in the order they appear on it. Default: 200.
• Skip images under (KB) (minImageKB): Images found on pages that are smaller than this are left out (icons, spacers, tracking pixels) and not charged; 0 keeps every image. Direct links are always kept. Default: 5.
• Maximum file size (MB) (maxFileMB): Larger files come back as a row with the reason, at no charge. Above 100 MB give the run more memory: 1 GB for every 150 MB of this limit (a few files can be in memory at once). Default: 100.
• Maximum results (maxItems): The run stops after this many result rows. It also caps what one run can cost. Default: 1000.
• Proxy configuration (proxyConfiguration): Leave as is: the run starts on the cheapest network the site accepts and switches to Apify Proxy automatically when the site blocks it. Pick proxy groups here only to force one specific network.

Example input

This is the input the actor is tested with every day.

{
  "pageUrls": [
    "https://en.wikipedia.org/wiki/Hummingbird"
  ],
  "fileTypes": [
    "image"
  ],
  "maxFilesPerPage": 20,
  "maxItems": 20
}

Run it from Python

The official apify-client package starts the run, waits for it and reads the rows. It needs your own Apify API token.

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")  # your own API token, from the Apify Console
run_input = {
    "pageUrls": ["https://en.wikipedia.org/wiki/Hummingbird"],
    "fileTypes": ["image"],
    "maxFilesPerPage": 20,
    "maxItems": 20,
}
run = client.actor("v0iddo/bulk-file-downloader").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Limits

• One row per file: the stored copy (fileUrl), the link it came from, the page it was found on and where on the page (img, srcset, link, video, og:image ...), its name, type, content type, size, SHA-256 checksum, and for images the width, height and alt text.
• The files are records in the run's key-value store; fileUrl opens them directly and works in the dataset table, in spreadsheets and in your own code. They are kept as long as your Apify plan keeps the run's storage.
• Images on pages come in the largest size the page offers in srcset, and lazy loading attributes (data-src and the like) are read, so the real image is taken instead of a placeholder.
• The same file under two links is stored once: the second row has duplicateOf and costs nothing. A link that appears on several pages is downloaded once.
• Files are read only from addresses whose robots.txt allows it, and not from hosts that close their content to machine learning agents in robots.txt. Such a file comes back as a row with the reason, at no charge. Only download files you have the right to copy.
• Pages are read without a browser: files a page shows only after running JavaScript (most social networks, many shops) are not found. Links to their files can still be given under File links.
• Only publicly visible data is collected. No login, no paywalled content, no personal contact details.
• The actor reads only pages that the site's robots.txt leaves open to automated access.
• Sites change their pages from time to time. The actor is checked automatically and repaired when a check fails.
• Each run sends one short usage record to the developer: the Actor and run IDs, the number of results, whether the run succeeded (and if not, the kind of error, such as "blocked" or "input problem"), whether the account is on a paid Apify plan, how the run was started (for example from the Console, the API or a schedule) and a one-way fingerprint of the account (a hash, not the account ID), so that repeat runs of one account count once. It contains nothing from your input or results.

Questions

Do I need an account on the site, cookies or an API key?

No. The actor reads only what every visitor of the site can see, without logging in. You only need an Apify account to start it.

How many results can one run return?

Up to 50,000 rows per run. Without a limit it stops at 1,000. Set Maximum results to the number you need; large jobs can also be split into several runs or scheduled.

Why did my run return no results?

The status message of the run says why: nothing matched the input, the input was not a page the actor understands, the page is closed to automated access by robots.txt, or the site refused every network the actor tried. Check the input against the example above and run it again with a small limit.

Can I run it on a schedule?

Yes. Save your input as a task and add a schedule in the Apify Console (daily, hourly or with a cron expression). Each run gets its own dataset, so you keep a history.

Which formats can I export?

JSON, CSV, Excel, XML, HTML table and RSS from the Output tab or the API. Integrations send the rows to Google Sheets, Slack, Make, Zapier, n8n or a webhook.

More

• A post that uses it: Facebook Ad Library API: which ads you get, which you don't
• Domain Availability Checker: Bulk RDAP
• All our scrapers on the Apify Store