This evening I came across a Google Search result for Game of Thrones Audiobooks and my immediate question was how can I download all these .mp3 files at once to my computer. As going through each of those directory and clicking on those files to download was, well a bit too boring and time consuming.

1. Wget Download All Files In Directory Listing

Download a List of Files at Once If you can’t find an entire folder of the downloads you want, wget can still help. Just put all of the download URLs into a single TXT file. Then point wget to that document with the -i option. Go ahead and copy WGET.exe to the System32 directory and restart your Command Prompt. Restart command terminal and test WGET. If you want to test WGET is working properly, restart your terminal and type: wget -h. If you’ve copied the file to the right place, you’ll see a help file appear with all of the available commands.

So I gave a thought to it and tried to play with the very famous shell command wget and that’s it! After few trials it gave me what I needed.

Explanation with each options

• wget: Simple Command to make CURL request and download remote files to our local machine.
• --execute='robots = off': This will ignore robots.txt file while crawling through pages. It is helpful if you’re not getting all of the files.
• --mirror: This option will basically mirror the directory structure for the given URL. It’s a shortcut for -N -r -l inf --no-remove-listing which means:
  • -N: don’t re-retrieve files unless newer than local
  • -r: specify recursive download
  • -l inf: maximum recursion depth (inf or 0 for infinite)
  • --no-remove-listing: don’t remove ‘.listing’ files
• --convert-links: make links in downloaded HTML or CSS point to local files
• --no-parent: don’t ascend to the parent directory
• --wait=5: wait 5 seconds between retrievals. So that we don’t thrash the server.
• <website-url>: This is the website url from where to download the files.

Wget Download All Files In Directory Listing

Happy Downloading