Metalinks for Download, Anyone?

Downloading large files from a server with http/ftp is becoming obsolete now. People are more inclined towards using torrent downloads because of the obvious advantages over server downloads such as speed limits, connection timeout errors, etc.

Torrents, however, are not so useful when downloading something new like, say, a new Linux distribution release. The seed:peer ratio is almost zero. The servers are overloaded. You neither get much speed from any particular server nor from torrents. How many times have you wished that you could supply more than a single server to your download manager to increase the speed a bit? Maybe mix torrents with the server downloads for even higher speeds? This is where Metalink steps in. Anthony Bryan came up with Metalink 11 years ago, but it still has not received the limelight it deserves.

In the simplest words, a Metalink is an XML file. This file contains the links to the various mirrors that host the file under consideration. It might also contain the location of the torrent of that file and/or magnet links. It also has an added advantage when used with some clients. The file can be simultaneously downloaded from various locations in the form of segments (like hash pieces in a torrent, but not the same). It also features automatic checksum verification of the downloaded data and, if any errors are found, there’s an automatic error correction feature also. Metalink supports both MD5SUM and SHA1SUM checksums as well as PGP signatures, which are embedded in the .metalink file itself.

A checksum is like a signature or a fingerprint of the file. If the downloaded file has no error then the checksum in the Metalink file will match the checksum of the downloaded file. In case the matching fails, the Metalink download client only downloads the segments with errors using different tools like rsync. For some clients, you may have to manually download the missing/corrupt data using tools such as rsync.

Creating Metalinks

Most distributions these days also provide a Metalink to the ISO files. If you do not have the Metalink or cannot find one, you can create it yourself. There are many Metalink generators available online or you can install one if you want. They generate the Metalink XML file. It contains XML data with a .metalink file extension.

The online Metalink generator at the official site of Metalink is good enough for generation and can be found at www.metalinker.org/generator. If you would like to install a generator or use local metalink generator scripts, then you can find those at www.metalinker.org/implementation.html.

For the more geeky folk, or those who want to have complete control over the Metalink file (there is not much to have control over, but just in case), the general Metalink XML format is given below:

<?xml version="1.0" encoding="UTF-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org">
   <files>
      <file name="example.ext">
         <verification>
            <hash type="md5">example-md5-hash</hash>
            <hash type="sha1">example-sha1-hash</hash>
            <signature type="pgp"/>
         </verification>
         <resources>
            <url type="ftp" location="us" preference="90">ftp://ftp.example.com/example.ext</url>
            <url type="ftp" location="uk" preference="75">ftp://ftp.example.net/example.ext</url>
            <url type="http" location="us" preference="80">http://example.com/example.ext</url>
            <url type="http" location="de" preference="10">http://example.net/example.ext</url>
            <url type="bittorrent" preference="100">http://example.org/example.ext.torrent</url>
            <url type="rsync"/>
            <url type="magnet"/>
            <url type="ed2k"/>
         </resources>
      </file>
   </files>
</metalink>

This is just the general format. You can also add additional fields like OS (OS specific files…the download manager will check your OS and automatically download the file tagged with your OS). The preference parameter has a value from 1 to 100. If a segment is available at multiple locations, then the segment will be downloaded from the location with the maximum priority. There is a provision to add publisher information like name, website, etc.

Downloading files using Metalinks

You cannot use your regular download clients to download files using Metalinks unless your client supports the Metalink format. If you are using Linux, then I would suggest using the aria2 download client. It has a command-line interface and comes packed with many features. (Almost all features supported by Metalink are also supported by aria2.) The DownThemAll Firefox extension can also handle Metalinks and is my personal favourite. If you would like to try a different client, the list of all clients supporting Metalink can be found at www.metalinker.org/implementation.html.

For the win formula

Metalink is a cool feature for downloading large files. The ability to avoid connection timeout errors and the package integrity check is something that gives Metalink some added bonus points. Plus, by grabbing chunks of files from multiple sources at once, it makes efficient use of available bandwidth.

Metalink can be better, of course. It still lacks some important features such as provision of different versions of the same file and an option to select one you want. One more feature lacking is dynamic preference value generation, depending on the load on the servers in the Metalink file. The server with the least load must be given first preference.

The advantages still outweigh the disadvantages, by far. Metalink is, as I said, a cool feature. I am amazed how I failed to notice it for so long.