Metalink

Last updated

Metalink
Metalink logo.svg
Filename extension
.meta4, .metalink
Internet media type
  • application/metalink4+xml,
  • application/metalink+xml
Type of formatFile distribution
Extended from XML, HTTP
Standard RFC 5854, RFC 6249

Metalink is an extensible metadata file format that describes one or more computer files available for download. It specifies files appropriate for the user's language and operating system; facilitates file verification and recovery from data corruption; and lists alternate download sources (mirror URIs).

Contents

The metadata is encoded in HTTP header fields and/or in an XML file with extension .meta4 or .metalink. The duplicate download locations provide reliability in case one method fails. Some clients also achieve faster download speeds by allowing different chunks/segments of each file to be downloaded from multiple resources at the same time (segmented downloading).

Metalink supports listing multiple partial and full file hashes along with PGP signatures. Most clients only support verifying MD5, SHA-1, and SHA-256, however. Besides FTP and HTTP mirror locations and rsync, it also supports listing the P2P methods BitTorrent, ed2k, magnet link or any other that uses a URI.

Development history

Metalink 3.0 was publicly released in 2005. It was designed to aid in downloading Linux ISO images and other large files on release day, when servers would be overloaded (each server would have to be tried manually) and to repair large downloads by replacing only the parts with errors instead of fully re-downloading them. It was initially adopted by download managers, and was used by open source projects such as OpenOffice.org and Linux distributions. A community developed around it, more download programs supported it (including proprietary ones) and it saw commercial adoption. In 2008, the community took their work to the Internet Engineering Task Force which resulted in Metalink 4.0 in 2010, described in a Standards Track RFC. [1] Metalink 3.0 (with the extension .metalink) and Metalink 4.0 (with the extension .meta4) are incompatible because they have a slightly different format. In 2011, another Standards Track RFC described Metalink in HTTP header fields. [2]

Client programs

ToolDebian-basedFedora-basedArch-basedBSD-basedWindows
aria2 [3] YesYesYes [4] Yes [5]
cURL No; built without Metalink supportNo; built without Metalink support
Wget Yes; v1.17 or later built with metalink supportYes; v1.17 or later built with metalink supportYes; v1.17 or later built with metalink supportYes; v1.17 or later built with metalink support
DownThemAll not in reponot in reponot in reponot in repo
FlashGot not in reponot in reponot in reponot in repo
Free Download Manager not in reponot in reponot in reponot in repo
GetRight not in reponot in reponot in reponot in repo
jDownloader not in reponot in reponot in reponot in repo
KGet YesYesYesYesNo
Phex not in reponot in reponot in reponot in repo
SmartFTP not in reponot in reponot in reponot in repo
easyMule not in reponot in reponot in reponot in repo
ZYpp not in reponot in reponot in reponot in repo

Client libraries

In use

Mandriva Linux has integrated Metalink into package management with urpmi and aria2. Fedora has integrated Metalink into package management with yum. openSUSE has integrated Metalink into package management with ZYpp and aria2.

Wubi, the Windows-based Ubuntu installer, uses Metadl (LGPL) to download Ubuntu ISO images and takes advantage of Metalink's features of higher availability and increased reliability. If there are errors in the download, they are repaired, instead of restarting the large download.

Appupdater (GPL) for Windows "is similar to apt-get or yum on Linux. It automates the process of installing and maintaining up to date versions of programs."[ citation needed ]

Currently, OpenOffice.org uses Metalinks to distribute their free office suite. cURL offers Metalinks of their downloads. UniProt Consortium also provides a Metalink for their large protein database. Dofus, a Flash MMORPG uses Metalinks for downloads and so does SageMath, the open-source mathematical software.

According to the main metalinker.org site, the software hosting and collaboration platform Origo generates Metalinks for all hosted releases. [6]

The following Linux distributions use Metalink for ISO image distribution:[ citation needed ] Adios, Bayanihan Linux, BeleniX, Berry Linux, BLAG Linux and GNU, Bluewhite64 Linux, Damn Small Linux, Fedora, GoboLinux, Granular Linux, KateOS, Linux Mint, openSUSE, Pardus Linux, PCLinuxOS, PuppyLinux, Sabayon Linux, StartCom Enterprise Linux, SUSE Linux, Ubuntu.

The following FreeBSD based distributions use Metalink for ISO image distribution: DesktopBSD, MidnightBSD, and TrueOS.

A guide to which clients support which Metalink features:

Client wget cURL DownThemAll Free Download Manager GetRight KGet Phex SmartFTP
Licensing Free Software (GNU General Public License) Free Software (MIT/X derivate license) Free Software (GNU General Public License) Free Software (GNU General Public License)Proprietary / $29.95 for Standard, $49.95 for Pro Free Software (GNU General Public License) Free Software (GNU General Public License)Proprietary / Free 30 day Evaluation

Multiple Editions/$39.99 Professional

InterfaceCLICLIGUIGUIGUIGUIGUIGUI
Metalink 4.0YesYesYesNoNoYesNoYes
Runs on Linux YesYesYesNoNoYesYesNo
Runs on Windows YesYesYesYesYesNoYesYes
Runs on Mac OS X YesYesYesNoNoNoYesNo
Resume Downloads ?YesYesYesYesYesYesYes [7]
Segmented Downloads? ?YesYesYesYesYesNo
Multiple File support? ?YesYesYesYesYesYes
"location"? ?YesNoNoNoNoNo
OS? ?YesYesNoNoNoNo
Language? ?YesNoNoNoNoNo
HTTP with Transparent Negotiation? ?YesNoNoNoNoNo
Metalink/HTTP? ?NoNoNoNoNoNo
HTTPYesYesYesYesYesYesYesYes
FTPYesYesYesYesYesYesYesYes
BitTorrent ?NoYesYesYesNoNo
magnet? ?NoNoNoNoYesNo
MD5 Hashes ?YesYesYesYesYesYes
SHA-1 Hashes ?YesYesYesYesYesYes
SHA-256 Hashes ?YesYesNoNoNoYes
Piece Hashes? ?NoNoNoNoNoNo
PGP Signatures? ?NoNoNoNoNoNo

No clients support rsync at this time.

Metalink is formatted as XML so it can be either hand-crafted or created with an XML library, useful for automation. These tools are specifically for Metalink.

Metalink 4.0 files have the extension .meta4 and are XML text files. They are served with the application/metalink4+xml Internet media type.

<?xml version="1.0" encoding="UTF-8"?><metalinkxmlns="urn:ietf:params:xml:ns:metalink"><published>2009-05-15T12:23:23Z</published><filename="example.ext"><size>14471447</size><identity>Example</identity><version>1.0</version><language>en</language><description>Adescriptionoftheexamplefilefordownload. </description><hashtype="sha-256">3d6fece8033d146d8611eab4f032df738c8c1283620fd02a1f2bfec6e27d590d</hash><urllocation="de"priority="1">ftp://ftp.example.com/example.ext</url><urllocation="fr"priority="1">http://example.com/example.ext</url><metaurlmediatype="torrent"priority="2">http://example.com/example.ext.torrent</metaurl></file></metalink>

Example Metalink/HTTP header fields

Metalink in HTTP header fields makes use of existing standard HTTP header fields such as ETags, Link header fields (for mirrors and P2P), and Instance Digests (for hashes).

Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5=" Link: <http://www2.example.com/example.ext>; rel=duplicate Link: <ftp://ftp.example.com/example.ext>; rel=duplicate Link: <http://example.com/example.ext.torrent>; rel=describedby; type="application/x-bittorrent" Link: <http://example.com/example.ext.meta4>; rel=describedby; type="application/metalink4+xml" Link: <http://example.com/example.ext.asc>; rel=describedby; type="application/pgp-signature" Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlODYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ== 

Metalink 3.0 files have the extension .metalink and are XML text files.

<?xml version="1.0" encoding="UTF-8"?><metalinkversion="3.0"xmlns="http://www.metalinker.org/"><files><filename="example.ext"><verification><hashtype="md5">example-md5-hash</hash><hashtype="sha1">example-sha1-hash</hash><signaturetype="pgp"/></verification><resources><urltype="ftp"location="us"preference="90">ftp://ftp.example.com/example.ext</url><urltype="ftp"location="uk"preference="90">ftp://ftp.example.net/example.ext</url><urltype="http"location="us"preference="90">http://example.com/example.ext</url><urltype="http"location="de"preference="90">http://example.net/example.ext</url><urltype="bittorrent"preference="100">http://example.org/example.ext.torrent</url><urltype="rsync"/><urltype="magnet"/><urltype="ed2k"/></resources></file></files></metalink>

See also

Related Research Articles

<span class="mw-page-title-main">Konqueror</span> Web browser and file manager

Konqueror is a free and open-source web browser and file manager that provides web access and file-viewer functionality for file systems. It forms a core part of the KDE Software Compilation. Developed by volunteers, Konqueror can run on most Unix-like operating systems. The KDE community licenses and distributes Konqueror under GNU GPL-2.0-or-later.

<span class="mw-page-title-main">Shareaza</span> Peer-to-peer file sharing application

Shareaza is a peer-to-peer file sharing client running under Microsoft Windows which supports the gnutella, Gnutella2 (G2), eDonkey, BitTorrent, FTP, HTTP and HTTPS network protocols and handles magnet links, ed2k links, and the now deprecated gnutella and Piolet links. It is available in 30 languages.

eDonkey2000

eDonkey2000 was (is) a peer-to-peer file sharing application developed by US company MetaMachine, using the Multisource File Transfer Protocol. It supported both the eDonkey2000 network and the Overnet network.

BitTorrent, also referred to as simply torrent, is a communication protocol for peer-to-peer file sharing (P2P), which enables users to distribute data and electronic files over the Internet in a decentralized manner. The protocol is developed and maintained by Rainberry, Inc., and was first released in 2001.

<span class="mw-page-title-main">Wget</span> Computer command line program for downloading

GNU Wget is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from "World Wide Web" and "get". It supports downloading via HTTP, HTTPS, and FTP.

cURL is a computer software project providing a library (libcurl) and command-line tool (curl) for transferring data using various network protocols. The name stands for "Client for URL".

<span class="mw-page-title-main">BitComet</span> BitTorrent, FTP and HTTP client

BitComet is a cross-protocol BitTorrent, HTTP and FTP client written in C++ for Microsoft Windows and available in 52 different languages. Its first public release was version 0.28. The current BitComet logo has been used since version 0.50.

<span class="mw-page-title-main">FileZilla</span> Free software, cross-platform file transfer protocol application

FileZilla is a free and open-source, cross-platform FTP application, consisting of FileZilla Client and FileZilla Server. Clients are available for Windows, Linux, and macOS. Both server and client support FTP and FTPS, while the client can in addition connect to SFTP servers. FileZilla's source code is hosted on SourceForge.

<span class="mw-page-title-main">Magnet URI scheme</span> Scheme that defines the format of magnet links

Magnet is a URI scheme that defines the format of magnet links, a de facto standard for identifying files (URN) by their content, via cryptographic hash value rather than by their location.

In computer hypertext, a URI fragment is a string of characters that refers to a resource that is subordinate to another, primary resource. The primary resource is identified by a Uniform Resource Identifier (URI), and the fragment identifier points to the subordinate resource.

<span class="mw-page-title-main">Gajim</span> Free instant messaging client for the XMPP protocol

Gajim is an instant messaging client for the XMPP protocol which uses the GTK toolkit. The name Gajim is a recursive acronym for Gajim's a jabber instant messenger. Gajim runs on Linux, BSD, macOS, and Microsoft Windows. Released under the GPL-3.0-only license, Gajim is free software. A 2009 round-up of similar software on Tom's Hardware found version 0.12.1 "the lightest and fastest jabber IM client".

BitTorrent is a proprietary adware BitTorrent client developed by Bram Cohen and Rainberry, Inc. used for uploading and downloading files via the BitTorrent protocol. BitTorrent was the first client written for the protocol. It is often nicknamed Mainline by developers denoting its official origins. Since version 6.0 the BitTorrent client has been a rebranded version of μTorrent. As a result, it is no longer open source. It is currently available for Microsoft Windows, Mac, Linux, iOS and Android. There are currently two versions of the software, "BitTorrent Classic" which inherits the historical version numbering, and "BitTorrent Web", which uses its own version numbering.

In computing, Java Web Start is a deprecated framework developed by Sun Microsystems that allows users to start application software for the Java Platform directly from the Internet using a web browser. The technology enables seamless version updating for globally distributed applications and greater control of memory allocation to the Java virtual machine.

Free Download Manager is a download manager for Windows, macOS, Linux and Android.

The following is a general comparison of BitTorrent clients, which are computer programs designed for peer-to-peer file sharing using the BitTorrent protocol.

CrushFTP is a proprietary multi-protocol, multi-platform file transfer server originally developed in 1999. CrushFTP is shareware with a tiered pricing model. It is targeted at home users on up to enterprise users.

curl-loader is an open-source software performance testing tool written in the C programming language.

<span class="mw-page-title-main">Lphant</span> Peer-to-peer sharing software

Lphant was a peer-to-peer file sharing client for the Microsoft Windows, Linux and Mac OS operating systems, which supports the eDonkey Network and the BitTorrent protocol. It was available in 19 languages. The name and logo of the original Lphant application has been replicated in a program called "Lphant 6.0".

In the BitTorrent file distribution system, a torrent file or meta-info file is a computer file that contains metadata about files and folders to be distributed, and usually also a list of the network locations of trackers, which are computers that help participants in the system find each other and form efficient distribution groups called swarms. Torrent files are normally named with the extension .torrent.

Mirror sites or mirrors are replicas of other websites. The concept of mirroring applies to network services accessible through any protocol, such as HTTP or FTP. Such sites have different URLs than the original site, but host identical or near-identical content. Mirror sites are often located in a different geographic region than the original, or upstream site. The purpose of mirrors is to reduce network traffic, improve access speed, ensure availability of the original site for technical or political reasons, or provide a real-time backup of the original site. Mirror sites are particularly important in developing countries, where internet access may be slower or less reliable.

References

  1. Bryan, Anthony; Tsujikawa, Tatsuhiro; McNab, Neil; Poeml, Peter (June 2010). The Metalink Download Description Format. IETF. doi: 10.17487/RFC5854 . RFC 5854 . Retrieved 2022-05-30.
  2. Bryan, Anthony; McNab, Neil; Tsujikawa, Tatsuhiro; Poeml, Peter; Nordstrom, Henrik (June 2011). Metalink/HTTP: Mirrors and Hashes. IETF. doi: 10.17487/RFC6249 . RFC 6249 . Retrieved 2022-05-30.
  3. Tsujikawa, Tatsuhiro. "aria2: The next generation download utility". GitHub . Archived from the original on 2021-07-27. Retrieved 2022-05-30.
  4. "Package Search". Arch Linux . 2021-08-21. Archived from the original on 2022-05-30. Retrieved 2022-05-30.
  5. "Aria2 Client 1.36.0". Chocolatey . 2021-08-21. Archived from the original on 2022-05-30. Retrieved 2022-05-30.
  6. https://www.metalinker.org/implementation.html
  7. "SmartFTP - Features". SmartFTP. Archived from the original on 2022-05-30. Retrieved 2022-05-30.