Type of site
|Area served||Worldwide (except China and Bahrain)|
|Launched||May 12, 1996 (private)|
October 24, 2001 (public)
|Written in||Java, Python|
The Wayback Machine is a digital archive of the World Wide Web. It was founded by the Internet Archive, a nonprofit library based in San Francisco, California. Created in 1996 and launched to the public in 2001, it allows the user to go "back in time" and see how websites looked in the past. Its founders, Brewster Kahle and Bruce Gilliat, developed the Wayback Machine to provide "universal access to all knowledge" by preserving archived copies of defunct web pages.
Since its creation in 1996, over 613 billion pages have been added to the archive. The service has also sparked controversy over whether creating archived pages without the owner's permission constitutes copyright infringement in certain jurisdictions.
The Wayback Machine began archiving cached web pages on May 12, 1996, [ failed verification ]with the goal of making the service public five years later.
Internet Archive founders Brewster Kahle and Bruce Gilliat launched the Wayback Machine in San Francisco, California,in October 2001, primarily to address the problem of website content vanishing whenever it gets changed or when a website is shut down. The service enables users to see archived versions of web pages across time, which the archive calls a "three-dimensional index". Kahle and Gilliat created the machine hoping to archive the entire Internet and provide "universal access to all knowledge". The name "Wayback Machine" is a reference to a fictional time-traveling and translation device, the "Wayback Machine", used by the characters Mister Peabody and Sherman in the animated cartoon The Adventures of Rocky and Bullwinkle and Friends . In one of the cartoon's segments, "Peabody's Improbable History", the characters used the machine to witness, participate in, and often alter famous events in history.
From 1996 to 2001, the information was kept on digital tape, with Kahle occasionally allowing researchers and scientists to tap into the "clunky" database.When the archive reached its fifth anniversary in 2001, it was unveiled and opened to the public in a ceremony at the University of California, Berkeley. By the time the Wayback Machine launched, it already contained over 10 billion archived pages. The data is stored on the Internet Archive's large cluster of Linux nodes. It revisits and archives new versions of websites on occasion (see technical details below). Sites can also be captured manually by entering a website's URL into the search box, provided that the website allows the Wayback Machine to "crawl" it and save the data.
On October 30, 2020, the Wayback Machine began fact-checking content.
Software has been developed to "crawl" the Web and download all publicly accessible information and data files on webpages, the Gopher hierarchy, the Netnews (Usenet) bulletin board system, and downloadable software.The information collected by these "crawlers" does not include all the information available on the Internet, since much of the data is restricted by the publisher or stored in databases that are not accessible. To overcome inconsistencies in partially cached websites, Archive-It.org was developed in 2005 by the Internet Archive as a means of allowing institutions and content creators to voluntarily harvest and preserve collections of digital content, and create digital archives.
Crawls are contributed from various sources, some imported from third parties and others generated internally by the Archive.For example, crawls are contributed by the Sloan Foundation and Alexa, crawls run by IA on behalf of NARA and the Internet Memory Foundation, mirrors of Common Crawl. The "Worldwide Web Crawls" have been running since 2010 and capture the global Web.
Documents and resources are stored with time stamp URLs such as
The frequency of snapshot captures varies per website.Websites in the "Worldwide Web Crawls" are included in a "crawl list", with the site archived once per crawl. A crawl can take months or even years to complete, depending on size. For example, "Wide Crawl Number 13" started on January 9, 2015, and completed on July 11, 2016. However, there may be multiple crawls ongoing at any one time, and a site might be included in more than one crawl list, so how often a site is crawled varies widely.
As of October 2019, users are limited to 5 archival requests and retrievals per minute.[ citation needed ][ why? ]
As technology has developed over the years, the storage capacity of the Wayback Machine has grown. In 2003, after only two years of public access, the Wayback Machine was growing at a rate of 12 terabytes/month. The data is stored on PetaBox rack systems custom designed by Internet Archive staff. The first 100TB rack became fully operational in June 2004, although it soon became clear that they would need much more storage than that.
The Internet Archive migrated its customized storage architecture to Sun Open Storage in 2009, and hosts a new data centre in a Sun Modular Datacenter on Sun Microsystems' California campus. As of 2009 [update] , the Wayback Machine contained approximately three petabytes of data and was growing at a rate of 100 terabytes each month.
A new, improved version of the Wayback Machine, with an updated interface and a fresher index of archived content, was made available for public testing in 2011, where captures appear in a calendar layout with circles whose width visualizes the number of crawls each day, but no marking of duplicates with asterisks or an advanced search page.A top toolbar has been added to facilitate navigating between captures. A bar chart visualizes the frequency of captures per month over the years. Features like "Changes", "Summary", and a graphical site map were added subsequently.
In March that year, it was said on the Wayback Machine forum that "the Beta of the new Wayback Machine has a more complete and up-to-date index of all crawled materials into 2010, and will continue to be updated regularly. The index driving the classic Wayback Machine only has a little bit of material past 2008, and no further index updates are planned, as it will be phased out this year."Also in 2011, the Internet Archive installed their sixth pair of PetaBox racks which increased the Wayback Machine's storage capacity by 700 terabytes.
In January 2013, the company announced a ground-breaking milestone of 240 billion URLs.
In October 2013, the company introduced the "Save a Page" featurewhich allows any Internet user to archive the contents of a URL, and quickly generates a permanent link unlike the preceding liveweb feature.
In December 2014, the Wayback Machine contained 435 billion web pages—almost nine petabytes of data, and was growing at about 20 terabytes a week.
In March 2015,[ date verification needed ] security researchers became aware of the threat posed by the service's unintentional hosting of malicious binaries from archived sites.
In July 2016, the Wayback Machine reportedly contained around 15 petabytes of data.
In September 2018, the Wayback Machine contained over 25 petabytes of data.
As of December 2020, the Wayback Machine contained over 70 petabytes of data.
|Wayback Machine by Year||Pages Archived (billion)|
Between October 2013 and March 2015, the website's global Alexa rank changed from 163to 208. In March 2019 the rank was at 244.
Historically, the Wayback Machine has respected the robots exclusion standard (robots.txt) in determining if a website would be crawled – or if already crawled, if its archives would be publicly viewable. Website owners had the option to opt-out of Wayback Machine through the use of robots.txt. It applied robots.txt rules retroactively; if a site blocked the Internet Archive, any previously archived pages from the domain were immediately rendered unavailable as well. In addition, the Internet Archive stated that "Sometimes, a website owner will contact us directly and ask us to stop crawling or archiving a site. We comply with these requests."In addition, the website says: "The Internet Archive is not interested in preserving or offering access to Web sites or other internet documents of persons who do not want their materials in the collection."
On 17 April 2017, reports surfaced of sites that had gone defunct and became parked domains that were using robots.txt to exclude themselves from search engines, resulting in them being inadvertently excluded from the Wayback Machine.The Internet Archive changed the policy to now require an explicit exclusion request to remove it from the Wayback Machine.
Wayback's retroactive exclusion policy is based in part upon Recommendations for Managing Removal Requests and Preserving Archival Integrity published by the School of Information Management and Systems at University of California, Berkeley in 2002, which gives a website owner the right to block access to the site's archives.Wayback has complied with this policy to help avoid expensive litigation.
The Wayback retroactive exclusion policy began to relax in 2017, when it stopped honoring robots.txt on U.S. government and military web sites for both crawling and displaying web pages. As of April 2017, Wayback is ignoring robots.txt more broadly, not just for U.S. government websites.
From its public launch in 2001, the Wayback Machine has been studied by scholars both for the ways it stores and collects data as well as for the actual pages contained in its archive. As of 2013, scholars had written about 350 articles on the Wayback Machine, mostly from the information technology, library science, and social science fields. Social science scholars have used the Wayback Machine to analyze how the development of websites from the mid-1990s to the present has affected the company's growth.
When the Wayback Machine archives a page, it usually includes most of the hyperlinks, keeping those links active when they just as easily could have been broken by the Internet's instability. Researchers in India studied the effectiveness of the Wayback Machine's ability to save hyperlinks in online scholarly publications and found that it saved slightly more than half of them.
"Journalists use the Wayback Machine to view dead websites, dated news reports, and changes to website contents. Its content has been used to hold politicians accountable and expose battlefield lies."In 2014, an archived social media page of Igor Girkin, a separatist rebel leader in Ukraine, showed him boasting about his troops having shot down a suspected Ukrainian military airplane before it became known that the plane actually was a civilian Malaysian Airlines jet (Malaysia Airlines Flight 17), after which he deleted the post and blamed Ukraine's military for downing the plane. In 2017, the March for Science originated from a discussion on Reddit that indicated someone had visited Archive.org and discovered that all references to climate change had been deleted from the White House website. In response, a user commented, "There needs to be a Scientists' March on Washington".
Furthermore, the site is used heavily for verification, providing access to references and content creation by Wikipedia editors.
In September 2020, a partnership was announced with Cloudflare to automatically archive websites served via its "Always Online" service, which will also allow it to direct users to its copy of the site if it cannot reach the original host.
In 2014 there was a six-month lag time between when a website was crawled and when it became available for viewing in the Wayback Machine.Currently, the lag time is 3 to 10 hours. The Wayback Machine offers only limited search facilities. Its "Site Search" feature allows users to find a site based on words describing the site, rather than words found on the web pages themselves.
Starting in April 2018, administrative staff members of the Wayback Machine's archive team have enforced the Quarter month rule, by occasionally deleting time intervals of 23 days or 39 days (3/4 and 5/4 of a month, respectively), to reduce the queue size.[ citation needed ]
In a 2009 case, Netbula, LLC v. Chordiant Software Inc., defendant Chordiant filed a motion to compel Netbula to disable the robots.txt file on its website that was causing the Wayback Machine to retroactively remove access to previous versions of pages it had archived from Netbula's site, pages that Chordiant believed would support its case.
Netbula objected to the motion on the ground that defendants were asking to alter Netbula's website and that they should have subpoenaed Internet Archive for the pages directly.An employee of Internet Archive filed a sworn statement supporting Chordiant's motion, however, stating that it could not produce the web pages by any other means "without considerable burden, expense and disruption to its operations."
Magistrate Judge Howard Lloyd in the Northern District of California, San Jose Division, rejected Netbula's arguments and ordered them to disable the robots.txt blockage temporarily in order to allow Chordiant to retrieve the archived pages that they sought.
In an October 2004 case, Telewizja Polska USA, Inc. v. Echostar Satellite, No. 02 C 3293, 65 Fed. R. Evid. Serv. 673 (N.D. Ill. October 15, 2004), a litigant attempted to use the Wayback Machine archives as a source of admissible evidence, perhaps for the first time. Telewizja Polska is the provider of TVP Polonia and EchoStar operates the Dish Network. Prior to the trial proceedings, EchoStar indicated that it intended to offer Wayback Machine snapshots as proof of the past content of Telewizja Polska's website. Telewizja Polska brought a motion in limine to suppress the snapshots on the grounds of hearsay and unauthenticated source, but Magistrate Judge Arlander Keys rejected Telewizja Polska's assertion of hearsay and denied TVP's motion in limine to exclude the evidence at trial.At the trial, however, District Court Judge Ronald Guzman, the trial judge, overruled Magistrate Keys' findings, and held that neither the affidavit of the Internet Archive employee nor the underlying pages (i.e., the Telewizja Polska website) were admissible as evidence. Judge Guzman reasoned that the employee's affidavit contained both hearsay and inconclusive supporting statements, and the purported web page, printouts were not self-authenticating.
Provided some additional requirements are met (e.g., providing an authoritative statement of the archivist), the United States patent office and the European Patent Office will accept date stamps from the Internet Archive as evidence of when a given Web page was accessible to the public. These dates are used to determine if a Web page is available as prior art for instance in examining a patent application.
There are technical limitations to archiving a website, and as a consequence, opposing parties in litigation can misuse the results provided by website archives. This problem can be exacerbated by the practice of submitting screenshots of web pages in complaints, answers, or expert witness reports when the underlying links are not exposed and therefore, can contain errors. For example, archives such as the Wayback Machine do not fill out forms and therefore, do not include the contents of non-RESTful e-commerce databases in their archives.
In Europe, the Wayback Machine could be interpreted as violating copyright laws. Only the content creator can decide where their content is published or duplicated, so the Archive would have to delete pages from its system upon request of the creator.The exclusion policies for the Wayback Machine may be found in the FAQ section of the site.
Some cases have been brought against the Internet Archive specifically for its Wayback Machine archiving efforts.
In late 2002, the Internet Archive removed various sites that were critical of Scientology from the Wayback Machine.An error message stated that this was in response to a "request by the site owner". Later, it was clarified that lawyers from the Church of Scientology had demanded the removal and that the site owners did not want their material removed.
In 2003, Harding Earley Follmer & Frailey defended a client from a trademark dispute using the Archive's Wayback Machine. The attorneys were able to demonstrate that the claims made by the plaintiff were invalid, based on the content of their website from several years prior. The plaintiff, Healthcare Advocates, then amended their complaint to include the Internet Archive, accusing the organization of copyright infringement as well as violations of the DMCA and the Computer Fraud and Abuse Act. Healthcare Advocates claimed that, since they had installed a robots.txt file on their website, even if after the initial lawsuit was filed, the Archive should have removed all previous copies of the plaintiff website from the Wayback Machine, however, some material continued to be publicly visible on Wayback.The lawsuit was settled out of court after Wayback fixed the problem.
Activist Suzanne Shell filed suit in December 2005, demanding Internet Archive pay her US$100,000 for archiving her website profane-justice.org between 1999 and 2004.Internet Archive filed a declaratory judgment action in the United States District Court for the Northern District of California on January 20, 2006, seeking a judicial determination that Internet Archive did not violate Shell's copyright. Shell responded and brought a countersuit against Internet Archive for archiving her site, which she alleges is in violation of her terms of service. On February 13, 2007, a judge for the United States District Court for the District of Colorado dismissed all counterclaims except breach of contract. The Internet Archive did not move to dismiss copyright infringement claims Shell asserted arising out of its copying activities, which would also go forward.
On April 25, 2007, Internet Archive and Suzanne Shell jointly announced the settlement of their lawsuit.The Internet Archive said it "...has no interest in including materials in the Wayback Machine of persons who do not wish to have their Web content archived. We recognize that Ms. Shell has a valid and enforceable copyright in her Web site and we regret that the inclusion of her Web site in the Wayback Machine resulted in this litigation." Shell said, "I respect the historical value of Internet Archive's goal. I never intended to interfere with that goal nor cause it any harm."
Between 2013 and 2016, a pornographic actor named Daniel Davydiuk tried to remove archived images of himself from the Wayback Machine's archive, first by sending multiple DMCA requests to the archive, and then by appealing to the Federal Court of Canada.
archive.org is currently blocked in China. [ needs update ] Since 2016, the website has been back, available in its entirety, although local commercial lobbyists are suing the Internet Archive in a local court to ban it on copyright grounds.After the Islamic State terrorist organization was banned, the Internet Archive had been blocked in its entirety in Russia as a host of an outreach video from that organization, for a short time in 2015–16.
Alison Macrina, director of the Library Freedom Project, notes that "while librarians deeply value individual privacy, we also strongly oppose censorship".
There is at least one case in which an article was removed from the archive shortly after it had been removed from its original website. A Daily Beast reporter had written an article that outed several gay Olympian athletes in 2016 after he had made a fake profile posing as a gay man on a dating app. The Daily Beast removed the article after it was met with widespread furor; not long after, the Internet Archive soon did as well, but emphatically stated that they did so for no other reason than to protect the safety of the outed athletes.
Other threats include natural disasters,destruction (remote or physical), manipulation of the archive's contents (see also: cyberattack, backup), problematic copyright laws and surveillance of the site's users.
Alexander Rose, executive director of the Long Now Foundation, suspects that in the long term of multiple generations "next to nothing" will survive in a useful way, stating, "If we have continuity in our technological civilization, I suspect a lot of the bare data will remain findable and searchable. But I suspect almost nothing of the format in which it was delivered will be recognizable" because sites "with deep back-ends of content-management systems like Drupal and Ruby and Django" are harder to archive.
In an article reflecting on the preservation of human knowledge, The Atlantic has commented that the Internet Archive, which describes itself to be built for the long-term,"is working furiously to capture data before it disappears without any long-term infrastructure to speak of."
Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. They are part of a web page's
head section. Multiple Meta elements with different attributes can be used on the same page. Meta elements can be used to specify page description, keywords and any other metadata not provided through the other
head elements and attributes.
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically operated by search engines for the purpose of Web indexing.
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.
The Internet Archive is an American digital library with the stated mission of "universal access to all knowledge". It provides free public access to collections of digitized materials, including websites, software applications/games, music, movies/videos, moving images, and millions of books. In addition to its archiving function, the Archive is an activist organization, advocating a free and open Internet. As of November 2021, the Internet Archive holds over 33 million books and texts, 7.3 million movies, videos and TV shows, 785,000 software programs, 13,901,000 audio files, 4 million images, and 627 billion web pages in the Wayback Machine.
Search engine optimization (SEO) is the process of improving the quality and quantity of website traffic to a website or a web page from search engines. SEO targets unpaid traffic rather than direct traffic or paid traffic. Unpaid traffic may originate from different kinds of searches, including image search, video search, academic search, news search, and industry-specific vertical search engines.
In the context of the World Wide Web, deep linking is the use of a hyperlink that links to a specific, generally searchable or indexed, piece of web content on a website, rather than the website's home page. The URL contains all the information needed to point to a particular item. Deep linking is different from mobile deep linking, which refers to directly linking to in-app content using a non-HTTP URI.
goatse.cx, often referred to simply as Goatse, was originally an Internet shock site. Its front page featured a picture, entitled
hello.jpg, showing a naked man, hunched over, widely stretching his anus with both of his hands to approximately the width of a grapefruit.
Alexa Internet, Inc. is an American web traffic analysis company based in San Francisco. It is a wholly owned subsidiary of Amazon.
Apache Nutch is a highly extensible and scalable open source web crawler software project.
The deep web, invisible web, or hidden web are parts of the World Wide Web whose contents are not indexed by standard web search-engines. This is in contrast to the "surface web", which is accessible to anyone using the Internet. Computer-scientist Michael K. Bergman is credited with coining the term in 2001 as a search-indexing term.
The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page. Reasons why one might want to use this meta tag include advising robots not to index a very large database, web pages that are very transitory, web pages that are under development, web pages that one wishes to keep slightly more private, or the printer and mobile-friendly versions of pages. Since the burden of honoring a website's noindex tag lies with the author of the search robot, sometimes these tags are ignored. Also the interpretation of the noindex tag is sometimes slightly different from one search engine company to the next.
An Internet bot, web robot, robot or simply bot, is a software application that runs automated tasks (scripts) over the Internet. Typically, bots perform tasks that are simple and repetitive much faster than a person could. The most extensive use of bots is for web crawling, in which an automated script fetches, analyzes and files information from web servers. More than half of all web traffic is generated by bots.
Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.
A search engine is a software system that is designed to carry out web searches. They search the World Wide Web in a systematic way for particular information specified in a textual web search query. The search results are generally presented in a line of results, often referred to as search engine results pages (SERPs) The information may be a mix of links to web pages, images, videos, infographics, articles, research papers, and other types of files. Some search engines also mine data available in databases or open directories. Unlike web directories, which are maintained only by human editors, search engines also maintain real-time information by running an algorithm on a web crawler. Internet content that is not capable of being searched by a web search engine is generally described as the deep web.
Web archiving is the process of collecting portions of the World Wide Web to ensure the information is preserved in an archive for future researchers, historians, and the public. Web archivists typically employ web crawlers for automated capture due to the massive size and amount of information on the Web. The largest web archiving organization based on a bulk crawling approach is the Wayback Machine, which strives to maintain an archive of the entire Web.
WebCite is an on-demand archive site, designed to digitally preserve scientific and educationally important material on the web by making snapshots of Internet contents as they existed at the time when a blogger or a scholar cited or quoted from it. The preservation service enables verifiability of claims supported by the cited sources even when the original web pages are being revised, removed, or disappear for other reasons, an effect known as link rot.
BotSeer was a Web-based information system and search tool used for research on Web robots and trends in Robot Exclusion Protocol deployment and adherence. It was created and designed by Yang Sun, Isaac G. Councill, Ziming Zhuang and C. Lee Giles. BotSeer is now inactive; the original URL was https://web.archive.org/web/20100208214818/http://botseer.ist.psu.edu/
Deletionpedia is an online archive wiki containing articles deleted from the English Wikipedia. Its version of each article includes a header with more information about the deletion such as whether a speedy deletion occurred, where the deletion discussion about the article can be found and which editor deleted the article. The original Deletionpedia operated from February to September 2008. The site was restarted under new management in December 2013.
Search engine cache is a cache of web pages that shows the page as it was when it was indexed by a web crawler. Cached versions of web pages can be used to view the contents of a page when the live version cannot be reached, has been altered or taken down.
We have added the ability to archive a page instantly and get back a permanent URL for that page in the Wayback Machine. This service allows anyone – wikipedia editors, scholars, legal professionals, students, or home cooks like me – to create a stable URL to cite, share or bookmark any information they want to still have access to in the future.
2015-03-25: Latest URLs hosted in this IP address detected by at least one URL scanner or malicious URL dataset. ... 2/62 2015-03-25 16:14:12 [complete URL redacted]/Renegotiating_TLS.pdf ... 1/62 2015-03-25 04:46:34 [complete URL redacted]/CBLightSetup.exeCS1 maint: location (link)
2015-03-25: Part of this site was listed for suspicious activity 138 time(s) over the past 90 days. ... What happened when Google visited this site? ... Of the 42410 pages we tested on the site over the past 90 days, 450 page(s) resulted in malicious software being downloaded and installed without user consent. The last time Google visited this site was on 2015-03-25, and the last time suspicious content was found on this site was on 2015-03-25. ... Malicious software includes 169 trojan(s), 126 virus, 43 backdoor(s).
1) Internet Archive's motion to dismiss Shell's counterclaim for conversion and civil theft (Second Cause of Action) is GRANTED, 2) Internet Archive's motion to dismiss Shell's counterclaim for breach of contract (Third Cause of Action) is DENIED; 3) Internet Archive's motion to dismiss Shell's counterclaim for Racketeering under RICO and COCCA (Fourth Cause of Action) is GRANTED.
Computers can enter into contracts on behalf of people. The Uniform Electronic Transactions Act (UETA) says that a 'contract may be formed by the interaction of electronic agents of the parties, even if no individual was aware of or reviewed the electronic agents' actions or the resulting terms and agreements.'
More importantly, held the court, Internet Archive's mere copying of Shell's site, and display thereof in its database, did not constitute the requisite exercise of dominion and control over defendant's property. Importantly, noted the court, the defendant at all times owned and operated her own site. Said the Court: 'Shell has failed to allege facts showing that Internet Archive exercised dominion or control over her website, since Shell's complaint states explicitly that she continued to own and operate the website while it was archived on the Wayback machine. Shell identifies no authority supporting the notion that copying documents is by itself enough of a deprivation of use to support conversion. Conversely, numerous circuits have determined that it is not.'
Both parties sincerely regret any turmoil that the lawsuit may have caused for the other. Neither Internet Archive nor Ms. Shell condones any conduct which may have caused harm to either party arising out of the public attention to this lawsuit. The parties have not engaged in such conduct and request that the public response to the amicable resolution of this litigation be consistent with their wishes that no further harm or turmoil be caused to either party.