Amazon S3

Last updated
Amazon S3
Amazon-S3-Logo.svg
Amazon S3 Logo
Type of site
 Cloud storage
Available in English
Owner Amazon.com
URL aws.amazon.com/s3/
IPv6 supportYes
CommercialYes
RegistrationRequired (included in free tier layer)
LaunchedMarch 14, 2006;17 years ago (2006-03-14)
Current statusActive

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface. [1] [2] Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its e-commerce network. [3] Amazon S3 can store any type of object, which allows uses like storage for Internet applications, backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage. AWS launched Amazon S3 in the United States on March 14, 2006, [1] [4] then in Europe in November 2007. [5]

Contents

Technical details

Design

Amazon S3 manages data with an object storage architecture [6] which aims to provide scalability, high availability, and low latency with high durability. [3] The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key. [7] Buckets can be managed using the console provided by Amazon S3, programmatically with the AWS SDK, or the REST application programming interface. Objects can be up to five terabytes in size. [8] [9] Requests are authorized using an access control list associated with each object bucket and support versioning [10] which is disabled by default. [11] Since buckets are typically the size of an entire file system mount in other systems, this access control scheme is very coarse-grained. In other words, unique access controls cannot be associated with individual files.[ citation needed ] Amazon S3 can be used to replace static web-hosting infrastructure with HTTP client-accessible objects, [12] index document support, and error document support. [13] The Amazon AWS authentication mechanism allows the creation of authenticated URLs, valid for a specified amount of time. Every item in a bucket can also be served as a BitTorrent feed. The Amazon S3 store can act as a seed host for a torrent and any BitTorrent client can retrieve the file. This can drastically reduce the bandwidth cost for the download of popular objects. A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in data mining operations. [14] There are various User Mode File System (FUSE)–based file systems for Unix-like operating systems (for example, Linux) that can be used to mount an S3 bucket as a file system. The semantics of the Amazon S3 file system are not that of a POSIX file system, so the file system may not behave entirely as expected. [15]

Amazon S3 storage classes

Amazon S3 offers nine different storage classes with different levels of durability, availability, and performance requirements. [16]

The Amazon S3 Glacier storage classes above are distinct from Amazon Glacier, which is a separate product with its own APIs.

File size limits

An object in S3 can be between 0 bytes and 5TB. If an object is larger than 5TB, it must be divided into chunks prior to uploading. When uploading, Amazon S3 allows a maximum of 5GB in a single upload operation; hence, objects larger than 5GB must be uploaded via the S3 multipart upload API. [18]

Uses

Notable users

S3 API and competing services

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface but are differentiated by their underlying technologies and business models. [29] A standard interface enables better competition from rival providers and allows economies of scale in implementation, among other benefits. [30]

History

At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3. AmazonS3TwoTrillionObjects.JPG
At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3.

Amazon Web Services introduced Amazon S3 in 2006. [31] [32]

DateNumber of Items Stored
October 200710 billion [33]
January 200814 billion [33]
October 200829 billion [34]
March 200952 billion [35]
August 200964 billion [36]
March 2010102 billion [37]
April 20132 trillion [38]
March 2021100 trillion [39]

In November 2017 AWS added default encryption capabilities at bucket level. [40]

See also

Related Research Articles

NetApp, Inc. is an intelligent data infrastructure company that provides unified data storage, integrated data services, and cloud operations (CloudOps) solutions to enterprise customers. The company is based in San Jose, California. It has ranked in the Fortune 500 from 2012 to 2021. Founded in 1992 with an initial public offering in 1995, NetApp offers cloud data services for management of applications and data both online and physically.

Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces.

<span class="mw-page-title-main">Amazon Web Services</span> On-demand cloud computing company

Amazon Web Services, Inc. (AWS) is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered, pay-as-you-go basis. Clients will often use this in combination with autoscaling. These cloud computing web services provide various services related to networking, compute, storage, middleware, IoT and other processing capacity, as well as software tools via AWS server farms. This frees clients from managing, scaling, and patching hardware and operating systems. One of the foundational services is Amazon Elastic Compute Cloud (EC2), which allows users to have at their disposal a virtual cluster of computers, with extremely high availability, which can be interacted with over the internet via REST APIs, a CLI or the AWS console. AWS's virtual computers emulate most of the attributes of a real computer, including hardware central processing units (CPUs) and graphics processing units (GPUs) for processing; local/RAM memory; hard-disk/SSD storage; a choice of operating systems; networking; and pre-loaded application software such as web servers, databases, and customer relationship management (CRM).

Cloud storage is a model of computer data storage in which data, said to be on "the cloud", is stored remotely in logical pools and is accessible to users over a network, typically the Internet. The physical storage spans multiple servers, and the physical environment is typically owned and managed by a cloud computing provider. These cloud storage providers are responsible for keeping the data available and accessible, and the physical environment secured, protected, and running. People and organizations buy or lease storage capacity from the providers to store user, organization, or application data.

<span class="mw-page-title-main">Amazon Elastic Compute Cloud</span> Cloud computing platform

Amazon Elastic Compute Cloud (EC2) is a part of Amazon.com's cloud-computing platform, Amazon Web Services (AWS), that allows users to rent virtual computers on which to run their own computer applications. EC2 encourages scalable deployment of applications by providing a web service through which a user can boot an Amazon Machine Image (AMI) to configure a virtual machine, which Amazon calls an "instance", containing any software desired. A user can create, launch, and terminate server-instances as needed, paying by the second for active servers – hence the term "elastic". EC2 provides users with control over the geographical location of instances that allows for latency optimization and high levels of redundancy. In November 2010, Amazon switched its own retail website platform to EC2 and AWS.

<span class="mw-page-title-main">Cloud computing</span> Form of shared Internet-based computing

Cloud computing is the on-demand availability of computer system resources, especially data storage and computing power, without direct active management by the user. Large clouds often have functions distributed over multiple locations, each of which is a data center. Cloud computing relies on sharing of resources to achieve coherence and typically uses a pay-as-you-go model, which can help in reducing capital expenses but may also lead to unexpected operating expenses for users.

<span class="mw-page-title-main">IBM Cloud Object Storage</span>

IBM Cloud Object Storage is a service offered by IBM for storing and accessing unstructured data. The object storage service can be deployed on-premises, as part of IBM Cloud Platform offerings, or in hybrid form. The offering can store any type of object which allows for uses like data archiving and backup, web and mobile applications, and as scalable, persistent storage for analytics. Interaction with Cloud Object Storage is based on Rest APIs.

An Amazon Machine Image (AMI) is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2.

Google Cloud Storage is a RESTful online file storage web service for storing and accessing data on Google Cloud Platform infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. It is an Infrastructure as a Service (IaaS), comparable to Amazon S3. Contrary to Google Drive and according to different service specifications, Google Cloud Storage appears to be more suitable for enterprises.

A cloud storage gateway is a hybrid cloud storage device, implemented in hardware or software, which resides at the customer premises and translates cloud storage APIs such as SOAP or REST to block-based storage protocols such as iSCSI or Fibre Channel or file-based interfaces such as NFS or SMB.

<span class="mw-page-title-main">Amazon Elastic Block Store</span> Cloud-based raw storage service

Amazon Elastic Block Store (EBS) provides raw block-level storage that can be attached to Amazon EC2 instances and is used by Amazon Relational Database Service (RDS). It is one of the two block-storage options offered by AWS, with the other being the EC2 Instance Store.

Amazon S3 Glacier is an online file storage web service that provides storage for data archiving and backup.

Zadara is a cloud computing company founded in 2011, with headquarters in Irvine, California. The company develops computer software that it markets as storage-as-a-service, which can be used for cloud or on-premises servers, a model sometimes called private cloud.

Scality is a global technology provider of software-defined storage (SDS) solutions, specializing in distributed file and object storage with cloud data management. Scality maintains offices in Paris (France), London (UK), San Francisco and Washington DC (USA), and Tokyo (Japan) and has employees in 14 countries.

AWS CloudFormation is a service provided by Amazon Web Services (AWS) that enables users to model and manage infrastructure resources in an automated and secure manner. Using CloudFormation, developers can define and provision AWS infrastructure resources using a JSON or YAML formatted Infrastructure as Code (IaC) template. The service was released on February 25, 2011.

This is a timeline of Amazon Web Services, which offers a suite of cloud computing services that make up an on-demand computing platform.

Amazon Neptune is a managed graph database product published by Amazon.com. It is used as a web service and is part of Amazon Web Services (AWS). It was announced on November 29, 2017. Amazon Neptune supports popular graph models property graph and W3C's RDF, and their respective query languages Apache TinkerPop's Gremlin, openCypher, and SPARQL, including other Amazon Web Services products.

<span class="mw-page-title-main">Rclone</span> Cloud storage management software

Rclone is an open source, multi threaded, command line computer program to manage or migrate content on cloud and other high latency storage. Its capabilities include sync, transfer, crypt, cache, union, compress and mount. The rclone website lists supported backends including S3 and Google Drive.

Amazon Elastic File System is a cloud storage service provided by Amazon Web Services (AWS) designed to provide scalable, elastic, concurrent with some restrictions, and encrypted file storage for use with both AWS cloud services and on-premises resources. Amazon EFS is built to be able to grow and shrink automatically as files are added and removed. Amazon EFS supports Network File System (NFS) versions 4.0 and 4.1 (NFSv4) protocol, and control access to files through Portable Operating System Interface (POSIX) permissions.

The AWS Cloud Development Kit is an open-source software development framework developed by Amazon Web Services (AWS) for defining and provisioning cloud infrastructure resources using familiar programming languages. The AWS CDK aims to improve the experience of working with Infrastructure as Code by providing higher-level, reusable constructs that enable developers to create and manage AWS resources more efficiently and with less boilerplate code compared to traditional configuration files like AWS CloudFormation templates.

References

Citations

  1. 1 2 "Amazon Web Services Launches "Amazon S3"" (Press release). 2006-03-14. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  2. ↑ Huang, Dijiang; Wu, Huijun (2017-09-08). Mobile Cloud Computing: Foundations and Service Models. Morgan Kaufmann. p. 67. ISBN   9780128096444. Archived from the original on 2018-11-15. Retrieved 2018-11-15.
  3. 1 2 "Cloud Object Storage - Store & Retrieve Data Anywhere - Amazon Simple Storage Service". Amazon Web Services, Inc. Archived from the original on 2018-05-17. Retrieved 2018-05-17.
  4. ↑ "5 Key Events in the history of Cloud Computing - DZone Cloud". dzone.com. Archived from the original on 2018-09-29. Retrieved 2018-09-28.
  5. ↑ "Amazon Web Services Offers European Storage for Amazon S3" (Press release). 2007-11-06. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  6. ↑ "What is Cloud Object Storage? – AWS". Amazon Web Services, Inc. 2019-10-16. Archived from the original on 2018-09-20. Retrieved 2018-07-09.
  7. ↑ "Tech Blog » Starting Websphere in Cloud and saving the data in S3". techblog.aasisvinayak.com. Archived from the original on 2010-03-12.
  8. ↑ "open-guides/og-aws". GitHub. Archived from the original on 2018-01-03. Retrieved 2018-05-17.
  9. ↑ "Error Responses - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2017-12-24. Retrieved 2018-05-21.
  10. ↑ "Using versioning in S3 buckets - Amazon Simple Storage Service".
  11. ↑ "Introduction to Amazon S3 - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2018-05-12. Retrieved 2018-05-17.
  12. ↑ "How to use Amazon S3 for Web Hosting". bucketexplorer.com. Archived from the original on 2008-04-08. Retrieved 2008-05-06.
  13. ↑ Amazon Simple Storage Service Archived 2011-02-20 at the Wayback Machine Docs.amazonwebservices.com. Retrieved on 2013-08-09.
  14. ↑ http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html Archived 2014-12-23 at the Wayback Machine Server Access Logging
  15. ↑ "Comparison of S3QL and other S3 file systems". Archived from the original on 2012-08-05. Retrieved 2012-06-29.
  16. ↑ "Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS". Amazon Web Services, Inc. Archived from the original on 2018-06-13. Retrieved 2018-05-17.
  17. ↑ "Announcing the new Amazon S3 Express One Zone high performance storage class | AWS News Blog". aws.amazon.com. 2023-11-28. Retrieved 2023-12-01.
  18. ↑ "How to Upload Large Files to S3". June 21, 2022. Archived from the original on October 1, 2022. Retrieved June 22, 2022.
  19. ↑ "Amazon S3: Show Me the Money". SmugMug Blog. SmugMug. November 10, 2006. Archived from the original on 2017-03-03. Retrieved 2017-03-03.
  20. ↑ "S3mper: Consistency in the Cloud". Archived from the original on 2016-04-24. Retrieved 2016-05-01.
  21. ↑ "Introduction to Amazon S3". Amazon. Archived from the original on 2017-12-25. Retrieved 28 December 2017.
  22. ↑ Hern, Alex (2017-02-02). "Amazon Web Services: the secret to the online retailer's future success". the Guardian. Archived from the original on 2018-05-02. Retrieved 2018-04-23.
  23. ↑ "AWS Case Study: reddit". aws.amazon.com. 2015. Archived from the original on 2015-03-17. Retrieved March 18, 2015.
  24. ↑ "What is Tahoe-LAFS-on-S3?". August 21, 2012. Archived from the original on 2013-05-06. Retrieved 2012-08-21.
  25. ↑ "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire". WIRED. Archived from the original on 2018-01-25. Retrieved 2018-04-23.
  26. ↑ "Dropbox saved almost $75 million over two years by building its own tech infrastructure". GeekWire. 2018-02-23. Archived from the original on 2018-04-23. Retrieved 2018-04-23.
  27. ↑ "Swiftype Explains Their Cloud Stack". July 1, 2013. Archived from the original on 2014-12-08. Retrieved 2014-12-08.
  28. ↑ Watters, Audrey (12 July 2010). "Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)". SAY Media, Inc. Archived from the original on 2013-02-17. Retrieved 19 December 2012.
  29. ↑ Crossroads of Information Technology Standards. Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council. Washington, DC: The National Academies Press, 1990. 1990. pp. 36–37. doi:10.17226/10440. ISBN   978-0-309-58171-4. Archived from the original on 2014-03-25. Retrieved 2014-03-25.{{cite book}}: CS1 maint: others (link)
  30. ↑ Overview of Amazon Web Services, 2018, https://docs.aws.amazon.com/whitepapers/latest/aws-overview/introduction.html Archived 2017-11-18 at the Wayback Machine
  31. ↑ Garfinkel, Simson L. 2007. An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS. Harvard Computer Science Group Technical Report TR-08-07. https://dash.harvard.edu/bitstream/handle/1/24829568/tr-08-07.pdf?sequence=1 Archived 2018-07-29 at the Wayback Machine
  32. 1 2 Vogels, Werner (2008-03-19). "Happy Birthday, Amazon S3!". All Things Distributed. Archived from the original on 2008-05-09. Retrieved 2008-05-23.
  33. ↑ "Amazon S3 - Busier Than Ever". 2008-10-08. Archived from the original on 2008-10-11. Retrieved 2008-10-09.
  34. ↑ "Celebrating S3's Third Birthday With Special Anniversary Pricing - Amazon Web Services". typepad.com. 31 March 2009. Archived from the original on 2011-07-07. Retrieved 2009-04-01.
  35. ↑ "Amazon's Head Start in the Cloud Pays Off". eweek.com.[ permanent dead link ]
  36. ↑ "Amazon S3 Now Hosts 100 Billion Objects". datacenterknowledge.com. 9 March 2010. Archived from the original on 2010-03-12. Retrieved 2010-03-09.
  37. ↑ "Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services". typepad.com. 18 April 2013.
  38. ↑ "Celebrate 15 Years of Amazon S3 with 'Pi Week' Livestream Events". amazon.com. 14 March 2021.
  39. ↑ "New Amazon S3 Encryption & Security Features". 6 November 2017.

Sources

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.