Original author(s) | Nick Craig-Wood [3] |
---|---|
Developer(s) | Nick Craig-Wood, [4] Ivan Andreev [5] |
Initial release | July 3, 2014 [6] |
Stable release | |
Repository | github |
Written in | Go [8] |
Operating system | Linux, Windows, macOS, FreeBSD, NetBSD, OpenBSD, Plan9, Solaris [8] |
Platform | Intel/AMD-64, Intel/AMD-32, ARM-32, ARM-64, MIPS-Big-Endian, MIPS-Little-Endian [8] |
Licence | MIT |
Website | rclone |
Rclone is an open source, multi threaded, command line computer program to manage or migrate content on cloud and other high latency storage. Its capabilities include sync, transfer, crypt, cache, union, compress and mount. The rclone website lists supported backends including S3 and Google Drive. [8]
Descriptions of rclone often carry the strapline "Rclone syncs your files to cloud storage". [8] Those prior to 2020 include the alternative "Rsync for Cloud Storage". [9]
Rclone is well known for its rclone sync
and rclone mount
commands. [10] It provides further management functions analogous to those ordinarily used for files on local disks, but which tolerate some intermittent and unreliable service. Rclone is commonly used with media servers such as Plex, [11] Emby or Jellyfin [12] to stream content direct from consumer file storage services. [11]
Official Ubuntu, Debian, Fedora, Gentoo, Arch, Brew, Chocolatey, and other package managers include rclone. [13]
Nick Craig-Wood was inspired by rsync. [14] [9] Concerns about the noise and power costs arising from home computer servers prompted him to embrace cloud storage and he began developing rclone as open source software in 2012 under the name swiftsync. [15] [16] [3]
Rclone was promoted to stable version 1.00 in July 2014. [6]
In May 2017, Amazon Drive barred new users of rclone and other upload utilities, citing security concerns. [17] Amazon Drive had been advertised as offering unlimited storage for £55 per year. Amazon's AWS S3 service continues to support new rclone users.
The original rclone logo was updated in September 2018. [18]
In March 2020, Nick Craig-Wood resigned from Memset Ltd, a cloud hosting company he founded, to focus on open source software. [19] [16] [20] [21]
Amazon's AWS April 2020 public sector blog explained how the Fred Hutch Cancer Research Center were using rclone in their Motuz tool to migrate very large biomedical research datasets in and out of AWS S3 object stores. [22]
In November 2020, rclone was updated to correct a weakness in the way it generated passwords. Passwords for encrypted remotes can be generated randomly by rclone or supplied by the user. In all versions of rclone from 1.49.0 to 1.53.2 the seed value for generated passwords was based on the number of seconds elapsed in the day, and therefore not truly random. CVE-2020-28924 recommended users upgrade to the latest version of rclone and check the passwords protecting their encrypted remotes. [23]
Release 1.55 of rclone in March 2021 included features sponsored by CERN and their CS3MESH4EOSC project. [24] The work was EU funded to promote vendor-neutral application programming interfaces and protocols for synchronisation and sharing of academic data on cloud storage. [25]
Rclone supports the following services as backends. There are others, built on standard protocols such as WebDAV or S3, that work. [8] WebDAV backends do not support rclone functionality dependent on server side checksum or modtime. [26]
Remotes are usually defined interactively from these backends, local disk, or memory (as S3), with rclone config
. Rclone can further wrap those remotes with one or more of alias
, chunk
, compress
, crypt
or union
, remotes.
Once defined, the remotes are referenced by other rclone commands interchangeably with the local drive. Remote names are followed by a colon to distinguish them from local drives. For example, a remote example_remote containing a folder, or pseudofolder, myfolder is referred to within a command as a path example_remote:/myfolder
. [36]
Rclone commands directly apply to remotes, or mount them for file access or streaming. With appropriate cache options the mount can be addressed as if a conventional, block level disk. Commands are provided to serve remotes over SFTP, HTTP, WebDAV, FTP and DLNA. [8] Commands can have sub-commands and flags. Filters determine which files on a remote that rclone commands are applied to. [37]
rclone rc
passes commands or new parameters to existing rclone sessions and has an experimental web browser interface. [38]
Rclone's crypt implements encryption of files at rest in cloud storage. It layers an encrypted remote over a pre-existing, cloud or other remote. Crypt is commonly [11] used to encrypt / decrypt media, for streaming, on consumer storage services such as Google Drive.
Rclone's configuration file contains the crypt password. The password can be lightly obfuscated, or the whole rclone.conf file can be encrypted. [39]
Crypt can either encrypt file content and name, or additionally full paths. In the latter case there is a potential clash with encryption for cloud backends, such as Microsoft OneDrive, having limited path lengths. [40] Crypt remotes do not encrypt object modification time or size. The encryption mechanism for content, name and path is available, for scrutiny, on the rclone website. Key derivation is with scrypt. [39]
These examples describe paths and file names but object keys behave similarly.
To recursively copy files from directory remote_stuff, at the remote xmpl, to directory stuff in the home folder:-
$ rclonecopy-v-Pxmpl:/remote_stuff~/stuff
-v
enables logging and -P
, progress information. By default rclone checks the file integrity (hash) after copy; can retry each file up to three times if the operation is interrupted; uses up to four parallel transfer threads, and does not apply bandwidth throttling.
Running the above command again copies any new or changed files at the remote to the local folder but, like default rsync behaviour, will not delete from the local directory, files which have been removed from the remote.
To additionally delete files from the local folder which have been removed from the remote - more like the behaviour of rsync with a --delete
flag:-
$ rclonesyncxmpl:/remote_stuff~/stuff
And to delete files from the source after they have been transferred to the local directory - more like the behaviour of rsync with a --remove-source-file
flag:-
$ rclonemovexmpl:/remote_stuff~/stuff
To mount the remote directory at a mountpoint in the pre-existing, empty stuff directory in the home directory (the ampersand at the end makes the mount command run as a background process):-
$ rclonemountxmpl:/remote_stuff~/stuff&
Default rclone syntax can be modified. Alternative transfer, filter, conflict and backend specific flags are available. Performance choices include number of concurrent transfer threads; chunk size; bandwidth limit profiling, and cache aggression. [37]
In 2018, University of Kentucky researchers published a conference paper comparing use of rclone and other command line, cloud data transfer agents for big data. [41] The paper was published as a result of funding by the National Science Foundation. [42]
Later that year, University of Utah's Center for High Performance Computing examined the impact of rclone options on data transfer rates. [43]
Examples are University of Maryland, [44] Iowa State University, [45] Trinity College Dublin, [46] NYU, [47] BYU, [48] Indiana University, [49] CSC Finland, [50] Utrecht University, [51] University of Nebraska, [52] University of Utah, [53] North Carolina State University, [54] Stony Brook, [55] Tulane University, [56] Washington State University, [57] Georgia Tech, [58] National Institutes of Health, [59] Wharton, [60] Yale, [61] Harvard, [62] Minnesota, [63] Michigan State, [64] Case Western Reserve University, [65] University of South Dakota, Northern Arizona University, [66] University of Pennsylvania, [67] Stanford, [68] University of Southern California, [69] UC Santa Barbara, [70] [71] UC Irvine, [72] UC Berkeley, [73] [74] and SURFnet. [75]
This section may lend undue weight to the idea that rclone is used for cybercrime, when most of the time it is used for innocuous purposes. Please help to create a more balanced presentation. Discuss and resolve this issue before removing this message. (January 2023) |
May 2020 reports stated rclone had been used by hackers to exploit Diebold Nixdorf ATMs with ProLock ransomware. [76] [77] The FBI issued a Flash Alert MI-000125-MW on May 4, 2020, in relation to the compromise. [78] They issued a further, related alert 20200901–001 in September 2020. Attackers had exfiltrated / encrypted data from organisations involved in healthcare, construction, finance, and legal services. Multiple US government agencies, and industrial entities were affected. Researchers established the hackers spent about a month exploring the breached networks, using rclone to archive stolen data to cloud storage, before encrypting the target system. [78] [79] [80] Reported targets included LaSalle County, and the city of Novi Sad. [81] [82]
The FBI warned January 2021, in Private Industry Notification 20210106–001, of extortion activity using Egregor ransomware and rclone. Organisations worldwide had been threatened with public release of exfiltrated data. In some cases rclone had been disguised under the name svchost. [83] Bookseller Barnes & Noble, US retailer Kmart, games developer Ubisoft and the Vancouver metro system have been reported as victims. [84]
An April 2021, cybersecurity investigation into SonicWall VPN zero-day vulnerability SNWLID-2021-0001 [85] by FireEye's Mandiant team established attackers UNC2447 used rclone for reconnaissance and exfiltration of victims' files. [86] Cybersecurity and Infrastructure Security Agency Analysis Report AR21-126A confirmed this use of rclone in FiveHands ransomware attacks. [87]
A June 2021, Microsoft Security Intelligence Twitter post identified use of rclone in BazaCall cyber attacks. [88] The miscreants sent emails encouraging recipients to contact a bogus call centre to cancel a paid for service. The call centre team then instructed victims to download a hostile file that installed malware on the target network, ultimately allowing use of rclone for covert extraction of potentially sensitive data.
In a 2021, Star Wars Day blog article, Managed Security Service Provider Red Canary announced Rclone Wars, an allusion to Clone Wars. The post notes illicit use of other legitimate file transfer utilities in exfiltrate and extort schemes but focuses on MEGAsync, MEGAcmd and rclone. To identify use of renamed rclone executables on compromised devices the authors suggest monitoring for distinctive rclone top level commands and command line flag strings such as remote:
and \\
. [89]
Rsync transfers files with other computers that have rsync installed. [90] It operates at the block, rather than file, level and has a delta algorithm so that it only needs to transfer changes in files. Rsync preserves file attributes and permissions. Rclone has a wider range of content management capabilities, and types of backend it can address, but only works at a whole file / object level. [91] [3] It does not currently preserve permissions and attributes. [92] Rclone is designed to have some tolerance of intermittent and unreliable connections or remote services. Its transfers are optimised for high latency networks. Rclone decides which of those whole files / objects to transfer after obtaining checksums, to compare, from the remote server. Where checksums are not available, rclone can use object size and timestamp. [93]
Rsync is single threaded. [94] Rclone is multi threaded with a user definable number of simultaneous transfers. [95] [96]
Rclone can pipe data between two completely remote locations, sometimes without local download. During an rsync transfer, one side must be a local drive. [90] [96]
Rclone ignores trailing slashes. Rsync requires their correct use. [97] Rclone filters require the use of **
to refer to the contents of a directory. Rsync does not. [98]
Eponymous cloud storage service rsync.net provides remote unix filesystems so that customers can run rsync and other standard Unix tools. [99] They also offer rclone only accounts. [100]
In 2016, a poster on Hacker News summarised rclone's relationship to rsync as:- (rclone) exists to give you rsync to things that aren't rsync. If you want to rsync to things that are rsync, use rsync. [101]
rsync is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems and is under the GPL-3.0-or-later license.
In software engineering, the terms frontend and backend refer to the separation of concerns between the presentation layer (frontend), and the data access layer (backend) of a piece of software, or the physical infrastructure or hardware. In the client–server model, the client is usually considered the frontend and the server is usually considered the backend, even when some presentation work is actually done on the server itself.
Quantum Corporation is a data storage, management, and protection company that provides technology to store, manage, archive, and protect video and unstructured data throughout the data life cycle. Their products are used by enterprises, media and entertainment companies, government agencies, big data companies, and life science organizations. Quantum is headquartered in San Jose, California and has offices around the world, supporting customers globally in addition to working with a network of distributors, VARs, DMRs, OEMs and other suppliers.
NetApp, Inc. is an American data infrastructure company that provides unified data storage, integrated data services, and cloud operations (CloudOps) solutions to enterprise customers. The company is based in Cork City, Ireland. It has ranked in the Fortune 500 from 2012 to 2021. Founded in 1992 with an initial public offering in 1995, NetApp offers cloud data services for management of applications and data both online and physically.
Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces.
A file-hosting service, also known as cloud-storage service, online file-storage provider, or cyberlocker, is an internet hosting service specifically designed to host user files. These services allow users to upload files that can be accessed over the internet after providing a username and password or other authentication. Typically, file hosting services allow HTTP access, and in some cases, FTP access. Other related services include content-displaying hosting services, virtual storage, and remote backup solutions.
Lustre is a type of parallel distributed file system, generally used for large-scale cluster computing. The name Lustre is a portmanteau word derived from Linux and cluster. Lustre file system software is available under the GNU General Public License and provides high performance file systems for computer clusters ranging in size from small workgroup clusters to large-scale, multi-site systems. Since June 2005, Lustre has consistently been used by at least half of the top ten, and more than 60 of the top 100 fastest supercomputers in the world, including the world's No. 1 ranked TOP500 supercomputer in November 2022, Frontier, as well as previous top supercomputers such as Fugaku, Titan and Sequoia.
Ransomware is a type of malware that permanently blocks access to the victim's personal data unless a "ransom" is paid. While some simple ransomware may lock the system without damaging any files, more advanced malware uses a technique called cryptoviral extortion. It encrypts the victim's files, making them inaccessible, and demands a ransom payment to decrypt them. In a properly implemented cryptoviral extortion attack, recovering the files without the decryption key is an intractable problem, and difficult-to-trace digital currencies such as paysafecard or Bitcoin and other cryptocurrencies are used for the ransoms, making tracing and prosecuting the perpetrators difficult.
Backblaze, Inc. is an American cloud storage and data backup company based in San Mateo, California. It was founded in 2007 by Gleb Budman and others. Its two main products are their B2 Cloud Storage and Computer Backup services, targeted at both business and personal markets.
This is a comparison of online backup services.
Cloud computing is the on-demand availability of computer system resources, especially data storage and computing power, without direct active management by the user. Large clouds often have functions distributed over multiple locations, each of which is a data center. Cloud computing relies on sharing of resources to achieve coherence and typically uses a pay-as-you-go model, which can help in reducing capital expenses but may also lead to unexpected operating expenses for users.
This is a comparison of notable file hosting services that are currently active. File hosting services are a particular kind of online file storage; however, various products that are designed for online file storage may not have features or characteristics that others designed for sharing files have.
ExpanDrive is a network filesystem client for MacOS, Microsoft Windows and Linux that facilitates mapping of local volume to many different types of cloud storage. When a server is mounted with ExpanDrive any program can read, write, and manage remote files as if they were stored locally. This is different from most File Transfer Clients because it is integrated into all applications on the operating system. It also does not require a file to be downloaded to access portions of the content. ExpanDrive is commercial software, at a cost of $49.95 per license. A 7-day, unrestricted demo is available for evaluation.
This is a list of file synchronization software for which there are Wikipedia articles.
Egnyte is a software company headquartered in Mountain View, California. It sells cloud-based content security, compliance, and collaboration tools for businesses. Egnyte was founded in 2007 with a focus on modernized file servers, but it has since shifted to selling tools that help users securely collaborate with coworkers and third parties.
iCloud is a cloud service operated by Apple Inc. Launched on October 12, 2011, iCloud enables users to store and sync data across devices, including Apple Mail, Apple Calendar, Apple Photos, Apple Notes, contacts, settings, backups, and files, to collaborate with other users, and track assets through Find My. It is built into iOS, iPadOS, watchOS, tvOS, macOS, and visionOS. iCloud may additionally be accessed through a limited web interface and Windows application.
ownCloud, a Kiteworks Company, is a free and open-source software project for content collaboration and sharing and syncing of files in distributed and federated enterprise scenarios. It allows companies and remote end-users to organize their documents on servers, computers, and mobile devices and work with them collaboratively while keeping a centrally organized and synchronized state.
Scality is a global technology provider of software-defined storage (SDS) solutions, specializing in distributed file and object storage with cloud data management. Scality maintains offices in Paris (France), London (UK), San Francisco and Washington DC (USA), and Tokyo (Japan) and has employees in 14 countries.
Nasuni Corporation is a privately-held hybrid cloud storage company with headquarters in Boston, Massachusetts.
MSP360, formerly CloudBerry Lab, is a software and application service provider company that develops online backup, remote desktop and file management products integrated with more than 20 cloud storage providers.