Darwin Core Archive

Last updated

Darwin Core Archive (DwC-A) is a biodiversity informatics data standard that makes use of the Darwin Core terms to produce a single, self-contained dataset for species occurrence, checklist, sampling event or material sample data. Essentially it is a set of text (CSV) files with a simple descriptor (meta.xml) to inform others how your files are organized. The format is defined in the Darwin Core Text Guidelines. [1] It is the preferred format for publishing data to the GBIF network.

Contents

Darwin Core

The Darwin Core standard [2] has been used to mobilize the vast majority of specimen occurrence and observational records within the GBIF network. [3] The Darwin Core standard was originally conceived to facilitate the discovery, retrieval, and integration of information about modern biological specimens, their spatio-temporal occurrence, and their supporting evidence housed in collections (physical or digital).

The Darwin Core today is broader in scope. It aims to provide a stable, standard reference for sharing information on biological diversity. As a glossary of terms, the Darwin Core provides stable semantic definitions with the goal of being maximally reusable in a variety of contexts. This means that Darwin Core may still be used in the same way it has historically been used, but may also serve as the basis for building more complex exchange formats, while still ensuring interoperability through a common set of terms.

Archive format

The central idea of an archive is that its data files are logically arranged in a star-like manner, with one core data file surrounded by any number of ’extensions’. Each extension record (or ‘extension file row’) points to a record in the core file; in this way, zero to many extension records can exist for each single core record, a more space-efficient method for data transfer than the alternative of including all the data within a single table which could otherwise contain many empty cells.

Details about recommended extensions can be found in their respective subsections and will be extensively documented in the GBIF registry, which will catalogue all available extensions.

Sharing entire datasets instead of using pageable web services like DiGIR and TAPIR allows much simpler and more efficient data transfer. For example, retrieving 260,000 records via TAPIR takes about nine hours, issuing 1,300 http requests to transfer 500 MB of XML-formatted data. The exact same dataset, encoded as DwC-A and zipped, becomes a 3 MB file. Therefore, GBIF highly recommends compressing an archive using ZIP or GZIP when generating a DwC-A.

An archive requires stable identifiers for core records, but not for extensions. For any kind of shared data it is therefore necessary to have some sort of local record identifiers. It's good practice to maintain – with the original data – identifiers that are stable over time and are not being reused after the record is deleted. If you can, please provide globally unique identifiers instead of local ones.

Archive descriptor

To be completed.


Dataset metadata

A Darwin Core Archive should contain a file containing metadata describing the whole dataset. The Ecological Metadata Language (EML) is the most common format for this, but simple Dublin Core files are being used too.

Related Research Articles

Dublin Core Standardized set of metadata elements

The Dublin Core, also known as the Dublin Core Metadata Element Set, is a set of fifteen "core" elements (properties) for describing resources. This fifteen-element Dublin Core has been formally standardized as ISO 15836, ANSI/NISO Z39.85, and IETF RFC 5013. The Dublin Core Metadata Initiative (DCMI), which formulates the Dublin Core, is a project of the Association for Information Science and Technology (ASIS&T), a non-profit organization. The core properties are part of a larger set of DCMI Metadata Terms. "Dublin Core" is also used as an adjective for Dublin Core metadata, a style of metadata that draws on multiple Resource Description Framework (RDF) vocabularies, packaged and constrained in Dublin Core application profiles.

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a protocol developed for harvesting metadata descriptions of records in an archive so that services can be built using metadata from many archives. An implementation of OAI-PMH must support representing metadata in Dublin Core, but may also support additional representations.

The Darwin Information Typing Architecture (DITA) specification defines a set of document types for authoring and organizing topic-oriented information, as well as a set of mechanisms for combining, extending, and constraining document types. It is an open standard that is defined and maintained by the OASIS DITA Technical Committee.

Learning object metadata Data model

Learning Object Metadata is a data model, usually encoded in XML, used to describe a learning object and similar digital resources used to support learning. The purpose of learning object metadata is to support the reusability of learning objects, to aid discoverability, and to facilitate their interoperability, usually in the context of online learning management systems (LMS).

The National Information Exchange Model (NIEM) is an XML-based information exchange framework from the United States. NIEM represents a collaborative partnership of agencies and organizations across all levels of government and with private industry. The purpose of this partnership is to effectively and efficiently share critical information at key decision points throughout the whole of the justice, public safety, emergency and disaster management, intelligence, and homeland security enterprise. NIEM is designed to develop, disseminate, and support enterprise-wide information exchange standards and processes that will enable jurisdictions to automate information sharing.

A representation term is a word, or a combination of words, that semantically represent the data type of a data element. A representation term is commonly referred to as a class word by those familiar with data dictionaries. ISO/IEC 11179-5:2005 defines representation term as a designation of an instance of a representation class As used in ISO/IEC 11179, the representation term is that part of a data element name that provides a semantic pointer to the underlying data type. A Representation class is a class of representations. This representation class provides a way to classify or group data elements.

This article describes the technical specifications of the OpenDocument office document standard, as developed by the OASIS industry consortium. A variety of organizations developed the standard publicly and make it publicly accessible, meaning it can be implemented by anyone without restriction. The OpenDocument format aims to provide an open alternative to proprietary document formats.

The Clinical Data Interchange Standards Consortium (CDISC) is a standards developing organization (SDO) dealing with medical research data linked with healthcare, to "enable information system interoperability to improve medical research and related areas of healthcare". The standards support medical research from protocol through analysis and reporting of results and have been shown to decrease resources needed by 60% overall and 70–90% in the start-up stages when they are implemented at the beginning of the research process.

The AgMES initiative was developed by the Food and Agriculture Organization (FAO) of the United Nations and aims to encompass issues of semantic standards in the domain of agriculture with respect to description, resource discovery, interoperability and data exchange for different types of information resources.

Biodiversity informatics is the application of informatics techniques to biodiversity information, such as taxonomy, biogeography or ecology. Modern computer techniques can yield new ways to view and analyze existing information, as well as predict future situations. Biodiversity informatics is a term that was only coined around 1992 but with rapidly increasing data sets has become useful in numerous studies and applications, such as the construction of taxonomic databases or geographic information systems. Biodiversity informatics contrasts with "bioinformatics", which is often used synonymously with the computerized handling of data in the specialized area of molecular biology.

Geospatial metadata is a type of metadata applicable to geographic data and information. Such objects may be stored in a geographic information system (GIS) or may simply be documents, data-sets, images or other objects, services, or related items that exist in some other native environment but whose features may be appropriate to describe in a (geographic) metadata catalog.

The Open Packaging Conventions (OPC) is a container-file technology initially created by Microsoft to store a combination of XML and non-XML files that together form a single entity such as an Open XML Paper Specification (OpenXPS) document. OPC-based file formats combine the advantages of leaving the independent file entities embedded in the document intact and resulting in much smaller files compared to normal use of XML.

A file format is a standard way that information is encoded for storage in a computer file. It specifies how bits are used to encode information in a digital storage medium. File formats may be either proprietary or free and may be either unpublished or open.

Metadata Data about data

Metadata is "data that provides information about other data". In other words, it is "data about data". Many distinct types of metadata exist, including descriptive metadata, structural metadata, administrative metadata, reference metadata, statistical metadata and legal metadata.

EPUB E-book file format

EPUB is an e-book file format that uses the ".epub" file extension. The term is short for electronic publication and is sometimes styled ePub. EPUB is supported by many e-readers, and compatible software is available for most smartphones, tablets, and computers. EPUB is a technical standard published by the International Digital Publishing Forum (IDPF). It became an official standard of the IDPF in September 2007, superseding the older Open eBook standard.

Darwin Core is an extension of Dublin Core for biodiversity informatics. It is meant to provide a stable standard reference for sharing information on biological diversity (biodiversity). The terms described in this standard are a part of a larger set of vocabularies and technical specifications under development and maintained by Biodiversity Information Standards (TDWG).

Plazi is a Swiss-based international non-profit association supporting and promoting the development of persistent and openly accessible digital bio-taxonomic literature. Plazi is cofounder of the Biodiversity Literature Repository and is maintaining this digital taxonomic literature repository at Zenodo to provide access to FAIR data converted from taxonomic publications using the TreatmentBank service, enhances submitted taxonomic treatments by creating a version in the XML format Taxpub, and educates about the importance of maintaining open access to scientific discourse and data. It is a contributor to the evolving e-taxonomy in the field of Biodiversity Informatics.

Mercury (metadata search system)

Mercury is a distributed metadata management, data discovery and access system. It is a scientific data search system to capture and manage biogeochemical and ecological data in support of the Earth science programs funded by the United States Department of Energy (DOE) and United States Geological Survey (USGS) - Department of Interior (DOI). Mercury was originally developed for NASA, but the consortium is now supported by USGS and the DOE. Ongoing development of Mercury is done through an informal consortium at Oak Ridge National Laboratory.

The Ebbe Nielsen Challenge is an international science competition conducted annually from 2015 onwards by the Global Biodiversity Information Facility (GBIF), with a set of cash prizes that recognize researcher(s)' submissions in creating software or approaches that successfully address a GBIF-issued challenge in the field of biodiversity informatics. It succeeds the Ebbe Nielsen Prize, which was awarded annually by GBIF between 2002 and 2014. The name of the challenge honours the memory of prominent entomologist and biodiversity informatics proponent Ebbe Nielsen, who died of a heart attack in the U.S.A. en route to the 2001 GBIF Governing Board meeting.

References

  1. Darwin Core Text Guidelines
  2. Wieczorek, John; D. Bloom; R. Guralnick; S. Blum; M. Döring; R. De Giovanni; T. Robertson; D. Vieglais (2012). "Darwin Core: An Evolving Community-developed Biodiversity Data Standard". PLoS ONE . 7 (1): e29715. Bibcode:2012PLoSO...729715W. doi: 10.1371/journal.pone.0029715 . PMC   3253084 . PMID   22238640.
  3. Darwin Core Archives – How-to Guide