C standard library (libc) |
---|
General topics |
Miscellaneous headers |
The C programming language has a set of functions implementing operations on strings (character strings and byte strings) in its standard library. Various operations, such as copying, concatenation, tokenization and searching are supported. For character strings, the standard library uses the convention that strings are null-terminated: a string of n characters is represented as an array of n + 1 elements, the last of which is a "NUL character" with numeric value 0.
The only support for strings in the programming language proper is that the compiler translates quoted string constants into null-terminated strings.
A string is defined as a contiguous sequence of code units terminated by the first zero code unit (often called the NUL code unit). [1] This means a string cannot contain the zero code unit, as the first one seen marks the end of the string. The length of a string is the number of code units before the zero code unit. [1] The memory occupied by a string is always one more code unit than the length, as space is needed to store the zero terminator.
Generally, the term string means a string where the code unit is of type char
, which is exactly 8 bits on all modern machines. C90 defines wide strings [1] which use a code unit of type wchar_t
, which is 16 or 32 bits on modern machines. This was intended for Unicode but it is increasingly common to use UTF-8 in normal strings for Unicode instead.
Strings are passed to functions by passing a pointer to the first code unit. Since char *
and wchar_t *
are different types, the functions that process wide strings are different than the ones processing normal strings and have different names.
String literals ("text"
in the C source code) are converted to arrays during compilation. [2] The result is an array of code units containing all the characters plus a trailing zero code unit. In C90 L"text"
produces a wide string. A string literal can contain the zero code unit (one way is to put \0
into the source), but this will cause the string to end at that point. The rest of the literal will be placed in memory (with another zero code unit added to the end) but it is impossible to know those code units were translated from the string literal, therefore such source code is not a string literal. [3]
Each string ends at the first occurrence of the zero code unit of the appropriate kind (char
or wchar_t
). Consequently, a byte string (char*
) can contain non-NUL characters in ASCII or any ASCII extension, but not characters in encodings such as UTF-16 (even though a 16-bit code unit might be nonzero, its high or low byte might be zero). The encodings that can be stored in wide strings are defined by the width of wchar_t
. In most implementations, wchar_t
is at least 16 bits, and so all 16-bit encodings, such as UCS-2, can be stored. If wchar_t
is 32-bits, then 32-bit encodings, such as UTF-32, can be stored. (The standard requires a "type that holds any wide character", which on Windows no longer holds true since the UCS-2 to UTF-16 shift. This was recognized as a defect in the standard and fixed in C++.) [4] C++11 and C11 add two types with explicit widths char16_t
and char32_t
. [5]
Variable-width encodings can be used in both byte strings and wide strings. String length and offsets are measured in bytes or wchar_t
, not in "characters", which can be confusing to beginning programmers. UTF-8 and Shift JIS are often used in C byte strings, while UTF-16 is often used in C wide strings when wchar_t
is 16 bits. Truncating strings with variable-width characters using functions like strncpy
can produce invalid sequences at the end of the string. This can be unsafe if the truncated parts are interpreted by code that assumes the input is valid.
Support for Unicode literals such as charfoo[512]="φωωβαρ";
(UTF-8) or wchar_tfoo[512]=L"φωωβαρ";
(UTF-16 or UTF-32, depends on wchar_t
) is implementation defined, [6] and may require that the source code be in the same encoding, especially for char
where compilers might just copy whatever is between the quotes. Some compilers or editors will require entering all non-ASCII characters as \xNN
sequences for each byte of UTF-8, and/or \uNNNN
for each word of UTF-16. Since C11 (and C++11), a new literal prefix u8
is available that guarantees UTF-8 for a bytestring literal, as in charfoo[512]=u8"φωωβαρ";
. [7] Since C++20 and C23, a char8_t
type was added that is meant to store UTF-8 characters and the types of u8 prefixed character and string literals were changed to char8_t
and char8_t[]
respectively.
In historical documentation the term "character" was often used instead of "byte" for C strings, which leads many[ who? ] to believe that these functions somehow do not work for UTF-8. In fact all lengths are defined as being in bytes and this is true in all implementations, and these functions work as well with UTF-8 as with single-byte encodings. The BSD documentation has been fixed to make this clear, but POSIX, Linux, and Windows documentation still uses "character" in many places where "byte" or "wchar_t" is the correct term.
Functions for handling memory buffers can process sequences of bytes that include null-byte as part of the data. Names of these functions typically start with mem
, as opposite to the str
prefix.
Most of the functions that operate on C strings are declared in the string.h
header (cstring
in C++), while functions that operate on C wide strings are declared in the wchar.h
header (cwchar
in C++). These headers also contain declarations of functions used for handling memory buffers; the name is thus something of a misnomer.
Functions declared in string.h
are extremely popular since, as a part of the C standard library, they are guaranteed to work on any platform which supports C. However, some security issues exist with these functions, such as potential buffer overflows when not used carefully and properly, causing the programmers to prefer safer and possibly less portable variants, out of which some popular ones are listed below. Some of these functions also violate const-correctness by accepting a const
string pointer and returning a non-const
pointer within the string. To correct this, some have been separated into two overloaded functions in the C++ version of the standard library.
Name | Notes |
---|---|
NULL | Macro expanding to the null pointer constant; that is, a constant representing a pointer value which is guaranteed not to be a valid address of an object in memory. |
wchar_t | Type used for a code unit in "wide" strings. On Windows, the only platform to use wchar_t extensively, it's defined as 16-bit [8] which was enough to represent any Unicode (UCS-2) character, but is now only enough to represent a UTF-16 code unit, which can be half a code point. On other platforms it is defined as 32-bit and a Unicode code point always fits. The C standard only requires that wchar_t be wide enough to hold the widest character set among the supported system locales [9] and be greater or equal in size to char, [10] |
wint_t | Integer type that can hold any value of a wchar_t as well as the value of the macro WEOF. This type is unchanged by integral promotions. Usually a 32 bit signed value. |
char8_t [11] | Part of the C standard since C23, in <uchar.h>, a type that is suitable for storing UTF-8 characters. [12] |
char16_t [13] | Part of the C standard since C11, [14] in <uchar.h>, a type capable of holding 16 bits even if wchar_t is another size. If the macro __STDC_UTF_16__ is defined as 1, the type is used for UTF-16 on that system. This is always the case in C23. [15] C++ does not define such a macro, but the type is always used for UTF-16 in that language. [16] |
char32_t [13] | Part of the C standard since C11, [17] in <uchar.h>, a type capable of holding 32 bits even if wchar_t is another size. If the macro __STDC_UTF_32__ is defined as 1, the type is used for UTF-32 on that system. This is always the case in C23. [15] C++ does not define such a macro, but the type is always used for UTF-32 in that language. [16] |
mbstate_t | Contains all the information about the conversion state required from one call to a function to the other. |
Byte string | Wide string | Description [note 1] | |
---|---|---|---|
String manipulation | strcpy [18] | wcscpy [19] | Copies one string to another |
strncpy [20] | wcsncpy [21] | Writes exactly n bytes, copying from source or adding nulls | |
strcat [22] | wcscat [23] | Appends one string to another | |
strncat [24] | wcsncat [25] | Appends no more than n bytes from one string to another | |
strxfrm [26] | wcsxfrm [27] | Transforms a string according to the current locale | |
String examination | strlen [28] | wcslen [29] | Returns the length of the string |
strcmp [30] | wcscmp [31] | Compares two strings (three-way comparison) | |
strncmp [32] | wcsncmp [33] | Compares a specific number of bytes in two strings | |
strcoll [34] | wcscoll [35] | Compares two strings according to the current locale | |
strchr [36] | wcschr [37] | Finds the first occurrence of a byte in a string | |
strrchr [38] | wcsrchr [39] | Finds the last occurrence of a byte in a string | |
strspn [40] | wcsspn [41] | Returns the number of initial bytes in a string that are in a second string | |
strcspn [42] | wcscspn [43] | Returns the number of initial bytes in a string that are not in a second string | |
strpbrk [44] | wcspbrk [45] | Finds in a string the first occurrence of a byte in a set | |
strstr [46] | wcsstr [47] | Finds the first occurrence of a substring in a string | |
strtok [48] | wcstok [49] | Splits a string into tokens | |
Miscellaneous | strerror [50] | — | Returns a string containing a message derived from an error code |
Memory manipulation | memset [51] | wmemset [52] | Fills a buffer with a repeated byte. Since C23, memset_explicit() was added to erase sensitive data. |
memcpy [53] | wmemcpy [54] | Copies one buffer to another. Since C23, memccpy() was added to efficiently concatenate strings. | |
memmove [55] | wmemmove [56] | Copies one buffer to another, possibly overlapping, buffer | |
memcmp [57] | wmemcmp [58] | Compares two buffers (three-way comparison) | |
memchr [59] | wmemchr [60] | Finds the first occurrence of a byte in a buffer | |
|
Name | Description |
---|---|
mblen [61] | Returns the number of bytes in the next multibyte character |
mbtowc [62] | Converts the next multibyte character to a wide character |
wctomb [63] | Converts a wide character to its multibyte representation |
mbstowcs [64] | Converts a multibyte string to a wide string |
wcstombs [65] | Converts a wide string to a multibyte string |
btowc [66] | Converts a single-byte character to wide character, if possible |
wctob [67] | Converts a wide character to a single-byte character, if possible |
mbsinit [68] | Checks if a state object represents initial state |
mbrlen [69] | Returns the number of bytes in the next multibyte character, given state |
mbrtowc [70] | Converts the next multibyte character to a wide character, given state |
wcrtomb [71] | Converts a wide character to its multibyte representation, given state |
mbsrtowcs [72] | Converts a multibyte string to a wide string, given state |
wcsrtombs [73] | Converts a wide string to a multibyte string, given state |
mbrtoc8 [74] | Converts the next multibyte character to a UTF-8 character, given state |
c8rtomb [75] | Converts a single code point from UTF-8 to a narrow multibyte character representation, given state |
mbrtoc16 [76] | Converts the next multibyte character to a UTF-16 character, given state |
c16rtomb [77] | Converts a single code point from UTF-16 to a narrow multibyte character representation, given state |
mbrtoc32 [78] | Converts the next multibyte character to a UTF-32 character, given state |
c32rtomb [79] | Converts a single code point from UTF-32 to a narrow multibyte character representation, given state |
These functions all need a mbstate_t object, originally in static memory (making the functions not be thread-safe) and in later additions the caller must maintain. This was originally intended to track shift states in the mb encodings, but modern ones such as UTF-8 do not need this. However these functions were designed on the assumption that the wc encoding is not a variable-width encoding and thus are designed to deal with exactly one wchar_t at a time, passing it by value rather than using a string pointer. As UTF-16 is a variable-width encoding, the mbstate_t has been reused to keep track of surrogate pairs in the wide encoding, though the caller must still detect and call mbtowc twice for a single character. [80] [81] [82] Later additions to the standard admit that the only conversion programmers are interested in is between UTF-8 and UTF-16 and directly provide this.
Byte string | Wide string | Description [note 1] |
---|---|---|
atof [83] | — | converts a string to a floating-point value ('atof' means 'ASCII to float') |
atoi atol atoll [84] | — | converts a string to an integer (C99) ('atoi' means 'ASCII to integer') |
strtof (C99) [85] strtod [86] strtold (C99) [87] | wcstof (C99) [88] wcstod [89] wcstold (C99) [90] | converts a string to a floating-point value |
strtol strtoll [91] | wcstol wcstoll [92] | converts a string to a signed integer |
strtoul strtoull [93] | wcstoul wcstoull [94] | converts a string to an unsigned integer |
|
The C standard library contains several functions for numeric conversions. The functions that deal with byte strings are defined in the stdlib.h
header (cstdlib
header in C++). The functions that deal with wide strings are defined in the wchar.h
header (cwchar
header in C++).
The functions strchr
, bsearch
, strpbrk
, strrchr
, strstr
, memchr
and their wide counterparts are not const-correct, since they accept a const
string pointer and return a non-const
pointer within the string. This has been fixed in C23. [95]
Also, since the Normative Amendment 1 (C95), atoxx
functions are considered subsumed by strtoxxx
functions, for which reason neither C95 nor any later standard provides wide-character versions of these functions. The argument against atoxx
is that they do not differentiate between an error and a 0
. [96]
Name | Source | Description |
---|---|---|
bzero [97] [98] | BSD | Fills a buffer with zero bytes, deprecated by memset |
memccpy [99] | SVID | Part of the C standard since C23, copies between two non-overlapping memory areas, stopping when a given byte is found. |
mempcpy [100] | GNU | a variant of memcpy returning a pointer to the byte following the last written byte |
strcasecmp [101] | BSD | case-insensitive version of strcmp |
strcat_s [102] | Windows | a variant of strcat that checks the destination buffer size before copying |
strcpy_s [103] | Windows | a variant of strcpy that checks the destination buffer size before copying |
strdup & strndup [104] | POSIX | Part of the C standard since C23, allocates and duplicates a string |
strerror_r [105] | POSIX 1, GNU | a variant of strerror that is thread-safe. The GNU version is incompatible with the POSIX one. |
stricmp [106] | Windows | case-insensitive version of strcmp |
strlcpy [107] | BSD | a variant of strcpy that truncates the result to fit in the destination buffer [108] |
strlcat [107] | BSD | a variant of strcat that truncates the result to fit in the destination buffer [108] |
strsignal [109] | POSIX:2008 | returns string representation of a signal code. Not thread safe. |
strtok_r [110] | POSIX | a variant of strtok that is thread-safe |
Despite the well-established need to replace strcat
[22] and strcpy
[18] with functions that do not allow buffer overflows, no accepted standard has arisen. This is partly due to the mistaken belief by many C programmers that strncat
and strncpy
have the desired behavior; however, neither function was designed for this (they were intended to manipulate null-padded fixed-size string buffers, a data format less commonly used in modern software), and the behavior and arguments are non-intuitive and often written incorrectly even by expert programmers. [108]
The most popular [lower-alpha 1] replacement are the strlcat
[111] and strlcpy
[112] functions, which appeared in OpenBSD 2.4 in December, 1998. [108] These functions always write one NUL to the destination buffer, truncating the result if necessary, and return the size of buffer that would be needed, which allows detection of the truncation and provides a size for creating a new buffer that will not truncate. For a long time they have not been included in the GNU C library (used by software on Linux), on the basis of allegedly being inefficient, [113] encouraging the use of C strings (instead of some superior alternative form of string), [114] [115] and hiding other potential errors. [116] [117] Even while glibc hadn't added support, strlcat and strlcpy have been implemented in a number of other C libraries including ones for OpenBSD, FreeBSD, NetBSD, Solaris, OS X, and QNX, as well as in alternative C libraries for Linux, such as libbsd, introduced in 2008, [118] and musl, introduced in 2011, [119] [120] and the source code added directly to other projects such as SDL, GLib, ffmpeg, rsync, and even internally in the Linux kernel. This did change in 2024, the glibc FAQ notes that as of glibc 2.38, the code has been committed [121] and thereby added. [122] These functions were standardized as part of POSIX.1-2024, [123] the Austin Group Defect Tracker ID 986 tracked some discussion about such plans for POSIX.
Sometimes memcpy
[53] or memmove
[55] are used, as they may be more efficient than strcpy
as they do not repeatedly check for NUL (this is less true on modern processors). Since they need a buffer length as a parameter, correct setting of this parameter can avoid buffer overflows.
As part of its 2004 Security Development Lifecycle, Microsoft introduced a family of "secure" functions including strcpy_s
and strcat_s
(along with many others). [124] These functions were standardized with some minor changes as part of the optional C11 (Annex K) proposed by ISO/IEC WDTR 24731. [125] These functions perform various checks including whether the string is too long to fit in the buffer. If the checks fail, a user-specified "runtime-constraint handler" function is called, [126] which usually aborts the program. [127] [128] These functions attracted considerable criticism because initially they were implemented only on Windows and at the same time warning messages started to be produced by Microsoft Visual C++ suggesting use of these functions instead of standard ones. This has been speculated by some to be an attempt by Microsoft to lock developers into its platform. [129] Experience with these functions has shown significant problems with their adoption and errors in usage, so the removal of Annex K is proposed for the next revision of the C standard. [130] Usage of memset_s
has been suggested as a way to avoid unwanted compiler optimizations. [131] [132]
strlcpy
, versus 38,644 uses of strcpy_s
(and 15,286,150 uses of strcpy
).[ citation needed ]In programming and information security, a buffer overflow or buffer overrun is an anomaly whereby a program writes data to a buffer beyond the buffer's allocated memory, overwriting adjacent memory locations.
Character encoding is the process of assigning numbers to graphical characters, especially the written characters of human language, allowing them to be stored, transmitted, and transformed using digital computers. The numerical values that make up a character encoding are known as "code points" and collectively comprise a "code space", a "code page", or a "character map".
In computer programming, a string is traditionally a sequence of characters, either as a literal constant or as some kind of variable. The latter may allow its elements to be mutated and the length changed, or it may be fixed. A string is generally considered as a data type and is often implemented as an array data structure of bytes that stores a sequence of elements, typically characters, using some character encoding. String may also denote more general arrays or other sequence data types and structures.
UTF-8 is a variable-length character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit.
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as "UCS-2" (for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names.
In computer and machine-based telecommunications terminology, a character is a unit of information that roughly corresponds to a grapheme, grapheme-like unit, or symbol, such as in an alphabet or syllabary in the written form of a natural language.
UTF-32 (32-bit Unicode Transformation Format) is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode code points, needing actually only 21 bits). In contrast, all other Unicode transformation formats are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value.
The C standard library or libc is the standard library for the C programming language, as specified in the ISO C standard. Starting from the original ANSI C standard, it was developed at the same time as the C library POSIX specification, which is a superset of it. Since ANSI C was adopted by the International Organization for Standardization, the C standard library is also called the ISO C library.
In computer programming, a null-terminated string is a character string stored as an array containing the characters and terminated with a null character. Alternative names are C string, which refers to the C programming language and ASCIIZ.
In computing, a locale is a set of parameters that defines the user's language, region and any special variant preferences that the user wants to see in their user interface. Usually a locale identifier consists of at least a language code and a country/region code. Locale is an important aspect of i18n.
printf is a C standard library function that formats text and writes it to standard output.
In Unix and Unix-like operating systems, iconv is a command-line program and a standardized application programming interface (API) used to convert between different character encodings. "It can convert from any of these encodings to any other, through Unicode conversion."
A wide character is a computer character datatype that generally has a size greater than the traditional 8-bit character. The increased datatype size allows for the use of larger coded character sets.
This article compares Unicode encodings in two types of environments: 8-bit-clean environments, and environments that forbid the use of byte values with the high bit set. Originally, such prohibitions allowed for links that used only seven data bits, but they remain in some standards and so some standard-conforming software must generate messages that comply with the restrictions. The Standard Compression Scheme for Unicode and the Binary Ordered Compression for Unicode are excluded from the comparison tables because it is difficult to simply quantify their size.
The C++ programming language has support for string handling, mostly implemented in its standard library. The language standard specifies several string types, some inherited from C, some designed to make use of the language's features, such as classes and RAII. The most-used of these is std::string.
bcrypt is a password-hashing function designed by Niels Provos and David Mazières, based on the Blowfish cipher and presented at USENIX in 1999. Besides incorporating a salt to protect against rainbow table attacks, bcrypt is an adaptive function: over time, the iteration count can be increased to make it slower, so it remains resistant to brute-force search attacks even with increasing computation power.
Microsoft was one of the first companies to implement Unicode in their products. Windows NT was the first operating system that used "wide characters" in system calls. Using the UCS-2 encoding scheme at first, it was upgraded to the variable-width encoding UTF-16 starting with Windows 2000, allowing a representation of additional planes with surrogate pairs. However Microsoft did not support UTF-8 in its API until May 2019.
In the C programming language, an escape sequence is specially delimited text in a character or string literal that represents one or more other characters to the compiler. It allows a programmer to specify characters that are otherwise difficult or impossible to specify in a literal.
crypt is a POSIX C library function. It is typically used to compute the hash of user account passwords. The function outputs a text string which also encodes the salt, and identifies the hash algorithm used. This output string forms a password record, which is usually stored in a text file.
Although C++ is one of the most widespread programming languages, many prominent software engineers criticize C++ arguing that it is overly complex and fundamentally flawed. Among the critics have been: Robert Pike, Joshua Bloch, Linus Torvalds, Donald Knuth, Richard Stallman, and Ken Thompson. C++ has been widely adopted and implemented as a systems language through most of its existence. It has been used to build many pieces of very important software.
{{cite web}}
: CS1 maint: location (link){{cite web}}
: CS1 maint: location (link){{cite web}}
: CS1 maint: location (link){{cite web}}
: CS1 maint: location (link)This [strlcpy and strlcat] API has been adopted by most modern operating systems and many standalone software packages [...]. The notable exception is the GNU standard C library, glibc, whose maintainer steadfastly refuses to include these improved APIs, labelling them "horribly inefficient BSD crap", despite prior evidence that they are faster is most cases than the APIs they replace.
Correct string handling means that you always know how long your strings are and therefore you can you memcpy (instead of strcpy).
{{cite web}}
: CS1 maint: location (link){{cite web}}
: CS1 maint: location (link)