DLL Hell

DLL hell is an umbrella term for the complications that arise when one works with dynamic-link libraries (DLLs) used with older Microsoft Windows operating systems, ^[1] particularly legacy 16-bit editions, which all run in a single memory space. DLL hell can appear in many different ways, wherein affected programs may fail to run correctly, if at all. It is the Windows ecosystem-specific form of the general concept dependency hell.

Problems

DLLs are Microsoft's implementation of shared libraries. Shared libraries allow common code to be bundled into a wrapper, the DLL, which is used by any application software on the system without loading multiple copies into memory. A simple example might be the GUI text editor, which is widely used by many programs. By placing this code in a DLL, all the applications on the system can use it without using more memory. This contrasts with static libraries, which are functionally similar but copy the code directly into the application. In this case, every application grows by the size of all the libraries it uses, and this can be quite large for modern programs.

The problem arises when the version of the DLL on the computer is different than the version that was used when the program was being created. DLLs have no built-in mechanism for backward compatibility, and even minor changes to the DLL can render its internal structure so different from previous versions that attempting to use them will generally cause the application to crash. Static libraries avoid this problem because the version that was used to build the application is included inside it, so even if a newer version exists elsewhere on the system, this does not affect the application.

A key reason for the version incompatibility is the structure of the DLL file. The file contains a directory of the individual methods (procedures, routines, etc.) contained within the DLL and the types of data they take and return. Even minor changes to the DLL code can cause this directory to be re-arranged, in which case an application that calls a particular method believing it to be the 4th item in the directory might end up calling an entirely different and incompatible routine, which would normally cause the application to crash.

There are several problems commonly encountered with DLLs, especially after numerous applications have been installed and uninstalled on a system. The difficulties include conflicts between DLL versions, difficulty in obtaining required DLLs, and having many unnecessary DLL copies.

Solutions to these problems were known even while Microsoft was writing the DLL system.^{[ citation needed ]} These have been incorporated into the .NET replacement, "Assemblies".

Incompatible versions

A particular version of a library can be compatible with some programs that use it and incompatible with others. Windows has been particularly vulnerable to this because of its emphasis on dynamic linking of C++ libraries and Object Linking and Embedding (OLE) objects. C++ classes export many methods, and a single change to the class, such as a new virtual method, can make it incompatible with programs that were built against an earlier version. Object Linking and Embedding has very strict rules to prevent this: interfaces are required to be stable, and memory managers are not shared. This is insufficient, however, because the semantics of a class can change. A bug fix for one application may result in the removal of a feature from another. Before Windows 2000, Windows was vulnerable to this because the COM class table was shared across all users and processes. Only one COM object in one DLL/EXE could be declared as having a specific global COM Class ID on a system. If any program needed to create an instance of that class, it got whatever was the current centrally registered implementation. As a result, an installation of a program that installed a new version of a common object might inadvertently break other programs that were previously installed.

DLL stomping

A common and troublesome problem occurs when a newly installed program overwrites a working system DLL with an earlier, incompatible version. Early examples of this were the ctl3d.dll and ctl3dv2.dll libraries for Windows 3.1: Microsoft-created libraries that third-party publishers would distribute with their software, but each distributing the version they developed with rather than the most recent version. ^[2] DLL stomping occurs because:

• Microsoft in the past distributed runtime DLLs as shared system components ^[3] (originally C:\WINDOWS and C:\WINDOWS\SYSTEM), as a way of efficiently sharing code in a shared-memory OS with limited RAM and disk space. Consequently, third-party developers also distributed these in such a manner.
• Application installers are typically executed in a privileged security context that has access to install DLLs into the system directories and to edit the system registry to register new DLLs as COM objects. A poorly written or misconfigured installer can therefore downgrade a system library on legacy versions of Windows, on which Windows File Protection or Windows Resource Protection does not roll back the change. On Windows Vista and later, only the "trusted installer" account can make changes to core operating-system libraries.
• Windows applications were permitted to include OS updates in their own installation programs. That is, many Microsoft DLLs are redistributable, meaning that the applications can include them if they need the services of the particular libraries.
• Before Windows Installer, Windows installers historically were commercial products; many people attempted to write their own installers, overlooking or mishandling versioning problems in the process.^{[ citation needed ]}
• Some development environments did not automatically add a version resource in their compiled libraries, so many developers overlooked this aspect. Checking file dates, overwriting existing files or skipping the copy operation if the DLL was already installed were the only options available instead of correct versioning.^{[ citation needed ]}
• Sometimes, the OS itself removed or replaced DLLs with older or obsolete versions. For example, Windows 2000 would install black-and-white printer DLLs on top of color-aware DLLs, if a black-and-white printer was installed after the color printer. ^[4]

Incorrect COM registration

In COM and other parts of Windows, prior to the introduction of side-by-side registry-free assemblies, ^[5] the Registry was used for determining which underlying DLL to use. If a different version of a module was registered, this DLL would be loaded instead of the expected one. This scenario could be caused by conflicting installations that register different versions of the same libraries, in which case the last installation would prevail.

Shared in-memory modules

16-bit versions of Windows (and Windows on Windows) load only one instance of any given DLL; all applications reference the same in-memory copy, until no applications are using it and it is unloaded from memory. (For 32-bit and 64-bit versions of Windows, inter-process sharing occurs only where different executables load a module from exactly the same directory; the code but not the stack is shared between processes through a process called "memory mapping".) Thus, even when the desired DLL is located in a directory where it can be expected to be found, such as in the system directory or the application directory, neither of these instances will be used if another application has started with an incompatible version from a third directory. This issue can manifest itself as a 16-bit application error that occurs only when applications are started in a specific order.

Lack of serviceability

In direct conflict with the DLL stomping problem: If updates to a DLL do not affect all applications that use it, then it becomes much harder to "service" the DLL – that is, to eliminate problems that exist in the current versions of the DLL. (Security fixes are a particularly compelling and painful case.) Instead of fixing just the latest version of the DLL, the implementor must ideally make their fixes and test them for compatibility on every released version of the DLL.

Causes

DLL incompatibility has been caused by:

• Memory constraints, combined with lack of separation of process memory space in 16-bit versions of Windows;
• Lack of enforced standard versioning, naming, and file-system location schemata for DLLs;
• Lack of an enforced standard method for software installation and removing (package management);
• Lack of centralized authoritative support for DLL application binary interface management and safeguards, allowing incompatible DLLs with the same file name and internal version numbers to be released;
• Oversimplified management tools, preventing the identification of changed or problematic DLLs by users and administrators;
• Developers breaking backward compatibility of functions in shared modules;
• Microsoft releasing out-of-band updates to operating-system runtime components;
• Inability of earlier versions of Windows to run side-by-side conflicting versions of the same library;
• Reliance on the current directory or %PATH% environment variable, both of which vary over time and from system to system, to find dependent DLLs (instead of loading them from an explicitly configured directory);
• Developers re-using the ClassIDs from sample applications for the COM interfaces of their applications, rather than generating their own new GUIDs.

DLL hell was a very common phenomenon on pre-Windows NT versions of Microsoft operating systems, the primary cause being that the 16-bit operating systems did not restrict processes to their own memory space, thereby not allowing them to load their own version of a shared module that they were compatible with. Application installers were expected to be good citizens and verify DLL version information before overwriting the existing system DLLs. Standard tools to simplify application deployment (which always involves shipping the dependent operating-system DLLs) were provided by Microsoft and other 3rd-party tools vendors. Microsoft even required application vendors to use a standard installer and have their installation program certified to work correctly, before being granted use of the Microsoft logo. The good-citizen installer approach did not mitigate the problem, as the rise in popularity of the Internet provided more opportunities to obtain non-conforming applications.

Use by malware

The ambiguity with which DLLs that are not fully qualified can be loaded in the Windows operating system has been exploited by malware in recent years,^{[ when? ]} opening a new class of vulnerability that affects applications from many different software vendors, as well as Windows itself. ^[6]

Solutions

Various forms of DLL hell have been solved or mitigated over the years.

Static linking

A simple solution to DLL hell in an application is to statically link all the libraries, i.e. to include the library version required in the program, instead of picking up a system library with a specified name. ^[7] This is common in C/C++ applications, where, instead of having to worry about which version of MFC42.DLL is installed, the application is compiled to be statically linked against the same libraries. This eliminates the DLLs entirely and is possible in standalone applications using only libraries that offer a static option, as Microsoft Foundation Class Library does. However, the main purpose of DLLs – runtime library sharing between programs to reduce memory overhead – is sacrificed; duplicating library code in several programs creates software bloat and complicates the deployment of security fixes or newer versions of dependent software.

Windows File Protection

The DLL overwriting problem (referred to as DLL Stomping by Microsoft) was somewhat reduced with Windows File Protection (WFP), ^[8] which was introduced in Windows 2000. ^[9] This prevents unauthorized applications from overwriting system DLLs, unless they use the specific Windows APIs that permit this. There may still be a risk that updates from Microsoft are incompatible with existing applications, but this risk is typically reduced in current versions of Windows through the use of side-by-side assemblies.

Third-party applications cannot stomp on OS files unless they bundle legitimate Windows updates with their installer, or if they disable the Windows File Protection service during installation, and on Windows Vista or later also take ownership of system files and grant themselves access. The SFC utility could revert these changes at any time.

Running conflicting DLLs simultaneously

The solutions here consist of having different copies of the same DLLs for each application, both on disk and in memory.

An easy manual solution to conflicts was placing the different versions of the problem DLL into the applications' folders, rather than a common system-wide folder. This works in general as long as the application is 32-bit or 64-bit, and that the DLL does not use shared memory. In the case of 16-bit applications, the two applications cannot be executed simultaneously on a 16-bit platform, or in the same 16-bit virtual machine under a 32-bit operating system. OLE prevented this before Windows 98 SE/2000, because earlier versions of Windows had a single registry of COM objects for all applications.

Windows 98 SE/2000 introduced a solution called side-by-side assembly , ^[10] which loads separate copies of DLLs for each application that requires them (and thus allows applications that require conflicting DLLs to run simultaneously). This approach eliminates conflicts by allowing applications to load unique versions of a module into their address space, while preserving the primary benefit of sharing DLLs between applications (i.e. reducing memory use) by using memory mapping techniques to share common code between different processes that do still use the same module. Yet DLLs using shared data between multiple processes cannot take this approach. ^[11] One negative side effect is that orphaned instances of DLLs may not be updated during automated processes.

Portable applications

Depending on the application architecture and runtime environment, portable applications may be an effective way to reduce some DLL problems, since every program bundles its own private copies of any DLLs it requires. ^[9] The mechanism relies on applications not fully qualifying the paths to dependent DLLs when loading them, and the operating system searching the executable directory before any shared location. ^[12] However this technique can also be exploited by malware, ^[13] and the increased flexibility may also come at the expense of security if the private DLLs are not kept up to date with security patches in the same way that the shared ones are.

Application virtualization can also allow applications to run in a "bubble", which avoids installing DLL files directly into the operating system.

Other countermeasures

There are other countermeasures to avoid DLL hell, some of which may have to be used simultaneously; some other features that help to mitigate the problem are:

• Installation tools are now bundled into Microsoft Visual Studio, one of the main environments for Windows development. These tools perform version checking before DLL installation, and can include predefined installation packages in a .MSI installation. This allows third party applications to integrate OS component updates without having to write their own installers for these components.
• System Restore can recover a system from a bad installation, including registry damage. Although this does not prevent the problem, it facilitates recovery therefrom.
• WinSxS (Windows Side-by-Side) directory, which allows multiple versions of the same libraries to co-exist.
• Run 16-bit applications in a separate memory space under a 32-bit version of Windows to allow two applications to use conflicting versions of the same DLL at the same time.
• Use a version of Windows that includes Windows File Protection. Windows Me and Windows 2000, both released in 2000, support this form of system file protection, as do Windows XP and Windows Server 2003. Its replacement, Windows Resource Protection, was introduced in Windows Vista and Windows Server 2008, and uses a different method of protecting system files from being changed.
• Registration-free COM: Windows XP introduced a new mode of COM object registration called "Registration-free COM". This feature makes it possible for applications that need to install COM objects to store all the required COM registry information in the application's own directory, instead of in the global system registry. Thus, it provides a mechanism for multiple versions of the same DLL to be registered at the same time by multiple applications (Microsoft calls this "Side-by-Side Assembly" ^[14] ). DLL hell can be substantially avoided using Registration-free COM, the only limitation being it requires at least Windows XP or later Windows versions and that it must not be used for EXE COM servers or system-wide components such as MDAC, MSXML, DirectX or Internet Explorer.
• Shipping the operating system with a capable package management system that is able to track the DLL dependencies, encouraging the use of the package manager and discouraging manual installation of DLLs. Windows Installer, included with Windows Me, Windows 2000 and all later versions provides this functionality.
• Having a central database or authority for DLL conflict resolution and software distribution. Changes to a library can be submitted to this authority; thus, it can make sure compatibility is preserved in the developed branches. If some older software is incompatible with the current library, the authority can provide a compatibility interface for it, or bundle the old version as a distinct package.
• If software developers need to customize a library, and if the main library release is unlikely to incorporate the changes that they need, they can ship the customized DLL for the program's private use (commonly by placing it in the program's private directory) or statically link the program against the customized library.
• While DLLs are best for modularizing applications and the system's components and as third-party libraries, their usage is not imperative in all cases on modern systems where memory is no longer a constraint. For example, if an application needs a library that will not be used anywhere else, it can be linked statically, with no space penalty and with a speed gain.
• Windows Vista and later use a special TrustedInstaller service to install operating system files. Other user accounts, including the SYSTEM, have no access to overwrite core system binaries. Windows 7 expands this functionality to some critical parts of the Registry.

See also

• Dependency hell
• Extension conflict
• Portable application
• Portable application creators
• JAR hell

References

1. "Avoiding DLL Hell: Introducing Application Metadata in the Microsoft .NET Framework". Microsoft. October 2000.
2. "A summary of CTL3D.DLL articles in Microsoft Support Knowledge Base". Microsoft.
3. Redistribution of the shared C runtime component in Visual C++ 2005 and in Visual C++ .NET.
4. KB 830490: HP Color LaserJet printer prints only in grayscale or in black-and-white on your Windows 2000 SP4-based computer.
5. Leslie Muller; Steve White (July 2005). "Registration-Free Activation of COM Components: A Walkthrough". Microsoft.
6. "Secure Loading of Libraries to Prevent DLL Preloading Attacks". Microsoft. 2011-06-11. Retrieved 2011-07-19.
7. Pfeiffer, Tim (1998-06-01). "Windows DLLs: Threat or Menace?". Dr. Dobb's Journal. Archived from the original on 2010-08-07. Retrieved 2010-07-07.
8. Windows File Protection and Windows.
9. Anderson, Rick (2000-01-11). "The End of DLL Hell". microsoft.com. Archived from the original on 2001-06-05. Retrieved 2010-07-07.
10. "Implementing Side-by-Side Component Sharing in Applications (Expanded)". Microsoft. Archived from the original on 10 December 2006. Retrieved 3 January 2013.
11. "How do I share data in my DLL with an application or with other DLLs?". Microsoft . Retrieved 2008-11-11.
12. Desitter, Arnaud (2007-06-15). "Using static and shared libraries across platforms; Row 9: Library Path". ArnaudRecipes. Archived from the original on 2008-06-01. Retrieved 2010-07-07.
13. "Secure loading of libraries to prevent DLL preloading attacks". Microsoft. Retrieved 16 Feb 2013.
14. Side-by-side Assemblies (Windows)

External links

• Getting Out of DLL Hell on Microsoft TechNet
• Simplifying Deployment and Solving DLL Hell with the .NET Framework on MSDN
• Avoiding DLL Hell: Introducing Application Metadata in the Microsoft .NET Framework by Matt Pietrek
• Dr. Dobb's on DLL hell (details on LoadLibraryEx)
• Joel on Software discussion Archived 2018-10-30 at the Wayback Machine
• Article on DLL hell