Open-source software security is the measure of assurance or guarantee in the freedom from danger and risk inherent to an open-source software system.
There are a variety of models and metrics to measure the security of a system. These are a few methods that can be used to measure the security of software systems.
It is argued that a system is most vulnerable after a potential vulnerability is discovered, but before a patch is created. By measuring the number of days between the vulnerability and when the vulnerability is fixed, a basis can be determined on the security of the system. There are a few caveats to such an approach: not every vulnerability is equally bad, and fixing a lot of bugs quickly might not be better than only finding a few and taking a little bit longer to fix them, taking into account the operating system, or the effectiveness of the fix. 
The Poisson process can be used to measure the rates at which different people find security flaws between open and closed source software. The process can be broken down by the number of volunteers Nv and paid reviewers Np. The rates at which volunteers find a flaw is measured by λv and the rate that paid reviewers find a flaw is measured by λp. The expected time that a volunteer group is expected to find a flaw is 1/(Nv λv) and the expected time that a paid group is expected to find a flaw is 1/(Np λp). 
By comparing a large variety of open source and closed source projects a star system could be used to analyze the security of the project similar to how Morningstar, Inc. rates mutual funds. With a large enough data set, statistics could be used to measure the overall effectiveness of one group over the other. An example of such as system is as follows: 
Coverity in collaboration with Stanford University has established a new baseline for open-source quality and security. The development is being completed through a contract with the Department of Homeland Security. They are utilizing innovations in automated defect detection to identify critical types of bugs found in software.  The level of quality and security is measured in rungs. Rungs do not have a definitive meaning, and can change as Coverity releases new tools. Rungs are based on the progress of fixing issues found by the Coverity Analysis results and the degree of collaboration with Coverity.  They start with Rung 0 and currently go up to Rung 2.
The project has been analyzed by Coverity's Scan infrastructure, but no representatives from the open-source software have come forward for the results. 
At rung 1, there is collaboration between Coverity and the development team. The software is analyzed with a subset of the scanning features to prevent the development team from being overwhelmed. 
There are 11 projects that have been analyzed and upgraded to the status of Rung 2 by reaching zero defects in the first year of the scan. These projects include: AMANDA, ntp, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba, and tcl. 
A number of podcasts cover Open-source software security:
Free software is computer software distributed under terms that allow users to run the software for any purpose as well as to study, change, and distribute it and any adapted versions. Free software is a matter of liberty, not price: users and computer programmers are free to do what they want with their copies of a free software regardless of how much is paid to obtain the program. Computer programs are deemed "free" if they give both programmers and end-users ultimate control over the software and, subsequently, over their devices.
In computing, source code is any collection of code, with or without comments, written using a human-readable programming language, usually as plain text. The source code of a program is specially designed to facilitate the work of computer programmers, who specify the actions to be performed by a computer mostly by writing source code. The source code is often transformed by an assembler or compiler into binary machine code that can be executed by the computer. The machine code might then be stored for execution at a later time. Alternatively, source code may be interpreted and thus immediately executed.
A software bug is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. The process of finding and fixing bugs is termed "debugging" and often uses formal techniques or tools to pinpoint bugs, and since the 1950s, some computer systems have been designed to also deter, detect or auto-correct various computer bugs during operations.
In software development, Linus's law is the assertion that "given enough eyeballs, all bugs are shallow".
Linux malware includes viruses, Trojans, worms and other types of malware that affect the Linux operating system. Linux, Unix and other Unix-like computer operating systems are generally regarded as very well-protected against, but not immune to, computer viruses.
Nmap is a free and open-source network scanner created by Gordon Lyon. Nmap is used to discover hosts and services on a computer network by sending packets and analyzing the responses.
A patch is a set of changes to a computer program or its supporting data designed to update, fix, or improve it. This includes fixing security vulnerabilities and other bugs, with such patches usually being called bugfixes or bug fixes. Patches are often written to improve the functionality, usability, or performance of a program.
In computing, hardening is usually the process of securing a system by reducing its surface of vulnerability, which is larger when a system performs more functions; in principle a single-function system is more secure than a multipurpose one. Reducing available ways of attack typically includes changing default passwords, the removal of unnecessary software, unnecessary usernames or logins, and the disabling or removal of unnecessary services.
Nessus is a proprietary vulnerability scanner developed by Tenable, Inc.
In computer security, a vulnerability is a weakness which can be exploited by a threat actor, such as an attacker, to cross privilege boundaries within a computer system. To exploit a vulnerability, an attacker must have at least one applicable tool or technique that can connect to a system weakness. In this frame, vulnerabilities are also known as the attack surface.
A penetration test, colloquially known as a pen test, pentest or ethical hacking, is an authorized simulated cyberattack on a computer system, performed to evaluate the security of the system. Not to be confused with a vulnerability assessment. The test is performed to identify both weaknesses, including the potential for unauthorized parties to gain access to the system's features and data, as well as strengths, enabling a full risk assessment to be completed.
Application security encompasses measures taken to improve the security of an application often by finding, fixing and preventing security vulnerabilities. Different techniques are used to surface such security vulnerabilities at different stages of an applications lifecycle such as design, development, deployment, upgrade, maintenance.
The Metasploit Project is a computer security project that provides information about security vulnerabilities and aids in penetration testing and IDS signature development. It is owned by Boston, Massachusetts-based security company Rapid7.
Dynamic program analysis is the analysis of computer software that is performed by executing programs on a real or virtual processor. For dynamic program analysis to be effective, the target program must be executed with sufficient test inputs to cover almost all possible outputs. Use of software testing measures such as code coverage helps ensure that an adequate slice of the program's set of possible behaviors has been observed. Also, care must be taken to minimize the effect that instrumentation has on the execution of the target program. Dynamic analysis is in contrast to static program analysis. Unit tests, integration tests, system tests and acceptance tests use dynamic testing.
OpenPAM is a BSD-licensed implementation of PAM used by FreeBSD, NetBSD, DragonFly BSD and macOS , and offered as an alternative to Linux PAM in certain Linux distributions.
Heartbleed is a security bug in the OpenSSL cryptography library, which is a widely used implementation of the Transport Layer Security (TLS) protocol. It was introduced into the software in 2012 and publicly disclosed in April 2014. Heartbleed may be exploited regardless of whether the vulnerable OpenSSL instance is running as a TLS server or client. It results from improper input validation in the implementation of the TLS heartbeat extension. Thus, the bug's name derives from heartbeat. The vulnerability is classified as a buffer over-read, a situation where more data can be read than should be allowed.
Shellshock, also known as Bashdoor, is a family of security bugs in the Unix Bash shell, the first of which was disclosed on 24 September 2014. Shellshock could enable an attacker to cause Bash to execute arbitrary commands and gain unauthorized access to many Internet-facing services, such as web servers, that use Bash to process requests.
Meltdown is a hardware vulnerability affecting Intel x86 microprocessors, IBM POWER processors, and some ARM-based microprocessors. It allows a rogue process to read all memory, even when it is not authorized to do so.
CodeSonar is a static code analysis tool from GrammaTech. CodeSonar is used to find and fix bugs and security vulnerabilities in source and binary code. It performs whole-program, inter-procedural analysis with abstract interpretation on C, C++, C#, Java, as well as x86 and ARM binary executables and libraries. CodeSonar is typically used by teams developing or assessing software to track their quality or security weaknesses. CodeSonar supports Linux, BSD, FreeBSD, NetBSD, MacOS and Windows hosts and embedded operating systems and compilers.