ModelOps

Last updated

ModelOps (model operations or model operationalization), as defined by Gartner, "is focused primarily on the governance and lifecycle management of a wide range of operationalized artificial intelligence (AI) and decision models, including machine learning, knowledge graphs, rules, optimization, linguistic and agent-based models" in Multi-Agent Systems. [1] "ModelOps lies at the heart of any enterprise AI strategy". [2] It orchestrates the model lifecycles of all models in production across the entire enterprise, from putting a model into production, then evaluating and updating the resulting application according to a set of governance rules, including both technical and business key performance indicators (KPI's). It grants business domain experts the capability to evaluate AI models in production, independent of data scientists. [3]

Contents

Illustrates the ModelOps cycle ModelOpsFlow.png
Illustrates the ModelOps cycle

A Forbes article promoted ModelOps: "As enterprises scale up their AI initiatives to become a true Enterprise AI organization, having full operationalized analytics capability puts ModelOps in the center, connecting both DataOps and DevOps." [4]

History

In a 2018 Gartner survey, 37% of respondents reported that they had deployed AI in some form; however, Gartner pointed out that enterprises were still far from implementing AI, citing deployment challenges. [5] Enterprises were accumulating undeployed, unused, and unrefreshed models, and manually deployed, often at a business unit level, increasing the risk exposure of the entire enterprise. [6] Independent analyst firm Forrester also covered this topic in a 2018 report on machine learning and predictive analytics vendors: “Data scientists regularly complain that their models are only sometimes or never deployed. A big part of the problem is organizational chaos in understanding how to apply and design models into applications. But another big part of the problem is technology. Models aren’t like software code because they need model management.” [7]

In December 2018, Waldemar Hummer and Vinod Muthusamy of IBM Research AI, proposed ModelOps as "a programming model for reusable, platform-independent, and composable AI workflows" on IBM Programming Languages Day. [8] In their presentation, they noted the difference between the application development lifecycle, represented by DevOps, and the AI application lifecycle. [9]

The goal for developing ModelOps was to address the gap between model deployment and model governance, ensuring that all models were running in production with strong governance, aligned with technical and business KPI's, while managing the risk. In their presentation, Hummer and Muthusamy described a programmatic solution for AI-aware staged deployment and reusable components that would enable model versions to match business apps, and which would include AI model concepts such as model monitoring, drift detection, and active learning. The solution would also address the tension between model performance and business KPI's, application and model logs, and model proxies and evolving policies. Various cloud platforms were part of the proposal. In June 2019, Hummer, Muthusamy, Thomas Rausch, Parijat Dube, and Kaoutar El Maghraoui presented a paper at the 2019 IEEE International Conference on Cloud Engineering (IC2E). [10] The paper expanded on their 2018 presentation, proposing ModelOps as a cloud-based framework and platform for end-to-end development and lifecycle management of artificial intelligence (AI) applications. In the abstract, they stated that the framework would show how it is possible to extend the principles of software lifecycle management to enable automation, trust, reliability, traceability, quality control, and reproducibility of AI model pipelines. [11] In March 2020, ModelOp, Inc. published the first comprehensive guide to ModelOps methodology. The objective of this publication was to provide an overview of the capabilities of ModelOps, as well as the technical and organizational requirements for implementing ModelOps practices. [12]

Use cases

One typical use case for ModelOps is in the financial services sector, where hundreds of time-series models are used to focus on strict rules for bias and auditability. In these cases, model fairness and robustness are critical, meaning the models have to be fair and accurate, and they have to run reliably. ModelOps automates the model lifecycle of models in production. Such automation includes designing the model lifecycle, inclusive of technical, business and compliance KPI's and thresholds, to govern and monitor the model as it runs, monitoring the models for bias and other technical and business anomalies, and updating the model as needed without disrupting the applications. ModelOps is the dispatcher that keeps all of the trains running on time and on the right track, ensuring risk control, compliance and business performance.

Another use case is the monitoring of a diabetic's blood sugar levels based on a patient's real-time data. The model that can predict hypoglycemia must be constantly refreshed with the current data, business KPI's and anomalies should be continuously monitored and must be available in a distributed environment, so the information is available on a mobile device as well as reporting to a larger system. The orchestration, governance, retraining, monitoring, and refreshing is done with ModelOps.

The ModelOps process

The ModelOps process focuses on automating the governance, management and monitoring of models in production across the enterprise, enabling AI and application developers to easily plug in lifecycle capabilities (such as bias-detection, robustness and reliability, drift detection, technical, business and compliance KPI's, regulatory constraints and approval flows) for putting AI models into production as business applications. The process starts with a standard representation of candidate models for production that includes a metamodel (the model specification) with all of the component and dependent pieces that go into building the model, such as the data, the hardware and software environments, the classifiers, and code plug-ins, and most importantly, the business and compliance/risk KPI's.

ModelOps: An evolution of MLOps

MLOps (machine learning operations) is a discipline that enables data scientists and IT professionals to collaborate and communicate while automating machine learning algorithms. It extends and expands on the principles of DevOps to support the automation of developing and deploying machine learning models and applications. [13] As a practice, MLOps involves routine machine learning (ML) models. However, the variety and uses of models have changed to include decision optimization models, optimization models, and transformational models that are added to applications. ModelOps is an evolution of MLOps that expands its principles to include not just the routine deployment of machine learning models but also the continuous retraining, automated updating, and synchronized development and deployment of more complex machine learning models. [14] ModelOps refers to the operationalization of all AI models, including the machine learning models with which MLOps is concerned. [15]

Related Research Articles

AnthillPro is a software tool originally developed and released as one of the first continuous integration servers. AnthillPro automates the process of building code into software projects and testing it to verify that project quality has been maintained. Software developers are able to identify bugs and errors earlier by using AnthillPro to track, collate, and test changes in real time to a collectively maintained body of computer code.

Business process management (BPM) is the discipline in which people use various methods to discover, model, analyze, measure, improve, optimize, and automate business processes. Any combination of methods used to manage a company's business processes is BPM. Processes can be structured and repeatable or unstructured and variable. Though not required, enabling technologies are often used with BPM.

RTTS is a professional services organization that provides software quality outsourcing, training, and resources for business applications. With offices in New York City, Philadelphia, Atlanta, and Phoenix, RTTS serves mid-sized to large corporations throughout North America. RTTS uses the software quality and test solutions from IBM, Hewlett Packard Enterprise, Microsoft and other vendors and open source tools to perform software performance testing, functional test automation, big data testing, data warehouse/ETL testing, mobile application testing, security testing and service virtualization.

<span class="mw-page-title-main">Reksoft</span>

Reksoft is a Russian multidisciplinary consulting and technology group of companies delivering a full cycle of digital transformation services, from business strategy formation and transformation management to deployment of information systems, custom software development and support, tailored industrial automation projects implementation. Reksoft Group includes the companies Reksoft, Reksoft Consulting, RNT Group, Reksoft.Labs.

HP Cloud Service Automation is cloud management software from Hewlett Packard Enterprise (HPE) that is used by companies and government agencies to automate the management of cloud-based IT-as-a-service, from order, to provision, and retirement. HP Cloud Service Automation orchestrates the provisioning and deployment of complex IT services such as of databases, middleware, and packaged applications. The software speeds deployment of application-based services across hybrid cloud delivery platforms and traditional IT environments.

HP Business Service Automation was a collection of software products for data center automation from the HP Software Division of Hewlett-Packard Company. The products could help Information Technology departments create a common, enterprise-wide view of each business service; enable the automation of change and compliance across all devices that make up a business service; connect IT processes and coordinate teams via common workflows; and integrate with monitoring and ticketing tools to form a complete, integrated business service management solution. HP now provides many of these capabilities as part of HP Business Service Management software and solutions.

HP Network Management Center (NMC) is a suite of integrated HP software used by network managers in information technology departments. The suite allows network operators to see, catalog and monitor the routers, switches, and other devices on their network. IT staff is alerted when a network device fails, and it predicts when a network node or connection point may go down. The suite was designed to address operational efficiency.

IBM OpenPages is an integrated governance, risk, and compliance (GRC) platform that enables companies to manage risk and regulatory challenges across the enterprise. The software was developed by American Computer Innovators in 1996.

Cloud management is the management of cloud computing products and services.

<span class="mw-page-title-main">Dynatrace</span> American technology company

Dynatrace, Inc. is a global technology company that provides a software observability platform based on artificial intelligence (AI) and automation. Dynatrace technologies are used to monitor, analyze, and optimize application performance, software development and security practices, IT infrastructure, and user experience for businesses and government agencies throughout the world.

Robotic process automation (RPA) is a form of business process automation that is based on software robots (bots) or artificial intelligence (AI) agents. RPA should not be confused with artificial intelligence as it is based on automotive technology following a predefined workflow. It is sometimes referred to as software robotics.

Infrastructure as code (IaC) is the process of managing and provisioning computer data center resources through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. The IT infrastructure managed by this process comprises both physical equipment, such as bare-metal servers, as well as virtual machines, and associated configuration resources. The definitions may be in a version control system, rather than maintaining the code through manual processes. The code in the definition files may use either scripts or declarative definitions, but IaC more often employs declarative approaches.

<span class="mw-page-title-main">DevOps toolchain</span> DevOps toolchain release package.

A DevOps toolchain is a set or combination of tools that aid in the delivery, development, and management of software applications throughout the systems development life cycle, as coordinated by an organisation that uses DevOps practices.

<span class="mw-page-title-main">Tricentis</span> Austrian software testing company

Tricentis is a software testing company founded in 2007 and headquartered in Austin, Texas. It provides software testing automation and software quality assurance products for enterprise software.

SAP Business Technology Platform is a platform as a service developed by SAP SE that offers a suite of services including database and data management, AI, analytics, application development, automation and integration all running on one unified platform.

<span class="mw-page-title-main">Oracle Cloud</span> Cloud computing service

Oracle Cloud is a cloud computing service offered by Oracle Corporation providing servers, storage, network, applications and services through a global network of Oracle Corporation managed data centers. The company allows these services to be provisioned on demand over the Internet.

DataOps is a set of practices, processes and technologies that combines an integrated and process-oriented perspective on data with automation and methods from agile software engineering to improve quality, speed, and collaboration and promote a culture of continuous improvement in the area of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics. DataOps applies to the entire data lifecycle from data preparation to reporting, and recognizes the interconnected nature of the data analytics team and information technology operations.

<span class="mw-page-title-main">MLOps</span> Approach to machine learning lifecycle management

MLOps or ML Ops is a paradigm that aims to deploy and maintain machine learning models in production reliably and efficiently. The word is a compound of "machine learning" and the continuous delivery practice (CI/CD) of DevOps in the software field. Machine learning models are tested and developed in isolated experimental systems. When an algorithm is ready to be launched, MLOps is practiced between Data Scientists, DevOps, and Machine Learning engineers to transition the algorithm to production systems. Similar to DevOps or DataOps approaches, MLOps seeks to increase automation and improve the quality of production models, while also focusing on business and regulatory requirements. While MLOps started as a set of best practices, it is slowly evolving into an independent approach to ML lifecycle management. MLOps applies to the entire lifecycle - from integrating with model generation, orchestration, and deployment, to health, diagnostics, governance, and business metrics.

Artificial Intelligence for IT Operations (AIOps) is a practice that uses artificial intelligence and machine learning to enhance and automate various aspects of IT operations. It is designed to optimize IT environments by analyzing large volumes of data generated by complex IT systems, including system logs, performance metrics, and network data. AIOps aims to streamline IT workflows, predict potential issues, automate incident response, and ultimately improve the performance and efficiency of enterprise IT environments.

Artificial Intelligence engineering is a technical discipline that focuses on the design, development, and deployment of AI systems. AI engineering involves applying engineering principles and methodologies to create scalable, efficient, and reliable AI-based solutions. It merges aspects of data engineering and software engineering to create real-world applications in diverse domains such as healthcare, finance, autonomous systems, and industrial automation.

References

  1. ↑ Brethenoux, Erick. "Gartner Glossary". Gartner. Retrieved 16 December 2020.
  2. ↑ Choudhary, Farhan. "Gartner Glossary". Gartner. Retrieved 6 August 2020.[ dead link ]
  3. ↑ Barot, Soyeb. "A Guidance Framework for Operationalizing Machine Learning". Gartner. Retrieved 6 August 2020.[ dead link ]
  4. ↑ Wu, Jun. "ModelOps Is The Key To Enterprise AI". Forbes. Retrieved 5 February 2021.
  5. ↑ "Gartner Survey Shows 37 Percent of Organizations Have Implemented AI in Some Form". Gartner Newsroom. 2018-01-21.
  6. ↑ Wu, Jun (2020), ModelOps Is the Key To Enterprise AI. (published 31 March 2020)
  7. ↑ "Multimodal Predictive Analytics and Machine Learning Solutions, Q3 2018" (PDF). IBM .
  8. ↑ "IBM Programming Languages Day".
  9. ↑ Waldemar Hummer and Vinod Muthusamy, A programming model for reusable, platform-independent, and composable AI workflows. IBM Research AI, 10 December 2018.
  10. ↑ "IEEE International Conference on Cloud Engineering (IC2E)".
  11. ↑ Hummer, Waldemar; Muthusamy, Vinod. ModelOps: Cloud-based Lifecycle Management for Reliable and Trusted AI. IEEE International Conference on Cloud Engineering. Parijat Dube, Kaoutar El Maghraoui. p. 1.
  12. ↑ "ModelOps Essentials: Best Practices for Success with Enterprise AI" (PDF). ModelOp. Retrieved 7 August 2020.
  13. ↑ Talagala, Nisha (2018-01-30). "Why MLOps (and not just ML) is your Business' New Competitive Frontier". AITrends.
  14. ↑ "Building ModelOps with intelligent automation for cloud-native apps accelerates growth with digital transformation initiatives". 451 Research.
  15. ↑ Barot, Soyeb. "Model Governance: The Crucial Tool for Business Success".
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.