These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
Type
Origin
Scope
SUBMIT A TOOL
If you have a tool that you think should be featured in the Catalogue of AI Tools & Metrics, we would love to hear from you!
SUBMITCarefulAI: Prompt-LLM Improvement Method (PLIM)
EducationalUnited KingdomUploaded on Dec 9, 2024PLIM is designed to make benchmarking and continuous monitoring of LLMs safer and more fit for purpose. This is particularly important in high-risk environments, e.g. healthcare, finance, insurance and defence. Having community-based prompts to validate models as fit for purpose is safer in a world where LLMs are not static.
Objective(s)
MLE-bench
TechnicalInternationalUploaded on Dec 6, 2024Evaluating machine learning agents on machine learning engineering.
Objective(s)
HarmBench
TechnicalInternationalUploaded on Nov 5, 2024A fast, scalable, and open-source framework for evaluating automated red teaming methods and LLM attacks/defenses. HarmBench has out-of-the-box support for transformers-compatible LLMs, numerous closed-source APIs, and several multimodal models.
Objective(s)
Probe: End-to-end AI Security Platform
TechnicalUnited StatesUploaded on Aug 2, 2024AI Security Platform for GenAI and Conversational AI applications. Probe enables security officers and developers identify, mitigate, and monitor AI system security.
Objective(s)
IEEE 3301-2022 - Adoption of Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) Technical Specification Artificial Intelligence Framework (AIF) Version 1
ProceduralUploaded on Jul 1, 2024The MPAI AI Framework (MPAI-AIF) Technical Specification specifies architecture, interfaces, protocols and Application Programming Interfaces (API) of an AI Framework (AIF), especially designed for execution of AI-based implementations, but also suitable for mixed AI and traditional data processing workflows.
Objective(s)
IEEE 3302-2022 - Adoption of Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) Technical Specification Context-based Audio Enhancement (CAE) Version 1.2
ProceduralUploaded on Jul 1, 2024MPAI-CAE V1.4 is a collection of four use cases to improve the user audio experience in a variety of situations.
Objective(s)
IEEE 3300-2022 - Adoption of Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) Technical Specification Multimodal Conversation Version 1.1
ProceduralUploaded on Jul 1, 2024This standard adopts MPAI Technical Specification Version 1.2 as an IEEE Standard.
Objective(s)
ITU-T Y.3177 - Architectural framework for artificial intelligence-based network automation for resource and fault management in future networks including IMT-2020
ProceduralUploaded on Jul 1, 2024This Recommendation specifies an architectural framework for network automation based on artificial intelligence (AI) for resource and fault management in future networks, including international mobile telecommunications-2020 (IMT-2020).
Objective(s)
ITU-T Y.3172 - Architectural framework for machine learning in future networks including IMT-2020
ProceduralUploaded on Jul 1, 2024Recommendation ITU-T Y.3172 specifies an architectural framework for machine learning (ML) in future networks including IMT-2020.
Objective(s)
ITU-T Y.3179 - Architectural framework for machine learning model serving in future networks including IMT-2020
ProceduralUploaded on Jul 1, 2024This Recommendation provides an architectural framework for machine learning (ML) models serving in future networks including IMT-2020, i.
Objective(s)
DIN SPEC 92001-1 - Artificial Intelligence - Life Cycle Processes and Quality Requirements - Part 1: Quality Meta Model
ProceduralUploaded on Jul 2, 2024This DIN SPEC in accordance with the PAS procedure has been drawn up by a DIN SPEC (PAS)-consortium set up on a temporary basis.
Objective(s)
IEC 62243:2012/IEEE Std 1232-2010 - Artificial intelligence exchange and service tie to all test environments (AI-ESTATE)
ProceduralUploaded on Jul 2, 2024The purpose of AI-ESTATE is to standardize interfaces for functional elements of an intelligent diagnostic reasoner and representations of diagnostic knowledge and data for use by such diagnostic reasoners
Objective(s)
ITU-T Y.3654 - Big data driven networking - Machine learning mechanism
ProceduralUploaded on Jul 2, 2024Recommendation ITU-T Y.3654 specifies the mechanisms of machine learning in big data driven networking (bDDN).
Objective(s)
ITU-T F.748.12 - Deep learning software framework evaluation methodology
ProceduralUploaded on Jul 2, 2024Evaluating deep learning software frameworks to help manufactures take full advantage of certain frameworks and avoid the disadvantages of others.
Objective(s)
ITU-T H.862.5 - Emotion enabled multimodal user interface based on artificial neural networks
ProceduralUploaded on Jul 2, 2024This Recommendation describes the functional entities and architecture for emotion enabled multimodal user interface based on artificial neural networks.
Objective(s)
ETSI GR ENI 010 V 1.1.1 - Experiential Networked Intelligence (ENI) - Evaluation of categories for AI application to Networks
Uploaded on Jul 2, 2024The purpose of this work item is to specify quantitative evaluation criteria of network autonomicity categories, which is defined in the published GR ENI 007.
Objective(s)
ETSI GR ENI 017 V 2.1.1 - Experiential Networked Intelligence (ENI) - Overview of Prominent Control Loop Architectures
ProceduralUploaded on Jul 2, 2024The purpose of the present document is to provide information on prominent control loop architectures that can be used in modular system design.
Objective(s)
ITU-T Y.3173 - Framework for evaluating intelligence levels of future networks including IMT-2020
ProceduralUploaded on Jul 2, 2024Recommendation ITU-T Y.3173 specifies a framework for evaluating the intelligence of future networks including IMT-2020 and a method for evaluating the intelligence levels of future networks including IMT-2020 is introduced.
Objective(s)
ITU-T M.3080 - Framework of artificial intelligence enhanced telecom operation and management (AITOM)
ProceduralUploaded on Jul 2, 2024This Recommendation provides a framework for artificial intelligence (AI) enhanced telecom operation and management (AITOM).
Objective(s)
ITU-T Y.3175 - Functional architecture of machine learning-based quality of service assurance for the IMT-2020 network
ProceduralUploaded on Jul 2, 2024This Recommendation specifies a functional architecture of quality of service (QoS) assurance based on machine learning (ML) for the international mobile telecommunications-2020 (IMT-2020) network.
Objective(s)
