Computerized technical systems, used in critical applications such as aviation or power grid management, must be trustworthy to reliably deliver the expected correct service. The academic and industrial communities developing software-based systems have produced several techniques to achieve their dependability or resilience of which safety is a major attribute. Software validation and verification techniques, such as error detection and recovery mechanisms, model checking, detection of incorrect or incomplete system knowledge, and resilience to unexpected changes due to environment or system dynamics, have been developed and used.

However, as decisions usually devoted to humans are being more and more delegated to machines, sometimes running computational algorithms based on learning techniques using data, operating in complex and evolving environments, new issues have to be considered. Should the AI “black-box” justify moving away from procedures that guarantee a trusted operation of the system? This is both an ethical and a technical question. Key features such as transparency, explainability and accountability take more importance. What technical and non-technical new measures should be taken then in the design process and in the governance of these systems?