напраВления модернизации образоВания The CDIO-FCDI-FFCD Rubrics for Evaluation of Three-Cycle Engineering Programs

. The aim of the paper is to propose Rubrics for self-evaluation of graduate and postgraduate engineering programs based on the FCDI (Forecast, Conceive, Design, Implement) Standards and FFCD (Foresight, Forecast, Conceive, Design) Standards by analogy with Rubrics for self-evaluation of undergraduate engineering programs based on the CDIO (Conceive, Design, Implement, Operate) Standards. The FCDI Standards and FFCD Standards were developed for Master’s and Doctoral engineering programs as a result of the CDIO approach evolution and by analogy with the CDIO Standards originally developed for Bachelor’s engineering programs. The CDIO/FCDI/FFCD Standards are recommended for the design and implementation of three-cycle engineering programs to train graduates for complex, innovative and research engineering activities, respectively, taking into account the features of the division of labor in the engineering profession. The 6-level scale Rubrics are helpful for evaluation the degree of Bachelor’s, Master’s and Doctoral engineering programs compliance with the recommendations of the CDIO, FCDI and FFCD Standards, respectively.


Introduction
At the turn of the 20th and 21st centuries, the CDIO concept of improving engineering education was developed, taking into account the realities of time and providing a balance between theory and practice. This concept was aimed at training engineers capable of working on the CDIO (Conceive, Design, Implement, Operate) stages of the life cycle of products, processes and systems. The CDIO Standards, containing recommendations on the planning of graduate's learning outcomes, curriculum design, learning technologies application, as well as the creation of material resources for program support and faculty development were offered to universities implementing basic (Bachelor's) engineering programs [1]. From the very beginning each of the 12 CDIO Standards was accompa-nied by Description, Rationale and Rubric for engineering program evaluation aligned with the CDIO Standards [2]. Universities that decide to implement the recommendations of the CDIO Standards using Rubrics carry out a selfevaluation of the programs for compliance with these standards. The Rubrics have been designed deliberately to encourage planning and allow universities various styles of CDIO Standards implementation and adoption. The Rubric is a table with the help of which on a 6-level scale it is possible to determine the degree of compliance of an engineering program with the recommendations of one or another CDIO Standard.
The CDIO Standards have become popular in universities of various countries. Currently, more than 140 universities located on all continents have united in the Worldwide CDIO Ini-tiative organization and exchange best practices in applying the CDIO approach to the design and implementation of engineering programs. More than a dozen Russian universities are among the participants of the organization [3]. Some of them actively use the CDIO Standards for the modernization of engineering programs and share their experience with foreign colleagues [4][5][6][7][8][9][10]. Universities that implement the CDIO Standards use Rubrics both for self-evaluation of the programs as a whole and for selfassessment of the curriculum elements ensuring the achievement of intended learning outcomes by students [11][12][13][14][15].
In 2014, the CDIO Standards have been revised and Rubrics have been further modified. However, these modifications have been relatively minor and have not changed the scope or the main contents of the standards. At the same time, as a result of widespread use of the CDIO Standards, proposals for their more significant modification began to appear [16; 17] including proposals to supplement the existing standards with new ones related to digital learning, diversity, engineering entrepreneurship, engineering ethics, internationalization & mobility, leadership, Master-level CDIO programs, multidisciplinary, collaborative skills, research-integrated education, sustainable development, etc. The need for the evolution of the CDIO approach has become apparent. The issue is being discussed in publications and at regular meetings of collaborators -participants of the Worldwide CDIO Initiative.

CDIO-FCDI-FFCD Models
In 2013, some Russian universities -participants of the Worldwide CDIO Initiative became members of "elite" group of 15 leading Russian universities -participants of so called "5-100 Russian academic excellence project". The goal of the project was to improve the quality and prestige of Russian higher education and bring at least 5 Russian universities from among the project participants into the 100 best universities in the world according to the three most authoritative world rankings: QS, TIMES and ARWU (http://5top100.com/). After some time, 6 more Russian universities entered the project.
As part of the "5-100 Russian academic excellence project", 21 Russian universities have focused on graduate and postgraduate higher education including Master's (MSc) and Doctoral (PhD) programs in engineering and technology. To implement the strategy focusing on graduate and postgraduate engineering education, universities needed a conceptual and methodological basis for improving the quality of MSc and PhD engineering programs. The CDIO approach could become such a basis. However, the CDIO Standards, originally developed for basic (undergraduate) engineering education and well-proven in the process of upgrading Bachelor's (BEng) programs, did not fully comply with MSc and PhD engineering programs. At Tomsk Polytechnic University with the participation of representatives of other Russian universities -Worldwide CDIO Initiative collaborators, relevant studies were conducted and it was proposed to evolve the CDIO approach and adapt it to graduate and postgraduate engineering education [18]. Further developments led to the creation of the CDIO-FCDI-FFCD Models for three-cycle engineering education [19].
Firstly, by analogy with the CDIO Syllabus (CDIO Standard 2), lists of intended learning outcomes (LOs) for graduates of MSc and PhD engineering programs were developed, which, unlike BEng programs graduates trained for complex engineering, should be focused on innovative and research engineering activities, respectively. In the formation of a list of intended LOs for Master's engineering programs, it was proposed to use the abbreviation FCDI (Forecast, Conceive, Design, Implement) instead of the abbreviation CDIO (Conceive, Design, Implement, Operate). The absence of "Operate" in a new abbreviation indicates that this kind of engineering activity (operation and maintenance of products, processes and systems) is not a priority for MSc program graduates. The presence of "Forecast" emphasizes the importance of forecasting potential needs of society in new products, processes and systems. In the formation of a list of intended LOs for Doctoral engineering programs it was proposed to use the abbreviation FFCD (Fore-sight, Forecast, Conceive, Design). The absence of "Implement" in the abbreviation indicates that participation in manufacturing is not a prior-

CDIO-FCDI-FFCD Standard 1
The CDIO/FCDI/FFCD Standard 1 regarding the context of undergraduate/graduate/ postgraduate engineering education is presented in Table 1. Table 2 shows the priority activities of the Bachelor's, Master's and Doctoral program graduates at the Foresight -Forecast -Conceive -Design -Implement -Operate stages, taking into account the system of division of labor in the engineering profession.
The degree of compliance of three-cycle engineering programs with the CDIO/FCDI/ FFCD Standard 1 is determined with the use of a 6-point scale based on the criteria presented in the Rubric (Table 3). Similar Rubrics ( Tables  6, 8, 10,12,14,16,18,20,22,24,26) are used to determine the degree of compliance of three-cycle engineering programs with other standards (Tables 4,7,9,11,13,15,17,19,21,23,25). There is a documented evidence that the CDIO/FCDI/FFCD principle is the context of the Bachelor/Master/ Doctor program and is implemented in all years of the program 3 The CDIO/FCDI/FFCD principle is implemented in one or more years of the Bachelor/Master/Doctor program 2 There is an explicit plan to transition to a CDIO/FCDI/FFCD context for the Bachelor/ Master/Doctor program 1 There is a willingness to adopt to a CDIO/FCDI/FFCD context for the Bachelor/Master/Doctor program 0 There is no plan to adopt the principle that CDIO/FCDI/FFCD is the context of education for the Bachelor/ Master/Doctor program Table 4 CDIO

CDIO FCDI FFCD
Specific, detailed learning outcomes for personal and interpersonal skills, and product, process, and system building skills, as well as disciplinary knowledge, consistent with program goals and validated by program stakeholders Specific, detailed learning outcomes for personal and interpersonal skills, and innovative product, process, and system design and development skills based on forecasting stakeholder needs, as well as interdisciplinary knowledge and teaching skills, consistent with program goals and validated by program stakeholders Specific, detailed learning outcomes for personal and interpersonal skills, and abilities to create scientific basis for innovative product, process, and system design and development, as well as transdisciplinary knowledge and pedagogical skills, consistent with program goals and validated by program stakeholders  Internal and external groups regularly review and revise program LOs and/or program goals based on CDIO/ FCDI/FFCD Syllabus and changes in stakeholder needs 4 Program LOs are aligned with CDIO/FCDI/FFCD Syllabus, and institutional vision and mission, and levels of proficiency are set for each outcome 3 Course and/or program LOs are validated with key program stakeholders, including faculty, students, alumni, and other stakeholders, and levels of proficiency are set for each outcome 2 A plan to incorporate explicit statements of LOs at course/module level as well as program outcomes is accepted by program leaders, faculty, and other stakeholders 1 The need to create or modify LOs at course/module level and program outcomes is recognized and such a process has been initiated 0 There are no explicit program LOs at course/module level nor program outcomes that cover knowledge and skills aligned with CDIO/FCDI/FFCD Syllabus Table 7 CDIO/FCDI/FFCD Standard 3 -Curricula

CDIO FCDI FFCD
A curriculum designed with mutually supporting disciplinary courses, with an explicit plan to integrate personal and interpersonal skills, and product, process, and system building skills A curriculum designed with mutually supporting interdisciplinary courses, as well as other elements (projects, internships, etc.), innovation and teaching activities with an explicit plan to integrate personal and interpersonal skills, and innovative product, process, and system design and development skills based on forecasting stakeholder needs A curriculum designed with mutually supporting transdisciplinary courses, as well as research and pedagogic activities with an explicit plan to integrate personal and interpersonal skills, and abilities to create scientific basis for innovative product, process, and system design and development using the methods of technological foresight Internal and external stakeholders regularly review the integrated curriculum and make recommendations and adjustments as needed 4 There is an evidence that the students have achieved the intended LOs aligned with CDIO/FCDI/FFCD Syllabus 3 The approved integrated curriculum concerning intended LOs aligned with CDIO/FCDI/FFCD Syllabus is in use 2 The curriculum that integrates LOs aligned with CDIO/FCDI/FFCD Syllabus is approved and a process has been initiated to implement the curriculum 1 The need to analyze the curriculum is recognized and initial mapping of LOs aligned with CDIO/FCDI/FFCD Syllabus is underway 0 The curriculum has no courses known to integrate LOs aligned with CDIO/FCDI/FFCD Syllabus

CDIO-FCDI-FFCD Standard 2
The CDIO/FCDI/FFCD Standard 2 regarding intended learning outcomes of the undergraduate/graduate/postgraduate engineering programs is presented in Table 4. Table 5 shows the list of intended learning outcomes (LOs) of the Bachelor's, Master's and Doctoral program graduates. The list can be supplemented by universities, taking into account the needs of key stakeholders, labor market requirements and other features of the university's mission.
Setting specific learning outcomes helps to ensure that students acquire the appropriate foundation for their future. Professional engineering organizations and industry representatives identified key attributes of Bachelors, Masters and Doctors of engineering both in technical and professional areas. Moreover, many evaluation and accreditation bodies expect engineering programs to identify program outcomes in terms of their graduates' knowledge, skills, and attitudes. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 2 is given in Table 6.
An integrated curriculum includes learning experiences that lead to the acquisition of personal and interpersonal skills, interwoven with the learning of disciplinary, interdisciplinary and transdisciplinary knowledge and its application in professional engineering. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 3 is given in Table 8.
Introductory course/workshop/seminar aims to stimulate students' interest in, and strengthen their motivation for, the field of complex/innovative/research engineering by focusing on the application of relevant disciplinary/interdisciplinary/transdisciplinary courses. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 4 is given in Table 10. Table 9 CDIO/FCDI/FFCD Standard 4 -Introduction to Engineering CDIO FCDI FFCD An introductory course that provides the framework for engineering practice in product, process, and system building, and introduces essential personal and interpersonal skills An introductory workshop that provides the framework for engineering practice in innovative product, process and system design and development based on forecasting the needs of stakeholders, as well as introduces essential personal and interpersonal skills An introductory seminar that provides the framework for engineering practice in creation of scientific basis for innovative product, process, and system design and development using the methods of technological foresight, as well as introduces essential personal and interpersonal skills There is a documented evidence that students have achieved the intended LOs of the introductory course/ workshop/seminar 3 An introductory course/workshop/seminar that includes engineering/innovation/research learning experiences and introduces essential personal and interpersonal skills has been implemented 2 A plan for an introductory course/workshop/seminar introducing a framework for engineering/innovation/ research practice has been approved and a process to implement the plan has been initiated 1 The need for an introductory course/workshop/seminar that provides the framework for engineering/ innovation/research practice is recognized and a planning process initiated 0 There is no introductory engineering course/workshop/seminar that provides a framework for engineering/ innovation/research practice and introduces key skills
Design-implement/innovation-design/research-design experiences are structured and sequenced to promote early success in complex/innovative/research engineering practice. The experiences also provide a solid foundation upon which to build deeper conceptual understanding of disciplinary/interdisciplinary/transdisciplinary skills. The emphasis on real-world contexts gives students opportunities to make connections between the scientific and technical content they are learning and their professional and career interests. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 5 is given in Table 12.
Workspaces and other learning environments that support hands-on learning are fundamental resources for learning to research, design and  Table 13 CDIO/FCDI/FFCD Standard 6 -Workspaces

CDIO FCDI FFCD
Workspaces and laboratories that support and encourage hands-on learning of product, process, and system building, disciplinary knowledge, and social learning Workspaces and laboratories that support and encourage innovative product, process, and system design and development, interdisciplinary knowledge, and social learning Workspaces and laboratories that support and encourage creation of the scientific basis for innovative products, processes and systems design and development, transdisciplinary knowledge, and social learning  The Bachelor/Master/Doctor program leaders, students, teachers and external stakeholders regularly evaluate the functionality and purposefulness of workspaces on learning and provide recommendations for improving them 4 Workspaces fully support all components of hands-on, scientific and technical knowledge, and skills learning 3 Development plans of workplaces are being implemented and some new or remodeled spaces are in use 2 Workspaces, their functionality and purposefulness for teaching are being evaluated by internal groups including stakeholders 1 The need for workspaces to support hands-on, scientific and technical knowledge, and skills activities is recognized and a process to address the need has been initiated 0 Workspaces are inadequate or inappropriate to support and encourage hands-on skills, scientific and technical knowledge, and social learning Integrated learning experiences that lead to the acquisition of disciplinary knowledge, as well as personal and interpersonal skills, and product, process, and system building skills Integrated learning experiences that lead to the acquisition of interdisciplinary knowledge, as well as personal and interpersonal skills, and innovative product, process, and system design and development skills based on forecasting stakeholder needs Integrated learning experiences that lead to the acquisition of transdisciplinary knowledge, as well as personal and interpersonal skills, and abilities to create scientific basis for innovative product, process, and system design and development using the methods of technological foresight Courses and other curriculum elements are regularly evaluated and revised regarding their integration of learning experiences and the impact of these experiences 4 There is an evidence of the impact of the implementation of integrated learning experiences according to the integrated curriculum plan 3 Integrated learning experiences are being implemented in courses and other elements across the curriculum according to the integrated curriculum plan 2 Courses and other curriculum elements plans with learning outcomes and activities that integrate personal and interpersonal skills with disciplinary/interdisciplinary/transdisciplinary knowledge have been approved 1 Courses and other curriculum elements plans have been benchmarked with respect to the integrated curriculum plan 0 There is no evidence of integrated learning of disciplines and skills develop products, processes, and systems. Students who have access to modern engineering tools, software, and laboratories have opportunities to develop the scientific and technical knowledge, skills, and attitudes that support product, process, and system researching, designing and developing competencies. These competencies are best developed in workspaces that are student-centered, user-friendly, acces-sible, and interactive. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 6 is given in Table 14.
The curriculum design and learning outcomes, prescribed in CDIO/FCDI/FFCD Standards 2 and 3 respectively can be realized only if there are corresponding pedagogical approaches that make dual use of student learning time. Furthermore, it is important that students recognize engineering faculty as role models of professional engineers and engineering researchers, instructing them in disciplinary/ interdisciplinary/transdisciplinary knowledge, personal and interpersonal skills, and product, process, and system research, design and development skills based on stakeholder needs. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/ FFCD Standard 7 is given in Table 16.
By engaging students in thinking about concepts, particularly new ideas, and requiring them to make an overt response, students not only learn more, they recognize for themselves what and how they learn. This process helps to increase students' motivation to achieve pro-gram learning outcomes and form habits of lifelong learning. With active learning, innovative and research methods, giving the maximum effect of education and training instructors can help students make connections among key concepts and facilitate the application of this knowledge to new settings. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 8 is given in Table 18.
If faculty members are expected to teach a curriculum of personal and interpersonal skills, and innovative product, process, and system research, design and development skills integrated with disciplinary/interdisciplinary/transdisciplinary knowledge, as described in CDIO/ FCDI/FFCD Standards 3, 4, 5, and 7, they as a group need to be competent in those skills. Engineering professors tend to be experts in the research and knowledge base of their respective disciplines and courses. Moreover, the rapid   Table 20.
If faculty members are expected to teach and assess in new ways, as described in CDIO/ FCDI/FFCD Standards 7, 8, and 11, they need opportunities to develop and improve these competencies. Many universities have faculty development programs and services that might be eager to collaborate with faculty in CDIO/FCDI/FFCD programs. In addition, if CDIO/FCDI/FFCD programs want to emphasize the importance of teaching, learning, and assessment, they must commit adequate resources for faculty development in these areas. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 10 is given in Table 22. Table 19 CDIO/FCDI/FFCD Standard 9 -Enhancement of Faculty Competence CDIO FCDI FFCD Actions that enhance faculty competence in personal and interpersonal skills, and product, process, and system building skills Actions that enhance faculty competence in personal and interpersonal skills, and innovative product, process, and system design and development skills Actions that enhance faculty competence in personal and interpersonal skills, and abilities to create scientific basis for innovative product, process, and system design and development  Actions that enhance faculty competence in providing integrated learning experiences, in using active experiential learning methods, and in assessing student learning Actions that enhance faculty competence in providing integrated learning experiences, in using active and innovative learning methods, and in assessing student learning Actions that enhance faculty competence in providing integrated learning experiences, in using active learning and research methods, and in assessing student learning
If we value personal and interpersonal skills, and product, process, and system research, design and development skills, and incorporate them into curriculum and learning experiences, then we must have effective assessment processes for measuring them. Different categories of learning outcomes require different assessment methods. Using a variety of assessment methods accommodates a broader range of learning styles, and increases the reliability and validity of the assessment data. As a result, determinations of students' achievement of the intended learning outcomes can be made with greater confidence. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/ FFCD Standard 11 is given in Table 24.
A key function of program evaluation is to determine the program's effectiveness and efficiency in reaching its intended goals. Evidence There is an evidence that the faculty is collective working on their competences in teaching, learning, and assessment methods 3 Faculty members participate continuously in faculty development in teaching, learning, and assessment methods 2 A systematic plan of faculty development in teaching, learning, and assessment methods is developed and budgeted 1 A need for enhancing teaching competences is recognized and accepted within the team 0 There are no programs or practices to enhance faculty teaching competence Assessment of student learning in personal and interpersonal skills, and product, process, and system building skills, as well as in disciplinary knowledge Assessment of student learning in personal and interpersonal skills, and innovative product, process, and system design and development skills, as well as in interdisciplinary knowledge Assessment of student learning in personal and interpersonal skills, and abilities to create scientific basis for innovative product, process, and system design and development, as well as in transdisciplinary knowledge Table 24 Rubric for CDIO/FCDI/FFCD Standard 11 Scale Criteria for Standard 11 5 Internal and external groups regularly review the use of learning assessment methods and make recommendations for continuous improvement 4 There is an evidence of aligned learning assessment methods 3 Learning assessment methods are aligned with the learning goals across the curriculum 2 There is a plan to align learning assessment methods with the curriculum 1 The need for the improvement of learning assessment methods is recognized 0 Learning assessment methods are inadequate, inappropriate or not aligned collected during the program evaluation process also serves as the basis of continuous program improvement. For example, if in an exit interview, a majority of students reported that they were not able to meet some specific learning outcome, a plan could be initiated to identify root causes and implement changes. Moreover, many external evaluators and accreditation bodies require regular and consistent program evaluation. The Rubric for evaluating programs for compliance with the recommendations of CDIO/FCDI/FFCD Standard 12 is given in Table 26.

Conclusion
Based on the presented Rubrics that form the hierarchy of levels of compliance of Bachelor's, Master's and Doctoral programs in the field of engineering and technology with the recommendations of the CDIO/FCDI/FFCD Standards, one can assess the quality of three-cycle training of graduates for complex, innovative and research engineering activities, respectively, taking into account the features of the division of labor in the engineering profession.

CDIO FCDI FFCD
A system that evaluates programs against these twelve CDIO standards, and provides feedback to students, faculty, and other stakeholders for the purposes of continuous improvement A system that evaluates programs against these twelve FCDI standards, and provides feedback to students, faculty, and other stakeholders for the purposes of continuous improvement A system that evaluates programs against these twelve FFCD standards, and provides feedback to students, faculty, and other stakeholders for the purposes of continuous improvement Table 26 Rubric for CDIO/FCDI/FFCD Standard 12 Scale Criteria for Standard 12   5 There is a documented evidence that systematic and continuous improvement is based on continuous program evaluation results

4
There is a documented evidence that program evaluation methods are being used with key stakeholders including students, faculty, program leaders, alumni and working life representatives 3 Program evaluation methods are being implemented across the program to gather data from majority of stakeholders, such as students, faculty, program leaders, alumni, working life representatives 2 A continuous program evaluation plan exists 1 The need for program evaluation is recognized and benchmarking of evaluation methods is in process 0 Program evaluation is non-existing