Evaluation process of the EXPRO project proposals

Aims:

• Professionalise and increase effectiveness of the project proposal evaluation system;
• mutually compare project proposals; comparison in scale of international excellence;
• standardise success criteria;
• reinforcement of transparency and evaluation’s independency.

The evaluation process meets the following requirements:

• statutory evaluation period of 8 months;
• two-stage evaluation system;
• the governing body is the Presidium of GACR;
• Discipline Committees (8) are expert advisory bodies according to Act No. 130/2002 Coll.;
• there shall be 4 independent reviews worked out by members of the relevant Discipline Committee (hereinafter “DC”) for each project proposal, in the case of an interdisciplinary project, one of the 4 reviews shall be written by a member of the secondary DC;
• there shall be at least 2 external reviews for projects proceeding to the second phase of the evaluation;
• participants in the evaluation process are bound by duty of confidentiality;
• neither an external reviewer nor a member of a DC shall be employed by any institution in the Czech Republic during the past 5 years.

Discipline Committee (DC) of EXPRO:

• a DC has a membership of 10 to 15 people, including a Chair;
• DC typically evaluates tens of project proposals;
• a foreign agency, chosen through a tender, presents a proposal for 17 candidates, foreign experts, for the membership of each DC;
• requirements on a member of a DC are: high level of expertise and professionalism mainly based on publication activity and other results, willingness and ability to evaluate a broader scope of the research field with hindsight, the ability to judge strictly by professional aspects;
• a member of a DC is nominated as an eminent scientist with broad insight;
• the members of the DCs are chosen and appointed by the GACR Presidium from the nomination of 17 proposed by the agency, on the basis of the spectrum of received project proposals, for a three-year period with the possibility of early membership termination.
Evaluation of project proposals (two-stage process, first stage has two phases)

OK-EX – 1st stage of evaluation prior to the DC meeting

Evaluation in the 1st stage

• each submitted project proposal shall be assigned to the DC proposed by the Applicant;
• the Chair of the DC shall go through each project proposal and should any of them thematically belong to another DC, he/she shall propose to transfer it before the rapporteurs are assigned – this requires the consent of the Chair of the respective DC and should it not be given, the project proposal shall be evaluated in the DC to which it has been submitted;
• each project proposal is to be evaluated by 4 members of the DC (rapporteurs), who write the reviews and decide on the score (in the case of an interdisciplinary project 1 of the 4 rapporteurs shall be from the secondary DC); the rapporteurs are chosen by the following procedure:
o the Chair or the Vice-Chair of the DC assigns 4 (in the case of interdisciplinary project 3, while the fourth shall be assigned by the Chairman of the secondary DC) rapporteurs for each project proposal;
o should the assigned rapporteur be in a conflict of interest, he/she is obliged to refuse and the Chair of the relevant DC shall assign a new rapporteur;
• rapporteurs shall examine the project proposal and draw up an independent review;
• each rapporteur shall independently evaluate the project proposal with respect to two criteria:
1. the excellency of the applicant and his capability to carry out the project,
2. the quality of the project, the significance and aim of the project and the way to achieve it;
• the rapporteurs shall evaluate the project proposals both verbally, filing in the text in the form, and numerically, using the 5-point scale (only whole points are admitted) giving one “point mark”;
• rapporteurs have been recommended to award no more than 10% with 5 points and at most 30% of the proposals with 4 or 5 points;
• a member of a DC shall not inform the other members of the DC about the project he/she is evaluating until ca. three days before the DC conference when the evaluations are made accessible to all the members of the DC;
• three days before the DC conference all the evaluations are made accessible to all the members of the DC excluding those in a conflict of interest via an electronic application;
• the application GRIS shall set the order of all projects of the relevant DC by the score achieved; the score is calculated by counting up the points awarded by each rapporteur (the quality of the applicant plus the quality of the project) and divide them by two, resulting order shall be set by the average scores of the project proposals calculated from the points awarded by all rapporteurs;
• each of the rapporteurs assigned to the given project proposal shall recommend 2–3 appropriate external reviewers for above-average projects via the application Expert Lookup.

Evaluation of DC-EXPRO in the first phase

• the DC conference shall be held via presented videoconference which will be attended by members of the relevant DC, a representative of the GACR Office department and a member of the Presidium relevant for the respective DC or his/her representative from the other members of the Presidium;
• the order and an auxiliary 50% limit shall be arranged by the application;
• the output from the DC meeting is a protocol of the meeting including a proposal for classification of the projects as proceeding/non-proceeding, this protocol is approved by the Chair; for the non-proceeding projects, the main rapporteur shall write an evaluation note after/during the meeting, which shall be approved by all the members of the DC via email and shall clearly state the collective evaluation of the DC; it is natural that, given the independency of the rapporteurs, the evaluation may vary – should the conclusion of the collective assessment of the DC be contrary to the assessment of some of the rapporteurs, such a discrepancy shall be explained in the project proposal evaluation note; in particular, the evaluation note of non-proceeding projects shall be unambiguous and it is recommended to compare the quality of the relevant projects with international standards and other evaluated project proposals;
• based on the recommendation of the rapporteurs, the DC shall decide to assign external reviewers to proceeding projects.

First stage, second phase of evaluation – prior to DC meeting

• at most three evaluations shall be obtained from external reviewers;
• the external reviewers shall be contacted by the application one by one;
• the process of external evaluation shall be closed immediately after three evaluations are obtained; should there be more of them, they are not to be taken into account (the members of the DC shall not be acquainted with them);
• the external reviewers shall use the same 5-point scale and the same criteria to evaluate the aims and quality of the project and the quality of the applicant as members of DC (i.e. each of the external reviewers independently evaluates the aims and quality of a project and the quality of an applicant, both awarded with 1 to 5 points);
• all the members of the DC (except for those in conflict of interest) shall acquaint themselves with all project proposals advancing to the second phase of evaluation (prior to DC meeting in Prague) inclusive:

o four evaluations by DC rapporteurs and
o all evaluations by external reviewers;

• each of the members of the DC independently evaluates the aims and quality of the project and the quality of the applicant again, awarding them with 1 to 5 points each;
• in the next step, the application GRIS calculates the average score of three decimal places, the evaluation of the members of the DC and of the external reviewers, which is taken into the calculation, too, have the same weigh;
• the score determines the rank of a project in the overall order of all projects of the respective DC, creating the base for the final verdict of the DC.

DC evaluation in the second phase

• the one-day meeting of the DC shall take place at the GACR offices in Prague in presence of all the members of the respective DC. One employee of the GACR Office and the respective member of the GACR Presidium, who may not take part in the discussion, shall attend;
• firstly, the project proposals shall be presented by the respective main rapporteurs;
• to change the order, the ad hoc value of the average score shall be changed, keeping it to three decimal places;
• the result shall be the final score determining the rank of all project proposals of the respective DC;
• concerning the projects proposed for funding, the DC shall discuss the adequacy of required funds and planned workload;
• the final order proposed by the DC shall be voted by respective DC and the result of the meeting shall be written down in the protocol;
• the course of the meeting shall be written down in a meeting report, the base document for the project proposal evaluation protocol; the external reviewers are often less critical than the members of the DC, should that happen, such a discrepancy in meanings is to be explained objectively and in detail;
• during/after the meeting, each of the main rapporteurs shall prepare conclusive evaluations of their respective projects, which shall be approved by the other members of the DC either on the spot or by email; as in the first phase, the project proposal evaluation note shall contain a clear conclusion of collective evaluation, shall be unambiguous and it is recommended to compare the quality of the relevant projects with international standards and other evaluated project proposals; should there be a discrepancy in the assessments, it shall be explained.

Second stage – the decision of the GACR Presidium according to § 21 (7) of Act No. 130/2002 Coll.

• The base for the discussion of the Presidium shall be the order of the projects across all the DC according to the scores assigned at the DC meetings;
• the Presidium shall be aware of the number of project proposals submitted to each DC, their ranking in their respective DC, and their financial requirements;
• the grounds for the final decision on funding made by the Presidium shall be the score, the order of project proposals within individual DC shall not be taken into account;
• should the Presidium decide to change the base order of the project proposals, a written justification shall be provided and published on the GACR website.

© 2016 GA ČR