Evaluation Process of Project Proposals

Aims:

  • Professionalise and increase effectiveness of the project proposal evaluation system;
  • mutually compare project proposals;
  • standardise success criteria;
  • strengthen the transparency and independence of evaluation;
  • identify a specific category of profoundly below-average projects (Cn).

The evaluation process meets the following requirements:

  • Statutory evaluation period of 8 months;
  • Three-stage evaluation system:
    the governing body is the Presidium,
    Discipline Committees are expert advisory bodies according to Act No. 130/2002 Coll.,
    evaluation Panels are the expert bodies of the Discipline Committees;
  • Statutory minimum of two independent evaluations for each project proposal;
  • at least one external review for projects proceeding to the second phase of the evaluation;
  • participants in the evaluation process bound by duty of confidentiality.

Panels:

  • including the Chair, the Panel has a membership of 8 to 12 panellists;
  • typically, a Panel evaluates dozens of project proposals;
  • proposals for new Panel members from leading experts can be submitted by legal and natural persons working in related scientific fields
  • Panel members requirements: high level of expertise mainly based on publication activity and other results, willingness and ability to evaluate a broader scope of the research field, the
  • ability to judge strictly on professional aspects;
  • a member of a Panel is nominated as an eminent scientist with broad insight, not as a representative of an organisation;
  • the members of the Panels are chosen and appointed for a two-year period with the possibility of early membership termination in case of non-compliance with the obligations under the Statute and Procedure of Discipline Committees and Evaluation Panels of the Czech Science Foundation;
  • membership proposal form shall contain:
    CV, expertise and summary, and citation rates of results,
    Employer allows to free the panellist to participate on the Panel meetings during work hours,
    binding consent of the nominee to actively participate on the Panel meetings within the set deadlines and related obligations;
  • the Panel members shall be chosen from the nominee list by the committee of:
    Presidium member of the relevant field,
    representative of the Research, Development and Innovation Council;
    representative of the GACR Scientific Advisory Board or of the GACR Supervisory Board;
  • A Panel member may submit a project proposal to the Panel of which he/she is a member (in such case the project proposal is additionally evaluated by external reviewers to guarantee the
    Panel’s accuracy and fairness);
  • the Chair and Vice-Chair may not submit a project proposal to the Panel under their own Discipline Committee;
  • during meetings, panellists having any conflict of interest shall leave the meeting room;
  • Discipline Committees are made up of Chairs and Vice-Chairs of individual Panels, named by the GACR Presidium, and suggested by the individual Panels via ballot.

 

Three-stage project proposal evaluation

First stage evaluation

Phase One

Evaluation in phase one

  • Each project proposal shall be evaluated by 4 members of the Panel (rapporteurs): two rapporteurs and two reviewers (in the case of an interdisciplinary project 1 of the 4 rapporteurs shall be from the secondary Panel).Rapporteurs are chosen by the following procedure:

main rapporteur shall be chosen by the Chair and Vice-Chair for each project proposal,
second rapporteur chosen by random selection,
both reviewers chosen by random selection,
selection of the rapporteurs and reviewers shall prevent assignment of the project to panellists who may have a conflict of interest;

  • rapporteurs shall evaluate the project proposal and draw up an independent review;
  • reviewers shall evaluate the project proposal;
  • each rapporteur and reviewer shall independently evaluate and select one of four provisional categories A1, B1, C1 and Cn for all projects assigned to them via the online grant information system. The ratio shall be at most 30% A1, at least 30% C1 + Cn and the rest for B1, for which:

A1 stands for a high quality project proposal, rapporteur or reviewer recommends the project to proceed to the next phase,
B1 stands for a quality project proposal, rapporteur or reviewer recommends the project to proceed to the next phase,
C1 stands for average or below average project proposal, rapporteur or reviewer does not recommend the project to proceed to the next phase,
Cn stands for a significantly below average project.

  • a member of a Panel shall not inform the other members of the Panel about the projects he/she is evaluating until three days before the Panel meeting – at that point evaluations are made accessible to all the members of the Panel;
  • three days before the Panel meeting all the evaluations are made accessible to all members of the Panel excluding those having a conflict of interest or their own project via an electronic application;
  • each of the rapporteurs assigned to the given project proposal shall recommend 2–3 appropriate external reviewers for projects ranked A1 or B1 via the web database Expert Lookup.

Panel evaluation in Phase 1

  • During the Panel meeting and Panel discussion the list of low quality project proposals is made. The result from the Panel meeting is the classification of the projects into categories. The total number of projects in category A1 shall be at most 30% and C1+Cn at least 30% of proposals evaluated by the Panel. The ratio between C1 and Cn is not prescribed. The output from the Panel meeting is a protocol of the meeting including a proposal for classification of the projects into A1, B1, C1 and Cn categories, this protocol is signed by all members of the Panel. For the projects in category C, the main rapporteur shall write an evaluation note. It shall clearly state the collective evaluation of the Panel. It is natural that, given the independency of the rapporteurs, the evaluation may vary – should the conclusion of the collective assessment of the Panel be contrary to the assessment of some of the rapporteurs, such a discrepancy shall be explained in the project proposal evaluation note. In particular, the evaluation note of C1 and Cn projects shall be unambiguous and it is recommended to compare the quality of the relevant projects with other evaluated project proposals.
  • Based on the recommendations of the rapporteurs, the Panel shall decide to assign external reviewers to projects in category A1 or B1.
  • The Panel shall present the following materials to the Discipline Committee:

meeting protocol with proposal of the classification of projects (A1, B1, C1 and Cn),
proposals classified as A1, B1, C1 and Cn by two or three rapporteurs, two reviewers, including the reviews,
proposal for external reviewers by the rapporteurs.

Discipline Committee evaluation in Phase 1

  • All members of the Discipline Committee shall acquaint themselves with the Panel meeting reports, rapporteur’s reviews with the classification proposals (A1, B1, C1 and Cn) by rapporteurs and reviewers and with proposals of external reviewers;
  • Discipline Committees shall discuss and propose rejection of project proposals which are not recommended to proceed to the second phase of the first stage (i.e. category C1 and Cn);
  • project proposals which were not classified as C1 or Cn shall be sent for the evaluation to the external reviewers by the GACR Office.

Phase 2

  • At least one review shall be obtained from an external reviewer;
  • all the members of the Panel shall acquaint themselves with all project proposals advancing to the phase 2 including:

two or three reviews by rapporteurs,
at least one review by the external reviewer,
four or five project evaluations from the first phase (classification into A1, B1);

  • prior to the Panel meeting, each of the Panel members independently classifies all the project proposals by their quality into three categories: A2, B2, C2 where A2 shall include at most 25% of project proposals in phase 2. The quota for B2 and C2 is not prescribed in phase 2
  • meaning of the categories:

A2 – high quality project, unambiguously recommended for funding,
B2 – quality project, recommended for funding,
C2 – average project proposal, not recommended for funding.

 

Panel evaluation in phase 2

  • As the first step, all the project proposals shall be presented by their main rapporteurs;
  • after thorough discussion, the Panel shall divide all project proposals into three categories: at most 25% which are recommended for funding (A2), those which are recommended for funding if sufficient funds are available (B2) and projects which are not recommended for funding (C2);
  • on the basis of the discussion and comparison of individual project proposals, the Panel shall propose an order of the projects categorised as A2 and B2;
  • the Panel shall discuss the adequacy of required funds and planned workload for projects in category A2 and B2;
  • the final order of projects in phase 2 shall be voted on by the Panel and the result of the vote shall be recorded in the meeting report;
  • the course of the meeting shall be written down in a meeting report – the base document for the project proposal evaluation protocol
  • External reviewers are often less critical evaluators than the members of the Panel. Should it happen that such a discrepancy in grading occurs, this shall be explained objectively and in detail;
  • at the end of the meeting, the statement of the Panel is written down in the evaluation note of each project proposal. As in phase 1, the project proposal evaluation note shall contain a clear conclusion of collective evaluation, shall be unambiguous (especially for projects in C2) and it is recommended to compare the quality of the relevant projects with other evaluated project proposals; should there be a discrepancy in the assessments, it shall be explained.

Second stage evaluation

The second stage of the evaluation process is carried out at the level of the relevant Discipline Committee. At this stage, the project proposals are not evaluated individually but are mutually compared from all Panels of the Discipline Committee. The ranking of projects is done by vote. The Discipline Committee’s proposed project ranking is then presented to the Presidium for the final decision.

Third stage evaluation

Taking into account the proposals of the Discipline Committees, the Presidium, in the presence of Discipline Committees’ Chairs, makes the final decision and announces the result of the public tender.

Addendum

The standards used in the evaluation of the quality of the project proposals and ranking of the project proposals to the A, B, C and Cn categories:
(1) For the evaluation of the project proposals, the following shall be assessed
a) above all the originality, quality and professional standard of the proposed grant project;
b) the capability of the organisation and co-organisation to execute the project with regard to their technical and institutional facilities;
c) the capability and qualifications of the applicant and potential co-applicant and their professional collaborators to execute the grant project, taking into account professional ability, creative contribution in the scientific field of the project proposal, and results in research and experimental development to date.
(2) For the evaluating of the quality and professional standard of the proposed grant project, the following shall be assessed:
a) aim of the project proposal – whether clear and specific aims have been defined; how demanding, significant and feasible they are;
b) whether it will contribute to the body of scientific knowledge in the fields covered under the expertise of the Evaluation Panel (content of the Panels is available at http://www.gacr.cz/poradni-organy/panely),
c) whether the breadth of the problem proposed for study is commensurate with the requested funding and project duration.
● Cn:
– project aims are vaguely defined or they cannot be identified,
– project lacks the basic features of basic research (clearly formulated hypothesis and aims are missing in relation to the current state of knowledge),
– project’s potential contribution to the knowledge in a given field is negligible or hardly identified,
– project is not based on an original idea (an applicant is either not sufficiently familiar with the state of knowledge or consciously replicates already done research);
d) proposed methods:
● in what way the applicant expects to achieve the stipulated aims and results (i.e. concept, preparation and suitability of the proposed methodology including the project timeline);
● the adequacy (in particular in terms of the workload and share of individual team members in the expected output of the grant project) and skill-set composition of the research team and the definition of the roles of individual team members in researching the problem;
● Cn:
– it is not evident from the project proposal how the aims (if formulated) will be achieved, what methods will be applied and how the team will proceed,
– it is not evident from the project proposal how members of the team will contribute to the achievement of the aims,
– the composition (size) of the research team is excessive, project may carry signs of hidden institutional funding,

e) outputs of the grant project – the expected publication activity and achievement of other types of basic research results as defined by Methodology 2013
● Cn:
– expected outputs of the project are inadequate or unclear,
– expected publication activity does not correspond to the previous publication activity of the applicant, team members (contradiction between promises and possibilities),
– proposed outputs do not fall into the results defined in Methodology 2013 for basic research;

f) foreign cooperation – expected involvement of foreign institutions in carrying out the grant project, sharing of equipment of collaborating institutions and use of complementary approaches and methodologies;

g) the progress, results and methods of previous grant projects of the organisation, co-organisations, applicant and co-applicant in (if any were conducted); taking into account any prior violations of the rules on the part of the organisation, co-organisation, applicant or co-applicant during economic management of the provided funding, fulfilment of all obligations laid down by the agreement or decision on provision of funding, and partial and final evaluation of such grant projects;
● Cn:
– the project is a “recycled” project rejected in previous tenders and was submitted into the tender in unchanged form that does not respect the objections or recommendations of expert bodies.

(3) In evaluating the proposed costs the following is assessed:
a) the appropriateness of the proposed costs and amount of full-time equivalents in relation to the project proposal and expected results,
b) the justifiability of the individual items of the proposed costs,
c) the amount of funding requested from the provider as a percentage of the total proposed costs (i.e. the proposed level of support),
d) fulfilment of the requirements of eligible costs according to the Tender Document.

Addendum 2

Cn form:
Reason for categorising the project proposal as Cn:
● the project does not sufficiently or at all define the “knowledge gap” which it intends to fill;
● the project does not have a clearly defined hypothesis and its verification method;
● the proposed methods in the project proposal are not adequate to verify the formulated hypothesis;
● the project expects results or aims which do not correspond with the defined hypothesis;
● its results or aims cannot be achieved by the planned research (by data collection and analysis);
● the project has been previously rejected in GACR tender for its inadequate quality and has been re-submitted without removing the deficiencies.

© 2016 GA ČR