Frequently-Asked Questions
In earlier communication I applied for the position of ‘panellist’. In later communication the employee of the Ministry informed me that I was selected as ‘evaluator’. Are these terms the same exact thing?
Yes, the term ‘evaluator’ and ‘panellist’ stand for the same position in our evaluation process, and the personnel responsible for the communication with evaluators (panelists) use both terms.
I was selected as evaluator into panel (or subpanel). What is expected from me?
Nature of your work really depends on whether you were selected as an evaluator within the Panel or Subpanel.
If you were selected member of Panel, your work will consist mostly of coordination and supervision of the work of evaluators within Subpanels which fall under your Panel. Also, you will be expected to integrate the necessary (required) procedures, and make decisions regarding the interdisciplinarity of evaluated scientific, artistic and other creative outputs. You will not evaluate these individual outputs.
If you were selected member of Subpanel, you will work on evaluation of scientific, artistic and other creative outputs of Slovak universities and research institutions. You are not expected to write individual reviews for each evaluated output, because the process of evaluation will be based on marking of quality of the outputs.
How panels and subpanels are expected to work, and when (and how often) are they expected to meet?
Panels and Subpanels are expected to schedule its meetings and organize the particular work tasks consensually – thus, as a group, while reflecting on time constraints and scientific orientations of evaluators. It means that the allocation of particular workload for each evaluator is expected to be decided in a discussion within the respective Panel/Subpanel, and that the schedule of meetings of panels and subpanels is expected to be decided in same way.
Do you have any more details about the amount of material to be reviewed by the subpanel, to help me understand the likely time commitment?
As the Periodic Evaluation of Research, Development, Artistic and Other Creative Activities is a pilot project, we are unable to estimate quantity of the work which you would undertake. We are, likewise, not able to provide you with number of outputs which you will be expected to review at the moment. The amount of workload for each individual evaluator is expected to be decided on Panel/Subpanel meetings consensually.
The amount of material for the review varies across the subpanels. Thus, quite naturally, there are Subpanels in which there is higher number of applications for the Periodical Evaluation by Slovak higher education and research institutions, and the Subpanels which cover fields of study with less applications. However, please note that not every Subpanel consists of same number of evaluators who share the workload. When deciding about the number of evaluators in particular subpanels Ministry of Education, Science, Research and Sport of the Slovak Republic had considered the expected quantity of applications for Periodical Evaluation for each Subpanel (study field). Hence, there will be no notable differences in the required amount of review work between evaluators from particular Subpanels.
In initial communication with Ministry of Education, Science, Research and Sport of the Slovak Republic it was stated that the process of Periodic Evaluation of Research, Development, Artistic and Other Creative Activities runs from the 1st of July, 2022. I was contacted later. Does it mean that the whole process is shifted into 2023, or that it runs in tighter time constraint (until the end of 2022)?
Originally, we had scheduled 6-month period (between 1st of July and 31st of December 2022), with having possible delays in mind. However, the process of evaluation runs between 1st of September and 1st of December. Within these dates all of the required tasks need to be carried out by Panels and Subpanels. Hence, within this period the required meetings of Panels and Subpanels are expected to take place, evaluation of all scientific, artistic and other creative outputs is expected to be done – as well as the creation of quality profiles for the applications from the Slovak higher education and research institutions.
What is the remuneration for the evaluators?
Gross remuneration is EUR 17,460.48 per panellist/evaluator. This sum will be transferred to evaluators during December in form of a single payment, once the work on their tasks will be carried out. In case the paperwork will not be in place by the end of December 2022, the remuneration will be transferred to evaluators in January 2023.
How am I supposed to evaluate the outputs, and what is the process? Are there some general criteria which are required to be observed in the process of evaluation of outputs?
Each scientific, artistic or other output has to be assessed by at least two evaluators from particular subpanel. The evaluators do not write reviews on individual outputs, but only grade them according to five quality grades, which are: ‘world-leading quality’, ‘internationally excellent quality’, ‘internationally recognised quality’, ‘nationally recognised quality’, and ‘below the standard of nationally recognised work’. After the assessment of quality of some single output is done (i.e., after at least two of evaluators within the subpanel agree on the grade that is to be awarded to this output), the quality grade may be uploaded into web-based software – but only by the chairs of each respective subpanels.
There are no general criteria which should be observed by all of the evaluators across all of the subpanels. As we consider each of the 28 evaluated fields of study specific in terms of criteria for quality, rigour and significance, we expect that each of the subpanels would develop its own criteria and methodology for the evaluation of outputs. These criteria for the evaluation and its methodology, then, are expected to be result of the consensual agreement within every subpanel, and are also expected to be observed by the evaluators consistently and uniformly in the applications (and its’ scientific, artistic, and other outputs) that belong to particular field of study (subpanel).
Is there some general guideline on how the workload should be distributed between the subpanel members?
Distribution of the workload within each one of subpanels follows no general set of rules. The way how the workload is distributed between the group of evaluators within particular subpanel depends solely on their consensual agreement. Subpanel may, for example, follow its members’ areas of expertise as a key for distribution of the workload – but it is really just one of many possible options.
What will be the result of the evaluation of particular application?
At the end of the evaluation, each application from the Slovak higher education and research institutions will be provided with quality profile. Each application’s quality profile will contain the information about the proportions of outputs within the application which belong into each of the five quality grades.
Where the evaluation will be carried out? Do I have access to some software?
The interface for evaluation of the applications and scientific (and other) outputs is called VER 2022 (Verification of Excellence in Research). Every evaluator (member of panel or subpanel) will have access to this web-based software via personal account, and will be able to see the progress of the evaluation for each application (and its outputs) in the respective field (subpanel). However, it is only the chairs of subpanels who will be able to award the grades to outputs (of which other subpanellists will provide him/her during subpanel meetings) in VER’s interface, and who will be able to upload the quality profiles for applications within their respective subpanels.
Do I need to consider all 3 dimensions of the evaluation (originality, significance, rigour)? Yes, all three must be considered in the evaluation, even though the output is eventually marked with just one grade (world standard, significant international level, international level, national level, unclassified), e.g. when the output seems great as far as significance is concerned, but less so within the rigour dimension, the latter must be taken into account as well – i.e. the evaluation must be complex.
Do the two of us evaluating an individual output report only one grade for the output?
Yes, that is correct, so You need to discuss and reconcile on the final grade.
Will some scientific teams (i.e. applications) have an advantage as to the selection of the outputs?
The selection of the outputs was done as objectively as possible (the principle of equal, even distribution, and the principle of representativeness, respectively), hence no worries about the advantages for the certain teams. They all have provided both excellent, as well as outputs of lower quality. We tried our best to set the standards of selection so that it is most fair to each and every application (scientific team being evaluated). You as evaluators merely focus on the outputs.
As far as the evaluation of abstracts (when the main body of the output is written in Slovak) is concerned, can I involve somebody that can speak Slovak in the evaluation by means of their helping us with reading the main body?
Unless the person proficient in Slovak language is one of the panellists or sub-panellists, You cannot involve the person (or any other third person, for that matter) in the evaluation process, therefore the answer is no.
Will the software to be used for evaluation accessible via regular web browser?
Yes, the VER 2022 SW will be accessible via regular web browser, so You do not need to download it, hence no worries whether it is compatible with the cyber security policy of Your respective Universities.
What is the responsibility of a panel and its members within the evaluation process?
You as panel members will have the responsibility to oversee the very process of evaluation carried out by Your respective sub-panels. Each sub-panel will choose their methodology of evaluation (this can differ across the sub-panels; but not within the individual sub-panel, i. e. once the sub-panel members determine their methodology, they all have to stick with it and follow it during the course of the evaluation process). If there is a huge discrepancy in grading between the two sub-panellists assessing the output, they ask the fellow sub-panellists to have a look at it. Should necessity still arise, they will ask for help You as the panel members.
Will the scientific teams (individual applications) see the individual grades they will be given for each output?
They will see only the final quality profile. They will not be familiar with what grade they were given for the individual outputs, nor by whom.
Can we suggest any recommendations for the future evaluation process?
Of course, we will welcome them at the end of the process, where You will be to provide us with the feedback on the whole process.
Will also the Panel members get an access to outputs via VER 2022 software?
Yes, 100%. You will determine whether some (we will give you heads-up which ones) are interdisciplinary and thus shifted for evaluation to another sub-panel.
Will there be also 4 panel meetings held?
Yes. You will be (inter alia) discussing whether the sub-panels follow the methodology of evaluation which they have previously agreed upon. As for the attendance on the meetings of Your assigned sub-panels, coordinate and set the dates of the meetings most convenient to each of You with the chairs of the individual sub-panels.
What am I supposed to do with the outputs that seemingly do not fall under the field of study in which the application has been submitted?
In case there are only few such outputs scattered across an application, You may grade them with the lowest possible mark, since they do not represent a relevant contribution to the declared field of study. However, should an entire application with all its outputs seem to be out of declared major, expertise or field of study, please do forward this application to be assessed by the main panel under which Your respective sub-panel falls under, and they will decide to which sub-panel the application should be forwarded for assessment.
What are we supposed to do with the output that was originally submitted within our field of study, marked as interdisciplinary, and it clearly belongs to the other field?
If such an output had not been declared as interdisciplinary by an applicant, you could legitimately assign it the lowest possible grade (as it does not belong to Your field – see the question and answer above). However, as this output was marked as interdisciplinary in another field of study, this second sub-panel will go through it, and is expected to propose the quality grade for it, too. This other subpanel will inform you about their stance in terms of quality of this piece. The final grade will then depend on the consensual agreement within Your subpanel, and on the fact whether or not you will take into account the proposed grade from the other subpanel.
How and when do we upload the final grades in the VER 2022 software?
Once You have the results for all the outputs of an individual application ready, the chair of the sub-panel can upload the results within the VER system and the system itself will generate the final quality profiles of the institution that submitted the application.
When does the contract become officially valid?
The contracts are at the moment circulating around various departments of the Ministry for the signatures. Once the process is finished, they will be made public and thus become officially valid.