While the holy book tells us not to judge [[1]](), human existence necessitates evaluation and decision-making. Our sensory organs constantly provide input, compelling us to respond to voluntary stimuli. This trait is evident in various societal roles: managers assess employee performance, investors evaluate business proposals, admissions officers rank applicants, and hiring managers prioritize candidates based on their qualifications. These examples underscore the inherent role of judgment in human interaction and decision-making processes.
As there is no free lunch, every judgment incurs a cost. Therefore, prudence dictates that decision-makers establish guardrails when making consequential decisions. Judgment yardsticks can vary, as the impact of some decisions may be long-lasting, while others are short-lived. The negative consequences of poor science are well-known, placing an undue burden on taxpayers. Many [flawed scientific works](https://www.heritage.org/poverty-and-inequality/report/big-government-policies-hurt-the-poor-and-how-address-them) have influenced government policies, leading to disastrous societal outcomes. This situation has necessitated the development of a principled approach to evaluating scientific research.
The rise of [paper mills](https://en.wikipedia.org/wiki/Research_paper_mill) churning out scholarly work at unprecedented scales, coupled with the emergence of [collusion rings](https://cacm.acm.org/opinion/collusion-rings-threaten-the-integrity-of-computer-science-research/), is worsening the quality of papers published in ostensibly prestigious venues. This status quo has encouraged some authors to indulge in data manipulation, overstating the merit of their work. But without sufficient policing, who can stop them? This blog aims to shed more light on making better judgments as part of objectively evaluating the quality of scientific publications (papers, projects).
## **Judging Experience**
Through divine providence, I have been privileged to have served as a reviewer for several top-notch scientific venues ([ICLR](https://iclr.cc/), [WWW](https://thewebconf.org/), [PoPETS](https://petsymposium.org/), and [ICML](https://icml.cc/)) and judged multiple science fairs ([Vancouver District Science Fair](https://blogs.vsb.bc.ca/vdsf/) (VDSF), [Greater Vancouver Regional Science Fair](https://gvrsf.ca/) (GVRSF)). The core goal for reviewing scientific works is to improve the quality of scientific publications through rigorous evaluation to ensure functionality and reproducibility.
There are criteria for judging work that share similarities based on the downstream task. When reviewing the work of postgraduate scientists, a higher level of rigor is expected. Conversely, for high school science contests, less rigor is required to encourage young people and foster the research attitude necessary to function at an expert level in the future.
We have proceeded in this work to develop a few guardrails to improve and evaluate the quality of scientific works. They include:
- Level of applicants: The academic pedigree of the authors of the work should be considered during the evaluation of the scientific work. Graduate students tend to anticipate harsh judgment on their work, while elementary school students should be judged with leniency to encourage them to stay in the STEM fields.
- Novelty: The work has to be unique, but in some cases, it may be the product of applying a standard method to a new field or finding more uses. It can also take the form of insight into a new set of experiments, case studies, and longitudinal studies.
- Clarity: Excellent writing to disseminate the information in a clear manner
- Grading of rubric: Ensure consistent grading for every project/paper
Let us discuss my lessons from reviewing conferences and how they differ from high school competitions.
- Reviewing Conference (Journal) Paper
- Judging Scientific Competition
# Reviewing Conference (Journal) Paper
Peer review is the hallmark of scientific validation. This practice has allowed scientists to validate claims before widespread distribution. When done properly, it can provide a medium for seeking feedback to improve work. Objective peer reviews that focus on the merit of the work are essential for acceptance in conferences or journals. Despite its positives, it can be a source of gatekeeping against non-mainstream ideas. We will provide a few guidelines to inform the peer review process and set a gold standard.
Reviewing is a volunteer-led activity orchestrated by fellow scientists; who are knowledgeable about the thesis of the papers under review. A typical paper needs a consensus among a pool of reviewers meeting a threshold for acceptance in a conference/journal.
Reviews may be Single-blind, Double-blind, and Triple-blind.
- Single-blind: reviewers know the authors of the paper, but the authors don’t know the reviewers
- Double-blind: reviewers and authors don’t even know each other (does not consider [preprints](https://en.wikipedia.org/wiki/Preprint) which may make the reviewers know the authors). This prevents bias when reviewed work may pose a conflict of interest.
- Triple-blind: Reviewers and authors don’t even know each other ( does consider [preprints](https://en.wikipedia.org/wiki/Preprint) which may make the reviewers know the authors). The author is prohibited from sharing their work as a [preprint](https://en.wikipedia.org/wiki/Preprint) or discussing their scientific work on social media to provide complete privacy where the author does not know the reviewers. Some scientific conferences provide mediums for rebuttals to facilitate two-way communication between reviewers and authors.
Scientific papers are evaluated based on correctness (validated findings), clarity (clear communication and coherence), and reproducibility. Some papers are not subject to full peer review and are desk-rejected to save reviewers time. Reasons for desk rejection include blatant formatting violations, inappropriate topics, and poor writing quality. The reviewing formats for reviewers include a summary of findings, strengths/weakness of the work, and reviewer’s recommendation which may consist of suggested edits (accept, reject, or revise) or other gradations of accept and reject (likely accept, weak accept, strong reject). Furthermore, if reviewers don't comment on a given scientific work, it might be a bad signal or a perfect job. On the contrary, when they have a lot of feedback on a given work, it is a positive signal and could be any of the following: a great idea but poor writing, a controversial idea, or sub-par work.
## Constraints
The original intent of reviewing scientific work is to serve as quality control and reduce the verification burden for future readers. Experience has shown that peer review can help advance science. Unfortunately, like any system, peer review can be abused, leading to various problems when reviewing standards are abused.
- Reviewers may reject work and unethically snoop on the idea owing to the cloak of anonymity. As a defense, a [preprint](https://en.wikipedia.org/wiki/Preprint) may help the author stake a legal claim on an idea before filing patents. This may be the disadvantage of the Triple-blind review form.
- The increasing emphasis on novelty over practical impact in academic research is diminishing the value of many papers. This "novelty for novelty's sake" approach often prioritizes mathematical rigor over real-world applications.
- The “public or perish mentality” has led to inundating conferences with low-quality papers and patronizing paper mills. This mentality is detrimental to the advancement of science.
- Undue emphasis on magazine-grade scientific typesetting work for paper by near-professional graphic designers. This formatting requirement puts an unnecessary burden on authors.
- Excess overspecialization of the authors in niche areas. Most technological advancement requires an interdisciplinary approach to problem-solving.
- Overwhelmed reviewers are volunteers and not paid. Rebuttal exerts a toll on the reviewer.
- Conflict of interest may occur due to rivalry between labs, individuals, and firms.
- Excessive cost of attending conferences, as it is getting monetized by commercial interest.
# Judging Scientific Competitions
Science fairs have become a popular way to introduce people to science at a grassroots level. Traditionally, these events have been in-person, but the post-COVID era has seen an increase in online components. While nothing can replace the in-person experience for participants, online formats can help recruit high-quality judges, overcoming geographical constraints. Hybrid formats are becoming increasingly popular for organizing science fairs.
Science fairs have the feel of tournaments where projects advance through the ranks until the final. Some local science fairs are a qualifying round for a regional and national fair. Let us consider the case of Vancouver, BC. [VDSF](https://blogs.vsb.bc.ca/vdsf/) is a qualifying round for GVRSF, and [GVRSF](https://gvrsf.ca/) is a qualifying round for [CWSF.](https://cwsf-espc.ca/). The participants are many students in the gifted science program, University transition program, public secondary schools, and home-schooling kids.
Science fair organizers are always seeking talented judges with varying knowledge and experience. These kinds of judges can enrich the student's learning experience. Ideally, projects are mapped to judges with the relevant background skills. However, there may be constraints where people get paired with projects and learn on the task.
### What can the student do to help the judges?
Each project has an abstract to give the judge an idea of the project description. Students present their work to help the judges understand their project and make realistic assessments of its merits. The final grade of a project is determined by seeking a consensus on the recommendation of judges.
### What is the judging process for science fairs?
Students present their work for a set duration, followed by a question-and-answer session. Judges then assign scores, provide feedback, and recommend advancement to the next round (local to national level). Submissions can be experiments, innovations, or studies, each evaluated based on its scientific approach. Novel work is considered an innovation, while studies involve data processing, analysis, and interpretation.
## Constraints
We have identified a few concerns that impact the experience of the judges and applicants as follows:
- Some projects are too risky to be demonstrated in a presentation setting due to safety concerns or lack of parental consent.
- Workload distribution among students presenting their work. There are difficulties with identifying the level of contribution of each team member in a project. As expected, contests aim to cultivate partnerships among students and not freeloaders. Any deviance is considered a misrepresentation
- Figuring out where the idea originated? Is it from their teacher, parents, or chatGPT? How useful is the idea?
## Grading rubric
A grading rubric is the same for every student, irrespective of their grade level in school. Younger students will be treated the same as older students without watering down the requirements. This arrangement cultivates the right mindset of excellence and eliminating mediocrity from an early age.
However, the project organizer may compensate for the participant's age to boost the morale of underaged applicants. The following unwritten rules can help improve the scoring of projects. They include:
- Be consistent with your grading and provide honest/unbiased feedback.
- Seek to identify students who are advanced beyond the capability of a high schooler by asking probing questions.
- Submit scores on the same day as the events to keep every context in mind.
# Conclusions
I have made the following observations while reviewing scientific works including:
- It is common to see kids have better projects than highly educated individuals based on my experience reviewing high school science fairs. As a result, I am convinced that autodidact owns the future.
- Reviewers can provide weaknesses and suggestions to strengthen the thesis by issuing pointers for future work.
- Peer reviews of papers can be a tool for scientists to validate or improve the viability of their ideas.
Anecdotal evidence suggests that many authors occasionally submit sub-par work in scientific conferences because of inherent bias in their self-evaluation. Therefore, the authors expected the reviewer's empathy in this situation.
Finally, there are transferrable judging skills in both scientific papers and competitions. Consequently, these skills can be reapplied in **software engineering activities** including source code review, business plans, architectural design, and developing better mentoring skills in apprentice-like setups typical in software craftsmanship.
[[1]](): Douay-Rheims Bible Matthew 7:1 “Judge not, that you may not be judged”
# **How to Cite this Article**
```
BibTeX Citation
@article{kodoh2024b,
author = {Odoh, Kenneth},
title = {The Act of Judging Scientific Works and Artefacts},
year = {2024},
note = {https://kenluck2001.github.io/blog_post/the_act_of_judging_scientific_works_and_artefacts.html}
}
```
1/17
Please feel free to donate to support my work by clicking
