Evaluation of PEAT.ai

QED | http://qed.ai

Introduction

We assess the performance of PEAT.ai, a German startup focusing on automated diagnosis of plant diseases, pests, and nutrient deficiencies, with a publicly advertised accuracy rating of 95 percent. PEAT's performance was measured on both

  • public datasets provided by PEAT.ai in their Github repository as examples, and
  • private datasets collected by our associates and staff from multiple East African countries.

In summary, PEAT.ai's accuracy on its own examples is 100 percent, but its accuracy on our dataset was 3 percent.

Code

Dependencies

In [92]:
import os
import time
import datetime
import yaml
from typing import List

import requests
import numpy as np
import pandas as pd
import nltk
from tqdm import tqdm_notebook
from fuzzywuzzy import fuzz

from IPython.display import Image
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

Load Configuration

In [90]:
config = yaml.safe_load(open("config.yml", "r"))
ENDPOINT = config['peat']['endpoint']
API_KEY = config['peat']['api_key']
In [91]:
col_names_ordered = ['filename', 'label', 'name', 'recognized', 'rank', 'similarity', 'scientific_name', 'peat_id', 'pic_id']
col_names_checked_ordered = ['filename', 'label', 'correct_diagnosis', 'name', 'recognized', 'rank', 'similarity', 'scientific_name', 'peat_id', 'pic_id']
col_names_grouped = ['label','correct_diagnosis','name','rank','similarity','recognized','scientific_name','peat_id']

Text Manipulation

In [7]:
def extract_path_part(path: str, part_to_extract_from: str) -> str:
    """Extracts a user-specified component of a path."""
    path_normalized = os.path.normpath(path.lower())
    if "dir" == part_to_extract_from:
        return os.path.basename(os.path.dirname(path_normalized))
    elif "file" == part_to_extract_from:
        return os.path.basename(path_normalized)
    else:
        raise ValueError("part_to_extract_from must be supplied.")
In [8]:
def extract_label(s: str) -> str:
    """Naive extraction of labels from a string."""
    s = os.path.splitext(s)[0]
    s = s.replace("_", " ")
    words = nltk.word_tokenize(s)
    return " ".join([w for w in words if w.isalpha()])
In [9]:
def extract_label_from_path(path: str, part_to_extract_from: str) -> str:
    """Naive extraction of labels from a user-specified component of a path."""
    part = extract_path_part(path, part_to_extract_from)
    label = extract_label(part)
    return label
In [10]:
def is_image(fn: str) -> bool:
    """Checks if the filename extension matches that of an image."""
    img_extensions = ["jpg", "png"]
    return any([fn.lower().endswith(ext) for ext in img_extensions])

Peat API

In [12]:
def diagnose(fn: str, variety: str=None):
    """Requests image diagnosis from PEAT.ai."""
    h = {"api_key": API_KEY}
    if variety:
        h['variety'] = variety
    try:
        # set filename explicitly to discourage snooping
        response = requests.post(
            ENDPOINT,
            files={'picture': ('image.jpg', open(fn, 'rb'), 'image/jpeg')},
            headers=h,
            timeout=40)
    except requests.exceptions.RequestException as e:
        print(e)
        return None
    else:
        json = response.json()
        return json
In [13]:
def diagnose_batch(dir_list: List[str], variety: str, part_to_extract_label_from: str, delay_seconds: int=0) -> list:
    """
    Fetch diagnoses of all images from a list of directories.
    part_to_extract_label_from can be "file" or "dir".
    """
    records = []
    for d in tqdm_notebook(dir_list):
        for fn in tqdm_notebook(os.listdir(d)):
            if is_image(fn):
                path = os.path.join(d,fn)
                response = diagnose(path, variety=variety)
                if 200 != response['code']:
                    print("Unexpected error: ", response)
                else:
                    try:
                        label = extract_label_from_path(path, part_to_extract_label_from)
                        datum = {**{'filename': fn, 'label': label}, **response}
                    except ValueError:
                        print("ValueError: ", path)
                        datum = response
                    records.append(datum)
                time.sleep(delay_seconds)
    return records
In [14]:
def unstack_diagnoses(diagnoses: list):
    """Unstack a list of diagnoses, to accommodate conversion into a DataFrame."""
    def return_dict_without_key(d, key):
        r = dict(d)
        del r[key]
        return r
    unstacked = []
    for d in diagnoses:
        for analysis in d['image_analysis']:
            unstacked.append( {**analysis, **return_dict_without_key(d, 'image_analysis')} )
    return unstacked

Image Processing

In [15]:
def manipulate_image(img):
    """Expects cv2 image as input. Returns manipulated cv2 image as output."""
    return cv2.flip(img, 1)
In [46]:
def manipulate_image_batch(input_root: str, input_dirs: list,
                           output_root: str):
    """Replicates subdirectories from input dir into output dir, with each image manipulated."""
    for d in tqdm_notebook(input_dirs):
        input_path = os.path.join(input_root, d)
        for fn in tqdm_notebook(os.listdir(input_path)):
            if is_image(fn):
                img = cv2.imread(os.path.join(input_path, fn))
                img_out = manipulate_image(img)
                output_path = os.path.join(output_root, d, fn)
                cv2.imwrite(output_path, img_out)

Data Frames and Performance Evaluation

In [17]:
def evaluate_dataset(dataset_name: str,
                     part_to_extract_label_from: str,
                     variety: str,
                     is_export_timestamped: bool=True):
    """
    Diagnose dataset and return results as a data frame, with truth labels.
    """
    dir_list = [
        os.path.join(config['data'][dataset_name]['root'], d)
        for d in config['data'][dataset_name]['dirs']
    ]
    output_dir = os.path.join(config['report']['dir'])

    diagnoses = diagnose_batch(
        dir_list,
        part_to_extract_label_from=part_to_extract_label_from,
        variety=variety)
    df = pd.DataFrame(unstack_diagnoses(diagnoses))

    report_basename = dataset_name
    if is_export_timestamped:
        report_basename = report_basename + "_" + datetime.datetime.now(
        ).strftime("%Y-%m-%d")
    df.to_csv(os.path.join(output_dir, report_basename + '.csv'))
    df.to_excel(os.path.join(output_dir, report_basename + '.xlsx'))

    return df
In [18]:
def has_shared_token(a: str, b: str):
    """Indicates whether or not the input strings share any tokens."""
    tokens_a = nltk.word_tokenize(a.lower())
    tokens_b = nltk.word_tokenize(b.lower())
    return not set(tokens_a).isdisjoint(tokens_b)
In [19]:
def is_fuzzy_match(a: str, b: str, threshold: float):
    """Indicates whether or not the inputs satisfy a fuzzy matching."""
    return fuzz.ratio(a.lower(), b.lower()) >= threshold
In [20]:
def augment_df(df,
               col_1_name: str,
               col_2_name: str,
               new_col_name: str,
               bivariate_function):
    """Returns dataframe with an extra column computed from two user-specified columns."""
    df_new = df.copy()
    s = pd.Series(
        map(lambda u, v: bivariate_function(u, v), 
            df[col_1_name],
            df[col_2_name]),
        index=df.index)
    df_new[new_col_name] = s
    return df_new

Performance Evaluation

Public data

In [50]:
df_public = evaluate_dataset('public', 'dir', 'MAIZE')
df_public_checked = augment_df(df_public, 'label', 'name', 'correct_diagnosis', has_shared_token)

In [51]:
df_public_checked_grouped = df_public_checked.groupby('filename')[col_names_grouped].apply(lambda df: df.reset_index(drop=True))
df_public_checked_grouped
Out[51]:
label correct_diagnosis name rank similarity recognized scientific_name peat_id
filename
PEAT_20160803_145433_d13411e4-330d-4d1d-97ab-e9613c9f4ea5.jpg 0 healthy maize True Healthy Maize 1 100 True Healthy 999999
PEAT_20160803_145508_85d9410f-5452-42da-b471-2d80a4e9d94e.jpg 0 healthy maize True Healthy Maize 1 99 True Healthy 999999
PEAT_20160803_145600_d13411e4-330d-4d1d-97ab-e9613c9f4ea5.jpg 0 healthy maize True Healthy Pea 1 97 True Healthy 999999
PEAT_20160803_145645_d13411e4-330d-4d1d-97ab-e9613c9f4ea5.jpg 0 healthy maize True Healthy Maize 1 99 True Healthy 999999
PEAT_20160803_145749_c6593fcd-0e86-443d-9f2b-e47f9aa76e8e.jpg 0 healthy maize True Healthy Maize 1 100 True Healthy 999999
PEAT_20160803_145921_d13411e4-330d-4d1d-97ab-e9613c9f4ea5.jpg 0 healthy maize True Healthy Maize 1 99 True Healthy 999999
PEAT_20160803_145933_a4c81506-f880-470b-ab89-f483d6bb699a.jpg 0 healthy maize True Healthy Maize 1 100 True Healthy 999999
PEAT_20160803_145952_c6593fcd-0e86-443d-9f2b-e47f9aa76e8e.jpg 0 healthy maize True Healthy Maize 1 99 True Healthy 999999
PEAT_20160803_150017_d13411e4-330d-4d1d-97ab-e9613c9f4ea5.jpg 0 healthy maize True Healthy Maize 1 100 True Healthy 999999
PEAT_20160803_150107_a4c81506-f880-470b-ab89-f483d6bb699a.jpg 0 healthy maize True Healthy Maize 1 99 True Healthy 999999
PEAT_20160831_175819_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 99 True Puccinia sorghi 100082
PEAT_20160831_175924_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 100 True Puccinia sorghi 100082
PEAT_20160831_180052_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 100 True Puccinia sorghi 100082
PEAT_20160831_180431_104c5fb4-60fa-4b99-bfa9-66c4db3abf1d.jpg 0 common rust True Common Rust 1 99 True Puccinia sorghi 100082
PEAT_20160831_180902_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 100 True Puccinia sorghi 100082
PEAT_20160831_181319_104c5fb4-60fa-4b99-bfa9-66c4db3abf1d.jpg 0 common rust True Common Rust 1 99 True Puccinia sorghi 100082
PEAT_20160831_181352_104c5fb4-60fa-4b99-bfa9-66c4db3abf1d.jpg 0 common rust True Common Rust 1 100 True Puccinia sorghi 100082
PEAT_20160831_184819_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 99 True Puccinia sorghi 100082
PEAT_20160831_185052_a9185344-b4a9-473a-b3bb-f88d8a7fd2b7.jpg 0 common rust True Common Rust 1 100 True Puccinia sorghi 100082
In [52]:
df_public_stats = df_public_checked.pivot_table(index='filename')[['correct_diagnosis']].apply(np.ceil)
df_public_stats.describe()
Out[52]:
correct_diagnosis
count 19.0
mean 1.0
std 0.0
min 1.0
25% 1.0
50% 1.0
75% 1.0
max 1.0

This shows that PEAT.ai correctly diagnoses 19 out of 19 of the images provided as examples in their public Github repository.

Public data, manipulated

In [35]:
input_root = conf['data']['public']['root']
input_dirs = conf['data']['public']['dirs']
output_root = conf['data']['public_manipulated']['root']
manipulate_image_batch(input_root, input_dirs, output_root)
In [ ]:
df_public_manipulated = evaluate_dataset('public_manipulated', 'dir', 'MAIZE')
In [226]:
df_public_manipulated_checked = augment_df(df_public_manipulated, 'label',
                                           'name', 'correct_diagnosis',
                                           has_shared_token)
df_public_manipulated_stats = df_public_manipulated_checked.pivot_table(
    index='filename')[['correct_diagnosis']].apply(np.ceil)
df_public_manipulated_stats.describe()
Out[226]:
correct_diagnosis
count 19.0
mean 1.0
std 0.0
min 1.0
25% 1.0
50% 1.0
75% 1.0
max 1.0

The diagnoses on the publicly provided training images are still correct after left-right flipping is applied.

TODO: Experiment with additional minor transformations, such as sharpening, blurring, and lighting change.

Private dataset

We now try our own maize imagery collected by associates in East Africa. In each case, the correct diagnosis is described by the filename and provided by a specialist with doctoral qualifications.

In [41]:
df_private = evaluate_dataset('private', 'file', 'MAIZE')

In [42]:
df_private_checked = augment_df(df_private, 'label', 'name',
                                'correct_diagnosis', has_shared_token)
In [43]:
df_private_checked_grouped = df_private_checked.groupby('filename')[col_names_grouped].apply(lambda df: df.reset_index(drop=True))
df_private_checked_grouped
Out[43]:
label correct_diagnosis name rank similarity recognized scientific_name peat_id
filename
Anthracnose.JPG 0 anthracnose False Pyricularia Leaf Spot 1 37 False Pyricularia grisea 100120
1 anthracnose True Anthracnose Leaf Blight 2 29 False Colletotrichum graminicola 100084
2 anthracnose False Healthy Raspberry 3 9 False Healthy 999999
Boron.JPG 0 boron False Spotted Stemborer 1 72 False Chilo partellus 600030
1 boron False Millet Downy Mildew 2 6 False Sclerospora graminicola 100004
Calcium_1.JPG 0 calcium False Spotted Stemborer 1 96 True Chilo partellus 600030
Calcium_2.JPG 0 calcium False Spotted Stemborer 1 99 True Chilo partellus 600030
Coccinia_Leaf_Blight.JPG 0 coccinia leaf blight False Millet Rust 1 81 True Puccinia substriata 100089
1 coccinia leaf blight False Healthy Millet 2 15 True Healthy 999999
Drought_1.JPG 0 drought False Spotted Stemborer 1 98 True Chilo partellus 600030
Drought_2.JPG 0 drought False Healthy Millet 1 99 True Healthy 999999
Drought_and_phosphorus.JPG 0 drought and phosphorus False Healthy Millet 1 99 True Healthy 999999
Eyespot.JPG 0 eyespot False Anthracnose Leaf Blight 1 47 False Colletotrichum graminicola 100084
1 eyespot False Common Rust 2 33 False Puccinia sorghi 100082
2 eyespot False Millet Rust 3 8 False Puccinia substriata 100089
Eyespot_1.JPG 0 eyespot False Anthracnose Leaf Blight 1 98 True Colletotrichum graminicola 100084
Eyespot_2.JPG 0 eyespot False Wheat Leaf Rust 1 90 True Puccinia triticina 100059
Gray_leaf_spot_1.JPG 0 gray leaf spot True Turcicum Leaf Blight 1 90 True Exserohilum turcicum 100065
1 gray leaf spot False Spotted Stemborer 2 7 True Chilo partellus 600030
Gray_leaf_spot_2.JPG 0 gray leaf spot False Millet Rust 1 90 True Puccinia substriata 100089
Magnesium_1.JPG 0 magnesium False Anthracnose Leaf Blight 1 75 True Colletotrichum graminicola 100084
1 magnesium False Maize Lethal Necrosis Disease 2 8 True MLND 200022
Magnesium_2.JPG 0 magnesium False Common Rust 1 60 False Puccinia sorghi 100082
1 magnesium False Pseudostem Weevil 2 25 False Odoiporus longicollis 600032
Magnesium_3.JPG 0 magnesium False Common Rust 1 74 False Puccinia sorghi 100082
1 magnesium False Pseudostem Weevil 2 8 False Odoiporus longicollis 600032
Magnesium_4.JPG 0 magnesium False Anthracnose Leaf Blight 1 78 True Colletotrichum graminicola 100084
1 magnesium False Common Rust 2 13 True Puccinia sorghi 100082
Maize_streak_virus.JPG 0 maize streak virus True Maize Lethal Necrosis Disease 1 72 False MLND 200022
1 maize streak virus False Millet Downy Mildew 2 11 False Sclerospora graminicola 100004
... ... ... ... ... ... ... ... ... ...
Nitrogen_and_aphids.JPG 2 nitrogen and aphids False Spotted Stemborer 3 14 False Chilo partellus 600030
3 nitrogen and aphids False Aphid Maize 4 7 False Aphidoidea family 600002
Phosphorus_1.JPG 0 phosphorus False Red Rot 1 99 True Glomerella tucumanensis 100052
Phosphorus_2_eyespot.JPG 0 phosphorus eyespot False Healthy Sorghum 1 86 True Healthy 999999
1 phosphorus eyespot False Anthracnose Leaf Blight 2 6 True Colletotrichum graminicola 100084
Phosphorus_3.JPG 0 phosphorus False grasshopper 1 25 False UNKNOWN 0
1 phosphorus False Red Rot 2 12 False Glomerella tucumanensis 100052
2 phosphorus False Anthracnose Leaf Blight 3 11 False Colletotrichum graminicola 100084
3 phosphorus False Spotted Stemborer 4 9 False Chilo partellus 600030
Phosphorus_4.JPG 0 phosphorus False Anthracnose Leaf Blight 1 66 False Colletotrichum graminicola 100084
1 phosphorus False agaric 2 29 False UNKNOWN 0
Phosphorus_5.JPG 0 phosphorus False papaya 1 42 False UNKNOWN 0
1 phosphorus False Anthracnose Leaf Blight 2 15 False Colletotrichum graminicola 100084
2 phosphorus False Healthy Sorghum 3 8 False Healthy 999999
3 phosphorus False Spotted Stemborer 4 6 False Chilo partellus 600030
Phosphorus_6.JPG 0 phosphorus False Blossom End Rot 1 59 False blossom end rot 700001
1 phosphorus False hot pepper 2 7 False UNKNOWN 0
2 phosphorus False Healthy Carrot 3 6 False Healthy 999999
Rust.JPG 0 rust True Common Rust 1 75 True Puccinia sorghi 100082
Rust_1.JPG 0 rust False Powdery Mildew on Mango 1 91 True Oidium mangiferae 100069
Rust_2.JPG 0 rust True Common Rust 1 90 True Puccinia sorghi 100082
Rust_3.JPG 0 rust True Millet Rust 1 53 False Puccinia substriata 100089
1 rust False Red Rot 2 46 False Glomerella tucumanensis 100052
Stemborer.JPG 0 stemborer False Anthracnose Leaf Blight 1 99 True Colletotrichum graminicola 100084
Stemborer_1.JPG 0 stemborer False Anthracnose Leaf Blight 1 99 True Colletotrichum graminicola 100084
Striga.JPG 0 striga False Tulipa gesneriana 1 38 False UNKNOWN 0
1 striga False Healthy Cherry 2 9 False Healthy 999999
2 striga False Healthy Chickpea 3 9 False Healthy 999999
Zinc_1.JPG 0 zinc False Corn Worm 1 54 False Helicoverpa zea 600040
1 zinc False Healthy Millet 2 40 False Healthy 999999

103 rows × 8 columns

The results now are much more diverse, presented using the dataframe above. The label column describes the correct diagnosis, provided by humans. The "name" column and everything to the right of it is provided by PEAT.ai. Note that PEAT.ai provides multiple diagnoses for the same image when it is not sure, ranking each of the diagnoses by a weight denoted as similarity.

The "correct_diagnosis" column indicates whether or not PEAT's diagnosis matched the diagnosis from human experts. To measure matching, we have adopted the most liberal metric of correctness possible, which we call "has_shared_token": there exists at least one word shared by both the label and the name columns. Note that under this metric, a label of "rust" and a diagnosis of "Millet Rust" will be treated as correct, even though the plant being imaged is not millet.

To quantify the accuracy of the diagnoses, we try two different metrics, the first of which is below:

  • At least one diagnosis shares a token with the correct diagnosis: Award PEAT.ai a point if any of the diagnoses returned for a given image share a token with the correct diagnosis, regardless of rank and weighting.

Implementation: Pivot the dataframe about each image, averaging the correct_diagnosis values across all diagnoses for that image. Apply the ceiling operator to all cells in the pivot table, and describe the resulting dataframe.

In [93]:
df_private_stats = df_private_checked.pivot_table(index='filename')[['correct_diagnosis']].apply(np.ceil)
df_private_stats
Out[93]:
correct_diagnosis
filename
Anthracnose.JPG 1.0
Boron.JPG 0.0
Calcium_1.JPG 0.0
Calcium_2.JPG 0.0
Coccinia_Leaf_Blight.JPG 0.0
Drought_1.JPG 0.0
Drought_2.JPG 0.0
Drought_and_phosphorus.JPG 0.0
Eyespot.JPG 0.0
Eyespot_1.JPG 0.0
Eyespot_2.JPG 0.0
Gray_leaf_spot_1.JPG 1.0
Gray_leaf_spot_2.JPG 0.0
Magnesium_1.JPG 0.0
Magnesium_2.JPG 0.0
Magnesium_3.JPG 0.0
Magnesium_4.JPG 0.0
Maize_streak_virus.JPG 1.0
Manganese.JPG 0.0
Manganese_2.JPG 0.0
Manganese_3.JPG 0.0
Nitrogen_1.JPG 0.0
Nitrogen_10.JPG 0.0
Nitrogen_11.JPG 0.0
Nitrogen_12.JPG 0.0
Nitrogen_13.JPG 0.0
Nitrogen_14.JPG 0.0
Nitrogen_15.JPG 0.0
Nitrogen_16.JPG 0.0
Nitrogen_17.JPG 0.0
Nitrogen_18.JPG 0.0
Nitrogen_2.JPG 0.0
Nitrogen_20.JPG 0.0
Nitrogen_21.JPG 0.0
Nitrogen_3.JPG 0.0
Nitrogen_4.JPG 0.0
Nitrogen_5.JPG 0.0
Nitrogen_6.JPG 0.0
Nitrogen_8.JPG 0.0
Nitrogen_9.JPG 0.0
Nitrogen_and_aphids.JPG 0.0
Phosphorus_1.JPG 0.0
Phosphorus_2_eyespot.JPG 0.0
Phosphorus_3.JPG 0.0
Phosphorus_4.JPG 0.0
Phosphorus_5.JPG 0.0
Phosphorus_6.JPG 0.0
Rust.JPG 1.0
Rust_1.JPG 0.0
Rust_2.JPG 1.0
Rust_3.JPG 1.0
Stemborer.JPG 0.0
Stemborer_1.JPG 0.0
Striga.JPG 0.0
Zinc_1.JPG 0.0
In [45]:
df_private_stats.describe()
Out[45]:
correct_diagnosis
count 55.000000
mean 0.109091
std 0.314627
min 0.000000
25% 0.000000
50% 0.000000
75% 0.000000
max 1.000000

Using this lenient assessment, only 10.9 percent of the images were diagnosed correctly.

Alternatively, a metric that is fairer to the user should incorporate PEAT's rankings, as follows:

  • Top-ranked diagnosis shares a token with the correct diagnosis: Award PEAT a point only if the most highly ranked diagnosis returned for a given image shares a token with the correct diagnosis.

Implementation: Construct a dataframe in which all records with rank not equal to 1 are deleted. Insert the correct_diagnosis column, and describe the resulting dataframe.

In [84]:
df_private_checked_top_rank = df_private_checked.rename(columns={'rank': 'ranking'})
df_private_checked_top_rank = df_private_checked_top_rank_only[df_private_checked_top_rank.ranking == 1]
df_private_checked_top_rank[df_private_checked_top_rank.correct_diagnosis == True]
Out[84]:
code eppo filename label name peat_id pic_id ranking recognized scientific_name similarity correct_diagnosis
17 200 SETOTU Gray_leaf_spot_1.JPG gray leaf spot Turcicum Leaf Blight 100065 90568a1f-a158-42cc-b835-f2e57470d0e3 1 True Exserohilum turcicum 90 True
28 200 UNKNOWN Maize_streak_virus.JPG maize streak virus Maize Lethal Necrosis Disease 200022 f4bbc688-439d-4ae3-90f4-de60dd6386de 1 False MLND 72 True
91 200 PUCCSO Rust.JPG rust Common Rust 100082 4440627d-c822-4162-9cdc-47cd144e7881 1 True Puccinia sorghi 75 True
93 200 PUCCSO Rust_2.JPG rust Common Rust 100082 1074baae-9c96-45d4-83cd-63875530015e 1 True Puccinia sorghi 90 True
94 200 UNKNOWN Rust_3.JPG rust Millet Rust 100089 548ea774-38c1-4af5-9dfc-8e56b2bf15db 1 False Puccinia substriata 53 True

Comments on the records above:

  • Turcicum Leaf Blight is not Gray Leaf Spot.
  • MSV and MLND are not the same.
  • Millet Rust should not be confused with Maize Rust.

After removing these three records from the list above, the diagnosis accuracy is 2/55 = 3%.