Full article title Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology
Journal Nature Communications
Author(s) Signoroni, Alberto; Ferrari, Alessandro; Lombardi, Stefano; Savardi, Mattia; Fontana, Stefania; Culbreath, Karissa
Author affiliation(s) University of Brescia, Copan WASP, Tricore Laboratories
Primary contact Email: alberto dot signoroni at unibs dot it
Year published 2023
Volume and issue 14
Article # 6874
DOI 10.1038/s41467-023-42563-1
ISSN 2041-1723
Distribution license Creative Commons Attribution 4.0 International
Website https://www.nature.com/articles/s41467-023-42563-1
Download https://www.nature.com/articles/s41467-023-42563-1.pdf (PDF)

Abstract

Full laboratory automation is revolutionizing work habits in an increasing number of clinical microbiology facilities worldwide, generating huge streams of digital images for interpretation. Contextually, deep learning (DL) architectures are leading to paradigm shifts in the way computers can assist with difficult visual interpretation tasks in several domains. At the crossroads of these epochal trends, we present a system able to tackle a core task in clinical microbiology, namely the global interpretation of diagnostic bacterial culture plates, including presumptive pathogen identification. This is achieved by decomposing the problem into a hierarchy of complex subtasks and addressing them with a multi-network architecture we call DeepColony. Working on a large stream of clinical data and a complete set of 32 pathogens, the proposed system is capable of effectively assisting plate interpretation with a surprising degree of accuracy in the widespread and demanding framework of urinary tract infections (UTIs). Moreover, thanks to the rich species-related generated information, DeepColony can be used for developing trustworthy clinical decision support services in laboratory automation ecosystems from local to global scale.

Keywords: bacteriology, clinical microbiology, computer science, urinary tract infection, laboratory automation, deep learning

Introduction

Microbiology is faced with tremendous questions, with bacterial , viral, and parasitic infections representing major threats to human health[1], in conjunction with concerns such as new species discovered annually[2], re-emerging pathogens[3][4], zoonotic infections[5], and antimicrobial resistance.[6] Correct and timely identification of pathogens is essential to effectively fight infections and the interpretation of bacterial cultures from human collected samples is a pivotal step in the clinical process. Over the last 30 years, molecular biology-based techniques (from polymerase chain reaction [PCR] to next-generation sequencing [NGS][7][8]), vibrational spectroscopy[9], and mass spectrometry (MS) diagnostic tools such as matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-ToF)[10][11] have also emerged as critical tools in the clinical microbiology laboratory (CML) for identification of pathogens. However, routine bacterial culture continues to be the mainstay for diagnosis of bacterial infectious diseases.[12] The continued use of culture in the CML[13] is due to its role in the recovery of viable organisms, the availability of a given organism for antimicrobial susceptibility testing (AST), the detection of unusual or unexpected pathogens, and the lower costs associated with culture-based methods compared with culture-independent methods.

Notwithstanding the benefits, culture interpretation is often a challenging undertaking even for the skilled microbiologist. Other specialties in clinical laboratory medicine that require subtle interpretation and complex visual diagnostic tasks have already shifted into a digital imaging ecosystem (e.g., urinalysis, hematology, and cytology). However, bacterial culture and culture plate interpretation have remained mostly unchanged from its origins in the nineteenth century. Thus, the entire culture process is still accompanied by the susceptibility to human factors and is still restricted by staff shift-availability, while the skilled workforce available to perform and read the cultures has decreased in the past few years.[14] In this resource-critical context, the recent advent and deployment of full laboratory automation (FLA) systems[15][16][17] has led to levels of automation and standardization of the physical culture steps (e.g., specimen processing, plate streaking, and incubation) along with a step of shooting the culture plates, which makes the plate images available for the application of digital analysis tools (Fig. 1a).


Fig1 Signoroni NatComm23 14.png

Figure 1. a The full laboratory automation (FLA) workflow and the role of the proposed solutions. b DeepColony hierarchical architecture of the operational and decision-making processes. c Example of “enumeration map” (level 0 output) where the cardinality of colony aggregates is indicated with colors and numbers. An “X” appears on colonies discarded because they are located in color distortion areas. d Example (on the same plate) of selected “good colonies” (level 1 output) with their presumptive ID (level 2 output). An “X” appears on colonies discarded because they are not single or too close to the plate border or to confluence areas. Detected species Si are indicated for each colony with colors and corresponding legend. e Outputs of level 3 and 4 are shown on the same plate. Gi refers to species ID updated after similarity-driven contextual grouping, while subsequent clinical relevance interpretation indicates a significant mixed flora plate, where best colonies eligible for picking are indicated with arrows. f Convolutional neural network (CNN) architecture for single colony presumptive ID (red box at level 2). g “Siamese CNN” scheme (red box at level 3) for contextual ID.

FLAs have already demonstrated clinical improvements through early growth detection, better culture quality, shorter turnaround times, and significant cost savings, with highly positive impacts on complementary downstream tasks (e.g., MALDI-ToF identification, AST, genome sequencing).[18][19][20][21] Despite this, considerable challenges still remain open within the digitized laboratory workflow[22][23] regarding the interpretation of culture images. Image analysis solutions in bacterial culture have been developed in specific contexts[18][24][25][26][27] and especially for the use of selective and differential media, including chromogenic agar and other differential media. However, algorithms developed using these methods already benefit from the biochemical properties of media and do not require broad identification of the many types of organisms seen in culture. Moreover, to implement such a tool, laboratories would be required to adopt a new media for their culture process, potentially incurring additional costs.

Few studies have demonstrated the development of an image analysis algorithm using a non-selective, non-differential media commonly used in the laboratory, such as sheep blood agar. Additionally, consistent and replicable culture interpretation remains a clinical challenge requiring significant expertise.[28] Therefore, there is a need for an additional clinical decision support tool that informs the technologist of the expected action based on observation of morphologies of grown organisms to ensure accurate and consistent culture interpretation on a large scale. Such an automated whole-plate interpretation has still not been tackled in its full complexity, and nothing even close to it has been prefigured until now.[16][17][23][29]

We approach these challenging objectives by leveraging the recent digital revolution of clinical microbiology culture and the strengths of deep learning (DL) for the solution of complex tasks.[30][31][32][33] In particular, compared to the most common solutions based on single convolutional neural networks (CNN)[34]], multi-network architectures (e.g., Siamese neural networks) are attractive in our case because of their ability to fit into contexts where decision-making processes are stratified into a complex structure.[35] The system must be designed to generate useful and easily interpretable information and to support expert decisions according to safety-by-design and human-in-the-loop (HITL) policies, aiming at achieving cost-effectiveness and skill-empowerment respectively. This requires an advanced ability to combine species-level identification and quantitation across a possibly wide range of clinically relevant pathogens; a skillful combination of multiple visual, cognitive, and procedural tasks; and a computational architecture capable of reproducing this complex environment involving both single-colony and whole-plate analyses.

Results

DeepColony architecture

We developed DeepColony, a hierarchical multi-network capable of handling all identification, quantitation, and interpretation stages, from the single colony to the whole plate level, whose nested architecture is depicted in Fig. 1b (milestones are indicated with green circles) and detailed later in the “Methods” section (DeepColony architecture). Firstly, in the challenging context of urinary tract infections (UTIs)[36], the bacterial species of isolated colonies are identified with a degree of accuracy approaching intrinsic visual discrimination limits for the majority of clinically relevant pathogens. Species identification stems from a dual process: a first “pathogen-aware–similarity-agnostic” approach focusing on single colony recognition (Fig. 1b, levels 0, 1 and 2, Fig. 1c, d), followed by a “similarity-aware–pathogen-agnostic” refinement step involving the global plate context (Fig. 1b, level 3, Fig. 1e). As a second milestone, the possibility to integrate the rich body of species-specific structured information generated from levels 0-3 into a rule-based decision system is demonstrated, with the aim of providing clinically accurate culture interpretations (Fig. 1b, level 4, Fig. 1e). Coherently to its action, DeepColony also produces workable information to guide best-colony picking (see Fig. 1e), which is essential for both the confirmation (e.g., MALDI-ToF identification) and the therapy targeting (e.g., antimicrobial susceptibility tests, AST) phases.

Colony-level dataset and phylogenetic grouping

A large labelled clinical dataset has been created for the training of DeepColony at the colony level. Starting from 1,351 unique plate images, a dataset of 26,213 isolated colony images was produced. These represent 32 UTI bacterial and fungal species, constituting 98% of the species that have been observed in three months of the clinical routine of a large CML, and are represented in their clinical variability. To avoid mislabeling, colonies were only drawn from pure flora cultures after MALDI-ToF identification to be used as ground truth-identification (GT-ID). Figure 2 depicts four examples for individual species, with the inclusion of polymorphisms and morphologic diversity, along with histograms indicating the number of available colonies in the testing subset of the dataset. All details about the acquisition and composition of this dataset are given in in the “Methods” section (Datasets). Phylogenetic relations between UTI microorganisms have been further considered aiming to arrange the 32 species into main groups. This is highlighted in the central part of Fig. 2, where a subdivision determined by the terminal branches of the diagram leads to 16 groups of clinical relevance.


Fig2 Signoroni NatComm23 14.png

Figure 2. Four images for each species are presented to exemplify commonly encountered morphological and spectral inter-species similarity and intra-strain polymorphism. Histograms indicate the number of available colony images in the dataset for each pathogen. The central diagram indicates phylogenetic links and proposed groupings (also delimited by the vertical lines between the pathogen names). Source data are provided as a Source Data file.

Single colony identification (level 0 to 2)

Quantitation of bacterial growth is the first step in the plate culture significance assessment. At level 0, in Fig. 1b, a DL-based colony counting method[25] is used to produce an “enumeration map” (depicted in Fig. 1c). At level 1, this map is used to select “good colonies,” i.e., colonies isolated from the confluent groupings and well developed among all single colonies on the plate (Fig. 1d). Importantly, level 1 takes into account species polymorphism and colony maturity when selecting the more reliable colonies. Following the selection of isolated colonies, level 2 completes the task of attributing a presumptive species-level identification to each selected bacterial colony. To this end, a CNN architecture operating on colony image segments has been designed to generate a “presumptive identification vector” (pIDv) consisting of a confidence-based ranking of most probable bacterial species (out of the 32 possible ones) for each image segment generated by level 1. The CNN scheme (with an example of the resulting pIDv) is shown in Fig. 1f, while technical details and the training using our colony-level dataset are described in the “Methods” section (Single colony identification). At the completion of levels 0-2 (Fig. 1d) single good colonies are identified for a “pathogen-aware–similarity-agnostic” description based on the most probable (Top-1 ID) species in pIDv for each colony.

Algorithm performance on the testing subset is presented in Fig. 3a by means of a 32 × 32 confusion matrix representing Top-1 ID (rows) versus GT-ID (columns), in absolute occurrence values, with an overall identification accuracy at this stage of 83.4% compared to the ground truth. Analysing the errors (off-diagonal entries), the main evidence is that most misclassifications follow a block-diagonal pattern. Given the non-casual ordering of species, this is evidence that errors mainly occur within the same phylogenetic group. This is confirmed by assessing the performance on the diagnostic/phylogenetic subdivision of 16 classes (Fig. 3b), where it can be observed that the overall accuracy increases to 88.4%. Given the objective of an assisted presumptive diagnosis and to further exploit the classification confidence information contained in the pIDv, it is of interest to include in our evaluation also Top-2 and Top-3 species accuracies, where Top-n accuracy is computed by considering samples as "correct" if the true species is listed in the n most probable classes (out of the possible 32) suggested by the network. The Top-2 accuracy is 92.3%, while for Top-3 a value of 95.5% is reached.

With morphologically similar organisms grouped, most of the residual misclassifications in Fig. 3b remain near the diagonal. This is a further evidence of the added value of the proposed diagnostic/phylogenetic grouping/ordering. For example, small colonies such as Lactobacillus species are sometimes confused with Aerococcus urinae, and vice-versa; Streptococcus agalactiae colonies are sometimes confused with Enterococcus faecalis or faecium and vice-versa. Similarly, few Streptococcus mitis colonies are misclassified as Pseudomonas aeruginosa, while some Enterococcus faecium and Candida species are labelled as Lactobacillus species. This morphological misclassification is similar to what may be observed by a technologist. Additionally, the naturally skewed composition of the colony-level dataset (only partially compensated for least represented classes) has a slight influence on the balance of the confusion between species. This is observable for Escherichia coli (constituting 16% of the dataset), which is more frequently erroneously identified than misclassified as another species. In summary, as far as level 2, by combining different interpretative criteria (diagnostic-phylogenetic interpretation and ID vector confidence exploitation), we are able to provide a first classification of organisms into very specific and clinically significant morphologic groups but, interestingly, we have not yet reached all what can be done to support microbiologists toward plate interpretation.


Fig3 Signoroni NatComm23 14.png

Figure 3. a 32 × 32 confusion matrix for single colony species ID (in absolute value). Phylogenetic group subdivisions are indicated with black lines. b 16 × 16 confusion matrix for single colony phylogenic group ID (in relative value referred to the group-based GT). c 32 × 32 matrix indicating relative increase/decrease (in absolute value and referred to a) after similarity-based smoothing. d 3D 16 × 16 matrix with column height indicating relative increment (blue)/decrement (pink) referred to baseline (green) matrix b. Source data are provided as a Source Data file.

Context-based Identification (level 3)

In levels 0-2, each single colony is assessed independently. However, in clinical application, the entire context of the plate must be assessed to determine if observed colonies are similar (pure culture) or different (mixed cultures). At level 3, a “similarity-aware—pathogen-agnostic” refinement step is applied, involving the global context of the plate and requiring the computation of a similarity metric. This requires an abstraction from the single colony perspective; now, visually similar colonies, which could possibly belong to different species, must be simultaneously considered and compared to the overall growth, with the aim of improving the consistency of the presumptive identification and of honing the precision of the tool in doubtful situations. This delicate task is addressed through a “context-based regularization” approach based on a non-linear similarity-driven embedding followed by a clustering in the embedded space that enables an enforcement of the identification consistency. For the dimensionality reduction embedding, a Siamese CNN[37][38] has been trained, whose architecture is shown in Fig. 1g, exploiting an extensive training set comprising 200,000 image pairs (plus 10,000 image pairs for validation) created specifically for this purpose. Once the isolated colonies are mapped in an embedding space, Mean-shift clustering[39] is used to define similarity-based colony clusters. Additional details are discussed in the “Methods” section (Context-based Identification). A final classification is given for each identified cluster by averaging the pIDv’s from level 2 within the cluster, and this allows an increase in classification consistency. As shown in Fig. 1e, the ID labels are now assigned to group of colonies instead of single colonies (and accordingly the pIDv’s).

This regularization method resulted effectively in improving overall identification performance and interpretation accuracy due to the quality and coherence of the embedding and clustering combination. On the 32 bacterial species, Top-1 accuracy increased to 90.6% (Top-2 96.6%, Top-3 97.6%) which is an increment of 7.2 percentage points (4.3 and 2.1 pps for Top-2 and Top-3 respectively). Figure 3c reports absolute values of increased (blue) or decreased (red) accuracy related to each species, compared to the level 2 values in Fig. 3a. Consistent with the significantly increased accuracy, blue squares are mainly localized on the matrix diagonal, while out of diagonal red squares show significant misclassification reductions. A 3D representation of variations in the 16×16 confusion matrix of the diagnostic/phylogenetic species aggregations is shown in Fig. 3d, where green columns correspond to Fig. 3c values, and the blue/pink bars indicate the increment/reduction portions. Improved accuracy (i.e., diagonal increments and out of diagonal reductions) is observed in the large majority of cases, with an overall accuracy of 93.8% (5.4 pps increase). As a mild side effect, the sum of misclassification of both Streptococcus mitis and oralis, instead of balancing between them, accounts for the slight increment in misclassification of viridans group streptococci (VGS) as Aerococcaceae and Pseudomonadacee (+7.9% and a + 1.1%, respectively) while the correct classification decreases (pink top on green column representing −5.6%).

In level 3, we thus demonstrate that applying a “similarity-aware–pathogen-agnostic” refinement improves the specificity of the organism identification, particularly in the context of mixed cultures.

Decision support to Culture Interpretation (level 4)

Individual colony interpretation and contextual interpretation provides information for the laboratory to use. However, the laboratory must ultimately synthesize that information to generate a laboratory result based on culture type and laboratory-specific rules. In DeepColony, this is achieved at level 4, where information from level 0 (segmentation and enumeration) and level 3 (pIDv) are combined to compute min-max colony counting range for each identified species on the plate. This opens up the possibility of assisting the decision-making process that is codified in the CML guidelines for the interpretation and subsequent management of culture plates.

A pictorial example of decision about plate significance is given in Fig. 1e, where the plate is interpreted as “mixed flora culture with two significant species.” One colony for each species is also indicated for possible colony picking, which is crucial for the following ID confirmation (by MALDI-ToF) and/or therapeutic assessment (by AST) phases. More in detail, the whole DeepColony decisional process is illustrated in Fig. 4 on three representative plates, exemplifying all the different possible outcomes (in presence of bacterial growth): significant pure flora plate, significant mixed flora plate, and non-significant mixed flora plate. Level 0 and level 1 steps are represented for Plate1 only (Fig. 4a–d), while level 2-4 (Fig. 4e–o) are exemplified on all the three plates. For level 2 plate images, single good colonies are marked with colored circles, indicating the Top-1 ID species of pIDv vector associated to each colony. For level 3, colored circles now indicate the Top-1 ID species of pIDv vector assigned to grouped colonies after context-based regularization. For Plate2 only, the presumptive identification scores related to Top-1 ID, Top-2 ID and Top-3 ID of the pIDv vector are shown for each identified species at level 2 and after group-based contextual regularization at level 3. Plate-level interpretation is generated at level 4 combining the produced species identification and the other quantitative info with the CML rules that, for our case, are depicted in Fig. 5a and detailed in the “Methods” section (Culture Plate Interpretation). For example, the presumptive presence of only two different species on Plate2 (according Top-1 ID of each group), together with CML rules, determine a “significant mixed flora” interpretation with the indication of the identified infective species. This comes with the generation of associated values, e.g., in terms of presumptive identification scores Top-1 ID, Top-2 ID and Top-3 ID (coherently with the first three confidence values of the pIDv vector) for each good colony and for the grouped species, as detailed in Table 1. Similarly, Plate1 is interpreted as “significant pure flora,” while Plate3 is interpreted “non-significant mixed flora.” An additional panel of nine plates, representative of other culture scenarios, is provided in Supplementary Fig. 1. The variegated visual and quantitative information generated constitutes a transparent and informative explainability pattern, allowing the technicians to well-interpret the various phases of the machine-driven decisional processes.


Fig4 Signoroni NatComm23 14.png

Figure 4. a Original P1; b P1 segmentation mask (level 0); c P1 enumeration map (level 0); d Good colonies on P1 (level 1); e, g, k level 2 Top-1 ID on P1, P2 and P3, respectively, where different colours are used to mark single good colonies according to the identified species as listed in the plate legends (Streptococcus agalactiae and Pseudomonas aeruginosa for Plate1 in e, Klebsiella oxytoca, Citrobacter freundii, Pseudomonas aeruginosa, or Klebsiella pneumoniae for P2 in g, eleven different species for P3 in k).; f, h, l level 3 contextual Top-1 ID on P1, P2 and P3 (Streptococcus agalactiae for Plate1 in f, Klebsiella oxytoca and Pseudomonas aeruginosa for P2 in h, eight species for P3 in l); i 3D representation of the first three entries (Top-1 ID, Top-2 ID, and Top-3 ID) of the level 2 pIDv for each good colony of P2 (different colors indicate different species) and anticipated indication of following grouping I and II; j the same 3D representation for grouped species operated in level 3; m, n, o Plate-level interpretation generated at level 4 by the combination of species-related information (ID and quantitation) and CML rules (for this case represented in the block diagram of Fig. 5a). Source data for i and j are provided as a Source Data file.

Fig5 Signoroni NatComm23 14.png

Figure 5. a Shared CML rules for plate significance attribution (here NGUF stands for Normal genitourinary flora). b DeepColony vs Laboratory performance. c Laboratory Impact of machine-assisted plate interpretation.

Table 1. Values of pIDv for each good colony and for grouped species referred to PLATE2 of Fig. 4. Confidence values (presumptive identification scores) are given on a 0 to 1 scale.
Colony n Top-1 ID pIDv value Top-2 ID pIDv value Top-3 ID pIDv value
1 K.oxytoca 0.9991 K.pneumoniae 0.0008 E.cloacae 0.0001
2 K.oxytoca 0.7754 E. aerogenes 0.2067 K.pneumoniae 0.0126
3 K.oxytoca 0.9111 E.cloacae 0.0446 K.pneumoniae 0.0439
4 K.oxytoca 0.6104 K.pneumoniae 0.38098 E.cloacae 0.0042
5 K.oxytoca 0.9974 K.pneumoniae 0.0024 E.cloacae 0.0001
6 K.pneumoniae 0.8237 K.oxytoca 0.1356 E.coli 0.0274
7 P.aeruginosa 0.7192 E.coli 0.2355 C.freundii 0.0427
8 C.freundii 0.5966 E.coli 0.355 P.aeruginosa 0.0435
Group of colonies n Top-1 ID pIDv value Top-2 ID pIDv value Top-3 ID pIDv value
I K.oxytoca 0.7382 K.pneumoniae 0.2107 E. aerogenes 0.0357
II P.aeruginosa 0.3813 C.freundii 0.3196 E.coli 0.2953

Plate-level decision (DeepColony vs humans) on a large clinical dataset

Given the ability to perform these complex interpretations, DeepColony has been applied to a large clinical dataset of 5,051 urine cultures acquired in a large US laboratory. The plate-level clinical dataset (more details in the “Methods” section [Datasets]) was acquired along with the technologists’ interpretation in terms of no growth, contaminated (mixed), or positive culture, according to the laboratory reading guidelines (the same depicted in Fig. 5a and adopted for DeepColony-guided interpretation at level 4). In Fig. 5b, DeepColony interpretation is compared with that of the technologists on the entire set of cultures to determine both class-based and overall human-machine agreement. Despite the intrinsic task difficulty, a high overall agreement of 95.4% (unweighted Kappa of 0.920) between machine-based and manual presumptive identifications is reached[40], with a class-based peak of 99.2% for the negative “no-growth” cultures, and a high agreement of 95.6% also reached for positive cultures. Out-of-diagonal values in the confusion matrix of Fig. 5b remain noticeably low and can be considered marginal, with the exception of the 194 plates, the 7.4% of the 2623 cultures, interpreted as positive by DeepColony, that were instead manually interpreted as contaminated. This lead to a lower agreement of 77.1% for the mixed growth or “contaminated” cultures due to a deliberately precautionary behavior, which is related to “safety by design” criteria: the upper bound of the estimated number of colonies identified within a species is considered, while additional qualitative considerations are limited to allow for technologist interpretation (who may deem a plate as non-significant despite some above-threshold colony forming unit [CFU] counting) aiming at reducing the risk of improperly discarding potentially positive cultures.

Taken together, this study shows that DeepColony can effectively classify and enumerate organisms to be combined with laboratory-based rule systems to augment the microbiologists’ clinical decisions on the interpretation of clinically significant cultures.

Discussion

Human-driven interpretation of culture plates requires the presumptive identification and quantitation of clinically relevant pathogens at both the colony and plate level. This reciprocal and dual perspective drove the design of the DeepColony architecture (Fig. 1), aimed at assisting such a challenging visual task and consequent analytical steps. In levels 0-2, each individual colony is assessed for identification, while at level 3 a contextual refinement of each colony in relation to other colonies is performed. Experienced microbiologists consider the relative variation of colony dimension, position, and maturation to identify (or exclude) polymorphism and thus reliably classify colonies. Trained on enough plates, our system contextually learns both these similarities and differences within a species, in order to correctly interpret not only inter-species similarity but also intra-species or better, since referred to the single plate, intra-strain polymorphism. This multi-level AI-driven action enables species-specific identification and quantitation, and this is unprecedented in a machine-assisted context. The rich body of generated structured information is then used to feed a decision support system compliant to the diagnostic procedures adopted within the CML (level 4) to provide an overall interpretation of the culture.

The results of this study demonstrate a high level of agreement between DeepColony and the interpretation given by the technologist, in particular for the negative cultures (>99% agreement). Among the positive cultures, DeepColony tended to interpret as positive a portion of cultures that were interpreted as contaminated by the laboratory. This is however due to safety-by-design (false negative culling) settings, since the system is here designed as a screening tool to correctly call true negatives, allowing some false positive results. This still provides an advantage in which repetitive work reviewing negative results is lightened, and professionals are better able to focus on the most relevant and/or critical cases. It also enables a series of improvements to the diagnostic process (e.g., traceability, higher degree of standardization, prioritized scheduling), which could result in increased personnel skill and rewards. Ultimately, by giving fully informative decision support and reliably speeding up presumptive diagnosis, a machine-assisted plate interpretation creates the conditions for significant positive repercussions, in terms of precision and timeliness, on the whole infection management workflow, with the prospect of improved and highly responsive patient infection care on a large scale.

Moreover, it is increasingly clear that data-driven artificial intelligence (AI) systems in critical sectors must not only have demonstrable technical functionalities but, to be of real support and to overcome the demands of the emerging regulations[41], they must be designed to guarantee trustworthiness thanks to adequate levels of transparency and explainability.[42] Considering the above observations, our work fits well in the context of liability and can be included in a broader framework for the development of assistive tools for FLA empowerment. This global picture is summarized in Fig. 5c, where the impact multiplication factors are evidenced and linked to the main traits of the proposed approach. DeepColony can be therefore configured to safely reduce the laboratory workload and to make decisions which are highly coherent with assigned interpretation guidelines, guiding and supporting the specialists in focusing on more relevant case reports with an increased level of information and assistance to better set the downstream steps.

An intrinsic limitation of DeepColony is the current inability to reliably identify species inside confluent areas. This forced us to introduce prudential ranges (i.e., confidence intervals) in the final phase of species-level concentration estimation. However, if there are multiple organism morphologies, the culture may be classified as “contaminated” or flagged for additional review. Because of the safety-by-design feature intrinsic to this system, this does not appear to be to the detriment of the consistency of the results. The substantial improvement we observed in pIDvs’ estimations led by the contextualized reasoning (which grounds on already reliable single-colony identifications) is further evidence of the fact that residual difficulties encountered by our system are coherent with the actual limits of expert visual interpretation: colonies from distinct species that are indistinguishable or, conversely, high polymorphic variability that risks inducing undesired differentiation within the same species. Additionally, the system can take full advantage of the high quality and resolution of the FLA digitized images, well over what is visible to the naked eye in traditional CML activities. Eventually, in this work we evaluated urine cultures that represent a large variety of organisms. However, there are other organisms that were not evaluated in this analysis that may be prevalent in other culture types (e.g., Streptococcus pneumoniae in sputum, Neisseria meningitidis in cerebrospinal fluid, Neisseria gonorrhea in genital). In these cases, the models would need to be adapted to reflect the range of organisms seen in a particular culture or region.

In conclusion, DeepColony is a unique framework for improving the efficiency and quality of massive routine activities and high-volume decisional procedures in a microbiological laboratory, with great potential to refine and reinforce the critical role of the microbiologist.

Methods

Datasets

The image datasets utilized in this study comprise both colony-level and plate-level data. These images come from high-resolution digital scans of cultured plates acquired from clinical specimens processed by WaspLab (by Copan WASP, Italy) FLA facilities. Digital images were produced using the WaspLab 16Mpixel tri-linear color camera performing pushbroom line-scanning under a calibrated white LED lighting system. Combined with high-quality telecentric optics, this guarantees low geometric distortion and high spatial resolution (24.5 μm/pixel).

Colony-level dataset

This clinical dataset was generated from plates cultured in collaborating labs in the US, where colony images were collected in a fully anonymized form, without any kind of interaction or interference with the laboratory and patient diagnostic/prognostic/therapeutic processes. Standard protocols for specimen processing, plate incubation, and digital image generation were followed.[19][20][21][22] Specifically, all plates, except for a small number of cultures derived from ATCC strains, were inoculated with 1 μl loop on REMEL blood agar from urine samples non-invasively collected. Plate incubation occurred in a CO2 atmosphere at 37 °C, and digital images of all plates were captured between 17 and 23 hours of incubation. Ground truth identification came from Vitek MALDI-ToF laboratory analyses. The size of colonies varies widely, ranging from approximately 0.4 mm (~20 pixels) to about 7 mm (~300 pixels) in diameter. Colony-level (i.e., single colony) images were provided using level 0 segmentation (Fig.1a)[25], which creates a mask that excluded confluent areas and colonies with underdeveloped morphology (due to proximity to highly confluent growth or the plate border) as they are not representative of the bacterial colony morphology. Additionally, pixels outside the main colony segment were set to zero to remove clutter (portions of other segments in the neighboring area). To standardize colony-level data, images containing single colonies were padded with a fixed 15 pixels margin along the longer edge, and a suitable margin on the other edge for making them square.

A quantitative description of the colony-level dataset is provided in Supplementary Table 1. From 1,351 pure flora plate images (1,321 from clinical specimens and 30 from ATCC), 26,213 isolated colony images (24,781 clinical and 1,432 from ATCC) were extracted. A 60/20/20 subdivision in training/validating/testing subsets was performed at the plate level (random selection among each of the 32 represented species) in order to avoid having colonies from the same plate appearing in different portions of the colony-level dataset. The table also provides information about the 16 proposed groups, which were derived from phylogenetic relations and diagnostic relevance criteria. In particular, phylogenetically related species—like Morganellaceae, Aereococaceae, Enterococcaceae—were collapsed together, as were pathogens where a joint identification could have a more diagnostic/clinical relevance than separate identifications (e.g., Enterobacteriaceae KES, grouping the pathogens Serratia marcescens and Enterobacter cloacae with Enterobacter aerogenes, Klebsiella oxytoca, and Klebsiella pneumoniae). However, in some cases, despite the phylogenetic relationship with neighbor species, the clinical relevance of a single species led us to maintain it as a separated category, e.g., Staphylococcus saprophyticus versus other coagulase-negative staphylococci (CNS), or Streptococcus agalactiae versus beta-haemolytic streptococci, or Candida albicans versus other non-Candida albicans species.

Plate-level clinical dataset

This comprises 5,051 WaspLab clinical images (from same number of plates) taken from the whole flow of about one week of examinations from urine sample cultures performed at TriCore Reference Laboratories in Albuquerque, NM (US). As for the colony-level dataset, all the plates were inoculated from urine clinical specimens using a 1 μl loop on REMEL blood agar, and incubated from 17 to 23 hours in CO2 at 37 °C. After screening them with a Sismex urine analyser, which identifies negative urine samples that can be excluded from the culturing process[43], plates were labeled by the staff on duty according to shared laboratory rules (depicted in Fig. 5a and detailed below), nominally adopted by both lab technologists/microbiologists and DeepColony, to determine the clinical significance of the culture.

DeepColony architecture

DeepColony is a comprehensive framework for colony identification and analysis conceived to operate in high-throughput CMLs. It is constituted of different CNNs that exchange information and cooperate in a hierarchical structure. As depicted in Fig. 1b, this structure is organized in five levels, from level 0 to level 4, each one addressing a specific aspect of the analysis, allowing to work at different interpretation scales, from the individual colonies to the whole plate. A pictorial representation of the outcomes of the different levels is presented in Fig. 1c–e, while a detailed algorithmic description of the whole process is given in the Supplementary Pseudocode.

At level 0, DeepColony determines the locations and quantifies the number of colonies present on the plate. The objective of this initial step is to provide essential information about the spatial distribution and abundance of colonies, being able to identify isolated colonies, to recognize and count touching colonies in small aggregates (up to six), to detect larger confluence areas and to possibly discard outliers (e.g., dirt material) or colonies growing too close to the plate borders. The colony segmentation and enumeration maps are generated by an approach that exploits a first CNN, which have been described in a previous publication[25] and will not be detailed here. As can be seen below, these are not only essential for the interpretation of individual growths but also contribute greatly to the interpretation of the entire plate. An example of the intermediate output of level 0 is shown in Fig. 1c.

At level 1, the system identifies isolated colonies that are good candidates for further analysis and identification. This is why, according the same criteria microbiologists use to select colonies for visual interpretation, the selection of “Good colonies” involves discarding colonies located close to large confluent areas or to other colonies, as they may struggle in obtaining proper nutrients for their own growth, and therefore may not have a sufficiently developed morphology indicative of the phenotypes of their species. Similarly, colonies in proximity of the plate border were also excluded, where not already excluded at level 0, this time because specular reflections and geometrical obstacles may make them unsuitable for the tasks. Colonies are discarded based on minimum distance criteria and exploiting a distance transform operated on the segmentation mask provided by level 0. In Fig. 1d, good colonies are all those circled and thus ready for the subsequent identification and interpretation steps. The core part of DeepColony is represented by levels from 2 to 4. Their role and main traits are summarized here, while their detailed description is given in the following paragraphs.

At level 2, DeepColony makes an initial identification using a CNN that assigns each colony to a particular species, based on visual appearance and growth characteristics. In Fig. 1d, three bacterial species have been identified on the plate and indicated with differently colored circles and symbols Si.

On this basis, level 3 improves the identification robustness and accuracy by incorporating a smoothing of the identification rankings produced by level 2 on the basis of a contextual analysis of the bacterial growth on the plate under analysis. An embedding of single colonies in a similarity space is performed using the feature vector generated by a branch of a Siamese CNN trained to recognize morphological variants of the same strain growing on a plate as similar (i.e., close to each other in the embedding space). This makes a subsequent clustering in the embedding space a meaningful way to regularize the species identification rankings within each cluster with a significant beneficial effect on the identification performance. A clear effect of this smoothing can be seen in Fig. 1e, where the number of identified strains is reduced thanks to the level 3 identification refinement.

Eventually, at level 4, DeepColony assesses the clinical significance of the entire plate and can assist in deciding which colonies should be picked for downstream phases, such as MALDI-ToF identification, secondary cultures, or antimicrobial susceptibility testing (AST). This decision-making process takes into account not only the identification results but also adheres to the specific laboratory guidelines to ensure a proper supportive interpretation in the context of use. Figure 1e indicates the final decision taken on the example plate and the suggested colony to pick for each identified species.

Single colony identification

The CNN for single colony identification schematized in Fig. 1f includes four convolutional layers and one fully connected layer. The hyperparameters of the architecture, including the number of convolutional layers, hidden units, and filter sizes, were carefully tuned using an extensive grid search. Interestingly, the final optimal CNN configuration is substantially equivalent to the best performing topology for cardinality classification also reported for level 0 analyses[25], thus confirming the adequacy of the designed deep neural network architecture and capacity to the problem at hand. Other tests made to justify and confirm other architectural choices (type of pooling and overall complexity) are reported in Supplementary Table 2. Further architectural details are available in Supplementary Note 1.

Context-based identification

Pathogenic species identification using visual data is challenging because of the numerosity of pathogens, the frequently high inter-species visual similarity, but also different forms of intra-strain morphological variability, in both colony dimensions and aspect, that frequently occur even on the same plate (see Supplementary Fig. 2). Hence the need to find a non-trivial similarity assessment strategy capable to both distinguishing visually similar different species and, at the same time, accommodate for different morphotypes of the same species. Even focusing on single colony images, the original data space is high-dimensional and too complex to directly define suitable mappings and/or metrics for assessing such species-related similarity assessment (this has been experimentally verified and reported in Supplementary Fig. 3). To overcome this challenge, a nonlinear similarity-driven embedding was introduced, wherein a data-driven mapping function is employed to convert high-dimensional data to a lower-dimensional space. This is followed by a clustering in the embedding space, where similarity clusters are exploited within a regularization strategy capable to improve, by a significant margin, the overall quality of colony identification, with obvious positive repercussion on quality and reliability of the pre-clinical interpretation of the culture plates. Both these steps are detailed below.

Nonlinear similarity-driven embedding

The adopted approach, based on a Siamese neural network[37][38], can learn a mapping function from colony images to a reduced-dimensionality space capable of generalizing on unseen data. This architecture, depicted in Fig. 1g, employs two Siamese-CNN subnetworks (S-CNN), which share the same weights while working on two different input images. Using a large training dataset of 200,000 pairs and a validation set of 10,000 pairs generated from the colony identification (level 2) dataset, the network was trained to identify genuine and impostor pairs. To ensure good interpretation conditioning of the plate context, same-strain/different-plate pairs were not considered for training purposes. As the goal of this embedding does not involve maximizing a specific accuracy metric, the model selection process was more indirect, with performance evaluated based on the subsequent clustering step. To follow the performance of the model during training, the loss function was monitored on the validation set, expecting it to decrease steadily. The dimension of the embedding space is defined by the last network layer of Fig. 1g and was experimentally set to 15. Further architectural and training details are available in Supplementary Note 2. Once the network has been trained to recognize strain similarities that are expressed on the same plate, one of the twin branches of the Siamese CNN is used to generate a feature vector to embed colonies in a plate-specific similarity space suitable for subsequent ID regularization.

Clustering (mean-shift) in the embedding space

To perform similarity-driven clustering in the embedding space a mean-shift technique[39] is adopted. This method was selected due to its ability to identify clusters with arbitrary shape and its minimal requirements in terms of feature space topology. Additionally, mean-shift clustering does not require strong a priori knowledge, which is often necessary for methods like k-means clustering. The computational efficiency of mean-shift clustering also made it a favorable choice. Gaussian kernels are adopted with bandwidth experimentally optimized on the dataset. To simulate mixed flora plate occurrences, virtual plates were generated containing up to three different strains on each plate. A total of 1,000 virtual plates were created to enable the construction of a realistic experimental setup to train and test our clustering-based approach. This was necessary since complete knowledge of every colony on mixed flora plates by MALDI-ToF identification is not available, and it would be unfeasible and/or not fully justifiable to obtain such information at a large scale. To compare the partition proposed by the clustering algorithm with the ground truth, an overall evaluation indicator v-measure is adopted[44] that combines entropy-based scores indicating homogeneity and completeness of the clustering (a deeper explanation regarding these clustering metrics is provided in Supplementary Note 3 while the obtained best parameters configuration is reported in Supplementary Table 3).

Culture plate interpretation

With information generated from level 0 to 3, both lab technicians and DeepColony can operate according to the same shared rules adopted by the CML to determine whether the growth on each plate is significant. Plate interpretation for clinical significance (level 4) is typically shaped by widely shared guidelines from which laboratories can derive their own specific protocols. The set of rules in use in our reference CML are depicted in Fig. 5a: (1) cultures with fewer than 10 colonies are considered no significant growth; (2) plates with 1–2 detected colony morphologies, or with ≥3 different colony morphologies where one pathogen is prevalent (over 50 CFU) over the normal (under 50 CFU) genito-urinary flora (NGUF), are considered positive, thus to be further evaluated for possible downstream steps (Maldi-ToF ID/AST); (3) otherwise, cultures are considered contaminated—that is, when they manifest ≥ 3 different colony morphologies without a particular pathogen that is prevalent over the others.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary, linked here (.pdf).

Abbreviations, acronyms, and initialisms

  • AI: artificial intelligence
  • AST: antimicrobial susceptibility testing
  • CFU: colony forming unit
  • CML: clinical microbiology laboratory
  • CNN: convolutional neural network
  • DL: deep learning
  • FLA: full laboratory automation
  • HITL: human-in-the-loop
  • MALDI-ToF: matrix-assisted laser desorption/ionization–time-of-flight mass spectrometry
  • MS: mass spectrometry
  • NGF: normal genitourinary flora
  • NGS: next-generations sequencing
  • pIDv: presumptive identification vector
  • PCR: polymerase chain reaction
  • S-CNN: Siamese convolutional neural network
  • UTI: urinary tract infection
  • VGS: viridans group streptococci

Supplementary information

Acknowledgements

This work was supported by the Italian Ministry of Research (Smart Factory project CTN01_00163_216730, A.S., A.F., S.L.) and by collaborative university-industry research activities with Copan WASP srl, Brescia, Italy. Authors want to thank Mario Savarese, Stefano Oliani, Giorgio Triva and Stefania Triva from Copan for their support in the realization of this work. This work is dedicated to the beloved memory of Daniele Triva.

Author contributions

A.S. and A.F. contributed equally. A.S. coordinated all the activities and the writing of the manuscript. A.F. led the development of the core technology and first validations. S.L. and M.S. contributed to both the architectural development and the final validation activities. S.F. contributed to the statistical analysis and representation of the experimental data and to the writing of the manuscript. K.C. conducted the urine culture interpretation portion of the study and participated in the writing of the manuscript.

Data availability

All data supporting the findings described in this manuscript are available in the article and in the Supplementary Information. Source data for all data figures are provided with this paper. The core colony image dataset is available on Figshare as https://doi.org/10.6084/m9.figshare.24203961. The use-case plate dataset is property of Tricore Laboratories (Albuquerque, New Mexico, USA), it was used under license for the current study and is not publicly available. Requests can be addressed to the corresponding author (expected response time two weeks) and access will require explicit permission from Tricore Labs. Source data are provided with this paper.

Code availability

A detailed pseudocode has been provided in the Supplementary Information. The implementation encompasses custom C blocks integrated into Python 2, with models structured in the Caffe framework. The code (architecture and trained models) are not publicly available due to some proprietary restrictions. They can be made available from the authors on request and with permission of Copan WASP.

Peer review

Competing interests

We declare no competing non-financial interests, but the following competing financial interests: A.S. and M.S. were supported by university-industry research contracts from Copan WASP srl (Brescia, Italy) in applying AI in lab automation imaging, A.F. and S.F. were Copan employees, S.L. is a Copan WASP employee, K.C. has been a clinical consultant for Copan WASP.

References

  1. ↑ Antimicrobial Resistance Division (AMR), Global Coordination and Partnership (GCP), Medicines Selection, IP and Affordability (MIA) (4 September 2017). "Prioritization of pathogens to guide discovery, research and development of new antibiotics for drug-resistant bacterial infections, including tuberculosis". World Health Organization. https://www.who.int/publications/i/item/WHO-EMP-IAU-2017.12. 
  2. ↑ Jones, Kate E.; Patel, Nikkita G.; Levy, Marc A.; Storeygard, Adam; Balk, Deborah; Gittleman, John L.; Daszak, Peter (1 February 2008). "Global trends in emerging infectious diseases" (in en). Nature 451 (7181): 990–993. doi:10.1038/nature06536. ISSN 0028-0836. PMC PMC5960580. PMID 18288193. https://www.nature.com/articles/nature06536. 
  3. ↑ Vouga, M.; Greub, G. (1 January 2016). "Emerging bacterial pathogens: the past and beyond" (in en). Clinical Microbiology and Infection 22 (1): 12–21. doi:10.1016/j.cmi.2015.10.010. PMC PMC7128729. PMID 26493844. https://linkinghub.elsevier.com/retrieve/pii/S1198743X1500909X. 
  4. ↑ Heymann, D.L.; Lee, V.J.M. (2015). "8.17 Emerging and re-emerging infections". Oxford Textbook of Global Public Health (6th ed.). Oxford University Press. pp. 1192–1205. ISBN 9780199661756. 
  5. ↑ Chikeka, I.; Dumler, J.S. (1 May 2015). "Neglected bacterial zoonoses" (in en). Clinical Microbiology and Infection 21 (5): 404–415. doi:10.1016/j.cmi.2015.04.022. PMC PMC4466158. PMID 25964152. https://linkinghub.elsevier.com/retrieve/pii/S1198743X15004358. 
  6. ↑ Brown, Eric D.; Wright, Gerard D. (1 January 2016). "Antibacterial drug discovery in the resistance era" (in en). Nature 529 (7586): 336–343. doi:10.1038/nature17042. ISSN 0028-0836. https://www.nature.com/articles/nature17042. 
  7. ↑ Didelot, Xavier; Bowden, Rory; Wilson, Daniel J.; Peto, Tim E. A.; Crook, Derrick W. (1 September 2012). "Transforming clinical microbiology with bacterial genome sequencing" (in en). Nature Reviews Genetics 13 (9): 601–612. doi:10.1038/nrg3226. ISSN 1471-0056. PMC PMC5049685. PMID 22868263. https://www.nature.com/articles/nrg3226. 
  8. ↑ Buchan, Blake W.; Ledeboer, Nathan A. (1 October 2014). "Emerging Technologies for the Clinical Microbiology Laboratory" (in en). Clinical Microbiology Reviews 27 (4): 783–822. doi:10.1128/CMR.00003-14. ISSN 0893-8512. PMC PMC4187641. PMID 25278575. https://journals.asm.org/doi/10.1128/CMR.00003-14. 
  9. ↑ Ho, Chi-Sing; Jean, Neal; Hogan, Catherine A.; Blackmon, Lena; Jeffrey, Stefanie S.; Holodniy, Mark; Banaei, Niaz; Saleh, Amr A. E. et al. (30 October 2019). "Rapid identification of pathogenic bacteria using Raman spectroscopy and deep learning" (in en). Nature Communications 10 (1): 4927. doi:10.1038/s41467-019-12898-9. ISSN 2041-1723. PMC PMC6960993. PMID 31666527. https://www.nature.com/articles/s41467-019-12898-9. 
  10. ↑ Seng, Piseth; Drancourt, Michel; Gouriet, Frédérique; La Scola, Bernard; Fournier, Pierre‐Edouard; Rolain, Jean Marc; Raoult, Didier (15 August 2009). "Ongoing Revolution in Bacteriology: Routine Identification of Bacteria by Matrix‐Assisted Laser Desorption Ionization Time‐of‐Flight Mass Spectrometry" (in en). Clinical Infectious Diseases 49 (4): 543–551. doi:10.1086/600885. ISSN 1058-4838. https://academic.oup.com/cid/article-lookup/doi/10.1086/600885. 
  11. ↑ Croxatto, Antony; Prod'hom, Guy; Greub, Gilbert (1 March 2012). "Applications of MALDI-TOF mass spectrometry in clinical diagnostic microbiology" (in en). FEMS Microbiology Reviews 36 (2): 380–407. doi:10.1111/j.1574-6976.2011.00298.x. ISSN 1574-6976. https://academic.oup.com/femsre/article-lookup/doi/10.1111/j.1574-6976.2011.00298.x. 
  12. ↑ Cornaglia G.; Raoult D. (2012). "Clinical microbiology: A future based on tradition". In Cornaglia, G., Courcol R., Herrmann, J.-L.. European Manual of Clinical Microbiology. European Society of Clinical Microbiology and Infectious Diseases. pp. 17–19. ISBN 9782878050264. 
  13. ↑ Belkumé, Alex van; Durand, raldine; Peyret, Michel; Chatellier, Sonia; Zambardi, Gilles; Schrenzel, Jacques; Shortridge, Dee; Engelhardt, Anette et al. (1 January 2013). "Rapid Clinical Bacteriology and Its Future Impact" (in en). Annals of Laboratory Medicine 33 (1): 14–27. doi:10.3343/alm.2013.33.1.14. ISSN 2234-3806. PMC PMC3535192. PMID 23301218. http://annlabmed.org/journal/view.html?doi=10.3343/alm.2013.33.1.14. 
  14. ↑ Garcia, Edna; Kundu, Iman; Kelly, Melissa; Soles, Ryan (5 July 2019). "The American Society for Clinical Pathology’s 2018 Vacancy Survey of Medical Laboratories in the United States" (in en). American Journal of Clinical Pathology 152 (2): 155–168. doi:10.1093/ajcp/aqz046. ISSN 0002-9173. https://academic.oup.com/ajcp/article/152/2/155/5499263. 
  15. ↑ Sautter, Robert L.; Thomson, Richard B. (1 May 2015). Gilligan, P. H.. ed. "Point-Counterpoint: Consolidated Clinical Microbiology Laboratories" (in en). Journal of Clinical Microbiology 53 (5): 1467–1472. doi:10.1128/JCM.02569-14. ISSN 0095-1137. PMC PMC4400739. PMID 25253793. https://journals.asm.org/doi/10.1128/JCM.02569-14. 
  16. ↑ 16.0 16.1 Rhoads, Daniel D.; Sintchenko, Vitali; Rauch, Carol A.; Pantanowitz, Liron (1 October 2014). "Clinical Microbiology Informatics" (in en). Clinical Microbiology Reviews 27 (4): 1025–1047. doi:10.1128/CMR.00049-14. ISSN 0893-8512. PMC PMC4187636. PMID 25278581. https://journals.asm.org/doi/10.1128/CMR.00049-14. 
  17. ↑ 17.0 17.1 Bailey, Adam L; Ledeboer, Nathan; Burnham, Carey-Ann D (1 May 2019). "Clinical Microbiology Is Growing Up: The Total Laboratory Automation Revolution" (in en). Clinical Chemistry 65 (5): 634–643. doi:10.1373/clinchem.2017.274522. ISSN 0009-9147. https://academic.oup.com/clinchem/article/65/5/634/5608051. 
  18. ↑ 18.0 18.1 Culbreath, Karissa; Piwonka, Heather; Korver, John; Noorbakhsh, Mir (18 February 2021). McElvania, Erin. ed. "Benefits Derived from Full Laboratory Automation in Microbiology: a Tale of Four Laboratories" (in en). Journal of Clinical Microbiology 59 (3): e01969–20. doi:10.1128/JCM.01969-20. ISSN 0095-1137. PMC PMC8106725. PMID 33239383. https://journals.asm.org/doi/10.1128/JCM.01969-20. 
  19. ↑ 19.0 19.1 Bourbeau, Paul P.; Ledeboer, Nathan A. (1 June 2013). Doern, Gary V.. ed. "Automation in Clinical Microbiology" (in en). Journal of Clinical Microbiology 51 (6): 1658–1665. doi:10.1128/JCM.00301-13. ISSN 0095-1137. PMC PMC3716104. PMID 23515547. https://journals.asm.org/doi/10.1128/JCM.00301-13. 
  20. ↑ 20.0 20.1 Ledeboer, Nathan A.; Dallas, Steven D. (1 September 2014). Gilligan, P. H.. ed. "Point-Counterpoint: The Automated Clinical Microbiology Laboratory: Fact or Fantasy?" (in en). Journal of Clinical Microbiology 52 (9): 3140–3146. doi:10.1128/JCM.00686-14. ISSN 0095-1137. PMC PMC4313129. PMID 24648549. https://journals.asm.org/doi/10.1128/JCM.00686-14. 
  21. ↑ 21.0 21.1 Doern, Christopher D.; Holfelder, Martin, Jorgensen, James H.; Carroll, Karen C.; Funke, Guido et al.., eds., "Automation and Design of the Clinical Microbiology Laboratory" (in en), Manual of Clinical Microbiology (Washington, DC, USA: ASM Press): 44–53, doi:10.1128/9781555817381.ch5, ISBN 978-1-68367-280-7, http://doi.wiley.com/10.1128/9781555817381.ch5 
  22. ↑ 22.0 22.1 Dauwalder, O.; Landrieve, L.; Laurent, F.; de Montclos, M.; Vandenesch, F.; Lina, G. (1 March 2016). "Does bacteriology laboratory automation reduce time to results and increase quality management?" (in en). Clinical Microbiology and Infection 22 (3): 236–243. doi:10.1016/j.cmi.2015.10.037. https://linkinghub.elsevier.com/retrieve/pii/S1198743X15009787. 
  23. ↑ 23.0 23.1 Egli, A.; Schrenzel, J.; Greub, G. (1 October 2020). "Digital microbiology" (in en). Clinical Microbiology and Infection 26 (10): 1324–1331. doi:10.1016/j.cmi.2020.06.023. PMC PMC7320868. PMID 32603804. https://linkinghub.elsevier.com/retrieve/pii/S1198743X20303670. 
  24. ↑ Rhoads, Daniel D.; Novak, Susan M.; Pantanowitz, Liron (1 January 2015). "A review of the current state of digital plate reading of cultures in clinical microbiology" (in en). Journal of Pathology Informatics 6 (1): 23. doi:10.4103/2153-3539.157789. PMC PMC4466785. PMID 26110091. https://linkinghub.elsevier.com/retrieve/pii/S2153353922004783. 
  25. ↑ 25.0 25.1 25.2 25.3 25.4 Ferrari, Alessandro; Lombardi, Stefano; Signoroni, Alberto (1 January 2017). "Bacterial colony counting with Convolutional Neural Networks in Digital Microbiology Imaging" (in en). Pattern Recognition 61: 629–640. doi:10.1016/j.patcog.2016.07.016. https://linkinghub.elsevier.com/retrieve/pii/S0031320316301650. 
  26. ↑ Kirn, Thomas J. (1 October 2016). Burnham, C.-A. D.. ed. "Commentary: Automatic Digital Plate Reading for Surveillance Cultures" (in en). Journal of Clinical Microbiology 54 (10): 2424–2426. doi:10.1128/JCM.01279-16. ISSN 0095-1137. PMC PMC5035426. PMID 27510833. https://journals.asm.org/doi/10.1128/JCM.01279-16. 
  27. ↑ Faron, Matthew L.; Buchan, Blake W.; Coon, Christopher; Liebregts, Theo; van Bree, Anita; Jansz, Arjan R.; Soucy, Genevieve; Korver, John et al. (1 October 2016). Burnham, C.-A. D.. ed. "Automatic Digital Analysis of Chromogenic Media for Vancomycin-Resistant-Enterococcus Screens Using Copan WASPLab" (in en). Journal of Clinical Microbiology 54 (10): 2464–2469. doi:10.1128/JCM.01040-16. ISSN 0095-1137. PMC PMC5035414. PMID 27413193. https://journals.asm.org/doi/10.1128/JCM.01040-16. 
  28. ↑ Glasson, John; Hill, Rhys; Summerford, Michael; Giglio, Steven (1 November 2016). Onderdonk, A. B.. ed. "Observations on Variations in Manual Reading of Cultures" (in en). Journal of Clinical Microbiology 54 (11): 2841–2841. doi:10.1128/JCM.01380-16. ISSN 0095-1137. PMC PMC5078566. PMID 27582515. https://journals.asm.org/doi/10.1128/JCM.01380-16. 
  29. ↑ Zhang, Jinghua; Li, Chen; Yin, Yimin; Zhang, Jiawei; Grzegorzek, Marcin (1 February 2023). "Applications of artificial neural networks in microorganism image analysis: a comprehensive review from conventional multilayer perceptron to popular convolutional neural network and potential visual transformer" (in en). Artificial Intelligence Review 56 (2): 1013–1070. doi:10.1007/s10462-022-10192-7. ISSN 0269-2821. PMC PMC9066147. PMID 35528112. https://link.springer.com/10.1007/s10462-022-10192-7. 
  30. ↑ LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey (28 May 2015). "Deep learning" (in en). Nature 521 (7553): 436–444. doi:10.1038/nature14539. ISSN 0028-0836. https://www.nature.com/articles/nature14539. 
  31. ↑ Szegedy, Christian; Wei Liu; Yangqing Jia; Sermanet, Pierre; Reed, Scott; Anguelov, Dragomir; Erhan, Dumitru; Vanhoucke, Vincent et al. (1 June 2015). "Going deeper with convolutions". 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Boston, MA, USA: IEEE): 1–9. doi:10.1109/CVPR.2015.7298594. ISBN 978-1-4673-6964-0. http://ieeexplore.ieee.org/document/7298594/. 
  32. ↑ Silver, David; Huang, Aja; Maddison, Chris J.; Guez, Arthur; Sifre, Laurent; van den Driessche, George; Schrittwieser, Julian; Antonoglou, Ioannis et al. (28 January 2016). "Mastering the game of Go with deep neural networks and tree search" (in en). Nature 529 (7587): 484–489. doi:10.1038/nature16961. ISSN 0028-0836. https://www.nature.com/articles/nature16961. 
  33. ↑ Litjens, Geert; Kooi, Thijs; Bejnordi, Babak Ehteshami; Setio, Arnaud Arindra Adiyoso; Ciompi, Francesco; Ghafoorian, Mohsen; van der Laak, Jeroen A.W.M.; van Ginneken, Bram et al. (1 December 2017). "A survey on deep learning in medical image analysis" (in en). Medical Image Analysis 42: 60–88. doi:10.1016/j.media.2017.07.005. https://linkinghub.elsevier.com/retrieve/pii/S1361841517301135. 
  34. ↑ Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E. (24 May 2017). "ImageNet classification with deep convolutional neural networks" (in en). Communications of the ACM 60 (6): 84–90. doi:10.1145/3065386. ISSN 0001-0782. https://dl.acm.org/doi/10.1145/3065386. 
  35. ↑ Zhang, Zizhao; Chen, Pingjun; McGough, Mason; Xing, Fuyong; Wang, Chunbao; Bui, Marilyn; Xie, Yuanpu; Sapkota, Manish et al. (13 May 2019). "Pathologist-level interpretable whole-slide cancer diagnosis with deep learning" (in en). Nature Machine Intelligence 1 (5): 236–245. doi:10.1038/s42256-019-0052-1. ISSN 2522-5839. https://www.nature.com/articles/s42256-019-0052-1. 
  36. ↑ Flores-Mireles, Ana L.; Walker, Jennifer N.; Caparon, Michael; Hultgren, Scott J. (1 May 2015). "Urinary tract infections: epidemiology, mechanisms of infection and treatment options" (in en). Nature Reviews Microbiology 13 (5): 269–284. doi:10.1038/nrmicro3432. ISSN 1740-1526. PMC PMC4457377. PMID 25853778. https://www.nature.com/articles/nrmicro3432. 
  37. ↑ 37.0 37.1 Bromley, Jane; Bentz, James W.; Bottou, Léon; Guyon, Isabelle; Lecun, Yann; Moore, Cliff; Säckinger, Eduard; Shah, Roopak (1 August 1993). "SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK" (in en). International Journal of Pattern Recognition and Artificial Intelligence 07 (04): 669–688. doi:10.1142/S0218001493000339. ISSN 0218-0014. https://www.worldscientific.com/doi/abs/10.1142/S0218001493000339. 
  38. ↑ 38.0 38.1 Chopra, S.; Hadsell, R.; LeCun, Y. (2005). "Learning a Similarity Metric Discriminatively, with Application to Face Verification". 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) (San Diego, CA, USA: IEEE) 1: 539–546. doi:10.1109/CVPR.2005.202. ISBN 978-0-7695-2372-9. http://ieeexplore.ieee.org/document/1467314/. 
  39. ↑ 39.0 39.1 Comaniciu, D.; Meer, P. (1 May 2002). "Mean shift: a robust approach toward feature space analysis". IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (5): 603–619. doi:10.1109/34.1000236. http://ieeexplore.ieee.org/document/1000236/. 
  40. ↑ Watson, P.F.; Petrie, A. (1 June 2010). "Method agreement analysis: A review of correct methodology" (in en). Theriogenology 73 (9): 1167–1179. doi:10.1016/j.theriogenology.2010.01.003. https://linkinghub.elsevier.com/retrieve/pii/S0093691X10000233. 
  41. ↑ Vokinger, Kerstin N.; Gasser, Urs (10 September 2021). "Regulating AI in medicine in the United States and Europe" (in en). Nature Machine Intelligence 3 (9): 738–739. doi:10.1038/s42256-021-00386-z. ISSN 2522-5839. PMC PMC7611759. PMID 34604702. https://www.nature.com/articles/s42256-021-00386-z. 
  42. ↑ Markus, Aniek F.; Kors, Jan A.; Rijnbeek, Peter R. (1 January 2021). "The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies" (in en). Journal of Biomedical Informatics 113: 103655. doi:10.1016/j.jbi.2020.103655. https://linkinghub.elsevier.com/retrieve/pii/S1532046420302835. 
  43. ↑ Erdman, Patrick; Anderson, Brian; Zacko, J. Christopher; Taylor, Kirk; Donaldson, Keri (1 November 2017). "The Accuracy of the Sysmex UF-1000i in Urine Bacterial Detection Compared With the Standard Urine Analysis and Culture" (in en). Archives of Pathology & Laboratory Medicine 141 (11): 1540–1543. doi:10.5858/arpa.2016-0520-OA. ISSN 0003-9985. http://meridian.allenpress.com/aplm/article/141/11/1540/65372/The-Accuracy-of-the-Sysmex-UF1000i-in-Urine. 
  44. ↑ Rosenberg, A.; Hirschberg, J. (June 2007). "V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure". Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). Association for Computational Linguistics. pp. 410–20. https://aclanthology.org/D07-1043/. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.

source: https://www.limswiki.org/index.php/Journal:Hierarchical_AI_enables_global_interpretation_of_culture_plates_in_the_era_of_digital_microbiology