Full article title Fostering reproducibility, reusability, and technology transfer in health informatics
Journal iScience
Author(s) Hauschild, Anne-Christin; Eick, Lisa; Wienbeck, Joachim; Heider, Dominik
Author affiliation(s) Philipps University of Marburg
Primary contact Email: dominik dot heider at uni-marburg dot de
Year published 2021
Volume and issue 24(7)
Article # 102803
DOI 10.1016/j.isci.2021.102803
ISSN 2589-0042
Distribution license Creative Commons Attribution 4.0 International
Website https://www.sciencedirect.com/science/article/pii/S2589004221007719
Download https://www.sciencedirect.com/science/article/pii/S2589004221007719/pdfft (PDF)

Abstract

Computational methods can transform healthcare. In particular, health informatics combined with artificial intelligence (AI) has shown tremendous potential when applied in various fields of medical research and has opened a new era for precision medicine. The development of reusable biomedical software for research or clinical practice is time-consuming and requires rigorous compliance with quality requirements as defined by international standards.

However, research projects rarely implement such measures, hindering smooth technology transfer to the research community or manufacturers, as well as reproducibility and reusability.

Here, we present a guideline for quality management systems (QMS) for academic organizations incorporating the essential components, while confining the requirements to an easily manageable effort. It provides a starting point to effortlessly implement a QMS tailored to specific needs and greatly facilitates technology transfer in a controlled manner, thereby supporting reproducibility and reusability.

Ultimately, the emerging standardized workflows can pave the way for an accelerated deployment in clinical practice.

Keywords: health informatics, bioinformatics, software engineering, software robustness


Abst1 Hauschild iScience2021 24-7.jpg

Graphical abstract

Introduction

Computational approaches offer new opportunities to transform health care. In particular, modern artificial intelligence (AI) and machine learning (ML) techniques have shown substantial potential when applied in various fields of medical research and therefore have opened up a new era for precision medicine. (Hawgood et al., 2015)[1] In the last decade, universities and other research organizations supported and encouraged by public funding agencies have allocated tremendous efforts to develop and enhance such predictive software models, algorithms, and “systems as medical devices” (SaMD) for clinical research and application.

A manifold of studies have proven AI to be advantageous for disease diagnosis, prognosis, and disease monitoring.[2][3] In cancer research, for instance, ML is used on omics data to gain deeper insights and understanding of the genetic and metabolic alterations that determine disease progression and enable tailored prognoses and monitoring.[4][5][6][7] Additionally, computational models on clinical information are used to assess individualized health risks, for instance, to identify high-risk patients for sepsis in intensive care units[8][9], the analysis of longitudinal data for the early detection of heart failure[10], or applications in infectious diseases.[11][12]

Once the outcome of this development is sufficiently mature and robust to be published and used in routine treatment or diagnosis, there is a need to transfer this knowledge to other research groups and, ultimately, clinical practice. Therefore, a straightforward technology transfer to a manufacturer of medical devices would be beneficial.[13][14]

Challenges of scientific software development for health

While the knowledge and technologies to develop effective and efficacious AI-driven clinical decision support systems exist, a manifold of pressing issues hinders further reuse in research and transfer to clinical practice. In software development projects in the industry, specialized developers typically work in large teams with considerable resources and are able to focus on usability and reusability.[15][16] In contrast, academic teams often consist of small groups of researchers (e.g., graduate students and postdoctoral scholars) who are typically not trained software engineers and only have temporary contracts and frequent turnover. Thus, individuals often develop software on a one-person-one-project basis.[13][16]

Moreover, funders and the academic hiring and promotion processes incentivize the pressure to publish and focus on “novelty” rather than software quality. Thus, it entices researchers to focus on theoretical aspects and proof of concept development.[13][17] Therefore, most researchers implement software in a prototype-centered manner, which often lacks quality checks such as systematic testing and may be published quickly.[18] However, these implementations often lack crucial qualities required for long-term reuse, such as documentation, usability, appropriate performance for real-life application, user-friendly interfaces, reusability, or minimized risk for potential users.[13][14]

Recently, scientific journals such as GigaScience or Biostatistics have promoted reproducibility and reusability by mandating FAIR principles (findability, accessibility, interoperability, and reusability). The concept of FAIR establishes a guideline for scientific data management and documentation.[13][19]

However, as recently surveyed by Pinto et al.[20], documentation of scientific software is one of the most significant “pain points.”[20] Journal publications are typically the primary source of documentation for scientific software and are quickly outdated by the agile software development style in academia.[16] Ideally, documentation should be detailed enough so that a developer with no prior knowledge of the project should make productive use of the software and use it for further development without biases or limitations.[13][18]

Another critical factor is accessibility. A lack of strict enforcement by journals, organizations, and funders has resulted in a loss of crucial data and software code.[13] According to an extensive analysis by Mangul et al., almost 28% of all resources linked in publications were not accessible, indicating poor maintenance. Moreover, a large proportion of software tested by Mangul et al. was not usable due to non-installability or lack of portability. The main inhibitors were storage locations outside of the journal’s directories or public versioning systems.[16][18]

Reproducibility and traceability are two of the most important aspects of biomedical and health informatics.[18][21] The lack of publicly available and comprehensible source code undermines the auditing of published methods and results. Additionally, the traceability of changes via version control is critical for reproducibility and reuse of research and code that replicators get to use.[18] These aspects ultimately undermine scientific rigor, transparency, and reproducibility.[13] The previously described accessibility, documentation, portability, and reusability factors are essential to ensure reproducibility and underpin trust in the scientific record of scientific software.[18]

Additionally, modern systems medicine approaches integrate all facets of private data such as electronic health records (EHR)[22], laboratory results[23], medical imaging[24], omics resources such as the cancer genome atlas (TCGA) or the gene expression omnibus[5][6][25][26], or pathway information.[4][6][7] However, sensitive patient data that enables an association of confidential personal information to single individuals underlies strict regulations such as the European General Data Protection Regulation (GDPR).[27] Therefore, the exchange within and among institutes is perceived as insurmountable, posing a roadblock hampering significant data-based medical innovations.

Quality management

The design and development of medical devices used in clinical practice, including software as a medical device, have to comply with regulations and standards that ensure the safety and quality of medical devices worldwide.[28][29] These regulations include standards concerning quality management (QM), the software development life cycle (including agile methods such as Scrum[30] or Kanban[31]), and risk management. However, such measures are not yet a standard in academic research work.[16][17][32]

The challenges in academic development procedures, as described before, affect technology transfer as well as reproducibility and reusability among researchers, as well as to the industry. It is usually restricted to transferring an intellectual method and know-how while discarding all created artifacts (e.g., documents for development planning and software code). In particular, manufacturers will need to understand the principles of the new method, train their staff members, and maintain design and development records when creating the final product. The record will include design documents, meeting minutes, and other documents produced during the development. This effort causes a significant overhead in the time and work they have to invest before the product is ready for the market.[33]

Introducing quality management systems (QMS) by industry has proven to guarantee continuous high quality among projects and procedures within an organization. Our goal is to facilitate this experience in academic environments. Establishing an academia-tailored QMS for research organizations can address the described challenges of software quality, documentation, accessibility, traceability, and reproducibility and help software development in research organizations reach higher quality and standards. These quality standards are particularly critical for software that is intended to be used in the medical context. We can significantly reduce software transfer barriers among researchers in academia and industry if these standards are established from the start.

Moreover, it has the potential to facilitate and speed up the transfer to clinical practice and thereby improve medical diagnosis or treatment of patients rapidly. However, our recommendations are based on standards that are common for medical device development but do not have the intention to shortcut any steps of the regular medical device development; this will always have to be done according to the strict rules and regulations that apply. Other benefits of a QMS for medical device development are also beneficial for research organizations when developing such software, including making responsibilities clearer, phasing in new team members quickly, and ensuring everyone knows how their work contributes to project and organizational goals.

Requirements for QM are defined in ISO 9001[34], in ISO 13485[35], and the Code of Federal Regulation (CFR) Title 21.[36] However, while ISO 9001 is generic and can be used in different fields, ISO 13485 is specific for QM in medical device development. The main differences are that ISO 9001 requires the organization to demonstrate continual improvement. In contrast, ISO 13485 only requires the certified organization to verify that the QMS is effectively implemented and maintained. Additionally, ISO 9001 includes customer satisfaction, which is not relevant to ISO 13485. There are also some other differences in ISO 13485, e.g., that the promotion and awareness of regulatory requirements is a management responsibility, that risk assessment and management need to be carried out (according to ISO 14971), and some others. Unfortunately, ISO 13485 is most likely outside of what is feasible for research groups in universities and other research organizations.

Our goal: Academia-tailored QMS

Until now, there has been no guidance on how to establish an academia-tailored QMS for research organizations that balances between not having any QMS at all and a wholly formulated QMS that complies with official regulatory standards. Here, we present a synopsis of relevant requirements in such standards and adjust these toward software development requirements in academia. We aim to embrace the benefits and facilitate technology transfer as much as possible while keeping the overhead for researchers and developers in a well manageable range. Thus, we tailor the content for use in universities or similar research organizations that focus on software in the medical setting.

However, the ideas presented here do not provide a shortcut when developing medical software. Any development of medical software must strictly follow the relevant regulations in the specific country or region. Therefore, our suggestions provide a starting point intended to be adapted to specific requirements as needed and we refer to it as MDx-ready (i.e., ready for transferring it into molecular diagnostics SaMD implementations). This flexibility is well in the mindset of most QMS: If there is a profound reason for the change, change it, but consider all relevant aspects and document the reasons and the changes.

QMS for medical software research

A vital aspect to be clarified when considering QM is the scope of the activities. In principle, QM is set up on an organizational rather than a project level. One of the key benefits of a QMS is that it provides projects with guiding principles for the setup, documentation, and more. The organization sets up the QMS, and the projects (and other procedures in the organization) use this QMS to ensure organization-wide quality and standards.

Universities and other research organizations often have a general QMS for the entire organization, primarily focusing on the quality of research and education. They cover topics such as hiring, waste disposal, publishing guidelines, and commitment to scientific standards. However, because of the range of specific subjects a QMS for medical software development needs to cover, the standard university regulations are typically neither helpful nor sufficient. A university-wide QMS lacks the particular requirements necessary in a customer-centered setting, such as for medical software development and defined in ISO 13485.

Traditionally, a company or organization defines a QMS to be applied in each project. In the academic setting, we recommend setting up a QMS, for instance, at the level of a research institute itself, an organizational unit such as a department, a sub-unit such as a research group, or agile organizations such as scientific consortia. For clarity, in this manuscript, we will use the general terms "organization," "institute," and "unit" interchangeably. Which level is most appropriate may differ on a case-by-case basis. However, it is vital to make a dedicated decision before starting to set up such quality guidelines. To give a practical demonstration of an academic QMS implementation, we provide an example on the level of a research organization (see Document S1, Supplemental information).

Content of the QMS

The QMS of an organization or research unit defines procedures and structures that are implemented in the organization to be seen and understood by everyone. It describes the basic structure of an organizational unit and defines the procedures that are implemented in the organization. Procedures in this context may be administrative processes, such as the employee training process, or technical procedures, such as storage and archiving processes. The QMS documents inputs, outputs, responsibilities, and activities for each procedure (see Figure 1).


Fig1 Hauschild iScience2021 24-7.jpg

Figure 1. Main components and organization of a quality management system

For an organization that works on research projects, the QMS should guide the execution of projects. In particular, it should include the basic structure, documentation, technical setup, responsibilities, and roles within projects. The guidance and policies of an organizational QMS ensure a controlled and defined project execution.

However, in a research environment, a well-designed QMS should be a guide, not a corset. Thus, the QMS should allow a significant degree of freedom to deviate from the recommended processes. Deviations are required to be documented, including a detailed justification and description of the alternative path chosen. It allows adaptations to different conditions while maintaining a solid level of traceability and reproducibility. Moreover, procedures necessary for a qualitative implementation and documentation that persons involved in a project commit to need to be described. Therefore, the QMS addresses all personnel involved in a project. Ultimately, by participating in the project, they commit themselves to comply with the procedures and regulations defined in the QMS.

Quality policies and objectives

International standards for QMS for medical devices, such as ISO 13485, require documenting an organization’s quality policy and quality objectives on the highest level. Clear quality policies and quality objectives will help understand what drives the organization and how it intends to reach its quality goals.[35]

A quality policy describes the high-level philosophy of the organization concerning quality. A quality objective is a measurable goal that is derived from quality policies. Quality objectives and policies are ultimately what distinguishes an organization from other organizations and what drives the organization on a high level. Personnel must be aware of the relevance and importance of their activities and how they contribute. The quality policies condense the overall goals, allowing staff members to align their work with these overarching goals.

Moreover, a research organization implementing a limited QMS should declare in the quality policy to what extent the execution of the QMS is mandatory (policy) for all projects and how it intends to enforce this decision (objective). In contrast to manufacturers of medical device software, a full QMS implementation and execution is not mandatory for research organizations in this article’s scope. However, they should deliberately choose whether and to what extent they use a QMS. Document S1, Section 1, Supplemental information gives example descriptions for “quality policies" and "quality objectives.”

Quality manual

The quality manual documents the core of the QMS. It defines the scope of the QMS and describes procedures and structures in the organization. We can consider it the “instruction handbook” of the QMS. Ideally, it should be possible to hand a copy of the quality manual to a new employee in the organization, enabling that person to work according to the QMS (in reality, this is typically not possible due to the size and complexity of a QMS).

Independent of the complexity and intended environment a QMS is created for, it should always contain a quality manual. The document comprises, for instance, information on:

  • the scope of the QMS (For whom is it relevant? Which procedures are covered?);
  • the structure of the QMS (One document? Multiple documents? How are multiple documents organized?); and
  • how and where the procedures of the QMS are stored, maintained, and archived.

Typically, a quality manual does not contain all the descriptions of all covered procedures, but includes references to other documents containing these descriptions. This structure makes it much easier to find and read the description of specific procedures. Also, it allows updating the definition of parts of the QMS rather than the entire QMS at once. An example “quality manual” can be found in Document S1, Section 2, Supplemental information.

Procedures to be documented

This article aims to give guidance on what subset of a QMS is helpful to implement to facilitate technology transfer from research to a manufacturer of medical devices, as well as for supporting reproducibility and reusability. So far, we have discussed the general aspects of a QMS, the following sections list, and briefly described which procedures we consider to be most important in this context. Established standards for QMS require a significantly more extensive set of procedures, and there are always good arguments for including or excluding a particular set of procedures.[34][35] We consider our recommendation (MDx-ready) as a starting point that can, and should, be adapted if there is any good reason to do so.

Document management

One of the most important things to aid the technology transfer to move forward from research to an industrial, full-blown medical product is documentation. It helps tremendously to have reliable, well-defined, structured, and (within the given possibilities) complete documentation.

The most fundamental aspect of document management is to define where and how documents are stored and managed. Different document storage systems are suitable; for instance, a file system, in a source code management system, such as git or subversion[37][38], in a content management system, or a dedicated document management system. If all projects separately decide how to store documents, managing and retrieving documents on an organizational level will be much more difficult. This is especially true considering the fast turnover of employees typical in research-oriented groups. Therefore, it is advisable to use the same document storage for all documents relevant to the organization and document it in the QMS. The documents that make up the QMS should be stored accordingly. Thus, the decision on a storage system should precede the development of the QMS.

Additional items that this part of a QMS should cover are, for instance:

  • How are documents identified consistently and unambiguously?
  • How can users access and modify documents?
  • How will previous versions of a document be available?
  • Is any approval needed when creating/storing a new document or a new version?
  • If permission is required, who needs to approve, and how is this done?
  • How can a user be sure to have the current version of a document?
  • Will documents be discarded at any time?

The set of procedures that cover these topics and others relevant to handling documents determines the document management section of the QMS. This section may also include the choice of a specific format or software for the work with documents.

We can distinguish between “documents” and “records,” where “documents” describe what is planned and “records” describe what happened. Documented procedures may support changes and updates, while records should never be altered after they were stored. For the sake of simplicity, we do not distinguish the two in the scope of this article. Document S1, Section 3, Supplemental information describes the procedures to be documented for the example QMS.

Project planning

Having some project planning is tremendously helpful, even in a research-oriented organization. It not only dramatically increases the likelihood of successful project completion, but it also makes it much easier to understand what happened in a project and why it happened, which in turn supports technology transfer significantly.

Therefore, we propose to define a set of procedures on how to handle project planning as part of a QMS proposal. However, not every tiny project has to implement comprehensive project planning. It only clarifies when to do which kind of planning and sets guidelines for the implementation.

Essential elements to be covered here include:

  • a procedure to systematically choose which parts of the QMS will be applied to the project (might be a limited subset for small projects);
  • a procedure that requires projects to check if everyone has the same understanding of the project goals;
  • a procedure that requires projects to define the roles and responsibilities within the project;
  • a set of documents that needs to be created for each project; and
  • a special focus on risk assessment and management (according to ISO 14971).

An example of project planning documentation can be found in Document S1, Section 4, Supplemental information.

Project execution

Every project is subject to specific circumstances, boundary conditions, internal and external requirements, and so on, leading to variability in execution. While this may or may not be beneficial for each project, it usually makes technology transfer, reproducibility, and reusability much more complicated; as part of the transfer, all of the specific setups for the project need to be understood. Having some level of standardization for all projects in an organization and having this standardization documented consistently will make the transfer, reproducibility, and reusability much more manageable. Besides this, it also provides many benefits that may not be visible to the individual projects at the time of start; having a higher chance of avoiding pitfalls using experience from previous similar setups is one of these. Therefore, we recommend defining procedures for project execution as part of our QMS for research organizations. More specifically, we recommend at least defining procedures for the following activities.

Requirements management: Each project should at least put thought into which inputs to consider for requirements, prioritize requirements, document requirements, and handle changes in requirements. A set of procedures that directs projects on how to spend these thoughts in a more formalized way should be part of the QMS.

Definition of the development environment and tools: It is probably not helpful to have all projects use the same development environment and set of tools. But it is beneficial to require the projects to decide what to use in a structured way and document this decision. Include the procedures on how to make the decision and what to document in the QMS.

Development process: As before, the most critical point is that selecting a specific development process (e.g., Scrum, Kanban, or Waterfall) is done consciously and that the reasons for the decision are documented.

Completion of the project: It must be clear when a project is completed. Again, this may depend strongly on the individual project. Some are complete after successful testing, others simply when time is over. The important thing is that everyone has the same understanding of when this is the case. Therefore, there should be a procedure that requires each project to define the criteria for completion and document this decision.

Documentation: The level of documentation created during the development (code comments? design document? test documentation?) may be subject to different needs. However, it is good to have firm guidelines here, as documentation is typically an unpopular and often neglected activity.

An example of project execution documentation and procedures can be found in Document S1, Section 5, Supplemental information.

Management

The success and quality of a venture or project primarily relies on the competencies of the contributing scientific and administrative personnel and their access to resources. Therefore, it is inevitable to provide them with the necessary know-how and competencies through appropriate education, training, skills, and experience processes. To ensure the consistent quality of all projects within an organization, the corresponding QMS should document such processes. More specifically, we recommend at least defining procedures for QMS training and qualification of personnel.

QMS training: The effort put into establishing a QMS is useless unless the staff members involved in the projects and the administration is trained on the QMS. It is essential that every person is aware of the QM system, knows where to find the QMS documents associated with their work, and knows how to act accordingly. Therefore, it should be documented how the management will ensure appropriate QMS training for new employees and initiate regular update training for all personnel when the QMS is changed. Moreover, it is essential to determine who is responsible for ensuring that every employee has the required QMS competencies and how training is implemented. The training will ensure that project teams are implementing a QMS in practice. However, while according to QM system standards like ISO 13485, such training processes require meticulous documentation of participation for each employee[35], we suggest that this is not mandatory in the academic environment.

Qualification of personnel: The management is responsible for ensuring that all staff members are qualified for their responsibilities. The process may include hiring staff members according to the requirements of the project and providing necessary training. The QMS should define corresponding procedures and responsibilities.

An example of management documentation and procedures can be found in Document S1, Section 6, Supplemental information.

Discussion

Computational and data science approaches, such as AI or ML, have opened a new era in health care and precision medicine such as diagnostics, prognostics, and monitoring. However, the development of such biomedical and health algorithms is very time-consuming. Moreover, transferring the knowledge and technology to clinical practice requires implementing all quality requirements defined in international standards.

Here, we discussed the difficulties of academic technology transfer, reproducibility, and reusability within research and from research to commercial manufacturing, with a particular focus on software that is intended to be used in a biomedical and clinical context. Therefore, the well-described challenges of documentation, traceability, transparency, accessibility, replicability, and reusability affecting software quality need addressing.[13][17][18]

To the best of our knowledge, we propose the first guidelines establishing an academia-tailored QMS for research organizations and units, which can significantly facilitate reproducibility and reusability of scientific software and speed up technology transfer in a controlled and predictable way. Only a few research organizations have thus far started to implement such processes.[39] The developed guidelines intend not to simplify the certification process but to support the manufacturer with gathering all required documentation for the certification process from the academic partners (MDx-ready). While a regulation-conforming QMS is not feasible for most research organizations, we proposed a checklist of selected elements of a QMS that will significantly benefit software quality and reusability while keeping the effort in a range that most organizations can easily handle. Thus, our proposal focuses on document management, project planning, execution, and the surrounding administrative procedures. However, depending on an organization’s specific needs, the set of elements in the QMS may need adjustment and differ from our suggestions.

Nevertheless, a commitment that a QMS is implemented and that elements are deliberately chosen to fit the organization is already a key factor in improving reproducibility and reusability, and facilitating technology transfer. In particular, community-wide adoption of such best practices for reproducibility would be critical to exploit the full potential of agile software development in the biomedical and life sciences.[13] Ultimately, to break the circle of constantly reinventing the wheel in academic software development, the reuse of scientific software is inevitable.[18] Our proposal provides a starting point for this, lowering the hurdle for research organizations to set up quality management.

Limitations of study

This article presents a guideline for QMS for academic organizations incorporating only the essential components while confining the requirements to an easily manageable effort. However, direct implementation of the entirety of ISO 13485 is most likely outside of what is feasible in universities and other organizations. The ideas presented here do not provide a shortcut when developing medical software. Any development of medical software must strictly follow the relevant regulations in the specific country or region. Therefore, our suggestions provide a starting point intended to be adapted to specific requirements as needed. We refer to it as "MDx-ready," which we define as "ready for transferring it into molecular diagnostics SaMD implementations."

Supplemental information

  • Document S1 (PDF). Includes Figure S1, Tables S1–S4, and Methods.

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 826078. This publication reflects only the authors’ view, and the European Commission is not responsible for any use that may be made of the information it contains.

Author contributions

All authors conceptualized the structure for the implementation guideline for a QMS in academia. A.-C.H.,L.E., and J.W. developed the guideline, designed the example, and wrote the manuscript. D.H. supervised the study and revised the manuscript.

Methods

All methods can be found in the accompanying transparent methods supplemental file.

Conflict of interest

The authors declare no competing interests.

References

  1. ↑ Hawgood, Sam; Hook-Barnard, India G.; O’Brien, Theresa C.; Yamamoto, Keith R. (12 August 2015). "Precision medicine: Beyond the inflection point" (in en). Science Translational Medicine 7 (300): 300ps17–300ps17. doi:10.1126/scitranslmed.aaa9970. ISSN 1946-6234. https://stm.sciencemag.org/lookup/doi/10.1126/scitranslmed.aaa9970. 
  2. ↑ Digital Health Center of Excellence (6 December 2017). "What are examples of Software as a Medical Device?". U.S. Food and Drug Administration. https://www.fda.gov/medical-devices/software-medical-device-samd/what-are-examples-software-medical-device. 
  3. ↑ Fatima, Meherwar; Pasha, Maruf (2017). "Survey of Machine Learning Algorithms for Disease Diagnostic" (in en). Journal of Intelligent Learning Systems and Applications 09 (01): 1. doi:10.4236/jilsa.2017.91001. http://www.scirp.org/journal/PaperInformation.aspx?PaperID=73781&#abstract. 
  4. ↑ 4.0 4.1 Batra, Richa; Alcaraz, Nicolas; Gitzhofer, Kevin; Pauling, Josch; Ditzel, Henrik J.; Hellmuth, Marc; Baumbach, Jan; List, Markus (1 December 2017). "On the performance of de novo pathway enrichment" (in en). npj Systems Biology and Applications 3 (1): 6. doi:10.1038/s41540-017-0007-2. ISSN 2056-7189. PMC PMC5445589. PMID 28649433. http://www.nature.com/articles/s41540-017-0007-2. 
  5. ↑ 5.0 5.1 Hauschild, A.-C.; Baumbach, J.I.; Baumbach, J. (2012). "Integrated statistical learning of metabolic ion mobility spectrometry profiles for pulmonary disease identification". Genetics and Molecular Research 11 (3): 2733–2744. doi:10.4238/2012.July.10.17. http://www.funpecrp.com.br/gmr/year2012/vol11-3/pdf/gmr2065.pdf. 
  6. ↑ 6.0 6.1 6.2 Jeanquartier, Fleur; Jean-Quartier, Claire; Kotlyar, Max; Tokar, Tomas; Hauschild, Anne-Christin; Jurisica, Igor; Holzinger, Andreas (2016), Holzinger, Andreas, ed., "Machine Learning for In Silico Modeling of Tumor Growth" (in en), Machine Learning for Health Informatics (Cham: Springer International Publishing) 9605: 415–434, doi:10.1007/978-3-319-50478-0_21, ISBN 978-3-319-50477-3, http://link.springer.com/10.1007/978-3-319-50478-0_21. Retrieved 2021-08-22 
  7. ↑ 7.0 7.1 Wiwie, Christian; Kuznetsova, Irina; Mostafa, Ahmed; Rauch, Alexander; Haakonsson, Anders; Barrio-Hernandez, Inigo; Blagoev, Blagoy; Mandrup, Susanne et al. (1 May 2019). "Time-Resolved Systems Medicine Reveals Viral Infection-Modulating Host Targets" (in en). Systems Medicine 2 (1): 1–9. doi:10.1089/sysm.2018.0013. ISSN 2573-3370. PMC PMC6524659. PMID 31119214. https://www.liebertpub.com/doi/10.1089/sysm.2018.0013. 
  8. ↑ Calvert, Jacob; Saber, Nicholas; Hoffman, Jana; Das, Ritankar (13 February 2019). "Machine-Learning-Based Laboratory Developed Test for the Diagnosis of Sepsis in High-Risk Patients" (in en). Diagnostics 9 (1): 20. doi:10.3390/diagnostics9010020. ISSN 2075-4418. PMC PMC6468682. PMID 30781800. http://www.mdpi.com/2075-4418/9/1/20. 
  9. ↑ Desautels, Thomas; Calvert, Jacob; Hoffman, Jana; Jay, Melissa; Kerem, Yaniv; Shieh, Lisa; Shimabukuro, David; Chettipally, Uli et al. (30 September 2016). "Prediction of Sepsis in the Intensive Care Unit With Minimal Electronic Health Record Data: A Machine Learning Approach" (in en). JMIR Medical Informatics 4 (3): e28. doi:10.2196/medinform.5909. ISSN 2291-9694. PMC PMC5065680. PMID 27694098. https://medinform.jmir.org/2016/3/e28/. 
  10. ↑ Chen, Robert; Stewart, Walter F.; Sun, Jimeng; Ng, Kenney; Yan, Xiaowei (1 October 2019). "Recurrent Neural Networks for Early Detection of Heart Failure From Longitudinal Electronic Health Record Data: Implications for Temporal Modeling With Respect to Time Before Diagnosis, Data Density, Data Quantity, and Data Type" (in en). Circulation: Cardiovascular Quality and Outcomes 12 (10). doi:10.1161/CIRCOUTCOMES.118.005114. ISSN 1941-7713. PMC PMC6814386. PMID 31610714. https://www.ahajournals.org/doi/10.1161/CIRCOUTCOMES.118.005114. 
  11. ↑ Heider, Dominik; Dybowski, Jan Nikolaj; Wilms, Christoph; Hoffmann, Daniel (1 December 2014). "A simple structure-based model for the prediction of HIV-1 co-receptor tropism" (in en). BioData Mining 7 (1): 14. doi:10.1186/1756-0381-7-14. ISSN 1756-0381. PMC PMC4124776. PMID 25120583. https://biodatamining.biomedcentral.com/articles/10.1186/1756-0381-7-14. 
  12. ↑ Riemenschneider, Mona; Hummel, Thomas; Heider, Dominik (1 December 2016). "SHIVA - a web application for drug resistance and tropism testing in HIV" (in en). BMC Bioinformatics 17 (1): 314. doi:10.1186/s12859-016-1179-2. ISSN 1471-2105. PMC PMC4994198. PMID 27549230. http://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1179-2. 
  13. ↑ 13.0 13.1 13.2 13.3 13.4 13.5 13.6 13.7 13.8 13.9 Brito, Jaqueline J; Li, Jun; Moore, Jason H; Greene, Casey S; Nogoy, Nicole A; Garmire, Lana X; Mangul, Serghei (1 June 2020). "Recommendations to enhance rigor and reproducibility in biomedical research" (in en). GigaScience 9 (6): giaa056. doi:10.1093/gigascience/giaa056. ISSN 2047-217X. PMC PMC7263079. PMID 32479592. https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giaa056/5849489. 
  14. ↑ 14.0 14.1 Riemenschneider, Mona; Wienbeck, Joachim; Scherag, André; Heider, Dominik (1 June 2018). "Data Science for Molecular Diagnostics Applications: From Academia to Clinic to Industry" (in en). Systems Medicine 1 (1): 13–17. doi:10.1089/sysm.2018.0002. ISSN 2573-3370. http://www.liebertpub.com/doi/10.1089/sysm.2018.0002. 
  15. ↑ Guellec, D.; van Pottelsberghe de la Potterie, B. (14 June 2000). "The Impact of Public R&D Expenditure on Business R&D" (in en). OECD Science, Technology and Industry Working Papers 2000/04. doi:10.1787/670385851815. https://www.oecd-ilibrary.org/science-and-technology/the-impact-of-public-r-d-expenditure-on-business-r-d_670385851815. 
  16. ↑ 16.0 16.1 16.2 16.3 16.4 Mangul, Serghei; Mosqueiro, Thiago; Abdill, Richard J.; Duong, Dat; Mitchell, Keith; Sarwal, Varuni; Hill, Brian; Brito, Jaqueline et al. (20 June 2019). "Challenges and recommendations to improve the installability and archival stability of omics computational tools" (in en). PLOS Biology 17 (6): e3000333. doi:10.1371/journal.pbio.3000333. ISSN 1545-7885. PMC PMC6605654. PMID 31220077. https://dx.plos.org/10.1371/journal.pbio.3000333. 
  17. ↑ 17.0 17.1 17.2 Mangul, Serghei; Martin, Lana S.; Eskin, Eleazar; Blekhman, Ran (1 December 2019). "Improving the usability and archival stability of bioinformatics software" (in en). Genome Biology 20 (1): 47, s13059–019–1649-8. doi:10.1186/s13059-019-1649-8. ISSN 1474-760X. PMC PMC6391762. PMID 30813962. https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1649-8. 
  18. ↑ 18.0 18.1 18.2 18.3 18.4 18.5 18.6 18.7 Lee, Graham; Bacon, Sebastian; Bush, Ian; Fortunato, Laura; Gavaghan, David; Lestang, Thibault; Morton, Caroline; Robinson, Martin et al. (1 February 2021). "Barely sufficient practices in scientific computing" (in en). Patterns 2 (2): 100206. doi:10.1016/j.patter.2021.100206. PMC PMC7892476. PMID 33659915. https://linkinghub.elsevier.com/retrieve/pii/S2666389921000167. 
  19. ↑ Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem et al. (1 December 2016). "The FAIR Guiding Principles for scientific data management and stewardship" (in en). Scientific Data 3 (1): 160018. doi:10.1038/sdata.2016.18. ISSN 2052-4463. PMC PMC4792175. PMID 26978244. http://www.nature.com/articles/sdata201618. 
  20. ↑ 20.0 20.1 Pinto, Gustavo; Wiese, Igor; Dias, Luiz Felipe (1 March 2018). "How do scientists develop scientific software? An external replication". 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER) (Campobasso: IEEE): 582–591. doi:10.1109/SANER.2018.8330263. ISBN 978-1-5386-4969-5. http://ieeexplore.ieee.org/document/8330263/. 
  21. ↑ Coiera, Enrico; Ammenwerth, Elske; Georgiou, Andrew; Magrabi, Farah (1 August 2018). "Does health informatics have a replication crisis?". Journal of the American Medical Informatics Association 25 (8): 963–968. doi:10.1093/jamia/ocy028. ISSN 1527-974X. PMC PMC6077781. PMID 29669066. https://doi.org/10.1093/jamia/ocy028. 
  22. ↑ Shickel, Benjamin; Tighe, Patrick James; Bihorac, Azra; Rashidi, Parisa (1 September 2018). "Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis". IEEE Journal of Biomedical and Health Informatics 22 (5): 1589–1604. doi:10.1109/JBHI.2017.2767063. ISSN 2168-2194. PMC PMC6043423. PMID 29989977. https://ieeexplore.ieee.org/document/8086133/. 
  23. ↑ Goecks, Jeremy; Jalili, Vahid; Heiser, Laura M.; Gray, Joe W. (1 April 2020). "How Machine Learning Will Transform Biomedicine" (in en). Cell 181 (1): 92–101. doi:10.1016/j.cell.2020.03.022. PMC PMC7141410. PMID 32243801. https://linkinghub.elsevier.com/retrieve/pii/S0092867420302841. 
  24. ↑ Anwar, Syed Muhammad; Majid, Muhammad; Qayyum, Adnan; Awais, Muhammad; Alnowami, Majdi; Khan, Muhammad Khurram (1 November 2018). "Medical Image Analysis using Convolutional Neural Networks: A Review" (in en). Journal of Medical Systems 42 (11): 226. doi:10.1007/s10916-018-1088-1. ISSN 0148-5598. http://link.springer.com/10.1007/s10916-018-1088-1. 
  25. ↑ Clough, Emily; Barrett, Tanya (2016), Mathé, Ewy; Davis, Sean, eds., "The Gene Expression Omnibus Database", Statistical Genomics (New York, NY: Springer New York) 1418: 93–110, doi:10.1007/978-1-4939-3578-9_5, ISBN 978-1-4939-3576-5, PMC PMC4944384, PMID 27008011, http://link.springer.com/10.1007/978-1-4939-3578-9_5. Retrieved 2021-08-22 
  26. ↑ Tomczak, Katarzyna; Czerwińska, Patrycja; Wiznerowicz, Maciej (2015). "The Cancer Genome Atlas (TCGA): An immeasurable source of knowledge". Współczesna Onkologia 1A: 68–77. doi:10.5114/wo.2014.47136. ISSN 1428-2526. PMC PMC4322527. PMID 25691825. http://www.termedia.pl/doi/10.5114/wo.2014.47136. 
  27. ↑ Voigt, Paul; von dem Bussche, Axel (2017) (in en). The EU General Data Protection Regulation (GDPR). Cham: Springer International Publishing. doi:10.1007/978-3-319-57959-7. ISBN 978-3-319-57958-0. http://link.springer.com/10.1007/978-3-319-57959-7. 
  28. ↑ Maak, Travis G.; Wylie, James D. (1 August 2016). "Medical Device Regulation: A Comparison of the United States and the European Union" (in en). Journal of the American Academy of Orthopaedic Surgeons 24 (8): 537–543. doi:10.5435/JAAOS-D-15-00403. ISSN 1067-151X. http://journals.lww.com/00124635-201608000-00004. 
  29. ↑ World Health Organization, ed. (2011). Development of medical device policies. WHO medical device technical series. Geneva, Switzerland: World Health Organization. ISBN 978-92-4-150163-7. 
  30. ↑ Schwaber, Ken; Beedle, Mike (2002). Agile software development with Scrum. Series in agile software development. Upper Saddle River, NJ: Prentice Hall. ISBN 978-0-13-067634-4. 
  31. ↑ Anderson, David J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, Washington: Blue Hole Press. ISBN 978-0-9845214-0-1. OCLC 699604917. https://www.worldcat.org/title/mediawiki/oclc/699604917. 
  32. ↑ Zamith, Manuel; Gonçalves, Gil (2018). "Towards an Agile Development Model for Certifiable Medical Device Software - Taking Advantage of the Medical Device Regulation:". Proceedings of the 13th International Conference on Software Technologies (Porto, Portugal: SCITEPRESS - Science and Technology Publications): 132–140. doi:10.5220/0006861301320140. ISBN 978-989-758-320-9. http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006861301320140. 
  33. ↑ Sharma, Arjun; Blank, Anthony; Patel, Parashar; Stein, Kenneth (1 March 2013). "Health care policy and regulatory implications on medical device innovations: A cardiac rhythm medical device industry perspective" (in en). Journal of Interventional Cardiac Electrophysiology 36 (2): 107–117. doi:10.1007/s10840-013-9781-y. ISSN 1383-875X. PMC PMC3606523. PMID 23474980. http://link.springer.com/10.1007/s10840-013-9781-y. 
  34. ↑ 34.0 34.1 "ISO 9001:2015 Quality management systems — Requirements". International Organization for Standardization. September 2015. https://www.iso.org/standard/62085.html. 
  35. ↑ 35.0 35.1 35.2 35.3 "ISO 13485:2016 Medical devices — Quality management systems — Requirements for regulatory purposes". International Organization for Standardization. March 2016. https://www.iso.org/standard/59752.html. 
  36. ↑ "CFR - Code of Federal Regulations Title 21". U.S. Food and Drug Administration. 1 April 2020. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm. 
  37. ↑ Collins-Sussman, B. (2002). "The subversion project: Building a better CVS". Linux Journal 2002 (94). doi:10.5555/513039.513042. https://dl.acm.org/doi/abs/10.5555/513039.513042. 
  38. ↑ Spinellis, Diomidis (1 May 2012). "Git". IEEE Software 29 (3): 100–101. doi:10.1109/MS.2012.61. ISSN 1937-4194. https://ieeexplore.ieee.org/document/6188603/. 
  39. ↑ Sapunar, Damir; Grković, Ivica; Lukšić, Davor; Marušić, Matko (23 May 2016). "The business process management software for successful quality management and organization: A case study from the University of Split School of Medicine" (in en). Acta Medica Academica 45 (1): 26. doi:10.5644/ama2006-124.153. ISSN 1840-2879. http://ama.ba/index.php/ama/article/view/268. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version—by design—lists them in order of appearance.

source: https://www.limswiki.org/index.php/Journal:Fostering_reproducibility,_reusability,_and_technology_transfer_in_health_informatics